This was a great post, and I appreciated the comments towards the end about the train to Crazy Town, like “Stops along the train to Crazy Town simply represent the place where we see the practical limits to a certain kind of quasi-formal reasoning.” In my own (but noticeably Yudkowsky-derived) words I was thinking that this applies in areas where “the probabilities are there to make clear our beliefs and values. If, on reflection, you find that the probabilities advocate for something you do not believe or value, then the model you built your probabilities on don’t capture your beliefs/values well enough.”
This is noticeably narrower than the AI x-risk prediction case, where I see the beliefs about possible/relevant models to be less clustered than beliefs about the set of human beliefs. And now I’m noticing that even here I might be trapped inside the Bayesian Mindset, as the previous sentence is basically a statement of credences over the spread of those sets of beliefs.
Have you had a chance to read Vanessa Kosoy and Diffractor’s work on Infra-Bayesianism? From what I can tell (I haven’t spent much time engaging with it myself yet, but), it’s very relevant here, as it wants to be a theory of learning that applies “when the hypothesis is not included in the hypothesis space”. Among other things, they talk about infradistributions: a set of probability distributions which would act like a prior over environments.
I haven’t read Kosoy & Diffractor’s stuff, but I will now!
FWIW I’m pretty skeptical that their framework will be helpful for making progress in practical epistemology (which I gather is not their main focus anyway?). That said, I’d be very happy to learn that I’m wrong here, so I’ll put some time into understanding what their approach is.
This was a great post, and I appreciated the comments towards the end about the train to Crazy Town, like “Stops along the train to Crazy Town simply represent the place where we see the practical limits to a certain kind of quasi-formal reasoning.” In my own (but noticeably Yudkowsky-derived) words I was thinking that this applies in areas where “the probabilities are there to make clear our beliefs and values. If, on reflection, you find that the probabilities advocate for something you do not believe or value, then the model you built your probabilities on don’t capture your beliefs/values well enough.”
This is noticeably narrower than the AI x-risk prediction case, where I see the beliefs about possible/relevant models to be less clustered than beliefs about the set of human beliefs. And now I’m noticing that even here I might be trapped inside the Bayesian Mindset, as the previous sentence is basically a statement of credences over the spread of those sets of beliefs.
Have you had a chance to read Vanessa Kosoy and Diffractor’s work on Infra-Bayesianism? From what I can tell (I haven’t spent much time engaging with it myself yet, but), it’s very relevant here, as it wants to be a theory of learning that applies “when the hypothesis is not included in the hypothesis space”. Among other things, they talk about infradistributions: a set of probability distributions which would act like a prior over environments.
I haven’t read Kosoy & Diffractor’s stuff, but I will now!
FWIW I’m pretty skeptical that their framework will be helpful for making progress in practical epistemology (which I gather is not their main focus anyway?). That said, I’d be very happy to learn that I’m wrong here, so I’ll put some time into understanding what their approach is.