This was a great post, and I appreciated the comments towards the end about the train to Crazy Town, like “Stops along the train to Crazy Town simply represent the place where we see the practical limits to a certain kind of quasi-formal reasoning.” In my own (but noticeably Yudkowsky-derived) words I was thinking that this applies in areas where “the probabilities are there to make clear our beliefs and values. If, on reflection, you find that the probabilities advocate for something you do not believe or value, then the model you built your probabilities on don’t capture your beliefs/values well enough.”
This is noticeably narrower than the AI x-risk prediction case, where I see the beliefs about possible/relevant models to be less clustered than beliefs about the set of human beliefs. And now I’m noticing that even here I might be trapped inside the Bayesian Mindset, as the previous sentence is basically a statement of credences over the spread of those sets of beliefs.
Have you had a chance to read Vanessa Kosoy and Diffractor’s work on Infra-Bayesianism? From what I can tell (I haven’t spent much time engaging with it myself yet, but), it’s very relevant here, as it wants to be a theory of learning that applies “when the hypothesis is not included in the hypothesis space”. Among other things, they talk about infradistributions: a set of probability distributions which would act like a prior over environments.
Yeah, I’m starting to believe that a severe limitation on Brier scores is this inability to use them in a forward-looking way. Brier scores reflect the performance of specific people on specific questions and using them as evidence for future prediction performance seems really fraught...but it’s the best we have as far as I can tell.