Fantastic post, puts into words a lot of my misgivings with how bayesianism is currently practised here.
I am unconvinced of the most important idea of the longtermist “bayesian mindset”: that good predictions in evidence rich, well established fields are transferable to predictions in highly speculative, uncertain fields. I am skeptical that being able to answer questions like “will the EU pass an AI act in the next year” is particularly highly correlated with correctly answering questions like “will a hypothetical future AGI defeat humanity in a hypothetical war”.
It seems to me like people can do well on the evidence questions because they are not super hard, and they can receive fairly quick feedback on errors. (For example, calibration is very hard to do on probabilities that are significantly less than 1% or more than 99%, so these type of questions are rarely tested). But when the plausibly correct answer for something varies by many order of magnitude, it seems fairly easy for groupthink and bias to seep in, because there are is not enough evidence to correct them.
This is not to say I don’t count good prediction as a point in someones favour, just that there are other factors I value more highly, like domain-relevant specific accomplishments and skills.
I’m sympathetic to the view that calibration on questions with larger bodies of obviously relevant evidence aren’t transferable to predictions on more speculative questions. Ultimately I believe that the amount of skill transfer is an open empirical question, though I think the absence of strong theorizing about the relevant mechanisms involved heavily counts against deferring to (e.g.) Metaculus predictions about AI timelines.
A potential note of disagreement on your final sentence. While I think focusing on calibration can Goodhart us away from some of the most important sources of epistemic insight, there are “predictions” (broadly construed) that I think we ought to weigh more highly than “domain-relevant specific accomplishments and skills”.
E.g., if you’re sympathetic to EA’s current focus on AI, then I think it’s sensible to think “oh, maybe Yudkowsky was onto something”, and upweight the degree to which you should engage in detail with his worldview, and potentially defer to the extent that you don’t possess a theory which jointly explains both his foresight and the errors you currently think he’s making.
My objection to ‘Bayesian Mindset’ and the use of subjective probabilities to communicate uncertainty is (in part) due to the picture imposed by the probabilistic mode of thinking, which is something like “you have a clear set of well-identified hypotheses, and the primary epistemic task is to become calibrated on such questions.” This leads me to suspect that EAs are undervaluing the ‘novel hypotheses generation’ component of predictions, though there is still a lot of value to be had from (novel) predictions.
Fantastic post, puts into words a lot of my misgivings with how bayesianism is currently practised here.
I am unconvinced of the most important idea of the longtermist “bayesian mindset”: that good predictions in evidence rich, well established fields are transferable to predictions in highly speculative, uncertain fields. I am skeptical that being able to answer questions like “will the EU pass an AI act in the next year” is particularly highly correlated with correctly answering questions like “will a hypothetical future AGI defeat humanity in a hypothetical war”.
It seems to me like people can do well on the evidence questions because they are not super hard, and they can receive fairly quick feedback on errors. (For example, calibration is very hard to do on probabilities that are significantly less than 1% or more than 99%, so these type of questions are rarely tested). But when the plausibly correct answer for something varies by many order of magnitude, it seems fairly easy for groupthink and bias to seep in, because there are is not enough evidence to correct them.
This is not to say I don’t count good prediction as a point in someones favour, just that there are other factors I value more highly, like domain-relevant specific accomplishments and skills.
Thanks :)
I’m sympathetic to the view that calibration on questions with larger bodies of obviously relevant evidence aren’t transferable to predictions on more speculative questions. Ultimately I believe that the amount of skill transfer is an open empirical question, though I think the absence of strong theorizing about the relevant mechanisms involved heavily counts against deferring to (e.g.) Metaculus predictions about AI timelines.
A potential note of disagreement on your final sentence. While I think focusing on calibration can Goodhart us away from some of the most important sources of epistemic insight, there are “predictions” (broadly construed) that I think we ought to weigh more highly than “domain-relevant specific accomplishments and skills”.
E.g., if you’re sympathetic to EA’s current focus on AI, then I think it’s sensible to think “oh, maybe Yudkowsky was onto something”, and upweight the degree to which you should engage in detail with his worldview, and potentially defer to the extent that you don’t possess a theory which jointly explains both his foresight and the errors you currently think he’s making.
My objection to ‘Bayesian Mindset’ and the use of subjective probabilities to communicate uncertainty is (in part) due to the picture imposed by the probabilistic mode of thinking, which is something like “you have a clear set of well-identified hypotheses, and the primary epistemic task is to become calibrated on such questions.” This leads me to suspect that EAs are undervaluing the ‘novel hypotheses generation’ component of predictions, though there is still a lot of value to be had from (novel) predictions.