A lot of political and organisation decision making is not based on how accurately you’ve predicted the world. Sad, but true.
I do not consider this sad, but just a fact about how decision-making works in the real world. Economists model humans as-if they use probabilities to make choices between well-defined options. If you model humans as making decisions in this way, then forecasting makes sense.
But the real world is not a multiple choice test offering pre-defined options which need probabilistic estimates. In real world decision-making, problem formulation and problem solving are the same cognitive process. There are no problems out there waiting for you to find them, rather, you have to define what is even a problem in the first place and how to think about it. Figuring out how to solve a problem is a process of sensemaking until you feel like you have a grip on the situation; like you know what is relevant, how it is relevant, what the moving parts are, how they interact, what your leverage point is, and what your values are. A probabilistic estimate can never offer that. A probabilistic estimate abstracts away so much that it actually leaves you feeling like you don’t know the space at all.
Dominic Cummings has mentioned this
During the pandemic, Dominic Cummings said some of the most useful stuff that he received and circulated in the British government was not forecasting, it was qualitative information explaining the general model of what’s going on, which enabled decision-makers to think more clearly about their options for action and the likely consequences. If you’re worried about a new disease outbreak, you don’t just want a percentage probability estimate about future case numbers, you want an explanation of how the virus is likely to spread, what you can do about it, how you can prevent it. Not the best estimate for how many COVID cases there will be in a month, but why forecasters believe there will be X COVID cases in a month. https://www.samstack.io/p/five-questions-for-michael-story
Forecasting is based on an as-if (read: wrong) model of decision-making. You wouldn’t decide when to make a left-hand-turn at a busy intersection based on some narrow probabilistic estimate (there is a 99% chance you won’t get t-boned if you turn now) because you want to understand the decision yourself, and that probabilistic estimate is missing so much (Will I T-bone someone? Are there people in the cross walk? Are the cars slowing down because the light already changed colors? Or are they speeding up because it already changed colors?)
In real world decision-making, framing/modeling is everything. How you take large problems and turn them into something tractable that the human mind can comprehend and reason about. But superforecasting assumes that problem away. It assumes away the most important aspect of decision-making; understanding.
I think this misunderstands forecasting or mischaracterises decision making.
Your brain absolutely does decide when to make a left-hand-turn at a busy intersection based on a probabilistic estimate. Your brain is just pretty reliable at making said judgments that you assume some binary choice was made. Subconciously you are weighing up the likelihood of various risks based on your senses observing them and concluding that there is a very low (or an appropriately low) probability of it going wrong. The easy way to test this is go to a zip line, one with a drunk operator and one with a sober operator. The zip line may look identically safe but your brain will (hopefully) consider the former more risky—not because the impact of something going wrong is different between the zip lines (both falls would hurt, if not kill you), but because the likelihood of something going wrong is different (as you’d calculate that a drunk operator may not be as cognitively aware of what they are doing). That’s a probabilistic assessment.
The difference is, most decisions we deal with in the world don’t need us to sit down and do a formal prediction process to ensure good decision making. However, most organisational, and especially those on policy and governance etc., do require it. They are concretely two predictions you are making:
1. What will the world be? (through the lens of what you care about, e.g. will there be an oil crisis?)
2. What influence on the world will my policy have? (e.g. if I send Trump a friendly email will he avoid attacking another petrostate?)
Viewing forecasting as just the probabilistic estimate is missing the entire benefit. Forecasting is the entire process. Stating your view probabilistic just allows you to understand your own and others uncertainty (and to provide a purity of accountability and incentive alignment). It only adds decision fatigue if you have not put in the right processes to interpret the result (such as upfront thresholds for action). If you are following an optimal predictive decision making process, you should be making your assumptions and your weighing up of information explicit. This is how you determine what is relevant, what isn’t relevant etc.
Dominic Cummings statement misses that everything he said there IS forecasting. What he’s actually saying is “the point estimate from forecasters was not useful—it was the explicit reasoning about the causal chain etc. from forecasters that was useful”.
However, benchmarking against accuracy ensures the incentives are correct and that decisions aren’t manipulated by elements that decrease the end outcomes efficacy.
If you conclude that it’d be better if decision makers had: 1) the most accurate view of the world; and 2) the most accurate view of how their actions may influence that world towards their objectives. Then I would stand by the statement that it’s sad.
I do not consider this sad, but just a fact about how decision-making works in the real world. Economists model humans as-if they use probabilities to make choices between well-defined options. If you model humans as making decisions in this way, then forecasting makes sense.
But the real world is not a multiple choice test offering pre-defined options which need probabilistic estimates. In real world decision-making, problem formulation and problem solving are the same cognitive process. There are no problems out there waiting for you to find them, rather, you have to define what is even a problem in the first place and how to think about it. Figuring out how to solve a problem is a process of sensemaking until you feel like you have a grip on the situation; like you know what is relevant, how it is relevant, what the moving parts are, how they interact, what your leverage point is, and what your values are. A probabilistic estimate can never offer that. A probabilistic estimate abstracts away so much that it actually leaves you feeling like you don’t know the space at all.
Dominic Cummings has mentioned this
Forecasting is based on an as-if (read: wrong) model of decision-making. You wouldn’t decide when to make a left-hand-turn at a busy intersection based on some narrow probabilistic estimate (there is a 99% chance you won’t get t-boned if you turn now) because you want to understand the decision yourself, and that probabilistic estimate is missing so much (Will I T-bone someone? Are there people in the cross walk? Are the cars slowing down because the light already changed colors? Or are they speeding up because it already changed colors?)
In real world decision-making, framing/modeling is everything. How you take large problems and turn them into something tractable that the human mind can comprehend and reason about. But superforecasting assumes that problem away. It assumes away the most important aspect of decision-making; understanding.
I think this misunderstands forecasting or mischaracterises decision making.
Your brain absolutely does decide when to make a left-hand-turn at a busy intersection based on a probabilistic estimate. Your brain is just pretty reliable at making said judgments that you assume some binary choice was made. Subconciously you are weighing up the likelihood of various risks based on your senses observing them and concluding that there is a very low (or an appropriately low) probability of it going wrong. The easy way to test this is go to a zip line, one with a drunk operator and one with a sober operator. The zip line may look identically safe but your brain will (hopefully) consider the former more risky—not because the impact of something going wrong is different between the zip lines (both falls would hurt, if not kill you), but because the likelihood of something going wrong is different (as you’d calculate that a drunk operator may not be as cognitively aware of what they are doing). That’s a probabilistic assessment.
The difference is, most decisions we deal with in the world don’t need us to sit down and do a formal prediction process to ensure good decision making. However, most organisational, and especially those on policy and governance etc., do require it. They are concretely two predictions you are making:
1. What will the world be? (through the lens of what you care about, e.g. will there be an oil crisis?)
2. What influence on the world will my policy have? (e.g. if I send Trump a friendly email will he avoid attacking another petrostate?)
Viewing forecasting as just the probabilistic estimate is missing the entire benefit. Forecasting is the entire process. Stating your view probabilistic just allows you to understand your own and others uncertainty (and to provide a purity of accountability and incentive alignment). It only adds decision fatigue if you have not put in the right processes to interpret the result (such as upfront thresholds for action). If you are following an optimal predictive decision making process, you should be making your assumptions and your weighing up of information explicit. This is how you determine what is relevant, what isn’t relevant etc.
Dominic Cummings statement misses that everything he said there IS forecasting. What he’s actually saying is “the point estimate from forecasters was not useful—it was the explicit reasoning about the causal chain etc. from forecasters that was useful”.
However, benchmarking against accuracy ensures the incentives are correct and that decisions aren’t manipulated by elements that decrease the end outcomes efficacy.
If you conclude that it’d be better if decision makers had: 1) the most accurate view of the world; and 2) the most accurate view of how their actions may influence that world towards their objectives. Then I would stand by the statement that it’s sad.