“we’ve found people don’t know what issues they should be focusing on, and rather than a probability estimate, they want help to identify the most prescient questions”.
Interesting point. Curious whether this alludes to a broader bottleneck within forecasting around question generation itself, rather than question answering/calibration. My vague impression (though I don’t know anything about the practical workflows of forecasting organisations) is that a lot of attention/status accrues to producing forecasts, whereas identifying and specifying the right questions may be at least as important (and potentially harder?) I’d be interested to hear how much effort (as a rough %) forecasting orgs currently devote to question discovery/specification versus answering already-defined questions.
Also wondering at what “level” this problem mainly exists. Is the issue:
identifying the broad strategic questions institutions should care about in the first place (e.g., cause prioritisation, emerging risks, etc), or
the more granular forecasting-design questions like framing, timelines, resolution criteria, etc?
(or perhaps both of these are linked too tightly in practice to separate?)
So my experience is that identifying/specifying/generating the right questions is at least 50% of the benefit, if not higher. There are lots of reasons organisations struggle with this, from: organisational incentives; incoherent steers and views from senior managers; lack of accountability and ownership; to simply not recognising that they are trying to do a prediction.
This is why forecasting funding that has focused on improving forecasting accuracy is flawed, because it doesn’t matter how accurate you are if your question isn’t of use to the decision making process.
The problem exists at both those “levels”, but the most important one to solve for an organisation is the first one. Issues with resolution criteria etc. decrease accuracy but as long as people’s rationale’s are explicit you can bridge that gap (i.e. I know why one person was higher and another was lower, it was because they both took the resolution to mean something slightly differently). But if the question/problem you are trying to solve in the first instance is wrong, then the whole thing is a waste of time and energy.
Thank you for posting this!
Interesting point. Curious whether this alludes to a broader bottleneck within forecasting around question generation itself, rather than question answering/calibration. My vague impression (though I don’t know anything about the practical workflows of forecasting organisations) is that a lot of attention/status accrues to producing forecasts, whereas identifying and specifying the right questions may be at least as important (and potentially harder?)
I’d be interested to hear how much effort (as a rough %) forecasting orgs currently devote to question discovery/specification versus answering already-defined questions.
Also wondering at what “level” this problem mainly exists. Is the issue:
identifying the broad strategic questions institutions should care about in the first place (e.g., cause prioritisation, emerging risks, etc), or
the more granular forecasting-design questions like framing, timelines, resolution criteria, etc?
(or perhaps both of these are linked too tightly in practice to separate?)
So my experience is that identifying/specifying/generating the right questions is at least 50% of the benefit, if not higher. There are lots of reasons organisations struggle with this, from: organisational incentives; incoherent steers and views from senior managers; lack of accountability and ownership; to simply not recognising that they are trying to do a prediction.
This is why forecasting funding that has focused on improving forecasting accuracy is flawed, because it doesn’t matter how accurate you are if your question isn’t of use to the decision making process.
The problem exists at both those “levels”, but the most important one to solve for an organisation is the first one. Issues with resolution criteria etc. decrease accuracy but as long as people’s rationale’s are explicit you can bridge that gap (i.e. I know why one person was higher and another was lower, it was because they both took the resolution to mean something slightly differently). But if the question/problem you are trying to solve in the first instance is wrong, then the whole thing is a waste of time and energy.