Thanks for posting this, I strongly upvoted it for these reasons:
It’s concise but a very high information-to-padding ratio, higher than I sometimes find on the forum[1]
IIDM is something I’m interested in, and seeing a high-quality post in this area is very welcome. I think it adds value to this area and brings attention to it on the Forum
The structure of ‘explanation—strength—weakness’ was very clear, in general I thought that the whole post was well structured but this made the post very easy to follow
As for the content of the post itself, I agree with most of it and I look forward to reading the references and links! My only comment for consideration would be, all 4 models seem to have an underlying weakness—that they are actually unlikely to be fully integrated into policy decision-making circles, and that acts as a bottleneck on applying any form on forecasting in the policy realm.
So my questions in that area would be:
Have there been any historical examples where forecasts where explicitly integrated into decision making, either in the public or private sectors? [My assumption is very little of both, and what there has is likely a lot more on the private than public side]
What are the empirical barriers to forecasting being adopted in public policy? Are there case studies of this being attempted and shut down, and in these cases where were the key points leading to a rejection of these models?
Are there any particular constituencies/polities where we might expect forecasting to have more of a foothold—where engagement by the EA IIDM community might lead to actual implementation?
And finally, I just want to end by saying again I thought it was a very good post :)
I’m curious about model 3 - the policy evaluation model.
I think this point is particularly insightful: “Conditional forecasting would also require policymakers to identify discrete and falsifiable goals of their policies, which would already be a major process improvement.”
But I don’t quite understand the thinking behind the following two points:
“generating discrete probabilities about the likely success of certain policy tools would incentivize decision-makers to engage with the logic underlying relevant forecasts.”—how exactly do you see this model changing the incentives policymakers face, relative to the status quo (which includes conditional forecasts sometimes being generated on the likes of Metaculus etc)?
“It would also provide the foundations for more active learning in the policymaking sphere: policymakers would be able to improve their policymaking skills by studying which of their interventions succeeded and failed.”—what’s the process you envision here that enables active leasing? If policymakers themselves are the ones that are making forecasts, and can see how their predictions compare to actual outcomes, then I can see where the learning comes in. But if policymakers are still just consumers of forecasts in this model, I don’t see how the supply of conditional forecasts would itself support policymakers’ learning.
Thanks in advance for any additional detail you can provide on this proposal!
Thanks for posting this, I strongly upvoted it for these reasons:
It’s concise but a very high information-to-padding ratio, higher than I sometimes find on the forum[1]
IIDM is something I’m interested in, and seeing a high-quality post in this area is very welcome. I think it adds value to this area and brings attention to it on the Forum
The structure of ‘explanation—strength—weakness’ was very clear, in general I thought that the whole post was well structured but this made the post very easy to follow
As for the content of the post itself, I agree with most of it and I look forward to reading the references and links! My only comment for consideration would be, all 4 models seem to have an underlying weakness—that they are actually unlikely to be fully integrated into policy decision-making circles, and that acts as a bottleneck on applying any form on forecasting in the policy realm.
So my questions in that area would be:
Have there been any historical examples where forecasts where explicitly integrated into decision making, either in the public or private sectors? [My assumption is very little of both, and what there has is likely a lot more on the private than public side]
What are the empirical barriers to forecasting being adopted in public policy? Are there case studies of this being attempted and shut down, and in these cases where were the key points leading to a rejection of these models?
Are there any particular constituencies/polities where we might expect forecasting to have more of a foothold—where engagement by the EA IIDM community might lead to actual implementation?
And finally, I just want to end by saying again I thought it was a very good post :)
Especially on my own posts....
I worked for the UK Civil Service and it was hard to push forecasting because:
Making markets is hard
Getting people to care about the numbers if they appear is hard
I think that the social problem of prediction market buy in is a bit harder than people generally think.
Michael story writes about it well here.
https://mwstory.substack.com/p/why-i-generally-dont-recommend-internal
Cool models. Thanks for writing this.
Thanks for writing this up, it’s very useful!
I’m curious about model 3 - the policy evaluation model.
I think this point is particularly insightful: “Conditional forecasting would also require policymakers to identify discrete and falsifiable goals of their policies, which would already be a major process improvement.”
But I don’t quite understand the thinking behind the following two points:
“generating discrete probabilities about the likely success of certain policy tools would incentivize decision-makers to engage with the logic underlying relevant forecasts.”—how exactly do you see this model changing the incentives policymakers face, relative to the status quo (which includes conditional forecasts sometimes being generated on the likes of Metaculus etc)?
“It would also provide the foundations for more active learning in the policymaking sphere: policymakers would be able to improve their policymaking skills by studying which of their interventions succeeded and failed.”—what’s the process you envision here that enables active leasing? If policymakers themselves are the ones that are making forecasts, and can see how their predictions compare to actual outcomes, then I can see where the learning comes in. But if policymakers are still just consumers of forecasts in this model, I don’t see how the supply of conditional forecasts would itself support policymakers’ learning.
Thanks in advance for any additional detail you can provide on this proposal!