I think there is a very realistic chance one of the results of this survey has been quite significantly misreported. Specifically, the responses to the question about the slow/âmoderate/ârapid progress scenarios.
Error #1, which I raised here, was that the probabilities were reported without qualification, when what should have been reported was the probability that the scenario would be the one that best matches reality. To their immense credit, the Forecasting Research Institute said they would correct this in a future version of the report. I thank them greatly for that.
Error #2, which Iâm not 100% sure yet is in fact an error, so letâs call it Possible Error #2, is that these donât seem to be probabilities at all. (I originally raised this possible error here.)
Respondents are asked to predict, in December 2030, âwhat percent of LEAP panelists will chooseâ each scenario (not with any probability). This implies that if they think thereâs, say, a 51% chance that 30% of LEAP panelists will choose the slow scenario, they should respond to the question by saying 30% will choose the slow scenario. If they think thereâs a 99% chance that 30% of LEAP panelists will choose the slow scenario, they should also respond by saying 30% will choose the slow scenario. In either case, the number in their answer is exactly the same, despite a 48-point difference in the probability they assign to this outcome. The report says that 30% is the probability respondents assign to the slow scenario, but itâs not clear that the respondentsâ probability is 30%.
The Forecasting Research Institute only asks for the predicted âvote shareâ for each scenario and not the estimated probabilities behind those vote share predictions. It doesnât seem possible to derive the respondentsâ probability estimates from the vote share predictions alone. By analogy, if FiveThirtyEightâs 2020 election forecast predicts that Joe Biden will win a 55% share of the national vote, this doesnât tell you what probability the model assigns to Biden winning the election (whether itâs, say, 70%, 80%, or 90%). The modelâs probability is certainly not 55%. To know the modelâs probability or guess at it, you would need information other than just the predicted vote share.
So, it seems plausible to me â although not yet certain â that this claim about what the respondents said the probability of each scenario is (with or without the âbest matchingâ qualifier) is incorrect because the respondents were not asked about probabilities in the first place. If there is a way to derive probabilities from what the respondents were asked, I donât know what it is.
[Edited on Nov. 21, 2025 at 1:05 PM Eastern to add: titotal apparently agrees.]
Update: the Forecasting Research Institute has changed the language in the report in response to this critique! (It seems like titotal played an important role in this. Thank you, titotal.)
On page 32, the report now gives the survey results in the same intersubjective resolution/âmetaprediction wording the question was asked in, rather than as an unqualified probability:
By 2030, the average expert thinks that 23% of LEAP panelists will say the state of AI most closely mirrors an (ârapidâ) AI progress scenario that matches some of these claims.
This is awesome! Iâm very happy to see this. Thanks to the Forecasting Research Institute for making this change.
I see also this EA Forum post has been updated in the same way, so thanks for that as well. Great to see.
Update #2: titotal has published a full breakdown of the error involving the intersubjective resolution/âmetaprediction framing of the survey question. Itâs a great post that explains the error very well. Many thanks to titotal for taking the time to write the post and for talking to the Forecasting Research Institute about this. Thanks again to the Forecasting Research Institute for revising the report and this post.
I think there is a very realistic chance one of the results of this survey has been quite significantly misreported. Specifically, the responses to the question about the slow/âmoderate/ârapid progress scenarios.
Error #1, which I raised here, was that the probabilities were reported without qualification, when what should have been reported was the probability that the scenario would be the one that best matches reality. To their immense credit, the Forecasting Research Institute said they would correct this in a future version of the report. I thank them greatly for that.
Error #2, which Iâm not 100% sure yet is in fact an error, so letâs call it Possible Error #2, is that these donât seem to be probabilities at all. (I originally raised this possible error here.)
Respondents are asked to predict, in December 2030, âwhat percent of LEAP panelists will chooseâ each scenario (not with any probability). This implies that if they think thereâs, say, a 51% chance that 30% of LEAP panelists will choose the slow scenario, they should respond to the question by saying 30% will choose the slow scenario. If they think thereâs a 99% chance that 30% of LEAP panelists will choose the slow scenario, they should also respond by saying 30% will choose the slow scenario. In either case, the number in their answer is exactly the same, despite a 48-point difference in the probability they assign to this outcome. The report says that 30% is the probability respondents assign to the slow scenario, but itâs not clear that the respondentsâ probability is 30%.
The Forecasting Research Institute only asks for the predicted âvote shareâ for each scenario and not the estimated probabilities behind those vote share predictions. It doesnât seem possible to derive the respondentsâ probability estimates from the vote share predictions alone. By analogy, if FiveThirtyEightâs 2020 election forecast predicts that Joe Biden will win a 55% share of the national vote, this doesnât tell you what probability the model assigns to Biden winning the election (whether itâs, say, 70%, 80%, or 90%). The modelâs probability is certainly not 55%. To know the modelâs probability or guess at it, you would need information other than just the predicted vote share.
So, it seems plausible to me â although not yet certain â that this claim about what the respondents said the probability of each scenario is (with or without the âbest matchingâ qualifier) is incorrect because the respondents were not asked about probabilities in the first place. If there is a way to derive probabilities from what the respondents were asked, I donât know what it is.
[Edited on Nov. 21, 2025 at 1:05 PM Eastern to add: titotal apparently agrees.]
Update: the Forecasting Research Institute has changed the language in the report in response to this critique! (It seems like titotal played an important role in this. Thank you, titotal.)
On page 32, the report now gives the survey results in the same intersubjective resolution/âmetaprediction wording the question was asked in, rather than as an unqualified probability:
This is awesome! Iâm very happy to see this. Thanks to the Forecasting Research Institute for making this change.
I see also this EA Forum post has been updated in the same way, so thanks for that as well. Great to see.
Update #2: titotal has published a full breakdown of the error involving the intersubjective resolution/âmetaprediction framing of the survey question. Itâs a great post that explains the error very well. Many thanks to titotal for taking the time to write the post and for talking to the Forecasting Research Institute about this. Thanks again to the Forecasting Research Institute for revising the report and this post.