Executive summary: The post argues that a subtle wording error in one LEAP survey question caused respondents and report authors to conflate three distinct questions, making the published statistic unsuitable as evidence about experts’ actual beliefs about future AI progress.
Key points:
The author says the report’s text described the statistic as if experts had been asked the probability of “rapid” AI progress (question 0), but the footnote actually summarized a different query about how LEAP panelists would vote (question 1).
The author states that the real survey item asked for the percentage of 2030 LEAP panelists who would choose “rapid” (question 2), which becomes a prediction of a future distribution rather than a probability of rapid progress.
The author argues that questions 0, 1, and 2 yield different numerical answers even under ideal reasoning, so treating responses to question 2 as if they reflected question 0 was an error.
The author claims that respondents likely misinterpreted the question, given its length, complexity, and lack of reminder about what was being asked.
The author reports that the LEAP team updated the document wording to reflect the actual question and discussed their rationale for scoreable questions but maintained that the issue does not affect major report findings.
The author recommends replacing the question with a direct probability-of-progress item plus additional scoreable questions to distinguish beliefs about AI progress from beliefs about panel accuracy.
This comment was auto-generated by the EA Forum Team. Feel free to point out issues with this summary by replying to the comment, andcontact us if you have feedback.
Executive summary: The post argues that a subtle wording error in one LEAP survey question caused respondents and report authors to conflate three distinct questions, making the published statistic unsuitable as evidence about experts’ actual beliefs about future AI progress.
Key points:
The author says the report’s text described the statistic as if experts had been asked the probability of “rapid” AI progress (question 0), but the footnote actually summarized a different query about how LEAP panelists would vote (question 1).
The author states that the real survey item asked for the percentage of 2030 LEAP panelists who would choose “rapid” (question 2), which becomes a prediction of a future distribution rather than a probability of rapid progress.
The author argues that questions 0, 1, and 2 yield different numerical answers even under ideal reasoning, so treating responses to question 2 as if they reflected question 0 was an error.
The author claims that respondents likely misinterpreted the question, given its length, complexity, and lack of reminder about what was being asked.
The author reports that the LEAP team updated the document wording to reflect the actual question and discussed their rationale for scoreable questions but maintained that the issue does not affect major report findings.
The author recommends replacing the question with a direct probability-of-progress item plus additional scoreable questions to distinguish beliefs about AI progress from beliefs about panel accuracy.
This comment was auto-generated by the EA Forum Team. Feel free to point out issues with this summary by replying to the comment, and contact us if you have feedback.