You correctly point out that “AI safety leaders” is a group that selects for high concern about AI, which means that the average is skewed towards high concern, relative to experts more generally.
I would like to add that the same is probably true (to a lesser extent) for AGI timeline estimates: People that think that AGI is very far away are less likely to think that AI safety is a pressing concern and are thus less motivated to become AI safety leaders. Also, people who are concerned about present-day AI risks, but don’t think AGI is imminent often call themselves “AI ethicists”, rather than AI safety people. These “AI ethicists” are unlikely to show up to a “summit on existential security”.
To be clear, I think it’s good to write this article, but we should always be mindful of selection effects when interpreting surveys.
Thanks for the comment, titotal. I agree the survey underestimates the variance in AI timelines and risk.
The AI Futures, which is know for AI 2027, had super broad timelines for artificial superintelligence (ASI) timelines on January 26. The difference between the 90th and 10th percentile was 168 years for Daniel Kokotajlo (2027 to 2195), and 137 years for Eli Lifland (2028 to 2165).
There is also huge variation in assessment of AI extinction risk. In the Existential Risk Persuasion Tournament (XPT), among domain experts and superforecasters, the 5th and 95th percentile AI extinction risk from 2023 to 2100 were 9.45*10^-7 and 37.0 % (excluding the 7 people who guessed a risk of exactly 0; here are the results).
I agree the survey underestimates the variance in AI timelines and risk.
The AI Futures, which is know for AI 2027, had super broad timelines for artificial superintelligence (ASI) timelines on January 26. The difference between the 90th and 10th percentile was 168 years for Daniel Kokotajlo (2027 to 2195), and 137 years for Eli Lifland (2028 to 2165).
Note that that is a different kind of distribution (one person’s beliefs) than the one reported here (many people’s medians)
Hi Buck. True. I still think the survey underestimates the variance in median AI timelines. Below are the results for the 2023 Expert Survey on Progress in AI (ESPAI). Half of the responses for the median date of full automation of tasks or occupations range from around 2045 to some date after 2120. In the survery of the post, half of the responses for the median date of AGI range from around 2032 to 2037. For the 25th percentile date of full automation, half of ESPAI’s responses range from around 2030 to 2100. In the survey of the post, half of the responses for the 25th percentile date of AGI range from around 2028 to 2032. AGI in the survey of the post does not have the exact same meaning as full automation of taks or occupations, but I am pretty confident my broad point stands if I am reading the graph below correctly.
Some vague questions are worth answering if they surface Information, but when the distribution is extremely broad, we should react epistemically with an update to the questions. From ESPAI23 to SES26 I see that being the case, where “all” becomes a more practical “90%”, which appears to 10x reduce the timeline uncertainty as it pertains to respondents’ 50% probability threshold.
Hi Emile. I see this is your 1st comment on the EA Forum. Welcome.
I think the difference in uncertainty is mostly explained by the surveys covering different people, not by ESPAI’s predictions having been made around 2.25 years earlier. ESPAI 2023 involved “2,778 researchers who had published in top-tier artificial intelligence (AI) venues”, and “took place in the fall of 2023″. The 2026 Summit on Existential Security (SES) involved “leaders and key thinkers in the x-risk and AI safety communities”, and “Survey data comes from the 59 respondents who consented to their answers being shared publicly”, and “was collected in February 2026”.
Buck, I would be curious to know what is your median time from weak AGI to artificial superintelligence (ASI) in this question from Metaculus, and your best guess for the (unconditional) probability of human extinction in the next 10 years.
I would be curious to know what is your median time from weak AGI to artificial superintelligence (ASI) in this question from Metaculus
I don’t want to take the time to really understand the operationalizations used in that question, sorry—I’m worried that the answer might depend substantially on finicky details. In general I think that the AI Futures Model is pretty similar to my views on how long it will take to go from “weak AGI” to “ASI”, for various definitions of those terms.
I think AI takeover in the next ten years is like 35% (😔), and conditional on takeover, I mostly agree with this; AI takeover is the main mechanism that causes me to expect massive human fatalities in the next decade.
Got it. Thanks. Here is what @Ryan Greenblatt says in the piece you linked about fatalities from AI takeover.
My guess is that, conditional on AI takeover, around 50% of currently living people die3 in expectation and literal human extinction4 is around 25% likely.5
Ryan, could you clarify what is the timeline for the 50 % of humans dying in expectation conditional on takeover?
The 25 % chance of human extinction refers to the 300 years following takeover, and excludes voluntary informed extinction, which I like because it would not be obviously bad. Here is footnote 4.
Concretely, literally every human in this universe is dead (under the definition of dead included in the prior footnote, so consensual uploads don’t count). And, this happens within 300 years of AI takeover and is caused by AI takeover. I’ll put aside outcomes where the AI later ends up simulating causally separate humans or otherwise instantiating humans (or human-like beings) which aren’t really downstream of currently living humans. I won’t consider it extinction if humans decide (in an informed way) to cease while they could have persisted or decide to modify themselves into very inhuman beings (again with informed consent etc.).
Thanks, Ryan. Buck and Ryan, do you see anybet I could make with any of you against short transformative AI (TAI) timelines, or what they supposedly imply, that is beneficial for both of us?
You correctly point out that “AI safety leaders” is a group that selects for high concern about AI, which means that the average is skewed towards high concern, relative to experts more generally.
I would like to add that the same is probably true (to a lesser extent) for AGI timeline estimates: People that think that AGI is very far away are less likely to think that AI safety is a pressing concern and are thus less motivated to become AI safety leaders. Also, people who are concerned about present-day AI risks, but don’t think AGI is imminent often call themselves “AI ethicists”, rather than AI safety people. These “AI ethicists” are unlikely to show up to a “summit on existential security”.
To be clear, I think it’s good to write this article, but we should always be mindful of selection effects when interpreting surveys.
Thanks for the comment, titotal. I agree the survey underestimates the variance in AI timelines and risk.
The AI Futures, which is know for AI 2027, had super broad timelines for artificial superintelligence (ASI) timelines on January 26. The difference between the 90th and 10th percentile was 168 years for Daniel Kokotajlo (2027 to 2195), and 137 years for Eli Lifland (2028 to 2165).
There is also huge variation in assessment of AI extinction risk. In the Existential Risk Persuasion Tournament (XPT), among domain experts and superforecasters, the 5th and 95th percentile AI extinction risk from 2023 to 2100 were 9.45*10^-7 and 37.0 % (excluding the 7 people who guessed a risk of exactly 0; here are the results).
Note that that is a different kind of distribution (one person’s beliefs) than the one reported here (many people’s medians)
Hi Buck. True. I still think the survey underestimates the variance in median AI timelines. Below are the results for the 2023 Expert Survey on Progress in AI (ESPAI). Half of the responses for the median date of full automation of tasks or occupations range from around 2045 to some date after 2120. In the survery of the post, half of the responses for the median date of AGI range from around 2032 to 2037. For the 25th percentile date of full automation, half of ESPAI’s responses range from around 2030 to 2100. In the survey of the post, half of the responses for the 25th percentile date of AGI range from around 2028 to 2032. AGI in the survey of the post does not have the exact same meaning as full automation of taks or occupations, but I am pretty confident my broad point stands if I am reading the graph below correctly.
Some vague questions are worth answering if they surface Information, but when the distribution is extremely broad, we should react epistemically with an update to the questions. From ESPAI23 to SES26 I see that being the case, where “all” becomes a more practical “90%”, which appears to 10x reduce the timeline uncertainty as it pertains to respondents’ 50% probability threshold.
Hi Emile. I see this is your 1st comment on the EA Forum. Welcome.
I think the difference in uncertainty is mostly explained by the surveys covering different people, not by ESPAI’s predictions having been made around 2.25 years earlier. ESPAI 2023 involved “2,778 researchers who had published in top-tier artificial intelligence (AI) venues”, and “took place in the fall of 2023″. The 2026 Summit on Existential Security (SES) involved “leaders and key thinkers in the x-risk and AI safety communities”, and “Survey data comes from the 59 respondents who consented to their answers being shared publicly”, and “was collected in February 2026”.
Buck, I would be curious to know what is your median time from weak AGI to artificial superintelligence (ASI) in this question from Metaculus, and your best guess for the (unconditional) probability of human extinction in the next 10 years.
I don’t want to take the time to really understand the operationalizations used in that question, sorry—I’m worried that the answer might depend substantially on finicky details. In general I think that the AI Futures Model is pretty similar to my views on how long it will take to go from “weak AGI” to “ASI”, for various definitions of those terms.
I think AI takeover in the next ten years is like 35% (😔), and conditional on takeover, I mostly agree with this; AI takeover is the main mechanism that causes me to expect massive human fatalities in the next decade.
Got it. Thanks. Here is what @Ryan Greenblatt says in the piece you linked about fatalities from AI takeover.
Ryan, could you clarify what is the timeline for the 50 % of humans dying in expectation conditional on takeover?
The 25 % chance of human extinction refers to the 300 years following takeover, and excludes voluntary informed extinction, which I like because it would not be obviously bad. Here is footnote 4.
I was imagining most deaths are within a few years, but full extinction may take considerably longer (see footnote that you quoted).
Thanks, Ryan. Buck and Ryan, do you see any bet I could make with any of you against short transformative AI (TAI) timelines, or what they supposedly imply, that is beneficial for both of us?