Hi Emile. I see this is your 1st comment on the EA Forum. Welcome.
I think the difference in uncertainty is mostly explained by the surveys covering different people, not by ESPAIās predictions having been made around 2.25 years earlier. ESPAI 2023 involved ā2,778 researchers who had published in top-tier artificial intelligence (AI) venuesā, and ātook place in the fall of 2023ā³. The 2026 Summit on Existential Security (SES) involved āleaders and key thinkers in the x-risk and AI safety communitiesā, and āSurvey data comes from the 59 respondents who consented to their answers being shared publiclyā, and āwas collected in February 2026ā.
Buck, I would be curious to know what is your median time from weak AGI to artificial superintelligence (ASI) in this question from Metaculus, and your best guess for the (unconditional) probability of human extinction in the next 10 years.
I would be curious to know what is your median time from weak AGI to artificial superintelligence (ASI) in this question from Metaculus
I donāt want to take the time to really understand the operationalizations used in that question, sorryāIām worried that the answer might depend substantially on finicky details. In general I think that the AI Futures Model is pretty similar to my views on how long it will take to go from āweak AGIā to āASIā, for various definitions of those terms.
I think AI takeover in the next ten years is like 35% (š), and conditional on takeover, I mostly agree with this; AI takeover is the main mechanism that causes me to expect massive human fatalities in the next decade.
Got it. Thanks. Here is what @Ryan Greenblatt says in the piece you linked about fatalities from AI takeover.
My guess is that, conditional on AI takeover, around 50% of currently living people die3 in expectation and literal human extinction4 is around 25% likely.5
Ryan, could you clarify what is the timeline for the 50 % of humans dying in expectation conditional on takeover?
The 25 % chance of human extinction refers to the 300 years following takeover, and excludes voluntary informed extinction, which I like because it would not be obviously bad. Here is footnote 4.
Concretely, literally every human in this universe is dead (under the definition of dead included in the prior footnote, so consensual uploads donāt count). And, this happens within 300 years of AI takeover and is caused by AI takeover. Iāll put aside outcomes where the AI later ends up simulating causally separate humans or otherwise instantiating humans (or human-like beings) which arenāt really downstream of currently living humans. I wonāt consider it extinction if humans decide (in an informed way) to cease while they could have persisted or decide to modify themselves into very inhuman beings (again with informed consent etc.).
Thanks, Ryan. Buck and Ryan, do you see anybet I could make with any of you against short transformative AI (TAI) timelines, or what they supposedly imply, that is beneficial for both of us?
Hi Emile. I see this is your 1st comment on the EA Forum. Welcome.
I think the difference in uncertainty is mostly explained by the surveys covering different people, not by ESPAIās predictions having been made around 2.25 years earlier. ESPAI 2023 involved ā2,778 researchers who had published in top-tier artificial intelligence (AI) venuesā, and ātook place in the fall of 2023ā³. The 2026 Summit on Existential Security (SES) involved āleaders and key thinkers in the x-risk and AI safety communitiesā, and āSurvey data comes from the 59 respondents who consented to their answers being shared publiclyā, and āwas collected in February 2026ā.
Buck, I would be curious to know what is your median time from weak AGI to artificial superintelligence (ASI) in this question from Metaculus, and your best guess for the (unconditional) probability of human extinction in the next 10 years.
I donāt want to take the time to really understand the operationalizations used in that question, sorryāIām worried that the answer might depend substantially on finicky details. In general I think that the AI Futures Model is pretty similar to my views on how long it will take to go from āweak AGIā to āASIā, for various definitions of those terms.
I think AI takeover in the next ten years is like 35% (š), and conditional on takeover, I mostly agree with this; AI takeover is the main mechanism that causes me to expect massive human fatalities in the next decade.
Got it. Thanks. Here is what @Ryan Greenblatt says in the piece you linked about fatalities from AI takeover.
Ryan, could you clarify what is the timeline for the 50 % of humans dying in expectation conditional on takeover?
The 25 % chance of human extinction refers to the 300 years following takeover, and excludes voluntary informed extinction, which I like because it would not be obviously bad. Here is footnote 4.
I was imagining most deaths are within a few years, but full extinction may take considerably longer (see footnote that you quoted).
Thanks, Ryan. Buck and Ryan, do you see any bet I could make with any of you against short transformative AI (TAI) timelines, or what they supposedly imply, that is beneficial for both of us?