This is far below pretty much any value any prominent AI safety person talks about, which are typically 10%+ or even up to ~90%. Does this give you pause? Or how do you explain this?
A footnote says the 0.15% number isn’t an actual forecast: “Participants were asked to indicate their intuitive impression of this risk, rather than develop a detailed forecast”. But superforecasters’ other forecasts are roughly consistent with 0.15% for extinction, so it still bears explaining.
In general I think superforecasters tend to anchor on historical trends, while AI safety people anchor on what’s physically possible or conceivable. Superforecasters get good accuracy compared to domain experts on most questions because domain experts in many fields don’t know how to use reference classes and historical trends well. But it’s done poorly recently because progress has accelerated—even in 2022 superforecasters’ median for the AI IMO gold medal was 2035, whereas it actually happened in 2025. Choosing a reference class for extinction is very difficult so people just rely on vibes.
Let’s take the question of whether world energy consumption will double year-over-year before 2045. In the full writeup, superforecasters, whose median is 0.35%, emphasized the huge difficulty in constructing terrestrial facilities to use that much energy:
Superforecasters generally expressed skepticism about a massive increase (doubling) in global energy consumption, due to this having low base rates and requiring unlikely technical breakthroughs.
Many rationales expressed skepticism that the rate of energy production could be scaled up so quickly even with advanced AI.
The breakthroughs thought to be needed are in energy production and distribution techniques.
A few superforecasters said that they thought fusion was the only plausible path, but even then other physical infrastructure might be limiting.
In contrast, I wrote about how doubling energy production in a year starting from self replicating robots in space just requires us to be more than ~0.1% efficient in refining asteroid raw material into solar panels and robots, and that it’s likely we get there eventually. I’m closer to 50% on this question.
Dyson swarms can have energy doubling times of *days*. The energy payback time of current solar panels on Earth is 1-2 years, in space there’s 8x more light than on Earth, and we’re >3 OOMs away from the minimum energy required to make solar panels (reducing SiO2 to Si).
I think to *not* get an energy doubling in one year by the time we exhaust the solar system’s energy, it would require a big slowdown (eg due to regulation or low energy demand) through about 15 OOMs of energy use, spanning from the first decently efficient self-replicating robots through Dyson swarms until we disassemble the gas planets for fusion fuel. Such a period would necessarily take decades or centuries to always be doubling slower than 1 year, which is basically an eternity when we have ASI.
The other factor is that AI safety people sometimes have a more inclusive definition of p(doom), that includes not just extinction but AIs seizing control of the world and colonizing the galaxy while leaving humans powerless.
Thank you a lot for this detailed answer. Especially points where superforecasters have provably been wrong on AI-related questions are very interesting and are certainly a very relevant argument against updating too much in their direction. Some kind of track record of superforecasters, experts, and public figures making predictions would be extremely interesting. Do you know whether something like this can be found somewhere?
To push back a bit against it being hard to find a good reference class and superforecasters having to rely on vibes: Yes, it might be hard, but aren’t superforecasters precisely those who have a great track record for finding a good methodology for making predictions, even when it’s hard? AI extinction is probably not the only question where making a forecast is tricky.
Sure, even a 0.15% probability by itself seems scary, though it might be low enough that you start wondering about trade-offs with delaying technological progress.
Apart from that, I would be interested how people with much higher P(doom) than that reconcile their belief with these numbers? Are there good reasons to believe that these numbers are not representative of the actual beliefs of superforecasters? Or that superforecasters are somehow systematically wrong or untrustworthy on this issue?
Hello everyone,
I have a question for those in the community that focus on AI safety: What do you make of superforecasters seemingly often having a very low P(doom)?
For example, in this survey (https://metr.org/blog/2025-08-20-forecasting-impacts-of-ai-acceleration/) superforecasters give a median P(doom) of 0.15% by 2100. You can find this number in the full write-up (https://docs.google.com/document/d/1QPvUlFG6-CrcZeXiv541pdt3oxNd2pTcBOOwEnSStRA/edit?usp=sharing), which is also linked in the blog post.
This is far below pretty much any value any prominent AI safety person talks about, which are typically 10%+ or even up to ~90%. Does this give you pause? Or how do you explain this?
A footnote says the 0.15% number isn’t an actual forecast: “Participants were asked to indicate their intuitive impression of this risk, rather than develop a detailed forecast”. But superforecasters’ other forecasts are roughly consistent with 0.15% for extinction, so it still bears explaining.
In general I think superforecasters tend to anchor on historical trends, while AI safety people anchor on what’s physically possible or conceivable. Superforecasters get good accuracy compared to domain experts on most questions because domain experts in many fields don’t know how to use reference classes and historical trends well. But it’s done poorly recently because progress has accelerated—even in 2022 superforecasters’ median for the AI IMO gold medal was 2035, whereas it actually happened in 2025. Choosing a reference class for extinction is very difficult so people just rely on vibes.
Let’s take the question of whether world energy consumption will double year-over-year before 2045. In the full writeup, superforecasters, whose median is 0.35%, emphasized the huge difficulty in constructing terrestrial facilities to use that much energy:
In contrast, I wrote about how doubling energy production in a year starting from self replicating robots in space just requires us to be more than ~0.1% efficient in refining asteroid raw material into solar panels and robots, and that it’s likely we get there eventually. I’m closer to 50% on this question.
The other factor is that AI safety people sometimes have a more inclusive definition of p(doom), that includes not just extinction but AIs seizing control of the world and colonizing the galaxy while leaving humans powerless.
Thank you a lot for this detailed answer. Especially points where superforecasters have provably been wrong on AI-related questions are very interesting and are certainly a very relevant argument against updating too much in their direction. Some kind of track record of superforecasters, experts, and public figures making predictions would be extremely interesting. Do you know whether something like this can be found somewhere?
To push back a bit against it being hard to find a good reference class and superforecasters having to rely on vibes: Yes, it might be hard, but aren’t superforecasters precisely those who have a great track record for finding a good methodology for making predictions, even when it’s hard? AI extinction is probably not the only question where making a forecast is tricky.
Edit: Just a few days ago, we got this here, which is very relevant: https://forum.effectivealtruism.org/posts/fp5kEpBkhWsGgWu2D/assessing-near-term-accuracy-in-the-existential-risk
0.15% by 2100 seems pretty scary (would probably suggest spending more resources on it then we currently do).
Sure, even a 0.15% probability by itself seems scary, though it might be low enough that you start wondering about trade-offs with delaying technological progress.
Apart from that, I would be interested how people with much higher P(doom) than that reconcile their belief with these numbers? Are there good reasons to believe that these numbers are not representative of the actual beliefs of superforecasters? Or that superforecasters are somehow systematically wrong or untrustworthy on this issue?