I currently give it something in the .1%-1% range.
For reference: My impression is that this is on the low end, relative to estimates that other people in the long-termist AI safety/governance community would give, but that it’s not uniquely low. It’s also, I think, more than high enough to justify a lot of work and concern.
I am curious whether you are, in general, more optimistic about x-risks [say, than Toby Ord]. What are your estimates of total and unforeseen anthropogenic risks in the next century?
Toby’s estimate for “unaligned artificial intelligence” is the only one that I meaningfully disagree with.
I would probably give lower numbers for the other anthropogenic risks as well, since it seems really hard to kill virtually everyone, and since the historical record suggests that permanent collapse is unlikely. (Complex civilizations were independently developed multiple times; major collapses, like the Bronze Age Collapse or fall of the Roman Empire, were reversed after a couple thousand years; it didn’t take that long to go from the Neolithic Revolution to the Industrial Revolution; etc.) But I haven’t thought enough about civilizational recovery or, for example, future biological weapons to feel firm in my higher level of optimism.
My impression is that this is on the low end, relative to estimates that other people in the long-termist AI safety/governance community would give, but that it’s not uniquely low.
Your estimate is the second lowest one I’ve come across, with the lower one being from someone (James Fodor) who I don’t think is in the longtermist AI safety/governance community (though they’re an EA and engage with longtermist thinking). But I’m only talking about the relatively small number of explicit, public estimates people have given, not all the estimates that relevant people would give, so I’d guess that your statement is accurate.
(Also, to be clear, I don’t mean to be imply we should be more skeptical of estimates that “stand out from the pack” than those that are closer to other estimates.)
I’m curious as to whether most of that .1-1% probability mass is on existential catastrophe via something like the classic Bostrom/Yudkowsky type scenario, vs something like what Christiano describes in What failure looks like, vs deliberate misuse of AI, vs something else? E.g., is it like you still see the classic scenarios as the biggest cause for concern here? Or is it like you now see those scenarios as extremely unlikely, yet have a residual sense that something as massive as AGI could cause massive bad consequences somehow?
You said in the podcast that the drop was ‘an order of magnitude’, so presumably your original estimate was 1-10%? I note that this is similar to Toby Ord’s in The Precipice (~10%) so perhaps that should be a good rule of thumb: if you are convinced by the classic arguments your estimate of existential catastrophe from AI should be around 10% and if you are unconvinced by specific arguments, but still think AI is likely to become very powerful in the next century, then it should be around 1%?
Those numbers sound pretty reasonable to me, but, since they’re roughly my own credences, it’s probably unsurprising that I’m describing them as “pretty reasonable” :)
On the other hand, depending on what counts as being “convinced” of the classic arguments, I think it’s plausible they actually support a substantially higher probability. I certainly know that some people assign a significantly higher than 10% chance to an AI-based existential catastrophe this century. And I believe that Toby’s estimate, for example, involved weighing up different possible views.
What do you think is the probability of AI causing an existential catastrophe in the next century?
I currently give it something in the .1%-1% range.
For reference: My impression is that this is on the low end, relative to estimates that other people in the long-termist AI safety/governance community would give, but that it’s not uniquely low. It’s also, I think, more than high enough to justify a lot of work and concern.
I am curious whether you are, in general, more optimistic about x-risks [say, than Toby Ord]. What are your estimates of total and unforeseen anthropogenic risks in the next century?
Toby’s estimate for “unaligned artificial intelligence” is the only one that I meaningfully disagree with.
I would probably give lower numbers for the other anthropogenic risks as well, since it seems really hard to kill virtually everyone, and since the historical record suggests that permanent collapse is unlikely. (Complex civilizations were independently developed multiple times; major collapses, like the Bronze Age Collapse or fall of the Roman Empire, were reversed after a couple thousand years; it didn’t take that long to go from the Neolithic Revolution to the Industrial Revolution; etc.) But I haven’t thought enough about civilizational recovery or, for example, future biological weapons to feel firm in my higher level of optimism.
Thanks for sharing your probability estimate; I’ve now added it to my database of existential risk estimates.
Your estimate is the second lowest one I’ve come across, with the lower one being from someone (James Fodor) who I don’t think is in the longtermist AI safety/governance community (though they’re an EA and engage with longtermist thinking). But I’m only talking about the relatively small number of explicit, public estimates people have given, not all the estimates that relevant people would give, so I’d guess that your statement is accurate.
(Also, to be clear, I don’t mean to be imply we should be more skeptical of estimates that “stand out from the pack” than those that are closer to other estimates.)
I’m curious as to whether most of that .1-1% probability mass is on existential catastrophe via something like the classic Bostrom/Yudkowsky type scenario, vs something like what Christiano describes in What failure looks like, vs deliberate misuse of AI, vs something else? E.g., is it like you still see the classic scenarios as the biggest cause for concern here? Or is it like you now see those scenarios as extremely unlikely, yet have a residual sense that something as massive as AGI could cause massive bad consequences somehow?
You said in the podcast that the drop was ‘an order of magnitude’, so presumably your original estimate was 1-10%? I note that this is similar to Toby Ord’s in The Precipice (~10%) so perhaps that should be a good rule of thumb: if you are convinced by the classic arguments your estimate of existential catastrophe from AI should be around 10% and if you are unconvinced by specific arguments, but still think AI is likely to become very powerful in the next century, then it should be around 1%?
Those numbers sound pretty reasonable to me, but, since they’re roughly my own credences, it’s probably unsurprising that I’m describing them as “pretty reasonable” :)
On the other hand, depending on what counts as being “convinced” of the classic arguments, I think it’s plausible they actually support a substantially higher probability. I certainly know that some people assign a significantly higher than 10% chance to an AI-based existential catastrophe this century. And I believe that Toby’s estimate, for example, involved weighing up different possible views.