I currently give it something in the .1%-1% range.
For reference: My impression is that this is on the low end, relative to estimates that other people in the long-termist AI safety/āgovernance community would give, but that itās not uniquely low. Itās also, I think, more than high enough to justify a lot of work and concern.
I am curious whether you are, in general, more optimistic about x-risks [say, than Toby Ord]. What are your estimates of total and unforeseen anthropogenic risks in the next century?
Tobyās estimate for āunaligned artificial intelligenceā is the only one that I meaningfully disagree with.
I would probably give lower numbers for the other anthropogenic risks as well, since it seems really hard to kill virtually everyone, and since the historical record suggests that permanent collapse is unlikely. (Complex civilizations were independently developed multiple times; major collapses, like the Bronze Age Collapse or fall of the Roman Empire, were reversed after a couple thousand years; it didnāt take that long to go from the Neolithic Revolution to the Industrial Revolution; etc.) But I havenāt thought enough about civilizational recovery or, for example, future biological weapons to feel firm in my higher level of optimism.
My impression is that this is on the low end, relative to estimates that other people in the long-termist AI safety/āgovernance community would give, but that itās not uniquely low.
Your estimate is the second lowest one Iāve come across, with the lower one being from someone (James Fodor) who I donāt think is in the longtermist AI safety/āgovernance community (though theyāre an EA and engage with longtermist thinking). But Iām only talking about the relatively small number of explicit, public estimates people have given, not all the estimates that relevant people would give, so Iād guess that your statement is accurate.
(Also, to be clear, I donāt mean to be imply we should be more skeptical of estimates that āstand out from the packā than those that are closer to other estimates.)
Iām curious as to whether most of that .1-1% probability mass is on existential catastrophe via something like the classic Bostrom/āYudkowsky type scenario, vs something like what Christiano describes in What failure looks like, vs deliberate misuse of AI, vs something else? E.g., is it like you still see the classic scenarios as the biggest cause for concern here? Or is it like you now see those scenarios as extremely unlikely, yet have a residual sense that something as massive as AGI could cause massive bad consequences somehow?
You said in the podcast that the drop was āan order of magnitudeā, so presumably your original estimate was 1-10%? I note that this is similar to Toby Ordās in The Precipice (~10%) so perhaps that should be a good rule of thumb: if you are convinced by the classic arguments your estimate of existential catastrophe from AI should be around 10% and if you are unconvinced by specific arguments, but still think AI is likely to become very powerful in the next century, then it should be around 1%?
Those numbers sound pretty reasonable to me, but, since theyāre roughly my own credences, itās probably unsurprising that Iām describing them as āpretty reasonableā :)
On the other hand, depending on what counts as being āconvincedā of the classic arguments, I think itās plausible they actually support a substantially higher probability. I certainly know that some people assign a significantly higher than 10% chance to an AI-based existential catastrophe this century. And I believe that Tobyās estimate, for example, involved weighing up different possible views.
What do you think is the probability of AI causing an existential catastrophe in the next century?
I currently give it something in the .1%-1% range.
For reference: My impression is that this is on the low end, relative to estimates that other people in the long-termist AI safety/āgovernance community would give, but that itās not uniquely low. Itās also, I think, more than high enough to justify a lot of work and concern.
I am curious whether you are, in general, more optimistic about x-risks [say, than Toby Ord]. What are your estimates of total and unforeseen anthropogenic risks in the next century?
Tobyās estimate for āunaligned artificial intelligenceā is the only one that I meaningfully disagree with.
I would probably give lower numbers for the other anthropogenic risks as well, since it seems really hard to kill virtually everyone, and since the historical record suggests that permanent collapse is unlikely. (Complex civilizations were independently developed multiple times; major collapses, like the Bronze Age Collapse or fall of the Roman Empire, were reversed after a couple thousand years; it didnāt take that long to go from the Neolithic Revolution to the Industrial Revolution; etc.) But I havenāt thought enough about civilizational recovery or, for example, future biological weapons to feel firm in my higher level of optimism.
Thanks for sharing your probability estimate; Iāve now added it to my database of existential risk estimates.
Your estimate is the second lowest one Iāve come across, with the lower one being from someone (James Fodor) who I donāt think is in the longtermist AI safety/āgovernance community (though theyāre an EA and engage with longtermist thinking). But Iām only talking about the relatively small number of explicit, public estimates people have given, not all the estimates that relevant people would give, so Iād guess that your statement is accurate.
(Also, to be clear, I donāt mean to be imply we should be more skeptical of estimates that āstand out from the packā than those that are closer to other estimates.)
Iām curious as to whether most of that .1-1% probability mass is on existential catastrophe via something like the classic Bostrom/āYudkowsky type scenario, vs something like what Christiano describes in What failure looks like, vs deliberate misuse of AI, vs something else? E.g., is it like you still see the classic scenarios as the biggest cause for concern here? Or is it like you now see those scenarios as extremely unlikely, yet have a residual sense that something as massive as AGI could cause massive bad consequences somehow?
You said in the podcast that the drop was āan order of magnitudeā, so presumably your original estimate was 1-10%? I note that this is similar to Toby Ordās in The Precipice (~10%) so perhaps that should be a good rule of thumb: if you are convinced by the classic arguments your estimate of existential catastrophe from AI should be around 10% and if you are unconvinced by specific arguments, but still think AI is likely to become very powerful in the next century, then it should be around 1%?
Those numbers sound pretty reasonable to me, but, since theyāre roughly my own credences, itās probably unsurprising that Iām describing them as āpretty reasonableā :)
On the other hand, depending on what counts as being āconvincedā of the classic arguments, I think itās plausible they actually support a substantially higher probability. I certainly know that some people assign a significantly higher than 10% chance to an AI-based existential catastrophe this century. And I believe that Tobyās estimate, for example, involved weighing up different possible views.