My estimates arenât low, I think thereâs very roughly about 40% chance weâll die because of AI this century. But here are some reasons why it isnât higher:
Creating a species vastly more intelligent than yourself seems highly unusual, nothing like it has happened before, so there need to be very good arguments for why itâs possible.
Having a species completely kill all other species is also very unusual, so there need to be very good arguments for why that would happen.
Perhaps AGI wonât be utility maximizing, LLMs donât seem to be very maximizing. If it has a good model of the world maybe itâll understand what we want and just give us that.
Perhaps weâll convince the world to slow down AI capabilities research and solve alignment in time.
There are good counterarguments to these which is why my p(doom) is so high, but they still add to my uncertainty.
In response to point 2 - if you see human civilization continuing to develop indefinitely without regard for other species, wouldnât other species be all extinct, except for maybe a select few?
Other species are instrumentally very useful to humans, providing ecosystem functions, food, and sources of material (including genetic material).
On the AI side, it seems possible that a powerful misaligned AGI would find ecosystems and/âor biological materials valuable, or that it would be cheaper to use humans for some tasks than machines. I think these factors would raise the odds that some humans (or human-adjacent engineered beings) survive in worlds dominated by such an AGI.
My estimates arenât low, I think thereâs very roughly about 40% chance weâll die because of AI this century. But here are some reasons why it isnât higher:
Creating a species vastly more intelligent than yourself seems highly unusual, nothing like it has happened before, so there need to be very good arguments for why itâs possible.
Having a species completely kill all other species is also very unusual, so there need to be very good arguments for why that would happen.
Perhaps AGI wonât be utility maximizing, LLMs donât seem to be very maximizing. If it has a good model of the world maybe itâll understand what we want and just give us that.
Perhaps weâll convince the world to slow down AI capabilities research and solve alignment in time.
There are good counterarguments to these which is why my p(doom) is so high, but they still add to my uncertainty.
In response to point 2 - if you see human civilization continuing to develop indefinitely without regard for other species, wouldnât other species be all extinct, except for maybe a select few?
âwithout regard for other speciesâ is doing a lot of work
Other species are instrumentally very useful to humans, providing ecosystem functions, food, and sources of material (including genetic material).
On the AI side, it seems possible that a powerful misaligned AGI would find ecosystems and/âor biological materials valuable, or that it would be cheaper to use humans for some tasks than machines. I think these factors would raise the odds that some humans (or human-adjacent engineered beings) survive in worlds dominated by such an AGI.
This seems pretty unlikely to me