My estimates aren’t low, I think there’s very roughly about 40% chance we’ll die because of AI this century. But here are some reasons why it isn’t higher:
Creating a species vastly more intelligent than yourself seems highly unusual, nothing like it has happened before, so there need to be very good arguments for why it’s possible.
Having a species completely kill all other species is also very unusual, so there need to be very good arguments for why that would happen.
Perhaps AGI won’t be utility maximizing, LLMs don’t seem to be very maximizing. If it has a good model of the world maybe it’ll understand what we want and just give us that.
Perhaps we’ll convince the world to slow down AI capabilities research and solve alignment in time.
There are good counterarguments to these which is why my p(doom) is so high, but they still add to my uncertainty.
In response to point 2 - if you see human civilization continuing to develop indefinitely without regard for other species, wouldn’t other species be all extinct, except for maybe a select few?
Other species are instrumentally very useful to humans, providing ecosystem functions, food, and sources of material (including genetic material).
On the AI side, it seems possible that a powerful misaligned AGI would find ecosystems and/or biological materials valuable, or that it would be cheaper to use humans for some tasks than machines. I think these factors would raise the odds that some humans (or human-adjacent engineered beings) survive in worlds dominated by such an AGI.
My estimates aren’t low, I think there’s very roughly about 40% chance we’ll die because of AI this century. But here are some reasons why it isn’t higher:
Creating a species vastly more intelligent than yourself seems highly unusual, nothing like it has happened before, so there need to be very good arguments for why it’s possible.
Having a species completely kill all other species is also very unusual, so there need to be very good arguments for why that would happen.
Perhaps AGI won’t be utility maximizing, LLMs don’t seem to be very maximizing. If it has a good model of the world maybe it’ll understand what we want and just give us that.
Perhaps we’ll convince the world to slow down AI capabilities research and solve alignment in time.
There are good counterarguments to these which is why my p(doom) is so high, but they still add to my uncertainty.
In response to point 2 - if you see human civilization continuing to develop indefinitely without regard for other species, wouldn’t other species be all extinct, except for maybe a select few?
“without regard for other species” is doing a lot of work
Other species are instrumentally very useful to humans, providing ecosystem functions, food, and sources of material (including genetic material).
On the AI side, it seems possible that a powerful misaligned AGI would find ecosystems and/or biological materials valuable, or that it would be cheaper to use humans for some tasks than machines. I think these factors would raise the odds that some humans (or human-adjacent engineered beings) survive in worlds dominated by such an AGI.
This seems pretty unlikely to me