As for existential risk, my current very tentative forecast is that the world state at the end of 2100 to look something like:
73% - the world in 2100 looks broadly like it does now (in 2023) in the same sense that the current 2023 world looks broadly like it did in 1946. That is to say of course there will be a lot of technological and sociological change between now and then but by the end of 2100 there still won’t be unprecedented explosive economic growth (e.g.., >30% GWP growth per year), no existential disaster, etc.
9% - the world is in a singleton state controlled by an unaligned rogue AI acting on its own initiative.
6% - the future is good for humans but our AI / post-AI society causes some other moral disaster (e.g., widespread abuse of digital minds, widespread factory farming)
5% - we get aligned AI, solve the time of perils, and have a really great future
4% - the world is in a singleton state controlled by an AI-enabled dictatorship that was initiated by some human actor misusing AI intentionally
1% - all humans are extinct due to an unaligned rogue AI acting on its own initiative
2% - all humans are extinct due to something else on this list (e.g., some other AI scenario, nukes, biorisk, unknown unknowns)
I think conditional on producing minimal menace AI by the end of 2070, there’s a 28% chance an existential risk would follow within the next 100 years that could be attributed to that AI system.
9% - the world is in a singleton state controlled by an unaligned rogue AI acting on its own initiative. …
1% - all humans are extinct due to an unaligned rogue AI acting on its own initiative
This is interesting and something I haven’t seen much expressed within EA. What is happening in the 8% where the humans are still around and the unaligned singleton rogue AI is acting on it’s own initiative? Does it just take decades to wipe all the humans out? Are there digital uploads of (some) humans for the purposes of information saving?[1] Is a ceiling on intelligence/capability hit upon by the AI which means humans retain some economic niches? Is the misalignment only partial, so that the AI somehow shares some of humanity’s values (enough to keep us around)?
I think conditional on producing minimal menace AI by the end of 2070, there’s a 28% chance an existential risk would follow within the next 100 years that could be attributed to that AI system.
As for existential risk, my current very tentative forecast is that the world state at the end of 2100 to look something like:
73% - the world in 2100 looks broadly like it does now (in 2023) in the same sense that the current 2023 world looks broadly like it did in 1946. That is to say of course there will be a lot of technological and sociological change between now and then but by the end of 2100 there still won’t be unprecedented explosive economic growth (e.g.., >30% GWP growth per year), no existential disaster, etc.
9% - the world is in a singleton state controlled by an unaligned rogue AI acting on its own initiative.
6% - the future is good for humans but our AI / post-AI society causes some other moral disaster (e.g., widespread abuse of digital minds, widespread factory farming)
5% - we get aligned AI, solve the time of perils, and have a really great future
4% - the world is in a singleton state controlled by an AI-enabled dictatorship that was initiated by some human actor misusing AI intentionally
1% - all humans are extinct due to an unaligned rogue AI acting on its own initiative
2% - all humans are extinct due to something else on this list (e.g., some other AI scenario, nukes, biorisk, unknown unknowns)
I think conditional on producing minimal menace AI by the end of 2070, there’s a 28% chance an existential risk would follow within the next 100 years that could be attributed to that AI system.
Though I don’t know how seriously you should take this, because forecasting >75 years into the future is very hard.
Also my views of this are very incomplete and in flux and I look forward to refining them and writing more about them publicly.
This is interesting and something I haven’t seen much expressed within EA. What is happening in the 8% where the humans are still around and the unaligned singleton rogue AI is acting on it’s own initiative? Does it just take decades to wipe all the humans out? Are there digital uploads of (some) humans for the purposes of information saving?[1] Is a ceiling on intelligence/capability hit upon by the AI which means humans retain some economic niches? Is the misalignment only partial, so that the AI somehow shares some of humanity’s values (enough to keep us around)?
Does this mean that you think we get alignment by default? Or alignment is on track to be solved on this timeline? Or somehow we survive misaligned AI (as per the above discrepancy between your estimates for singleton unaligned rogue AI and human extinction)? As per my previous comment, I think the default outcome of AGI is doom with high likelihood (and haven’t received any satisfactory answers to the question If your AGI x-risk estimates are low, what scenarios make up the bulk of your expectations for an OK outcome?
This still seems like pretty much an existential catastrophe in my book, even if it isn’t technically extinction.
Thanks for elaborating, Peter! Do you mind sharing how you obtained those probabilities? Are they your subjective guesses?