9% - the world is in a singleton state controlled by an unaligned rogue AI acting on its own initiative. …
1% - all humans are extinct due to an unaligned rogue AI acting on its own initiative
This is interesting and something I haven’t seen much expressed within EA. What is happening in the 8% where the humans are still around and the unaligned singleton rogue AI is acting on it’s own initiative? Does it just take decades to wipe all the humans out? Are there digital uploads of (some) humans for the purposes of information saving?[1] Is a ceiling on intelligence/capability hit upon by the AI which means humans retain some economic niches? Is the misalignment only partial, so that the AI somehow shares some of humanity’s values (enough to keep us around)?
I think conditional on producing minimal menace AI by the end of 2070, there’s a 28% chance an existential risk would follow within the next 100 years that could be attributed to that AI system.
This is interesting and something I haven’t seen much expressed within EA. What is happening in the 8% where the humans are still around and the unaligned singleton rogue AI is acting on it’s own initiative? Does it just take decades to wipe all the humans out? Are there digital uploads of (some) humans for the purposes of information saving?[1] Is a ceiling on intelligence/capability hit upon by the AI which means humans retain some economic niches? Is the misalignment only partial, so that the AI somehow shares some of humanity’s values (enough to keep us around)?
Does this mean that you think we get alignment by default? Or alignment is on track to be solved on this timeline? Or somehow we survive misaligned AI (as per the above discrepancy between your estimates for singleton unaligned rogue AI and human extinction)? As per my previous comment, I think the default outcome of AGI is doom with high likelihood (and haven’t received any satisfactory answers to the question If your AGI x-risk estimates are low, what scenarios make up the bulk of your expectations for an OK outcome?
This still seems like pretty much an existential catastrophe in my book, even if it isn’t technically extinction.