Existing nuclear weapon infrastructure, especially ICBMs, could be manipulated by a powerful AI to further its goals (which may well be orthogonal to our goals).
Smart things are not dangerous because they have access to human-built legacy nukes. Smart things are dangerous because they are smarter than you.
I expect that the most efficient way to kill everyone is via the biotech->nanotech->tiny diamondoid bacteria hopping the jetstream and replicating using CHON and sunlight->everybody falling over dead 3 days after it gets smart. I don’t expect it would use nukes if they were there.
Smart AIs are not dangerous because somebody built guns for them, smart AIs are not dangerous because cars are connected to the Internet, smart AIs are not dangerous because they can steal existing legacy weapons infrastructure, smart AIs are dangerous because they are smarter than you and can think of better stuff to do.
Sure, but I’m not sure this particular argument means working on nuclear safety is as important as working on AI. We could get rid of all nuclear weapons and a powerful AI could just remake them, or make far worse weapons that we can’t even conceive of now. Unless we destroy absolutely everything I’m sure a powerful unaligned AI will be able to wreak havoc, and the best way to prevent that seems to me to ensure AI is aligned in the first place!
Compare the number of steps required for an agent to initiate the launch of existing missiles to the number of steps required for an agent to build & use a missile-launching infrastructure de novo.
Not sure why number of steps is important. If we’re talking about very powerful unaligned AI it’s going to wreak havoc in any case. From a longtermist point of view it doesn’t matter if it takes it a day, a month, or a year to do so.
Ah cmon it still shifts the P(doom) distribution a bit.
Consider us having some solid countermeasures with OODA loops of ~days. If we delay doom by y days, then some number of countermeasures can fire where otherwise they wouldn’t get to fire at all.
(Though this assumes an imperfectly timed treacherous turn, before it’s unstoppable.)
Maybe. I was thinking that the point at which a rogue AI is powerful enough to take control of existing nuclear weapons is the point at which we’re already completely screwed, but I could be wrong.
NC3 early warning systems are susceptible to error signals, and the chain of command hasn’t always been v secure (and may not be today), so it wouldn’t necessarily be that hard for a relatively unsophisticated AGI to spoof and trigger a nuclear war:* certainly easier than many other avenues that would involve cracking scientific problems.
(*which is another thing from hacking to the level of “controlling” the arsenal and being able to retarget it at will, which would probably require a more advanced capability, where the risk from the nuclear avenue might perhaps be redundant compared to risks from other, direct avenues).
Incidentally, at CSER I’ve been working with co-authors on a draft chapter that explores “military AI as cause or compounder of global catastrophic risk”, and one of the avenues also involves discussion of what we call “weapons/arsenal overhang”, so this is an interesting topic that I’d love to discuss more
Right, that covers hard takeoff or long-con treachery—but there are scenarios where we uncover the risk before strict “prepotence”. And imo we should maintain a distribution over a big set of such scenarios at the moment.
Yeah, understandable but I would also push back. Mining / buying your own uranium and building a centrifuge to enrich it and putting it into a missile is difficult for even rogue nations like Iran. An advanced AI system might just be lines of code in a computer that can use the internet and output text or speech, but with no robotics system to give it physical capacity. From that point of view, building your own nukes seems much more difficult than hacking into an existing ICBM system.
I agree that the current nuclear weapon situation makes AI catastrophe more likely on the margin, and said as much here (The paragraph “You might reply: The thing that went wrong in this scenario is not the out-of-control AGI, it’s the fact that humanity is too vulnerable! And my response is: Why can’t it be both? …”)
That said, I do think the nuclear situation is a rather small effect (on AI risk specifically), in that there are many different paths for an intelligent motivated agent to cause chaos and destruction. Even if triggering nuclear war is the lowest-hanging fruit for a hypothetical future AGI aspiring to destroy humanity (it might or might not be, I dunno), I think other fruits are hanging only slightly higher, like causing extended blackouts, arranging for the release of bio-engineered plagues, triggering non-nuclear great power war (if someday nuclear weapons are eliminated), mass spearphishing / hacking, mass targeted disinformation, etc., even leaving aside more exotic things like nanobots. Solving all these problems would be that much harder (still worth trying!), and anyway we need to solve AI alignment one way or the other, IMO. :)
What does “overhang” mean in this context?
Existing nuclear weapon infrastructure, especially ICBMs, could be manipulated by a powerful AI to further its goals (which may well be orthogonal to our goals).
Smart things are not dangerous because they have access to human-built legacy nukes. Smart things are dangerous because they are smarter than you.
I expect that the most efficient way to kill everyone is via the biotech->nanotech->tiny diamondoid bacteria hopping the jetstream and replicating using CHON and sunlight->everybody falling over dead 3 days after it gets smart. I don’t expect it would use nukes if they were there.
Smart AIs are not dangerous because somebody built guns for them, smart AIs are not dangerous because cars are connected to the Internet, smart AIs are not dangerous because they can steal existing legacy weapons infrastructure, smart AIs are dangerous because they are smarter than you and can think of better stuff to do.
Some back-and-forth on this between Eliezer & me in this thread.
Sure, but I’m not sure this particular argument means working on nuclear safety is as important as working on AI. We could get rid of all nuclear weapons and a powerful AI could just remake them, or make far worse weapons that we can’t even conceive of now. Unless we destroy absolutely everything I’m sure a powerful unaligned AI will be able to wreak havoc, and the best way to prevent that seems to me to ensure AI is aligned in the first place!
Compare the number of steps required for an agent to initiate the launch of existing missiles to the number of steps required for an agent to build & use a missile-launching infrastructure de novo.
Not sure why number of steps is important. If we’re talking about very powerful unaligned AI it’s going to wreak havoc in any case. From a longtermist point of view it doesn’t matter if it takes it a day, a month, or a year to do so.
Ah cmon it still shifts the P(doom) distribution a bit.
Consider us having some solid countermeasures with OODA loops of ~days. If we delay doom by y days, then some number of countermeasures can fire where otherwise they wouldn’t get to fire at all.
(Though this assumes an imperfectly timed treacherous turn, before it’s unstoppable.)
Maybe. I was thinking that the point at which a rogue AI is powerful enough to take control of existing nuclear weapons is the point at which we’re already completely screwed, but I could be wrong.
NC3 early warning systems are susceptible to error signals, and the chain of command hasn’t always been v secure (and may not be today), so it wouldn’t necessarily be that hard for a relatively unsophisticated AGI to spoof and trigger a nuclear war:* certainly easier than many other avenues that would involve cracking scientific problems.
(*which is another thing from hacking to the level of “controlling” the arsenal and being able to retarget it at will, which would probably require a more advanced capability, where the risk from the nuclear avenue might perhaps be redundant compared to risks from other, direct avenues).
Incidentally, at CSER I’ve been working with co-authors on a draft chapter that explores “military AI as cause or compounder of global catastrophic risk”, and one of the avenues also involves discussion of what we call “weapons/arsenal overhang”, so this is an interesting topic that I’d love to discuss more
Ok thanks this makes more sense to me now
Right, that covers hard takeoff or long-con treachery—but there are scenarios where we uncover the risk before strict “prepotence”. And imo we should maintain a distribution over a big set of such scenarios at the moment.
Yeah, understandable but I would also push back. Mining / buying your own uranium and building a centrifuge to enrich it and putting it into a missile is difficult for even rogue nations like Iran. An advanced AI system might just be lines of code in a computer that can use the internet and output text or speech, but with no robotics system to give it physical capacity. From that point of view, building your own nukes seems much more difficult than hacking into an existing ICBM system.
I agree that the current nuclear weapon situation makes AI catastrophe more likely on the margin, and said as much here (The paragraph “You might reply: The thing that went wrong in this scenario is not the out-of-control AGI, it’s the fact that humanity is too vulnerable! And my response is: Why can’t it be both? …”)
That said, I do think the nuclear situation is a rather small effect (on AI risk specifically), in that there are many different paths for an intelligent motivated agent to cause chaos and destruction. Even if triggering nuclear war is the lowest-hanging fruit for a hypothetical future AGI aspiring to destroy humanity (it might or might not be, I dunno), I think other fruits are hanging only slightly higher, like causing extended blackouts, arranging for the release of bio-engineered plagues, triggering non-nuclear great power war (if someday nuclear weapons are eliminated), mass spearphishing / hacking, mass targeted disinformation, etc., even leaving aside more exotic things like nanobots. Solving all these problems would be that much harder (still worth trying!), and anyway we need to solve AI alignment one way or the other, IMO. :)
Gotcha, thanks!