Not sure why number of steps is important. If we’re talking about very powerful unaligned AI it’s going to wreak havoc in any case. From a longtermist point of view it doesn’t matter if it takes it a day, a month, or a year to do so.
Ah cmon it still shifts the P(doom) distribution a bit.
Consider us having some solid countermeasures with OODA loops of ~days. If we delay doom by y days, then some number of countermeasures can fire where otherwise they wouldn’t get to fire at all.
(Though this assumes an imperfectly timed treacherous turn, before it’s unstoppable.)
Maybe. I was thinking that the point at which a rogue AI is powerful enough to take control of existing nuclear weapons is the point at which we’re already completely screwed, but I could be wrong.
NC3 early warning systems are susceptible to error signals, and the chain of command hasn’t always been v secure (and may not be today), so it wouldn’t necessarily be that hard for a relatively unsophisticated AGI to spoof and trigger a nuclear war:* certainly easier than many other avenues that would involve cracking scientific problems.
(*which is another thing from hacking to the level of “controlling” the arsenal and being able to retarget it at will, which would probably require a more advanced capability, where the risk from the nuclear avenue might perhaps be redundant compared to risks from other, direct avenues).
Incidentally, at CSER I’ve been working with co-authors on a draft chapter that explores “military AI as cause or compounder of global catastrophic risk”, and one of the avenues also involves discussion of what we call “weapons/arsenal overhang”, so this is an interesting topic that I’d love to discuss more
Right, that covers hard takeoff or long-con treachery—but there are scenarios where we uncover the risk before strict “prepotence”. And imo we should maintain a distribution over a big set of such scenarios at the moment.
Yeah, understandable but I would also push back. Mining / buying your own uranium and building a centrifuge to enrich it and putting it into a missile is difficult for even rogue nations like Iran. An advanced AI system might just be lines of code in a computer that can use the internet and output text or speech, but with no robotics system to give it physical capacity. From that point of view, building your own nukes seems much more difficult than hacking into an existing ICBM system.
Not sure why number of steps is important. If we’re talking about very powerful unaligned AI it’s going to wreak havoc in any case. From a longtermist point of view it doesn’t matter if it takes it a day, a month, or a year to do so.
Ah cmon it still shifts the P(doom) distribution a bit.
Consider us having some solid countermeasures with OODA loops of ~days. If we delay doom by y days, then some number of countermeasures can fire where otherwise they wouldn’t get to fire at all.
(Though this assumes an imperfectly timed treacherous turn, before it’s unstoppable.)
Maybe. I was thinking that the point at which a rogue AI is powerful enough to take control of existing nuclear weapons is the point at which we’re already completely screwed, but I could be wrong.
NC3 early warning systems are susceptible to error signals, and the chain of command hasn’t always been v secure (and may not be today), so it wouldn’t necessarily be that hard for a relatively unsophisticated AGI to spoof and trigger a nuclear war:* certainly easier than many other avenues that would involve cracking scientific problems.
(*which is another thing from hacking to the level of “controlling” the arsenal and being able to retarget it at will, which would probably require a more advanced capability, where the risk from the nuclear avenue might perhaps be redundant compared to risks from other, direct avenues).
Incidentally, at CSER I’ve been working with co-authors on a draft chapter that explores “military AI as cause or compounder of global catastrophic risk”, and one of the avenues also involves discussion of what we call “weapons/arsenal overhang”, so this is an interesting topic that I’d love to discuss more
Ok thanks this makes more sense to me now
Right, that covers hard takeoff or long-con treachery—but there are scenarios where we uncover the risk before strict “prepotence”. And imo we should maintain a distribution over a big set of such scenarios at the moment.
Yeah, understandable but I would also push back. Mining / buying your own uranium and building a centrifuge to enrich it and putting it into a missile is difficult for even rogue nations like Iran. An advanced AI system might just be lines of code in a computer that can use the internet and output text or speech, but with no robotics system to give it physical capacity. From that point of view, building your own nukes seems much more difficult than hacking into an existing ICBM system.