(“AI can have bad consequences” as a motivation for AI safety--> Yes, but AI can have bad consequences in meaningfully different ways!)
Here’s some frame confusion that I see a lot, that I think leads to confused intuitions (especially when trying to reason about existential risk from advanced AI, as opposed to today’s systems):
1. There’s (weapons) -- tech like nuclear weapons or autonomous weapons that if used correctly involve people dying. (Tech like this exists)
2. There’s (misuse) -- tech was intentioned to be anywhere from beneficial <> neutral <> seems high on offense-defense balance, but it wasn’t designed for harm. Examples here include social media, identification systems, language models, surveillance systems. (Tech like this exists)
3. There’s (advanced AI pursuing instrumental incentives --> causing existential risk), which is not about misuse, it’s about the *system itself* being an optimizer and seeking power (humans are not the problem here, the AI itself is, once the AI is sufficiently advanced). (Tech like this does not exist)
You can say “AI is bad” for all of them, and they’re all problems, but they’re different problems and should be thought of separately. (1) is a problem (autonomous weapons is the AI version of it) but is pretty independent from (3). Technical AI safety discussion is mostly about the power-seeking agent issue (3). (2) is a problem all the time for all tech (though some tech lends itself more to this than others). They’re all going to need to get solved, but at least (1) and (2) are problems humanity has any experience with (and so we have at least some structures in place to deal with them, and people are aware these are problems).
(“AI can have bad consequences” as a motivation for AI safety--> Yes, but AI can have bad consequences in meaningfully different ways!)
Here’s some frame confusion that I see a lot, that I think leads to confused intuitions (especially when trying to reason about existential risk from advanced AI, as opposed to today’s systems):
1. There’s (weapons) -- tech like nuclear weapons or autonomous weapons that if used correctly involve people dying. (Tech like this exists)
2. There’s (misuse) -- tech was intentioned to be anywhere from beneficial <> neutral <> seems high on offense-defense balance, but it wasn’t designed for harm. Examples here include social media, identification systems, language models, surveillance systems. (Tech like this exists)
3. There’s (advanced AI pursuing instrumental incentives --> causing existential risk), which is not about misuse, it’s about the *system itself* being an optimizer and seeking power (humans are not the problem here, the AI itself is, once the AI is sufficiently advanced). (Tech like this does not exist)
You can say “AI is bad” for all of them, and they’re all problems, but they’re different problems and should be thought of separately. (1) is a problem (autonomous weapons is the AI version of it) but is pretty independent from (3). Technical AI safety discussion is mostly about the power-seeking agent issue (3). (2) is a problem all the time for all tech (though some tech lends itself more to this than others). They’re all going to need to get solved, but at least (1) and (2) are problems humanity has any experience with (and so we have at least some structures in place to deal with them, and people are aware these are problems).