I am concerned that at some point in the next few decades, well-meaning and smart people who work on AGI research and development, alignment and governance will become convinced they are in an existential race with an unsafe and misuse-prone opponent [emphasis added].
Most people who’ve thought about AI risk, I think, would agree that most of the risk comes not from misuse risk, but from accident risk (i.e., not realizing the prepotent AI one is deploying is misaligned).[1] Therefore, being convinced the opponent is misuse-prone is actually not necessary, I don’t think, to believe one is in an existential race. All that’s necessary is to believe there is an opponent at all.
I’d define a prepotent AI system (or cooperating collection of systems) as one that cannot be controlled by humanity, and which is at least as powerful as humanity as a whole with respect to shaping the world. (By this definition, such an AI system need not be superintelligent, or even generally intelligent or economically transformative. It may have powerful capabilities in a narrow domain that enable prepotence, such as technological autonomy, replication speed, or social manipulation.)
Thanks for this very illuminating post.
One thing:
Most people who’ve thought about AI risk, I think, would agree that most of the risk comes not from misuse risk, but from accident risk (i.e., not realizing the prepotent AI one is deploying is misaligned).[1] Therefore, being convinced the opponent is misuse-prone is actually not necessary, I don’t think, to believe one is in an existential race. All that’s necessary is to believe there is an opponent at all.
I’d define a prepotent AI system (or cooperating collection of systems) as one that cannot be controlled by humanity, and which is at least as powerful as humanity as a whole with respect to shaping the world. (By this definition, such an AI system need not be superintelligent, or even generally intelligent or economically transformative. It may have powerful capabilities in a narrow domain that enable prepotence, such as technological autonomy, replication speed, or social manipulation.)