Fwiw, this section made me feel like it should be thought through more deeply:
Luisa Rodriguez: [...] How confident are you that leaders in the countries that are set up to race and are already racing a little bit are going to see this as close to existential?
Daniel Kokotajlo: I think it will be existential if one side is racing and the other side isn’t. And even if they don’t see that yet, by the time they have superintelligences, then they will see it — because the superintelligences, being superintelligent, will be able to correctly identify this strategic consideration, and probably communicate that to the humans around them.
Luisa Rodriguez: Right. Once you have AGI, the AGI is like, “This is existential. We should do this big wartime effort to create a robot economy that’s going to give us this big advantage.”
Daniel Kokotajlo: That’s right. [...]
Some random thoughts:
Leaders already abstractly believe that AI is existentially important (e.g. Putin since 2017 apparently) and don’t really behave accordingly.
I think it’s very likely that there are already advisors in China and the U.S. that frequently argue for the existential importance of AI / ASI to the leader or people close to them. It’s unclear to me that e.g. Trump would trust an ASI much more than personal advisors.
I think it’s unclear that an ASI, especially from the current paradigm, will advise to race, and probably will strongly urge for cooperation and at most limited racing, e.g. defensive acceleration.
Thanks for the interesting interview!
Fwiw, this section made me feel like it should be thought through more deeply:
Some random thoughts:
Leaders already abstractly believe that AI is existentially important (e.g. Putin since 2017 apparently) and don’t really behave accordingly.
I think it’s very likely that there are already advisors in China and the U.S. that frequently argue for the existential importance of AI / ASI to the leader or people close to them. It’s unclear to me that e.g. Trump would trust an ASI much more than personal advisors.
I think it’s unclear that an ASI, especially from the current paradigm, will advise to race, and probably will strongly urge for cooperation and at most limited racing, e.g. defensive acceleration.