If you don’t think misalignment automatically equals extinction, then the argument doesn’t work. The neutral world is now competing with “neutral world where the software fucks up and kills people sometimes”, which seems to be worse.
That is fair. I still think the idea that aligned superintelligent AI in the wrong hands can be very bad may be under-appreciated. The implication is that something like moral circle expansion seems very important at the moment to help mitigate these risks. And of course work to ensure that countries with better values win the race to powerful AI.
If you don’t think misalignment automatically equals extinction, then the argument doesn’t work. The neutral world is now competing with “neutral world where the software fucks up and kills people sometimes”, which seems to be worse.
That is fair. I still think the idea that aligned superintelligent AI in the wrong hands can be very bad may be under-appreciated. The implication is that something like moral circle expansion seems very important at the moment to help mitigate these risks. And of course work to ensure that countries with better values win the race to powerful AI.