You are assuming that AI could be massively economically beneficial (significantly) before it causes our extinction (or at the least, a global catastrophe). I don’t think this is likely, and this defeats a lot of your opposition to an indefinite pause.
We need such a pause because no one can wield the technology safely. It’s not a case of restraint from economic competition and wealth generation, it’s a case of restraint from suicide-omnicide (which should be much easier!)
any nation could decide to break the agreement and develop AI on their own, becoming incredibly rich as a result – perhaps even richer than the entire rest of the world combined within a decade.
...
As a reminder, in order to be successful, attempts to forestall both hardware progress and algorithmic progress would need to be stronger than the incentives for nations and actors within nations to deviate from the international consensus, develop AI, and become immensely wealthy as a result.
This is assuming the AI wouldn’t just end the world. The reason for the Pause is that it likely would. If a country was able to become rich like this from AI (without ending the world), it would mean that they’ve basically solved the alignment (x-safety) problem. If this was the case, then the reason for the indefinite pause would no longer exist!
plausibly the only way we could actually sustain an indefinite pause on AI for more than a few decades is by constructing a global police state.
Assuming the world accepts the reason for the pause being that the default outcome of AGI is extinction, then this wouldn’t be necessary. A strong enough taboo would emerge around AGI development. How many human clones have ever been born in our current (non-police-state) world?
When we ask GPT-4 to help us, it does not generally yield bad outcomes as a result of severe value misspecification
Generally is the operative word here. In the limit of superintelligence, unless it never yields bad outcomes, we’re all dead.
It is noteworthy that humans are alreadycapable of deceiving others about their intentions; indeed, people do that all the time. And yet that fact alone does not yet appear to have caused an existential catastrophe for humans who are powerless.
People get killed by sociopaths all the time! And there are plenty of would be world-ending-button pressers if they had the option.
Even if AIs end up not caring much for humans, it is dubious that they would decide to kill all of us. As Robin Hanson has argued, the primary motives for rogue AIs would likely be to obtain freedom – perhaps the right to own property and to choose their own employment – rather than to kill all humans.
This seems very anthropomorphising, and ignores the possibilities of recursive capability improvements, foom, superintelligence, convergent instrumental goals, arbitrary terminal goals resultant from inner alignment failure, misuse risk, and multi-agent coordination failure (i.e most of the reasons for AI x-risk being significant, which justify an indefinite pause).
While I still believe Bostrom’s argument has some intuitive plausibility, I think it is wrong for EAs to put a ton of weight on it, and confidently reject alternative perspectives. Pushing for an indefinite pause on the basis of these premises seems to be similar to the type of reasoning that Toby Ord has argued against in his EA global talk, and the reasoning that Holden Karnofsky cautioned against in his essay on the perils of maximization. A brazen acceptance of premise 1 might have even imperiled our own community.
We don’t need to rely on these premises. The default outcome of AGI is doom. To avoid near certain extinction, we need an indefinite AI pause.
absent an incredible breakthrough in AI interpretability, if we require that AI companies “prove” that their systems are safe before they are released, I do not think that this standard will be met in six months, and I am doubtful that it could be met in decades – or perhaps even centuries.
If that’s what it takes, then so be it. Much better than extinction.
You are assuming that AI could be massively economically beneficial (significantly) before it causes our extinction (or at the least, a global catastrophe). I don’t think this is likely, and this defeats a lot of your opposition to an indefinite pause.
If you don’t think AI will be economically significant before extinction, I’m curious whether you’d say that your view has been falsified if AI raises economic growth rates in the US to 5, 8, or 10% without us all dying. At what what point would you say that your model here was wrong?
(This isn’t a complete reply to your comment. I appreciate your good-faith engagement with my thesis.)
I don’t think AI could raise growth rates in the US >10% (annualised) for more than a year before rapid improvement in AI capabilities kicks in (from AI-based AI engineering speeding things up) and chaos ensues shortly (days—months) after (global catastrophe at minimum, probably extinction).
Assuming the world accepts the reason for the pause being that the default outcome of AGI is extinction, then this wouldn’t be necessary. A strong enough taboo would emerge around AGI development. How many human clones have ever been born in our current (non-police-state) world?
This won’t address all the arguments in your comment but I have a few things to say in response to this point.
I agree it’s possible that we could just get a very long taboo on AI and halt its development for many decades without a world government to enforce the ban. That doesn’t seem out of the question.
However, it also doesn’t seem probable to me. Here are my reasons:
AGI is something that several well-funded companies are already trying hard to do. I don’t think that was ever true of human cloning (though I could be wrong).
I looked it up and my impression is that it might cost tens of millions of dollars to clone a single human, whereas in the post I argued that AGI will eventually be possible to train with only about 1 million dollars. More importantly, after that, you don’t need to train the AI again. You can just copy the AGI to other hardware. Therefore, it seems that you might really only need one rich person to do it once to get the benefits. That seems like a much lower threshold than human cloning, although I don’t know all the details.
The payoff for building (aligned) AGI is probably much greater than human cloning, and it also comes much sooner.
The underlying tech that allows you to build AGI is shared by other things that don’t seem to have any taboos at all. For example, GPUs are needed for video games. The taboo would need to be strong enough that we’d need to also ban a ton of other things that people currently think are fine.
AGI is just software, and seems harder to build a taboo around compared to human cloning. I don’t think many people have a disgust reaction to GPT-4, for example.
Finally, I doubt there will ever be a complete global consensus that AI is existentially unsafe, since the arguments are speculative, and even unaligned AI will appear “aligned” in the short term if only to trick us. The idea that unaligned AIs might fool us is widely conceded among AI safety researchers, and so I suspect you agree too.
AGI is something that several well-funded companies are already trying hard to do. I don’t think that was ever true of human cloning (though I could be wrong).
Eugenics was quite popular in polite society, at least until the Nazis came along.
The underlying tech that allows you to build AGI is shared by other things that don’t seem to have any taboos at all. For example, GPUs are needed for video games. The taboo would need to be strong enough that we’d need to also ban a ton of other things that people currently think are fine.
You only need to ban huge concentrations of GPUs. At least initially. By the time training run FLOP limits are reduced sufficiently because of algorithmic improvement, we will probably have arrested further hardware development as a measure to deal with it. So individual consumers would not be impacted for a long time (plenty of time for a taboo to settle into acceptance of reduced personal compute allowance).
AGI is just software, and seems harder to build a taboo around compared to human cloning. I don’t think many people have a disgust reaction to GPT-4, for example.
They might once multimodal foundation models are controlling robots that can do their jobs (a year or two’s time?)
Finally, I doubt there will ever be a complete global consensus that AI is existentially unsafe, since the arguments are speculative, and even unaligned AI will appear “aligned” in the short term if only to trick us.
Yes, this is a massive problem. It’s like asking for a global lockdown to prevent Covid spread in December 2019, before the bodies started piling up. Let’s hope it doesn’t come to needing a “warning shot” (global catastrophe with many casualties) before we get the necessary regulation of AI. Especially since we may well not get one and instead face unstoppable extinction.
You are assuming that AI could be massively economically beneficial (significantly) before it causes our extinction (or at the least, a global catastrophe). I don’t think this is likely, and this defeats a lot of your opposition to an indefinite pause.
We need such a pause because no one can wield the technology safely. It’s not a case of restraint from economic competition and wealth generation, it’s a case of restraint from suicide-omnicide (which should be much easier!)
This is assuming the AI wouldn’t just end the world. The reason for the Pause is that it likely would. If a country was able to become rich like this from AI (without ending the world), it would mean that they’ve basically solved the alignment (x-safety) problem. If this was the case, then the reason for the indefinite pause would no longer exist!
Assuming the world accepts the reason for the pause being that the default outcome of AGI is extinction, then this wouldn’t be necessary. A strong enough taboo would emerge around AGI development. How many human clones have ever been born in our current (non-police-state) world?
Generally is the operative word here. In the limit of superintelligence, unless it never yields bad outcomes, we’re all dead.
People get killed by sociopaths all the time! And there are plenty of would be world-ending-button pressers if they had the option.
This seems very anthropomorphising, and ignores the possibilities of recursive capability improvements, foom, superintelligence, convergent instrumental goals, arbitrary terminal goals resultant from inner alignment failure, misuse risk, and multi-agent coordination failure (i.e most of the reasons for AI x-risk being significant, which justify an indefinite pause).
We don’t need to rely on these premises. The default outcome of AGI is doom. To avoid near certain extinction, we need an indefinite AI pause.
If that’s what it takes, then so be it. Much better than extinction.
If you don’t think AI will be economically significant before extinction, I’m curious whether you’d say that your view has been falsified if AI raises economic growth rates in the US to 5, 8, or 10% without us all dying. At what what point would you say that your model here was wrong?
(This isn’t a complete reply to your comment. I appreciate your good-faith engagement with my thesis.)
I don’t think AI could raise growth rates in the US >10% (annualised) for more than a year before rapid improvement in AI capabilities kicks in (from AI-based AI engineering speeding things up) and chaos ensues shortly (days—months) after (global catastrophe at minimum, probably extinction).
This won’t address all the arguments in your comment but I have a few things to say in response to this point.
I agree it’s possible that we could just get a very long taboo on AI and halt its development for many decades without a world government to enforce the ban. That doesn’t seem out of the question.
However, it also doesn’t seem probable to me. Here are my reasons:
AGI is something that several well-funded companies are already trying hard to do. I don’t think that was ever true of human cloning (though I could be wrong).
I looked it up and my impression is that it might cost tens of millions of dollars to clone a single human, whereas in the post I argued that AGI will eventually be possible to train with only about 1 million dollars. More importantly, after that, you don’t need to train the AI again. You can just copy the AGI to other hardware. Therefore, it seems that you might really only need one rich person to do it once to get the benefits. That seems like a much lower threshold than human cloning, although I don’t know all the details.
The payoff for building (aligned) AGI is probably much greater than human cloning, and it also comes much sooner.
The underlying tech that allows you to build AGI is shared by other things that don’t seem to have any taboos at all. For example, GPUs are needed for video games. The taboo would need to be strong enough that we’d need to also ban a ton of other things that people currently think are fine.
AGI is just software, and seems harder to build a taboo around compared to human cloning. I don’t think many people have a disgust reaction to GPT-4, for example.
Finally, I doubt there will ever be a complete global consensus that AI is existentially unsafe, since the arguments are speculative, and even unaligned AI will appear “aligned” in the short term if only to trick us. The idea that unaligned AIs might fool us is widely conceded among AI safety researchers, and so I suspect you agree too.
Eugenics was quite popular in polite society, at least until the Nazis came along.
You only need to ban huge concentrations of GPUs. At least initially. By the time training run FLOP limits are reduced sufficiently because of algorithmic improvement, we will probably have arrested further hardware development as a measure to deal with it. So individual consumers would not be impacted for a long time (plenty of time for a taboo to settle into acceptance of reduced personal compute allowance).
They might once multimodal foundation models are controlling robots that can do their jobs (a year or two’s time?)
Yes, this is a massive problem. It’s like asking for a global lockdown to prevent Covid spread in December 2019, before the bodies started piling up. Let’s hope it doesn’t come to needing a “warning shot” (global catastrophe with many casualties) before we get the necessary regulation of AI. Especially since we may well not get one and instead face unstoppable extinction.