Hardware development restriction would be nice, but it’s not necessary for a successful moratorium (at least for the next few years) given already proposed compute governance schemes. There are only a handful of large hardware manufacturers and data centre vendors who would need to be regulated into building in detection and remote kill switches into their products to ensure training runs over a certain threshold of compute aren’t completed. And training FLOP limits could be regularly ratcheted down to account for algorithmic improvements. (Eventually hardware development restrictions would come in once the FLOP limits threaten becoming too accessible/cheap to reach to be easily enforceable otherwise).
Hardware overhang is likely
Not with an indefinite global pause that is only lifted following a global consensus on an alignment solution sufficient for x-safety (and this is the only kind of moratorium that is being seriously discussed as a solution to AI x-risk). I think a lot of objections to Pause are based on the idea that it would be of fixed time limit. This is obviously unrealistic—when has there ever been an international treaty or moratorium that had a fixed expiry date?
I hope you’ll agree that my predictions are plausible, and are grounded in how humans and governments have behaved historically.
This does not seem very like how {nuclear, bio, chemical} weapons treaties or CFC or climate change treaties have gone.
One thing you haven’t factored is a taboo forming on AGI/ASI development that would accompany any Pause. This would overcome a lot of your objections / failure modes. Where are all the non-human-cloning-ban countries?
Addressing some of your objections:
Hardware development restriction would be nice, but it’s not necessary for a successful moratorium (at least for the next few years) given already proposed compute governance schemes. There are only a handful of large hardware manufacturers and data centre vendors who would need to be regulated into building in detection and remote kill switches into their products to ensure training runs over a certain threshold of compute aren’t completed. And training FLOP limits could be regularly ratcheted down to account for algorithmic improvements. (Eventually hardware development restrictions would come in once the FLOP limits threaten becoming too accessible/cheap to reach to be easily enforceable otherwise).
Not with an indefinite global pause that is only lifted following a global consensus on an alignment solution sufficient for x-safety (and this is the only kind of moratorium that is being seriously discussed as a solution to AI x-risk). I think a lot of objections to Pause are based on the idea that it would be of fixed time limit. This is obviously unrealistic—when has there ever been an international treaty or moratorium that had a fixed expiry date?
This does not seem very like how {nuclear, bio, chemical} weapons treaties or CFC or climate change treaties have gone.
One thing you haven’t factored is a taboo forming on AGI/ASI development that would accompany any Pause. This would overcome a lot of your objections / failure modes. Where are all the non-human-cloning-ban countries?