I hope to write a post about this at some point, but since you raise some of these arguments, I think the most important cruxes for a pause are:
It seems like in many people’s models, the reason the “snap back” is problematic is that the productivity of safety research is much higher when capabilities are close to the danger zone, both because the AIs that we’re using to do safety research are better and because the AIs that we’re doing the safety research on are more similar to the ones in the danger zone. If the “snap back” reduces the amount of calendar time during which we think AI safety research will be most productive in exchange for giving us more time overall, this could easily be net negative. On the other hand, a pause might just “snap back” to somewhere on the capabilities graph that’s still outside the danger zone, and lower than it would’ve been without the pause for the reasons you describe.
A huge empirical uncertainty I have is: how elastic is the long-term supply curve of compute? If, on one extreme end, the production of computing hardware for the next 20 years is set in stone, then at the end of the pause there would be a huge jump in how much compute a developer could use to train a model, which seems pretty likely to produce a destabilizing/costly jump. At the other end, if compute supply were very responsive to expected AI progress and a pause would mean a big cut to e.g. Nvidia’s R&D budget and TSMC shelved plans for a leading-node fab or two as a result, the jump would be much less worrying in expectation. I’ve heard that the industry plans pretty far in advance because of how much time and money it takes to build a fab (and how much coordination is required between the different parts of the supply chain), but it seems like at this point a lot of the future expected revenue to be won from designing the next generations of GPUs comes from their usefulness for training huge AI systems, so it seems like there should at least be some marginal reduction in long-term capacity if there were a big regulatory response.
@tlevin I would be interested in you writing up this post, though I’d be even more interested in hearing your thoughts on the regulatory proposal Thomas is proposing.
Note that both of your points seem to be arguing against a pause, whereas my impression is that Thomas’s post focuses more on implementing a national regulatory body.
(I read Thomas’s post as basically saying like “eh, I know there’s an AI pause debate going on, but actually this pause stuff is not as important as getting good policies. Specifically, we should have a federal agency that does licensing for frontier AI systems, hardware monitoring for advanced chips, and tracking of risks. If there’s an AI-related emergency or evidence of imminent danger, then the agency can activate emergency powers to swiftly respond.”
I think the “snap-back” point and the “long-term supply curve of compute” point seem most relevant to a “should we pause?” debate, but they seem less relevant to Thomas’s regulatory body proposal. Let me know if you think I’m missing something, though!)
Any realistic pause would only be lifted once there is a consensus on a potential solution to x-safety (or at least, say, full solutions to all jailbreaks, mechanistic interpretability and alignment up to the (frozen) frontier). If compute limits are in place during the pause, they can gradually be ratcheted up, with evals performed on models trained at each step, to avoid any such sudden snap back.
Thanks for writing this up!
I hope to write a post about this at some point, but since you raise some of these arguments, I think the most important cruxes for a pause are:
It seems like in many people’s models, the reason the “snap back” is problematic is that the productivity of safety research is much higher when capabilities are close to the danger zone, both because the AIs that we’re using to do safety research are better and because the AIs that we’re doing the safety research on are more similar to the ones in the danger zone. If the “snap back” reduces the amount of calendar time during which we think AI safety research will be most productive in exchange for giving us more time overall, this could easily be net negative. On the other hand, a pause might just “snap back” to somewhere on the capabilities graph that’s still outside the danger zone, and lower than it would’ve been without the pause for the reasons you describe.
A huge empirical uncertainty I have is: how elastic is the long-term supply curve of compute? If, on one extreme end, the production of computing hardware for the next 20 years is set in stone, then at the end of the pause there would be a huge jump in how much compute a developer could use to train a model, which seems pretty likely to produce a destabilizing/costly jump. At the other end, if compute supply were very responsive to expected AI progress and a pause would mean a big cut to e.g. Nvidia’s R&D budget and TSMC shelved plans for a leading-node fab or two as a result, the jump would be much less worrying in expectation. I’ve heard that the industry plans pretty far in advance because of how much time and money it takes to build a fab (and how much coordination is required between the different parts of the supply chain), but it seems like at this point a lot of the future expected revenue to be won from designing the next generations of GPUs comes from their usefulness for training huge AI systems, so it seems like there should at least be some marginal reduction in long-term capacity if there were a big regulatory response.
@tlevin I would be interested in you writing up this post, though I’d be even more interested in hearing your thoughts on the regulatory proposal Thomas is proposing.
Note that both of your points seem to be arguing against a pause, whereas my impression is that Thomas’s post focuses more on implementing a national regulatory body.
(I read Thomas’s post as basically saying like “eh, I know there’s an AI pause debate going on, but actually this pause stuff is not as important as getting good policies. Specifically, we should have a federal agency that does licensing for frontier AI systems, hardware monitoring for advanced chips, and tracking of risks. If there’s an AI-related emergency or evidence of imminent danger, then the agency can activate emergency powers to swiftly respond.”
I think the “snap-back” point and the “long-term supply curve of compute” point seem most relevant to a “should we pause?” debate, but they seem less relevant to Thomas’s regulatory body proposal. Let me know if you think I’m missing something, though!)
Any realistic pause would only be lifted once there is a consensus on a potential solution to x-safety (or at least, say, full solutions to all jailbreaks, mechanistic interpretability and alignment up to the (frozen) frontier). If compute limits are in place during the pause, they can gradually be ratcheted up, with evals performed on models trained at each step, to avoid any such sudden snap back.