2. Increasing the chance of a “fast takeoff” in which one or a handful of AIs rapidly and discontinuously become more capable, concentrating immense power in their hands.
You don’t actually discuss concentrating power, I think. (You just say fast takeoff is bad because it makes alignment harder, which is the same as your 1.)
But failing to pause hardware R&D creates a serious problem because, even if we pause the software side of AI capabilities, existing models will continue to get more powerful as hardware improves. Language models are much stronger when they’re allowed to “brainstorm” many ideas, compare them, and check their own work— see the Graph of Thoughts paper for a recent example. Better hardware makes these compute-heavy inference techniques cheaper and more effective.
Two clarifications (I know you know—for the others’ benefit):
(a) Software progress includes training-time and inference-time improvements like better prompting or agent scaffolding. You’re considering a pause on training-time improvements. “Existing models will continue to get more powerful” as inference-time compute and software improve.
(b) Failing to pause hardware R&D may create training-time compute overhang. I agree with you that existing models will be able to leverage better hardware at inference time, so it probably doesn’t create a big inference-time compute overhang. So failing to pause is not “a serious problem” in the context of inference-time compute, I think.
Good post.
Small things:
You don’t actually discuss concentrating power, I think. (You just say fast takeoff is bad because it makes alignment harder, which is the same as your 1.)
Two clarifications (I know you know—for the others’ benefit):
(a) Software progress includes training-time and inference-time improvements like better prompting or agent scaffolding. You’re considering a pause on training-time improvements. “Existing models will continue to get more powerful” as inference-time compute and software improve.
(b) Failing to pause hardware R&D may create training-time compute overhang. I agree with you that existing models will be able to leverage better hardware at inference time, so it probably doesn’t create a big inference-time compute overhang. So failing to pause is not “a serious problem” in the context of inference-time compute, I think.