One of the three major threads in this post (I think) is feedback loops & takeoff: for safety, causing capabilities to increase more gradually and have more time with more capable systems is important, relative to total time until powerful systems appear. By default, capabilities would increase gradually. A pause would create an “overhang” and would not be sustained forever; when the pause ends, the overhang entails that capabilities increase rapidly.
I kinda agree. I seem to think rapid increase in training compute is less likely, would be smaller, and would be less bad than you do. Some of the larger cruxes:
Magnitude of overhang: it seems the size of the largest training run largely isn’t about the cost of compute. Why hasn’t someone done a billion-dollar LLM training run, why did we only recently break $10M? I don’t know but I’d guess you can’t effectively (i.e. you get sharply diminishing returns for doing more than a couple orders of magnitude more than models that have been around for a while), or it’s hard to get a big cluster to parallelize and so the training run would take years, or something. Relevant meme:
Magnitude of overhang: endogeneity. AI progress improves AI progress, for reasons like Examples of AI Improving AI and normal iterating and learning from experience. This means takeoff is faster than otherwise, especially in no-pause worlds. So a pause makes fast takeoff worse but not as much as we’d naively think.
Badness of overhang: I seem to think total-time is more important relative to time-with-powerful-models than you, such that I’d accept a small overhang in exchange for a moderate amount of timeline. Shrug. This is probably because (a) I’m more pessimistic about alignment than you and (b) I’m more optimistic about current alignment research being useful for aligning powerful AI. Probably it’s not worth
A pause would create an “overhang” and would not be sustained forever
This is too strong. Some pauses could be sustained through AGI, obviating the overhang problem. For example, if you pause slightly below AGI, you get to AGI via algorithmic improvements and inference-time compute increases—the pause doesn’t end and overhang isn’t an issue.
One of the three major threads in this post (I think) is feedback loops & takeoff: for safety, causing capabilities to increase more gradually and have more time with more capable systems is important, relative to total time until powerful systems appear. By default, capabilities would increase gradually. A pause would create an “overhang” and would not be sustained forever; when the pause ends, the overhang entails that capabilities increase rapidly.
I kinda agree. I seem to think rapid increase in training compute is less likely, would be smaller, and would be less bad than you do. Some of the larger cruxes:
Magnitude of overhang: it seems the size of the largest training run largely isn’t about the cost of compute. Why hasn’t someone done a billion-dollar LLM training run, why did we only recently break $10M? I don’t know but I’d guess you can’t effectively (i.e. you get sharply diminishing returns for doing more than a couple orders of magnitude more than models that have been around for a while), or it’s hard to get a big cluster to parallelize and so the training run would take years, or something. Relevant meme:
Magnitude of overhang: endogeneity. AI progress improves AI progress, for reasons like Examples of AI Improving AI and normal iterating and learning from experience. This means takeoff is faster than otherwise, especially in no-pause worlds. So a pause makes fast takeoff worse but not as much as we’d naively think.
Badness of overhang: I seem to think total-time is more important relative to time-with-powerful-models than you, such that I’d accept a small overhang in exchange for a moderate amount of timeline. Shrug. This is probably because (a) I’m more pessimistic about alignment than you and (b) I’m more optimistic about current alignment research being useful for aligning powerful AI. Probably it’s not worth
I discuss my cruxes in Cruxes for overhang (also relevant: Cruxes on US lead for some domestic AI regulation).
This is too strong. Some pauses could be sustained through AGI, obviating the overhang problem. For example, if you pause slightly below AGI, you get to AGI via algorithmic improvements and inference-time compute increases—the pause doesn’t end and overhang isn’t an issue.