Great set of posts (including the âhow farâ and âhow suddenâ related ones). I only skimmed the parts I had read drafts of, but still have a few comments, mostly minor:
1. Accelerating progress framing
We define âaccelerating AI progressâ as âeach increment of capability advancement (e.g. GPT-3 â GPT-4) happens more quickly than the lastâ.
I am a bit skeptical of this definition, both because it is underspecified, and Iâm not sure it is pointing at the most important thing.
Underspecified: how many GPT jumps need to be in the âeach quicker than the lastâ regime? This seems more than just a semantic quibble, as clearly the one-time speedup leads to at least one GPT jump being faster, and the theoretical limits lead to this eventually stopping, but Iâm not sure where within this you want to call it âacceleratingâ.
Framing: Basically, we are trying to condense a whole graph into a few key numbers, so this might be quite loss-y and we need to focus on the variables that are most strategically important, which I think are:
Timeline: date that transition period starts
Suddenness: time in transition period
Plateau height: in effective compute, defining the plateau as when rate of progress drops back below 2025 levels.
Plateau date: how long it takes to get there.
Iâm not sure there is an important further question of whether the line is curving up or down between the transition period and the plateau (or more precisely, when it transitions from curving up (as in the transition period) to curving down (as in the plateau)). I suppose âacceleratingâ could include plateauing quite quickly, and decelerating could include still going very fast and reaching a very high plateau quickly, which to most people wouldnât intuitively feel like âdecelerationâ.
2. Max rate of change
Theoretical limits for the speed of progress are 100X as fast as recent progress.
It would be good to flag in the main text that the justification for this is in Appendix 2 (initially I thought it was a bare asertion). Also, it is interesting that in @kokotajlodâs scenario the âwildly superintelligentâ AI maxes out at 1 million-fold AI R&D speedup; I commented to them on a draft that this seemed implausibly high to me. I have no particular take on whether 100x is too low or too high as the theoretical max, but it would be interesting to work out why there is this Forethought vs AI Futures difference.
3. Error in GPT OOMS calculations
Algorithmic improvements compound multiplicatively rather than additively, so the formula in column G I think should be 3^years rather than 3*years?
This also clears up the current mismatch between columns G and H. Most straightforward would be for column H to be log10(G), same as column F. But since log(a^b) = b*log(a), once you make the correction to column G you get out column H = log(3^years) = years * log(3) = 0.48*years. Which is close to what you currently have of 0.4 * years, I assume there was just a rounding error somewhere.
This wonât end up changing the main results though.
4. Physical limits
Regarding the effective physical limits of each feedback loop, perhaps it is worth noting that your estimates are very well grounded and high-confidence for the chip production feedback loop as we know more or less exactly the energy output of the sun. But the other two are super speculative. Which is fine, they are just quite different types of estimates, so we should remember to rely far less on them.
5. End of the transition period
Currently, this is set at when AIs are almost as productive (9/â10) as humans, but it would make more sense to me to end it when AIs are markedly superior to humans, e.g. 10x.
Maybe I am misunderstanding elasticities though, I only have a lay non-economistâs grasp of them.
Overall it might be more intuitive to define the transition period in terms of how useful one additional human researcher vs AI researcher is, from human being 10x better to the AI being 10x better.
Defining what âone AI researcherâ is could be tricky, maybe we could use the pace of human thought in tokens per second as a way to standardise.
Great set of posts (including the âhow farâ and âhow suddenâ related ones). I only skimmed the parts I had read drafts of, but still have a few comments, mostly minor:
1. Accelerating progress framing
I am a bit skeptical of this definition, both because it is underspecified, and Iâm not sure it is pointing at the most important thing.
Underspecified: how many GPT jumps need to be in the âeach quicker than the lastâ regime? This seems more than just a semantic quibble, as clearly the one-time speedup leads to at least one GPT jump being faster, and the theoretical limits lead to this eventually stopping, but Iâm not sure where within this you want to call it âacceleratingâ.
Framing: Basically, we are trying to condense a whole graph into a few key numbers, so this might be quite loss-y and we need to focus on the variables that are most strategically important, which I think are:
Timeline: date that transition period starts
Suddenness: time in transition period
Plateau height: in effective compute, defining the plateau as when rate of progress drops back below 2025 levels.
Plateau date: how long it takes to get there.
Iâm not sure there is an important further question of whether the line is curving up or down between the transition period and the plateau (or more precisely, when it transitions from curving up (as in the transition period) to curving down (as in the plateau)). I suppose âacceleratingâ could include plateauing quite quickly, and decelerating could include still going very fast and reaching a very high plateau quickly, which to most people wouldnât intuitively feel like âdecelerationâ.
2. Max rate of change
It would be good to flag in the main text that the justification for this is in Appendix 2 (initially I thought it was a bare asertion). Also, it is interesting that in @kokotajlodâs scenario the âwildly superintelligentâ AI maxes out at 1 million-fold AI R&D speedup; I commented to them on a draft that this seemed implausibly high to me. I have no particular take on whether 100x is too low or too high as the theoretical max, but it would be interesting to work out why there is this Forethought vs AI Futures difference.
3. Error in GPT OOMS calculations
Algorithmic improvements compound multiplicatively rather than additively, so the formula in column G I think should be 3^years rather than 3*years?
This also clears up the current mismatch between columns G and H. Most straightforward would be for column H to be log10(G), same as column F. But since log(a^b) = b*log(a), once you make the correction to column G you get out column H = log(3^years) = years * log(3) = 0.48*years. Which is close to what you currently have of 0.4 * years, I assume there was just a rounding error somewhere.
This wonât end up changing the main results though.
4. Physical limits
Regarding the effective physical limits of each feedback loop, perhaps it is worth noting that your estimates are very well grounded and high-confidence for the chip production feedback loop as we know more or less exactly the energy output of the sun. But the other two are super speculative. Which is fine, they are just quite different types of estimates, so we should remember to rely far less on them.
5. End of the transition period
Currently, this is set at when AIs are almost as productive (9/â10) as humans, but it would make more sense to me to end it when AIs are markedly superior to humans, e.g. 10x.
Maybe I am misunderstanding elasticities though, I only have a lay non-economistâs grasp of them.
Overall it might be more intuitive to define the transition period in terms of how useful one additional human researcher vs AI researcher is, from human being 10x better to the AI being 10x better.
Defining what âone AI researcherâ is could be tricky, maybe we could use the pace of human thought in tokens per second as a way to standardise.
(Finally, Fn2 is missing a link.)