By some accounts, growth from 0.0 to 4.0 is infinite growth, which is infinitely faster than Mooreâs law!
More seriously, I didnât really think through precisely whether artificial intelligence could be increasing faster than Mooreâs law. I guess in theory it could. I forgot that Mooreâs law speed actually isnât that impressive on its own. It has to compound over decades to be impressive.
If I eat a sandwich today and eat two sandwiches tomorrow, the growth rate in my sandwich consumption is astronomically faster than Mooreâs law. But what matters is if the growth rate continues and compounds long-term.
The bigger picture is how to measure general intelligence or âfluid intelligenceâ in a way that makes sense. The Elo rating of AlphaGo probably increased faster than Mooreâs law from 2014 to 2017. But we donât see the Elo rating of AlphaGo as a measure of AGI, or else AGI would have already been achieved in 2015.
I think essentially all of these benchmarks and metrics for LLM performance are like the Elo rating of AlphaGo in this respect. They are measuring a narrow skill.
More seriously, I didnât really think through precisely whether artificial intelligence could be increasing faster than Mooreâs law.
Fair enough, but in that case I feel kind of confused about what your statement âProgress does not seem like a fast exponential trend, faster than Mooreâs lawâ was intended to imply.
If the claim you are making is âAGI by 2030 will require some growth faster than Mooreâs lawâ then the good news is that almost everyone agrees with you but the bad news is that everyone already agrees with you so this point is not really cruxy to anyone.
Maybe you have an additional claim like â...and growth faster than mooreâs law is unlikely?â If so, I would encourage you to write that because I think that is the kind of thing that would engage with peopleâs cruxes!
Progress does not seem like a fast exponential trend, faster than Mooreâs law and laying the groundwork for an intelligence explosion. Progress seems actually pretty slow and incremental, with a moderate improvement from GPT-3.5 to GPT-4, and another moderate improvement from GPT-4 to o3-mini. The decline in the costs of running the models or the increase in the compute used to be train models is probably happening faster than Mooreâs law, but not the actual intelligence of the models.
To remove the confusing part about Mooreâs law, I could re-word it like this:
Progress toward AGI does not seem very fast, not fast enough to lay the groundwork for an intelligence explosion within anything like 5 years. Progress seems actually pretty slow and incremental, with a moderate improvement from GPT-3.5 to GPT-4, and another moderate improvement from GPT-4 to o3-mini. The decline in the costs of running the models or the increase in the compute used to be train models is probably very large, but the actual intelligence of the models seems to be increasing only a bit with each new major version.
I think this conveys my meaning better than what I wrote originally, and it avoids getting into the Mooreâs law topic.
The Mooreâs law topic is a bit of an unnecessary rabbit hole. A lot of things increase faster than Mooreâs law during a short window of time, but few increase at a CAGR of 41% (or whatever Mooreâs lawâs CAGR is) for decades. Thereâs all kinds of ways to mis-apply the analogy of Mooreâs law.
People have made jokes about this kind of thing before, like The Economist sarcastically forecasting in 2006 based on then-recent trends that a 14-blade razor would be released by 2010.
I also think of David Deutschâs book The Beginning of Infinity, in which he rails against the practice of uncritically extrapolating past trends forward, and his TED Talk where he does a bit of the same.
By some accounts, growth from 0.0 to 4.0 is infinite growth, which is infinitely faster than Mooreâs law!
More seriously, I didnât really think through precisely whether artificial intelligence could be increasing faster than Mooreâs law. I guess in theory it could. I forgot that Mooreâs law speed actually isnât that impressive on its own. It has to compound over decades to be impressive.
If I eat a sandwich today and eat two sandwiches tomorrow, the growth rate in my sandwich consumption is astronomically faster than Mooreâs law. But what matters is if the growth rate continues and compounds long-term.
The bigger picture is how to measure general intelligence or âfluid intelligenceâ in a way that makes sense. The Elo rating of AlphaGo probably increased faster than Mooreâs law from 2014 to 2017. But we donât see the Elo rating of AlphaGo as a measure of AGI, or else AGI would have already been achieved in 2015.
I think essentially all of these benchmarks and metrics for LLM performance are like the Elo rating of AlphaGo in this respect. They are measuring a narrow skill.
Fair enough, but in that case I feel kind of confused about what your statement âProgress does not seem like a fast exponential trend, faster than Mooreâs lawâ was intended to imply.
If the claim you are making is âAGI by 2030 will require some growth faster than Mooreâs lawâ then the good news is that almost everyone agrees with you but the bad news is that everyone already agrees with you so this point is not really cruxy to anyone.
Maybe you have an additional claim like â...and growth faster than mooreâs law is unlikely?â If so, I would encourage you to write that because I think that is the kind of thing that would engage with peopleâs cruxes!
So, what I originally wrote is:
To remove the confusing part about Mooreâs law, I could re-word it like this:
I think this conveys my meaning better than what I wrote originally, and it avoids getting into the Mooreâs law topic.
The Mooreâs law topic is a bit of an unnecessary rabbit hole. A lot of things increase faster than Mooreâs law during a short window of time, but few increase at a CAGR of 41% (or whatever Mooreâs lawâs CAGR is) for decades. Thereâs all kinds of ways to mis-apply the analogy of Mooreâs law.
People have made jokes about this kind of thing before, like The Economist sarcastically forecasting in 2006 based on then-recent trends that a 14-blade razor would be released by 2010.
I also think of David Deutschâs book The Beginning of Infinity, in which he rails against the practice of uncritically extrapolating past trends forward, and his TED Talk where he does a bit of the same.