a major paradigm shift that enables massive progress without significant increases in effective compute
I am confused hereâI thought by definition if capabilities have gone up, that is an increase in effective compute? As to estimate effective compute we just ask how much more compute would you need to use in the old paradigm to get the same performance. Maybe you mean the answer is âinfinityâ as in no amount of compute in the old paradigm would get performance as good as after the paradigm shift?
Iâm saying that if there is such a new paradigm then we could have >10 years worth of AI progress at rates of 2020-5, and >10 OOMs of effective compute growth according to the old paradigm. But that, perhaps, within the new paradigm these gains are achieved while the efficiency of AI algs only increases slightly. E.g. a new paradigm where each doubling of compute increases capabilities as much as 1000X does today. Then, measured within the new paradigm, the algorithmic progress might seem like just a couple of OOMs so âeffective computeâ isnât rising fast, but relative to the old paradigm progress (and effective compute growth) is massive.
Fwiw, this X thread discusses the units Iâm using for âyear of Ai progressâ, and Eli Lifland gives a reasonable alternative. Either work as a way to understand the framework.
I am confused hereâI thought by definition if capabilities have gone up, that is an increase in effective compute? As to estimate effective compute we just ask how much more compute would you need to use in the old paradigm to get the same performance. Maybe you mean the answer is âinfinityâ as in no amount of compute in the old paradigm would get performance as good as after the paradigm shift?
Thanks, good Q.
Iâm saying that if there is such a new paradigm then we could have >10 years worth of AI progress at rates of 2020-5, and >10 OOMs of effective compute growth according to the old paradigm. But that, perhaps, within the new paradigm these gains are achieved while the efficiency of AI algs only increases slightly. E.g. a new paradigm where each doubling of compute increases capabilities as much as 1000X does today. Then, measured within the new paradigm, the algorithmic progress might seem like just a couple of OOMs so âeffective computeâ isnât rising fast, but relative to the old paradigm progress (and effective compute growth) is massive.
Fwiw, this X thread discusses the units Iâm using for âyear of Ai progressâ, and Eli Lifland gives a reasonable alternative. Either work as a way to understand the framework.