I tweeted about this fairly breathlessly, but congratulations on this—its a really important bit of research. Not to get too carried away, but this plausibly could prove to be one of the most important AI governance reports of the year, even though its only February. I’m very excited by it.
The reason is: if its right, then big AI models will cost $100m+ *each* to train by 2030. So forget about academia, start-ups, open-source collectives or individuals: they can’t keep up! For good or ill, Big Tech will be the only game in town.
This has massive implications for the research field, business models, company strategy investors, regulation, antitrust, government investment/industrial strategy, AI governance, threat models, arms control etc.
I just want to add caution on taking the extrapolations too seriously. The linear extrapolation is not my all-things-considered view of what is going to happen, and the shaded region is just the uncertainty in the linear regression trendline rather than my subjective uncertainty in the estimates.
I agree with you inasmuch as I expect the initial costs of state-of-the-art models to get well out of reach for actors other than big tech (if we include labs with massive investment like OpenAI), and states, by 2030. I still have significant uncertainty about this though. Plausibly, the biggest players in AI won’t be willing to spend $100M just on the computation for a final training run as soon as 2030. We still don’t have a great understanding of what hardware and software progress will be like in future (though Epoch hasworked on this). Maybe efficiency improves faster than expected and/or there just won’t be worthwhile gains from spending so much in order to compete.
Also, I’d like to be clear about what it means to “keep up”. I expect those lower-resourced types of actors won’t keep up in the sense that they won’t be the first to advance state-of-the-art on the most important AI capabilities. But the cost of a given ML system falls over time and that is a big driver of how AI capabilities diffuse.
I tweeted about this fairly breathlessly, but congratulations on this—its a really important bit of research. Not to get too carried away, but this plausibly could prove to be one of the most important AI governance reports of the year, even though its only February. I’m very excited by it.
The reason is: if its right, then big AI models will cost $100m+ *each* to train by 2030. So forget about academia, start-ups, open-source collectives or individuals: they can’t keep up! For good or ill, Big Tech will be the only game in town.
This has massive implications for the research field, business models, company strategy investors, regulation, antitrust, government investment/industrial strategy, AI governance, threat models, arms control etc.
Thanks Haydn!
I just want to add caution on taking the extrapolations too seriously. The linear extrapolation is not my all-things-considered view of what is going to happen, and the shaded region is just the uncertainty in the linear regression trendline rather than my subjective uncertainty in the estimates.
I agree with you inasmuch as I expect the initial costs of state-of-the-art models to get well out of reach for actors other than big tech (if we include labs with massive investment like OpenAI), and states, by 2030. I still have significant uncertainty about this though. Plausibly, the biggest players in AI won’t be willing to spend $100M just on the computation for a final training run as soon as 2030. We still don’t have a great understanding of what hardware and software progress will be like in future (though Epoch has worked on this). Maybe efficiency improves faster than expected and/or there just won’t be worthwhile gains from spending so much in order to compete.
Also, I’d like to be clear about what it means to “keep up”. I expect those lower-resourced types of actors won’t keep up in the sense that they won’t be the first to advance state-of-the-art on the most important AI capabilities. But the cost of a given ML system falls over time and that is a big driver of how AI capabilities diffuse.