Youāre talking about research rather than scaling here, right? Do you think there is more funding for fundamental AI research now than in 2020? What about for non-LLM fundamental AI research?
Based on our compute and cost estimates for OpenAIās released models from Q2 2024 through Q1 2025, the majority of OpenAIās R&D compute in 2024 was likely allocated to research, experimental training runs, or training runs for unreleased models, rather than the final, primary training runs of released models like GPT-4.5, GPT-4o, and o3.
Thatās kind of interesting in its own right, but I wouldnāt say that money allocated toward training compute for LLMs is the same idea as money allocated to fundamental AI research, if thatās what you were intending to say.
Itās uncontroversial that OpenAI spends a lot on research, but Iām trying to draw a distinction between fundamental research, which, to me, connotes things that are more risky, uncertain, speculative, explorative, and may take a long time to pay off, and research that can be quickly productized.
I donāt understand the details of what Epoch AI is trying to say, but I would be curious to learn.
Do unreleased models include as-yet unreleased models such as GPT-5? (The timeframe is 2024 and OpenAI didnāt release GPT-5 until 2025.) Would it also include o4? (Is there still going to be an o4?) Or is it specifically models that are never intended to be released? Iām guessing itās just everything that hasnāt been released yet, since I donāt know how Epoch AI would have any insight into what OpenAI intends to release or not.
Iām also curious how much trial and error goes into training for LLMs. Does OpenAI often abort training runs or find the results to be disappointing? How many partial or full training runs go into training one model? For example, what percentage of the overall cost is the $400 million estimated for the final training run of GPT-4.5? 100%? 90%? 50%? 10%?
Overall, this estimate from Epoch AI doesnāt seem to tell us much about what amount of money or compute OpenAI is allocating to fundamental research vs. R&D that can quickly be productized.
Most of OpenAIās 2024 compute went to experiments
This is what Epoch AI says about its estimates:
Thatās kind of interesting in its own right, but I wouldnāt say that money allocated toward training compute for LLMs is the same idea as money allocated to fundamental AI research, if thatās what you were intending to say.
Itās uncontroversial that OpenAI spends a lot on research, but Iām trying to draw a distinction between fundamental research, which, to me, connotes things that are more risky, uncertain, speculative, explorative, and may take a long time to pay off, and research that can be quickly productized.
I donāt understand the details of what Epoch AI is trying to say, but I would be curious to learn.
Do unreleased models include as-yet unreleased models such as GPT-5? (The timeframe is 2024 and OpenAI didnāt release GPT-5 until 2025.) Would it also include o4? (Is there still going to be an o4?) Or is it specifically models that are never intended to be released? Iām guessing itās just everything that hasnāt been released yet, since I donāt know how Epoch AI would have any insight into what OpenAI intends to release or not.
Iām also curious how much trial and error goes into training for LLMs. Does OpenAI often abort training runs or find the results to be disappointing? How many partial or full training runs go into training one model? For example, what percentage of the overall cost is the $400 million estimated for the final training run of GPT-4.5? 100%? 90%? 50%? 10%?
Overall, this estimate from Epoch AI doesnāt seem to tell us much about what amount of money or compute OpenAI is allocating to fundamental research vs. R&D that can quickly be productized.