Based on our compute and cost estimates for OpenAI’s released models from Q2 2024 through Q1 2025, the majority of OpenAI’s R&D compute in 2024 was likely allocated to research, experimental training runs, or training runs for unreleased models, rather than the final, primary training runs of released models like GPT-4.5, GPT-4o, and o3.
That’s kind of interesting in its own right, but I wouldn’t say that money allocated toward training compute for LLMs is the same idea as money allocated to fundamental AI research, if that’s what you were intending to say.
It’s uncontroversial that OpenAI spends a lot on research, but I’m trying to draw a distinction between fundamental research, which, to me, connotes things that are more risky, uncertain, speculative, explorative, and may take a long time to pay off, and research that can be quickly productized.
I don’t understand the details of what Epoch AI is trying to say, but I would be curious to learn.
Do unreleased models include as-yet unreleased models such as GPT-5? (The timeframe is 2024 and OpenAI didn’t release GPT-5 until 2025.) Would it also include o4? (Is there still going to be an o4?) Or is it specifically models that are never intended to be released? I’m guessing it’s just everything that hasn’t been released yet, since I don’t know how Epoch AI would have any insight into what OpenAI intends to release or not.
I’m also curious how much trial and error goes into training for LLMs. Does OpenAI often abort training runs or find the results to be disappointing? How many partial or full training runs go into training one model? For example, what percentage of the overall cost is the $400 million estimated for the final training run of GPT-4.5? 100%? 90%? 50%? 10%?
Overall, this estimate from Epoch AI doesn’t seem to tell us much about what amount of money or compute OpenAI is allocating to fundamental research vs. R&D that can quickly be productized.
This is what Epoch AI says about its estimates:
That’s kind of interesting in its own right, but I wouldn’t say that money allocated toward training compute for LLMs is the same idea as money allocated to fundamental AI research, if that’s what you were intending to say.
It’s uncontroversial that OpenAI spends a lot on research, but I’m trying to draw a distinction between fundamental research, which, to me, connotes things that are more risky, uncertain, speculative, explorative, and may take a long time to pay off, and research that can be quickly productized.
I don’t understand the details of what Epoch AI is trying to say, but I would be curious to learn.
Do unreleased models include as-yet unreleased models such as GPT-5? (The timeframe is 2024 and OpenAI didn’t release GPT-5 until 2025.) Would it also include o4? (Is there still going to be an o4?) Or is it specifically models that are never intended to be released? I’m guessing it’s just everything that hasn’t been released yet, since I don’t know how Epoch AI would have any insight into what OpenAI intends to release or not.
I’m also curious how much trial and error goes into training for LLMs. Does OpenAI often abort training runs or find the results to be disappointing? How many partial or full training runs go into training one model? For example, what percentage of the overall cost is the $400 million estimated for the final training run of GPT-4.5? 100%? 90%? 50%? 10%?
Overall, this estimate from Epoch AI doesn’t seem to tell us much about what amount of money or compute OpenAI is allocating to fundamental research vs. R&D that can quickly be productized.