It’s the crux between you and Ajeya, because you’re relatively more in agreement on the other numbers. But I think that adopting the xpt numbers on these other variables would slow down your own timelines notably, because of the almost complete lack of increase in spending.
That said, if the forecasters agreed with your compute requirements, they would probably also forecast higher spending.
The XPT forecasters are so in the dark about compute spending that I just pretend they gave more reasonable numbers. I’m honestly baffled how they could be so bad. The most aggressive of them thinks that in 2025 the most expensive training run will be $70M, and that it’ll take 6+ years to double thereafter, so that in 2032 we’ll have reached $140M training run spending… do these people have any idea how much GPT-4 cost in 2022?!?!? Did they not hear about the investments Microsoft has been making in OpenAI? And remember that’s what the most aggressive among them thought! The conservatives seem to be living in an alternate reality where GPT-3 proved that scaling doesn’t work and an AI winter set in in 2020.
Remember these predictions were made in summer 2022, before ChatGPT, before the big Microsoft investment and before any serious info about GPT-4. They’re still low, but not ridiculous.
And then GPT-3 happened, and was widely regarded to be a huge success and proof that scaling is a good idea etc.
So the amount of compute-spending that the most aggressive forecasters think could be spent on a single training run in 2032… is about 25% as much compute-spending as Microsoft gave OpenAI starting in 2019, before GPT-3 and before the scaling hypothesis. The most aggressive forecasters.
It’s the crux between you and Ajeya, because you’re relatively more in agreement on the other numbers. But I think that adopting the xpt numbers on these other variables would slow down your own timelines notably, because of the almost complete lack of increase in spending.
That said, if the forecasters agreed with your compute requirements, they would probably also forecast higher spending.
The XPT forecasters are so in the dark about compute spending that I just pretend they gave more reasonable numbers. I’m honestly baffled how they could be so bad. The most aggressive of them thinks that in 2025 the most expensive training run will be $70M, and that it’ll take 6+ years to double thereafter, so that in 2032 we’ll have reached $140M training run spending… do these people have any idea how much GPT-4 cost in 2022?!?!? Did they not hear about the investments Microsoft has been making in OpenAI? And remember that’s what the most aggressive among them thought! The conservatives seem to be living in an alternate reality where GPT-3 proved that scaling doesn’t work and an AI winter set in in 2020.
Perhaps this should be a top-level comment.
Remember these predictions were made in summer 2022, before ChatGPT, before the big Microsoft investment and before any serious info about GPT-4. They’re still low, but not ridiculous.
Fair, but still: In 2019 Microsoft invested a billion dollars in OpenAI, roughly half of which was compute: Microsoft invests billions more dollars in OpenAI, extends partnership | TechCrunch
And then GPT-3 happened, and was widely regarded to be a huge success and proof that scaling is a good idea etc.
So the amount of compute-spending that the most aggressive forecasters think could be spent on a single training run in 2032… is about 25% as much compute-spending as Microsoft gave OpenAI starting in 2019, before GPT-3 and before the scaling hypothesis. The most aggressive forecasters.