“All that is stopping them being even more powerful is spending on compute. Google & Microsoft are worth $1-2T each, and $10B can buy ~100x the compute used for GPT-4. Think about this: it means we are already well into hardware overhang territory[5].”
I broadly agree with the point that compute could be scaled up significantly, but I want to add a few notes about the claim that $10B buys 100x the compute of GPT-4.
Altman said “more” when asked if GPT-4 had cost $100M to train. We don’t know how much more. But PaLM seems to have only cost $9M-$23M so $100M is probably reasonable.
If OpenAI was buying up 100x the compute of GPT-4, maybe that would be a big enough spike in demand for GPUs that they would become more expensive. I’m pretty uncertain about what to expect there, but I estimated that PaLM used the equivalent of 0.01% of the world’s current GPU/TPU computing capacity for 2 months. GPT-4 seems to be bigger than PaLM, so 100x the compute used for it might be the equivalent of more than 1% of the world’s existing GPU/TPU computing capacity for 2 months.
Thanks! Interested in getting more stats on global GPU/TPU capacity (there seems to be a dearth of good stats available). In your AI Impacts report, you mention paywalled reports. How much are they to access? (DM me and I can pay).
Epoch AI estimate that GPT-4 costs only $40M (not sure if that is today’s price, with cost improvements since last year?)
100x the compute used for it might be the equivalent of more than 1% of the world’s existing GPU/TPU computing capacity for 2 months.
1% seems well within reach for the biggest players :(
Great post!
I broadly agree with the point that compute could be scaled up significantly, but I want to add a few notes about the claim that $10B buys 100x the compute of GPT-4.
Altman said “more” when asked if GPT-4 had cost $100M to train. We don’t know how much more. But PaLM seems to have only cost $9M-$23M so $100M is probably reasonable.
If OpenAI was buying up 100x the compute of GPT-4, maybe that would be a big enough spike in demand for GPUs that they would become more expensive. I’m pretty uncertain about what to expect there, but I estimated that PaLM used the equivalent of 0.01% of the world’s current GPU/TPU computing capacity for 2 months. GPT-4 seems to be bigger than PaLM, so 100x the compute used for it might be the equivalent of more than 1% of the world’s existing GPU/TPU computing capacity for 2 months.
Thanks! Interested in getting more stats on global GPU/TPU capacity (there seems to be a dearth of good stats available). In your AI Impacts report, you mention paywalled reports. How much are they to access? (DM me and I can pay).
Epoch AI estimate that GPT-4 costs only $40M (not sure if that is today’s price, with cost improvements since last year?)
1% seems well within reach for the biggest players :(