Nvidia’s moat comes from a few things. As you pointed out, they have CUDA, which is a proprietary set of APIs for running parallelised math operations. But they also have the best performing chips on the market by a long way. This is not merely a function of having strong optimisation on the software side (possibly replicable by o3 but I would need to see more evidence to be convinced that an LLM would be good at optimisation), or on the hardware side (much, MUCH trickier for an LLM given that a lot of the hardware has to operate on nanometre scale, which can be hard to simulate), but also because having the most money and a strong track record & relationship means they can get preferential access to next-gen fabs at TSMC.
It is also true that the recent boom has increased investment into running CUDA code on other GPUs. The SCALE project is one such example. This implies (a) the bottleneck is not about replicating CUDA’s functionality (which it does), but more about replicating its performance (they might have gains to make there) and/or (b) that the actual moat really does lie in the hardware. Again, probably a mix of both.
However, this hasn’t stopped other companies from making progress here. I think it’s indicative that Deepseek v3 was allegedly trained for less than $10m. If this is true, it suggests to me that:
Frontier labs might be currently using their hardware very inefficiently, and if these efficiencies were to be capitalised on, demand for Nvidia hardware would reduce (both by using less of their GPUs, but also because you wouldn’t need the best of the best to do well)
If it turns out to be cheap to train good LLMs, captured value might shift back to frontier labs, or even to downstream applications. This would reduce Nvidia’s pricing power.
Also, it looks like the competition is catching up anyway. It seems like it’s very reasonable to do inference on Apple or Google chips (Apple Intelligence runs on M2-series chips, these also have top TSMC node access; Google run a lot of inference on their own TPUs). I was particularly impressed that you can run a 600B+ parameter model on 8 Mac Minis, not even running Apple’s best chips. Even if it’s only inference, that’s a huge chunk of the market that might fall to competitors soon.
So I’m not exactly counting on Nvidia to hold, but I think it will be for other reasons than automation. Even if you are very AI-pilled, we still live in the world where market dynamics are much stronger than labour automation effects. For now :)
Interesting. Seems like the obvious way to resolve this disagreement would be to check benchmarks for SCALE. 6 months ago, in this reddit thread, they said benchmarks will “become publicly available in the near future”. So, join their discord if you’re holding Nvidia stock, I guess?
Regarding decreased demand for GPUs—I agree Deepseek v3 is evidence that demand will decrease. But o3 seems like evidence that demand will increase, since we’ll be doing more test-time compute. Unclear which evidence is more compelling.
Well put! I will add that the competition includes companies like Amazon creating their own chips and others designing silicon specialised for inference (like Groq (not to be confused with xAI’s model Grok))
Nvidia’s moat comes from a few things. As you pointed out, they have CUDA, which is a proprietary set of APIs for running parallelised math operations. But they also have the best performing chips on the market by a long way. This is not merely a function of having strong optimisation on the software side (possibly replicable by o3 but I would need to see more evidence to be convinced that an LLM would be good at optimisation), or on the hardware side (much, MUCH trickier for an LLM given that a lot of the hardware has to operate on nanometre scale, which can be hard to simulate), but also because having the most money and a strong track record & relationship means they can get preferential access to next-gen fabs at TSMC.
It is also true that the recent boom has increased investment into running CUDA code on other GPUs. The SCALE project is one such example. This implies (a) the bottleneck is not about replicating CUDA’s functionality (which it does), but more about replicating its performance (they might have gains to make there) and/or (b) that the actual moat really does lie in the hardware. Again, probably a mix of both.
However, this hasn’t stopped other companies from making progress here. I think it’s indicative that Deepseek v3 was allegedly trained for less than $10m. If this is true, it suggests to me that:
Frontier labs might be currently using their hardware very inefficiently, and if these efficiencies were to be capitalised on, demand for Nvidia hardware would reduce (both by using less of their GPUs, but also because you wouldn’t need the best of the best to do well)
If it turns out to be cheap to train good LLMs, captured value might shift back to frontier labs, or even to downstream applications. This would reduce Nvidia’s pricing power.
Also, it looks like the competition is catching up anyway. It seems like it’s very reasonable to do inference on Apple or Google chips (Apple Intelligence runs on M2-series chips, these also have top TSMC node access; Google run a lot of inference on their own TPUs). I was particularly impressed that you can run a 600B+ parameter model on 8 Mac Minis, not even running Apple’s best chips. Even if it’s only inference, that’s a huge chunk of the market that might fall to competitors soon.
So I’m not exactly counting on Nvidia to hold, but I think it will be for other reasons than automation. Even if you are very AI-pilled, we still live in the world where market dynamics are much stronger than labour automation effects. For now :)
There’s a thread here about hardware between companies, with johnswentworth arguing AMD had better hardware than Nvidia.
Interesting. Seems like the obvious way to resolve this disagreement would be to check benchmarks for SCALE. 6 months ago, in this reddit thread, they said benchmarks will “become publicly available in the near future”. So, join their discord if you’re holding Nvidia stock, I guess?
Thanks.
Regarding decreased demand for GPUs—I agree Deepseek v3 is evidence that demand will decrease. But o3 seems like evidence that demand will increase, since we’ll be doing more test-time compute. Unclear which evidence is more compelling.
Well put! I will add that the competition includes companies like Amazon creating their own chips and others designing silicon specialised for inference (like Groq (not to be confused with xAI’s model Grok))