Re scaling current methods: The hundreds of billions figure we quoted does require more context not in our piece; SemiAnalysis explains in a bit more detail how they get to that number (eg assuming training in 3mo instead of 2 years).
That’s hundreds of billions with current hardware. (Actually, not even current hardware, but the A100 which is last-gen; the H100 should already do substantially better.) But HW price-performance currently doubles every ~2 years. Yes, Moore’s Law may be slowing, but I’d be surprised if we don’t get another OOM improvement in price-performance during the next decade, especially given the insatiable demand for effective compute these days.
We don’t want to haggle over the exact scale before it becomes infeasible, though—even if we get another 2 OOM in, we wanted to emphasize with our argument that ‘the current method route’ 1) requires regular scientific breakthroughs of the pre-TAI sort, and 2) even if we get there doesn’t guarantee capabilities that look like magic compared to what we have now, depending on how much you believe in emergence. Both would be bottlenecks.
Yeah, I agree things would be a lot slower without algorithmic breakthroughs. Those do seem to be happening at a pretty good pace though (not just looking at ImageNet, but also looking at ML research subjectively). I’d assume they’ll keep happening at the same rate so long as the number of people (and later, possibly AIs) focused on finding them keeps growing at the same rate.
That’s hundreds of billions with current hardware. (Actually, not even current hardware, but the A100 which is last-gen; the H100 should already do substantially better.) But HW price-performance currently doubles every ~2 years. Yes, Moore’s Law may be slowing, but I’d be surprised if we don’t get another OOM improvement in price-performance during the next decade, especially given the insatiable demand for effective compute these days.
Yeah, I agree things would be a lot slower without algorithmic breakthroughs. Those do seem to be happening at a pretty good pace though (not just looking at ImageNet, but also looking at ML research subjectively). I’d assume they’ll keep happening at the same rate so long as the number of people (and later, possibly AIs) focused on finding them keeps growing at the same rate.