Bottlenecks. AI progress relies on improvements in search, computation, storage and so on (each of these areas breaks down into many subcomponents). Progress could be slowed down by any of these subcomponents: if any of these are difficult to speed up, then AI progress will be much slower than we would naively expect. The classic metaphor here concerns the speed a liquid can exit a bottle, which is rate-limited by the narrow space near the opening. AI systems may run into bottlenecks if any essential components cannot be improved quickly (see Aghion et al., 2019).
This seems false to me.
I think it’s helpful to consider the interaction of compute, algorithms, data, and financial/human capital.
I’m not sure that many people think that “search” or “storage” are important levers for computing
I guess RAM isn’t implausible as a bottleneck, but I probably wouldn’t decouple that from chip progress more broadly.
Compute
progress seems driven by many small improvements instead of one large change. There are many ideas that might work when designing chips, manufacturing equipment, etc., and progress in general seems to be fairly steady and regular. and progress is pretty distributed
Algorithms
again, the story I tend to hear from people inside the labs, as well as various podcasts and Twitter, is that many ideas might work, and it’s mostly a case of testing various ideas empirically in a compute-efficient manner
and performance gains can come from multiple places, e.g., performance engineering, better implementations of components, architectural improvements, hyperparameter search … which are approximately independent of each other
Data—I think data is a more plausible bottleneck—it seems to me that either synthetic generation works or it doesn’t.
That said my main issue is that you shouldn’t consider any of these factors as “independent bottlenecks.” If there isn’t enough data, you can try to develop more data-efficient algorithms or dedicate more compute to producing more data. If you’re struggling to make progress on algorithms, you can just keep scaling up, throwing more data and compute at the problem, etc.
I do think bottlenecks may exist, and identifying them is an important step in determining how to forecast, regulate, and manage AI progress, but, I don’t think interactions between AI progress inputs should be used as an argument against a fast take-off or approximately monotonically increasing rates of AI progress up to extremely powerful AI.
This seems false to me.
I think it’s helpful to consider the interaction of compute, algorithms, data, and financial/human capital.
I’m not sure that many people think that “search” or “storage” are important levers for computing
I guess RAM isn’t implausible as a bottleneck, but I probably wouldn’t decouple that from chip progress more broadly.
Compute
progress seems driven by many small improvements instead of one large change. There are many ideas that might work when designing chips, manufacturing equipment, etc., and progress in general seems to be fairly steady and regular. and progress is pretty distributed
Algorithms
again, the story I tend to hear from people inside the labs, as well as various podcasts and Twitter, is that many ideas might work, and it’s mostly a case of testing various ideas empirically in a compute-efficient manner
and performance gains can come from multiple places, e.g., performance engineering, better implementations of components, architectural improvements, hyperparameter search … which are approximately independent of each other
Data—I think data is a more plausible bottleneck—it seems to me that either synthetic generation works or it doesn’t.
That said my main issue is that you shouldn’t consider any of these factors as “independent bottlenecks.” If there isn’t enough data, you can try to develop more data-efficient algorithms or dedicate more compute to producing more data. If you’re struggling to make progress on algorithms, you can just keep scaling up, throwing more data and compute at the problem, etc.
I do think bottlenecks may exist, and identifying them is an important step in determining how to forecast, regulate, and manage AI progress, but, I don’t think interactions between AI progress inputs should be used as an argument against a fast take-off or approximately monotonically increasing rates of AI progress up to extremely powerful AI.