Something I didn’t really touch on the interview is factors that might push in the direction of generality. I’ve never considered user-friendliness as a factor that might be important—but I think you’re right at least about the case of GPT-3. I also agree that empirical work investigating the value of generality will probably be increasingly useful.
Some other potential factors that might count in favor of generality:
*It seems like limited data availability can push in the direction of generality. For example: If we wanted to create a system capable of producing Shakespearean sonnets, and we had a trillion examples of Shakespearean sonnets, I imagine that the best and most efficient way to create this system would be to train it only on Shakespearean sonnets. But, since we don’t have that many Shakespearean sonnets, it of course ends up being useful to first train the system on a more inclusive corpus of English-language text (as in the case of GPT-3) and then fine-tune it on the smaller Shakespeare dataset. In this way, creating general systems can end up being useful (or even necessary) for creating systems that can perform specific tasks. (Although this argument is consistent with more general systems being used in training, but more narrow systems ultimately being deployed.)
*If you’re pretty unsure what tasks you’ll want AI systems to perform in some context—and it’s slow or costly to create new narrow AI systems, to figure out what existing narrow AI system would be appropriate for the tasks that come up, or to switch to using new narrow AI systems—then it may simply be more efficient to use very general AI systems that can handle a wide range of tasks.
*If you use multiple dinstinct systems to get some job done, there’s a cost to coordinating them, which might avoided if you use a single more unified system. For example, as a human analogy, if three people people want to cook a meal together, then some energy is going to need to go into deciding who does what, keeping track of each person’s progress, etc. The costs of coordinating multiple specialized units can sometimes outweigh the benefits of specialization.
I think the “CAIS response” to the latter two points would probably be that AI-driven R&D processes might eventually get really good at quickly spinning up new AI systems, as needed, and coordinating the use of multiple systems as needed. Personally unsure whether or not I find that compelling, in the long run.
Limited data availability and generality in practice now: this paper ( https://arxiv.org/abs/2006.16668 ) about how improving translation performance for “low resource” languages with not many training examples available relies on “positive language transfer” from training on other languages.
Ben here: Great post!
Something I didn’t really touch on the interview is factors that might push in the direction of generality. I’ve never considered user-friendliness as a factor that might be important—but I think you’re right at least about the case of GPT-3. I also agree that empirical work investigating the value of generality will probably be increasingly useful.
Some other potential factors that might count in favor of generality:
*It seems like limited data availability can push in the direction of generality. For example: If we wanted to create a system capable of producing Shakespearean sonnets, and we had a trillion examples of Shakespearean sonnets, I imagine that the best and most efficient way to create this system would be to train it only on Shakespearean sonnets. But, since we don’t have that many Shakespearean sonnets, it of course ends up being useful to first train the system on a more inclusive corpus of English-language text (as in the case of GPT-3) and then fine-tune it on the smaller Shakespeare dataset. In this way, creating general systems can end up being useful (or even necessary) for creating systems that can perform specific tasks. (Although this argument is consistent with more general systems being used in training, but more narrow systems ultimately being deployed.)
*If you’re pretty unsure what tasks you’ll want AI systems to perform in some context—and it’s slow or costly to create new narrow AI systems, to figure out what existing narrow AI system would be appropriate for the tasks that come up, or to switch to using new narrow AI systems—then it may simply be more efficient to use very general AI systems that can handle a wide range of tasks.
*If you use multiple dinstinct systems to get some job done, there’s a cost to coordinating them, which might avoided if you use a single more unified system. For example, as a human analogy, if three people people want to cook a meal together, then some energy is going to need to go into deciding who does what, keeping track of each person’s progress, etc. The costs of coordinating multiple specialized units can sometimes outweigh the benefits of specialization.
I think the “CAIS response” to the latter two points would probably be that AI-driven R&D processes might eventually get really good at quickly spinning up new AI systems, as needed, and coordinating the use of multiple systems as needed. Personally unsure whether or not I find that compelling, in the long run.
Limited data availability and generality in practice now: this paper ( https://arxiv.org/abs/2006.16668 ) about how improving translation performance for “low resource” languages with not many training examples available relies on “positive language transfer” from training on other languages.
It seems like the hyperlink of the arxiv webpage is invalid (i.e. when you click on the arxiv link).
Fixed! Whoops.