Coming back to the point about data. Whilst Epoch gathered some data showing that the stock high quality text data might soon be exhausted, their overall conclusion is that there is only a “20% chance that the scaling (as measured in training compute) of ML models will significantly slow down by 2040 due to a lack of training data.”. Regarding Jacob Buckman’s point about chess, he actually outlines a way around that (training data provided by narrow AI). As a counter to the wider point about the need for active learning, see DeepMind’s Adaptive Agent and the Voyager “lifelong learning” Minecraft agent, both of which seem like impressive steps in this direction.
Coming back to the point about data. Whilst Epoch gathered some data showing that the stock high quality text data might soon be exhausted, their overall conclusion is that there is only a “20% chance that the scaling (as measured in training compute) of ML models will significantly slow down by 2040 due to a lack of training data.”. Regarding Jacob Buckman’s point about chess, he actually outlines a way around that (training data provided by narrow AI). As a counter to the wider point about the need for active learning, see DeepMind’s Adaptive Agent and the Voyager “lifelong learning” Minecraft agent, both of which seem like impressive steps in this direction.