An implicit assumption made by all the biological anchor hypotheses is that the overwhelming majority of the computational cost of training will come from running the model that is being trained, rather than from running its training environment.
This is clearly the case for a transformative model which only operates on text, code, images, audio, and video since in that case the “environment” (the strings of tokens or pixels being processed) requires a negligible amount of computation and memory compared to what is required for a large model. Additionally, as I mentioned above, it seems possible that some highly abstract mathematical environments which are very cheap to run could nonetheless be very rich and support extremely intelligent agents. I think this is likely to be sufficient for training a transformative model, although I am not confident.
If reinforcement learning in a rich simulated world (e.g. complex physics or other creatures) is required to train a transformative model, it is less clear whether model computation will dominate the computation of the environment. Nonetheless, I still believe this is likely. My understanding is that the computation used to run video game-playing agents is currently in the same ballpark as the computation used to run the game engine. Given these models are far from perfect play, there is likely still substantial room to improve on those same environments with a larger model. It doesn’t seem likely that the computational cost of environments will need to grow faster than the computational cost of agents going forward. (If several intelligent agents must interact with one another in the environment, it seems likely that all agents can be copies of the same model.)
In the main report, I assume that the computation required to train a transformative model under this path can be well-approximated by FHKP, where F is the model’s FLOP / subj sec, H is the model’s horizon length in subjective seconds, P is the parameter count of the model, and and K describe scaling behavior. I do not add an additional term for the computational cost of running the environment.
Thanks for pointing out that section. I agree that the section discusses the issue. But I am left unsatisfied about whether it defeats it.
In particular,
“I think this is likely to be sufficient for training a transformative model, although I am not confident. ”
“it is less clear whether model computation will dominate the computation of the environment. Nonetheless, I still believe this is likely”
don’t seem hugely confident. And that’s fine, the report is already pretty long.
But then even if the report is not at fault I am kind of unsatisfied about the evolutionary anchor part being used as an actual upper bound—not sure whether people are actually doing that all that often, but Eli’s comment below seems to indicate that it might be, and I remember it being used that way on a couple of occasions.
This seems like a reasonable assumption for other anchors such as the Lifetime and the Neural Network Horizon anchors, which assume that training environments for TAI are similar to training environments used for AI today. But it seems much more difficult to justify for the evolution anchor, which Ajeya admits would be far more computationally intensive than storing text or simulating a deterministic Atari game.
This post argues that the evolutionary environment is similarly or more complex than the brains of the organisms within it, while the second paragraph of the above quotation disagrees. Neither argument seems detailed enough to definitively answer the question, so I’d be interested to read any further research on the two questions proposed in the post:
Coming up with estimates of what the least fine-grained world that we would expect might be able to produce intelligent life if we simulated natural selection in it.
Calculating how much compute it would take to in fact simulate it.
But it seems much more difficult to justify for the evolution anchor, which Ajeya admits would be far more computationally intensive than storing text or simulating a deterministic Atari game.
The evolution anchor involves more compute than the other anchors (because you need to get so many more data points and train the AI on them), but it’s not obvious to me that it requires a larger proportion of compute spent on the environment than the other anchors. Like, it seems plausible to me that the evolution anchor looks more like having the AI play pretty simple games for an enormously long time, rather than having a complicated physically simulated environment.
Ajeya’s report addresses this in the “What if training data or environments will be a bottleneck?” section, in particular in the “Having computation to run training environments” subsection:
Thanks for pointing out that section. I agree that the section discusses the issue. But I am left unsatisfied about whether it defeats it.
In particular,
“I think this is likely to be sufficient for training a transformative model, although I am not confident. ”
“it is less clear whether model computation will dominate the computation of the environment. Nonetheless, I still believe this is likely”
don’t seem hugely confident. And that’s fine, the report is already pretty long.
But then even if the report is not at fault I am kind of unsatisfied about the evolutionary anchor part being used as an actual upper bound—not sure whether people are actually doing that all that often, but Eli’s comment below seems to indicate that it might be, and I remember it being used that way on a couple of occasions.
This seems like a reasonable assumption for other anchors such as the Lifetime and the Neural Network Horizon anchors, which assume that training environments for TAI are similar to training environments used for AI today. But it seems much more difficult to justify for the evolution anchor, which Ajeya admits would be far more computationally intensive than storing text or simulating a deterministic Atari game.
This post argues that the evolutionary environment is similarly or more complex than the brains of the organisms within it, while the second paragraph of the above quotation disagrees. Neither argument seems detailed enough to definitively answer the question, so I’d be interested to read any further research on the two questions proposed in the post:
Coming up with estimates of what the least fine-grained world that we would expect might be able to produce intelligent life if we simulated natural selection in it.
Calculating how much compute it would take to in fact simulate it.
The evolution anchor involves more compute than the other anchors (because you need to get so many more data points and train the AI on them), but it’s not obvious to me that it requires a larger proportion of compute spent on the environment than the other anchors. Like, it seems plausible to me that the evolution anchor looks more like having the AI play pretty simple games for an enormously long time, rather than having a complicated physically simulated environment.
Fair enough. Both seem plausible to me, we’d probably need more evidence to know which one would require more compute.