I maybe disagree somewhat on the position on scale? I expect that you could get GPT-3 with older architectures but significantly more parameters + data + compute. I’d still agree with “we wouldn’t have GPT-3” because the extra cost would mean we wouldn’t have trained it by now, but plausibly Blake thinks “even with additional parameters + data + compute beyond what we would have had, other architectures wouldn’t have worked”? So I don’t disagree with any of the explicitly stated positions but it’s plausible I disagree with the worldview.
I maybe also disagree with the position on long-term credit assignment? I again agree with the stated point that current RL algorithms mostly can’t do long-term credit assignment when trained from scratch on an environment. But maybe Blake thinks this is important to get powerful AI systems, whereas I wouldn’t say that.
I think he would agree with “we wouldn’t have GPT-3 from an economical perspective”. I am not sure whether he would agree with a theoretical impossibility. From the transcript:
“Because a lot of the current models are based on diffusion stuff, not just bigger transformers. If you didn’t have diffusion models [and] you didn’t have transformers, both of which were invented in the last five years, you wouldn’t have GPT-3 or DALL-E. And so I think it’s silly to say that scale was the only thing that was necessary because that’s just clearly not true.”
To be clear, the part about the credit assignment problem was mostly when discussing the research at his lab, and he did not explicitly argue that the long-term credit assignment problem was evidence that training powerful AI systems is hard. I included the quote because it was relevant, but it was not an “argument” per se.
Interested to hear more about why long-term credit assignment isn’t needed for powerful AI. I think it depends how you quantify those things and I’m pretty unsure about this myself.
Is it because there is already loads of human-generated data which implicitly embody or contain enough long-term credit assignment? Or is it that long-term credit assignment is irrelevant for long-term reasoning? Or maybe long-term reasoning isn’t needed for ‘powerful AI’?
We’re tackling the problem “you tried out a long sequence of actions, and only at the end could you tell whether the outcomes were good or not, and now you have to figure out which actions ”.
Some approaches to this that don’t involve “long-term credit assignment” as normally understood by RL practitioners:
Have humans / other AI systems tell you which of the actions were useful. (One specific way this could be achieved is to use humans / AI systems to provide a dense reward, kinda like in summarizing books from human feedback.)
Supervise the AI system’s reasoning process rather than the outcomes it gets (e.g. like chain-of-thought prompting but with more explicit supervision).
Just don’t even bother, do regular old self-supervised learning on a hard task; in order to get good performance maybe the model has to develop “general intelligence” (i.e. something akin to the algorithms humans use in order to do long-term planning; after all our long-term planning doesn’t work via trial and error).
I think it’s also plausible that (depending on your definitions) long-term reasoning isn’t needed for powerful AI.
I maybe disagree somewhat on the position on scale? I expect that you could get GPT-3 with older architectures but significantly more parameters + data + compute. I’d still agree with “we wouldn’t have GPT-3” because the extra cost would mean we wouldn’t have trained it by now, but plausibly Blake thinks “even with additional parameters + data + compute beyond what we would have had, other architectures wouldn’t have worked”? So I don’t disagree with any of the explicitly stated positions but it’s plausible I disagree with the worldview.
I maybe also disagree with the position on long-term credit assignment? I again agree with the stated point that current RL algorithms mostly can’t do long-term credit assignment when trained from scratch on an environment. But maybe Blake thinks this is important to get powerful AI systems, whereas I wouldn’t say that.
I think he would agree with “we wouldn’t have GPT-3 from an economical perspective”. I am not sure whether he would agree with a theoretical impossibility. From the transcript:
To be clear, the part about the credit assignment problem was mostly when discussing the research at his lab, and he did not explicitly argue that the long-term credit assignment problem was evidence that training powerful AI systems is hard. I included the quote because it was relevant, but it was not an “argument” per se.
Thanks Rohin, I second almost all of this.
Interested to hear more about why long-term credit assignment isn’t needed for powerful AI. I think it depends how you quantify those things and I’m pretty unsure about this myself.
Is it because there is already loads of human-generated data which implicitly embody or contain enough long-term credit assignment? Or is it that long-term credit assignment is irrelevant for long-term reasoning? Or maybe long-term reasoning isn’t needed for ‘powerful AI’?
We’re tackling the problem “you tried out a long sequence of actions, and only at the end could you tell whether the outcomes were good or not, and now you have to figure out which actions ”.
Some approaches to this that don’t involve “long-term credit assignment” as normally understood by RL practitioners:
Have humans / other AI systems tell you which of the actions were useful. (One specific way this could be achieved is to use humans / AI systems to provide a dense reward, kinda like in summarizing books from human feedback.)
Supervise the AI system’s reasoning process rather than the outcomes it gets (e.g. like chain-of-thought prompting but with more explicit supervision).
Just don’t even bother, do regular old self-supervised learning on a hard task; in order to get good performance maybe the model has to develop “general intelligence” (i.e. something akin to the algorithms humans use in order to do long-term planning; after all our long-term planning doesn’t work via trial and error).
I think it’s also plausible that (depending on your definitions) long-term reasoning isn’t needed for powerful AI.