I’m quite confused, given the fact that all of the weights in the transformer are frozen after training and RLHF, why it’s called learning at all. The model certainly isn’t learning anything.
I would frame it as: the model is learning but then forgetting what it’s learned (due to its inability to move anything from working/short-term memory to long-term memory). That’s something that we see in learning in humans as well (one example: I’ve learned an enormous number of six-digit confirmation codes, each of which I remember just long enough to enter it into the website that’s asking for it), although of course not so consistently.
I would frame it as: the model is learning but then forgetting what it’s learned (due to its inability to move anything from working/short-term memory to long-term memory). That’s something that we see in learning in humans as well (one example: I’ve learned an enormous number of six-digit confirmation codes, each of which I remember just long enough to enter it into the website that’s asking for it), although of course not so consistently.