I’m not really referring to hardware here, in pre-training and RLHF the model weights are being changed and updated
Sure, I was just using this as an example. I should have made this more clera.
Here is a version of the exact same paragraph you wrote but for activations and incontext learning:
in pre-training and RLHF the model activations are being changed and updated by each layer, and that’s where the ‘in-context learning’ (if we want to call it that) comes in—the activations are being updated/optimized to better predict the next token and understand the text. The layers learned to in-context learn (update the activations) across a wide variety of data in pretraining.
Fair enough if you want to say “the model isn’t learning, the activations are learning”, but then you should also say “short term (<1 minute) learning in humans isn’t the brain learning, it is the transient neural state learning”.
I’ll have to dive into the technical details here I think, but the mystery of in-context learning has certainly shot up my reading list, and I really appreciate that link btw! It seems Blaine has some of the similary a-priori scepticism that I do towards it, but the right way for me to proceed is dive into the empirical side and see if my ideas hold water there.
Sure, I was just using this as an example. I should have made this more clera.
Here is a version of the exact same paragraph you wrote but for activations and incontext learning:
in pre-training and RLHF the model activations are being changed and updated by each layer, and that’s where the ‘in-context learning’ (if we want to call it that) comes in—the activations are being updated/optimized to better predict the next token and understand the text. The layers learned to in-context learn (update the activations) across a wide variety of data in pretraining.
(We can show transformers learning to optimization in [very toy cases](https://www.lesswrong.com/posts/HHSuvG2hqAnGT5Wzp/no-convincing-evidence-for-gradient-descent-in-activation#Transformers_Learn_in_Context_by_Gradient_Descent__van_Oswald_et_al__2022_).)
Fair enough if you want to say “the model isn’t learning, the activations are learning”, but then you should also say “short term (<1 minute) learning in humans isn’t the brain learning, it is the transient neural state learning”.
I’ll have to dive into the technical details here I think, but the mystery of in-context learning has certainly shot up my reading list, and I really appreciate that link btw! It seems Blaine has some of the similary a-priori scepticism that I do towards it, but the right way for me to proceed is dive into the empirical side and see if my ideas hold water there.