Deep learning is strongly biased toward networks that generalize the way humans want— otherwise, it wouldn’t be economically useful.
I noticed you switched here from talking about “SGD” to talking about “deep learning”. That seems dodgy. I think you are neglecting the possible implicit regularization effect of SGD.
I don’t work at OpenAI, but my prior is that insofar as ChatGPT generalizes, it’s the result of many years of research into regularization being applied during its training. (The fact that the term ‘regularization’ doesn’t even appear in this post seems like a big red flag.)
We’ve figured out now how to train neural networks so they generalize, and we could probably figure out how to train neural networks without schemers if we put in similar years of effort. But in the same way that the very earliest neural networks were (likely? I’m no historian) overfit by default, it seems reasonable to wonder if the very earliest neural networks large enough to have schemers will have schemers by default.
I noticed you switched here from talking about “SGD” to talking about “deep learning”. That seems dodgy. I think you are neglecting the possible implicit regularization effect of SGD.
I don’t work at OpenAI, but my prior is that insofar as ChatGPT generalizes, it’s the result of many years of research into regularization being applied during its training. (The fact that the term ‘regularization’ doesn’t even appear in this post seems like a big red flag.)
We’ve figured out now how to train neural networks so they generalize, and we could probably figure out how to train neural networks without schemers if we put in similar years of effort. But in the same way that the very earliest neural networks were (likely? I’m no historian) overfit by default, it seems reasonable to wonder if the very earliest neural networks large enough to have schemers will have schemers by default.