I would expect advanced AI systems to still be improveable in a way that humans are not. You might lose all ability to see inside the AI’s thinking process, but you could still make hyperparameter tweaks. Humans you can also make hyperparameter tweaks, but unless your think AIs will take 20 years to train, it still seems easier than comparable human improvement.
Fair point. It seems that the central property of AI systems this arguments rests on is their speed, or the time until you get feedback. I agree it seems likely that AI training time (and then ability to evaluate performance on withheld test data or similar) in wall-clock speed will be shorter than feedback loops for humans (e.g. education reforms, genetic engineering, …).
However, some ways in which this could fail to enable rapid self-improvement:
The speed advantage could be offset by other differences, e.g. even less interpretable “thinking processes”.
Performance at certain tasks may be bottlenecked by feedback from slow real-world interactions. (If sim2real transfer doesn’t work well for some tasks.)
I would expect advanced AI systems to still be improveable in a way that humans are not. You might lose all ability to see inside the AI’s thinking process, but you could still make hyperparameter tweaks. Humans you can also make hyperparameter tweaks, but unless your think AIs will take 20 years to train, it still seems easier than comparable human improvement.
Fair point. It seems that the central property of AI systems this arguments rests on is their speed, or the time until you get feedback. I agree it seems likely that AI training time (and then ability to evaluate performance on withheld test data or similar) in wall-clock speed will be shorter than feedback loops for humans (e.g. education reforms, genetic engineering, …).
However, some ways in which this could fail to enable rapid self-improvement:
The speed advantage could be offset by other differences, e.g. even less interpretable “thinking processes”.
Performance at certain tasks may be bottlenecked by feedback from slow real-world interactions. (If sim2real transfer doesn’t work well for some tasks.)