This suggests that we should schedule a time to talk in person, and/or an adversarial collaboration trying to write a version of the argument that you’re thinking of.
Sounds good, I’ll just clarify my position in this response, rather than arguing against your claims.
So then I guess your response is something like “But everyone forgetting to eat food is a crazy scenario, whereas the naive extrapolation of the thing we’re currently doing is the default scenario”.
It’s more like “there isn’t any intellectual work to be done / field building to do / actors to coordinate to get everyone to eat”.
Whereas in the AI case, I don’t know how we’re going to fix the problem I outlined; and as far as I can tell nor does anyone else in the AI community, and therefore there is intellectual work to be done.
We are already at significantly-better-than-human optimisation
Sorry, by optimization there I meant something more like “intelligence”. I don’t really care whether it comes from better SGD, some hardcoded planning algorithm, or a mesa optimizer; the question is whether it is significantly more capable than humans at pursuing goals.
I thought our opinions were much more similar.
I think our predictions of how the world will go concretely are similar; but I’d guess that I’m happier with abstract arguments that depend on fuzzy intuitive concepts than you are, and find them more compelling than more concrete ones that depend on a lot of specific details.
Sounds good, I’ll just clarify my position in this response, rather than arguing against your claims.
It’s more like “there isn’t any intellectual work to be done / field building to do / actors to coordinate to get everyone to eat”.
Whereas in the AI case, I don’t know how we’re going to fix the problem I outlined; and as far as I can tell nor does anyone else in the AI community, and therefore there is intellectual work to be done.
Sorry, by optimization there I meant something more like “intelligence”. I don’t really care whether it comes from better SGD, some hardcoded planning algorithm, or a mesa optimizer; the question is whether it is significantly more capable than humans at pursuing goals.
I think our predictions of how the world will go concretely are similar; but I’d guess that I’m happier with abstract arguments that depend on fuzzy intuitive concepts than you are, and find them more compelling than more concrete ones that depend on a lot of specific details.