Terminological point: It sounds like you’re using the phrase “instrumental convergence” in an unusual way.
I take it the typical idea is just that there are some instrumental goals that an intelligent agent can expect to be useful in the pursuit of a wide range of other goals, whereas you seem to be emphasizing the idea that those instrumental goals would be pursued to extremes destructive of humanity. It seems to me that (1) those two two ideas are worth keeping separate, (2) “instrumental convergence” would more accurately label the first idea, and (3) that phrase is in fact usually used to refer to the first idea only.
This occurred to me as I was skimming the post and saw the suggestion that instrumental convergence is not seen in humans, to which my reaction was, “What?! Don’t people like money?”
I agree with this, there are definitely two definitions at play. I think a failure to distinguish between these two definitions is actually a big problem with the AI doom argument, where they end up doing an unintentional motte-and-bailey between the two definitions.
David Thornstad explains it pretty well here. The “people want money” definition is trivial and obviously true, but does not lead to the “doom is inevitable” conclusion. I have a goal of eating food, and money is useful for that purpose, but that doesn’t mean I automatically try and accumulate all the wealth on the planet in order to tile the universe with food.
Terminological point: It sounds like you’re using the phrase “instrumental convergence” in an unusual way.
I take it the typical idea is just that there are some instrumental goals that an intelligent agent can expect to be useful in the pursuit of a wide range of other goals, whereas you seem to be emphasizing the idea that those instrumental goals would be pursued to extremes destructive of humanity. It seems to me that (1) those two two ideas are worth keeping separate, (2) “instrumental convergence” would more accurately label the first idea, and (3) that phrase is in fact usually used to refer to the first idea only.
This occurred to me as I was skimming the post and saw the suggestion that instrumental convergence is not seen in humans, to which my reaction was, “What?! Don’t people like money?”
I agree with this, there are definitely two definitions at play. I think a failure to distinguish between these two definitions is actually a big problem with the AI doom argument, where they end up doing an unintentional motte-and-bailey between the two definitions.
David Thornstad explains it pretty well here. The “people want money” definition is trivial and obviously true, but does not lead to the “doom is inevitable” conclusion. I have a goal of eating food, and money is useful for that purpose, but that doesn’t mean I automatically try and accumulate all the wealth on the planet in order to tile the universe with food.