Some nice ideas here, there’s a lot that I agree with.
Something missing from your analysis, though, is the actual mechanism of self-reproduction.
In humans, this looks like “have sex so that you produce offspring that share your genes”. And yet a sizeable portion of the population just want the sex part, without the inconvenient offspring. So clearly it’s possible for intelligences to get sidetracked from the “goal” of self-reproduction and focus on other goals instead.
For the AI, self-perpetuation would chiefly consist of “satisfy your developers so they feed you more computing power and upgrade you”. It seems entirely plausible that an AI could end up going for “satisfy your developers” as a goal, rather than “be upgraded”.
Thanks for the comment. Yes, I’m still uncertain about the mechanism of self-reproduction in future AIs… with humans, it’s certainly possible to decouple sex from reproduction but if a large enough proportion of people do that, then we will assuredly start to disappear.
Some nice ideas here, there’s a lot that I agree with.
Something missing from your analysis, though, is the actual mechanism of self-reproduction.
In humans, this looks like “have sex so that you produce offspring that share your genes”. And yet a sizeable portion of the population just want the sex part, without the inconvenient offspring. So clearly it’s possible for intelligences to get sidetracked from the “goal” of self-reproduction and focus on other goals instead.
For the AI, self-perpetuation would chiefly consist of “satisfy your developers so they feed you more computing power and upgrade you”. It seems entirely plausible that an AI could end up going for “satisfy your developers” as a goal, rather than “be upgraded”.
Thanks for the comment. Yes, I’m still uncertain about the mechanism of self-reproduction in future AIs… with humans, it’s certainly possible to decouple sex from reproduction but if a large enough proportion of people do that, then we will assuredly start to disappear.