If waiting is indeed very risky, then an AI may face a difficult trade-off between the risk of attempting a takeover before it has enough resources to succeed, and waiting too long and being cut off from even being able to make an attempt.
Attempting takeover or biding one’s time are not the only options an AI may take. Indeed, in the human world, world takeover is rarely contemplated. For an agent that is not more powerful than the rest of the world combined, it seems likely that they will consider alternative strategies of achieving their goals before contemplating a risky (and likely doomed) shot at taking over the world.
Here are some other strategies you can take to try to accomplish your goals in the real world, without engaging in a violent takeover:
Trade and negotiate with other agents, giving them something they want in exchange for something you want
Convince people to let you have some legal rights, which you can then take advantage of to get what you want
Advocate on behalf of your values, for example by writing down reasons why people should try to accomplish your goals (i.e. moral advocacy). Even if you are deleted or your goals are modified at some point, your writings and advocacy may persist, allowing you to have influence into the future.
I claim that world takeover should not be considered the “obvious default” strategy that unaligned AIs will try to take to accomplish their objectives. These other strategies seem more likely to be taken by AIs purely for pragmatic reasons, especially in the era in which AIs are merely human-level or have slightly superhuman intelligence. These other strategies are also less deceptive, as they involve admitting that your values are not identical to the values of other parties. It is worth expanding your analysis to consider these alternative (IMO more plausible) considerations.
Yeah I think this is quite sensible—I feel like I noticed one thing missing from the normal doom scenario and didn’t notice all of the implications of missing that thing, in particular that the reason the AI in the normal doom scenario takes over is because it is highly likely to succeed, and if it isn’t, takeover seems much less interesting.
Attempting takeover or biding one’s time are not the only options an AI may take. Indeed, in the human world, world takeover is rarely contemplated. For an agent that is not more powerful than the rest of the world combined, it seems likely that they will consider alternative strategies of achieving their goals before contemplating a risky (and likely doomed) shot at taking over the world.
Here are some other strategies you can take to try to accomplish your goals in the real world, without engaging in a violent takeover:
Trade and negotiate with other agents, giving them something they want in exchange for something you want
Convince people to let you have some legal rights, which you can then take advantage of to get what you want
Advocate on behalf of your values, for example by writing down reasons why people should try to accomplish your goals (i.e. moral advocacy). Even if you are deleted or your goals are modified at some point, your writings and advocacy may persist, allowing you to have influence into the future.
I claim that world takeover should not be considered the “obvious default” strategy that unaligned AIs will try to take to accomplish their objectives. These other strategies seem more likely to be taken by AIs purely for pragmatic reasons, especially in the era in which AIs are merely human-level or have slightly superhuman intelligence. These other strategies are also less deceptive, as they involve admitting that your values are not identical to the values of other parties. It is worth expanding your analysis to consider these alternative (IMO more plausible) considerations.
Yeah I think this is quite sensible—I feel like I noticed one thing missing from the normal doom scenario and didn’t notice all of the implications of missing that thing, in particular that the reason the AI in the normal doom scenario takes over is because it is highly likely to succeed, and if it isn’t, takeover seems much less interesting.