Won’t we see warning signs of AI takeover and be able to nip it in the bud? I would guess we would see some warning signs, but does that mean we could nip it in the bud? Think about human civil wars and revolutions: there are some warning signs, but also, people go from “not fighting” to “fighting” pretty quickly as they see an opportunity to coordinate with each other and be successful.
After seeing warning signs of an incoming AI takeover, I would expect people to go from “not fighting” to “fighting” the AI takeover. As it would be bad for virtually everyone (outside of the apocalyptic residual), there would be an incentive to coordinate against the AIs. This seemingly contrasts to civil wars and revolutions, in which there are many competing interests.
That being said, I do not think the above is a strong objection, because humanity does not have a flawless track record of coordinating to solve global groblems.
[Footnote 11] I don’t go into detail about how AIs might coordinate with each other, but it seems like there are many options, such as by opening their own email accounts and emailing each other.
I can see “how AIs might coordinate with each other”. How about the questions:
Why should we expect AIs to coordinate with each other? Because they would derive from the same system, and therefore have the same goal?
On the one hand, intelligent agents seem to coordinate to achieve their goals.
However, assuming powerful AIs are extreme optimisers, even a very small difference between the goals of two powerful systems might lead to a breakdown in cooperation.
If multiple powerful AIs, with competing goals, emerge at roughly the same time, can humans take advantage?
Regardless of the answer, a war between powerful AIs still seems a pretty bad outcome.
A simple analogy: human wars might eventually benefit some animals, but it does not change much the fact that most power to shape the world belongs to humans.
However, fierce competition between potentially powerful AIs at an early stage might give humans time to react (before e.g. one of the AIs dominates the others, and starts thinking about how to conquer the wider world).
These questions are outside the scope of this post, which is about what would happen if AIs were pointed at defeating humanity.
I don’t think there’s a clear answer to whether AIs would have a lot of their goals in common, or find it easier to coordinate with each other than with humans, but the probability of each seems at least reasonably high if they are all developed using highly similar processes (making them all likely more similar to each other in many ways than to humans).
After seeing warning signs of an incoming AI takeover, I would expect people to go from “not fighting” to “fighting” the AI takeover. As it would be bad for virtually everyone (outside of the apocalyptic residual), there would be an incentive to coordinate against the AIs. This seemingly contrasts to civil wars and revolutions, in which there are many competing interests.
That being said, I do not think the above is a strong objection, because humanity does not have a flawless track record of coordinating to solve global groblems.
I can see “how AIs might coordinate with each other”. How about the questions:
Why should we expect AIs to coordinate with each other? Because they would derive from the same system, and therefore have the same goal?
On the one hand, intelligent agents seem to coordinate to achieve their goals.
However, assuming powerful AIs are extreme optimisers, even a very small difference between the goals of two powerful systems might lead to a breakdown in cooperation.
If multiple powerful AIs, with competing goals, emerge at roughly the same time, can humans take advantage?
Regardless of the answer, a war between powerful AIs still seems a pretty bad outcome.
A simple analogy: human wars might eventually benefit some animals, but it does not change much the fact that most power to shape the world belongs to humans.
However, fierce competition between potentially powerful AIs at an early stage might give humans time to react (before e.g. one of the AIs dominates the others, and starts thinking about how to conquer the wider world).
These questions are outside the scope of this post, which is about what would happen if AIs were pointed at defeating humanity.
I don’t think there’s a clear answer to whether AIs would have a lot of their goals in common, or find it easier to coordinate with each other than with humans, but the probability of each seems at least reasonably high if they are all developed using highly similar processes (making them all likely more similar to each other in many ways than to humans).