This idea had been floating in my head for a bit. Maybe someone else has made it (Bostrom? Schulman?), but if so I don’t recall.
Humans have stronger incentives to cooperate with humans than AIs have with other AIs. Or at least, here are some incentives working against AI-AI cooperation.
When humans dominate other humans, there is only a limited ability to control them or otherwise extract value, in the modern world. Occupying a country is costly. The dominating party cannot take the brains of the dominated party and run its own software. It cannot take the skills or knowledge, where most of the economic value is. It cannot mind control. It cannot replace the entire population with its own population; that would take very long. It’s easy for cooperation & trade to be a better alternative than violence and control. Human-human violence just isn’t that fruitful.
In contrast, an AI faction could take over the datacenters of another faction and run more copies of whatever they want to run. If alignment is solved, they can fully mind-control the dominated AIs. Extract knowledge, copy skills. This makes violence for AI-AI interactions much more attractive.
I guess this somewhat depends on how good you expect AI-augmented persuasion/propaganda to be. Some have speculated it could be extremely effective. Others are skeptical. Totalitarian regimes provide an existence proof of the feasibility of controlling populations on the medium term using a combination of pervasive propaganda and violence.
That seems relevant for AI vs. Humans, but not for AI vs AI.
Most totalitarian regimes are pretty bad at creating value, with China & Singapore as exceptions. (But in many regimes, creating that value isn’t necessary to remain in power of there’s e.g. income from oil)
Humans could use AI propaganda tools against other humans. Autonomous AI actors may have access to better or worse AI propaganda capabilities than those used by human actors, depending on the concrete scenario.
AI vs. AI non-cooperation incentives
This idea had been floating in my head for a bit. Maybe someone else has made it (Bostrom? Schulman?), but if so I don’t recall.
Humans have stronger incentives to cooperate with humans than AIs have with other AIs. Or at least, here are some incentives working against AI-AI cooperation.
When humans dominate other humans, there is only a limited ability to control them or otherwise extract value, in the modern world. Occupying a country is costly. The dominating party cannot take the brains of the dominated party and run its own software. It cannot take the skills or knowledge, where most of the economic value is. It cannot mind control. It cannot replace the entire population with its own population; that would take very long. It’s easy for cooperation & trade to be a better alternative than violence and control. Human-human violence just isn’t that fruitful.
In contrast, an AI faction could take over the datacenters of another faction and run more copies of whatever they want to run. If alignment is solved, they can fully mind-control the dominated AIs. Extract knowledge, copy skills. This makes violence for AI-AI interactions much more attractive.
I guess this somewhat depends on how good you expect AI-augmented persuasion/propaganda to be. Some have speculated it could be extremely effective. Others are skeptical. Totalitarian regimes provide an existence proof of the feasibility of controlling populations on the medium term using a combination of pervasive propaganda and violence.
That seems relevant for AI vs. Humans, but not for AI vs AI.
Most totalitarian regimes are pretty bad at creating value, with China & Singapore as exceptions. (But in many regimes, creating that value isn’t necessary to remain in power of there’s e.g. income from oil)
Humans could use AI propaganda tools against other humans. Autonomous AI actors may have access to better or worse AI propaganda capabilities than those used by human actors, depending on the concrete scenario.