Hey kbog,
Thanks for this. I think this is well argued. If I may, I’d like to pick some holes. I’m not sure if they are sufficient to swing the argument the other way, but I don’t think they’re trivial either.
I’m going to use autonomy in weapons systems in favour of LAWs for reasons argued here(see Takeaway 1).
As far as I can tell, almost all considerations you give are to inter-state conflict. The intra-state consequences are not explored and I think they deserve to be. Fully autonomous weapons systems potentially obviate the need for a mutually beneficial social contract between the regimes in control of the weapons and the populations over which they rule. All dissent becomes easy to crush. This is patently bad in itself, but it also has consequences for interstate conflict; with less approval needed to go to war, inter-state conflict may increase.
The introduction of weapons systems with high degrees of autonomy poses an arguably serious risk of geopolitical turbulence: it is not clear that all states will develop the capability to produce highly autonomous weapons systems. Those that do not will have to purchase them from technologically-more advanced allies willing to sell them. States that find themselves outside of such alliances will be highly vulnerable to attack. This may motivate a nontrivial reshuffling of global military alliances, the outcomes of which are hard to predict. For those without access to these new powerful weapons, one risk mitigation strategy is to develop nuclear weapons, potentially motivating nuclear proliferation.
On your point:
The logic here is a little bit gross, since it’s saying that we should make sure that ordinary soldiers like me die for the sake of the greater good of manipulating the political system and it also implies that things like body armor and medics should be banned from the battlefield, but I won’t worry about that here because this is a forum full of consequentialists and I honestly think that consequentialist arguments are valid anyway.
My argument here isn’t hugely important but I take some issue with the analogies. I prefer thinking in terms of both actors agreeing on acceptable level of vulnerability in order to reduce the risk of conflict. In this case, a better analogy is to the Cold War agreement not to build comprehensive ICBM defenses, an analogy which would come out in favour of limiting autonomy in weapons systems. But neither of us are placing much importance on this point overall.
I’d like to unpack this point a little bit:
Third, you might say that LAWs will prompt an arms race in AI, reducing safety. But faster AI development will help us avoid other kinds of risks unrelated to AI, and it will expedite humanity’s progress and expansion towards a future with exponentially growing value. Moreover, there is already substantial AI development in civilian sectors as well as non-battlefield military use, and all of these things have competitive dynamics. AGI would have such broad applications that restricting its use in one or two domains is unlikely to make a large difference; after all, economic power is the source of all military power, and international public opinion has nontrivial importance in international relations, and AI can help nations beat their competitors at both.
I believe discourse on AI risks often conflates ‘AI arms race’ with ‘race to the finish’. While these races are certainly linked, and therefore the conflation justified in some senses, I think it trips up the argument in this case. In an AI arms race, we should be concerned about the safety of non-AGI systems, which may be neglected in an arms race scenario. This weakens the argument that highly autonomous weapons systems might lead to fewer civilian casualties, as this is likely the sort of safety measure that might be neglected when racing to develop weapons systems capable of out-doing the ever more capable weapons of one’s rival.
The second sentence only holds if the safety issue is solved, so I don’t accept the argument that it will help humanity reach a future exponentially growing in value (at least insofar as we’re talking about the long run future, as there may be some exponential progress in the near-term).
It could simply be my reading, but I’m not entirely clear on the point made across the third and fourth sentences, and I don’t think they give a compelling case that we shouldn’t try to avoid military application or avoid exacerbating race dynamics.
Lastly, while I think you’ve given a strong case to soften opposition to advancing autonomy in weapons systems, the argument against any regulation of these weapons hasn’t been made. Not all actors seek outright bans, and I think it’d be worth acknowledging that (contrary to the title) there are some undesirable things with highly autonomous weapons systems and that we should like to impose some regulations on them such as, for example, some minimum safety requirements that help reduce civilian casualties.
Overall, I think the first point I made should cause serious pause, and it’s the largest single reason I don’t agree with your overall argument, as many good points as you make here.
(And to avoid any suspicions: despite arguing on his side, coming from the same city, and having the same rare surname, I am of no known relation to Noel Sharkey of the Stop Killer Robots Campaign, though I confess a pet goal to meet him for a pint one day.)
Hmm, everything that I mentioned applies to interstate conflict, but they don’t all only apply to interstate conflict. Intrastate conflicts might be murkier and harder to analyze, and I think they are something to be looked at, but I’m not sure how much it would modify the main points. The assumptions of the expected utility theory of conflict do get invalidated.
Fully autonomous weapons systems potentially obviate the need for a mutually beneficial social contract between the regimes in control of the weapons and the populations over which they rule. All dissent becomes easy to crush.
Well, firstly, I am of the opinion that most instances of violent resistance against governments in history were unjustified, and that a general reduction in revolutionary violence would do more good than harm. Peaceful resistance is more effective at political change than violent resistance anyway (https://www.psychologytoday.com/blog/sex-murder-and-the-meaning-life/201404/violent-versus-nonviolent-revolutions-which-way-wins). You could argue that governments will become more oppressive and less responsive to peaceful resistance if they have better security against hypothetical revolutions, though I don’t have a large expectation for this to happen, at least in the first world.
Second, this doesn’t have much to do with autonomous weapons in particular. It applies to all methods by which the government can suppress dissent, all military and police equipment.
Third, lethal force is a small and rare part of suppressing protests and dissent as long as full-fledged rebellion doesn’t break out. Modern riot police are equipped with nonlethal weapons; we can expect that any country with the ability to deploy robots would have professional capabilities for riot control and the deployment of nonlethal weapons. And crowd control is based more on psychology and appearances than application of kinetic force.
Finally, even when violent rebellion does break out, nonstate actors such as terrorists and rebels are outgunned anyway. Governments trying to pacify rebellions need to work with the local population, gather intelligence, and assert their legitimacy in the eyes of the populace. Lethal autonomous weapons are terrible for all of these things. They would be very good for the application of quick precise firepower at low risk to friendly forces, but that is far from the greatest problem faced by governments seeking to suppress dissent.
The one thing that implies that rebellion would become less frequent in a country with LAWs is that an army of AGI robots could allow leadership to stop a rebellion without worrying about the loyalty of police and soldiers. By that time, probably we should just make sure that machines have ethical guidelines not to kill their own people, support evil governments and similar things. I can see this being a problem, but it’s a little too far out and speculative to make plans around it.
This is patently bad in itself, but it also has consequences for interstate conflict; with less approval needed to go to war, inter-state conflict may increase.
The opposite is at least as likely. Nations often go to war in order to maintain legitimacy in the eyes of the population. Argentina’s Falklands venture was a good example of this ‘diversionary foreign policy’ (https://en.wikipedia.org/wiki/Diversionary_foreign_policy).
The introduction of weapons systems with high degrees of autonomy poses an arguably serious risk of geopolitical turbulence: it is not clear that all states will develop the capability to produce highly autonomous weapons systems. Those that do not will have to purchase them from technologically-more advanced allies willing to sell them. States that find themselves outside of such alliances will be highly vulnerable to attack. This may motivate a nontrivial reshuffling of global military alliances, the outcomes of which are hard to predict.
How would AI be any different here from other kinds of technological progress? And I don’t think that the advent of new military technology has major impacts on geopolitical alliances. I actually cannot think of a case where alliances shifted because of new military technology. Military exports and license production are common among non-allies, and few alliances lack advanced industrial powers; right now there are very few countries in the world which are not on good enough terms with at least one highly developed military power to buy weapons from them.
In an AI arms race, we should be concerned about the safety of non-AGI systems, which may be neglected in an arms race scenario. This weakens the argument that highly autonomous weapons systems might lead to fewer civilian casualties, as this is likely the sort of safety measure that might be neglected when racing to develop weapons systems capable of out-doing the ever more capable weapons of one’s rival.
But the same dynamic is present when nations compete with non-AI weapons. The demand for potent firepower implies that systems will cause collateral damage and that soldiers will not be as trained or disciplined on ROE as they could be.
The second sentence only holds if the safety issue is solved, so I don’t accept the argument that it will help humanity reach a future exponentially growing in value (at least insofar as we’re talking about the long run future, as there may be some exponential progress in the near-term).
Well, of course nothing matters if there is an existential catastrophe. But you can’t go into this with the assumption that AI will cause an existential catastrophe. It likely won’t, and in all those scenarios, quicker AI development is likely better. Does this mean that AI should be developed quicker, all-things-considered? I don’t know, I’m just saying that overall it’s not clear that it should be developed more slowly.
It could simply be my reading, but I’m not entirely clear on the point made across the third and fourth sentences, and I don’t think they give a compelling case that we shouldn’t try to avoid military application or avoid exacerbating race dynamics.
I just mean that military use is a comparatively small part of the overall pressure towards quicker AI development.
Lastly, while I think you’ve given a strong case to soften opposition to advancing autonomy in weapons systems, the argument against any regulation of these weapons hasn’t been made. Not all actors seek outright bans, and I think it’d be worth acknowledging that (contrary to the title) there are some undesirable things with highly autonomous weapons systems and that we should like to impose some regulations on them such as, for example, some minimum safety requirements that help reduce civilian casualties.
There are things that are wrong with AI weapons in that they are, after all, weapons, and there is always something wrong with weapons. But I think there is nothing that makes AI weapons overall worse than ordinary ones.
I don’t think that regulating them is necessarily bad. I did say at the end that testing, lobbying, international watchdogs, etc are the right direction to go in. I haven’t thought this through, but my first instinct is to say that autonomous systems should simply follow all the same regulations and laws that soldiers do today. Whenever a nation ratifies an international treaty on military conduct, such as the Geneva Convention, its norms should apply to autonomous systems as well as soldiers. That sounds sufficient to me, at first glance.
Hey kbog, Thanks for this. I think this is well argued. If I may, I’d like to pick some holes. I’m not sure if they are sufficient to swing the argument the other way, but I don’t think they’re trivial either.
I’m going to use autonomy in weapons systems in favour of LAWs for reasons argued here(see Takeaway 1).
As far as I can tell, almost all considerations you give are to inter-state conflict. The intra-state consequences are not explored and I think they deserve to be. Fully autonomous weapons systems potentially obviate the need for a mutually beneficial social contract between the regimes in control of the weapons and the populations over which they rule. All dissent becomes easy to crush. This is patently bad in itself, but it also has consequences for interstate conflict; with less approval needed to go to war, inter-state conflict may increase.
The introduction of weapons systems with high degrees of autonomy poses an arguably serious risk of geopolitical turbulence: it is not clear that all states will develop the capability to produce highly autonomous weapons systems. Those that do not will have to purchase them from technologically-more advanced allies willing to sell them. States that find themselves outside of such alliances will be highly vulnerable to attack. This may motivate a nontrivial reshuffling of global military alliances, the outcomes of which are hard to predict. For those without access to these new powerful weapons, one risk mitigation strategy is to develop nuclear weapons, potentially motivating nuclear proliferation.
On your point:
My argument here isn’t hugely important but I take some issue with the analogies. I prefer thinking in terms of both actors agreeing on acceptable level of vulnerability in order to reduce the risk of conflict. In this case, a better analogy is to the Cold War agreement not to build comprehensive ICBM defenses, an analogy which would come out in favour of limiting autonomy in weapons systems. But neither of us are placing much importance on this point overall.
I’d like to unpack this point a little bit:
I believe discourse on AI risks often conflates ‘AI arms race’ with ‘race to the finish’. While these races are certainly linked, and therefore the conflation justified in some senses, I think it trips up the argument in this case. In an AI arms race, we should be concerned about the safety of non-AGI systems, which may be neglected in an arms race scenario. This weakens the argument that highly autonomous weapons systems might lead to fewer civilian casualties, as this is likely the sort of safety measure that might be neglected when racing to develop weapons systems capable of out-doing the ever more capable weapons of one’s rival.
The second sentence only holds if the safety issue is solved, so I don’t accept the argument that it will help humanity reach a future exponentially growing in value (at least insofar as we’re talking about the long run future, as there may be some exponential progress in the near-term).
It could simply be my reading, but I’m not entirely clear on the point made across the third and fourth sentences, and I don’t think they give a compelling case that we shouldn’t try to avoid military application or avoid exacerbating race dynamics.
Lastly, while I think you’ve given a strong case to soften opposition to advancing autonomy in weapons systems, the argument against any regulation of these weapons hasn’t been made. Not all actors seek outright bans, and I think it’d be worth acknowledging that (contrary to the title) there are some undesirable things with highly autonomous weapons systems and that we should like to impose some regulations on them such as, for example, some minimum safety requirements that help reduce civilian casualties.
Overall, I think the first point I made should cause serious pause, and it’s the largest single reason I don’t agree with your overall argument, as many good points as you make here.
(And to avoid any suspicions: despite arguing on his side, coming from the same city, and having the same rare surname, I am of no known relation to Noel Sharkey of the Stop Killer Robots Campaign, though I confess a pet goal to meet him for a pint one day.)
Hmm, everything that I mentioned applies to interstate conflict, but they don’t all only apply to interstate conflict. Intrastate conflicts might be murkier and harder to analyze, and I think they are something to be looked at, but I’m not sure how much it would modify the main points. The assumptions of the expected utility theory of conflict do get invalidated.
Well, firstly, I am of the opinion that most instances of violent resistance against governments in history were unjustified, and that a general reduction in revolutionary violence would do more good than harm. Peaceful resistance is more effective at political change than violent resistance anyway (https://www.psychologytoday.com/blog/sex-murder-and-the-meaning-life/201404/violent-versus-nonviolent-revolutions-which-way-wins). You could argue that governments will become more oppressive and less responsive to peaceful resistance if they have better security against hypothetical revolutions, though I don’t have a large expectation for this to happen, at least in the first world.
Second, this doesn’t have much to do with autonomous weapons in particular. It applies to all methods by which the government can suppress dissent, all military and police equipment.
Third, lethal force is a small and rare part of suppressing protests and dissent as long as full-fledged rebellion doesn’t break out. Modern riot police are equipped with nonlethal weapons; we can expect that any country with the ability to deploy robots would have professional capabilities for riot control and the deployment of nonlethal weapons. And crowd control is based more on psychology and appearances than application of kinetic force.
Finally, even when violent rebellion does break out, nonstate actors such as terrorists and rebels are outgunned anyway. Governments trying to pacify rebellions need to work with the local population, gather intelligence, and assert their legitimacy in the eyes of the populace. Lethal autonomous weapons are terrible for all of these things. They would be very good for the application of quick precise firepower at low risk to friendly forces, but that is far from the greatest problem faced by governments seeking to suppress dissent.
The one thing that implies that rebellion would become less frequent in a country with LAWs is that an army of AGI robots could allow leadership to stop a rebellion without worrying about the loyalty of police and soldiers. By that time, probably we should just make sure that machines have ethical guidelines not to kill their own people, support evil governments and similar things. I can see this being a problem, but it’s a little too far out and speculative to make plans around it.
The opposite is at least as likely. Nations often go to war in order to maintain legitimacy in the eyes of the population. Argentina’s Falklands venture was a good example of this ‘diversionary foreign policy’ (https://en.wikipedia.org/wiki/Diversionary_foreign_policy).
How would AI be any different here from other kinds of technological progress? And I don’t think that the advent of new military technology has major impacts on geopolitical alliances. I actually cannot think of a case where alliances shifted because of new military technology. Military exports and license production are common among non-allies, and few alliances lack advanced industrial powers; right now there are very few countries in the world which are not on good enough terms with at least one highly developed military power to buy weapons from them.
But the same dynamic is present when nations compete with non-AI weapons. The demand for potent firepower implies that systems will cause collateral damage and that soldiers will not be as trained or disciplined on ROE as they could be.
Well, of course nothing matters if there is an existential catastrophe. But you can’t go into this with the assumption that AI will cause an existential catastrophe. It likely won’t, and in all those scenarios, quicker AI development is likely better. Does this mean that AI should be developed quicker, all-things-considered? I don’t know, I’m just saying that overall it’s not clear that it should be developed more slowly.
I just mean that military use is a comparatively small part of the overall pressure towards quicker AI development.
There are things that are wrong with AI weapons in that they are, after all, weapons, and there is always something wrong with weapons. But I think there is nothing that makes AI weapons overall worse than ordinary ones.
I don’t think that regulating them is necessarily bad. I did say at the end that testing, lobbying, international watchdogs, etc are the right direction to go in. I haven’t thought this through, but my first instinct is to say that autonomous systems should simply follow all the same regulations and laws that soldiers do today. Whenever a nation ratifies an international treaty on military conduct, such as the Geneva Convention, its norms should apply to autonomous systems as well as soldiers. That sounds sufficient to me, at first glance.