On flash-war risks, I think a key variable is what the actual forcing function is on decision speed and the key outcome you care about is the decision quality.
Fights where escalation is more constrained by decision making speed than weapon speed are where we should expect flash war dynamics. These could include: short-range conflicts, cyber wars, the use of directed energy weapons, influence operations/propaganda battles, etc.
For nuclear conflict, unless some country gets extremely good at stealth, strategic deception, and synchronized mass incapacitation/counterforce, there will still be warning and a delay before impact. The only reasons to respond faster than dictated by the speed of the adversary weapons and delays in your own capacity to act would be if doing so could further reduce attrition, or enable better retaliation… but I don’t see much offensive prospect for that. If the other side is doing a limited strike, then you want to delay escalation/just increase survivability, if the other side is shooting for an incapacitating strike, then their commitment will be absolutely massive and their pre-mobilization high, so retaliation would be your main option left at that point anyway. Either way, you might get bombers off the ground and cue up missile defenses, but for second strike I don’t see that much advantage to speeds faster than those imposed by the attacker, especially given the risk of acting on false alarm. This logic seems to be clearly present in all the near miss cases: there is the incentive to wait for more information from more sensors.
Improving automation in sensing quality, information fusion, and attention rationing would all seem useful for finding false alarms faster. In general it would be interesting to see more attention put into AI-enabled de-escalation, signaling, and false alarm reduction.
I think most of the examples of nuclear risk near misses favor the addition of certain types of autonomy, namely those that increase sensing redundancy and thus contribute to to improving decision quality and expanding the length of the response window. To be concrete:
For the Stanislav example: if the lights never start flashing in the first place because of the lack of radar return (e.g. if the Soviets had more space-based sensors), then there’d be no time window for Stanislav to make a disastrous mistake. The more diverse and high quality sensors you have, and the better feature detection you have, the more accurate a picture you will have and the harder it will be for the other side to trick you.
If during the Cuban missile crisis, the submarine which Arkhipov was on knew that the U.S. was merely dropping signaling charges (not attacking), then there would have been no debate about nuclear attack: the Soviets would have just known they’d been found.
In the training tape false alarm scenario: U.S. ICBMs can wait to respond because weapon arrival is not instant, satellite sensors all refute the false alarm: catastrophe averted. If you get really redundant sensor systems that can autonomously refute false alarms, you don’t get such a threatening alert in the first place, just a warning that something is broken in your overall warning infrastructure: this is exactly what you want.
Full automation of NC3 is basically a decision to attack, and something you’d only want to activate at the end of a decision window where you are confident that you are being attacked.
Thanks for engaging so closely with the report! I really appreciate this comment.
Agreed on the weapon speed vs. decision speed distinction — the physical limits to the speed of war are real. I do think, however, that flash wars can make non-flash wars more likely (eg cyber flash war unintentionally intrudes on NC3 system components, that gets misinterpreted as preparation for a first strike, etc.). I should have probably spelled that out more clearly in the report.
I think we actually agree on the broader point — it is possible to leverage autonomous systems and AI to make the world safer, to lengthen decision-making windows, to make early warning and decision-support systems more reliable.
But I don’t think that’s a given. It depends on good choices. The key questions for us are therefore: How do we shape the future adoption of these systems to make sure that’s the world we’re in? How can we trust that our adversaries are doing the same thing? How can we make sure that our confidence in some of these systems is well-calibrated to their capabilities? That’s partly why a ban probably isn’t the right framing.
I also think this exchange illustrates why we need more research on the strategic stability questions.
Thanks for this analysis, I found this a very interesting report! As we’ve discussed, there are a number of convergent lines of analysis, which Di Cooke, Kayla Matteucci and I also came to for our research paper ‘Military Artificial Intelligence as Contributor to Global Catastrophic Risk’ on the EA Forum ( link ; SSRN).
Although by comparison we focused more on the operational and logistical limits to producing and using LAWS swarms en masse, and we sliced the nuclear risk escalation scenarios slightly different. We also put less focus on the question of ‘given this risk portfolio, what governance interventions are more/less useful’.
This is part of ongoing work (including a larger project and article that also examines the military developers/operators angle on AGI alignment/misuse risks, and the ‘arsenal overhang (extant military [& nuclear] infrastructures) as a contributor to misalignment risk’ arguments (for the latter, see also some of Michael Aird’s discussion here), though that had to be cut from this chapter for reasons of length and focus.
I always don’t know if it is appropriate to put links on own articles in the comments. Will it be seen as just self-advertising? Or they may contribute to discussion?
I believe by your definition, lethal autonomous weapon systems already exist and are widely in use by the US military. For example, the CIWS system will fire on targets like rapidly moving nearby ships without any human intervention.
It’s tricky because there is no clear line between “autonomous” and “not autonomous”. Is a land mine autonomous because it decides to explode without human intervention? Well, land mines could have more and more advanced heuristics slowly built into them. At what point does it become autonomous?
I’m curious what ethical norms you think should apply to a system like the CIWS, designed to autonomously engage, but within a relatively restricted area, i.e. “there’s something coming fast toward our battleship, let’s shoot it out of the air even though the algorithm doesn’t know exactly what it is and we don’t have time to get a human into the loop”.
Thank you for your comment and thanks for reading :)
The key question for us is not “what is autonomy?” — that’s bogged down the UN debates for years — but rather “what are the systemic risks of certain military AI applications, including a spectrum of autonomous capabilities?” I think many systems around today are better thought of as closer to “automated” than truly “autonomous,” as I mention in the report, but again, I think that binary distinctions like that are less salient than many people think. What we care about is the multi-dimensional problem of more and more autonomy in more and more systems, and how that can destabilize the international system.
I agree with your point that it’s a tricky definitional problem. In point 3 under the section on the “Killer Robot Ban” in the report, one of the key issues there is “The line between autonomous and automated systems is blurry.” I think you’re pointing to a key problem with how people often think about this issue.
I’m sorry I won’t be able to give a satisfying answer about “ethical norms” as it’s a bit outside the purview of the report, which focuses more on strategic stability and GCRs. (I will say that I think the idea of “human in the loop” is not the solution it’s often made out to be, given some of the issues with speed and cognitive biases discussed in the report). There are some people doing good work on related questions in international humanitarian law though that will give a much more interesting answer.
Great report! Looking forward to digging into it more.
It definitely makes sense to focus on (major) states. However a different intervention I don’t think I saw in the piece is about targeting the private sector—those actually developing the tech. E.g. Reprogramming war by Pax for Peace, a Dutch NGO. They describe the project as follows:
“This is part of the PAX project aimed at dissuading the private sector from contributing to the development of lethal autonomous weapons. These weapons pose a serious threat to international peace and security, and would violate fundamental legal and ethical principles. PAX aims to engage with the private sector to help prevent lethal autonomous weapons from becoming a reality. In a series of four reports we look into which actors could potentially be involved in the development of these weapons. Each report looks at a different group of actors, namely states, the tech sector, universities & research institutes, and arms producers. This project is aimed at creating awareness in the private sector about the concerns related to lethal autonomous weapons, and at working with private sector actors to develop guidelines and regulations to ensure their work does not contribute to the development of these weapons.”
It follows fairly successful investor campaigns on e.g. cluster munitions. This project could form the basis for shareholder activism or divestment by investors, and/or wider activism by the AI community by students, researchers, employees, etc—building on eg FLI’s “we won’t work on LAWS” pledge.
I’d be interested in your views on that kind of approach.
That’s a great point. I think you’re right — I should have dug a bit deeper on how the private sector fits into this.
I think cyber is an example where the private sector has really helped to lead — like Microsoft’s involvement at the UN debates, the Paris Call, the Cybersecurity Tech Accord, and others — and maybe that’s an example of how industry stakeholders can be engaged.
I also think that TEVV-related norms and confidence building measures would probably involve leading companies.
I still broadly thinking that states are the lever to target at this stage in the problem, given that they would be (or are) driving demand. I am also always a little unsure about using cluster munitions as an example of success — both because I think autonomous weapons are just a different beast in terms of military utility, and of course because of the breaches (including recently).
Thank you again for pointing out that hole in the report!
I don’t think its a hole at all, I think its quite reasonable to focus on major states. The private sector approach is a different one with a whole different set of actors/interventions/literature—completely makes sense that its outside the scope of this report. I was just doing classic whatabouterism, wondering about your take on a related but seperate approach.
Btw I completely agree with you about cluster munitions.
The cluster munitions divestment example seems plausibly somewhat more successful in the West, but not elsewhere (e.g. the companies that remain on the “Hall of Shame” list). I’d expect something similar here if the pressure against LAWs were narrow (e.g. against particular types with low strategic value). Decreased demand does seem more relevant than decreased investment though.
If LAWs are stigmatized entirely, and countries like the U.S. don’t see a way to tech their way out to sustain advantage, then you might not get the same degree of influence in the first place since demand remains.
I find it interesting that the U.S. wouldn’t sign the Convention on Cluster Munitions, but also doesn’t seem to be buying or selling any more. One implication might be that the stigma disincentivizes change/tech progress: since more discriminant cluster munitions would be stigmatized as well. I presume this reduces the number of such weapons, but increases the risk of collateral damage per weapon by slowing the removal of older, more indiscriminate/failure prone weapons from arsenals.
While in principle, you could drive down the civilian harm with new smaller bomblets that reliably deactivate themselves if they don’t find a military target, as far as I can tell, to the degree that the U.S. is replacing cluster bombs, it is just doing so with big indiscriminate bombs (BLU 136/BLU134) that will just shower a large target area with fragments.
On flash-war risks, I think a key variable is what the actual forcing function is on decision speed and the key outcome you care about is the decision quality.
Fights where escalation is more constrained by decision making speed than weapon speed are where we should expect flash war dynamics. These could include: short-range conflicts, cyber wars, the use of directed energy weapons, influence operations/propaganda battles, etc.
For nuclear conflict, unless some country gets extremely good at stealth, strategic deception, and synchronized mass incapacitation/counterforce, there will still be warning and a delay before impact. The only reasons to respond faster than dictated by the speed of the adversary weapons and delays in your own capacity to act would be if doing so could further reduce attrition, or enable better retaliation… but I don’t see much offensive prospect for that. If the other side is doing a limited strike, then you want to delay escalation/just increase survivability, if the other side is shooting for an incapacitating strike, then their commitment will be absolutely massive and their pre-mobilization high, so retaliation would be your main option left at that point anyway. Either way, you might get bombers off the ground and cue up missile defenses, but for second strike I don’t see that much advantage to speeds faster than those imposed by the attacker, especially given the risk of acting on false alarm. This logic seems to be clearly present in all the near miss cases: there is the incentive to wait for more information from more sensors.
Improving automation in sensing quality, information fusion, and attention rationing would all seem useful for finding false alarms faster. In general it would be interesting to see more attention put into AI-enabled de-escalation, signaling, and false alarm reduction.
I think most of the examples of nuclear risk near misses favor the addition of certain types of autonomy, namely those that increase sensing redundancy and thus contribute to to improving decision quality and expanding the length of the response window. To be concrete:
For the Stanislav example: if the lights never start flashing in the first place because of the lack of radar return (e.g. if the Soviets had more space-based sensors), then there’d be no time window for Stanislav to make a disastrous mistake. The more diverse and high quality sensors you have, and the better feature detection you have, the more accurate a picture you will have and the harder it will be for the other side to trick you.
If during the Cuban missile crisis, the submarine which Arkhipov was on knew that the U.S. was merely dropping signaling charges (not attacking), then there would have been no debate about nuclear attack: the Soviets would have just known they’d been found.
In the training tape false alarm scenario: U.S. ICBMs can wait to respond because weapon arrival is not instant, satellite sensors all refute the false alarm: catastrophe averted. If you get really redundant sensor systems that can autonomously refute false alarms, you don’t get such a threatening alert in the first place, just a warning that something is broken in your overall warning infrastructure: this is exactly what you want.
Full automation of NC3 is basically a decision to attack, and something you’d only want to activate at the end of a decision window where you are confident that you are being attacked.
Thanks for engaging so closely with the report! I really appreciate this comment.
Agreed on the weapon speed vs. decision speed distinction — the physical limits to the speed of war are real. I do think, however, that flash wars can make non-flash wars more likely (eg cyber flash war unintentionally intrudes on NC3 system components, that gets misinterpreted as preparation for a first strike, etc.). I should have probably spelled that out more clearly in the report.
I think we actually agree on the broader point — it is possible to leverage autonomous systems and AI to make the world safer, to lengthen decision-making windows, to make early warning and decision-support systems more reliable.
But I don’t think that’s a given. It depends on good choices. The key questions for us are therefore: How do we shape the future adoption of these systems to make sure that’s the world we’re in? How can we trust that our adversaries are doing the same thing? How can we make sure that our confidence in some of these systems is well-calibrated to their capabilities? That’s partly why a ban probably isn’t the right framing.
I also think this exchange illustrates why we need more research on the strategic stability questions.
Thanks again for the comment!
Thanks for this analysis, I found this a very interesting report! As we’ve discussed, there are a number of convergent lines of analysis, which Di Cooke, Kayla Matteucci and I also came to for our research paper ‘Military Artificial Intelligence as Contributor to Global Catastrophic Risk’ on the EA Forum ( link ; SSRN).
Although by comparison we focused more on the operational and logistical limits to producing and using LAWS swarms en masse, and we sliced the nuclear risk escalation scenarios slightly different. We also put less focus on the question of ‘given this risk portfolio, what governance interventions are more/less useful’.
This is part of ongoing work (including a larger project and article that also examines the military developers/operators angle on AGI alignment/misuse risks, and the ‘arsenal overhang (extant military [& nuclear] infrastructures) as a contributor to misalignment risk’ arguments (for the latter, see also some of Michael Aird’s discussion here), though that had to be cut from this chapter for reasons of length and focus.
I always don’t know if it is appropriate to put links on own articles in the comments. Will it be seen as just self-advertising? Or they may contribute to discussion?
I looked at these problems in two articles:
Could slaughterbots wipe out humanity? Assessment of the global catastrophic risk posed by autonomous weapons
and
Military AI as a Convergent Goal of Self-Improving AI
I believe by your definition, lethal autonomous weapon systems already exist and are widely in use by the US military. For example, the CIWS system will fire on targets like rapidly moving nearby ships without any human intervention.
https://en.wikipedia.org/wiki/Phalanx_CIWS
It’s tricky because there is no clear line between “autonomous” and “not autonomous”. Is a land mine autonomous because it decides to explode without human intervention? Well, land mines could have more and more advanced heuristics slowly built into them. At what point does it become autonomous?
I’m curious what ethical norms you think should apply to a system like the CIWS, designed to autonomously engage, but within a relatively restricted area, i.e. “there’s something coming fast toward our battleship, let’s shoot it out of the air even though the algorithm doesn’t know exactly what it is and we don’t have time to get a human into the loop”.
Hi Kevin,
Thank you for your comment and thanks for reading :)
The key question for us is not “what is autonomy?” — that’s bogged down the UN debates for years — but rather “what are the systemic risks of certain military AI applications, including a spectrum of autonomous capabilities?” I think many systems around today are better thought of as closer to “automated” than truly “autonomous,” as I mention in the report, but again, I think that binary distinctions like that are less salient than many people think. What we care about is the multi-dimensional problem of more and more autonomy in more and more systems, and how that can destabilize the international system.
I agree with your point that it’s a tricky definitional problem. In point 3 under the section on the “Killer Robot Ban” in the report, one of the key issues there is “The line between autonomous and automated systems is blurry.” I think you’re pointing to a key problem with how people often think about this issue.
I’m sorry I won’t be able to give a satisfying answer about “ethical norms” as it’s a bit outside the purview of the report, which focuses more on strategic stability and GCRs. (I will say that I think the idea of “human in the loop” is not the solution it’s often made out to be, given some of the issues with speed and cognitive biases discussed in the report). There are some people doing good work on related questions in international humanitarian law though that will give a much more interesting answer.
Thanks again!
Great report! Looking forward to digging into it more.
It definitely makes sense to focus on (major) states. However a different intervention I don’t think I saw in the piece is about targeting the private sector—those actually developing the tech. E.g. Reprogramming war by Pax for Peace, a Dutch NGO. They describe the project as follows:
It follows fairly successful investor campaigns on e.g. cluster munitions. This project could form the basis for shareholder activism or divestment by investors, and/or wider activism by the AI community by students, researchers, employees, etc—building on eg FLI’s “we won’t work on LAWS” pledge.
I’d be interested in your views on that kind of approach.
Hi Haydn,
That’s a great point. I think you’re right — I should have dug a bit deeper on how the private sector fits into this.
I think cyber is an example where the private sector has really helped to lead — like Microsoft’s involvement at the UN debates, the Paris Call, the Cybersecurity Tech Accord, and others — and maybe that’s an example of how industry stakeholders can be engaged.
I also think that TEVV-related norms and confidence building measures would probably involve leading companies.
I still broadly thinking that states are the lever to target at this stage in the problem, given that they would be (or are) driving demand. I am also always a little unsure about using cluster munitions as an example of success — both because I think autonomous weapons are just a different beast in terms of military utility, and of course because of the breaches (including recently).
Thank you again for pointing out that hole in the report!
I don’t think its a hole at all, I think its quite reasonable to focus on major states. The private sector approach is a different one with a whole different set of actors/interventions/literature—completely makes sense that its outside the scope of this report. I was just doing classic whatabouterism, wondering about your take on a related but seperate approach.
Btw I completely agree with you about cluster munitions.
The cluster munitions divestment example seems plausibly somewhat more successful in the West, but not elsewhere (e.g. the companies that remain on the “Hall of Shame” list). I’d expect something similar here if the pressure against LAWs were narrow (e.g. against particular types with low strategic value). Decreased demand does seem more relevant than decreased investment though.
If LAWs are stigmatized entirely, and countries like the U.S. don’t see a way to tech their way out to sustain advantage, then you might not get the same degree of influence in the first place since demand remains.
I find it interesting that the U.S. wouldn’t sign the Convention on Cluster Munitions, but also doesn’t seem to be buying or selling any more. One implication might be that the stigma disincentivizes change/tech progress: since more discriminant cluster munitions would be stigmatized as well. I presume this reduces the number of such weapons, but increases the risk of collateral damage per weapon by slowing the removal of older, more indiscriminate/failure prone weapons from arsenals.
https://www.washingtonpost.com/news/checkpoint/wp/2016/09/02/why-the-last-u-s-company-making-cluster-bombs-wont-produce-them-anymore/
While in principle, you could drive down the civilian harm with new smaller bomblets that reliably deactivate themselves if they don’t find a military target, as far as I can tell, to the degree that the U.S. is replacing cluster bombs, it is just doing so with big indiscriminate bombs (BLU 136/BLU134) that will just shower a large target area with fragments.