Thanks Matthijs! I really like this report and will edit a link into the post.
oeg
Karma: 192
What term to use for AI in different policy contexts?
Examining pathways through which narrow AI systems might increase the likelihood of nuclear war
I’ve just made a correction to the table here. I had previously copy-pasted the wrong values, meaning that the column “Mean negotiating time” had the values for the maximum (rather than mean) time that it took to get an agreement for each reference class. Sorry for the error.
Prospects for AI safety agreements between countries
“Risk Awareness Moments” (Rams): A concept for thinking about AI governance interventions
Thank you very much for writing this. I broadly agree with your post, but I probably put less weight than you do on the historical record. I think the crux is that I assign a higher probability to there being close calls that we don’t yet know about, but which would make the picture look very different.[1] Here are a few reasons for thinking this:
The post says that documents generally get declassified after 25 years, implying that we would hear about them eventually. This is a US rule though – other countries are less transparent, suggesting that we are undercounting their close calls.[2]
Even if documents do get declassified, it’s non-trivial to actually identify close calls. In “The Limits of Safety”, Scott Sagan documents several nuclear close calls.[3] He says that the cases he discovered were probably “just the tip of the iceberg” (p270) – essentially because the archive work to find them was so difficult.[4]
It wouldn’t necessarily be obvious to researchers that a given incident was a close call. Nuclear weapons are often thought about in terms of normal accident theory, which posits that complex systems fail in unpredictable ways.
Out of 10 close calls, 3 relate primarily to the USSR or Russia, 1 relates to both the USSR and the USA, and 6 relate primarily to the USA.[5] I can imagine that the USA and USSR/Russia would have a disproportionate share of close calls (for example because they have by far the most nuclear weapons). But it would be very surprising to me for the US to have 6 such cases, while the other nuclear weapons states – China, France, Israel, India, North Korea, Pakistan, and the UK – combined have 0.
- ^
This depends upon the claim that the cases that we know about are not necessarily representative of the entire universe of cases.
- ^
A similar point is made in this report from Chatham House (p3).
- ^
Sagan is interested in a broader range of close calls than the one in the post, but from memory I think Sagan was the first researcher to publicly identify the “Missiles over Georgia” and “power outage” cases.
- ^
Sagan also did interviews with relevant people, and submitted requests under the Freedom of Information Act. These are described in the introduction to Limits of Safety.
- ^
USSR/Russia: Petrov, Able Archer, Norwegian Rocket. USSR and USA: Cuban Missile Crisis. USA: remaining 6.
This reading list is an excellent place to start for getting a sense of China x AI (though it doesn’t have that much about China’s political objectives in general).