christian.r

• christian.r 4 Mar 2022 19:01 UTC

  11 points

  0 ∶ 0

  on: The Fu­ture Fund’s Pro­ject Ideas Competition

  A Project Candor for Global Catastrophic Risks

  Biorisk and Recovery from Catastrophe, Values and Reflective Processes, Effective Altruism

  This is a proposal to fund a large-scale public communications project on global catastrophic risks (GCRs), modeled on the Eisenhower administration’s Project Candor. Project Candor was a Cold War public relations campaign to “inform the public of the realities of the ‘Age of Peril’” (see Unclassified 1953 Memo from Eisenhower Library). Policymakers were concerned that the public did not yet understand that the threats from nuclear weapons and the Soviet Union had inaugurated a new era in human history: the Age of Peril. Today, at the precipice, the Age of Peril continues with possible risks from engineered pandemics, thermonuclear exchange, great power war, and more. Voting behavior and public discourse, however, do not seem attuned to these risks. A new privately-funded Project Candor would communicate to the public the nature of the threats, their probabilities, and what we can do about them. This proposal is related to “a fund for movies and documentaries” and “new publications on the most pressing issues,” but differs in that it would be a unified and coordinated campaign across multiple media.

• christian.r 4 Mar 2022 19:18 UTC

  8 points

  0 ∶ 0

  on: The Fu­ture Fund’s Pro­ject Ideas Competition

  Creative Arms Control

  Biorisk and Recovery from Catastrophe

  This is a proposal to fund research efforts on “creative arms control,” or non-treaty-based international governance mechanisms. Traditional arms control—formal treaty-based international agreements—has fallen out of favor among some states, to the extent that some prominent policymakers have asked whether we’ve reached “The End of Arms Control.”^[1] Treaties are difficult to negotiate and may be poorly suited to some fast-moving issues like autonomous weapons, synthetic biology, and cyber operations; by the time traditional arms control is negotiated, the technology may have outpaced the regulations. Partly for this reason, states and private actors alike have increasingly turned to informal “norms” processes (e.g. on cyber), codes of conduct and technical agreements, or confidence-building measures (CBMs). How well does such “creative arms control” work? Is it a suitable instrument for regulating emerging technologies this century? How hard is it to turn a norms process into a verification-based treaty regime? Research on these questions is still thin, and greater funding could therefore be very valuable for future regulation on GCR/​X-risk-related technologies.

  1. ^

     For a while, it seemed like the 2021 extension of New START had invalidated Ambassador Brooks’s points in that 2020 article. I think Russia’s 2022 invasion of Ukraine is going to make agreement on formal arms control very difficult again.

Rad­i­cal Ca­reer Tran­si­tions: Skunk Bax­ter and the Bare­foot Contessa

• christian.r 4 Mar 2022 22:47 UTC

  3 points

  0 ∶ 0

  on: The Fu­ture Fund’s Pro­ject Ideas Competition

  Experimental Wargames for Great Power War and Biological Warfare

  Biorisk and Recovery from Catastrophe, Epistemic Institutions

  This is a proposal to fund a series of “experimental wargames,” on great power war and biological warfare. Wargames have long been a standard tool of think tanks, the military, and the academic IR world since the early Cold War. Until recently, however, these games were largely used to uncover unknown unknowns and help with scenario planning. Most such games continue to be unscientific exercises. Recent work on “experimental wargames” (see, e.g. this paper on drones and escalation), however, has leveraged wargaming methods with randomly-assigned groups and varying scenarios to see how decision-makers will react in hypothetical crisis situations. A series of well-designed experimental wargames on crisis decision-making in a great power confrontation or during a biological attack could help identify weaknesses, quantify risks, and uncover cognitive biases at work in high-pressure decision-making. Additionally, they would have the added benefit of raising awareness about global catastrophic risks.

• christian.r 8 Mar 2022 16:37 UTC

  24 points

  0 ∶ 0

  on: Com­par­ing top fore­cast­ers and do­main experts

  Thank you for writing this overview! I think it’s very useful. A few notes on the famous “30%” claim:

  • Part of the problem with fully understanding the performance of IC analysts is that much of the information about the tournaments and the ICPM is classified.

  • What originally happened is that someone leaked info about ACE to David Ignatius, who then published it in his column. (The IC never denied the claim.^[1]) The document you cite is part of a case study by MITRE that’s been approved for public release.

  One under-appreciated takeaway that you hint at is that prediction markets (rather than non-market aggregation platforms) are poorly suited to classified environments. Here’s a quote from a white paper I co-wrote last year:^[2]

  “Prediction markets are especially challenging to implement in classified environments because classified markets will necessarily have large limitations on participation, requiring the use of algorithmic correctives to solve liquidity problems. Good liquidity, like that of a well-functioning stock market, is difficult to achieve in prediction markets like the ICPM, requiring prediction markets to have corrective tools like setting liquidity parameters and using automated market makers, which attempt to simulate efficient market behavior in electronic prediction markets.”

  More broadly, I would like to push back a little against the idea that your point 3(a) ( whether supers outperform IC analysts) is really much evidence for or against 3 (whether supers outperform domain experts).

  First, the IARPA tournaments asked a wide range of questions, but intelligence analysts tend to be specialized. If you’re looking at the ICPM, are you really looking at the performance of domain experts? Or are you looking at e.g. an expert on politics in the Horn of Africa trying to forecast the price of the Ruble? On the one hand, since participants self-selected which questions they answered, we might expect domain experts to stick to their domain. On the other, analysts might have seen it as a “game,” a “break,” or “professional development”—in short, an opportunity to try their hand something outside their expertise. The point is that we simply don’t know whether the ICPM really reflects “expert” opinion.

  Second, I am inclined to believe that comparisons between IC analysts and supers may tell us more about the secrecy heuristic than about forecaster performance. From the same white paper:

  “Experimental research on using secrecy as a heuristic for informational quality demonstrates that people tend to weigh secret information more heavily than publicly available information, viewing secret information as higher quality than public information.^[3] Secrecy does matter, especially in situations where information asymmetry exists, but a pervasive secrecy bias may negatively affect the accuracy of a classified crowd in some cases.”

  I personally see much of the promise of forecasting platforms not as a tool for beating experts, but as a tool for identifying them more reliably (more reliably than by the usual signals, like a PhD).

  1. ^

     Tetlock discusses this a bit in Chapter 4 of Superforecasting.

  2. ^

     Keeping Score: A New Approach to Geopolitical Forecasting, https://​​global.upenn.edu/​​sites/​​default/​​files/​​perry-world-house/​​Keeping%20Score%20Forecasting%20White%20Paper.pdf.

  3. ^

     Travers et al., “The Secrecy Heuristic,” https://​​www.jstor.org/​​stable/​​43785861.

  What links here?

  • Gavin's comment on Com­par­ing top fore­cast­ers and do­main experts by Gavin (8 Mar 2022 18:35 UTC; 3 points)

• christian.r 8 Mar 2022 20:10 UTC

  5 points

  0 ∶ 0

  in reply to: Gavin’s comment on: Com­par­ing top fore­cast­ers and do­main experts

  Thanks Gavin! That makes sense on how you view this and (3).

• christian.r 9 May 2022 12:12 UTC

  4 points

  0 ∶ 0

  on: Space gov­er­nance—prob­lem profile

  Hi Fin!

  This is great. Thank you for writing it up and posting it! I gave it a strong upvote.

  (TLDR for what follows: I think this is very neglected, but I’m highly uncertain about tractability of formal treaty-based regulation)

  As you know, I did some space policy-related work at a think tank about a year ago, and one of the things that surprised us most is how neglected the issue is — there are only a handful of organizations seriously working on it, and very few of them are the kinds of well-connected and -respected think tanks that actually influence policy (CSIS is one). This is especially surprising because — as Jackson Wagner writes below — so much of space governance runs through U.S. policy. Anyway, I think that’s another point in favor of working on this!

  As I think I mentioned when we talked about space stuff a little while ago, I’m a bit skeptical about tractability of “traditional” (ie formal, treaty-based) arms control. You note some of the challenges in the 80K version of the write up. Getting the major powers to agree to anything right now, let alone something as sensitive as space tech, seems unlikely. Moreover, the difficulties of verification and ease of cheating are high, as they are with all dual-use technology. Someone can come up with a nice “debris clean up” system that just happens to also be a co-orbital ASAT, for example.

  But I think there are other mechanisms for creating “rules of the orbit” — that’s the word Simonetta di Pippo, the director of UNOOSA used at a workshop I helped organize last year. (https://​​global.upenn.edu/​​sites/​​default/​​files/​​perry-world-house/​​Dipippo_SpaceWorkshop.pdf)

  Cyber is an example where a lot of actors have apparently decided that treaty-based arms control isn’t going to cut it (in part for political reasons, in part because the tech moves so fast), but there are still serious attempts at creating norms and regulation (https://​​carnegieendowment.org/​​2020/​​02/​​26/​​cyberspace-and-geopolitics-assessing-global-cybersecurity-norm-processes-at-crossroads-pub-81110). That includes standard setting and industry-driven processes, which feel especially appropriate in space, where private actors play such an important role. We have a report on autonomous weapons and AI-enabled warfare coming out soon at Founders Pledge, and I think that’s another space where people put too much emphasis on treaty-based regulation and neglect norms and confidence building measures for issues where great powers can agree on risk reduction.

  Again, I think this is a great write up, and love that you are drawing attention to these issues. Thank you!

• christian.r 19 May 2022 10:35 UTC

  74 points

  0 ∶ 0

  on: Are you re­ally in a race? The Cau­tion­ary Tales of Szilárd and Ellsberg

  Hi Haydn,

  This is awesome! Thank you for writing and posting it. I especially liked the description of the atmosphere at RAND, and big +1 on the secrecy heuristic being a possibly big problem.^[1] Some people think it helps explain intelligence analysts’ underperformance in the forecasting tournaments, and I think there might be something to that explanation.

  We have a report on autonomous weapons systems and military AI applications coming out soon (hopefully later today) that gets into the issue of capability (mis)perception in arms races too, and your points on competition with China are well taken.

  What I felt was missing from the post was the counterfactual: what if the atomic scientists’ and defense intellectuals’ worst fears about their adversaries had been correct? It’s not hard to imagine. The USSR did seem poised to dominate in rocket capabilities at the time of Sputnik.

  I think there’s some hindsight bias going on here. In the face of high uncertainty about an adversary’s intentions and capabilities, it’s not obvious to me that skepticism is the right response. Rather, we should weigh possible outcomes. In the Manhattan Project case, one of those possible outcomes was that a murderous totalitarian regime would be the first to develop nuclear weapons, become a permanent regional hegemon, or worse, a global superpower. I think the atomic scientists’ and U.S. leadership’s decision then was the right one, given their uncertainties at the time.

  I think it would be especially interesting to see whether misperception is actually more common historically. But I think there are examples of “racing” where assessments were accurate or even under-confident (as you mention, thermonuclear weapons).

  Thanks again for writing this! I think you raise a really important question — when is AI competition “suboptimal”?^[2]

  1. ^

     https://​​www.jstor.org/​​stable/​​43785861

  2. ^

     In Charles Glaser’s sense (https://​​www.belfercenter.org/​​sites/​​default/​​files/​​files/​​publication/​​glaser.pdf)

  What links here?

  • Why poli­cy­mak­ers should be­ware claims of new “arms races” (Bul­letin of the Atomic Scien­tists) by christian.r (14 Jul 2022 13:38 UTC; 55 points)

• christian.r 19 May 2022 13:10 UTC

  13 points

  0 ∶ 0

  in reply to: HaydnBelfield’s comment on: Are you re­ally in a race? The Cau­tion­ary Tales of Szilárd and Ellsberg

  Thank you for the reply! I definitely didn’t mean to mischaracterize your opinions on that case :)

  Agreed, a project like that would be great. Another point in favor of your argument that this is a dynamic to watch out for on AI competition is if verifying claims of superiority is harder for software (along the lines of Missy Cummings’s “The AI That Wasn’t There” https://​​tnsr.org/​​roundtable/​​policy-roundtable-artificial-intelligence-and-international-security/​​#essay2). That seems especially vulnerable to misperceptions

• christian.r 19 May 2022 14:07 UTC

  3 points

  0 ∶ 0

  in reply to: HaydnBelfield’s comment on: Risks from Au­tonomous Weapon Sys­tems and Mili­tary AI

  Hi Haydn,

  That’s a great point. I think you’re right — I should have dug a bit deeper on how the private sector fits into this.

  I think cyber is an example where the private sector has really helped to lead — like Microsoft’s involvement at the UN debates, the Paris Call, the Cybersecurity Tech Accord, and others — and maybe that’s an example of how industry stakeholders can be engaged.

  I also think that TEVV-related norms and confidence building measures would probably involve leading companies.

  I still broadly thinking that states are the lever to target at this stage in the problem, given that they would be (or are) driving demand. I am also always a little unsure about using cluster munitions as an example of success — both because I think autonomous weapons are just a different beast in terms of military utility, and of course because of the breaches (including recently).

  Thank you again for pointing out that hole in the report!

• christian.r 20 May 2022 0:43 UTC

  11 points

  0 ∶ 0

  in reply to: Kevin Lacker’s comment on: Risks from Au­tonomous Weapon Sys­tems and Mili­tary AI

  Hi Kevin,

  Thank you for your comment and thanks for reading :)

  The key question for us is not “what is autonomy?” — that’s bogged down the UN debates for years — but rather “what are the systemic risks of certain military AI applications, including a spectrum of autonomous capabilities?” I think many systems around today are better thought of as closer to “automated” than truly “autonomous,” as I mention in the report, but again, I think that binary distinctions like that are less salient than many people think. What we care about is the multi-dimensional problem of more and more autonomy in more and more systems, and how that can destabilize the international system.

  I agree with your point that it’s a tricky definitional problem. In point 3 under the section on the “Killer Robot Ban” in the report, one of the key issues there is “The line between autonomous and automated systems is blurry.” I think you’re pointing to a key problem with how people often think about this issue.

  I’m sorry I won’t be able to give a satisfying answer about “ethical norms” as it’s a bit outside the purview of the report, which focuses more on strategic stability and GCRs. (I will say that I think the idea of “human in the loop” is not the solution it’s often made out to be, given some of the issues with speed and cognitive biases discussed in the report). There are some people doing good work on related questions in international humanitarian law though that will give a much more interesting answer.

  Thanks again!

• christian.r 20 May 2022 16:53 UTC

  6 points

  0 ∶ 0

  in reply to: Gentzel’s comment on: Risks from Au­tonomous Weapon Sys­tems and Mili­tary AI

  Thanks for engaging so closely with the report! I really appreciate this comment.

  Agreed on the weapon speed vs. decision speed distinction — the physical limits to the speed of war are real. I do think, however, that flash wars can make non-flash wars more likely (eg cyber flash war unintentionally intrudes on NC3 system components, that gets misinterpreted as preparation for a first strike, etc.). I should have probably spelled that out more clearly in the report.

  I think we actually agree on the broader point — it is possible to leverage autonomous systems and AI to make the world safer, to lengthen decision-making windows, to make early warning and decision-support systems more reliable.

  But I don’t think that’s a given. It depends on good choices. The key questions for us are therefore: How do we shape the future adoption of these systems to make sure that’s the world we’re in? How can we trust that our adversaries are doing the same thing? How can we make sure that our confidence in some of these systems is well-calibrated to their capabilities? That’s partly why a ban probably isn’t the right framing.

  I also think this exchange illustrates why we need more research on the strategic stability questions.

  Thanks again for the comment!

Build­ing a Bet­ter Dooms­day Clock

• christian.r 26 May 2022 13:12 UTC

  6 points

  0 ∶ 0

  in reply to: Leo’s comment on: Build­ing a Bet­ter Dooms­day Clock

  Thank you! I also really struggle with the clock metaphor. It seems to have just gotten locked in as the Bulletin took off in the early Cold War. The time bomb is a great suggestion — it communicates the idea much better

Why poli­cy­mak­ers should be­ware claims of new “arms races” (Bul­letin of the Atomic Scien­tists)

• christian.r 7 Aug 2022 12:43 UTC

  5 points

  0 ∶ 0

  on: Is­lands, nu­clear win­ter, and trade dis­rup­tion as a hu­man ex­is­ten­tial risk factor

  Really enjoyed reading this and learned a lot. Thank you for writing it! I’m especially intrigued by the proposal for regional alliances in table 6 — including the added bit about expansionist regional powers in the co-benefits column of the linked supplemental version of the table.

  I was curious about one part of the paper on volcanic eruptions. You wrote that eg “Indonesia harbours many of the world’s large volcanoes from which an ASRS could originate (eg, Toba and Tambora eruptions).” Just eyeballing maps of the biggest known volcanoes, the overlap with some island refuges seems concerning. Do we know what ash deposit models say about how much these places would get covered in ash for various kinds of volcanic eruptions in the region and what this would mean for infrastructure and agriculture?

  Thanks!

• christian.r 8 Aug 2022 11:21 UTC

  1 point

  0 ∶ 0

  in reply to: Matt Boyd’s comment on: Is­lands, nu­clear win­ter, and trade dis­rup­tion as a hu­man ex­is­ten­tial risk factor

  Thanks, Matt! That makes sense

• christian.r 15 Aug 2022 18:22 UTC

  10 points

  0 ∶ 0

  on: The Parable of the Boy Who Cried 5% Chance of Wolf

  Thanks for writing this! I think it’s great. Reminds me of another wild animal metaphor about high-stakes decision-making under uncertainty—Reagan’s 1984 “Bear in the Woods” campaign ad:

  There is a bear in the woods. For some people, the bear is easy to see. Others don’t see it at all. Some people say the bear is tame. Others say it’s vicious and dangerous. Since no one can really be sure who’s right, isn’t it smart to be as strong as the bear—if there is a bear?

  I think that kind of reasoning is helpful when communicating about GCRs and X-risks.

• christian.r 16 Sep 2022 16:08 UTC

  18 points

  2 ∶ 0

  on: The Pug­wash Con­fer­ences and the Anti-Bal­lis­tic Mis­sile Treaty as a case study of Track II diplomacy

  Hi Rani, it’s great to see the report out. It’s good to have this clear deep dive on the canonical case. I especially like that it points to some attributes of track II dialogues that we should pay special attention to when evaluating them as potential interventions. Great work!

Sixty years af­ter the Cuban Mis­sile Cri­sis, a new era of global catas­trophic risks