Hi! My name is Joris, I’m a university group organizer at PISE (the group at Erasmus University Rotterdam, the Netherlands), and I’m currently writing my thesis for my bachelor’s degree in PPE (Philosophy, Politics & Economics). I’m helping Rumtin from GCR Policy study practical policy change on x-risks and GCRs, and concretely I’m exploring the conditions and characteristics of successful policy change for extreme risks.
Policy change on low-likelihood-high-impact risks is challenging to achieve, so finding and studying cases where it has been achieved can help inform the work of the x-risk studies field. The purpose of this research is to find patterns across case studies that can tell us how extreme risk policy occurs. By looking at the key characteristics of past policy efforts, we hope to provide insights on how policy for extreme risks occurs, and to provide guidance to researchers if they wish to do policy engagement.
I have found a number of case studies from which to draw lessons, but I want to broaden my search. I’m reaching out here to ask the following: what would be useful cases of extreme risk policy to study? These case studies should be for policies that were implemented to understand, govern, or reduce extreme risks (not simply existential or globally catastrophic risks). For example, case studies could include: NASA’s asteroid tracking program; US Defense Department’s UFO detection program; closing the hole in the ozone layer; and the joint US-Soviet smallpox and polio vaccination program[1].
We want to look at some other examples (risk- and country-agnostic) to identify common factors that led to policy change, such as individual policy leadership, precipitating events, or other shifts in societal or cultural landscape. Our hypothesis is that precipitating events, particularly small-scale versions of the risk, are the critical factor in shaping policy in that area. For example, a quick review of the literature on NASA’s asteroid tracking program identifies a number of drivers for its establishment, including: Comet Shoemaker–Levy 9 hitting Jupiter in 1994, scientific consensus forming around an asteroid causing the dinosaur extinction (including with the discovery of the Chicxulub crater)[2], the releases of blockbusters Deep Impact and Armageddon[3], and the efforts of individual linked to Congress[4]
If you know of any significant extreme risk-related policy successes, we would love to hear. Bonus points if you know of literature (or people) that can help us understand why and how this policy came about!
If you’re not comfortable commenting, feel free to email me at joris [dot] pijpers [at] gmail [dot] com!
- ^
- ^
See The Precipice, p. 69-70
- ^
Carl Schulman talks about this in this 80.000 Hours podcast episode: https://80000hours.org/podcast/episodes/carl-shulman-common-sense-case-existential-risks/
- ^
Nunn-Lugar; see quick summary here: https://www.openphilanthropy.org/blog/ai-governance-grantmaking
Setting aside whether or not such risks were actually significant, perhaps planetary protection could be an interesting example of where bureaucracies spent time and money to mitigate unknown risks from e.g., extraterrestrial contamination.
The 2014 NIH moratorium on funding gain-of-function research (which was lifted in 2017)
The Asilomar Conference on Recombinant DNA, which Katja Grace has a report on: https://intelligence.org/2015/06/30/new-report-the-asilomar-conference-a-case-study-in-risk-mitigation/
Fixing the Ozone Layer should provide a whole host of important insights here.
The US decision to end their biological weapon program
The South African decision to end their nuclear weapons program and destroy their nuclear arsenal
The Nuclear Non-Proliferation Treaty (sort of; it’s unclear how much of an impact it has had, but these posts [1, 2] provide some insight)
Some more examples of risks which were probably not extreme*, but which elicited strong policy responses:
Y2K (though this might count as an extreme risk in the context of corporate governance)
Nuclear power plant accidents (in particular Three Mile Island and Chernobyl)
GMOs (both risks to human health and to the environment; see e.g. legislation in the EU, India, and Hawai’i)
various food additives (e.g. Red No. 2)
many, many novel drugs/pharmaceuticals (thalidomide, opioids, DES, Fen-phen, Seldane, Rezulin, Vioxx, Bextra, Baycol...)
*I’m not really sure how you’re defining “extreme risk”, but the examples you gave all have meaningfully life-changing implications for >10s of millions of people. These examples are lesser in severity and/or scope, but seem like they still caused strong policy responses due to overestimated risk (though this warrants being careful about ex ante vs. ex post risk) and/or unusually high concern about the topic.
Military/weapons technologies, in particular nuclear weapons, biological weapons, chemical weapons, and cyberattacks
Several infectious diseases, including COVID-19, Ebola, SARS, MERS, swine flu, HIV/AIDS, etc.
Gene-edited humans (see coverage of / responses to the twins modified by He Jiankui)