Haydn has been a Research Associate and Academic Project Manager at the University of Cambridge’s Centre for the Study of Existential Risk since Jan 2017.
HaydnBelfield
Karma: 3,749
Thanks for sharing this! One thing I thought you might be interested in is the Pope and Pontifical Academy of Sciences involvement with work on climate change, and biodiversity.
In a recent post (Lord Martin Rees: an appreciation) I noted:
“One particularly notable example on climate is his work at the Vatican. In May 2014, he helped Sir Partha Dasgupta co-organise a major workshop with the Pontifical Academy of Sciences on climate change. After the workshop, Sir Partha spoke to the Pope directly and encouraged him to include climate change in his speeches and to urge people to be better stewards of the planet. The workshop underpinned a major report published in April 2015 by the Vatican. The report in turn partly informed the May 2015 Laudato si’ Papal Encyclical, which focussed on the impending threat of climate change and was influential in encouraging the 1.3 billion Catholics worldwide to support for the Paris Agreement, agreed in December 2015.”
I agree that if I could only recommend one book, it would probably be the Precipice - as its more up-to-date and comprehensive. I was thinking a wider bibliography / reading list. However, I really would prioritise the two Ted talks as short, interesting, credible intros.
I’ve got a real soft spot for “Our Final Century: Will Civilisation Survive the Twenty-first Century?” as it was the book that originally got me interested in existential risk. I still think its really important for the field, and is usefully included alongside 2008′s GCR and Bostrom’s 2002 paper. We’re actually working on an “updated after 20 years” version of the book, hopefully out next year.
Yes he’s said this very consistently for years. Its interesting for like the insider’s insider (Astronomer Royal!) to advocate an outside game.
As for everything, I suspect the answer is “you need both, determining which is most helpful on the margin depends on the specific details of each case and your own personal fit”.
Probably just Oxford vs Cambridge founder effects/path dependency.
EDIT: By ‘Oxford founder effects’ I mean something like “many of the early xrisk researchers came up through Oxford & naturally tend to cite Oxford folks; and two of the best book-length recent intros are from Toby and Will at Oxford; so the introductory materials are skewed towards Oxford”.
Hi Rob, thanks for responding.
I agree that eventually, individuals may be able to train (or more importantly run exfiltrated models) advanced AI that is very dangerous. I expect that before that, it will be within the reach of richer, bigger groups. Today, it requires more compute/better techniques than we have available. At some point in the coming years/decades it will be within the reach of major states’ budgets, then smaller states and large companies, and then smaller and smaller groups until its within the reach of individuals. That’s the same process that many, many other technologies have followed. If that’s right, what does that suggest we need? Agreement between the major states, then non-proliferation agreements, then regulation and surveillance banning corporate/individuals.
On governments not being major players in cutting-edge AI research today. This is certainly true. I think cyber might be a relevant analogy here. Much of the development and deployment of cyberattacks has been by the private sector (companies and contractors in the US, often criminals for some autocracies). Nevertheless, the biggest cyberattacks (Stuxnet, NotPetya, etc) are directed by the governments of major states—i.e. the P5 of US, Russia, UK, France and China. Its possible that something similar happens for AI.
In terms of how long international agreements take, I think 50 years is a bit pessimistic. I would take arms control agreements as possible comparisons. Take 1972′s nuclear and biological weapons agreements. The ideas behind deterrence were largely developed around 1960 (Schelling 1985; Adler 1992), and then made into an international agreement in 1972. It might even have happened sooner, under LBJ, had the USSR not invaded Czechoslovakia on 20th August 1968, a day before SALT was supposed to start. On biological weapons, the UK proposed the BWC in August 1968, and it was signed in 1972 as well. New START took about 2 years. So in general, bilateral arms control style agreements with monitoring and verification can be agreed in less than 5 years.
To take the nuclear 1960s analogy, we could loosely think of ourselves as being in early 1962: we’ve come up with the concerns if not the specific agreements, and some decision-makers and politicians are on board. We haven’t yet had a major AI warning shot like the Cuban Missile Crisis (which began 60 years ago yesterday!), we haven’t yet had the confidence-building measures like the 1963 Hotline Agreement, and haven’t yet proposed or begun the equivalent of SALT. All that’s might be to come in the next few years/decades.
This won’t be an easy project by any means, but I don’t think we can yet say its completely infeasible—more research, and the attempt itself, is needed.
Just a quick note to say that I think planning on a pivotal act is risky and dangerous, and we just don’t yet know how feasible or infeasible “some healthy and competent worldwide collaboration steering the transition” is—more research is needed.
As I say in The Rival AI Deployment Problem: a Pre-deployment Agreement as the least-bad response,
“while it may seem unlikely at this stage, a predeployment agreement might be the least-bad option – and at least worthy of more study and reflection. In particular, more research should be done into possible clauses in a pre-deployment agreement, and into possibilities for AI development monitoring and verification.”
I think he quit his PhD actually. So you could ask him why, and what factors people should consider when choosing to do a PhD or deciding to change while on it.
<Before that he did a PhD in the Philosophy of Machine Learning at Cambridge, on the topic of “to what extent is the development of artificial intelligence analogous to the biological and cultural evolution of human intelligence?”>
I’m very pro framing this as an externality. Doesn’t just help with left-leaning people, it can also be helpful for talking to other audiences, such as those immersed in economics or antitrust/competition law.
For more on this risk, see this interesting recent book: Dark Skies: Space Expansionism, Planetary Geopolitics, and the Ends of Humanity (Jun. 2020) Daniel Deudney
https://academic-oup-com.ezp.lib.cam.ac.uk/book/33656?login=true
https://www.amazon.co.uk/Dark-Skies-Expansionism-Planetary-Geopolitics/dp/0190903341
This is really fascinating and useful work, thanks for putting it together (and everyone who contributed)!
Oof this comment was a shame to read—I downvoted it. Ad hominem attack and no discussion of the content of the paper.
Also, the paper has ten authors and got through Nature peer-review—seems a stretch to write it off as just two people’s ideology.
Just to respond to the nuclear winter point.
I actually think the EA world has been pretty good epistemically on winter: appropriately humble and exploratory, mostly funding research to work out how big a problem it is, not basing big claims on (possibly) unsettled science. The argument for serious action on reducing nuclear risk doesn’t rely on claims about nuclear winter—though nuclear winter would really underline its importance. The Rethink Priorities report you critique talks at length about the debate over winter, which is great. See also 80,000 Hours profile, which is similarly cautious/hedged.
The EA world has been the major recent funder of research on nuclear winter: OpenPhil in 2017, 2020, perhaps Longview, and soon FLI. The research has advanced considerably since 2016. Indeed, most of the research ever published on nuclear winter has been published in the last few years, using the latest climate modelling. The most recent papers are getting published in Nature. I would disagree that theres a “reliance on papers that have a number of obvious flaws”.
So as I see it the main phenomenon is that there’s just much more being posted on the forum. I think there’s two factors behind that 1) community growth and 2) strong encouragement to post on the Forum. Eg there’s lots of encouragement to post on the forum from: the undergraduate introductory/onboarding fellowships, the AGI/etc ‘Fundamentals’ courses, the SERI/CERI/etc Summer Fellowships, or this or this (h/t John below).
The main phenomenon is that there is a lot more posted on the forum, mostly from newer/more junior people. It could well be the case that the average quality of posts has gone down. However, I’m not so sure that the quality of the best posts has gone down, and I’m not so sure that there are fewer of the best posts every month. Nevertheless, spotting the signal from the noise has become harder.
But then the forum serves several purposes. To take two of them: One (which is the one commenters here are most focussed on) is “signal”—producing really high-quality content—and its certainly got harder to find that. But another purpose is more instrumental—its for more junior people to demonstrate their writing/reasoning ability to potential employees. Or its to act as an incentive/endgoal for them to do some research—where the benefit is more that they see whether its a fit for them or not, but they wouldn’t actually do the work if it wasn’t structured towards writing something public.
So the main thing that those of us who are looking for “signal” need to do is find better/new ways to do so. The curated posts are a postive step in this direction, as are the weekly summaries and the monthly summaries.
This is really great work! Very clearly structured and written, persuasively argued and (fairly) well supported by the evidence.
I’m currently doing my PhD/DPhil on the history of arms control agreements, and 1972 is one of my four case-studies. So obviously I think its really important and interesting, and that more people should know about it – and I have a lot of views on the subject! So I’ve got a few thoughts on methodology, further literature and possible extensions which I’ll share below. But they’re all to adding to what is excellent work.
Methodology
Its a bit unclear to me what your claim is for the link between these Track II discussions and the ultimate outcome of the two 1972 agreements. Its not that they were sufficient (needed SALT negotiations, and even then needed Kissinger/Dobrynin backchannel). Is it that the discussions were necessary for the outcome? Or just that they contributed in a positive way? I would be interested in your view.
The limitations section is good. But I think you could have been even clearer on the limits and strengths of a ‘single N’ approach. The limits are how much this can be generalised to the entire ‘universe of cases’. However, single N also has strengths—its most useful for developing and exploring mechanisms. So I think you could frame your contribution as exploring and deepening an analysis of the mechanisms. For example, something like “Two main mechanisms are proposed in the literature, this case study provides strong evidence for mechanism 1 (conveying new conceptions/ideas) and demonstrates how it works”.
On another point, I’d be concerned that if you chose this case because it was one of the most successful Track II cases you’d be ‘selecting on the dependent variable’ (apologies for the political science jargon – something like “cherrypicked for having a particular outcome”) . Can you justify your motivation and case-selection differently, for example as one of (the?) biggest and most sustained Track 2 dialogues? e.g. you say: “when the first Pugwash conference happened in 1957, there were either no, or almost no, other opportunities for Soviet and American scientists to have conversations about security policy and nuclear issues”
Further literature
Schelling. What Went Wrong with Arms Control?.
Adler + Schelling are great on the US side of the story. I assume you would be familiar with them, but I don’t see them cited. If you haven’t read them, you’re in for a treat – they’re great, and largely agree with you.
If you want to go down a tangent, you might want to engage with new line of argument that many US nuclear policymakers never accepted the parity of MAD, but continued seeking advantage (Green and Long 2017; Green 2020; Lieber and Press 2006, 2020; Long and Green 2015).
As a sidenote, I’m curious why so much of the research on the two nuclear 1972 agreements focusses on ABM. ABM is the more intellectually interesting and counterintuitive. But its not clear to me it was *more important* then the limits on offensive weapons though.
Next steps/possible extensions
My impression is your main audiences are funders (and to a lesser extent general researchers and activists) within GCR. However if you wanted to adapt it, this very plausibly could be a paper. Its already a paper length, ~8,000 words. If you wanted to go down that route, there’s a few things I’d do:
I’d cut most of the personal best guesses (“it seems likely to me” etc).
I think the notes are really great and interesting! If you incorporated some of them in the text of the piece you could deepen some of your claims in section 4, slim down the other sections.
Have a paragraph or two placing this piece within wider IR theoretical debates on constructivism, epistemic communities, going against systemic theories to open the ‘black box’ of the unitary state, etc.
If you wanted to continue this research, you could contrast this case with a similar conference and see what the difference in outcomes was; or try and draw up a list of the whole universe of cases (all major Track II dialogues).
Hmm I strongly read it as focussed on magnitude 7. Eg In the paper they focus on magnitude 7 eruptions, and the 1⁄6 this century probability: “The last magnitude-7 event was in Tambora, Indonesia, in 1815.” / “Given the estimated recurrence rate for a magnitude-7 event, this equates to more than US$1 billion per year.” This would be corroborated by their thread, Forum post, and previous work, which emphasise 7 & 1⁄6.
Sorry to be annoying/pedantic about this. I’m being pernickety as I view a key thrust of their research as distinguishing 7 from 8. We can’t just group magnitude 7 (1/6 chance) along with magnitude 8 and write them off as a teeny 1⁄14,000 chance. We need to distinguish 7 from 8, consider their severity/probability seperately, and prioritise them differently.
Hi Pablo and Matthew, just a quick one:
“Michael Cassidy and Lara Mani warn about the risk from huge volcanic eruptions. Humanity devotes significant resources to managing risk from asteroids, and yet very little into risk from supervolcanic eruptions, despite these being substantially more likely. The absolute numbers are nonetheless low; super-eruptions are expected roughly once every 14,000 years. Interventions proposed by the authors include better monitoring of eruptions, investments in preparedness, and research into geoengineering to mitigate the climatic impacts of large eruptions or (most speculatively) into ways of intervening on volcanoes directly to prevent eruptions.”
However, their Nature paper is about magnitude 7 eruptions, which may have a probability this century of 1⁄6, not supervolcanic eruptions (magnitude 8), which as you point out have a much lower probability.
I think its a fascinating paper that in a prominent, rigorous and novel way applies importance/neglectedness/tractability to a comparison of two hazards:
“Over the next century, large-scale volcanic eruptions are hundreds of times more likely to occur than are asteroid and comet impacts, put together. The climatic impact of these events is comparable, yet the response is vastly different. ‘Planetary defence’ receives hundreds of millions of dollars in funding each year, and has several global agencies devoted to it. [...] By contrast, there is no coordinated action, nor large-scale investment, to mitigate the global effects of large-magnitude eruptions. This needs to change.”
Thanks for pulling it together over so many years, and reading through a truly mindboggling array of papers & reports!
Yep totally fair point, my examples were about pieces. However, note that the quote you pulled out referred to ‘good work in the segments’ (though this is quite a squirmy lawyerly point for me to make). Also, interestingly 2019-era Will was a bit more skeptical of xrisk—or at least wrote a piece exploring that view.
I’m a bit wary of naming specific people whose views I know personally but haven’t expressed them publicly, so I’ll just give some orgs who mostly work in those two segments, if you don’t mind:
‘Long-term + EA’: the APPG for Future Generations does a lot of work here, and I’d add Tyler John’s work (here & here), plausibly Beckstead’s thesis.
‘Xrisk + EA’: my impression is that some of the more normy groups OpenPhil have funded are here, working with the EA community on xrisk topics, but not necessarily buying longtermism.
I agree that Effective Altruism and the existential risk prevention movement are not the same thing. Let me use this as an opportunity to trot out my Venn diagrams again. The point is that these communities and ideas overlap but don’t necessarily imply each other—you don’t have to agree to all of them because you agree with one of them, and there are good people doing good work in all the segments.
This is interesting: https://www.reuters.com/technology/exclusive-behind-ftxs-fall-battling-billionaires-failed-bid-save-crypto-2022-11-10/
In particular, here’s another hypothesis for why Binance withdrew:
How did Binance have such leverage over FTX?