Haydn has been a Research Associate and Academic Project Manager at the University of Cambridge’s Centre for the Study of Existential Risk since Jan 2017.
HaydnBelfield
Karma: 3,751
Absolutely well-deserved, well done Eirik and the rest of the Norwegian community.
Really helpful contribution: focuses on the key issues, balanced, evidenced and has concrete next steps. Thanks.
This is interesting: https://www.reuters.com/technology/exclusive-behind-ftxs-fall-battling-billionaires-failed-bid-save-crypto-2022-11-10/
In particular, here’s another hypothesis for why Binance withdrew:
By ditching the deal, Binance had also avoided the regulatory scrutiny that would likely have accompanied the takeover, which Zhao had flagged as a likelihood in a memo to employees that he posted on Twitter.
Financial regulators around the world have issued warnings about Binance for operating without a license or violating money laundering laws. The U.S. Justice Department is investigating Binance for possible money laundering and criminal sanctions violations. Reuters reported last month that Binance had helped Iranian firms trade $8 billion since 2018 despite U.S. sanctions, part of a series of articles this year by the news agency on the exchange’s financial crime compliance.
How did Binance have such leverage over FTX?
When FTX in May 2021 applied for a license in Gibraltar for a subsidiary, it had to submit information about its major shareholders, but Binance stonewalled FTX’s requests for help, according to messages and emails between the exchanges seen by Reuters.
Between May and July, FTX lawyers and advisors wrote to Binance at least 20 times for details on Zhao’s sources of wealth, banking relationships, and ownership of Binance, the messages show.
In June 2021, however, an FTX lawyer told Binance’s chief financial officer that Binance wasn’t “engaging with us properly” and they risked “severely disrupting an important project for us.” A Binance legal officer responded to FTX to say she was trying to get a response from Zhao’s personal assistant, but the requested information was “too general” and they may not provide everything.
By July of that year, Bankman-Fried had tired of waiting. He bought back Zhao’s stake in FTX for about $2 billion, the person with direct knowledge of the deal said. Two months later, with Binance no longer involved, Gibraltar’s regulator granted FTX a license.
That sum was paid to Binance, in part, in FTX’s own coin, FTT, Zhao said last Sunday—a holding he would later order Binance to sell, precipitating the crisis at FTX.
Thanks for sharing this! One thing I thought you might be interested in is the Pope and Pontifical Academy of Sciences involvement with work on climate change, and biodiversity.
In a recent post (Lord Martin Rees: an appreciation) I noted:
“One particularly notable example on climate is his work at the Vatican. In May 2014, he helped Sir Partha Dasgupta co-organise a major workshop with the Pontifical Academy of Sciences on climate change. After the workshop, Sir Partha spoke to the Pope directly and encouraged him to include climate change in his speeches and to urge people to be better stewards of the planet. The workshop underpinned a major report published in April 2015 by the Vatican. The report in turn partly informed the May 2015 Laudato si’ Papal Encyclical, which focussed on the impending threat of climate change and was influential in encouraging the 1.3 billion Catholics worldwide to support for the Paris Agreement, agreed in December 2015.”
I agree that if I could only recommend one book, it would probably be the Precipice - as its more up-to-date and comprehensive. I was thinking a wider bibliography / reading list. However, I really would prioritise the two Ted talks as short, interesting, credible intros.
I’ve got a real soft spot for “Our Final Century: Will Civilisation Survive the Twenty-first Century?” as it was the book that originally got me interested in existential risk. I still think its really important for the field, and is usefully included alongside 2008′s GCR and Bostrom’s 2002 paper. We’re actually working on an “updated after 20 years” version of the book, hopefully out next year.
Yes he’s said this very consistently for years. Its interesting for like the insider’s insider (Astronomer Royal!) to advocate an outside game.
As for everything, I suspect the answer is “you need both, determining which is most helpful on the margin depends on the specific details of each case and your own personal fit”.
Probably just Oxford vs Cambridge founder effects/path dependency.
EDIT: By ‘Oxford founder effects’ I mean something like “many of the early xrisk researchers came up through Oxford & naturally tend to cite Oxford folks; and two of the best book-length recent intros are from Toby and Will at Oxford; so the introductory materials are skewed towards Oxford”.
Lord Martin Rees: an appreciation
Hi Rob, thanks for responding.
I agree that eventually, individuals may be able to train (or more importantly run exfiltrated models) advanced AI that is very dangerous. I expect that before that, it will be within the reach of richer, bigger groups. Today, it requires more compute/better techniques than we have available. At some point in the coming years/decades it will be within the reach of major states’ budgets, then smaller states and large companies, and then smaller and smaller groups until its within the reach of individuals. That’s the same process that many, many other technologies have followed. If that’s right, what does that suggest we need? Agreement between the major states, then non-proliferation agreements, then regulation and surveillance banning corporate/individuals.
On governments not being major players in cutting-edge AI research today. This is certainly true. I think cyber might be a relevant analogy here. Much of the development and deployment of cyberattacks has been by the private sector (companies and contractors in the US, often criminals for some autocracies). Nevertheless, the biggest cyberattacks (Stuxnet, NotPetya, etc) are directed by the governments of major states—i.e. the P5 of US, Russia, UK, France and China. Its possible that something similar happens for AI.
In terms of how long international agreements take, I think 50 years is a bit pessimistic. I would take arms control agreements as possible comparisons. Take 1972′s nuclear and biological weapons agreements. The ideas behind deterrence were largely developed around 1960 (Schelling 1985; Adler 1992), and then made into an international agreement in 1972. It might even have happened sooner, under LBJ, had the USSR not invaded Czechoslovakia on 20th August 1968, a day before SALT was supposed to start. On biological weapons, the UK proposed the BWC in August 1968, and it was signed in 1972 as well. New START took about 2 years. So in general, bilateral arms control style agreements with monitoring and verification can be agreed in less than 5 years.
To take the nuclear 1960s analogy, we could loosely think of ourselves as being in early 1962: we’ve come up with the concerns if not the specific agreements, and some decision-makers and politicians are on board. We haven’t yet had a major AI warning shot like the Cuban Missile Crisis (which began 60 years ago yesterday!), we haven’t yet had the confidence-building measures like the 1963 Hotline Agreement, and haven’t yet proposed or begun the equivalent of SALT. All that’s might be to come in the next few years/decades.
This won’t be an easy project by any means, but I don’t think we can yet say its completely infeasible—more research, and the attempt itself, is needed.
Just a quick note to say that I think planning on a pivotal act is risky and dangerous, and we just don’t yet know how feasible or infeasible “some healthy and competent worldwide collaboration steering the transition” is—more research is needed.
As I say in The Rival AI Deployment Problem: a Pre-deployment Agreement as the least-bad response,
“while it may seem unlikely at this stage, a predeployment agreement might be the least-bad option – and at least worthy of more study and reflection. In particular, more research should be done into possible clauses in a pre-deployment agreement, and into possibilities for AI development monitoring and verification.”
I think he quit his PhD actually. So you could ask him why, and what factors people should consider when choosing to do a PhD or deciding to change while on it.
<Before that he did a PhD in the Philosophy of Machine Learning at Cambridge, on the topic of “to what extent is the development of artificial intelligence analogous to the biological and cultural evolution of human intelligence?”>
I’m very pro framing this as an externality. Doesn’t just help with left-leaning people, it can also be helpful for talking to other audiences, such as those immersed in economics or antitrust/competition law.
For more on this risk, see this interesting recent book: Dark Skies: Space Expansionism, Planetary Geopolitics, and the Ends of Humanity (Jun. 2020) Daniel Deudney
https://academic-oup-com.ezp.lib.cam.ac.uk/book/33656?login=true
https://www.amazon.co.uk/Dark-Skies-Expansionism-Planetary-Geopolitics/dp/0190903341
This is really fascinating and useful work, thanks for putting it together (and everyone who contributed)!
The Rival AI Deployment Problem: a Pre-deployment Agreement as the least-bad response
Oof this comment was a shame to read—I downvoted it. Ad hominem attack and no discussion of the content of the paper.
Also, the paper has ten authors and got through Nature peer-review—seems a stretch to write it off as just two people’s ideology.
Just to respond to the nuclear winter point.
I actually think the EA world has been pretty good epistemically on winter: appropriately humble and exploratory, mostly funding research to work out how big a problem it is, not basing big claims on (possibly) unsettled science. The argument for serious action on reducing nuclear risk doesn’t rely on claims about nuclear winter—though nuclear winter would really underline its importance. The Rethink Priorities report you critique talks at length about the debate over winter, which is great. See also 80,000 Hours profile, which is similarly cautious/hedged.
The EA world has been the major recent funder of research on nuclear winter: OpenPhil in 2017, 2020, perhaps Longview, and soon FLI. The research has advanced considerably since 2016. Indeed, most of the research ever published on nuclear winter has been published in the last few years, using the latest climate modelling. The most recent papers are getting published in Nature. I would disagree that theres a “reliance on papers that have a number of obvious flaws”.
So as I see it the main phenomenon is that there’s just much more being posted on the forum. I think there’s two factors behind that 1) community growth and 2) strong encouragement to post on the Forum. Eg there’s lots of encouragement to post on the forum from: the undergraduate introductory/onboarding fellowships, the AGI/etc ‘Fundamentals’ courses, the SERI/CERI/etc Summer Fellowships, or this or this (h/t John below).
The main phenomenon is that there is a lot more posted on the forum, mostly from newer/more junior people. It could well be the case that the average quality of posts has gone down. However, I’m not so sure that the quality of the best posts has gone down, and I’m not so sure that there are fewer of the best posts every month. Nevertheless, spotting the signal from the noise has become harder.
But then the forum serves several purposes. To take two of them: One (which is the one commenters here are most focussed on) is “signal”—producing really high-quality content—and its certainly got harder to find that. But another purpose is more instrumental—its for more junior people to demonstrate their writing/reasoning ability to potential employees. Or its to act as an incentive/endgoal for them to do some research—where the benefit is more that they see whether its a fit for them or not, but they wouldn’t actually do the work if it wasn’t structured towards writing something public.
So the main thing that those of us who are looking for “signal” need to do is find better/new ways to do so. The curated posts are a postive step in this direction, as are the weekly summaries and the monthly summaries.
This is really great work! Very clearly structured and written, persuasively argued and (fairly) well supported by the evidence.
I’m currently doing my PhD/DPhil on the history of arms control agreements, and 1972 is one of my four case-studies. So obviously I think its really important and interesting, and that more people should know about it – and I have a lot of views on the subject! So I’ve got a few thoughts on methodology, further literature and possible extensions which I’ll share below. But they’re all to adding to what is excellent work.
Methodology
Its a bit unclear to me what your claim is for the link between these Track II discussions and the ultimate outcome of the two 1972 agreements. Its not that they were sufficient (needed SALT negotiations, and even then needed Kissinger/Dobrynin backchannel). Is it that the discussions were necessary for the outcome? Or just that they contributed in a positive way? I would be interested in your view.
The limitations section is good. But I think you could have been even clearer on the limits and strengths of a ‘single N’ approach. The limits are how much this can be generalised to the entire ‘universe of cases’. However, single N also has strengths—its most useful for developing and exploring mechanisms. So I think you could frame your contribution as exploring and deepening an analysis of the mechanisms. For example, something like “Two main mechanisms are proposed in the literature, this case study provides strong evidence for mechanism 1 (conveying new conceptions/ideas) and demonstrates how it works”.
On another point, I’d be concerned that if you chose this case because it was one of the most successful Track II cases you’d be ‘selecting on the dependent variable’ (apologies for the political science jargon – something like “cherrypicked for having a particular outcome”) . Can you justify your motivation and case-selection differently, for example as one of (the?) biggest and most sustained Track 2 dialogues? e.g. you say: “when the first Pugwash conference happened in 1957, there were either no, or almost no, other opportunities for Soviet and American scientists to have conversations about security policy and nuclear issues”
Further literature
Schelling. What Went Wrong with Arms Control?.
Adler + Schelling are great on the US side of the story. I assume you would be familiar with them, but I don’t see them cited. If you haven’t read them, you’re in for a treat – they’re great, and largely agree with you.
If you want to go down a tangent, you might want to engage with new line of argument that many US nuclear policymakers never accepted the parity of MAD, but continued seeking advantage (Green and Long 2017; Green 2020; Lieber and Press 2006, 2020; Long and Green 2015).
As a sidenote, I’m curious why so much of the research on the two nuclear 1972 agreements focusses on ABM. ABM is the more intellectually interesting and counterintuitive. But its not clear to me it was *more important* then the limits on offensive weapons though.
Next steps/possible extensions
My impression is your main audiences are funders (and to a lesser extent general researchers and activists) within GCR. However if you wanted to adapt it, this very plausibly could be a paper. Its already a paper length, ~8,000 words. If you wanted to go down that route, there’s a few things I’d do:
I’d cut most of the personal best guesses (“it seems likely to me” etc).
I think the notes are really great and interesting! If you incorporated some of them in the text of the piece you could deepen some of your claims in section 4, slim down the other sections.
Have a paragraph or two placing this piece within wider IR theoretical debates on constructivism, epistemic communities, going against systemic theories to open the ‘black box’ of the unitary state, etc.
If you wanted to continue this research, you could contrast this case with a similar conference and see what the difference in outcomes was; or try and draw up a list of the whole universe of cases (all major Track II dialogues).
I really appreciate the explorative, curious, open and constructive approach Toby!
On ‘what are some important results that a complex model produces’, one nice example is a focus on vulnerability. That is, focus on improving general resilience, as well as preventing and mitigating particular hazards. This has apparently become best practice in many companies—e.g. rather than just listing hazards, focus also on having adequate capital reserves and some slack/redundancy in one’s supply chains.
Matt Boyd and Nick Wilson have done some great complex-model-ish work looking at the resilience of island nations to a range of scenarios. One thing that turned up is that Aotearoa New Zealand has lots of food production, but transport of that food is reliant on road transport, and the country closed its only oil refinery. Having an oil refinery might increase its resilience/decrease its vulnerability.
I don’t think that point would have necessarily come up in a ‘simple-model’ approach, but its concrete, tractable, important and plausibly a good thing to suggest the govt act on.
Of course, you touch on vulnerabilities in the Precipice. Nevertheless, its fun to wonder what a sequel would look like with each chapter framed around a critical system/vulnerability (food, health, communications) rather than each around a particular hazard.