Haydn has been a Research Associate and Academic Project Manager at the University of Cambridge’s Centre for the Study of Existential Risk since Jan 2017.
HaydnBelfield
Karma: 3,749
If this one goes well, would be good to organise one in Europe as well.
I wanted to post to say I agree, EAGxLatinAmerica was wonderful—so exciting to meet such interesting people doing great work!
Also...
“I met Rob Wiblin at lunch and I didn’t recognize him.”
Ha rekt
I appreciate this quick and clear statement from CEA.
We came up with our rankings seperately, but when we compared it turned out we agreed on the top 4 + honourable mention. We then worked on the texts together.
I think the crucial thing is funding levels.
It was only by October 1941 (after substantial nudging from the British) that Roosevelt approved serious funding. As a reminder, I’m particularly interested in ‘sprint’ projects with substantial funding: for example those in which the peak year funding reached 0.4% of GDP (Stine, 2009, see also Grace, 2015).
So to some extent they were in a race 1939-1942, but I would suggest it wasn’t particularly intense, it wasn’t a sprint race.
These tragic tradeoffs also worry me deeply. Existential wagering is to me one of the more worrying, but also possible to avoid. However, the tradeoff between existential co-option and blackmail seems particularly hard to avoid for AI.
I think my point is more like “if anyone gets anywhere near advanced AI, governments will have something to say about it—they will be a central player in shaping its development and deployment.” It seems very unlikely to me that governments would not notice or do anything about such a potentially transformative technology. It seems very unlikely to me that a company could train and deploy an advanced AI system of the kind you’re thinking about without governments regulating and directing it. On funding specifically, I would probably be >50% on governments getting involved in meaningful private-public collaboration if we get closer to substantial leaps in capabilities (though it seems unlikely to me that AI progress will get to that point by 2030).
On your regulation question, I’d note that the EU AI Act, likely to pass next year, already proposes the following requirements applying to companies providing (eg selling, licensing or selling access to) ‘general purpose AI systems’ (eg large foundation models):
Risk Management System
Data and data governance
Technical documentation
Record-keeping
Transparency and provision of information to users
Human oversight
Accuracy, robustness and cybersecurity
So they’ll already have to do (post-training) safety testing before deployment. Regulating the training of these models is different and harder, but even that seems plausible to me at some point, if the training runs become ever huger and potentially more consequential. Consider the analogy that we regulate biological experiments.
Strongly agree, upvoted.
Just a minor point on the Putin quote, as it comes up so often, he was talking to a bunch of schoolkids, encouraging them to do science and technology. He said similarly supportive things about a bunch of other technologies. I’m at >90% he wasn’t referring to AGI. He’s not even that committed to AI leadership: he’s taken few actions indicating serious interest in ‘leading in AI’. Indeed, his Ukraine invasion has cut off most of his chip supplies and led to a huge exodus of AI/CS talent. It was just an off-the-cuff rhetorical remark.
This is a really useful and interesting post that I’m glad you’ve written! I agree with a lot of it, but I’ll mention one bit I’m less sure about.
I think we can have more nuance about governments “being in the race” or their “policy having strong effects”. I agree that pre-2030, a large, centralised, government-run development programme like the Apollo Project is less likely (I assume this is the central thing you have in your mind). However there are other ways governments could be involved, including funding, regulating and ‘directing’ development and deployment.
I think cyber weapons and cyber defence is a useful comparison. Much of the development—and even deployment—is led by the private sector: defence contractors in the US, criminals in some other states. Nevertheless, much of it is funded, regulated and directed by states. People didn’t think this would happen in the late 1990s and 2000s—they thought it would be private sector led. But nevertheless with cyber, we’re now in a situation where the major states (e.g. those in the P5, with big economies, militaries and nuclear weapons) have the preponderance of cyber power—they have directed and are responsible for all the largest cyber attacks (Stuxnet, 2016 espionage, NotPetya, WannaCry etc). It’s a public-private partnership, but states are in the driving seat.
Something similar might happen with AI this side of 2030, without the situation resembling the Apollo Project.
For much more on this, Jade Leung’s thesis is great: Who will govern artificial intelligence? Learning from the history of strategic politics in emerging technologies
Sure—happy to. Deleted.
I agree
Remarkable interview. One key section (people should read the whole thing!):
When you talk about your mistakes, you talk about your intent. Your mother once described you as a “take-no-prisoners utilitarian.” Shouldn’t your intent be irrelevant?
Except to the extent that it’s predictive of the future. But yeah, at the end of the day, I do think that, what happened happened, and whatever I was thinking or not thinking or trying to do or not trying to do, it happened. And that sucks. That’s really bad. A lot of people got hurt. And I think that, thinking about why it happened, there are some perspectives from which it matters, including trying to figure out what to do going forward. But that doesn’t change the fact that it happened. And as you said, I’m not expecting people to say, Oh, that’s all good then. Sam didn’t intend for me to lose money. I don’t miss that money anymore. That’s not how it works.
One of your close personal mentors, the effective altruism philosopher Will MacAskill, has disavowed you. Have you talked with him since?
I haven’t talked with him. [Five second pause.] I don’t blame him. [20-second pause and false starts.] I feel incredibly bad about the impact of this on E.A. and on him, and more generally on all of the things I had been wanting to support. At the end of the day, this isn’t any of their faults.
This fucked up a lot of their plans, and a lot of plans that people had to do a lot of good for the world. And that’s terrible. And to your point, from a consequentialist perspective, what happened was really bad. And independent of intent or of anything like that, it’s still really bad.
Have you talked with your brother Gabe, who ran your Guarding Against Pandemics group? Are you worried, frankly, that you might have ruined his career too?
It doesn’t feel good either. Like, none of these things feel good.
Have you apologized to him?
Yeah, I spent a lot of last month apologizing, but I don’t know how much the apologies mean to people at the end of the day. Because what happened happened, and it’s cold comfort in a lot of ways.
I don’t want to put words in his mouth. I feel terrible about what happened to all the things he’s trying to do. He’s family, and he’s been supportive even when he didn’t have to be. But I don’t know what’s going through his head from his perspective, and I don’t want to put words in it.
Do you think someone like you deserves to go to jail? On a moral level, doesn’t someone who has inflicted so much pain—intent be damned—deserve it? There are a lot of people incarcerated in this country for far less.
What happens happens. That’s not up to me.
I can tell you what I think personally, viscerally, and morally feels right to me. Which is that I feel like I have a duty to sort of spend the rest of my life doing what I can to try and make things right as I can.
You shocked a lot of people when you referred in a recent interview to the “dumb game that we woke Westerners play.” My understanding is that you were talking about corporate social responsibility and E.S.G., not about effective altruism, right?
That’s right.
To what extent do you feel your image and donations gave you cover? I know you say you didn’t do anything wrong intentionally. But I wonder how much you were in on the joke.
Gave me cover to do what, though? I think what I was in on, so to speak, was that a lot of the C.S.R. stuff was bullshit. Half of that was always just branding, and I think that’s true for most companies. And to some extent everyone knew, but it was a game everyone played together. And it’s a dumb game.
Yeah the breakeven point is a super rough figure, originally wasn’t going to include it. Paying for staff and food would push out, more events or hiring out for events would bring closer, etc. Main thing I wanted to add to the conversation is a sense of how expensive workshops and retreats are.
Wilton Park and West Court are both historic, so thought they’d be good comparisons (and ones I know).
Just for context on event costs
Wilton Park
They do lots of workshops on international security. Their events cost around £54,000 for two nights.
(see page 14 of their Annual Report: “In 2020⁄21, we delivered 128 (76 in 2019⁄20) events at average net revenue of £13k (£54k in 2019⁄20). The lower average net revenue this year was due to the reduced income generated from virtual events compared to that generated by face to face events in 2019⁄20. Virtual events are shorter, generally lasting half a day, compared to face to face events which are generally for two nights.”)
West Court, Jesus College, Cambridge
I’ve been to several academic workshops and conferences here. Their prices are, for a 24 hour (overnight) rate:
West Court single ensuite from £205 Let’s say 100 attendees overnight for 3 days (a weekend workshop) in the cheapest rooms: £200*100*3 = £60,000.
Shakeel offers the further examples of “traditional specialist conference centres, e.g. Oberwolfach, The Rockefeller Foundation Bellagio Center or the Brocher Foundation.”
50 events (one a week) like these a year would cost £3m (£60,000*50=£3,000,000). So break even (assuming £15m was the actual cost) in 5 years—quicker if they paid less, which seems likely.
No idea if this is a good use of money, just sharing some information for context.
I think a lot of the disagreements in the comments is coming down to different conceptions of headhunting. Dan, you refer to targeting/specific/direct outreach to particular individuals, but that doesn’t seem the crucial difference, its in the intent, tone and incentives.
“Hey X person, you’re doing a great job at your current job. You might be totally happy at your current job, but I thought I’d flag this cool new opportunity that seems really impactful—happy to discuss why it might be a good fit” seems fine.
Giving a hard sell, strongly denigrating the current employer or being strongly incentivised for switches (eg paid a commission) seems way less fine.
Thanks for this speech Gideon, an important point and one that I obviously agree with a lot. I thought I’d just throw in a point about policy advocacy. One benefit of the simple/siloed/hazard-centric approach is that that really is how government departments, academic fields and NGO networks are often structured. There’s a nuclear weapons field of academics, advocates and military officials that barely interacts with even the the biological weapons field.
Of course, one thing that ‘complex-model’ thinking can hopefully do is identify new approaches to reduce risk and/or new affected parties and potential coalition partners—such as bringing in DEFRA and thinking about food networks.
As a field, we need to be able to zoom in and out, focus on different levels of analysis to spot possible solutions.
I really appreciate the explorative, curious, open and constructive approach Toby!
On ‘what are some important results that a complex model produces’, one nice example is a focus on vulnerability. That is, focus on improving general resilience, as well as preventing and mitigating particular hazards. This has apparently become best practice in many companies—e.g. rather than just listing hazards, focus also on having adequate capital reserves and some slack/redundancy in one’s supply chains.
Matt Boyd and Nick Wilson have done some great complex-model-ish work looking at the resilience of island nations to a range of scenarios. One thing that turned up is that Aotearoa New Zealand has lots of food production, but transport of that food is reliant on road transport, and the country closed its only oil refinery. Having an oil refinery might increase its resilience/decrease its vulnerability.
I don’t think that point would have necessarily come up in a ‘simple-model’ approach, but its concrete, tractable, important and plausibly a good thing to suggest the govt act on.
Of course, you touch on vulnerabilities in the Precipice. Nevertheless, its fun to wonder what a sequel would look like with each chapter framed around a critical system/vulnerability (food, health, communications) rather than each around a particular hazard.
Absolutely well-deserved, well done Eirik and the rest of the Norwegian community.
Really helpful contribution: focuses on the key issues, balanced, evidenced and has concrete next steps. Thanks.
One reading of this is that Open Phil’s new ‘bar for funding’ is that
So (very roughly!) a new project would need to be ranked as better than the average longtermist grant over the past 18 months in order to be funded.
Is that a wild misrepresentation?