Haydn has been a Research Associate and Academic Project Manager at the University of Cambridge’s Centre for the Study of Existential Risk since Jan 2017.
HaydnBelfield
Karma: 3,738
Helpful post on the upcoming red-teaming event, thanks for putting it together!
Minor quibble—“That does mean I’m left with the slightly odd conclusion that all that’s happened is the Whitehouse has endorsed a community red-teaming event at a conference.”
I mean they did also announce $140m, that’s pretty good! That’s to say, the two other announcements seem pretty promising.
The funding through the NSF to launch 7 new National AI Research Institutes is promising, especially the goal for these to provide public goods such as reseach into climate, agriculture, energy, public health, education, and cybersecurity. $140m is, for example, more than the UK’s Foundation Models Taskforce £100m ($126m).
The final announcement was that in summer 2023 the OMB will be releasing draft policy guidance for the use of AI systems by the US government. This sounds excruciatinly boring, but will be important, as the federal government is such a bigger buyer/procurer and setter of standards. In the past, this guidance has been “you have to follow NIST standards”, which gives those standards a big carrot. The EU AI Act is more stick, but much of the high-risk AI it focusses on is use by governments (education, health, recruitment, police, welfare etc) and they’re developing standards too. So far amount of commonality across the two. To make another invidious UK comparison, the AI white paper says that a year from now, they’ll put out a report considering the need for statutory interventions. So we’ve got neither stick, carrot or standards...
“Today’s announcements include:
New investments to power responsible American AI research and development (R&D). The National Science Foundation is announcing $140 million in funding to launch seven new National AI Research Institutes. This investment will bring the total number of Institutes to 25 across the country, and extend the network of organizations involved into nearly every state. These Institutes catalyze collaborative efforts across institutions of higher education, federal agencies, industry, and others to pursue transformative AI advances that are ethical, trustworthy, responsible, and serve the public good. In addition to promoting responsible innovation, these Institutes bolster America’s AI R&D infrastructure and support the development of a diverse AI workforce. The new Institutes announced today will advance AI R&D to drive breakthroughs in critical areas, including climate, agriculture, energy, public health, education, and cybersecurity.
Public assessments of existing generative AI systems. The Administration is announcing an independent commitment from leading AI developers, including Anthropic, Google, Hugging Face, Microsoft, NVIDIA, OpenAI, and Stability AI, to participate in a public evaluation of AI systems, consistent with responsible disclosure principles—on an evaluation platform developed by Scale AI—at the AI Village at DEFCON 31. This will allow these models to be evaluated thoroughly by thousands of community partners and AI experts to explore how the models align with the principles and practices outlined in the Biden-Harris Administration’s Blueprint for an AI Bill of Rights and AI Risk Management Framework. This independent exercise will provide critical information to researchers and the public about the impacts of these models, and will enable AI companies and developers take steps to fix issues found in those models. Testing of AI models independent of government or the companies that have developed them is an important component in their effective evaluation.
Policies to ensure the U.S. government is leading by example on mitigating AI risks and harnessing AI opportunities. The Office of Management and Budget (OMB) is announcing that it will be releasing draft policy guidance on the use of AI systems by the U.S. government for public comment. This guidance will establish specific policies for federal departments and agencies to follow in order to ensure their development, procurement, and use of AI systems centers on safeguarding the American people’s rights and safety. It will also empower agencies to responsibly leverage AI to advance their missions and strengthen their ability to equitably serve Americans—and serve as a model for state and local governments, businesses and others to follow in their own procurement and use of AI. OMB will release this draft guidance for public comment this summer, so that it will benefit from input from advocates, civil society, industry, and other stakeholders before it is finalized.”
Yes this was my thought as well. I’d love a book from you Jeff but would really (!!) love one from both of you (+ mini-chapters from the kids?).
I don’t know the details of your current work, but it seems worth writing one chapter as a trial run, and if you think its going well (and maybe has good feedback) considering taking 6 months or so off.
Am I right that in this year and a half, you spent ~$2 million (£1.73m)? Seems reasonable not to continue this if you don’t think its impactful
per month per year Total Rent $70,000 $840,000 $1,260,000 Food & drink $35,000 $420,000 $630,000 Contractor $5,000 $60,000 $90,000 Staff time $6,250 $75,000 $112,500 Total $185,000 $2,220,000 $2,092,500
He listed GovAI on this (very good!) post too: https://www.lesswrong.com/posts/5hApNw5f7uG8RXxGS/the-open-agency-model
Yeah dunno exactly what the nature of his relationship/link
He’s at GovAI.
The point is that he reused the term, and didn’t redact it by e.g. saying “n------!!!!” or “the n-word”.
3 notes on the discussion in the comments.
1. OP is clearly talking about the last 4 or so years, not FHI in eg 2010 to 2014. So quality of FHI or Bostrom as a manager in that period is not super relevant to the discussion. The skills needed to run a small, new, scrappy, blue-sky-thinking, obscure group are different from a large, prominent, policy-influencing organisation in the media spotlight.
2. The OP is not relitigating the debate over the Apology (which I, like Miles, have discussed elsewhere) but instead is pointing out the practical difficulties of Bostrom staying. Commenters may have different views from the University, some FHI staff, FHI funders and FHI collaborators—that doesn’t mean FHI wouldn’t struggle to engage these key stakeholders.
3. In the last few weeks the heads of Open Phil and CEA have stepped aside. Before that, the leadership of CSER and 80,000 Hours has changed. There are lots of other examples in EA and beyond. Leadership change is normal and good. While there aren’t a huge number of senior staff left at FHI, presumably either Ord or Sandberg could step up (and do fine given administrative help and willingness to delegate) - or someone from outside like Greaves plausibly could be Director.
This is very exciting work! Really looking forward to the first research output, and what the team goes on to do. I hope this team gets funding—if I were a serious funder I would support this.
For some context on the initial Food Security project, readers might want to take a glance at how South America does in this climate modelling from Xia et al, 2022.
Fig. 4: Food intake (kcal per capita per day) in Year 2 after different nuclear war soot injections.
You’re leaving as you’ve led: being a great positive role model (in this case on mental health), being humble, and focussing on what’s best for the community and the wider impact it seeks to achieve. Congratulations and well done Max!
Good summary!
I’ve got a book chapter on this topic coming out on March 30 in How Worlds Collapse: What History, Systems, and Complexity Can Teach Us About Our Modern World and Fragile Future. Chapter 4. Collapse, Recovery, and Existential Risk—Haydn Belfield
The basic takeaway is that collapse->extinction and unrecoverable collapse are both unlikely, but can’t be ruled out. The more important question is “what kind of ethical and political systems might be dominant in a recovered world?”—those systems might be much worse for our future progress than our current world’s dominant ethical and political systems.
Anyway, I’ll share more when the book is out!
Yeah that’s fair. Depends on the particular researcher, they’re quite eclectic. Some are even further removed, like the difference between scientists and literary criticism of a novel about scientists (see e.g. this paper on Frederic Jameson).
One of their Directors Thomas Meier came to our most recent Cambridge Conference on Catastrophic Risk (2022). They’ve also got some good people on their board like Elaine Scarry.
I would note that my sense is that they’re a bit more focussed on analysing ‘apocalyptic imaginaries’ from a sociological and criticial theory perspective. See for example their first journal issue, which is mostly critical analysis of narratives of apocalypse in fiction or conspiracy theories (rather than e.g. climate modelling of nuclear winter). They strike me as somewhat similar to the Centre for the Critical Study of Apocalyptic and Millenarian Movements. Maybe a crude analagous distinction would be between scientists and philosophers of science?
On the youtube video, I wasn’t super impressed by that talk. It seemed more interested in pathologising research on global risks than engaging on the object level, similar to some of the more lurid recent work from Torres and Gebru. But I’m going to Schwarz’s talk this Friday in Cambridge so hopefully will be able to dig deeper.
Just quickly wanted to add Ryan’s shortform post.
This was a really interesting and useful read! Posting the summary from the end of the post, as I found it helpful:
To summarise, here are the most important factors informing my view:
China’s research on the important sub-domains of AI (such as transformer architectures and deep learning) are less impressive than headlines might otherwise indicate.
I suspect China’s economic growth will slow down considerably unless its political and economic system changes in a more pluralistic direction (and even then, that might not be enough). This will make spending on large, speculative projects like TAI more difficult to justify politically, provided basic needs are not being met and growth is stagnant.
China might create interesting narrow applications of AI in domains such as surveillance and other consumer products, but this might not be enough to propel their labs and firms to the frontier.
China has a massive problem with producing compute, and the proposed solutions do not seem to be sufficient to emerge at the cutting-edge of semiconductor manufacturing.
Some of the most promising short-to-medium-term paths towards TAI will require access to gargantuan volumes of computing power, and the US government has recently taken decisive action to prevent China from accessing it.
First mover advantages are real: China is not in a great position to emerge at the front any time soon, and the most important actors from a governance point of view are probably the ones who are most likely to develop TAI first.
China will struggle with talent if it primarily relies on Chinese-born scientists, who tend to return to China at low rates. A more liberal China with higher quality-of-life might attract foreign workers, but creating such a society is not exactly easy, nor is it necessarily desirable for many government elites.
Here are some things that would cause me to update in the other direction:
China manages to avoid a huge growth slowdown, and this cumulative economic growth makes the Chinese economy truly enormous, and solidifies the CCP’s political power even further.
China begins producing leading research in important TAI sub-domains, or is able to closely follow the West. A cutting-edge algorithmic or architectural discovery coming out of China would be particularly interesting in this respect.
China’s centralised access to data gives them a massive advantage over the West and their pesky data protection laws in the long-run.
Creating narrow AI products and services for the Chinese market proves to be insanely profitable, enough to flush Chinese labs with more cash than their Western counterparts.
China solves its semiconductor struggles and begins taking a large chunk of the semiconductor market, or manages to emerge as a frontrunner in an alternative computing paradigm such as quantum computing.
Creating TAI requires less compute than previously thought, or is possible to do with the kind of inferior-generation semiconductors that China can produce domestically.
First-mover advantages are not as important as once thought, and the Chinese government can spend in an attempt to narrow the gap.
China begins repatriating researchers who go abroad at much greater rates, or manages to start attracting non-Chinese talent in considerable numbers.
The Chinese government decides to throw a much larger portion of their GDP towards AI than other comparatively sized economies, and it turns out that money can buy AI progress.
I tweeted about this fairly breathlessly, but congratulations on this—its a really important bit of research. Not to get too carried away, but this plausibly could prove to be one of the most important AI governance reports of the year, even though its only February. I’m very excited by it.
The reason is: if its right, then big AI models will cost $100m+ *each* to train by 2030. So forget about academia, start-ups, open-source collectives or individuals: they can’t keep up! For good or ill, Big Tech will be the only game in town.
This has massive implications for the research field, business models, company strategy investors, regulation, antitrust, government investment/industrial strategy, AI governance, threat models, arms control etc.
They made ~142 grants in that 18 month period. Assuming some multiple grants, that’s still maybe 100-120 grantees to contact to ask whether they want to opt-in or not. Presumably most grantees will want to see, if not dispute, their tiered ranking before they opt in to publishing it. This will all take a fair amount of time—and perhaps time at a senior level: eg the relevant relationship-holder (presumably the Program Officer) will need to contact the grantees, and then the CEO of the grantee will want to see the ranking and perhaps dispute it. It also runs a fair risk of damaging relationships with grantees.
So I would not be surprised if OpenPhil did not release the full tiered ranking. What they could do is release the list they considered (or confirm if I or others are correct in our attempted replication). Then we can at least know the ‘universe of cases’ they considered.
Notable that Mary Robinson (Chair of The Elders and former UN High Commissioner for Human Rights) and Ban Ki-moon (Deputy Chair of The Elders and former Secretary-General of the United Nations) were both at the 2020 and 2023 Doomsday Clock announcements:
https://thebulletin.org/2020/01/press-release-it-is-now-100-seconds-to-midnight/
I looked at OpenPhil grants tagged as “Longtermism” in 2021 or 2022 (from here or the downloadable spreadsheet). I count 142 grants made since the beginning of July 2021. The bar isn’t being set by grants per se, but as a percentage: “% of our longtermist grantmaking over the last 18 months (by dollars)”. These 142 grants are about $235m, so half would be $117m.
For context, some grants are tiny, e.g. the smallest is $2,400. The top 10 represent over half that funding ($129m).
Centre for Effective Altruism — Biosecurity Coworking Space Centre for Effective Altruism Biosecurity & Pandemic Preparedness 5,318,000 Aug-22 Effective Altruism Funds — Re-Granting Support Center for Effective Altruism Effective Altruism Community Growth (Longtermism) 7,084,000 Feb-22 Centre for Effective Altruism — Harvard Square Coworking Space Center for Effective Altruism Effective Altruism Community Growth (Longtermism) 8,875,000 Aug-22 Redwood Research — General Support Redwood Research Potential Risks from Advanced AI 9,420,000 Nov-21 Good Forever — Regranting for Biosecurity Projects Good Forever Foundation Biosecurity & Pandemic Preparedness 10,000,000 Feb-22 Redwood Research — General Support (2022) Redwood Research Potential Risks from Advanced AI 10,700,000 Aug-22 Californians Against Pandemics — California Pandemic Early Detection and Prevention Act Ballot Initiative Californians Against Pandemics Biosecurity & Pandemic Preparedness 11,100,000 Oct-21 Massachusetts Institute of Technology — AI Trends and Impacts Research (2022) Massachusetts Institute of Technology Potential Risks from Advanced AI 13,277,348 Mar-22 Funding for AI Alignment Projects Working With Deep Learning Systems Potential Risks from Advanced AI 14,459,002 Apr-22 Center for Security and Emerging Technology — General Support (August 2021) Center for Security and Emerging Technology Potential Risks from Advanced AI 38,920,000 Aug-21 Edit: cut the following text thanks to Linch’s spot:
Though I note that you mentioned “we also included a number of grants now-defunct FTX-associated funders had made.”—it would be helpful for you to at least release the spreadsheet of grants you considered, even if for very understandable reasons you don’t want to publish the Tier ranking.
It seems like the majority of individual grantees (over several periods) are doing academic-related research.
Can Caleb or other fund managers say more about why “One heuristic we commonly use (especially for new, unproven grantees) is to offer roughly 70% of what we anticipate the grantee would earn in an industry role” rather than e.g. “offer roughly the same as what we anticipate the grantee would earn in academia”?
See e.g. the UC system salary scale: