I direct the AI:Futures and Responsibility Programme (https://www.ai-far.org/) at the University of Cambridge, which works on AI strategy, safety and governance. I also work on global catastrophic risks with the Centre for the Study of Existential Risk and AI strategy/policy with the Centre for the Future of Intelligence.
Sean_o_h
Karma: 3,933
Examples of fairly critical remarks about Anthropic’s actions. All more recent than your timeframe, because twitter’s search function is a pain and because my memory for twitter etc discussion is less good.
https://x.com/S_OhEigeartaigh/status/2029475839654388069 (re: gullible bunch memo)
https://x.com/S_OhEigeartaigh/status/2026957849994108990 (re: walking back RSP commitments)https://x.com/S_OhEigeartaigh/status/2019518744561873286 (re: safety evaluations)
(I am followed by some prominent US policy people, including the outgoing white house AI adviser, so if there were a bias against people who criticise Anthropic, I would expect to be punished pretty harshly)
As one datapoint, this is from 2024, and I saw no subsequent evidence from core AI safety funders (including CG) that it influenced their willingness to fund me (I’ve also been regularly critical of Anthropic, and the other companies, on twitter and elsewhere throughout this time).
https://www.lcfi.ac.uk/news-events/blog/post/reflections-on-machines-of-loving-grace
Yes, me too on Mick’s points.
This is a slow-burn solution, but the most effective support and rebuttals will come from people who aren’t EAs, but are just fair/principled, and have had enough exposure to EA to know when attacks are unfair. E.g. See Dean Ball this week. So the more surface area EAs can create with those sorts of people, the better the position EA is in. For example, I think Andy Masley’s datacenter water use posts created a lot of surface ara with such people and has been better for the EA ‘brand’ than any specific rebuttal.
(A part of this strategy involves, as a general principle, “behave with as much dignity, integrity and fairness as possible, even when others aren’t). (Admittedly my own responses usually involve gently poking fun, but I do try to stay good natured).
Thank you for your important work! You’re probably already speaking to them, but in this broader community and adjacent, I recommend the work of:
For ‘role of Africa in frontier AI safety’ isseuesCecil Abungu’s team’s work at ILINA https://www.ilinaprogram.org/research
Sumaya Nur Adan’s work, with her new CASA centre and also with OMS AIGI in Oxford https://aigi.ox.ac.uk/people/sumaya-nur-adan/
For more locally-focused safety work, the African Hub on AI Safety, peace and security
https://ai.uct.ac.za/african-hub-ai-safety-peace-security and relatedly this white paper from the Global Centre on AI Governance
https://www.globalcenter.ai/research/toward-an-african-agenda-for-ai-safety
(among others, of course).
This is appalling to read and I’m so sorry Fran. I don’t know what else to say, but I felt I should at least say that.
Maybe, but this also seems like the kind of extremely broadly salient thing where it would be more difficult for EAs to make a big difference on the margins with their work and funding compared to ‘regular’ EA causes. (though people should also focus time and money on things important to them)
Seems worth someone tracking who the major shareholdrs are and how many voting rights they hold—e.g. I’d bet the house that Jaan Tallinn would be against this, so it’d be good to know if there are enough to support him to ward against possibilities like this.
I agree especially with the first of these (third is outside my scope of expertise). There’s a lot of work that feels chronically underserved and unsupported in ‘implementation-focused policy advocacy and communication’.
E.g. standards are correctly mentioned. Participation in standards development processes is an unsexy, frequently tedious and time-consuming processes. The big companies have people whose day jobs is basically sitting in standards bodies like ISO making sure the eventual standards reflect company priorities. AI safety/governance has a few people fitting it in alongside a million other priorities.e.g. the EU AI office is still struggling to staff the AI Office, one of the most important safety implementation offices out there. e.g. I serve on an OECD AI working group, an international working group with WIC, have served on GPAI, PAI, and regularly advise UN and national governments. You can make a huge difference on these things especially if you have the time to develop proposals/recommendations and follow up between meetings. But I see the same faces from academia & cvil society at all of these—all of us exhausted, trying to fit in as much as we can, alongside research, management, fundraising, teaching+student mentorship + essays (for the academics).
Some of this is that it takes time as an independent to reach the level of seniority/recognition to be invited to these working groups. But my impression from being on funding review panels that many of the people who are well-placed to do so still have an uphill battle to get funding for themselves or the support they need. It helps if you’ve had something flashy and obviously impactful (e.g. AI-2027) but there’s a ton of behind-the-scenes work (that I think is much harder for funders to assess, and sometimes harder for philanthropists to get excited about) that an ecosystem with a chance of actually steering this properly and acting as the necessary counterweight to commercial pressures needs. Time and regular participation at national government level (a bunch of them!) plus US, EU, UN, OECD, G7/G20/G77, Africa, Belt&Road/ANSO, GPAI, ITU, ISO, things like Singapore SCAI & much more. Great opportunities for funders (including small funders).
Useful data and analysis thanks, though I’d note that from a TAI/AI risk-focused perspective I would expect the non-safety figures to overcount for some of these orgs. E.g. CFI (where I work) is in there at 25 FTE, but that covers a very broad range of AI governance/ethics/humanities topics, where only a subset (maybe a quarter?) would be specifically relevant to TAI governance (specifically a big chunk of Kinds of Intelligence that mainly does technical evaluation/benchmarking work, but advises policy on the basis of this, and the AI:FAR group). I would expect similar with some of the other ‘broader’ groups e.g. Ada.
Also in both categories I don’t follow the rationale for including GDM but not the other frontier companies with safety/governance teams e.g. Anthropic, OpenAI, xAI (admittedly more minimal). I can see a rationale for including all or none of them.
The text of the plan is here:
http://hk.ocmfa.gov.cn/eng/xjpzxzywshd/202507/t20250729_11679232.htm
Features a section on AI safety:
”Advancing the governance of AI safety. We need to conduct timely risk assessment of AI and propose targeted prevention and response measures to establish a widely recognized safety governance framework. We need to explore categorized and tiered management approaches, build a risk testing and evaluation system for AI, and promote the sharing of information as well as the development of emergency response of AI safety risks and threats. We need to improve data security and personal information protection standards, and strengthen the management of data security in processes such as the collection of training data and model generation. We need to increase investment in technological research and development, implement secure development standards, and enhance the interpretability, transparency, and safety of AI. We need to explore traceability management systems for AI services to prevent the misuse and abuse of AI technologies. We need to advocate for the establishment of open platforms to share best practices and promote international cooperation on AI safety governance worldwide.”
I will say though that I really enjoyed this—and it definitely imparts the, ah, appropriate degree of scepticism i might want potential applicants to have ;)
I’ve a similar concern to Geoffrey’s.
When I clicked on the video last week, there’s a prominent link to careers, then jobs. At the time, 3 of the top 5 were at AGI companies (Anthropic, OpenAI, GDM). I eventually found the ‘should you work at AGI labs?’ link, but it was much less obvious. This is the funnel that tens of thousands of people will be following (assuming 1% of people watching the video consider a change of career).
80K has for a long time pushed safety & governance at AGI companies as a top career path. While some of the safety work may have been very safety-dominant, a lot of it has in practice helped companies ship product, and advanced capabilities in doing so (think RLHF etc—see https://arxiv.org/pdf/2312.08039 for more discussion). This is inevitably a higher likelihood in a commercial setting than in e.g. academia.
Policy and governance roles have done some good, but have in practice also contributed to misplaced trust in companies and greater support for e.g. self-governance than might otherwise have been the case. Anecdotally, I became more trusting of OpenAI after working with their researchers on Towards Trustworthy AI (https://arxiv.org/pdf/2004.07213), in light of them individually signing onto (and in some cases proposing) mechanisms such as whistleblower mechanisms and other independent oversight mechanisms. At the same time unbeknownst to them, OpenAI leadership were building clauses into their contracts to strip them of their equity if they criticised the company on leaving. I expect the act by safety-focused academics like myself of coauthoring the report with openAI policy people will have also had the effect of increasing the perceptions of trustworthiness of OpenAI.
By now, almost everyone concerned about safety seems to have left OpenAI, often citing concerns over the ethics, safety-committedness and responsibilit-committedness of leadership. This includes everyone on Towards Trustworthy AI, and I expect many of the people funneled there by 80K. ( I get the impression from speaking to some of them that they feel they were being used as ‘useful idiots’). One of the people who left over concerns was Daniel Kokatajlo himself—who indeed had to give up 85% of his family’s net worth (temporarily, I believe) in order to be critical of OpenAI.
Another consequence of this funnel is that it’s contributed to the atrophy of the academic AI safety and governance pipeline, and loss of interest in supporting this part of the space by funders (‘isn’t the most exciting work happening inside the companies anyway?’). The most ethically-motivated people, who might otherwise have taken the hit of academic salaries and precarity, had a green light to go for the companies. This has contributed to the atrophy of the independent critical role, and government advisory role, that academia could have played in frontier AI governance.
There’s a lot more worth reflecting on than is captured in the ‘should you work at AI labs/companies’ article. While I’ve focused on OpenAI to be concrete here, the underlying issues apply to some degree across frontier AI companies.
And there is a lot more that a properly reflective 80K could be doing here. E.g.
Don’t have the tech company roles coming up at the very top the week you release your video
Have these high-risk roles flagged somehow, or in a separate category
Make it so the roles are accessed through the ‘should you work at tech companies’ article or in conjunction with it, so that people are more likely to read it
Heck, you could even have a little survey to answer before accessing these high risk roles, like when you’re investing in a high-risk asset
(this is all just off the top of my head, I’m sure there are better suggestions).
It’s long been part of 80k’s strategy to put people in high-consequence positions in the hope they can do good, and exert influence around them. It is a high-risk strategy with pretty big potential downsides. There have now been multiple instances in which this plan has been shown not to survive contact with the kind of highly agentic, skilled-at-wielding-power individuals who end up in CEO-and-similar positions (I can think of a couple of Sams, for instance). If 80k is going to be pointing a lot of in-expectation young and inexperienced people in these directions, it might benefit from being a little more reflective about how it does it.
I don’t think it’s impossible to do good from within companies, but I do expect you need to be skilful, sophisticated, and somewhat experienced. These are AGI companies. Their goal is to build AGI sooner than their competitors build AGI. Their leadership are extremely focused on this. Whether the role is in governance, or safety, it’s reasonable to expect ultimately that you as an employee will be expected to help them do that (certainly not hinder them)
Really inspiring. Grateful to you for writing this up.
Thanks Rían, I appreciate it. And to be fair, this is from my perspective as much a me thing as it is an Oli thing. Like, I don’t think the global optimal solution is an EA forum that’s a cuddly little safe space for me. But we all have to make the tradeoffs that make most sense for us individually, and this kind of thing is costly for me.
One other observation that might explain some of the different perceptions on ‘blame’ here.
I don’t think Oxford’s bureaucracy/administration is good, and I think it did behave very badly at points*. But overall, I don’t think Oxford’s bureaucracy/behaviour was a long way outside what you would expect for the reference class of thousand-year-old-institutions with >10,000 employees. And Nick knew that was what it was, chose to be situated there, and did benefit (particularly in the early days) from the reputation boost. I think there is some reasonable expectation that having made that choice, he would put some effort into either figuring out how to operate effectively within its constraints, or take it somewhere else.
(*it did at point have the feeling of grinding inevitability of a failing marriage, where beyond a certain point everything one side did was perceived in the worst light and with maximal irritation by the other side, going in both directions, which contributed to bad behaviour I think).
It wasn’t carefully chosen. It was the term used by the commenter I was replying to. I was a little frustrated, because it was another example of a truth-seeking enquiry by Milena getting pushed down the track of only-considering-answers-in-which-all-the-agency/wrongness-is-on-the-university side (including some pretty unpleasant options relating to people I’d worked with (‘parasitic egregore/siphon money’).
>Did Oxford think it was a reputation risk? Were the other philosophers jealous of the attention and funding FHI got? Was a beaurocratic parasitic egregore putting up roadblocks to siphon off money to itself? Garden variety incompetence?
So I just did copy and paste on the most relevant phrase, but flipped it. Bit blunter and more smart-arse than I would normally be (as you’ve presumably seen from my writing, I normally caveat to a probably-tedious degree), but I was finding it hard to challenge the simplistic fhi-good-uni-bad narrative. It was one line, I didn’t think much about it.
I remain of the view that the claim is true/a reasonable interpretation, but de novo / in a different context I would have phrased differently.
Deleting this because on re-reading I think I’m just repeating myself, but in a more annoyed way. Thanks for checking with other people, I’ll leave it at that.
Thank you. I’m grateful you checked with other people. Yes, I do think there is some history rewriting and mythologising going on here compared to my own memory of how things were, and this bothers me because I think the truth does matter.There is a very real sense in which Nick had a pretty sweet setup at Oxford, in terms of having the power and influence to do an unusual thing. And there were a bunch of people around him working insanely hard to help make that happen. I also do think there is a degree to which, yes, Nick blew it. I don’t really want to dwell on this because it feels a bit like bad-mouthing FHI at its funeral. And it’s not the whole story. But it is a source of some frustration to me, because I did not have that position of power and influence in trying to do somewhat similar things, and have spent years banging my head against various walls, and I would have liked to see the FHI story go well. That is not to say i would have made all the right decisions had I had that power and influence (I’m sure I would not), or that I did make all the right decisions in the situations I was in. But I still think I am within my rights to have a view as someone who was actually there for 4 years doing the thing.As well as all the good stuff, Nick was unusually pedantic and stubborn about a huge range of things, many of them (to my lights) relatively unimportant, from expensive cups to fonts to refusing to follow processes that would not realistically have impeded FHI’s intellectual activities to follow. And so many things would get framed as a battle to be won against the Faculty/University, where a bit of cooperation would have gone a long way. Play stupid games, win stupid prizes. It sucked up huge amounts of FHI time, huge amounts of Faculty time, and huge amounts of social capital, which made it harder to stand ground/get cooperation on the stuff that mattered. And it compounded over time. You don’t trust me, but do some digging and I think you’ll find it. There were two sides to this thing.All of this I am saying based on a lot of context and experience at FHI. Rather than question or challenge me on my original point, your immediate reaction was two multi-paragraph posts seemingly aimed at publicly discrediting me in every way—repeatedly saying that you don’t consider me trustworthy; that my career’s work has been bad for the world and therefore my takes shouldn’t be listened to; that I am some sort of malign intellectual influence who is somehow responsible for intellectual attacks on other people*. To me this doesn’t look like truth-seeking behaviour. It looks more like an effort to discredit a person who challenged the favoured narrative.Evenafterbeing told by someone who actually was at FHI that I was a big part of making it work, your response seems to imply that if you did some digging into my time at FHI, you would find that actually my influence turned out to be negative and harmful. Well do that digging, see if that’s what you find. I worked damn hard there, took personal risks, and did good work. You want to claim that’s false, you can show some evidence.*And no, I don’t think these things are equivalently harsh. I criticised Nick for bureaucratic mistakes. Nobody respects Nick primarily for his administrative/bureaucratic relationship skills. They respect him for other things, which I have praised on other occasions. Your personal go at me targeted pretty much every aspect of why people might respect me or consider me worth listening to. That is fundamentally different.
For what it’s worth, I’m (at least partly) sympathetic to Oli’s position here. If nothing else, from my end I’m not confident that the combined time usage of:
[Oli producing book-length critique of CSER/Leverhulme, or me personally, depending] +
[me producing presumably book-length response] +
[further back and forth] +
[a whole lot of forum readers trying to unpick the disagreements]
is overall worth it, particularly given (a) it seems likely to me there are some worldview/cultural differences that would take time to unpick and (b) I will be limited in what I can say on certain matters by professional constraints/norms.
I think this might be one of the LTFF writeups Oli mentions (apologies if wrong), and seems like a good place to start:
https://forum.effectivealtruism.org/posts/an9GrNXrdMwBJpHeC/long-term-future-fund-august-2019-grant-recommendations-1#Addendum__Thoughts_on_a_Strategy_Article_by_the_Leadership_of_Leverhulme_CFI_and_CSER
And as to the claim “I also wouldn’t be surprised if Sean’s takes were ultimately responsible for a good chunk of associated pressure and attacks on people’s intellectual integrity” it seems like some of this is based on my online comments/writing. I don’t believe I’ve ever deleted anything on the EA forum, LW, or very much on twitter/linkedin (the online mediums I use), my papers are all online, and so again a decent place to start is to search for my username and come to their own conclusions.
And while ongoing commentary on their safety/policy actions might not count for your criteria, this paper has been influential enough with policymakers that it probably counts (and goes against aspects of Anthropic’s stance on China that underlies a lot of their positions). https://papers.ssrn.com/abstract=5278644