I direct the AI:Futures and Responsibility Programme (https://www.ai-far.org/) at the University of Cambridge, which works on AI strategy, safety and governance. I also work on global catastrophic risks with the Centre for the Study of Existential Risk and AI strategy/policy with the Centre for the Future of Intelligence.
Sean_o_h
Karma: 3,878
Seems worth someone tracking who the major shareholdrs are and how many voting rights they hold—e.g. I’d bet the house that Jaan Tallinn would be against this, so it’d be good to know if there are enough to support him to ward against possibilities like this.
I agree especially with the first of these (third is outside my scope of expertise). There’s a lot of work that feels chronically underserved and unsupported in ‘implementation-focused policy advocacy and communication’.
E.g. standards are correctly mentioned. Participation in standards development processes is an unsexy, frequently tedious and time-consuming processes. The big companies have people whose day jobs is basically sitting in standards bodies like ISO making sure the eventual standards reflect company priorities. AI safety/governance has a few people fitting it in alongside a million other priorities.e.g. the EU AI office is still struggling to staff the AI Office, one of the most important safety implementation offices out there. e.g. I serve on an OECD AI working group, an international working group with WIC, have served on GPAI, PAI, and regularly advise UN and national governments. You can make a huge difference on these things especially if you have the time to develop proposals/recommendations and follow up between meetings. But I see the same faces from academia & cvil society at all of these—all of us exhausted, trying to fit in as much as we can, alongside research, management, fundraising, teaching+student mentorship + essays (for the academics).
Some of this is that it takes time as an independent to reach the level of seniority/recognition to be invited to these working groups. But my impression from being on funding review panels that many of the people who are well-placed to do so still have an uphill battle to get funding for themselves or the support they need. It helps if you’ve had something flashy and obviously impactful (e.g. AI-2027) but there’s a ton of behind-the-scenes work (that I think is much harder for funders to assess, and sometimes harder for philanthropists to get excited about) that an ecosystem with a chance of actually steering this properly and acting as the necessary counterweight to commercial pressures needs. Time and regular participation at national government level (a bunch of them!) plus US, EU, UN, OECD, G7/G20/G77, Africa, Belt&Road/ANSO, GPAI, ITU, ISO, things like Singapore SCAI & much more. Great opportunities for funders (including small funders).
Useful data and analysis thanks, though I’d note that from a TAI/AI risk-focused perspective I would expect the non-safety figures to overcount for some of these orgs. E.g. CFI (where I work) is in there at 25 FTE, but that covers a very broad range of AI governance/ethics/humanities topics, where only a subset (maybe a quarter?) would be specifically relevant to TAI governance (specifically a big chunk of Kinds of Intelligence that mainly does technical evaluation/benchmarking work, but advises policy on the basis of this, and the AI:FAR group). I would expect similar with some of the other ‘broader’ groups e.g. Ada.
Also in both categories I don’t follow the rationale for including GDM but not the other frontier companies with safety/governance teams e.g. Anthropic, OpenAI, xAI (admittedly more minimal). I can see a rationale for including all or none of them.
The text of the plan is here:
http://hk.ocmfa.gov.cn/eng/xjpzxzywshd/202507/t20250729_11679232.htm
Features a section on AI safety:
”Advancing the governance of AI safety. We need to conduct timely risk assessment of AI and propose targeted prevention and response measures to establish a widely recognized safety governance framework. We need to explore categorized and tiered management approaches, build a risk testing and evaluation system for AI, and promote the sharing of information as well as the development of emergency response of AI safety risks and threats. We need to improve data security and personal information protection standards, and strengthen the management of data security in processes such as the collection of training data and model generation. We need to increase investment in technological research and development, implement secure development standards, and enhance the interpretability, transparency, and safety of AI. We need to explore traceability management systems for AI services to prevent the misuse and abuse of AI technologies. We need to advocate for the establishment of open platforms to share best practices and promote international cooperation on AI safety governance worldwide.”
I will say though that I really enjoyed this—and it definitely imparts the, ah, appropriate degree of scepticism i might want potential applicants to have ;)
I’ve a similar concern to Geoffrey’s.
When I clicked on the video last week, there’s a prominent link to careers, then jobs. At the time, 3 of the top 5 were at AGI companies (Anthropic, OpenAI, GDM). I eventually found the ‘should you work at AGI labs?’ link, but it was much less obvious. This is the funnel that tens of thousands of people will be following (assuming 1% of people watching the video consider a change of career).
80K has for a long time pushed safety & governance at AGI companies as a top career path. While some of the safety work may have been very safety-dominant, a lot of it has in practice helped companies ship product, and advanced capabilities in doing so (think RLHF etc—see https://arxiv.org/pdf/2312.08039 for more discussion). This is inevitably a higher likelihood in a commercial setting than in e.g. academia.
Policy and governance roles have done some good, but have in practice also contributed to misplaced trust in companies and greater support for e.g. self-governance than might otherwise have been the case. Anecdotally, I became more trusting of OpenAI after working with their researchers on Towards Trustworthy AI (https://arxiv.org/pdf/2004.07213), in light of them individually signing onto (and in some cases proposing) mechanisms such as whistleblower mechanisms and other independent oversight mechanisms. At the same time unbeknownst to them, OpenAI leadership were building clauses into their contracts to strip them of their equity if they criticised the company on leaving. I expect the act by safety-focused academics like myself of coauthoring the report with openAI policy people will have also had the effect of increasing the perceptions of trustworthiness of OpenAI.
By now, almost everyone concerned about safety seems to have left OpenAI, often citing concerns over the ethics, safety-committedness and responsibilit-committedness of leadership. This includes everyone on Towards Trustworthy AI, and I expect many of the people funneled there by 80K. ( I get the impression from speaking to some of them that they feel they were being used as ‘useful idiots’). One of the people who left over concerns was Daniel Kokatajlo himself—who indeed had to give up 85% of his family’s net worth (temporarily, I believe) in order to be critical of OpenAI.
Another consequence of this funnel is that it’s contributed to the atrophy of the academic AI safety and governance pipeline, and loss of interest in supporting this part of the space by funders (‘isn’t the most exciting work happening inside the companies anyway?’). The most ethically-motivated people, who might otherwise have taken the hit of academic salaries and precarity, had a green light to go for the companies. This has contributed to the atrophy of the independent critical role, and government advisory role, that academia could have played in frontier AI governance.
There’s a lot more worth reflecting on than is captured in the ‘should you work at AI labs/companies’ article. While I’ve focused on OpenAI to be concrete here, the underlying issues apply to some degree across frontier AI companies.
And there is a lot more that a properly reflective 80K could be doing here. E.g.
Don’t have the tech company roles coming up at the very top the week you release your video
Have these high-risk roles flagged somehow, or in a separate category
Make it so the roles are accessed through the ‘should you work at tech companies’ article or in conjunction with it, so that people are more likely to read it
Heck, you could even have a little survey to answer before accessing these high risk roles, like when you’re investing in a high-risk asset
(this is all just off the top of my head, I’m sure there are better suggestions).
It’s long been part of 80k’s strategy to put people in high-consequence positions in the hope they can do good, and exert influence around them. It is a high-risk strategy with pretty big potential downsides. There have now been multiple instances in which this plan has been shown not to survive contact with the kind of highly agentic, skilled-at-wielding-power individuals who end up in CEO-and-similar positions (I can think of a couple of Sams, for instance). If 80k is going to be pointing a lot of in-expectation young and inexperienced people in these directions, it might benefit from being a little more reflective about how it does it.
I don’t think it’s impossible to do good from within companies, but I do expect you need to be skilful, sophisticated, and somewhat experienced. These are AGI companies. Their goal is to build AGI sooner than their competitors build AGI. Their leadership are extremely focused on this. Whether the role is in governance, or safety, it’s reasonable to expect ultimately that you as an employee will be expected to help them do that (certainly not hinder them)
Really inspiring. Grateful to you for writing this up.
Thanks Rían, I appreciate it. And to be fair, this is from my perspective as much a me thing as it is an Oli thing. Like, I don’t think the global optimal solution is an EA forum that’s a cuddly little safe space for me. But we all have to make the tradeoffs that make most sense for us individually, and this kind of thing is costly for me.
One other observation that might explain some of the different perceptions on ‘blame’ here.
I don’t think Oxford’s bureaucracy/administration is good, and I think it did behave very badly at points*. But overall, I don’t think Oxford’s bureaucracy/behaviour was a long way outside what you would expect for the reference class of thousand-year-old-institutions with >10,000 employees. And Nick knew that was what it was, chose to be situated there, and did benefit (particularly in the early days) from the reputation boost. I think there is some reasonable expectation that having made that choice, he would put some effort into either figuring out how to operate effectively within its constraints, or take it somewhere else.
(*it did at point have the feeling of grinding inevitability of a failing marriage, where beyond a certain point everything one side did was perceived in the worst light and with maximal irritation by the other side, going in both directions, which contributed to bad behaviour I think).
It wasn’t carefully chosen. It was the term used by the commenter I was replying to. I was a little frustrated, because it was another example of a truth-seeking enquiry by Milena getting pushed down the track of only-considering-answers-in-which-all-the-agency/wrongness-is-on-the-university side (including some pretty unpleasant options relating to people I’d worked with (‘parasitic egregore/siphon money’).
>Did Oxford think it was a reputation risk? Were the other philosophers jealous of the attention and funding FHI got? Was a beaurocratic parasitic egregore putting up roadblocks to siphon off money to itself? Garden variety incompetence?
So I just did copy and paste on the most relevant phrase, but flipped it. Bit blunter and more smart-arse than I would normally be (as you’ve presumably seen from my writing, I normally caveat to a probably-tedious degree), but I was finding it hard to challenge the simplistic fhi-good-uni-bad narrative. It was one line, I didn’t think much about it.
I remain of the view that the claim is true/a reasonable interpretation, but de novo / in a different context I would have phrased differently.
Deleting this because on re-reading I think I’m just repeating myself, but in a more annoyed way. Thanks for checking with other people, I’ll leave it at that.
Thank you. I’m grateful you checked with other people. Yes, I do think there is some history rewriting and mythologising going on here compared to my own memory of how things were, and this bothers me because I think the truth does matter.There is a very real sense in which Nick had a pretty sweet setup at Oxford, in terms of having the power and influence to do an unusual thing. And there were a bunch of people around him working insanely hard to help make that happen. I also do think there is a degree to which, yes, Nick blew it. I don’t really want to dwell on this because it feels a bit like bad-mouthing FHI at its funeral. And it’s not the whole story. But it is a source of some frustration to me, because I did not have that position of power and influence in trying to do somewhat similar things, and have spent years banging my head against various walls, and I would have liked to see the FHI story go well. That is not to say i would have made all the right decisions had I had that power and influence (I’m sure I would not), or that I did make all the right decisions in the situations I was in. But I still think I am within my rights to have a view as someone who was actually there for 4 years doing the thing.As well as all the good stuff, Nick was unusually pedantic and stubborn about a huge range of things, many of them (to my lights) relatively unimportant, from expensive cups to fonts to refusing to follow processes that would not realistically have impeded FHI’s intellectual activities to follow. And so many things would get framed as a battle to be won against the Faculty/University, where a bit of cooperation would have gone a long way. Play stupid games, win stupid prizes. It sucked up huge amounts of FHI time, huge amounts of Faculty time, and huge amounts of social capital, which made it harder to stand ground/get cooperation on the stuff that mattered. And it compounded over time. You don’t trust me, but do some digging and I think you’ll find it. There were two sides to this thing.All of this I am saying based on a lot of context and experience at FHI. Rather than question or challenge me on my original point, your immediate reaction was two multi-paragraph posts seemingly aimed at publicly discrediting me in every way—repeatedly saying that you don’t consider me trustworthy; that my career’s work has been bad for the world and therefore my takes shouldn’t be listened to; that I am some sort of malign intellectual influence who is somehow responsible for intellectual attacks on other people*. To me this doesn’t look like truth-seeking behaviour. It looks more like an effort to discredit a person who challenged the favoured narrative.Evenafterbeing told by someone who actually was at FHI that I was a big part of making it work, your response seems to imply that if you did some digging into my time at FHI, you would find that actually my influence turned out to be negative and harmful. Well do that digging, see if that’s what you find. I worked damn hard there, took personal risks, and did good work. You want to claim that’s false, you can show some evidence.*And no, I don’t think these things are equivalently harsh. I criticised Nick for bureaucratic mistakes. Nobody respects Nick primarily for his administrative/bureaucratic relationship skills. They respect him for other things, which I have praised on other occasions. Your personal go at me targeted pretty much every aspect of why people might respect me or consider me worth listening to. That is fundamentally different.
For what it’s worth, I’m (at least partly) sympathetic to Oli’s position here. If nothing else, from my end I’m not confident that the combined time usage of:
[Oli producing book-length critique of CSER/Leverhulme, or me personally, depending] +
[me producing presumably book-length response] +
[further back and forth] +
[a whole lot of forum readers trying to unpick the disagreements]
is overall worth it, particularly given (a) it seems likely to me there are some worldview/cultural differences that would take time to unpick and (b) I will be limited in what I can say on certain matters by professional constraints/norms.
I think this might be one of the LTFF writeups Oli mentions (apologies if wrong), and seems like a good place to start:
https://forum.effectivealtruism.org/posts/an9GrNXrdMwBJpHeC/long-term-future-fund-august-2019-grant-recommendations-1#Addendum__Thoughts_on_a_Strategy_Article_by_the_Leadership_of_Leverhulme_CFI_and_CSER
And as to the claim “I also wouldn’t be surprised if Sean’s takes were ultimately responsible for a good chunk of associated pressure and attacks on people’s intellectual integrity” it seems like some of this is based on my online comments/writing. I don’t believe I’ve ever deleted anything on the EA forum, LW, or very much on twitter/linkedin (the online mediums I use), my papers are all online, and so again a decent place to start is to search for my username and come to their own conclusions.
And I guess I should just say directly. I do wish it were possible to raise (specific) critical points on matter like faculty relations where I have some direct insight and discuss these, without immediate escalation to counterclaims that my career’s work has been bad for the world, that I am not to be trusted, and and that my influence is somehow responsible for attacks on people’s intellectual integrity. It’s very stressful and upsetting.
I suffer from (mild) social anxiety. That is not uncommon. This kind of very forceful interaction is valuable for some people but is difficult and costly for others to engage with. I am going to engage less with EA forum/LW as a result of this and a few similar interactions, and I am especially going to be more hesitant to be critical of EA/LW sacred cows. I imagine, given what you have said about my takes, that this will be positive from your perspective. So be it. But you might also consider the effect it will have on others who might be psychologically similar, and whose takes you might consider more valuable.
>”and would update a good amount on reports by people who were actually there, especially in the later years.”
For takes from people you might trust more than me, you might consider reaching out to Owen Cotton-Barratt, Niel Bowerman, or Page Hedley, all of whom played relevant roles later than me.
Thanks Habryka. My reason for commenting is that a one-sided story is being told here about the administrative/faculty relationship stuff, both by FHI and in the discussion here, and I feel it to be misleading in its incompleteness. It appears Carrick and I disagree and I respect his views, but I think many people who worked at FHI felt it to be severely administratively mismanaged for a long time. I felt presenting that perspective was important for trying to draw the right lessons.
I agree with the general point that maintaining independence under this kind of pressure is extremely hard, that there are difficult tradeoffs to make. I believe Nick made many of the right decisions in maintaining integrity and independence, and sometimes incurred costly penalties to do so that likely contributed to the administrative/bureaucratic tensions with the faculty. However, I think part of what is happening here is that some quite different things from working-inside-fhi-perspective are being conflated under broad ‘heading’ (intellectual integrity/independence) which sometimes overlapped, but often relatively minimally, and can be usefully disaggregated—intellectual vision and integrity; following administrative process for your hosting organisation; bureaucratic relationship management.
Pick your battles. If you’re going to be ‘weird’ along one dimension, it often makes sense to try to be ‘easy’ along others. The really important dimension was the intellectual independence. During my time FHI constantly incurred heavy costs for being uncooperative on many administrative and bureaucratic matters that I believe did not affect the intellectual element, or only minimally, often resulting in using up far more of FHI’s own team’s time than otherwise.
One anecdote. When I arrived at FHI in 2011, there was a head of admin at philosophy (basically running the faculty) called Tom (I think). His name was mud at FHI; the petty administrative tyrant who was thwarting everything FHI wanted to do. So I went and got to know him. Turns out the issue was fixed by my having a once a month meeting with him to talk through what we wanted to do, and figure out how to do it. Nearly everything we wanted to to do could be done, but sometimes following a process that FHI hadn’t been following, or looping in someone who needed to be aware. Not doing this had been causing him huge administrative hassle and extra workload. After that, he was regularly working overtime to help us on deadline occasions. On one occasion, he was (I’m sure) the only admin in Oxford working on Easter Monday, using the Oxford ‘authority’ to help us sort out a visa problem for a researcher’s wife unexpectedly stuck at an airport and panicking. A lot of that kind of thing. (*note, I expect that later in FHI’s time frictions were sufficiently entrenched to prevent these kinds of positive feedbacks)
I don’t particularly wish to have a referendum on my integrity, or a debate over whether CSER and CFI have been good or not. On the former, people can read my comment, your criticism, and make their own mind up how much to ‘trust’ me, or ask others who worked at FHI; the latter is a separate conversation where I am somewhat constrained in what I can say.
But briefly, for the same reasons that I think it’s important not to take the wrong lessons: I don’t agree that CSER and CFI have been bad for the world. They are also quite different than what my own visions for them would have been (in some ways good no doubt, in some ways bad perhaps). If you are to draw the direct comparison, I think it’s worth noting that Nick and I were in very different positions that afforded different freedoms. I took up the role at CSER somewhat reluctantly at Nick’s encouragement. I was too junior to play the kind of role that Nick played at FHI from Cambridge’s perspective (nowhere near being a professor), and there was already a senior board in place of professors mostly uninvolved with this field, and with quite different perspectives to mine. The founder whose perspectives most aligned with my own took a hands off role, for what I think are sound reasons. The extent to which this might come to limit my own intellectual and strategic relevance became apparent to me in 2015, and I spoke to Nick about resigning and doing something else; he persuaded me that staying and providing what intellectual and strategic direction I could appeared the highest value thing I could do. In hindsight, had my goal been to realise my own intellectual and strategic vision I would have been better served to continue direct academic work longer, progress to professor, and start something smaller a little later. In practice, my role required executing a shared vision in which my influence was one of many; or at CFI developing one of several distinct programmes.
With that said, I’m entirely confident that you are right that there were intellectual and strategic decisions I made that were the wrong ones, and where I judged the tradeoffs incorrectly. I’m also confident that had I been in Nick’s position, there are correct decisions that he made that I would not have had the intellectual courage to make or stick with in the face of opposition. And as I noted in a previous comment, I think elements of Nick’s personality in terms of stubborn-ness and uncompromising-ness on the way he wanted to do things contributed both to the intellectual independence and the administrative/bureaucratic problems; I just wish they could have been more selectively applied. (I also don’t think Nick made all the right intellectual and strategic decisions, but that, again, is a different discussion).
Re: incompetence in terms of faculty relationship, I believe the comment is correct and I stand by it. But it is of course only one part of the story (one i wanted not to be lost). And how strongly I hold that may be coloured by my own feelings. FHI was something that was important to me too, and that I put years of hard work into supporting. Even as late as 2022 I was working with Oxford to try to find solutions. I feel that there were many unforced errors, and I am frustrated.
(With apologies, I’m leaving for research meetings in China tomorrow, so will likely not have time to reply for a few weeks).
Having worked there and interfaced with the Faculty for 4 years, yes, I would expect garden variety incompetence on Bostrom’s part in terms of managing the relationship was a big part; I would predict the single biggest contributer to the eventual outcome.
Strong +1 re: ‘hero’ work culture. especially for ops staff. This was one of the things that bothered me while there and contributed to my moving on—an (admittedly very nice) attitude of praising (especially admin/management) people who were working stupidly hard/long, rather than actually investing in fixing a clearly dysfunctional situation. And while it might not have been possible to fix later on due to embedded animosity/frustration on both sides ⇒ hiring freeze etc, it certainly was early on when I was there.
The admin load issue was not just about the faculty. And the breakdown of relationship with the faculty was really was not one-sided, at least when I was there (and I think I succeeded in semi-rescuing some of the key relationships (oxford martin school, faculty of philosophy) while I was there, at least temporarily).
I can’t imagine it helped in winning allies in Oxford, but relationship with Faculty/University was already highly dysfunctional. (I was consulted as part of a review re: FHI’s position within Oxford and various options before said personal controversies).
Also, there is famously quite a lot of antisemitism on the left and far left. Sidestepping the academic debate on whether antisemitism is or is not technically a form of racism, it seem strange to me to claim that racism-and-adjacent only exist on the right.
(for avoidance of doubt, I agree with the OP that Hanania seems racist, and not a good ally for this community)
Maybe, but this also seems like the kind of extremely broadly salient thing where it would be more difficult for EAs to make a big difference on the margins with their work and funding compared to ‘regular’ EA causes. (though people should also focus time and money on things important to them)