I direct the AI:Futures and Responsibility Programme (https://www.ai-far.org/) at the University of Cambridge, which works on AI strategy, safety and governance. I also work on global catastrophic risks with the Centre for the Study of Existential Risk and AI strategy/policy with the Centre for the Future of Intelligence.
Sean_o_h
Karma: 3,748
Thanks Rían, I appreciate it. And to be fair, this is from my perspective as much a me thing as it is an Oli thing. Like, I don’t think the global optimal solution is an EA forum that’s a cuddly little safe space for me. But we all have to make the tradeoffs that make most sense for us individually, and this kind of thing is costly for me.
One other observation that might explain some of the different perceptions on ‘blame’ here.
I don’t think Oxford’s bureaucracy/administration is good, and I think it did behave very badly at points*. But overall, I don’t think Oxford’s bureaucracy/behaviour was a long way outside what you would expect for the reference class of thousand-year-old-institutions with >10,000 employees. And Nick knew that was what it was, chose to be situated there, and did benefit (particularly in the early days) from the reputation boost. I think there is some reasonable expectation that having made that choice, he would put some effort into either figuring out how to operate effectively within its constraints, or take it somewhere else.
(*it did at point have the feeling of grinding inevitability of a failing marriage, where beyond a certain point everything one side did was perceived in the worst light and with maximal irritation by the other side, going in both directions, which contributed to bad behaviour I think).
It wasn’t carefully chosen. It was the term used by the commenter I was replying to. I was a little frustrated, because it was another example of a truth-seeking enquiry by Milena getting pushed down the track of only-considering-answers-in-which-all-the-agency/wrongness-is-on-the-university side (including some pretty unpleasant options relating to people I’d worked with (‘parasitic egregore/siphon money’).
>Did Oxford think it was a reputation risk? Were the other philosophers jealous of the attention and funding FHI got? Was a beaurocratic parasitic egregore putting up roadblocks to siphon off money to itself? Garden variety incompetence?
So I just did copy and paste on the most relevant phrase, but flipped it. Bit blunter and more smart-arse than I would normally be (as you’ve presumably seen from my writing, I normally caveat to a probably-tedious degree), but I was finding it hard to challenge the simplistic fhi-good-uni-bad narrative. It was one line, I didn’t think much about it.
I remain of the view that the claim is true/a reasonable interpretation, but de novo / in a different context I would have phrased differently.
Deleting this because on re-reading I think I’m just repeating myself, but in a more annoyed way. Thanks for checking with other people, I’ll leave it at that.
Thank you. I’m grateful you checked with other people. Yes, I do think there is some history rewriting and mythologising going on here compared to my own memory of how things were, and this bothers me because I think the truth does matter.There is a very real sense in which Nick had a pretty sweet setup at Oxford, in terms of having the power and influence to do an unusual thing. And there were a bunch of people around him working insanely hard to help make that happen. I also do think there is a degree to which, yes, Nick blew it. I don’t really want to dwell on this because it feels a bit like bad-mouthing FHI at its funeral. And it’s not the whole story. But it is a source of some frustration to me, because I did not have that position of power and influence in trying to do somewhat similar things, and have spent years banging my head against various walls, and I would have liked to see the FHI story go well. That is not to say i would have made all the right decisions had I had that power and influence (I’m sure I would not), or that I did make all the right decisions in the situations I was in. But I still think I am within my rights to have a view as someone who was actually there for 4 years doing the thing.As well as all the good stuff, Nick was unusually pedantic and stubborn about a huge range of things, many of them (to my lights) relatively unimportant, from expensive cups to fonts to refusing to follow processes that would not realistically have impeded FHI’s intellectual activities to follow. And so many things would get framed as a battle to be won against the Faculty/University, where a bit of cooperation would have gone a long way. Play stupid games, win stupid prizes. It sucked up huge amounts of FHI time, huge amounts of Faculty time, and huge amounts of social capital, which made it harder to stand ground/get cooperation on the stuff that mattered. And it compounded over time. You don’t trust me, but do some digging and I think you’ll find it. There were two sides to this thing.All of this I am saying based on a lot of context and experience at FHI. Rather than question or challenge me on my original point, your immediate reaction was two multi-paragraph posts seemingly aimed at publicly discrediting me in every way—repeatedly saying that you don’t consider me trustworthy; that my career’s work has been bad for the world and therefore my takes shouldn’t be listened to; that I am some sort of malign intellectual influence who is somehow responsible for intellectual attacks on other people*. To me this doesn’t look like truth-seeking behaviour. It looks more like an effort to discredit a person who challenged the favoured narrative.Evenafterbeing told by someone who actually was at FHI that I was a big part of making it work, your response seems to imply that if you did some digging into my time at FHI, you would find that actually my influence turned out to be negative and harmful. Well do that digging, see if that’s what you find. I worked damn hard there, took personal risks, and did good work. You want to claim that’s false, you can show some evidence.*And no, I don’t think these things are equivalently harsh. I criticised Nick for bureaucratic mistakes. Nobody respects Nick primarily for his administrative/bureaucratic relationship skills. They respect him for other things, which I have praised on other occasions. Your personal go at me targeted pretty much every aspect of why people might respect me or consider me worth listening to. That is fundamentally different.
For what it’s worth, I’m (at least partly) sympathetic to Oli’s position here. If nothing else, from my end I’m not confident that the combined time usage of:
[Oli producing book-length critique of CSER/Leverhulme, or me personally, depending] +
[me producing presumably book-length response] +
[further back and forth] +
[a whole lot of forum readers trying to unpick the disagreements]
is overall worth it, particularly given (a) it seems likely to me there are some worldview/cultural differences that would take time to unpick and (b) I will be limited in what I can say on certain matters by professional constraints/norms.
I think this might be one of the LTFF writeups Oli mentions (apologies if wrong), and seems like a good place to start:
https://forum.effectivealtruism.org/posts/an9GrNXrdMwBJpHeC/long-term-future-fund-august-2019-grant-recommendations-1#Addendum__Thoughts_on_a_Strategy_Article_by_the_Leadership_of_Leverhulme_CFI_and_CSER
And as to the claim “I also wouldn’t be surprised if Sean’s takes were ultimately responsible for a good chunk of associated pressure and attacks on people’s intellectual integrity” it seems like some of this is based on my online comments/writing. I don’t believe I’ve ever deleted anything on the EA forum, LW, or very much on twitter/linkedin (the online mediums I use), my papers are all online, and so again a decent place to start is to search for my username and come to their own conclusions.
And I guess I should just say directly. I do wish it were possible to raise (specific) critical points on matter like faculty relations where I have some direct insight and discuss these, without immediate escalation to counterclaims that my career’s work has been bad for the world, that I am not to be trusted, and and that my influence is somehow responsible for attacks on people’s intellectual integrity. It’s very stressful and upsetting.
I suffer from (mild) social anxiety. That is not uncommon. This kind of very forceful interaction is valuable for some people but is difficult and costly for others to engage with. I am going to engage less with EA forum/LW as a result of this and a few similar interactions, and I am especially going to be more hesitant to be critical of EA/LW sacred cows. I imagine, given what you have said about my takes, that this will be positive from your perspective. So be it. But you might also consider the effect it will have on others who might be psychologically similar, and whose takes you might consider more valuable.
>”and would update a good amount on reports by people who were actually there, especially in the later years.”
For takes from people you might trust more than me, you might consider reaching out to Owen Cotton-Barratt, Niel Bowerman, or Page Hedley, all of whom played relevant roles later than me.
Thanks Habryka. My reason for commenting is that a one-sided story is being told here about the administrative/faculty relationship stuff, both by FHI and in the discussion here, and I feel it to be misleading in its incompleteness. It appears Carrick and I disagree and I respect his views, but I think many people who worked at FHI felt it to be severely administratively mismanaged for a long time. I felt presenting that perspective was important for trying to draw the right lessons.
I agree with the general point that maintaining independence under this kind of pressure is extremely hard, that there are difficult tradeoffs to make. I believe Nick made many of the right decisions in maintaining integrity and independence, and sometimes incurred costly penalties to do so that likely contributed to the administrative/bureaucratic tensions with the faculty. However, I think part of what is happening here is that some quite different things from working-inside-fhi-perspective are being conflated under broad ‘heading’ (intellectual integrity/independence) which sometimes overlapped, but often relatively minimally, and can be usefully disaggregated—intellectual vision and integrity; following administrative process for your hosting organisation; bureaucratic relationship management.
Pick your battles. If you’re going to be ‘weird’ along one dimension, it often makes sense to try to be ‘easy’ along others. The really important dimension was the intellectual independence. During my time FHI constantly incurred heavy costs for being uncooperative on many administrative and bureaucratic matters that I believe did not affect the intellectual element, or only minimally, often resulting in using up far more of FHI’s own team’s time than otherwise.
One anecdote. When I arrived at FHI in 2011, there was a head of admin at philosophy (basically running the faculty) called Tom (I think). His name was mud at FHI; the petty administrative tyrant who was thwarting everything FHI wanted to do. So I went and got to know him. Turns out the issue was fixed by my having a once a month meeting with him to talk through what we wanted to do, and figure out how to do it. Nearly everything we wanted to to do could be done, but sometimes following a process that FHI hadn’t been following, or looping in someone who needed to be aware. Not doing this had been causing him huge administrative hassle and extra workload. After that, he was regularly working overtime to help us on deadline occasions. On one occasion, he was (I’m sure) the only admin in Oxford working on Easter Monday, using the Oxford ‘authority’ to help us sort out a visa problem for a researcher’s wife unexpectedly stuck at an airport and panicking. A lot of that kind of thing. (*note, I expect that later in FHI’s time frictions were sufficiently entrenched to prevent these kinds of positive feedbacks)
I don’t particularly wish to have a referendum on my integrity, or a debate over whether CSER and CFI have been good or not. On the former, people can read my comment, your criticism, and make their own mind up how much to ‘trust’ me, or ask others who worked at FHI; the latter is a separate conversation where I am somewhat constrained in what I can say.
But briefly, for the same reasons that I think it’s important not to take the wrong lessons: I don’t agree that CSER and CFI have been bad for the world. They are also quite different than what my own visions for them would have been (in some ways good no doubt, in some ways bad perhaps). If you are to draw the direct comparison, I think it’s worth noting that Nick and I were in very different positions that afforded different freedoms. I took up the role at CSER somewhat reluctantly at Nick’s encouragement. I was too junior to play the kind of role that Nick played at FHI from Cambridge’s perspective (nowhere near being a professor), and there was already a senior board in place of professors mostly uninvolved with this field, and with quite different perspectives to mine. The founder whose perspectives most aligned with my own took a hands off role, for what I think are sound reasons. The extent to which this might come to limit my own intellectual and strategic relevance became apparent to me in 2015, and I spoke to Nick about resigning and doing something else; he persuaded me that staying and providing what intellectual and strategic direction I could appeared the highest value thing I could do. In hindsight, had my goal been to realise my own intellectual and strategic vision I would have been better served to continue direct academic work longer, progress to professor, and start something smaller a little later. In practice, my role required executing a shared vision in which my influence was one of many; or at CFI developing one of several distinct programmes.
With that said, I’m entirely confident that you are right that there were intellectual and strategic decisions I made that were the wrong ones, and where I judged the tradeoffs incorrectly. I’m also confident that had I been in Nick’s position, there are correct decisions that he made that I would not have had the intellectual courage to make or stick with in the face of opposition. And as I noted in a previous comment, I think elements of Nick’s personality in terms of stubborn-ness and uncompromising-ness on the way he wanted to do things contributed both to the intellectual independence and the administrative/bureaucratic problems; I just wish they could have been more selectively applied. (I also don’t think Nick made all the right intellectual and strategic decisions, but that, again, is a different discussion).
Re: incompetence in terms of faculty relationship, I believe the comment is correct and I stand by it. But it is of course only one part of the story (one i wanted not to be lost). And how strongly I hold that may be coloured by my own feelings. FHI was something that was important to me too, and that I put years of hard work into supporting. Even as late as 2022 I was working with Oxford to try to find solutions. I feel that there were many unforced errors, and I am frustrated.
(With apologies, I’m leaving for research meetings in China tomorrow, so will likely not have time to reply for a few weeks).
Having worked there and interfaced with the Faculty for 4 years, yes, I would expect garden variety incompetence on Bostrom’s part in terms of managing the relationship was a big part; I would predict the single biggest contributer to the eventual outcome.
Strong +1 re: ‘hero’ work culture. especially for ops staff. This was one of the things that bothered me while there and contributed to my moving on—an (admittedly very nice) attitude of praising (especially admin/management) people who were working stupidly hard/long, rather than actually investing in fixing a clearly dysfunctional situation. And while it might not have been possible to fix later on due to embedded animosity/frustration on both sides ⇒ hiring freeze etc, it certainly was early on when I was there.
The admin load issue was not just about the faculty. And the breakdown of relationship with the faculty was really was not one-sided, at least when I was there (and I think I succeeded in semi-rescuing some of the key relationships (oxford martin school, faculty of philosophy) while I was there, at least temporarily).
I can’t imagine it helped in winning allies in Oxford, but relationship with Faculty/University was already highly dysfunctional. (I was consulted as part of a review re: FHI’s position within Oxford and various options before said personal controversies).
Also, there is famously quite a lot of antisemitism on the left and far left. Sidestepping the academic debate on whether antisemitism is or is not technically a form of racism, it seem strange to me to claim that racism-and-adjacent only exist on the right.
(for avoidance of doubt, I agree with the OP that Hanania seems racist, and not a good ally for this community)
April Fools is
A strange game.
The only high-utility move is
Not to play.
Huge congratulations, you have made the world better. Thank you.
I quite liked it, but I’d happily give up praise posts if it meant not having the denouncement posts.
Supervolcanoes being unlikely to be a human extinction risk was also my conclusion when I looked into it for an extinction risk review (currently under peer review) late last year, from speaking to volcanologists—McGraw (2024) was not released at that point so I’m grateful for this analysis and to be pointed to the paper.
Datapoint: I put money in my pension.
I agree. I suspect that responses to calls for evidence over the years played a big role in introducing and normalising xrisk research ideas in the UK context, before the big moves we’ve seen in the last year.
e.g. a few representative examples
(2016) https://data.parliament.uk/writtenevidence/committeeevidence.svc/evidencedocument/science-and-technology-committee/robotics-and-artificial-intelligence/written/32690.pdf(2017)
(2022)
https://www.longtermresilience.org/post/future-of-compute-review-submission-of-evidence
And many more.
Not a perfect translation, but I like proto-EA and leading Irish language poet Sean O Riordain writing a poem about moral circle expansion back in 1971. (It reads a lot better in the original language).
https://comhar.ie/iris/81/5/ni-ceadmhach-neamhshuim/
Apathy Is Out
There’s not a fly, moth, bee,
man, or woman created by God
whose welfare’s not our responsibility;
to ignore their predicament
isn’t on.There’s not a madman in Mad Valley
we shouldn’t sit with
and keep company,
since
he’s sick in the head
on our behalf.There’s not a place, stream or bush, however remote;
or a flagstone
north, south, east or west
that we shouldn’t consider
without affection and empathy.
No matter how far South Africa,
no matter how distant the moon,
they’re part of us by right:
there’s not a single spot anywhere
we’re not a part of. We issue from everywhere.
Really inspiring. Grateful to you for writing this up.