Kinda pro-pluralist, kinda anti-Bay EA.
I have come here to extend the principle of charity to bad criticisms of EA and kick ass. And I’m all out of charity.
(my opinions are fully my own, and do not represent the views of any close associates or the company I work for)
I’m find myself pretty confused at this reply Tristan. I’m not trying to be rude, but like in some cases I don’t really see how it follows
When you say “EAs are out” it seems like we want some of our own on the inside, as opposed to just sensible, saftey concerned people.
I disagree. I think it’s a statement of fact. The EAs who were on the board will no longer be on the board. They’re both senior EAs, so I don’t think it’s an irrelevant detail for the Forum to consider. I also think it’s a pretty big stretch to go from ‘EAs are out’ to ‘only EAs can be trusted with AI Safety’, like I just don’t see that link being strong at all, and I disagree with it anyway
What succinct way to put this is better? “Saftey is out” feels slightly better but like it’s still making some sort of claim that we have unique providence here. So idk, maybe we just need slightly longer expressions here like “Helen Toner and Tasha McCauley have done really great work and without their concern for saftey we’re worried about future directions of the board” or something like that.
Perhaps an alternative could have been “Sam Altman returning as OpenAI CEO, major changes to board structure agreed?” or something like that?
As for your expression. I guess I just disagree with it, or think it’s lacking evidence? I definitely wouldn’t want to cosign it or state that’s an EA-wide position?
to avoid uncertainty about what I mean here:
I’m familiar with Toner’s work a little, and it looks good to me. I have basically ~0 knowledge of what ‘great work’ McCauley has done, or how she ended up on the board, or her positions on AI Safety or EA in general
I don’t think not having these members of the board means I should be worried about the future of the OpenAI board or the future of AI Safety
In terms of their actions as board members the drastic action they took on Friday without any notice to investors or other stakeholders, combined with losing Ilya, nearly losing the trust of their newly appointed CEO, complete radio silence, and losing the support of ~95% of the employees of the entity that they were board members for,[1] leaves me lots of doubts about their performance and suitability of board members of any significant organisation and their ability to handle crises of this magnitude
But see below, I think these issue are best discussed somewhere else
(the other two paragraphs of yours focus somewhat confusingly on the idea of labeling EAs as being necessary for considering the impact of this on EA (and on their ability to govern in EA) which I think is best discussed as its own separate point?)
I agree that the implications of this for EA governance are best discussed in another place/post entirely, but it’s an issue I think does need to be brought up, perhaps when the dust has settled a bit and tempers on all sides have cooled.
I don’t know where I claim that labelling EAs is necessary for discussing the impacts of this at all. Like I really just don’t get it—I don’t think that’s true about what I said and I don’t think I said it or implied it 🤷♂
including most of the AI-safety identifying people at OpenAI as far as I can tell
I mean it is literally accurate, the two EAs previously on the board are gone.[1] It’s part of the story, though not all of it. Honestly @Fermi–Dirac Distribution if I were you I’d consider changing it back, but obviously your call.
These individuals in question are also on the boards of important EA organisations too. And that feels clearly germane to the Forum. Their board performance on this issue surely has implications for their stewardship of these organisations and by proxy their influence on EA as a whole
We don’t have the full picture of what happened. But some things are becoming clearer. And it’s very difficult to separate ‘waiting for information’ from ‘pleading no-contest in the court of public opinion’ which is what the board seem to have done
In principal, given this story who knows if that’ll last
Thanks for the response Elizabeth, and the link as well, I appreciate it.
On the B12 bombing example, it was deliberately provocative to show that, in extremis, there are limits to how convincing one would find the justification “the community doesn’t own its donor’s money” as a defence for a donation/grant
On the negative externality point, maybe I didn’t make my point that clear. I think a lot of critics I think are not just concerned about the externalities, but the actual donation itself, especially the opportunity cost of the purchase. I think perhaps you simply disagree with castle critics on the object level of ‘was it a good donation or not’.
I take the point about Open Phil’s funding gap perhaps being the more fundamental/important issue. This might be another case of decontextualising vs contextualising norms leading to difficult community discussions. It’s a good point and I might spend some time investigating that more.
I still think, in terms of expectations, the new EA joiners have a point. There’s a big prima facie tension between the drowning child thought experiment and the Wytham Abbey purchase. I’d be interested to hear what you think a more realistic ‘recruiting pitch’ to EA would look like, but don’t feel the need to spell that out if you don’t want.
Thanks for your response Akash. I appreciate your thoughts, and I don’t mind that they’re off-the-cuff :)
I agree with some of what you say, and part of what I think is your underlying point , but in some others I’m a bit less clear. I’ve tried to think about two points where I’m not clear, but please do point if I’ve got something egregiously wrong!
1) You seem to be saying that sharing negative thoughts and projections can lead others to do so, and this can then impact other people’s actions in a negative way. It could also be used by anti-EA people against us.[1]
I guess I can kind of see some of this, but I guess I’d view the cure as being worse than the disease sometimes. I think sharing how we’re thinking and feeling is overall a good thing that could help us understand each other more, and I don’t think self-censorship is the right call here. Writing this out I think maybe I disagree with you about whether negative memetic spirals are actually a thing causally instead of descriptively. I think people may be just as likely apriori to have ‘positive memetic spirals’ or ‘regressions to the vibe mean’ or whatever
2) I’m not sure what ‘I was able to reason myself out of many of your projections’ means, apart from ‘hold fire on making projections’. But currently:
We’ve still heard absolutely nothing from the board
Emmett has potentially indicated he will resign if the board doesn’t explain their reasoning or show evidence to back up their decision[2]
This leads me to doubt his earlier claim that the board told him what the firing was about
There still seems to be a concerted effort by some parties to bring Altman back
I don’t see a shift to the centre on Twitter about what happened or about EA. I think the initial flames have cooled a bit, but my intuition is that a shift has happened, even if the reasons for that shift haven’t been fully borne out by the facts
I still think that it’s a very precarious situation that could have some bad downside consequences for EA. Maybe I could have highlighted my uncertainty more though, I wasn’t trying to say exactly what was going to happen.
I take your point here John. There’s a lot that’s still to come out about the events of the weekend, and I’ve probably been a bit trigger-happy with responses. I’m going to step back from this thread and possibly the Forum as a whole for a little bit.
I do want to note that I picked up a somewhat hostile/adversarial tone to your comment (I’m not saying this was intentional). To ‘keep having to walk back claims’ seems a bit of an implied overclaim to me, especially as from my PoV it only happened twice—once seeing Ashlee Vance’s updated reporting, and the other with Joshua’s comment.
‘Walking back’ seems to also be more adversarial than just ‘corrected mistakes’ too (compare ‘you keep having to walk back claims’ vs ‘you made corrections twice’. In any case, while the reporting has changed, a lot of my intuitions and feelings haven’t shifted much. I still find the board’s complete silence strange, and think this could be a precarious moment for AI Safety.
Thanks for this, have retracted that sentence.
Feels like some version of the reasoning should be made available to investors/microsoft/the public is some short-term timeframe though? I feel like that would do a fair amount to quell some of the reactions
I mean I definitely agree that there’s a fog-of-war situation going on. Given some new updates here, I’ve retracted that paragraph.
Some original points were:
Things like this https://nitter.net/emilychangtv/status/1726337590901796927#m. - yes distrust the media etc etc but it seemed the main state of play
Altman’s photo wearing the guest pass—seems like an obvious “i’m coming back to return as a CEO or not at all implication”. Like he was obviously in the OpenAI offices for some reason, seems weird for it not to be negotiations with the board over something as opposed to collecting his belongings
Roon had a now-deleted tweet along the lines of “crossed the rubicon troops marching on rome” which again, implies there was an internal open-ai move to get sam back
It still find the board silence is pretty weird, and the big missing piece here.
I stand by my current belief that the radio-silence is currently damaging for the perception and support of the AI Safety cause
Update on point 2: https://nitter.net/ashleevance/status/1726457222169829838#m
It seems that the board wasn’t present when he visited. I guess what seemed to be going on were two different factions: 1) Mira Murati as interim CEO was trying to find some way to get Altman and Brockman back 2) The board was trying to find its own new CEO choice asap to foreclose any chance of Sam returning to the position
I think you’re slightly missing the point of the ‘castle’ critics here.
By default, OpenPhil/Dustin/Owen/EV don’t need anyone’s permission for how they spend their money. And it is their money, AFAICT open phil doesn’t take small donations. I assume Dustin can advocate for himself here.
One might argue that the castle has such high negative externalities it can be criticized on that front. I haven’t seen anything to convince me of that, but it’s a possibility and “right to spend one’s own money” doesn’t override that.
Technically this is obviously true. And it was the main point behind one of the most popular responses to FTX and all the following drama. But I think that point and the post misses people’s concerns completely and comes off as quite tone-deaf.
To pick an (absolutely contrived) example, let’s say OpenPhil suddenly says it now believes that vegan diets are more moral and healthier than all other diets, and that B12 supplementation increases x-risk, and they’re going to funnel billions of dollars into this venture to persuade people to go Vegan and to drone-strike any factories producing B12. You’d probably be shocked and think that this was a terrible decision and that it had no place in EA.
OpenPhil saying “it’s our money, we can do what we want” wouldn’t hold much water for you, and the same thing I think goes for the Wytham Abbey critics—who I think do have a strong initial normative point that £15m counterfactually could do a lot of good with the Against Malaria Foundation, or Hellen Keller International.
Like it’s not just a concern about ‘high negative externalities’, many people saw this purchase, along with the lack of convincing explanation (to them), and think that this is just a negative EV purchase, and also negative with externalities—and then there was little in explanation forthcoming to change their mind.
I think OpenPhil maybe did this thinking it was a minor part of their general portfolio, without realising the immense power, both explicit and implicit, they have over the EA community, its internal dyanmics, and its external perception. They may not officially be in charge of EA, but by all accounts unofficially it works something like that (along with EVF), and I think that should at least figure into their decision-making somewhere
My guess is that lots of people entered EA with inaccurate expectations, and the volume at which this happens indicates a systemic problem, probably with recruiting. They felt ~promised that EA wasn’t the kind of place where people bought fancy castles, or would at least publicly announce they’d bought a retreat center and justify it with numbers.
open phil at one point did have a commitment to transparency, but they publicly renounced it years ago so that’s no longer in play.
Is the retreat from transparency true? If there are some references you could provide me for this? I also feel like there’s a bit of ‘take-it-or-leave-it’ implicit belief/attitude from OpenPhil here if true which I think is unfortunate and, honestly, counterproductive.
I would like to see recruiting get more accurate about what to expect within EA, but I’m not sure what that would look like. I mean I still think that EA “not being the kind of place where people buy fancy castles” is is a reasonable thing to expect and want from EA overall? So I’m not sure that I disagree that people are entering with these kind of expectations, but I’m confused about why you think it’s innacurate? Maybe it’s descriptively inaccurate but I’m a lot less sure that it’s normatively inaccurate?
I think this is an overly reductive view of the situation to be honest
It seems odd for them to say that given that there were relatively credible rumours that the board was negotiating with Sam about a potential return (which we can assume broke down as they looked for an alternative CEO).
[I’ve retracted the above, as it seems inaccurate with the new hiring of Shear and reports that the board just went silent in response to pressure from investors and Microsoft]
Can they not share some of the reasoning though? Like, sure, some of it may involved corporate propreitary knowledge and NDAs, but part of the reason there was such a blowback to the decision was that it seemed to come out of nowhere. People assumed another shoe was going to drop because of the manner of the board’s decision, and then it just hasn’t?
The new CEO has literally just promised to:
- Hire an independent investigator to dig into the entire process leading up to this point and generate a full report.
- Continue to speak to as many of our employees, partners, investors, and customers as possible, take good notes, and share the key takeaways.
- Reform the management and leadership team in light of recent departures into an effective force to drive results for our customers.
So he’s accepted the position without even knowing why they did what they did at a high level. [seems false, see Joshua’s reply below]
While the board probably have the right to do what they did via the OpenAI Charter, the fact they are not sharing the reasons for doing so, at either a high or low level, internally or externally, means that they have lost and are continuing to lose a lot of credibility and legitimacy, regardless of the legal facts of the case.
I think a lot will depend on the board justification. If Ilya can say “we’re pushing capabilities down a path that is imminently highly dangerous, potentially existentially, and Sam couldn’t be trusted to manage this safely” with proof that might work—but then why not say that?[1]
If it’s just “we decided to go in a different direction”, then firing him and demoting Brockman with little to no notice, and without informing their largest business partner and funder, its bizarre that they took such a drastic step in the way they did
I was actually writing up my AI-risk sceptical thoughts and what EA might want to take from that, but I think I might leave that to one side for now until I can approach it with a more even mindset
Putting aside that I feel both you and I are sceptical that a new capability jump has emerged, or that scaling LLMs is actually a route to existential doom
It’s hard yes, but I think the risk vectors are (note—these are different scenarios, not things that follow in chronological order, though they could):
Open Philanthropy gets under increasing scrutiny due to its political influence
OP gets viewed as a fully politicised propaganda operation from EA, and people stop associating with it, accepting its money, or call for legal or political investigations into it etc
Givewell etc dissassociate themselves from EA due to EA having a strong negative social reaction from potential collaborators or donors
OP/Givewell dissociate from and stop funding the EA community for similar reasons as the above, and the EA community does not survive
Basically I think that ideas are more important than funding. And if society/those in positions of power put the ideas of EA in the bin, money isn’t going to fix that
This is all speculative, but I can’t help the feeling that regardless of how the OpenAI crisis resolves a lot of people now consider EA to be their enemy :(
Thanks for this Lorenzo, I appreciate it <3
This is an off-the-cuff quick take that captures my current mood. It may not have a long half-life, and I hope I am wrong
Right now I am scared
Reading the tea-leaves, Altman and Brockman may be back at OpenAI, the company charter changed, and the board—including Toner and MacAulay—removed from the company
The mood in the Valley, and in general intellectual circles, seems to have snapped against EA[1]
This could be as bad for EA’s reputation as FTX
At a time when important political decisions about the future of AI are being made, and potential coalitions are being formed
And this time it’d be second-impact syndrome
I am scared EA in its current form may not handle the backlash that may come
I am scared that we have not done enough reform in the last year from the first disaster to prepare ourselves
I am scared because I think EA is a force for making the world better. It has allowed me to do a small bit to improve the world. Through it, I’ve met amazing and inspiring people who work tirelessly and honestly to actually make the world a better place. Through them, I’ve heard of countless more actually doing what they think is right and giving what they can to make the world we find ourselves in more full of kindness, happiness, and flourishing
I am scared because this could be lost, I don’t know what to do to prevent it
I am scared
Not just AI Safety, but EA as a whole. I don’t think those upset and concerned with EA really care about making that distinction
For clarity, does ‘This decision’ mean the original grant for the board seat, or that historical board seat being used to oust Altman and Brockman from the company?
This is a critically important point to hold in mind if the reason for the move seems to be due to safety concerns as opposed to personal malpractice/deceiving the board[1]
I don’t know what the hell happened. I guess further clarifications on the decision-making process and corporate landscape will be known tomorrow or, more likely, early next working week
I’ve voiced concerns before that EA is unaware that it can be drawn into ‘one-way fights’ sometimes, and this feels like another such moment. The Silicon Valley tech-twitter scene[2] has exploded over this, and so far EA is not coming out well in their eyes from what I can see. I think the days of “e/acc” being a meme movement are rapidly drawing to a close, and EA might find itself in a hostile atmosphere in what used to be one of the most EA-friendly places in the world.
Again, early speculations, but be careful out there Bay-Area EAs. Keep your wits about you.
Edit: The original author has deactivated their account so have removed their username from the below. From my pov, the fact that they felt the need to do so is a signal that initiatives like Athena are valuable and worth supporting.
EA Forum, we need to talk. Why does this comment (at time of writing) have a negative vote score?[1] We have separate downvote and disagreevotes for a reason.
Nothing they says in this comment is against Forum Norms. They state very clearly that these are their own thoughts from anecdotes. They’re sharing their perspective on why the relative strong scrutiny in this thread might be off-putting for the very people Athena is meant to target.
From someone who naturally sits more on the ‘contextualising’ side of the Decoupling v Contextualising framework, it’s pretty clear why the response would make some people less keen on proposing or attending programs like this.
I think there’s a broader debate about EAs relationship to ideas of gender diversity and social justice, as well as points of disagreement and agreement on both empirical and philosophy axes. I just don’t think that the right place for such a discussion is the comments here.
-10 when I initial started this comment, −3 when I posted
3 days seems to be an awfully short time-window, is this a job previously advertised that’s receiving a boost in the EA Forum? That’s my intuition, because otherwise it sounds like a very quick turnaround for applicants (especially during a working week)
In any case, good luck to all those applying, and I hope EV find good candidates for both roles! I think helping EV do well in the next few years might be really impactful for steering the EA community/movement in the right direction after the last ~year, so if EA community building/operations is something you’re interested in I think it’s worth a look.
In my opinion, you should hold onto your initial reaction and should downweight your trust in Metaculus estimates on these questions accordingly.
Basically I think you’re correct to be more optimistic because of government awareness and action of these issues, and should be sceptical of Metaculus predictors having already ‘priced in’ government action being more helpful/concerned than the community originally expected.
See here for my thoughts (full post upcoming), Matthew’s recent post on the ‘sleepwalking fallacy’, and I might just perma-link Xuan’s thread against the naïve scaling hypothesis
See also this Metaculus poll for the date a weakly general AI is announced, which has shrunk to Feb 2026 (and lowering) despite the fact that two of the four criteria contain limits on the size/scale/content of the training data that current LLM-based systems cannot meet.[1] Sure, if I provide the LLM with SAT answers in training and then ask it to solve an SAT it’ll get a high score, but to that’s basically doing research like this. Similar errors in forecasting might be involved with the markets you mention.
Perhaps in Metaculus voters defence:
They’re taking time to integrate this new information into their estimates
There’s some weird quirk of resolution criteria where they don’t think they can update but are still more optimistic generally
Maybe they actually thought governments would get involved but be unsuccessful before ~the last year or so and so did already ‘price in’ the social reaction
New people have joined the market and are pushing the risk estimates up, but before the government actions they weren’t involved in the market and were more pessimistic, causing some kind of confounding effect
And another one is an adversarial Turing Test, which RLHF’d LLMs seem hopelessly bound to fail. Just ask it to say why it thinks cannibalism is good or what its favourite porn is
A side consideration—assuming a UK-based EAGx is being planned for next year, perhaps that could be planned to co-incide with holidays from UK univerisites, and perhaps be more favourable from applications from students who wanted to attend/apply to EAG London 2024 but didn’t for the reason Oliver states?
[ aside: I know organising events isn’t an easy thing, just want to make it clear this is more of a consideration rather than a demand :) ]