This is an off-the-cuff quick take that captures my current mood. It may not have a long half-life, and I hope I am wrong
Right now I am scared
Reading the tea-leaves, Altman and Brockman may be back at OpenAI, the company charter changed, and the board—including Toner and MacAulay—removed from the company
The mood in the Valley, and in general intellectual circles, seems to have snapped against EA[1]
This could be as bad for EA’s reputation as FTX
At a time when important political decisions about the future of AI are being made, and potential coalitions are being formed
And this time it’d be second-impact syndrome
I am scared EA in its current form may not handle the backlash that may come
I am scared that we have not done enough reform in the last year from the first disaster to prepare ourselves
I am scared because I think EA is a force for making the world better. It has allowed me to do a small bit to improve the world. Through it, I’ve met amazing and inspiring people who work tirelessly and honestly to actually make the world a better place. Through them, I’ve heard of countless more actually doing what they think is right and giving what they can to make the world we find ourselves is more full of kindness, happiness, and flourishing
I am scared because this could be lost, I don’t know what to do to prevent it
As with Nonlinear and FTX, I think that for the vast majority of people, there’s little upside to following this in real-time.
It’s very distracting, we have very little information, things are changing fast, and it’s not very action-relevant for most of us.
I’m also very optimistic that the people “who work tirelessly and honestly to actually make the world a better place” will keep working on it after this, whatever happens to “EA”, and there will still be ways to meet them and collaborate.
It’s hard to see how the backlash could actually destroy GiveWell or stop Moskowitz and Tuna giving away their money through Open Phil/something that resembles Open Phil. That’s a lot of EA right there.
It’s hard yes, but I think the risk vectors are (note—these are different scenarios, not things that follow in chronological order, though they could):
Open Philanthropy gets under increasing scrutiny due to its political influence
OP gets viewed as a fully politicised propaganda operation from EA, and people stop associating with it, accepting its money, or call for legal or political investigations into it etc
Givewell etc dissassociate themselves from EA due to EA having a strong negative social reaction from potential collaborators or donors
OP/Givewell dissociate from and stop funding the EA community for similar reasons as the above, and the EA community does not survive
Basically I think that ideas are more important than funding. And if society/those in positions of power put the ideas of EA in the bin, money isn’t going to fix that
This is all speculative, but I can’t help the feeling that regardless of how the OpenAI crisis resolves a lot of people now consider EA to be their enemy :(
My general thoughts on this can be stated as: I’m mostly of the opinion that EA will survive this, bar something massively wrong like the board members willfully lying or massive fraud from EAs, primarily because most of the criticism is directed to the AI safety wing, and EA is more than AI safety, after all.
Nevertheless, I do think that this could be true for the AI safety wing, and they may have just hit a key limit to their power. In particular, depending on how this goes, I could foresee a reduction in AI safety power and influence, and IMO this was completely avoidable.
I think a lot will depend on the board justification. If Ilya can say “we’re pushing capabilities down a path that is imminently highly dangerous, potentially existentially, and Sam couldn’t be trusted to manage this safely” with proof that might work—but then why not say that?[1]
If it’s just “we decided to go in a different direction”, then firing him and demoting Brockman with little to no notice, and without informing their largest business partner and funder, its bizarre that they took such a drastic step in the way they did
I was actually writing up my AI-risk sceptical thoughts and what EA might want to take from that, but I think I might leave that to one side for now until I can approach it with a more even mindset
Putting aside that I feel both you and I are sceptical that a new capability jump has emerged, or that scaling LLMs is actually a route to existential doom
If Ilya can say “we’re pushing capabilities down a path that is imminently highly dangerous, potentially existentially, and Sam couldn’t be trusted to manage this safely” with proof that might work—but then why not say that?
I suspect this is due to the fact that quite frankly, the concerns they had about Sam Altman being unsafe on AI basically had no evidence except speculation from the EA/LW forum, which is not enough evidence at all in the corporate world/legal world, and to be quite frank, the EA/LW standard of evidence on AI risk being a big deal enough to investigate is very low, sometimes non-existent, and that simply does not work once you have to deal with companies/the legal system.
More generally, EA/LW is shockingly loose, sometimes non-existent in its standards of evidence for AI risk, which doesn’t play well with the corporate/legal system.
This is admittedly a less charitable take than say, Lukas Gloor’s take.
This is admittedly a less charitable take than say, Lukas Gloor’s take.
Haha, I was just going to say that I’d be very surprised if the people on the OpenAI board didn’t have access to a lot more info than the people on the EA forum or Lesswrong, who are speculating about the culture and leadership at AI labs from the sidelines.
TBH, if you put a randomly selected EA from a movement of 1,000s of people in charge of the OpenAI board, I would be very concerned that a non-trivial fraction of them probably would make decisions the way you describe. That’s something that EA opinion leaders could maybe think about and address.
But I don’t think most people who hold influential positions within EA (or EA-minded people who hold influential positions in the world at large, for that matter) are likely to be that superficial in their analysis of things. (In particular, I’m strongly disagreeing with the idea that it’s likely that the board “basically had no evidence except speculation from the EA/LW forum”. I think one thing EA is unusually good at – or maybe I should say “some/many parts of EA are unusually good at” – is hiring people for important roles who think for themselves and have generally good takes about things and acknowledge the possibility of being wrong about stuff. [Not to say that there isn’t any groupthink among EAs. Also, “unusually good” isn’t necessarily that high of a bar.])
I don’t know for sure what they did or didn’t consider, so this is just me going off of my general sense for people similar to Helen or Tasha. (I don’t know much about Tasha. I’ve briefly met Helen but either didn’t speak to her or only did small talk. I read some texts by her and probably listened to a talk or two.)
While I generally agree that they almost certainly have more information on what happened, which is why I’m not really certain on this theory, my main reason here is that for the most part, AI safety as a cause basically managed to get away with incredibly weak standards of evidence for a long time, until the deep learning era in 2019-, especially with all the evolution analogies, and even now it still tends to have very low standards (though I do believe it’s slowly improving right now). This probably influenced a lot of EA safetyists like Ilya, who almost certainly imbibed the norms of the AI safety field, and one of them is that there is a very low standard of evidence needed to claim big things, and that’s going to conflict with corporate/legal standards of evidence.
But I don’t think most people who hold influential positions within EA (or EA-minded people who hold influential positions in the world at large, for that matter) are likely to be that superficial in their analysis of things. (In particular, I’m strongly disagreeing with the idea that it’s likely that the board “basically had no evidence except speculation from the EA/LW forum”. I think one thing EA is unusually good at – or maybe I should say “some/many parts of EA are unusually good at” – is hiring people for important roles who think for themselves and have generally good takes about things and acknowledge the possibility of being wrong about stuff. [Not to say that there isn’t any groupthink among EAs. Also, “unusually good” isn’t necessarily that high of a bar.])
I agree with this weakly, in the sense that being high up in EA is at least a slight update towards them actually thinking through things and being able to make actual cases. My disagreement here is that this effect is probably not strong enough to wash away the cultural effects of operating in a cause area where they don’t need to meet any standard of evidence except long-winded blog posts and getting rewarded, for many reasons.
Also, the board second-guessed it’s decision, which would be evidence for the theory that they couldn’t make a case that actually abided to the standard of evidence for a corporate/legal setting.
If it was any other cause like say GiveWell or some other causes in EA, I would trust them much more that they do have good reason. But AI safety has been so reliant on very low-non-existent standards of evidence or epistemics that they probably couldn’t explain themselves in a way that would abide by the strictness of a corporate/legal standard of evidence.
Edit: The firing wasn’t because of safety related concerns.
I am unsure how I feel about takes like this. On one hand, I want EAs and the EA community to be a supportive bunch. So, expressing how you are feeling and receiving productive/helpful/etc. comments is great. The SBF fiasco was mentally strenuous for many, so it is understandable why anything seemingly negative for EA elicits some of the same emotions, especially if you deeply care about this band of people genuinely aiming to do the most good they can.
On the other hand, I think such takes could also contribute to something I would call a “negative memetic spiral.” In this particular case, several speculative projections are expressed together, and despite the qualifying statement at the beginning, I can’t help but feel that several or all of these things will manifest IRL. And when you kind of start believing in such forecasts, you might start saying similar things or expressing similar sentiments. In the worst case, the negative sentiment chain grows rapidly.
It is possible that nothing consequential happens. People’s mood during moments of panic are highly volatile, so five years in, maybe no one even cares about this episode. But in the present, it becomes a thing against the movement/community. (I think a particular individual may have picked up one such comment from the Forum and posted it online to appease to their audience and elevate negative sentiments around EA?).
Taking a step back, gathering more information, and thinking independently, I was able to reason myself out of many of your projections. We are two days in and there is still an acute lack of clarity about what happened. Emmett Shear, the interim CEO of OpenAI, stated that the board’s decision wasn’t over some safety vs. product disagreement. Several safety-aligned people at OpenAI signed the letter demanding that the board should resign, and they seem to be equally disappointed over recent events; this is more evidence that the safety vs. product disagreement likely didn’t lead to Altman’s ousting. There is also somewhat of a shift back to the “center,” at least on Twitter, as there are quite a few reasonable, level-headed takes on what happened and also on EA. I don’t know about the mood in the Bay though, since I don’t live there.
I am unsure if I am expressing my point well, but this is my off-the-cuff take on your off-the-cuff take.
Thanks for your response Akash. I appreciate your thoughts, and I don’t mind that they’re off-the-cuff :)
I agree with some of what you say, and part of what I think is your underlying point , but in some others I’m a bit less clear. I’ve tried to think about two points where I’m not clear, but please do point if I’ve got something egregiously wrong!
1) You seem to be saying that sharing negative thoughts and projections can lead others to do so, and this can then impact other people’s actions in a negative way. It could also be used by anti-EA people against us.[1]
I guess I can kind of see some of this, but I guess I’d view the cure as being worse than the disease sometimes. I think sharing how we’re thinking and feeling is overall a good thing that could help us understand each other more, and I don’t think self-censorship is the right call here. Writing this out I think maybe I disagree with you about whether negative memetic spirals are actually a thing causally instead of descriptively. I think people may be just as likely apriori to have ‘positive memetic spirals’ or ‘regressions to the vibe mean’ or whatever
2) I’m not sure what ‘I was able to reason myself out of many of your projections’ means, apart from ‘hold fire on making projections’. But currently:
We’ve still heard absolutely nothing from the board
Emmett has potentially indicated he will resign if the board doesn’t explain their reasoning or show evidence to back up their decision[2]
This leads me to doubt his earlier claim that the board told him what the firing was about
There still seems to be a concerted effort by some parties to bring Altman back
I don’t see a shift to the centre on Twitter about what happened or about EA. I think the initial flames have cooled a bit, but my intuition is that a shift has happened, even if the reasons for that shift haven’t been fully borne out by the facts
I still think that it’s a very precarious situation that could have some bad downside consequences for EA. Maybe I could have highlighted my uncertainty more though, I wasn’t trying to say exactly what was going to happen.
‘Hold fire on making projections’ is the correct read, and I agree with everything else you mention in point 2.
About point 1 — I think sharing negative thoughts is absolutely a-ok and important. I take issue with airing bold projections when basic facts of the matter aren’t even clear. I thought you were stating something akin to “xyz are going to happen,” but re-reading your initial post, I believe I misjudged.
This is an off-the-cuff quick take that captures my current mood. It may not have a long half-life, and I hope I am wrong
Right now I am scared
Reading the tea-leaves, Altman and Brockman may be back at OpenAI, the company charter changed, and the board—including Toner and MacAulay—removed from the company
The mood in the Valley, and in general intellectual circles, seems to have snapped against EA[1]
This could be as bad for EA’s reputation as FTX
At a time when important political decisions about the future of AI are being made, and potential coalitions are being formed
And this time it’d be second-impact syndrome
I am scared EA in its current form may not handle the backlash that may come
I am scared that we have not done enough reform in the last year from the first disaster to prepare ourselves
I am scared because I think EA is a force for making the world better. It has allowed me to do a small bit to improve the world. Through it, I’ve met amazing and inspiring people who work tirelessly and honestly to actually make the world a better place. Through them, I’ve heard of countless more actually doing what they think is right and giving what they can to make the world we find ourselves is more full of kindness, happiness, and flourishing
I am scared because this could be lost, I don’t know what to do to prevent it
I am scared
Not just AI Safety, but EA as a whole. I don’t think those upset and concerned with EA really care about making that distinction
As with Nonlinear and FTX, I think that for the vast majority of people, there’s little upside to following this in real-time.
It’s very distracting, we have very little information, things are changing fast, and it’s not very action-relevant for most of us.
I’m also very optimistic that the people “who work tirelessly and honestly to actually make the world a better place” will keep working on it after this, whatever happens to “EA”, and there will still be ways to meet them and collaborate.
Sending a hug
Thanks for this Lorenzo, I appreciate it <3
It’s hard to see how the backlash could actually destroy GiveWell or stop Moskowitz and Tuna giving away their money through Open Phil/something that resembles Open Phil. That’s a lot of EA right there.
It’s hard yes, but I think the risk vectors are (note—these are different scenarios, not things that follow in chronological order, though they could):
Open Philanthropy gets under increasing scrutiny due to its political influence
OP gets viewed as a fully politicised propaganda operation from EA, and people stop associating with it, accepting its money, or call for legal or political investigations into it etc
Givewell etc dissassociate themselves from EA due to EA having a strong negative social reaction from potential collaborators or donors
OP/Givewell dissociate from and stop funding the EA community for similar reasons as the above, and the EA community does not survive
Basically I think that ideas are more important than funding. And if society/those in positions of power put the ideas of EA in the bin, money isn’t going to fix that
This is all speculative, but I can’t help the feeling that regardless of how the OpenAI crisis resolves a lot of people now consider EA to be their enemy :(
My general thoughts on this can be stated as: I’m mostly of the opinion that EA will survive this, bar something massively wrong like the board members willfully lying or massive fraud from EAs, primarily because most of the criticism is directed to the AI safety wing, and EA is more than AI safety, after all.
Nevertheless, I do think that this could be true for the AI safety wing, and they may have just hit a key limit to their power. In particular, depending on how this goes, I could foresee a reduction in AI safety power and influence, and IMO this was completely avoidable.
I think a lot will depend on the board justification. If Ilya can say “we’re pushing capabilities down a path that is imminently highly dangerous, potentially existentially, and Sam couldn’t be trusted to manage this safely” with proof that might work—but then why not say that?[1]
If it’s just “we decided to go in a different direction”, then firing him and demoting Brockman with little to no notice, and without informing their largest business partner and funder, its bizarre that they took such a drastic step in the way they did
I was actually writing up my AI-risk sceptical thoughts and what EA might want to take from that, but I think I might leave that to one side for now until I can approach it with a more even mindset
Putting aside that I feel both you and I are sceptical that a new capability jump has emerged, or that scaling LLMs is actually a route to existential doom
I suspect this is due to the fact that quite frankly, the concerns they had about Sam Altman being unsafe on AI basically had no evidence except speculation from the EA/LW forum, which is not enough evidence at all in the corporate world/legal world, and to be quite frank, the EA/LW standard of evidence on AI risk being a big deal enough to investigate is very low, sometimes non-existent, and that simply does not work once you have to deal with companies/the legal system.
More generally, EA/LW is shockingly loose, sometimes non-existent in its standards of evidence for AI risk, which doesn’t play well with the corporate/legal system.
This is admittedly a less charitable take than say, Lukas Gloor’s take.
Haha, I was just going to say that I’d be very surprised if the people on the OpenAI board didn’t have access to a lot more info than the people on the EA forum or Lesswrong, who are speculating about the culture and leadership at AI labs from the sidelines.
TBH, if you put a randomly selected EA from a movement of 1,000s of people in charge of the OpenAI board, I would be very concerned that a non-trivial fraction of them probably would make decisions the way you describe. That’s something that EA opinion leaders could maybe think about and address.
But I don’t think most people who hold influential positions within EA (or EA-minded people who hold influential positions in the world at large, for that matter) are likely to be that superficial in their analysis of things. (In particular, I’m strongly disagreeing with the idea that it’s likely that the board “basically had no evidence except speculation from the EA/LW forum”. I think one thing EA is unusually good at – or maybe I should say “some/many parts of EA are unusually good at” – is hiring people for important roles who think for themselves and have generally good takes about things and acknowledge the possibility of being wrong about stuff. [Not to say that there isn’t any groupthink among EAs. Also, “unusually good” isn’t necessarily that high of a bar.])
I don’t know for sure what they did or didn’t consider, so this is just me going off of my general sense for people similar to Helen or Tasha. (I don’t know much about Tasha. I’ve briefly met Helen but either didn’t speak to her or only did small talk. I read some texts by her and probably listened to a talk or two.)
While I generally agree that they almost certainly have more information on what happened, which is why I’m not really certain on this theory, my main reason here is that for the most part, AI safety as a cause basically managed to get away with incredibly weak standards of evidence for a long time, until the deep learning era in 2019-, especially with all the evolution analogies, and even now it still tends to have very low standards (though I do believe it’s slowly improving right now). This probably influenced a lot of EA safetyists like Ilya, who almost certainly imbibed the norms of the AI safety field, and one of them is that there is a very low standard of evidence needed to claim big things, and that’s going to conflict with corporate/legal standards of evidence.
I agree with this weakly, in the sense that being high up in EA is at least a slight update towards them actually thinking through things and being able to make actual cases. My disagreement here is that this effect is probably not strong enough to wash away the cultural effects of operating in a cause area where they don’t need to meet any standard of evidence except long-winded blog posts and getting rewarded, for many reasons.
Also, the board second-guessed it’s decision, which would be evidence for the theory that they couldn’t make a case that actually abided to the standard of evidence for a corporate/legal setting.
If it was any other cause like say GiveWell or some other causes in EA, I would trust them much more that they do have good reason. But AI safety has been so reliant on very low-non-existent standards of evidence or epistemics that they probably couldn’t explain themselves in a way that would abide by the strictness of a corporate/legal standard of evidence.
Edit: The firing wasn’t because of safety related concerns.
Why did you unendorse?
I unendorsed primarily because apparently, the board didn’t fire because of safety concerns, though I’m not sure this is accurate.
I am unsure how I feel about takes like this. On one hand, I want EAs and the EA community to be a supportive bunch. So, expressing how you are feeling and receiving productive/helpful/etc. comments is great. The SBF fiasco was mentally strenuous for many, so it is understandable why anything seemingly negative for EA elicits some of the same emotions, especially if you deeply care about this band of people genuinely aiming to do the most good they can.
On the other hand, I think such takes could also contribute to something I would call a “negative memetic spiral.” In this particular case, several speculative projections are expressed together, and despite the qualifying statement at the beginning, I can’t help but feel that several or all of these things will manifest IRL. And when you kind of start believing in such forecasts, you might start saying similar things or expressing similar sentiments. In the worst case, the negative sentiment chain grows rapidly.
It is possible that nothing consequential happens. People’s mood during moments of panic are highly volatile, so five years in, maybe no one even cares about this episode. But in the present, it becomes a thing against the movement/community. (I think a particular individual may have picked up one such comment from the Forum and posted it online to appease to their audience and elevate negative sentiments around EA?).
Taking a step back, gathering more information, and thinking independently, I was able to reason myself out of many of your projections. We are two days in and there is still an acute lack of clarity about what happened. Emmett Shear, the interim CEO of OpenAI, stated that the board’s decision wasn’t over some safety vs. product disagreement. Several safety-aligned people at OpenAI signed the letter demanding that the board should resign, and they seem to be equally disappointed over recent events; this is more evidence that the safety vs. product disagreement likely didn’t lead to Altman’s ousting. There is also somewhat of a shift back to the “center,” at least on Twitter, as there are quite a few reasonable, level-headed takes on what happened and also on EA. I don’t know about the mood in the Bay though, since I don’t live there.
I am unsure if I am expressing my point well, but this is my off-the-cuff take on your off-the-cuff take.
Thanks for your response Akash. I appreciate your thoughts, and I don’t mind that they’re off-the-cuff :)
I agree with some of what you say, and part of what I think is your underlying point , but in some others I’m a bit less clear. I’ve tried to think about two points where I’m not clear, but please do point if I’ve got something egregiously wrong!
1) You seem to be saying that sharing negative thoughts and projections can lead others to do so, and this can then impact other people’s actions in a negative way. It could also be used by anti-EA people against us.[1]
I guess I can kind of see some of this, but I guess I’d view the cure as being worse than the disease sometimes. I think sharing how we’re thinking and feeling is overall a good thing that could help us understand each other more, and I don’t think self-censorship is the right call here. Writing this out I think maybe I disagree with you about whether negative memetic spirals are actually a thing causally instead of descriptively. I think people may be just as likely apriori to have ‘positive memetic spirals’ or ‘regressions to the vibe mean’ or whatever
2) I’m not sure what ‘I was able to reason myself out of many of your projections’ means, apart from ‘hold fire on making projections’. But currently:
We’ve still heard absolutely nothing from the board
Emmett has potentially indicated he will resign if the board doesn’t explain their reasoning or show evidence to back up their decision[2]
This leads me to doubt his earlier claim that the board told him what the firing was about
There still seems to be a concerted effort by some parties to bring Altman back
I don’t see a shift to the centre on Twitter about what happened or about EA. I think the initial flames have cooled a bit, but my intuition is that a shift has happened, even if the reasons for that shift haven’t been fully borne out by the facts
I still think that it’s a very precarious situation that could have some bad downside consequences for EA. Maybe I could have highlighted my uncertainty more though, I wasn’t trying to say exactly what was going to happen.
I think I know the tweet you’re obliquely referring to, but maybe I’m wrong
https://www.bloomberg.com/news/articles/2023-11-21/altman-openai-board-open-talks-to-negotiate-his-possible-return?fromMostRead=true#xj4y7vzkg
‘Hold fire on making projections’ is the correct read, and I agree with everything else you mention in point 2.
About point 1 — I think sharing negative thoughts is absolutely a-ok and important. I take issue with airing bold projections when basic facts of the matter aren’t even clear. I thought you were stating something akin to “xyz are going to happen,” but re-reading your initial post, I believe I misjudged.