Anyone with thoughts on what went wrong with EA’s involvement in OpenAI? It’s probably too late to apply any lessons to OpenAI itself, but maybe not too late elsewhere (e.g., Anthropic)?
At the risk of sounding, it’s really not clear to me that anything “went wrong”—from my outside perspective, it’s not like there was a clear mess-up on the part of EA’s anywhere here, just a difficult situation managed to the best of people’s abilities.
That doesn’t mean that it’s not worth pondering whether there’s any aspect that had been handled badly, or more broadly what one can take away from this situation (although we should beware to over-update on single notable events). But, not knowing the counterfactuals, and absent a clear picture of what things “going right” would have looked like, it’s not evident that this should be chalked up as a failing on the part of EA.
The thing that stands out to me as clearly seeming to go wrong is the lack of communication from the board during the whole debacle. Given that the final best guess at the reasoning for their decision seems like something could have explained[1], it does seem like an own goal that they didn’t try to do so at the time.
They were getting clear pressure from the OpenAI employees to do this for instance, this was one of the main complaints in the employee letter, and from talking to a couple of OAI employees I’m fairly convinced that this was sincere (i.e. they were just as in the dark as everyone else, and this was at least one of their main frustrations).
I’ve heard a few people make a comparison to other CEO-stepping-down situations, where it’s common for things to be relatively hush hush and “taking time out to spend with their family”. I think this isn’t a like for like comparison, because in those cases it’s usually a mutual agreement between the board and the CEO for them both to save face and preserve the reputation of the company. In the case of a sudden unilateral firing it seems more important to have your reasoning ready to explain publicly (or even privately, to the employees).
It’s possible of course that there are some secret details that explain this behaviour, but I don’t think there’s any reason to be overly charitable in assuming this. If there was some strategic tradeoff that the board members were making it’s hard to see what they were trading off against because they don’t seem to have ended up with anything in the deal[2]. I also don’t find “safety-related secret” explanations that compelling because I don’t see why they couldn’t have said this (that there was a secret, not what it was). Everyone involved was very familiar with the idea that AI safety infohazards might exist so this would have been a comprehensible explanation.
If I put myself in the position of the board members I can much more easily imagine feeling completely out of my depth in the situation that happened and ill-advisedly doubling down on this strategy of keeping quiet. It’s also possible they were getting bad advice to this effect, as lawyers tend to tell you to keep quiet, and there is general advice out there to “not engage with the twitter mob”.
Several minor fibs from Sam, saying different things to different board members to try and manipulate them. This does technically fit with the “not consistently candid” explanation but that was very cryptic without further clarification and examples
To frame this the other way, if they had kept quiet and then been given some lesser advisory position in the company afterwards you could more easily reason that some face-saving dealing had gone on
I wish I could signal boost this comment more. D’Angelo, Toner, and McCauley had a reason which was enough to persuade Ilya to take the drastic move of summarily removing Brockman from the board and then firing Sam, without any foresight and no communication whatsoever to the rest of the company, OpenAI stakeholders, or even Emmett (he threatened to resign unless they told him,[1] though not sure where that left off), which lost them legitimacy both internally and externally, and eventually lost them everything.
I honestly don’t have a plausible reason for it, or an belief they might have had which would help to square this circle, especially since I don’t buy the “AGI has been achieved internally” stuff. I honestly think that your explanation of not realising it was going to go so nuclear,[2] and then just doing whatever the lawyers told them, is what happened. But if so, the lawyers’ strategy of “say nothing, do nothing, just collect evidence and be quiet” was just an absolute disaster both for their own strategy and EA’s reputation and legitimacy as a whole.[3] It honestly just seems like staggering incompetency to me,[4] and the continued silence is just the most perplexing part of the entire saga, and I’m still (obviously) exercised by it.
Toner and McCauley remain as 2 members of the 5 person advisory board for The Centre for the Governance of AI. Any implications are left as an exercise for the reader.
The dangers of policy set by lawyers reminds me of the Heads of Harvard, UPenn, and MIT having a disaster in front of Congress about the Israel/Palestine/free-speech/genocide hearing, possibly because they were repeating some legally cleared lines about what the could say to limit liability instead of saying “genocide is bad and advocating it is against our code of conduct”
I think answers to this are highly downstream of object-level positions.
If you think timelines are short and scaled-up versions of current architectures will lead to AGI, then ‘what went wrong’ is contributing to vastly greater chance of extinction.
If you don’t agree with the above, then ‘what went wrong’ is overly dragging EA’s culture and perception to be focused on AI-Safety, and causing great damage to all of EA (even non-AI-Safety parts) when the OpenAI board saga blew up in Toner & McCauleys’ faces.
Lessons are probably downstream of this diagnosis.
My general lesson aligns with Bryan’s recent post—man is EA bad about communicating what it is, and despite the OpenAI fiasco not being an attempted EA-coup motivated by Pascal’s mugging longtermist concerns, it seems so many people have that as a ‘cached explanation’ of what went on. Feels to me like that is a big own goal and was avoidable.
Also on OpenAI, I think it’s bad that people like Joshua Achiam who do good work at OpenAI seem to really dislike EA. That’s a really bad sign—feels like the AI Safety community could have done more not to alienate people like him maybe.
Anyone with thoughts on what went wrong with EA’s involvement in OpenAI? It’s probably too late to apply any lessons to OpenAI itself, but maybe not too late elsewhere (e.g., Anthropic)?
At the risk of sounding, it’s really not clear to me that anything “went wrong”—from my outside perspective, it’s not like there was a clear mess-up on the part of EA’s anywhere here, just a difficult situation managed to the best of people’s abilities.
That doesn’t mean that it’s not worth pondering whether there’s any aspect that had been handled badly, or more broadly what one can take away from this situation (although we should beware to over-update on single notable events). But, not knowing the counterfactuals, and absent a clear picture of what things “going right” would have looked like, it’s not evident that this should be chalked up as a failing on the part of EA.
The thing that stands out to me as clearly seeming to go wrong is the lack of communication from the board during the whole debacle. Given that the final best guess at the reasoning for their decision seems like something could have explained[1], it does seem like an own goal that they didn’t try to do so at the time.
They were getting clear pressure from the OpenAI employees to do this for instance, this was one of the main complaints in the employee letter, and from talking to a couple of OAI employees I’m fairly convinced that this was sincere (i.e. they were just as in the dark as everyone else, and this was at least one of their main frustrations).
I’ve heard a few people make a comparison to other CEO-stepping-down situations, where it’s common for things to be relatively hush hush and “taking time out to spend with their family”. I think this isn’t a like for like comparison, because in those cases it’s usually a mutual agreement between the board and the CEO for them both to save face and preserve the reputation of the company. In the case of a sudden unilateral firing it seems more important to have your reasoning ready to explain publicly (or even privately, to the employees).
It’s possible of course that there are some secret details that explain this behaviour, but I don’t think there’s any reason to be overly charitable in assuming this. If there was some strategic tradeoff that the board members were making it’s hard to see what they were trading off against because they don’t seem to have ended up with anything in the deal[2]. I also don’t find “safety-related secret” explanations that compelling because I don’t see why they couldn’t have said this (that there was a secret, not what it was). Everyone involved was very familiar with the idea that AI safety infohazards might exist so this would have been a comprehensible explanation.
If I put myself in the position of the board members I can much more easily imagine feeling completely out of my depth in the situation that happened and ill-advisedly doubling down on this strategy of keeping quiet. It’s also possible they were getting bad advice to this effect, as lawyers tend to tell you to keep quiet, and there is general advice out there to “not engage with the twitter mob”.
Several minor fibs from Sam, saying different things to different board members to try and manipulate them. This does technically fit with the “not consistently candid” explanation but that was very cryptic without further clarification and examples
To frame this the other way, if they had kept quiet and then been given some lesser advisory position in the company afterwards you could more easily reason that some face-saving dealing had gone on
I wish I could signal boost this comment more. D’Angelo, Toner, and McCauley had a reason which was enough to persuade Ilya to take the drastic move of summarily removing Brockman from the board and then firing Sam, without any foresight and no communication whatsoever to the rest of the company, OpenAI stakeholders, or even Emmett (he threatened to resign unless they told him,[1] though not sure where that left off), which lost them legitimacy both internally and externally, and eventually lost them everything.
I honestly don’t have a plausible reason for it, or an belief they might have had which would help to square this circle, especially since I don’t buy the “AGI has been achieved internally” stuff. I honestly think that your explanation of not realising it was going to go so nuclear,[2] and then just doing whatever the lawyers told them, is what happened. But if so, the lawyers’ strategy of “say nothing, do nothing, just collect evidence and be quiet” was just an absolute disaster both for their own strategy and EA’s reputation and legitimacy as a whole.[3] It honestly just seems like staggering incompetency to me,[4] and the continued silence is just the most perplexing part of the entire saga, and I’m still (obviously) exercised by it.
Toner and McCauley remain as 2 members of the 5 person advisory board for The Centre for the Governance of AI. Any implications are left as an exercise for the reader.
https://www.bloomberg.com/news/articles/2023-11-21/altman-openai-board-open-talks-to-negotiate-his-possible-return
But how could they not realise this???
The dangers of policy set by lawyers reminds me of the Heads of Harvard, UPenn, and MIT having a disaster in front of Congress about the Israel/Palestine/free-speech/genocide hearing, possibly because they were repeating some legally cleared lines about what the could say to limit liability instead of saying “genocide is bad and advocating it is against our code of conduct”
Nothing has really changed my mind here
I think answers to this are highly downstream of object-level positions.
If you think timelines are short and scaled-up versions of current architectures will lead to AGI, then ‘what went wrong’ is contributing to vastly greater chance of extinction.
If you don’t agree with the above, then ‘what went wrong’ is overly dragging EA’s culture and perception to be focused on AI-Safety, and causing great damage to all of EA (even non-AI-Safety parts) when the OpenAI board saga blew up in Toner & McCauleys’ faces.
Lessons are probably downstream of this diagnosis.
My general lesson aligns with Bryan’s recent post—man is EA bad about communicating what it is, and despite the OpenAI fiasco not being an attempted EA-coup motivated by Pascal’s mugging longtermist concerns, it seems so many people have that as a ‘cached explanation’ of what went on. Feels to me like that is a big own goal and was avoidable.
Also on OpenAI, I think it’s bad that people like Joshua Achiam who do good work at OpenAI seem to really dislike EA. That’s a really bad sign—feels like the AI Safety community could have done more not to alienate people like him maybe.