It might help to imagine a hard takeoff scenario using only known sorts of NN & scaling effects… (LW crosspost, with >82 comments)
It Looks Like You’re Trying To Take Over The World
In A.D. 20XX. Work was beginning. “How are you gentlemen !!”… (Work. Work never changes; work is always hell.)
Specifically, a MoogleBook researcher has gotten a pull request from Reviewer #2 on his new paper in evolutionary search in auto-ML, for error bars on the auto-ML hyperparameter sensitivity like larger batch sizes, because more can be different and there’s high variance in the old runs with a few anomalously high performance values. (“Really? Really? That’s what you’re worried about?”) He can’t see why worry, and wonders what sins he committed to deserve this asshole Chinese (given the Engrish) reviewer, as he wearily kicks off yet another HQU experiment...
EDIT: this is going a bit viral, and it seems like many of the readers have missed key parts of the reporting. I wrote this as a reply to Wei Dai and a high-level summary for people who were already familiar with the details; I didn’t write this for people who were unfamiliar, and I’m not going to reference every single claim in it, as I have generally referenced them in my prior comments/tweets and explained the details & inferences there. If you are unaware of aspects like ‘Altman was trying to get Toner fired’ or pushing out Hoffman or how Slack was involved in Sutskever’s flip or why Sutskever flip-flopped back, still think Q* matters, haven’t noticed the emphasis put on the promised independent report, haven’t read the old NYer Altman profile or Labenz’s redteam experience etc., it may be helpful to catch up by looking at other sources; my comments have been primarily on LW since I’m not a heavy EAF user, plus my usual excerpts.
It was a pretty weak hand. There is this pervasive attitude that Sam Altman could have been dispensed with easily by the OA Board if it had been more competent; this strange implicit assumption that Altman is some johnny-come-lately where the Board screwed up by hiring him. Commenters seem to ignore the long history here—if anything, it was he who screwed up by hiring the Board!
Altman co-founded OA. He was the face in initial coverage and 1 of 2 board members (with Musk). He was a major funder of it. Even Elon Musk’s main funding of OA was through an Altman vehicle. He kicked out Musk when Musk decided he needed to be in charge of OA. Open Philanthropy (OP) only had that board seat and made a donation because Altman invited them to, and he could personally have covered the $30m or whatever OP donated for the seat; and no one cared or noticed when OP let the arrange lapse after the initial 3 years. (I had to contact OP to confirm this when someone doubted that the seat was no longer controlled by OP.) He thought up, drafted, and oversaw the entire for-profit thing in the first place, including all provisions related to board control. He voted for all the board members, filling it back up from when it was just him (& Greg Brockman at one point IIRC). He then oversaw and drafted all of the contracts with MS and others, while running the for-profit and eschewing equity in the for-profit. He designed the board to be able to fire the CEO because, to quote him, “the board should be able to fire me”. He interviewed every person OA hired, and used his networks to recruit for OA. And so on and so forth
Credit where credit is due—Altman may not have believed the scaling hypothesis like Dario Amodei, may not have invented PPO like John Schulman, may not have worked on DL from the start like Ilya Sutskever, may not have created GPT like Alec Radford, may not have written & optimized any code like Brockman’s—but the 2023 OA organization is fundamentally his work.
The question isn’t, “how could EAers* have ever let Altman take over OA and possibly kick them out”, but entirely the opposite: “how did EAers ever get any control of OA, such that they could even possibly kick out Altman?” Why was this even a thing given that OA was, to such an extent, an Altman creation?
The answer is: “because he gave it to them.” Altman freely and voluntarily handed it over to them.
So you have an answer right there to why the Board was willing to assume Altman’s good faith for so long, despite everyone clamoring to explain how (in hindsight) it was so obvious that the Board should always have been at war with Altman and regarding him as an evil schemer out to get them. But that’s an insane way for them to think! Why would he undermine the Board or try to take it over, when he was the Board at one point, and when he made and designed it in the first place? Why would he be money-hungry when he refused all the equity that he could so easily have taken—and in fact, various partner organizations wanted him to have in order to ensure he had ‘skin in the game’? Why would he go out of his way to make the double non-profit with such onerous & unprecedented terms for any investors, which caused a lot of difficulties in getting investment and Microsoft had to think seriously about, if he just didn’t genuinely care or believe any of that? Why any of this?
(None of that was a requirement, or even that useful to OA for-profit. Other double systems like Mozilla or Hershey don’t have such terms, they’re just normal corporations with a lot of shares owned by a non-profit, is all. OA for-profit could’ve been the same way. Certainly, if all of this was for PR reasons or some insidious decade-long scheme of Altman to ‘greenwash’ OA, it was a spectacular failure—nothing has occasioned more confusion and bad PR for OA than the double structure or capped-profit. See, for example, my shortly-before-the-firing Twitter argument with well-known AI researcher Delip Rao who repeatedly stated & doubled down on the claim that the OA non-profit legally owned the OA for-profit was not just factually wrong but misinformation. He helpfully linked to a page about political misinformation & propaganda campaigns online in case I had any doubt about what the term ‘misinformation’ meant.)
What happened is, broadly: ‘Altman made the OA non/for-profits and gifted most of it to EA with the best of intentions, but then it went so well & was going to make so much money that he had giver’s remorse, changed his mind, and tried to quietly take it back; but he had to do it by hook or by crook, because the legal terms said clearly “no takesie backsies”’. Altman was all for EA and AI safety and an all-powerful nonprofit board being able to fire him, and was sincere about all that, until OA & the scaling hypothesis succeeded beyond his wildest dreams†, and he discovered it was inconvenient for him and convinced himself that the noble mission now required him to be in absolute control, never mind what restraints on himself he set up years ago—he now understands how well-intentioned but misguided he was and how he should have trusted himself more. (Insert Garfield meme here.)
No wonder the board found it hard to believe! No wonder it took so long to realize Altman had flipped on them, and it seemed Sutskever needed Slack screenshots showing Altman blatantly lying to them about Toner before he finally, reluctantly, flipped. The Altman you need to distrust & assume bad faith of & need to be paranoid about stealing your power is also usually an Altman who never gave you any power in the first place! I’m still kinda baffled by it, personally.
He concealed this change of heart from everyone, including the board, gradually began trying to unwind it, overplayed his hand at one point—and here we are.
So, what could the EA faction of the board have done? …Not much, really. They only ever had the power that Altman gave them in the first place.
* I don’t really agree with this framing of Sutskever/Toner/McCauley/D’Angelo as “EA”, but for the sake of argument, I’ll go with this labeling.
† Please try to cast your mind back to when Altman et al would be planning all this in 2018-2019, with OA rapidly running out of cash after the mercurial Musk’s unexpected-but-inevitable betrayal, its DRL projects like OA5 remarkable research successes but commercially worthless, and just some interesting results like GPT-1 and then GPT-2-small coming out of their unsupervised learning backwater from Alec Radford tinkering around with RNNs and then these new ‘Transformer’ things. The idea that OA might somehow be worth over ninety billion dollars, yes, that’s ‘billion’ with a ‘b’ in scarcely 3 years would have been insane, absolutely insane, not a single person in the AI world would have taken you seriously if you had suggested that and if you emailed any of them asking about how plausible that was, they would have added an email filter to send your future emails to the trash bin. It is very easy to be sincerely full of the best intentions and discuss how to structure your double corporation to deal with windfalls like growing to a $1000b market cap when no one really expects that to happen, certainly not in the immediate future… Thus, no one is also sitting around going, ‘well wait, we required the board to not own equity, but if the company is worth even fraction of our long-term targets, and it’s recruiting with stock options like usual, then each employee is going to have, like, $10m or even $100m of pseudo-equity in the OA for-profit. That seems… problematic. Do we need to do something about it?’