Will Hurd is plausibly quite concerned about AI Risk[1]. It’s hard to know for sure because his campaign website is framed in the language of US-China competition (and has unfortunate-by-my-lights suggestions like “Equip the Military and Intelligence Community with Advanced AI”), but I think a lot of the proposed policies are relevant to AI risk.
Shivon Zilis left OpenAI allegedly because she had Elon Musk’s children and this was seen as a COI[2]. To the extent that there’s bad blood still between Altman and Musk, if instead of framing the board’s decisions are “doomer” vs “non-doomer”, we frame it as as “skeptical of giving Sam free rein” and “fine with letting Sam do whatever”, there’s a reasonable case that Zilis agrees with Musk enough that she would not side with the pro-Sam faction.
(EDIT 2023/12/04: Changed wording to be slightly more precise and slightly less strong) So there’s at least some evidence that any or all of Hoffman/Hurd/Zilis (the 3 board members that left recently) would’ve opposed Sam trying to ouster Toner. Far from certain, but I’d currently say[3] >50% (EDIT: that at least one of them would be opposed). Especially if it turns out that one or all of them were themselves pushed out by Altman and they started sharing notes. Of course, ousting Altman in retaliation is a pretty big move, and the more politically savvy ones might’ve found a better compromise solution.
His wikipedia page says “On September 20, 2023, Hurd unveiled a detailed plan for how he would regulate AI if elected President, comparing AI to nuclear fusion, and proposing creating a branch of the executive to deal solely and directly on the issue of AI, and proposing strict regulations on civilian AI usage.” The last one in particular doesn’t sound necessarily conducive to OpenAI/Microsoft’s advanced AI ambitions.
Of course, this counterfactual is hard to verify. The Twitter backlash + OAI revolt probably means people would be hesitant to be publicly pro-Toner, now.
I haven’t seen any coverage of the double structure or Anthropic exit which suggests that Amodei helped think up or write the double structure. Certainly, the language they use around the Anthropic public benefit corporation indicates they all think, at least post-exit, that the OA double structure was a terrible idea (eg. see the end of this article).
You don’t know that. They seem to have often had near majorities, rather than being a token 1 or 2 board members.
By most standards, Karnofsky and Sutskever are ‘doomers’, and Zillis is likely a ‘doomer’ too as that is the whole premise of Neuralink and she was a Musk representative (which is why she was pushed out after Musk turned on OA publicly and began active hostilities like breaking contracts with OA). Hoffman’s views are hard to characterize, but he doesn’t seem to clearly come down as an anti-doomer or to be an Altman loyalist. (Which would be a good enough reason for Altman to push him out; and for a charismatic leader, neutralizing a co-founder is always useful, for the same reason no one would sell life insurance to an Old Bolshevik in Stalinist Russia.)
If I look at the best timeline of the board composition I’ve seen thus far, at a number of times post-2018, it looks like there was a ‘near majority’ or even outright majority. For example, 2020-12-31 has either a tie or an outright majority for either side depending on how you assume Sutskever & Hoffman (Sutskever?/Zilis/Karnofsky/D’Angelo/McCauley vs Hoffman? vs Altman/Brockman), and with the 2021-12-31 list the Altman faction needs to pick up every possible vote to match the existing 5 ‘EA’ faction (Zilis/Karnofsky/D’Angelo/McCauley/Toner vs Hurd?/Sutskever?/Hoffman? vs Brockman/Altman) although this has to be wrong because the board maxes out at 7 according to the bylaws so it’s unclear how exactly the plausible majorities evolved over time.
I am reasonably confident Helen replaced Holden as a board member, so I don’t think your 2021-12-31 list is accurate. Maybe there was a very short period where they were both on the board, but I heard the intention was for Helen to replace Holden.
The main thing that I doubt is that Sam knew at the time that he was gifting the board to doomers. Ilya was a loyalist and non-doomer when appointed. Elon was I guess some mix of doomer and loyalist at the start. Given how AIS worries generally increased in SV circles over time, more likely than not some of D’Angelo, Hoffman, and Hurd moved toward the “doomer” pole over time.
Ilya has always been a doomer AFAICT, he was just loyal to Altman personally, who recruited him to OA. (I can tell you that when I spent a few hours chatting with him in… 2017 or something? a very long time ago, anyway—I don’t remember him dismissing the dangers or being pollyannaish.) ‘Superalignment’ didn’t come out of nowhere or surprise anyone about Ilya being in charge. Elon was… not loyal to Altman but appeared content to largely leave oversight of OA to Altman until he had one of his characteristic mood changes, got frustrated and tried to take over. In any case, he surely counts as a doomer by the time Zilis is being added to the board as his proxy. D’Angelo likewise seems to have consistently, in his few public quotes, been concerned about the danger.
A lot of people have indeed moved towards the ‘doomer’ pole but much of that has been timelines: AI doom in 2060 looks and feels a lot different from AI doom in 2027.
Hmm, OK. Back when I met Ilya, about 2018, he was radiating excitement that his next idea would create AGI, and didn’t seem sensitive to safety worries. I also thought it was “common knowledge” that his interest in safety increased substantially between 2018-22, and that’s why I was unsurprised to see him in charge of superalignment.
Re Elon-Zillis, all I’m saying is that it looked to Sam like the seat would belong to someone loyal to him at the time the seat was created.
You may well be right about D’Angelo and the others.
Hm, maybe it was common knowledge in some areas? I just always took him for being concerned. There’s not really any contradiction between being excited about your short-term work and worried about long-term risks. Fooling yourself about your current idea is an important skill for a researcher. (You ever hear the joke about Geoff Hinton? He suddenly solves how the brain works, at long last, and euphorically tells his daughter; she replies: “Oh Dad—not again!”)
Just judging from his Twitter feed, I got the weak impression D’Angelo is somewhat enthusiastic about AI and didn’t catch any concerns about existential safety.
Nitpicks:
I think Dario and others would’ve also been involved in setting up the corporate structure
Sam never gave the “doomer” faction a near majority. That only happened because 2-3 “non-doomers” left and Ilya flipped.
Re 2: It’s plausible, but I’m not sure that this is true. Points against:
Reid Hoffman was reported as being specifically pushed out by Altman: https://www.semafor.com/article/11/19/2023/reid-hoffman-was-privately-unhappy-about-leaving-openais-board
Will Hurd is plausibly quite concerned about AI Risk[1]. It’s hard to know for sure because his campaign website is framed in the language of US-China competition (and has unfortunate-by-my-lights suggestions like “Equip the Military and Intelligence Community with Advanced AI”), but I think a lot of the proposed policies are relevant to AI risk.
Shivon Zilis left OpenAI allegedly because she had Elon Musk’s children and this was seen as a COI[2]. To the extent that there’s bad blood still between Altman and Musk, if instead of framing the board’s decisions are “doomer” vs “non-doomer”, we frame it as as “skeptical of giving Sam free rein” and “fine with letting Sam do whatever”, there’s a reasonable case that Zilis agrees with Musk enough that she would not side with the pro-Sam faction.
(EDIT 2023/12/04: Changed wording to be slightly more precise and slightly less strong) So there’s at least some evidence that any or all of Hoffman/Hurd/Zilis (the 3 board members that left recently) would’ve opposed Sam trying to ouster Toner. Far from certain, but I’d currently say[3] >50% (EDIT: that at least one of them would be opposed). Especially if it turns out that one or all of them were themselves pushed out by Altman and they started sharing notes. Of course, ousting Altman in retaliation is a pretty big move, and the more politically savvy ones might’ve found a better compromise solution.
His wikipedia page says “On September 20, 2023, Hurd unveiled a detailed plan for how he would regulate AI if elected President, comparing AI to nuclear fusion, and proposing creating a branch of the executive to deal solely and directly on the issue of AI, and proposing strict regulations on civilian AI usage.” The last one in particular doesn’t sound necessarily conducive to OpenAI/Microsoft’s advanced AI ambitions.
Convoluted wording because of “executives claimed that they were born via in vitro fertilization (IVF).”
Of course, this counterfactual is hard to verify. The Twitter backlash + OAI revolt probably means people would be hesitant to be publicly pro-Toner, now.
I haven’t seen any coverage of the double structure or Anthropic exit which suggests that Amodei helped think up or write the double structure. Certainly, the language they use around the Anthropic public benefit corporation indicates they all think, at least post-exit, that the OA double structure was a terrible idea (eg. see the end of this article).
You don’t know that. They seem to have often had near majorities, rather than being a token 1 or 2 board members.
By most standards, Karnofsky and Sutskever are ‘doomers’, and Zillis is likely a ‘doomer’ too as that is the whole premise of Neuralink and she was a Musk representative (which is why she was pushed out after Musk turned on OA publicly and began active hostilities like breaking contracts with OA). Hoffman’s views are hard to characterize, but he doesn’t seem to clearly come down as an anti-doomer or to be an Altman loyalist. (Which would be a good enough reason for Altman to push him out; and for a charismatic leader, neutralizing a co-founder is always useful, for the same reason no one would sell life insurance to an Old Bolshevik in Stalinist Russia.)
If I look at the best timeline of the board composition I’ve seen thus far, at a number of times post-2018, it looks like there was a ‘near majority’ or even outright majority. For example, 2020-12-31 has either a tie or an outright majority for either side depending on how you assume Sutskever & Hoffman (Sutskever?/Zilis/Karnofsky/D’Angelo/McCauley vs Hoffman? vs Altman/Brockman), and with the 2021-12-31 list the Altman faction needs to pick up every possible vote to match the existing 5 ‘EA’ faction (Zilis/Karnofsky/D’Angelo/McCauley/Toner vs Hurd?/Sutskever?/Hoffman? vs Brockman/Altman) although this has to be wrong because the board maxes out at 7 according to the bylaws so it’s unclear how exactly the plausible majorities evolved over time.
I am reasonably confident Helen replaced Holden as a board member, so I don’t think your 2021-12-31 list is accurate. Maybe there was a very short period where they were both on the board, but I heard the intention was for Helen to replace Holden.
The main thing that I doubt is that Sam knew at the time that he was gifting the board to doomers. Ilya was a loyalist and non-doomer when appointed. Elon was I guess some mix of doomer and loyalist at the start. Given how AIS worries generally increased in SV circles over time, more likely than not some of D’Angelo, Hoffman, and Hurd moved toward the “doomer” pole over time.
Ilya has always been a doomer AFAICT, he was just loyal to Altman personally, who recruited him to OA. (I can tell you that when I spent a few hours chatting with him in… 2017 or something? a very long time ago, anyway—I don’t remember him dismissing the dangers or being pollyannaish.) ‘Superalignment’ didn’t come out of nowhere or surprise anyone about Ilya being in charge. Elon was… not loyal to Altman but appeared content to largely leave oversight of OA to Altman until he had one of his characteristic mood changes, got frustrated and tried to take over. In any case, he surely counts as a doomer by the time Zilis is being added to the board as his proxy. D’Angelo likewise seems to have consistently, in his few public quotes, been concerned about the danger.
A lot of people have indeed moved towards the ‘doomer’ pole but much of that has been timelines: AI doom in 2060 looks and feels a lot different from AI doom in 2027.
Hmm, OK. Back when I met Ilya, about 2018, he was radiating excitement that his next idea would create AGI, and didn’t seem sensitive to safety worries. I also thought it was “common knowledge” that his interest in safety increased substantially between 2018-22, and that’s why I was unsurprised to see him in charge of superalignment.
Re Elon-Zillis, all I’m saying is that it looked to Sam like the seat would belong to someone loyal to him at the time the seat was created.
You may well be right about D’Angelo and the others.
Hm, maybe it was common knowledge in some areas? I just always took him for being concerned. There’s not really any contradiction between being excited about your short-term work and worried about long-term risks. Fooling yourself about your current idea is an important skill for a researcher. (You ever hear the joke about Geoff Hinton? He suddenly solves how the brain works, at long last, and euphorically tells his daughter; she replies: “Oh Dad—not again!”)
Just judging from his Twitter feed, I got the weak impression D’Angelo is somewhat enthusiastic about AI and didn’t catch any concerns about existential safety.