Holden Karnofsky

• Holden Karnofsky 21 Jun 2024 0:49 UTC

  3 points

  0 ∶ 0

  in reply to: Pablo’s comment on: Case stud­ies on so­cial-welfare-based stan­dards in var­i­ous industries

  Thanks for pointing this out! Should be resolved now.

Case stud­ies on so­cial-welfare-based stan­dards in var­i­ous industries

• Holden Karnofsky 13 May 2024 21:03 UTC

  42 points

  4 ∶ 0

  Error

  The value
    NIL
  is not of type
    SIMPLE-STRING
  when binding #:USER-ID162

• Holden Karnofsky 13 May 2024 20:59 UTC

  51 points

  11 ∶ 1

  in reply to: Adam_Scholl’s comment on: Join­ing the Carnegie En­dow­ment for In­ter­na­tional Peace

  My spouse isn’t currently planning to divest the full amount of her equity. Some factors here: (a) It’s her decision, not mine. (b) The equity has important voting rights, such that divesting or donating it in full could have governance implications. (c) It doesn’t seem like this would have a significant marginal effect on my real or perceived conflict of interest: I could still not claim impartiality when married to the President of a company, equity or no. With these points in mind, full divestment or donation could happen in the future, but there’s no immediate plan to do it.

  The bottom line is that I have a significant conflict of interest that isn’t going away, and I am trying to help reduce AI risk despite that. My new role will not have authority over grants or other significant resources besides my time and my ability to do analysis and make arguments. People encountering any analysis and arguments will have to decide how to weigh my conflict of interest for themselves, while considering arguments and analysis on the merits.

  For whatever it’s worth, I have publicly said that the world would pause AI development if it were all up to me, and I make persistent efforts to ensure people I’m interacting with know this. I also believe the things I advocate for would almost universally have a negative expected effect (if any effect) on the value of the equity I’m exposed to. But I don’t expect everyone to agree with this or to be reassured by it.

Join­ing the Carnegie En­dow­ment for In­ter­na­tional Peace

Good job op­por­tu­ni­ties for helping with the most im­por­tant century

We’re Not Ready: thoughts on “paus­ing” and re­spon­si­ble scal­ing policies

• Holden Karnofsky 10 Oct 2023 14:38 UTC

  5 points

  0 ∶ 0

  in reply to: Zeek’s comment on: This Can’t Go On

  Digital content requires physical space too, just relatively small amounts. E.g., physical resources/​atoms are needed to make the calculations associated with digital interactions. At some point the number of digital interactions will be capped, and the question will be how much they can be made better and better. More on the latter here: https://​​www.cold-takes.com/​​more-on-multiple-world-size-economies-per-atom/​​

3 lev­els of threat obfuscation

• Holden Karnofsky 10 Jun 2023 6:21 UTC

  8 points

  1 ∶ 2

  in reply to: Sol3:2’s comment on: A Play­book for AI Risk Re­duc­tion (fo­cused on mis­al­igned AI)

  I expected readers to assume that my wife owned significant equity in Anthropic; I’ve now edited the post to state this explicitly (and also added a mention of her OpenAI equity, which I should’ve included before and have included in the past). I don’t plan to disclose the exact amount and don’t think this is needed for readers to have sufficient context on my statements here.

• Holden Karnofsky 10 Jun 2023 6:16 UTC

  12 points

  2 ∶ 0

  in reply to: finnhambly’s comment on: A Play­book for AI Risk Re­duc­tion (fo­cused on mis­al­igned AI)

  Sorry, I didn’t mean to dismiss the importance of the conflict of interest or say it isn’t affecting my views.

  I’ve sometimes seen people reason along the lines of “Since Holden is married to Daniela, this must mean he agrees with Anthropic on specific issue X,” or “Since Holden is married to Daniela, this must mean that he endorses taking a job at Anthropic in specific case Y.” I think this kind of reasoning is unreliable and has been incorrect in more than one specific case. That’s what I intended to push back against.

A Play­book for AI Risk Re­duc­tion (fo­cused on mis­al­igned AI)

• Holden Karnofsky 4 Jun 2023 15:21 UTC

  3 points

  0 ∶ 0

  in reply to: Koen Holtman’s comment on: Seek­ing (Paid) Case Stud­ies on Standards

  Thanks! I’m looking for case studies that will be public; I’m agnostic about where they’re posted beyond that. We might consider requests to fund confidential case studies, but this project is meant to inform broader efforts, so confidential case studies would still need to be cleared for sharing with a reasonable set of people, and the funding bar would be higher.

• Holden Karnofsky 2 Jun 2023 22:51 UTC

  2 points

  0 ∶ 0

  in reply to: DanielFilan’s comment on: Tak­ing a leave of ab­sence from Open Philan­thropy to work on AI safety

  I think this was a goof due to there being a separate hardcover version, which has now been removed—try again?

Seek­ing (Paid) Case Stud­ies on Standards

• Holden Karnofsky 23 Mar 2023 21:44 UTC

  2 points

  0 ∶ 0

  in reply to: Arden Koehler’s comment on: My takes on the FTX situ­a­tion will (mostly) be cold, not hot

  To give a rough idea, I basically mean anyone who is likely to harm those around them (using a common-sense idea of doing harm) and/​or “pollute the commons” by having an outsized and non-consultative negative impact on community dynamics. It’s debatable what the best warning signs are and how reliable they are.

• Holden Karnofsky 22 Mar 2023 22:44 UTC

  83 points

  7 ∶ 3

  on: Time Ar­ti­cle Dis­cus­sion—“Effec­tive Altru­ist Lead­ers Were Re­peat­edly Warned About Sam Bankman-Fried Years Be­fore FTX Col­lapsed”

  Re: “In the weeks leading up to that April 2018 confrontation with Bankman-Fried and in the months that followed, Mac Aulay and others warned MacAskill, Beckstead and Karnofsky about her co-founder’s alleged duplicity and unscrupulous business ethics” -

  I don’t remember Tara reaching out about this, and I just searched my email for signs of this and didn’t see any. I’m not confident this didn’t happen, just noting that I can’t remember or easily find signs of it.

  In terms of what I knew/​learned 2018 more generally, I discuss that here.

  What links here?

  • Holden Karnofsky’s re­cent com­ments on FTX by Lizka (24 Mar 2023 11:44 UTC; 149 points)

• Holden Karnofsky 22 Mar 2023 22:40 UTC

  14 points

  4 ∶ 0

  in reply to: CuriousEA’s comment on: Tak­ing a leave of ab­sence from Open Philan­thropy to work on AI safety

  For context, my wife is the President and co-founder of Anthropic, and formerly worked at OpenAI.

  80% of her equity in Anthropic is (not legally bindingly) pledged for donation. None of her equity in OpenAI is. She may pledge more in the future if there is a tangible compelling reason to do so.

  I plan to be highly transparent about my conflict of interest, e.g. I regularly open meetings by disclosing it if I’m not sure the other person already knows about it, and I’ve often mentioned it when discussing related topics on Cold Takes.

  I also plan to discuss the implications of my conflict of interest for any formal role I might take. It’s possible that my role in helping with safety standards will be limited to advising with no formal powers (it’s even possible that I’ll decide I simply can’t work in this area due to the conflict of interest, and will pursue one of the other interventions I’ve thought about).

  But right now I’m just exploring options and giving non-authoritative advice, and that seems appropriate. (I’ll also note that I expect a lot of advice and opinions on standards to come from people who are directly employed by AI companies; while this does present a conflict of interest, and a more direct one than mine, I think it doesn’t and can’t mean they are excluded from relevant conversations.)

  What links here?

  • Zach Stein-Perlman's comment on An­thropic lead­er­ship conversation by Zach Stein-Perlman (LessWrong; 22 Dec 2024 15:30 UTC; 14 points)
  • Lukas Finnveden's comment on Front-Load Giv­ing Be­cause of An­thropic Donors? by jefftk (LessWrong; 6 Dec 2025 0:36 UTC; 8 points)

• Holden Karnofsky 22 Mar 2023 22:37 UTC

  32 points

  6 ∶ 0

  in reply to: Ozzie Gooen’s comment on: Some com­ments on re­cent FTX-re­lated events

  There was no one with official responsibility for the relationship between FTX and the EA community. I think the main reason the two were associated was via FTX’s/​Sam having a high profile and talking a lot about EA—that’s not something anyone else was able to control. (Some folks did ask him to do less of this.)

  It’s also worth noting that we generally try to be cautious about power dynamics as a funder, which means we are hesitant to be pushy about most matters. In particular, I think one of two major funders in this space attacking the other, nudging grantees to avoid association and funding from it, etc. would’ve been seen as strangely territorial behavior absent very strong evidence of misconduct.

  That said: as mentioned in another comment, with the benefit of hindsight, I wish I’d reasoned more like this: “This person is becoming very associated with effective altruism, so whether or not that’s due to anything I’ve done, it’s important to figure out whether that’s a bad thing and whether proactive distancing is needed.”

  What links here?

  • Holden Karnofsky’s re­cent com­ments on FTX by Lizka (24 Mar 2023 11:44 UTC; 149 points)

• Holden Karnofsky 22 Mar 2023 22:35 UTC

  28 points

  4 ∶ 1

  in reply to: Sharmake’s comment on: Some com­ments on re­cent FTX-re­lated events

  In 2018, I heard accusations that Sam had communicated in ways that left people confused or misled, though often with some ambiguity about whether Sam had been confused himself, had been inadvertently misleading while factually accurate, etc. I put some effort into understanding these concerns (but didn’t spend a ton of time on it; Open Phil didn’t have a relationship with Sam or Alameda).

  I didn’t hear anything that sounded anywhere near as bad as what has since come out about his behavior at FTX. At the time I didn’t feel my concerns rose to the level where it would be appropriate or fair to publicly attack or condemn him. The whole situation did make me vaguely nervous, and I spoke with some people about it privately, but I never came to a conclusion that there was a clearly warranted (public) action.

  What links here?

  • Holden Karnofsky’s re­cent com­ments on FTX by Lizka (24 Mar 2023 11:44 UTC; 149 points)
  • Holden Karnofsky's comment on Time Ar­ti­cle Dis­cus­sion—“Effec­tive Altru­ist Lead­ers Were Re­peat­edly Warned About Sam Bankman-Fried Years Be­fore FTX Col­lapsed” by Nathan Young (22 Mar 2023 22:44 UTC; 83 points)