I’m a bit confused about people suggesting this is defendable.
”I’m not the expert on effective altruism. I don’t identify with that terminology. My impression is that it’s a bit of an outdated term”.
There are three statements here
1. I’m not the expert on effective altruism—Its hard to see this as anything other than a lie. She’s married to Holden Karnofsky and knows ALL about Effective Altruism. She would probably destroy me on a “Do you understand EA” quiz.… I wonder how@Holden Karnofskyfeels about this?
2. I don’t identify with that terminology. - yes trueat least now! Maybe she’s still got some residual warmth for us deep in her heart?
3.My impression is that it’s a bit of an outdated term”. - Her husband set up 2 of the biggest EA (or heavily EA based) institutions that are still going strong today. On what planet is it an “outdated” term? Perhaps on the planet where your main goal is growing and defending your company?
In addition to the clear associations from the OP, from Their wedding page 2017 seemingly written by Daniela “We are both excited about effective altruism: using evidence and reason to figure out how to benefit others as much as possible, and taking action on that basis. For gifts we’re asking for donations to charities recommended by GiveWell, an organization Holden co-founded.”
If you want to distance yourself from EA, do it and be honest. If you’d rather not comment, don’t comment. But don’t obfuscate and lie pretending you don’t know about EA and downplay the movement.
I’m all for giving people the benefit of the doubt, but there doesn’t seem to be reasonable doubt here.
I don’t love raising this as its largely speculation on my part, but there might still be a undercurrent of copium within the EA community by people who backed, or still back Anthropic as the “best” of the AI acceleration bunch (which they quite possibly are) and want to hold that close after failing with Open AI…
Everything you say is correct I think, but I think in more normal circles, pointing out the inconsistency between someone’s wedding page and their corporate PR bullshit would seem a bit weird and obsessive and mean. I don’t find it so, but I think ordinary people would get a bad vibe from it.
That’s interesting I think I might move in different circles. Most people I know would not really understand the concept of there being a PR world where your present different things from your personal life
Perhaps you move in more corporate or higher flying circles where this kind of disconnect is normal and where its fine to have a public/private communication disconnect which is considered rude to challenge? Interesting!
fwiw I think in any circle I’ve been a part of critiquing someone publicly based on their wedding website would be considered weird/a low blow. (Including corporate circles.) [1]
I think there is a level of influence at which everything becomes fair game, e.g. Donald Trump can’t really expect a public/private communication disconnect. I don’t think that’s true of Daniela, although I concede that her influence over the light cone might not actually be that much lower than Trump’s.
Wow again I just haven’t moved in circles where this would even be considered. Only the most elite 0.1 percent of people can even have a meaningful “public private disconnect” as you have to have quite a prominent public profile for that to even be an issue. Although we all have a “public profile” in theory, very few people are famous/powerful enough for it to count.
I don’t think I believe in a public/private disconnect but I’ll think about it some more. I believe in integrity and honesty in most situations, especially when your are publicly disparaging a movement. If you have chosen to lie and smear a movement with”My impression is that it’s a bit of an outdated term” then I think this makes what you say a bit more fair game than for other statements where you aren’t low-key attacking a group of well meaning people.
Only the most elite 0.1 percent of people can even have a meaningful “public private disconnect” as you have to have quite a prominent public profile for that to even be an issue.
Hmm yeah, that’s kinda my point? Like complaining about your annoying coworker anonymously online is fine, but making a public blog post like “my coworker Jane Doe sucks for these reasons” would be weird, people get fired for stuff like that. And referencing their wedding website would be even more extreme.
(Of course, most people’s coworkers aren’t trying to reshape the lightcone without public consent so idk, maybe different standards should apply here. I can tell you that a non-trivial number of people I’ve wanted to hire for leadership positions in EA have declined for reasons like “I don’t want people critiquing my personal life on the EA Forum” though.)
No one is critiquing Daniela’s personal life though, they’re critiquing something about her public life (ie her voluntary public statements to journalists) for contradicting what she’s said in her personal life. Compare this with a common reason people get cancelled where the critique is that there’s something bad in their personal life, and people are disappointed that the personal life doesn’t reflect the public persona- in this case it’s the other way around.
most people’s coworkers aren’t trying to reshape the lightcone without public consent so idk, maybe different standards should apply here
Exactly. Daniela and the senior leadership at one of the frontier AI labs are not the same as someone’s random office colleague. There’s a clear public interest angle here in terms of understanding the political and social affiliations of powerful and influential people—which is simply absent in the case you describe.
That’s interesting and I’m sad to hear about people declining jobs due those reasons. On the other hand though some leadership jobs might not be the right job fit if they’re not up for that kind of critique. I would imagine though there are a bunch of ways to avoid the “EA limelight” for many positions though, of course not public facing ones.
Slight quibble though I would consider “Jane Doe sucks for these reasons” an order of magnitude more objectionable than quoting a wedding website to make a point. Maybe wedding website are sacrosanct in a way in missing tho...
the other hand though some leadership jobs might not be the right job fit if they’re not up for that kind of critique
Yeah, this used to be my take but a few iterations of trying to hire for jobs which exclude shy awkward nerds from consideration when the EA candidate pool consists almost entirely of shy awkward nerds has made the cost of this approach quite salient to me.
Agree with Ben that this makes it harder to find folks for leadership positions.
In addition to excluding shy awkward nerds, you’re also actively selecting for a bunch of personality traits, not all of which are unalloyed positives.
By analogy, I think there’s a very strong argument that very high levels of scrutiny are fair game for politicians but I’m not particularly thrilled with what that does to our candidate pool.
(I don’t know of a great way to resolve this tension.)
There’s already been much critique of your argument here, but I will just say that by the “level of influence” metric, Daniela shoots it out of the park compared to Donald Trump. I think it is entirely uncontroversial and perhaps an understatement to claim the world as a whole and EA in particular has a right to know & discuss pretty much every fact about the personal, professional, social, and philosophical lives of the group of people who, by their own admission, are literally creating God. And are likely to be elevated to a permanent place of power & control over the universe for all of eternity.
Such a position should not be a pleasurable job with no repercussions on the level of privacy or degree of public scrutiny on your personal life. If you are among this group, and this level of scrutiny disturbs you, perhaps you shouldn’t be trying to “reshape the lightcone without public consent” or knowledge.
I agree with this being weird / a low blow in general, but not in this particular case. The crux with your footnote may be that I see this as more than a continuum.
I think someone’s interest in private communications becomes significantly weaker as they assume a position of great power over others, conditioned on the subject matter of the communication being a matter of meaningful public interest. Here, I think an AI executive’s perspective on EA is a matter of significant public interest.
Second, I do not find a wedding website to be a particularly private form of communication compared to (e.g.) a private conversation with a romantic partner. Audience in the hundreds, no strong confidentiality commitment, no precautions to prevent public access.
The more power the individual has over others, the wider the scope of topics that are of legitimate public interest for the others to bring up and the narrower the scope of communications that citing would be a weird / low. So what applies to major corporate CEOs with significant influence over the future would not generally apply to most people.
Compare this to paparazzi, who hound celebrities (who do not possess CEO-level power) for material that is not of legitimate public interest, and often under circumstances in which society recognizes particularly strong privacy rights.
I’m reminded of the NBA basketball-team owner who made some racist basketball-related comments to his affair partner, who leaked them. My recollection is that people threw shade on the affair partner (who arguably betrayed his confidences), but few people complained about showering hundreds of millions of dollars worth of tax consequences on the owner by forcing the sale of his team against his will. Unlike comments to a medium-size audience on a website, the owner’s comments were particularly private (to an intimate figure, 1:1, protected from non-consensual recording by criminal law).
I think you shouldn’t assume that people are “experts” on something just because they’re married to someone who is an expert, even when (like Daniela) they’re smart and successful.
I agree that these statements are not defensible. I’m sad to see it. There’s maybe some hope that the person making these statements was just caught off guard and it’s not a common pattern at Antrhopic to obfuscate things with that sort of misdirection. (Edit: Or maybe the journalist was fishing for quotes and made it seem like they were being more evasive than they actually were.)
I don’t get why they can’t just admit that Anthropic’s history is pretty intertwined with EA history. They could still distance themselves from “EA as the general public perceives it” or even “EA-as-it-is-now.”
For instance, they could flag that EA maybe has a bit of a problem with “purism”—like, some vocal EAs in this comment section and elsewhere seem to think it is super obvious that Anthropic has been selling out/became too much of a typical for-profit corporation. I didn’t myself think that this was necessarily the case because I see a lot of valid tradeoffs that Anthropic leadership is having to navigate, and the armchair quarterbacks EAs seem to be failing to take that into account? However, the communications highlighted in the OP made me update that Anthropic leadership probably does lack the integrity needed to do complicated power-seeking stuff that has the potential to corrupt. (If someone can handle the temptions from power, they should at the very least be able to handle the comparatively easy dynamics of don’t willingly distort the truth as you know it.)
Anthropic leadership probably does lack the integrity needed to do complicated power-seeking stuff that has the potential to corrupt.
Yes. It’s sad to see, but Anthropic is going the same way as OpenAI, despite being founded by a group that split from OpenAI over safety concerns. Power (and money) corrupts. How long until another group splits from Anthropic and the process repeats? Or actually, one can hope that such a group splitting from Anthropic might actually have integrity and instead work on trying to stop the race.
What surprises me about this whole situation is that people seem surprised at the executive leadership at a corporation worth an estimated $61.5B would engage in big-corporation PR-speak. The base rate for big-corporation execs engaging in such conduct in their official capacities seems awfully close to 100%. Hence, it does not feel like anything to update on for me.
I’m getting the sense that a decent number of people assume that being “EA aligned” is somehow a strong inoculant against the temptations of money and power. Arguably the FTX scandal—which after all involved multiple EAs, not just SBF—should have already caused people to update on how effective said inoculant is, at least when billions of dollars were floating around.[1]
This is not to suggest that most EAs would act in fraudulent ways if surrounded by billions of dollars, but it does provide evidence that EAs are not super-especially resistant to the corrosive effects of money and power at that level of concentration. FTX was only one cluster of people, but how many people have been EAs first and then been exposed to the amount of money/power that FTX or Anthropic had/have?
What surprises me about this whole situation is that people seem surprised at the executive leadership at a corporation worth an estimated $61.5B would engage in big-corporation PR-speak. The base rate for big-corporation execs engaging in such conduct in their official capacities seems awfully close to 100%.
Hm, good point. This gives me pause, but I’m not sure what direction to update in. Like, maybe I should update “corporate speak is just what these large orgs do and it’s more like a fashion thing than a signal of their (lack of) integrity on things that matter most.” Or maybe I should update in the direction you suggest, namely “if an org grows too much, it’s unlikely to stay aligned with its founding character principles.”
I’m getting the sense that a decent number of people assume that being “EA aligned” is somehow a strong inoculant against the temptations of money and power.
I would have certainly thought so. If anything can be an inoculant against those temptations, surely a strong adherence to a cause greater than oneself packaged in lots warnings against biases and other ways humans can go wrong (as is the common message in EA and rationalist circles) seems like the best hope for it? If you don’t think it can be a strong inoculant, that makes you pretty cynical, no? (I think cynicism is often right, so this isn’t automatically a rejection of your position. I just want to flag that yours is a claim with quite strong implications on its own.)
Arguably the FTX scandal—which after all involved multiple EAs, not just SBF—should have already caused people to update on how effective said inoculant is, at least when billions of dollars were floating around.
If you were just talking about SBF, then I’d say your point is weak because he probably wasn’t low on dark triad traits to start out with. But you emphasizing how other EAs around him were also involved (the direct co-conspirators at Alameda and FTX) is a strong point.
Still, in my mind this would probably have gone very differently with the same group of people minus SBF and with a leader with a stronger commitment and psychological disposition towards honesty. (I should flag that parts of Caroline Ellison’s blog also gave me vibes of “seems to like having power too much”—but at least it’s more common for young people to later change/grow.) That’s why I don’t consider it a huge update for “power corrupts”. To me, it’s a reinforcement of “it matters to have good leadership.”
My worldview(?) is that “power corrupts” doesn’t apply equally to every leader and that we’d be admitting defeat straight away if we stopped trying to do ambitious things. There doesn’t seem to be a great way to do targeted ambitious things without some individual acquiring high amounts of power in the process.(?) We urgently need to do a better job at preventing that those who end up with a lot of power are almost always those with kind of shady character. The fact that we’re so bad at this suggests that these people are advantaged at some aspects of ambitious leadership, which makes the whole thing a lot harder. But that doesn’t mean it’s impossible.
I concede that there’s a sense in which this worldview of mine is not grounded in empiricism—I haven’t even looked into the matter from that perspective. Instead, it’s more like a commitment to a wager: “If this doesn’t work, what else are we supposed to do?” I’m not interested in concluding that the best we can do is criticise the powers that be from the sidelines.
Of course, if leaders exhibit signs of low integrity, like in this example of Anthropic’s communications, it’s important not to let this slide. The thing I want to push back against is an attitude of “person x or org y has acquired so much power, surely that means that they’re now corrupted,” and this leading to no longer giving them the benefit of the doubt/not trying to see the complexities of their situation when they do something that looks surprising/disappointing/suboptimal. With great power comes great responsiblity, including a responsibility to not mess up your potential for doing even more good later on. Naturally, this does come with lots of tradeoffs and it’s not always easy to infer from publicly visible actions and statements whether an org is still culturally on track. (That said, I concede that you can often tell quite a lot about someone’s character/an org’s culture based on how/whether they communicate nuances, which is sadly why I’ve had some repeated negative updates about Anthropic lately.)
When I speak of a strong inoculant, I mean something that is very effective in preventing the harm in question—such as the measles vaccine. Unless there were a measles case at my son’s daycare, or a family member were extremely vulnerable to measles, the protection provided by the strong inoculant is enough that I can carry on with life without thinking about measles.
In contrast, the influenza vaccine is a weak inoculant—I definitely get vaccinated because I’ll get infected less and hospitalized less without it. But I’m not surprised when I get the flu. If I were at great risk of serious complications from the flu, then I’d only use vaccination as one layer of my mitigation strategy (and without placing undue reliance on it.) And of course there are strengths in between those two.
I’d call myself moderately cynical. I think history teaches us that the corrupting influence of power is strong and that managing this risk has been a struggle. I don’t think I need to take the position that no strong inoculant exists. It is enough to assert that—based on centuries of human experience across cultures—our starting point should be that inoculants as weak until proven otherwise by sufficient experience. And when one of the star pupils goes so badly off the rails, along with several others in his orbit, that adds to the quantum of evidence I think is necessary to overcome the general rule.
I’d add that one of the traditional ways to mitigate this risk is to observe the candidate over a long period of time in conjunction with lesser levels of power. Although it doesn’t always work well in practice, you do get some ability to measure the specific candidate’s susceptibility in lower-stakes situations. It may not be popular to say, but we just won’t have had the same potential to observe people in their 20s and 30s in intermediate-power situations that we often will have had for the 50+ crowd. Certainly people can and do fake being relatively unaffected by money and power for many years, but it’s harder to pull off than for a shorter period of time.
If anything can be an inoculant against those temptations, surely a strong adherence to a cause greater than oneself packaged in lots warnings against biases and other ways humans can go wrong (as is the common message in EA and rationalist circles) seems like the best hope for it?
Maybe. But on first principles, one might have also thought that belief in an all-powerful, all-knowing deity who will hammer you if you fall out of line would be a fairly strong inoculant. But experience teaches us that this is not so!
Also, if I had to design a practical philosophy that was maximally resistant to corruption, I’d probably ground it on virtue ethics or deontology rather than give so much weight to utilitarian considerations. The risk of the newly-powerful person deceiving themselves may be greater for a utilitarian.
--
As you imply, the follow-up question is where we go from here. I think there are three possible approaches to dealing with a weak or moderate-strength inoculant:
In some cases, a sober understanding of how strong or weak the inoculant is should lead to a decision not to proceed with a project at all.
In other cases, a sober understanding of the inoculant affects how we should weight further measures to mitigate the risk of corrupting influence versus maximizing effectiveness.
For instance, I think you’re onto something with “these people are advantaged at some aspects of ambitious leadership.” If I’m permitted a literary analogy, one could assign more weight to how much a would-be powerholder has The Spirit of Frodo in deciding who to entrust with great power. Gandalf tells us that Bilbo (and thus Frodo) were meant to have the ring, and not by its maker. The problem is that Frodo would probably make a lousy CEO in a competitive, fast-moving market, and I’m not sure you can address that without also removing something of what makes him best-suited to bear the Ring.
In still other cases, there isn’t a good alternative and there aren’t viable mitigating factors. But acknowledging the risk that is being taken is still important; it ensures we are accounting for all the risks, reminds us to prepare contingency plans, and so on.
My point is that doing these steps well requires a reasonably accurate view of inoculant strength. And I got the sense that the community is more confident in EA-as-inoculant than the combination of general human experience and the limited available evidence on EA-as-inoculant warrants.
Are you saying bc it’s not “surprising” it should be allowed? This rhetorical move of shaming your opponent for not having already gotten used to and therefore tolerating someone doing bad things I always find bizarre.
No—that something is unsurprising, even readily predictable, does not imply anything about whether it is OK.
The fact that people seem surprised by the presence of corpspeak here does make me concerned that they may have been looking at the world with an assumption that “aligned” people are particularly resistant to the corrosive effects of money and power. That, in my opinion, is a dangerous assumption to make—and is not one I would find well-supported by the available evidence. Our models of the world should assume that at least the significant majority of people will be adversely and materially influenced by exposure to high concentrations of money and power, and we need to plan accordingly.
Convincing such people that Anthropic is doing corpspeak and not just being perfectly reasonable or justified by 3D chess (with ultimate EA goals) would be a lot of progress...
It’s a huge problem in EA that people don’t take CoI that seriously as something that affect their thinking. They think they can solve every problem explicitly intellectually so corruption by money won’t happen to them.
Yeah this is pretty damning. At this point, being pro-Anthropic is like being pro-FTX in early 2022.
(EDITED to add, following David Manheim’s recommendation: downvoters and disagree-voters note: there was actually far less publicly available information to update on FTX being bad in early 2022.)
I don’t think it is like being pro-FTX in early 2022
1) Back then hardly anyone knew about the FTX issues. Here we’re discussing issues where there is a lot of public information 2) SBF was hiding a mass fraud that was clearly both illegal and immorral. Here we are not discussing illegailities or fraud, but whether a company is being properly honest, transparent and safe? 3) SBF was a promotor of EA and to some degree held up on an EA pedestal. Here Anthropic is the opposite, trying to distance themselves from the movement.
Fair points. I was more thinking in broad terms of supporting something that will most likely turn out hugely negative. I think it’s pretty clear already that Anthropic is massively negative expected value for the future of humanity. And we’ve already got the precedent of OpenAI and how that’s gone (and Anthropic seems to be going the same way in broad terms—i.e. not caring about endangering 8 billion people’s lives with reckless AGI/ASI development).
I just think Anthropic leaders being un-candid about their connection to EA is pretty weak evidence that they’re doing fraud or something like what FTX did. (It’s positive evidence, but weak.)
I was thinking less in terms of fraud/illegality, and more in terms of immorality/negative externalities (i.e. they are recklessly endangering everyone’s lives).
I’m a bit confused about people suggesting this is defendable.
”I’m not the expert on effective altruism. I don’t identify with that terminology. My impression is that it’s a bit of an outdated term”.
There are three statements here
1. I’m not the expert on effective altruism—Its hard to see this as anything other than a lie. She’s married to Holden Karnofsky and knows ALL about Effective Altruism. She would probably destroy me on a “Do you understand EA” quiz.… I wonder how @Holden Karnofsky feels about this?
2. I don’t identify with that terminology. - yes true at least now! Maybe she’s still got some residual warmth for us deep in her heart?
3. My impression is that it’s a bit of an outdated term”. - Her husband set up 2 of the biggest EA (or heavily EA based) institutions that are still going strong today. On what planet is it an “outdated” term? Perhaps on the planet where your main goal is growing and defending your company?
In addition to the clear associations from the OP, from Their wedding page 2017 seemingly written by Daniela “We are both excited about effective altruism: using evidence and reason to figure out how to benefit others as much as possible, and taking action on that basis. For gifts we’re asking for donations to charities recommended by GiveWell, an organization Holden co-founded.”
If you want to distance yourself from EA, do it and be honest. If you’d rather not comment, don’t comment. But don’t obfuscate and lie pretending you don’t know about EA and downplay the movement.
I’m all for giving people the benefit of the doubt, but there doesn’t seem to be reasonable doubt here.
I don’t love raising this as its largely speculation on my part, but there might still be a undercurrent of copium within the EA community by people who backed, or still back Anthropic as the “best” of the AI acceleration bunch (which they quite possibly are) and want to hold that close after failing with Open AI…
Everything you say is correct I think, but I think in more normal circles, pointing out the inconsistency between someone’s wedding page and their corporate PR bullshit would seem a bit weird and obsessive and mean. I don’t find it so, but I think ordinary people would get a bad vibe from it.
That’s interesting I think I might move in different circles. Most people I know would not really understand the concept of there being a PR world where your present different things from your personal life
Perhaps you move in more corporate or higher flying circles where this kind of disconnect is normal and where its fine to have a public/private communication disconnect which is considered rude to challenge? Interesting!
fwiw I think in any circle I’ve been a part of critiquing someone publicly based on their wedding website would be considered weird/a low blow. (Including corporate circles.) [1]
I think there is a level of influence at which everything becomes fair game, e.g. Donald Trump can’t really expect a public/private communication disconnect. I don’t think that’s true of Daniela, although I concede that her influence over the light cone might not actually be that much lower than Trump’s.
Wow again I just haven’t moved in circles where this would even be considered. Only the most elite 0.1 percent of people can even have a meaningful “public private disconnect” as you have to have quite a prominent public profile for that to even be an issue. Although we all have a “public profile” in theory, very few people are famous/powerful enough for it to count.
I don’t think I believe in a public/private disconnect but I’ll think about it some more. I believe in integrity and honesty in most situations, especially when your are publicly disparaging a movement. If you have chosen to lie and smear a movement with”My impression is that it’s a bit of an outdated term” then I think this makes what you say a bit more fair game than for other statements where you aren’t low-key attacking a group of well meaning people.
Hmm yeah, that’s kinda my point? Like complaining about your annoying coworker anonymously online is fine, but making a public blog post like “my coworker Jane Doe sucks for these reasons” would be weird, people get fired for stuff like that. And referencing their wedding website would be even more extreme.
(Of course, most people’s coworkers aren’t trying to reshape the lightcone without public consent so idk, maybe different standards should apply here. I can tell you that a non-trivial number of people I’ve wanted to hire for leadership positions in EA have declined for reasons like “I don’t want people critiquing my personal life on the EA Forum” though.)
No one is critiquing Daniela’s personal life though, they’re critiquing something about her public life (ie her voluntary public statements to journalists) for contradicting what she’s said in her personal life. Compare this with a common reason people get cancelled where the critique is that there’s something bad in their personal life, and people are disappointed that the personal life doesn’t reflect the public persona- in this case it’s the other way around.
Exactly. Daniela and the senior leadership at one of the frontier AI labs are not the same as someone’s random office colleague. There’s a clear public interest angle here in terms of understanding the political and social affiliations of powerful and influential people—which is simply absent in the case you describe.
That’s interesting and I’m sad to hear about people declining jobs due those reasons. On the other hand though some leadership jobs might not be the right job fit if they’re not up for that kind of critique. I would imagine though there are a bunch of ways to avoid the “EA limelight” for many positions though, of course not public facing ones.
Slight quibble though I would consider “Jane Doe sucks for these reasons” an order of magnitude more objectionable than quoting a wedding website to make a point. Maybe wedding website are sacrosanct in a way in missing tho...
Yeah, this used to be my take but a few iterations of trying to hire for jobs which exclude shy awkward nerds from consideration when the EA candidate pool consists almost entirely of shy awkward nerds has made the cost of this approach quite salient to me.
There are trade-offs to everything 🤷♂️
Agree with Ben that this makes it harder to find folks for leadership positions.
In addition to excluding shy awkward nerds, you’re also actively selecting for a bunch of personality traits, not all of which are unalloyed positives.
By analogy, I think there’s a very strong argument that very high levels of scrutiny are fair game for politicians but I’m not particularly thrilled with what that does to our candidate pool.
(I don’t know of a great way to resolve this tension.)
100 percent man
There’s already been much critique of your argument here, but I will just say that by the “level of influence” metric, Daniela shoots it out of the park compared to Donald Trump. I think it is entirely uncontroversial and perhaps an understatement to claim the world as a whole and EA in particular has a right to know & discuss pretty much every fact about the personal, professional, social, and philosophical lives of the group of people who, by their own admission, are literally creating God. And are likely to be elevated to a permanent place of power & control over the universe for all of eternity.
Such a position should not be a pleasurable job with no repercussions on the level of privacy or degree of public scrutiny on your personal life. If you are among this group, and this level of scrutiny disturbs you, perhaps you shouldn’t be trying to “reshape the lightcone without public consent” or knowledge.
I agree with this being weird / a low blow in general, but not in this particular case. The crux with your footnote may be that I see this as more than a continuum.
I think someone’s interest in private communications becomes significantly weaker as they assume a position of great power over others, conditioned on the subject matter of the communication being a matter of meaningful public interest. Here, I think an AI executive’s perspective on EA is a matter of significant public interest.
Second, I do not find a wedding website to be a particularly private form of communication compared to (e.g.) a private conversation with a romantic partner. Audience in the hundreds, no strong confidentiality commitment, no precautions to prevent public access.
The more power the individual has over others, the wider the scope of topics that are of legitimate public interest for the others to bring up and the narrower the scope of communications that citing would be a weird / low. So what applies to major corporate CEOs with significant influence over the future would not generally apply to most people.
Compare this to paparazzi, who hound celebrities (who do not possess CEO-level power) for material that is not of legitimate public interest, and often under circumstances in which society recognizes particularly strong privacy rights.
I’m reminded of the NBA basketball-team owner who made some racist basketball-related comments to his affair partner, who leaked them. My recollection is that people threw shade on the affair partner (who arguably betrayed his confidences), but few people complained about showering hundreds of millions of dollars worth of tax consequences on the owner by forcing the sale of his team against his will. Unlike comments to a medium-size audience on a website, the owner’s comments were particularly private (to an intimate figure, 1:1, protected from non-consensual recording by criminal law).
No, I don’t move in corporate circles.
Not when the issue is “knowledge of and identification with something by name”
I think you shouldn’t assume that people are “experts” on something just because they’re married to someone who is an expert, even when (like Daniela) they’re smart and successful.
Well it’s not really an assumption, is it? We have very good reason to think she’s downplaying her knowledge.
I agree that these statements are not defensible. I’m sad to see it. There’s maybe some hope that the person making these statements was just caught off guard and it’s not a common pattern at Antrhopic to obfuscate things with that sort of misdirection. (Edit: Or maybe the journalist was fishing for quotes and made it seem like they were being more evasive than they actually were.)
I don’t get why they can’t just admit that Anthropic’s history is pretty intertwined with EA history. They could still distance themselves from “EA as the general public perceives it” or even “EA-as-it-is-now.”
For instance, they could flag that EA maybe has a bit of a problem with “purism”—like, some vocal EAs in this comment section and elsewhere seem to think it is super obvious that Anthropic has been selling out/became too much of a typical for-profit corporation. I didn’t myself think that this was necessarily the case because I see a lot of valid tradeoffs that Anthropic leadership is having to navigate, and the armchair quarterbacks EAs seem to be failing to take that into account? However, the communications highlighted in the OP made me update that Anthropic leadership probably does lack the integrity needed to do complicated power-seeking stuff that has the potential to corrupt. (If someone can handle the temptions from power, they should at the very least be able to handle the comparatively easy dynamics of don’t willingly distort the truth as you know it.)
Yes. It’s sad to see, but Anthropic is going the same way as OpenAI, despite being founded by a group that split from OpenAI over safety concerns. Power (and money) corrupts. How long until another group splits from Anthropic and the process repeats? Or actually, one can hope that such a group splitting from Anthropic might actually have integrity and instead work on trying to stop the race.
What surprises me about this whole situation is that people seem surprised at the executive leadership at a corporation worth an estimated $61.5B would engage in big-corporation PR-speak. The base rate for big-corporation execs engaging in such conduct in their official capacities seems awfully close to 100%. Hence, it does not feel like anything to update on for me.
I’m getting the sense that a decent number of people assume that being “EA aligned” is somehow a strong inoculant against the temptations of money and power. Arguably the FTX scandal—which after all involved multiple EAs, not just SBF—should have already caused people to update on how effective said inoculant is, at least when billions of dollars were floating around.[1]
This is not to suggest that most EAs would act in fraudulent ways if surrounded by billions of dollars, but it does provide evidence that EAs are not super-especially resistant to the corrosive effects of money and power at that level of concentration. FTX was only one cluster of people, but how many people have been EAs first and then been exposed to the amount of money/power that FTX or Anthropic had/have?
(I know I’m late again replying to this thread.)
Hm, good point. This gives me pause, but I’m not sure what direction to update in. Like, maybe I should update “corporate speak is just what these large orgs do and it’s more like a fashion thing than a signal of their (lack of) integrity on things that matter most.” Or maybe I should update in the direction you suggest, namely “if an org grows too much, it’s unlikely to stay aligned with its founding character principles.”
I would have certainly thought so. If anything can be an inoculant against those temptations, surely a strong adherence to a cause greater than oneself packaged in lots warnings against biases and other ways humans can go wrong (as is the common message in EA and rationalist circles) seems like the best hope for it? If you don’t think it can be a strong inoculant, that makes you pretty cynical, no? (I think cynicism is often right, so this isn’t automatically a rejection of your position. I just want to flag that yours is a claim with quite strong implications on its own.)
If you were just talking about SBF, then I’d say your point is weak because he probably wasn’t low on dark triad traits to start out with. But you emphasizing how other EAs around him were also involved (the direct co-conspirators at Alameda and FTX) is a strong point.
Still, in my mind this would probably have gone very differently with the same group of people minus SBF and with a leader with a stronger commitment and psychological disposition towards honesty. (I should flag that parts of Caroline Ellison’s blog also gave me vibes of “seems to like having power too much”—but at least it’s more common for young people to later change/grow.) That’s why I don’t consider it a huge update for “power corrupts”. To me, it’s a reinforcement of “it matters to have good leadership.”
My worldview(?) is that “power corrupts” doesn’t apply equally to every leader and that we’d be admitting defeat straight away if we stopped trying to do ambitious things. There doesn’t seem to be a great way to do targeted ambitious things without some individual acquiring high amounts of power in the process.(?) We urgently need to do a better job at preventing that those who end up with a lot of power are almost always those with kind of shady character. The fact that we’re so bad at this suggests that these people are advantaged at some aspects of ambitious leadership, which makes the whole thing a lot harder. But that doesn’t mean it’s impossible.
I concede that there’s a sense in which this worldview of mine is not grounded in empiricism—I haven’t even looked into the matter from that perspective. Instead, it’s more like a commitment to a wager: “If this doesn’t work, what else are we supposed to do?”
I’m not interested in concluding that the best we can do is criticise the powers that be from the sidelines.
Of course, if leaders exhibit signs of low integrity, like in this example of Anthropic’s communications, it’s important not to let this slide. The thing I want to push back against is an attitude of “person x or org y has acquired so much power, surely that means that they’re now corrupted,” and this leading to no longer giving them the benefit of the doubt/not trying to see the complexities of their situation when they do something that looks surprising/disappointing/suboptimal. With great power comes great responsiblity, including a responsibility to not mess up your potential for doing even more good later on. Naturally, this does come with lots of tradeoffs and it’s not always easy to infer from publicly visible actions and statements whether an org is still culturally on track. (That said, I concede that you can often tell quite a lot about someone’s character/an org’s culture based on how/whether they communicate nuances, which is sadly why I’ve had some repeated negative updates about Anthropic lately.)
When I speak of a strong inoculant, I mean something that is very effective in preventing the harm in question—such as the measles vaccine. Unless there were a measles case at my son’s daycare, or a family member were extremely vulnerable to measles, the protection provided by the strong inoculant is enough that I can carry on with life without thinking about measles.
In contrast, the influenza vaccine is a weak inoculant—I definitely get vaccinated because I’ll get infected less and hospitalized less without it. But I’m not surprised when I get the flu. If I were at great risk of serious complications from the flu, then I’d only use vaccination as one layer of my mitigation strategy (and without placing undue reliance on it.) And of course there are strengths in between those two.
I’d call myself moderately cynical. I think history teaches us that the corrupting influence of power is strong and that managing this risk has been a struggle. I don’t think I need to take the position that no strong inoculant exists. It is enough to assert that—based on centuries of human experience across cultures—our starting point should be that inoculants as weak until proven otherwise by sufficient experience. And when one of the star pupils goes so badly off the rails, along with several others in his orbit, that adds to the quantum of evidence I think is necessary to overcome the general rule.
I’d add that one of the traditional ways to mitigate this risk is to observe the candidate over a long period of time in conjunction with lesser levels of power. Although it doesn’t always work well in practice, you do get some ability to measure the specific candidate’s susceptibility in lower-stakes situations. It may not be popular to say, but we just won’t have had the same potential to observe people in their 20s and 30s in intermediate-power situations that we often will have had for the 50+ crowd. Certainly people can and do fake being relatively unaffected by money and power for many years, but it’s harder to pull off than for a shorter period of time.
Maybe. But on first principles, one might have also thought that belief in an all-powerful, all-knowing deity who will hammer you if you fall out of line would be a fairly strong inoculant. But experience teaches us that this is not so!
Also, if I had to design a practical philosophy that was maximally resistant to corruption, I’d probably ground it on virtue ethics or deontology rather than give so much weight to utilitarian considerations. The risk of the newly-powerful person deceiving themselves may be greater for a utilitarian.
--
As you imply, the follow-up question is where we go from here. I think there are three possible approaches to dealing with a weak or moderate-strength inoculant:
In some cases, a sober understanding of how strong or weak the inoculant is should lead to a decision not to proceed with a project at all.
In other cases, a sober understanding of the inoculant affects how we should weight further measures to mitigate the risk of corrupting influence versus maximizing effectiveness.
For instance, I think you’re onto something with “these people are advantaged at some aspects of ambitious leadership.” If I’m permitted a literary analogy, one could assign more weight to how much a would-be powerholder has The Spirit of Frodo in deciding who to entrust with great power. Gandalf tells us that Bilbo (and thus Frodo) were meant to have the ring, and not by its maker. The problem is that Frodo would probably make a lousy CEO in a competitive, fast-moving market, and I’m not sure you can address that without also removing something of what makes him best-suited to bear the Ring.
In still other cases, there isn’t a good alternative and there aren’t viable mitigating factors. But acknowledging the risk that is being taken is still important; it ensures we are accounting for all the risks, reminds us to prepare contingency plans, and so on.
My point is that doing these steps well requires a reasonably accurate view of inoculant strength. And I got the sense that the community is more confident in EA-as-inoculant than the combination of general human experience and the limited available evidence on EA-as-inoculant warrants.
Are you saying bc it’s not “surprising” it should be allowed? This rhetorical move of shaming your opponent for not having already gotten used to and therefore tolerating someone doing bad things I always find bizarre.
No—that something is unsurprising, even readily predictable, does not imply anything about whether it is OK.
The fact that people seem surprised by the presence of corpspeak here does make me concerned that they may have been looking at the world with an assumption that “aligned” people are particularly resistant to the corrosive effects of money and power. That, in my opinion, is a dangerous assumption to make—and is not one I would find well-supported by the available evidence. Our models of the world should assume that at least the significant majority of people will be adversely and materially influenced by exposure to high concentrations of money and power, and we need to plan accordingly.
Convincing such people that Anthropic is doing corpspeak and not just being perfectly reasonable or justified by 3D chess (with ultimate EA goals) would be a lot of progress...
It’s a huge problem in EA that people don’t take CoI that seriously as something that affect their thinking. They think they can solve every problem explicitly intellectually so corruption by money won’t happen to them.
Not sure if I misunderstand something but the wedding page seems from 2017? (It reads “October 21, 2017” at the top.)
Apologies corrected
Yeah this is pretty damning. At this point, being pro-Anthropic is like being pro-FTX in early 2022.
(EDITED to add, following David Manheim’s recommendation: downvoters and disagree-voters note: there was actually far less publicly available information to update on FTX being bad in early 2022.)
I don’t think it is like being pro-FTX in early 2022
1) Back then hardly anyone knew about the FTX issues. Here we’re discussing issues where there is a lot of public information
2) SBF was hiding a mass fraud that was clearly both illegal and immorral. Here we are not discussing illegailities or fraud, but whether a company is being properly honest, transparent and safe?
3) SBF was a promotor of EA and to some degree held up on an EA pedestal. Here Anthropic is the opposite, trying to distance themselves from the movement.
Seems very different to me.
Fair points. I was more thinking in broad terms of supporting something that will most likely turn out hugely negative. I think it’s pretty clear already that Anthropic is massively negative expected value for the future of humanity. And we’ve already got the precedent of OpenAI and how that’s gone (and Anthropic seems to be going the same way in broad terms—i.e. not caring about endangering 8 billion people’s lives with reckless AGI/ASI development).
I just think Anthropic leaders being un-candid about their connection to EA is pretty weak evidence that they’re doing fraud or something like what FTX did. (It’s positive evidence, but weak.)
I was thinking less in terms of fraud/illegality, and more in terms of immorality/negative externalities (i.e. they are recklessly endangering everyone’s lives).
I think almost nobody had the info needed to predict FTX besides the perpetrators. I think we already know all we need to oppose Anthropic.
Downvoters note: there was actually far less publicly available information to update on FTX being bad in early 2022.
You should add an edit to clarify the the claim, not just reply.