What surprises me about this whole situation is that people seem surprised at the executive leadership at a corporation worth an estimated $61.5B would engage in big-corporation PR-speak. The base rate for big-corporation execs engaging in such conduct in their official capacities seems awfully close to 100%. Hence, it does not feel like anything to update on for me.
I’m getting the sense that a decent number of people assume that being “EA aligned” is somehow a strong inoculant against the temptations of money and power. Arguably the FTX scandal—which after all involved multiple EAs, not just SBF—should have already caused people to update on how effective said inoculant is, at least when billions of dollars were floating around.[1]
This is not to suggest that most EAs would act in fraudulent ways if surrounded by billions of dollars, but it does provide evidence that EAs are not super-especially resistant to the corrosive effects of money and power at that level of concentration. FTX was only one cluster of people, but how many people have been EAs first and then been exposed to the amount of money/power that FTX or Anthropic had/have?
What surprises me about this whole situation is that people seem surprised at the executive leadership at a corporation worth an estimated $61.5B would engage in big-corporation PR-speak. The base rate for big-corporation execs engaging in such conduct in their official capacities seems awfully close to 100%.
Hm, good point. This gives me pause, but I’m not sure what direction to update in. Like, maybe I should update “corporate speak is just what these large orgs do and it’s more like a fashion thing than a signal of their (lack of) integrity on things that matter most.” Or maybe I should update in the direction you suggest, namely “if an org grows too much, it’s unlikely to stay aligned with its founding character principles.”
I’m getting the sense that a decent number of people assume that being “EA aligned” is somehow a strong inoculant against the temptations of money and power.
I would have certainly thought so. If anything can be an inoculant against those temptations, surely a strong adherence to a cause greater than oneself packaged in lots warnings against biases and other ways humans can go wrong (as is the common message in EA and rationalist circles) seems like the best hope for it? If you don’t think it can be a strong inoculant, that makes you pretty cynical, no? (I think cynicism is often right, so this isn’t automatically a rejection of your position. I just want to flag that yours is a claim with quite strong implications on its own.)
Arguably the FTX scandal—which after all involved multiple EAs, not just SBF—should have already caused people to update on how effective said inoculant is, at least when billions of dollars were floating around.
If you were just talking about SBF, then I’d say your point is weak because he probably wasn’t low on dark triad traits to start out with. But you emphasizing how other EAs around him were also involved (the direct co-conspirators at Alameda and FTX) is a strong point.
Still, in my mind this would probably have gone very differently with the same group of people minus SBF and with a leader with a stronger commitment and psychological disposition towards honesty. (I should flag that parts of Caroline Ellison’s blog also gave me vibes of “seems to like having power too much”—but at least it’s more common for young people to later change/grow.) That’s why I don’t consider it a huge update for “power corrupts”. To me, it’s a reinforcement of “it matters to have good leadership.”
My worldview(?) is that “power corrupts” doesn’t apply equally to every leader and that we’d be admitting defeat straight away if we stopped trying to do ambitious things. There doesn’t seem to be a great way to do targeted ambitious things without some individual acquiring high amounts of power in the process.(?) We urgently need to do a better job at preventing that those who end up with a lot of power are almost always those with kind of shady character. The fact that we’re so bad at this suggests that these people are advantaged at some aspects of ambitious leadership, which makes the whole thing a lot harder. But that doesn’t mean it’s impossible.
I concede that there’s a sense in which this worldview of mine is not grounded in empiricism—I haven’t even looked into the matter from that perspective. Instead, it’s more like a commitment to a wager: “If this doesn’t work, what else are we supposed to do?” I’m not interested in concluding that the best we can do is criticise the powers that be from the sidelines.
Of course, if leaders exhibit signs of low integrity, like in this example of Anthropic’s communications, it’s important not to let this slide. The thing I want to push back against is an attitude of “person x or org y has acquired so much power, surely that means that they’re now corrupted,” and this leading to no longer giving them the benefit of the doubt/not trying to see the complexities of their situation when they do something that looks surprising/disappointing/suboptimal. With great power comes great responsiblity, including a responsibility to not mess up your potential for doing even more good later on. Naturally, this does come with lots of tradeoffs and it’s not always easy to infer from publicly visible actions and statements whether an org is still culturally on track. (That said, I concede that you can often tell quite a lot about someone’s character/an org’s culture based on how/whether they communicate nuances, which is sadly why I’ve had some repeated negative updates about Anthropic lately.)
When I speak of a strong inoculant, I mean something that is very effective in preventing the harm in question—such as the measles vaccine. Unless there were a measles case at my son’s daycare, or a family member were extremely vulnerable to measles, the protection provided by the strong inoculant is enough that I can carry on with life without thinking about measles.
In contrast, the influenza vaccine is a weak inoculant—I definitely get vaccinated because I’ll get infected less and hospitalized less without it. But I’m not surprised when I get the flu. If I were at great risk of serious complications from the flu, then I’d only use vaccination as one layer of my mitigation strategy (and without placing undue reliance on it.) And of course there are strengths in between those two.
I’d call myself moderately cynical. I think history teaches us that the corrupting influence of power is strong and that managing this risk has been a struggle. I don’t think I need to take the position that no strong inoculant exists. It is enough to assert that—based on centuries of human experience across cultures—our starting point should be that inoculants as weak until proven otherwise by sufficient experience. And when one of the star pupils goes so badly off the rails, along with several others in his orbit, that adds to the quantum of evidence I think is necessary to overcome the general rule.
I’d add that one of the traditional ways to mitigate this risk is to observe the candidate over a long period of time in conjunction with lesser levels of power. Although it doesn’t always work well in practice, you do get some ability to measure the specific candidate’s susceptibility in lower-stakes situations. It may not be popular to say, but we just won’t have had the same potential to observe people in their 20s and 30s in intermediate-power situations that we often will have had for the 50+ crowd. Certainly people can and do fake being relatively unaffected by money and power for many years, but it’s harder to pull off than for a shorter period of time.
If anything can be an inoculant against those temptations, surely a strong adherence to a cause greater than oneself packaged in lots warnings against biases and other ways humans can go wrong (as is the common message in EA and rationalist circles) seems like the best hope for it?
Maybe. But on first principles, one might have also thought that belief in an all-powerful, all-knowing deity who will hammer you if you fall out of line would be a fairly strong inoculant. But experience teaches us that this is not so!
Also, if I had to design a practical philosophy that was maximally resistant to corruption, I’d probably ground it on virtue ethics or deontology rather than give so much weight to utilitarian considerations. The risk of the newly-powerful person deceiving themselves may be greater for a utilitarian.
--
As you imply, the follow-up question is where we go from here. I think there are three possible approaches to dealing with a weak or moderate-strength inoculant:
In some cases, a sober understanding of how strong or weak the inoculant is should lead to a decision not to proceed with a project at all.
In other cases, a sober understanding of the inoculant affects how we should weight further measures to mitigate the risk of corrupting influence versus maximizing effectiveness.
For instance, I think you’re onto something with “these people are advantaged at some aspects of ambitious leadership.” If I’m permitted a literary analogy, one could assign more weight to how much a would-be powerholder has The Spirit of Frodo in deciding who to entrust with great power. Gandalf tells us that Bilbo (and thus Frodo) were meant to have the ring, and not by its maker. The problem is that Frodo would probably make a lousy CEO in a competitive, fast-moving market, and I’m not sure you can address that without also removing something of what makes him best-suited to bear the Ring.
In still other cases, there isn’t a good alternative and there aren’t viable mitigating factors. But acknowledging the risk that is being taken is still important; it ensures we are accounting for all the risks, reminds us to prepare contingency plans, and so on.
My point is that doing these steps well requires a reasonably accurate view of inoculant strength. And I got the sense that the community is more confident in EA-as-inoculant than the combination of general human experience and the limited available evidence on EA-as-inoculant warrants.
Are you saying bc it’s not “surprising” it should be allowed? This rhetorical move of shaming your opponent for not having already gotten used to and therefore tolerating someone doing bad things I always find bizarre.
No—that something is unsurprising, even readily predictable, does not imply anything about whether it is OK.
The fact that people seem surprised by the presence of corpspeak here does make me concerned that they may have been looking at the world with an assumption that “aligned” people are particularly resistant to the corrosive effects of money and power. That, in my opinion, is a dangerous assumption to make—and is not one I would find well-supported by the available evidence. Our models of the world should assume that at least the significant majority of people will be adversely and materially influenced by exposure to high concentrations of money and power, and we need to plan accordingly.
Convincing such people that Anthropic is doing corpspeak and not just being perfectly reasonable or justified by 3D chess (with ultimate EA goals) would be a lot of progress...
It’s a huge problem in EA that people don’t take CoI that seriously as something that affect their thinking. They think they can solve every problem explicitly intellectually so corruption by money won’t happen to them.
What surprises me about this whole situation is that people seem surprised at the executive leadership at a corporation worth an estimated $61.5B would engage in big-corporation PR-speak. The base rate for big-corporation execs engaging in such conduct in their official capacities seems awfully close to 100%. Hence, it does not feel like anything to update on for me.
I’m getting the sense that a decent number of people assume that being “EA aligned” is somehow a strong inoculant against the temptations of money and power. Arguably the FTX scandal—which after all involved multiple EAs, not just SBF—should have already caused people to update on how effective said inoculant is, at least when billions of dollars were floating around.[1]
This is not to suggest that most EAs would act in fraudulent ways if surrounded by billions of dollars, but it does provide evidence that EAs are not super-especially resistant to the corrosive effects of money and power at that level of concentration. FTX was only one cluster of people, but how many people have been EAs first and then been exposed to the amount of money/power that FTX or Anthropic had/have?
(I know I’m late again replying to this thread.)
Hm, good point. This gives me pause, but I’m not sure what direction to update in. Like, maybe I should update “corporate speak is just what these large orgs do and it’s more like a fashion thing than a signal of their (lack of) integrity on things that matter most.” Or maybe I should update in the direction you suggest, namely “if an org grows too much, it’s unlikely to stay aligned with its founding character principles.”
I would have certainly thought so. If anything can be an inoculant against those temptations, surely a strong adherence to a cause greater than oneself packaged in lots warnings against biases and other ways humans can go wrong (as is the common message in EA and rationalist circles) seems like the best hope for it? If you don’t think it can be a strong inoculant, that makes you pretty cynical, no? (I think cynicism is often right, so this isn’t automatically a rejection of your position. I just want to flag that yours is a claim with quite strong implications on its own.)
If you were just talking about SBF, then I’d say your point is weak because he probably wasn’t low on dark triad traits to start out with. But you emphasizing how other EAs around him were also involved (the direct co-conspirators at Alameda and FTX) is a strong point.
Still, in my mind this would probably have gone very differently with the same group of people minus SBF and with a leader with a stronger commitment and psychological disposition towards honesty. (I should flag that parts of Caroline Ellison’s blog also gave me vibes of “seems to like having power too much”—but at least it’s more common for young people to later change/grow.) That’s why I don’t consider it a huge update for “power corrupts”. To me, it’s a reinforcement of “it matters to have good leadership.”
My worldview(?) is that “power corrupts” doesn’t apply equally to every leader and that we’d be admitting defeat straight away if we stopped trying to do ambitious things. There doesn’t seem to be a great way to do targeted ambitious things without some individual acquiring high amounts of power in the process.(?) We urgently need to do a better job at preventing that those who end up with a lot of power are almost always those with kind of shady character. The fact that we’re so bad at this suggests that these people are advantaged at some aspects of ambitious leadership, which makes the whole thing a lot harder. But that doesn’t mean it’s impossible.
I concede that there’s a sense in which this worldview of mine is not grounded in empiricism—I haven’t even looked into the matter from that perspective. Instead, it’s more like a commitment to a wager: “If this doesn’t work, what else are we supposed to do?”
I’m not interested in concluding that the best we can do is criticise the powers that be from the sidelines.
Of course, if leaders exhibit signs of low integrity, like in this example of Anthropic’s communications, it’s important not to let this slide. The thing I want to push back against is an attitude of “person x or org y has acquired so much power, surely that means that they’re now corrupted,” and this leading to no longer giving them the benefit of the doubt/not trying to see the complexities of their situation when they do something that looks surprising/disappointing/suboptimal. With great power comes great responsiblity, including a responsibility to not mess up your potential for doing even more good later on. Naturally, this does come with lots of tradeoffs and it’s not always easy to infer from publicly visible actions and statements whether an org is still culturally on track. (That said, I concede that you can often tell quite a lot about someone’s character/an org’s culture based on how/whether they communicate nuances, which is sadly why I’ve had some repeated negative updates about Anthropic lately.)
When I speak of a strong inoculant, I mean something that is very effective in preventing the harm in question—such as the measles vaccine. Unless there were a measles case at my son’s daycare, or a family member were extremely vulnerable to measles, the protection provided by the strong inoculant is enough that I can carry on with life without thinking about measles.
In contrast, the influenza vaccine is a weak inoculant—I definitely get vaccinated because I’ll get infected less and hospitalized less without it. But I’m not surprised when I get the flu. If I were at great risk of serious complications from the flu, then I’d only use vaccination as one layer of my mitigation strategy (and without placing undue reliance on it.) And of course there are strengths in between those two.
I’d call myself moderately cynical. I think history teaches us that the corrupting influence of power is strong and that managing this risk has been a struggle. I don’t think I need to take the position that no strong inoculant exists. It is enough to assert that—based on centuries of human experience across cultures—our starting point should be that inoculants as weak until proven otherwise by sufficient experience. And when one of the star pupils goes so badly off the rails, along with several others in his orbit, that adds to the quantum of evidence I think is necessary to overcome the general rule.
I’d add that one of the traditional ways to mitigate this risk is to observe the candidate over a long period of time in conjunction with lesser levels of power. Although it doesn’t always work well in practice, you do get some ability to measure the specific candidate’s susceptibility in lower-stakes situations. It may not be popular to say, but we just won’t have had the same potential to observe people in their 20s and 30s in intermediate-power situations that we often will have had for the 50+ crowd. Certainly people can and do fake being relatively unaffected by money and power for many years, but it’s harder to pull off than for a shorter period of time.
Maybe. But on first principles, one might have also thought that belief in an all-powerful, all-knowing deity who will hammer you if you fall out of line would be a fairly strong inoculant. But experience teaches us that this is not so!
Also, if I had to design a practical philosophy that was maximally resistant to corruption, I’d probably ground it on virtue ethics or deontology rather than give so much weight to utilitarian considerations. The risk of the newly-powerful person deceiving themselves may be greater for a utilitarian.
--
As you imply, the follow-up question is where we go from here. I think there are three possible approaches to dealing with a weak or moderate-strength inoculant:
In some cases, a sober understanding of how strong or weak the inoculant is should lead to a decision not to proceed with a project at all.
In other cases, a sober understanding of the inoculant affects how we should weight further measures to mitigate the risk of corrupting influence versus maximizing effectiveness.
For instance, I think you’re onto something with “these people are advantaged at some aspects of ambitious leadership.” If I’m permitted a literary analogy, one could assign more weight to how much a would-be powerholder has The Spirit of Frodo in deciding who to entrust with great power. Gandalf tells us that Bilbo (and thus Frodo) were meant to have the ring, and not by its maker. The problem is that Frodo would probably make a lousy CEO in a competitive, fast-moving market, and I’m not sure you can address that without also removing something of what makes him best-suited to bear the Ring.
In still other cases, there isn’t a good alternative and there aren’t viable mitigating factors. But acknowledging the risk that is being taken is still important; it ensures we are accounting for all the risks, reminds us to prepare contingency plans, and so on.
My point is that doing these steps well requires a reasonably accurate view of inoculant strength. And I got the sense that the community is more confident in EA-as-inoculant than the combination of general human experience and the limited available evidence on EA-as-inoculant warrants.
Are you saying bc it’s not “surprising” it should be allowed? This rhetorical move of shaming your opponent for not having already gotten used to and therefore tolerating someone doing bad things I always find bizarre.
No—that something is unsurprising, even readily predictable, does not imply anything about whether it is OK.
The fact that people seem surprised by the presence of corpspeak here does make me concerned that they may have been looking at the world with an assumption that “aligned” people are particularly resistant to the corrosive effects of money and power. That, in my opinion, is a dangerous assumption to make—and is not one I would find well-supported by the available evidence. Our models of the world should assume that at least the significant majority of people will be adversely and materially influenced by exposure to high concentrations of money and power, and we need to plan accordingly.
Convincing such people that Anthropic is doing corpspeak and not just being perfectly reasonable or justified by 3D chess (with ultimate EA goals) would be a lot of progress...
It’s a huge problem in EA that people don’t take CoI that seriously as something that affect their thinking. They think they can solve every problem explicitly intellectually so corruption by money won’t happen to them.