“Due to this, I should act on the assumption EAs aren’t more trustworthy than average. Previously I acted as if they were. I now think the typical EA probably is more trustworthy than average – EA attracts some very kind and high integrity people – but it also attracts plenty of people with normal amounts of pride, self-delusion and self-interest, and there’s a significant minority who seem more likely to defect or be deceptive than average. The existence of this minority, and because it’s hard to tell who is who, means you need to assume that someone might be untrustworthy by default (even if “they’re an EA”). This doesn’t mean distrusting everyone by default – I still think it’s best to default to being cooperative – but it’s vital to have checks for and ways to exclude dangerous actors, especially in influential positions (i.e. trust, but verify).”
To build on and perhaps strengthen what you are saying. I think a central problem of utilitarian communities is: they will be disproportionately people on the spectrum, psychopaths and naive utilitarians.
I think there are reasons to think that in communities with lots of utilitarians/consequentialists, bad actors will not just be at the average societal level, but that they will be overrepresented. A lot of psychologyresearch suggests that psychopaths disproportionately have utilitarian intuitions. By my count, at least two of the first ~100 EAs were psychopaths (SBF and one other that I know of, without looking very hard). (Of course not all utilitarians are psychopaths).
In addition to this, utilitarian communities will often obviously attract a disproportionate number of naive utilitarians who will be tempted to do bad actions.
This is all compounded by the fact that utilitarian communities will also be disproportionately comprised of people on the autism spectrum. This is obvious to anyone acquainted with EA and my offhand explanation would be something about the attraction to data and quantification. The people on the spectrum get exploited by/do not stand up to the psychopaths and naive utilitarians because they are not very good at interpersonal judgements and soft skills, and lack a good theory of mind.
With respect to judgment of character, I think we should maybe expect EAs to be below average because of the prevalence of autism.
All of this suggests that, as you recommend, in communities with lots of consequentialists, there needs to be very large emphasis on virtues and common sense norms. If people show signs of not being high integrity, they should be strongly disfavoured by default. I don’t think this happens at the moment, and SBF is not the only example.
It could be true that bad actors are overrepresented in EA, and so it makes sense to have checks and balances for people in power. Perhaps honesty is not capturing all people are meaning with trustworthy, but I think the norms of honesty in EA are much stronger than in mainstream society. Behaviors such as admitting mistakes, being willing to update, admitting that someone else might be better for the job, recognizing uncertainty, admitting weaknesses in arguments, etc are much less common in mainstream society. I think EA selects for more honest people and also pushes people in the movement to be more honest.
I downvoted and want to explain my reasoning briefly: the conclusions presented are too strong, and the justifications don’t necessarily support them.
We simply don’t have enough experience or data points to say what the “central problem” in a utilitarian community will be. The one study cited seems suggestive at best. People on the spectrum are, well, on a spectrum, and so is their behavior; how they react will not be as monolithic as suggested.
All that being said, I softly agree with the conclusion (because I think this would be true for any community).
All of this suggests that, as you recommend, in communities with lots of consequentialists, there needs to be very large emphasis on virtues and common sense norms.
I’m not sure I agree with the conclusion, because people with dark triad personalities may be better than average at virtue signalling and demonstrating adherence to norms.
I think there should probably be a focus on principles, standards and rules that can be easily recalled by a person in a chaotic situation (e.g. put on your mask before helping others). And that these should be designed with limiting downside risk and risk of ruin in mind.
My intuition is that the rule “disfavour people who show signs of being low integrity” is a bad one, as:
it relies on ability to compare person to idealised person rather than behaviour to rule, and the former is much more difficult to reason about
it’s moving the problem elsewhere not solving it
it’s likely to reduce diversity and upside potential of the community
it doesn’t mitigate the risk when a bad actor passes the filter
I’d favour starting from the premise that everyone has the potential to act without integrity and trying to design systems than mitigate this risk.
What systems could be designed to mitigate risks of people taking advantage of others? What about spreading the knowledge of how we are influenced? With this knowledge, we can recognize these behaviors and turn off our auto pilots so we can defend ourselves from bad actors. Or will that knowledge being widespread lead to some people using this knowledge to do more damage?
This is excellent thank you.
“Due to this, I should act on the assumption EAs aren’t more trustworthy than average. Previously I acted as if they were. I now think the typical EA probably is more trustworthy than average – EA attracts some very kind and high integrity people – but it also attracts plenty of people with normal amounts of pride, self-delusion and self-interest, and there’s a significant minority who seem more likely to defect or be deceptive than average. The existence of this minority, and because it’s hard to tell who is who, means you need to assume that someone might be untrustworthy by default (even if “they’re an EA”). This doesn’t mean distrusting everyone by default – I still think it’s best to default to being cooperative – but it’s vital to have checks for and ways to exclude dangerous actors, especially in influential positions (i.e. trust, but verify).”
To build on and perhaps strengthen what you are saying. I think a central problem of utilitarian communities is: they will be disproportionately people on the spectrum, psychopaths and naive utilitarians.
I think there are reasons to think that in communities with lots of utilitarians/consequentialists, bad actors will not just be at the average societal level, but that they will be overrepresented. A lot of psychology research suggests that psychopaths disproportionately have utilitarian intuitions. By my count, at least two of the first ~100 EAs were psychopaths (SBF and one other that I know of, without looking very hard). (Of course not all utilitarians are psychopaths).
In addition to this, utilitarian communities will often obviously attract a disproportionate number of naive utilitarians who will be tempted to do bad actions.
This is all compounded by the fact that utilitarian communities will also be disproportionately comprised of people on the autism spectrum. This is obvious to anyone acquainted with EA and my offhand explanation would be something about the attraction to data and quantification. The people on the spectrum get exploited by/do not stand up to the psychopaths and naive utilitarians because they are not very good at interpersonal judgements and soft skills, and lack a good theory of mind.
With respect to judgment of character, I think we should maybe expect EAs to be below average because of the prevalence of autism.
All of this suggests that, as you recommend, in communities with lots of consequentialists, there needs to be very large emphasis on virtues and common sense norms. If people show signs of not being high integrity, they should be strongly disfavoured by default. I don’t think this happens at the moment, and SBF is not the only example.
It could be true that bad actors are overrepresented in EA, and so it makes sense to have checks and balances for people in power. Perhaps honesty is not capturing all people are meaning with trustworthy, but I think the norms of honesty in EA are much stronger than in mainstream society. Behaviors such as admitting mistakes, being willing to update, admitting that someone else might be better for the job, recognizing uncertainty, admitting weaknesses in arguments, etc are much less common in mainstream society. I think EA selects for more honest people and also pushes people in the movement to be more honest.
I downvoted and want to explain my reasoning briefly: the conclusions presented are too strong, and the justifications don’t necessarily support them.
We simply don’t have enough experience or data points to say what the “central problem” in a utilitarian community will be. The one study cited seems suggestive at best. People on the spectrum are, well, on a spectrum, and so is their behavior; how they react will not be as monolithic as suggested.
All that being said, I softly agree with the conclusion (because I think this would be true for any community).
I’m not sure I agree with the conclusion, because people with dark triad personalities may be better than average at virtue signalling and demonstrating adherence to norms.
I think there should probably be a focus on principles, standards and rules that can be easily recalled by a person in a chaotic situation (e.g. put on your mask before helping others). And that these should be designed with limiting downside risk and risk of ruin in mind.
My intuition is that the rule “disfavour people who show signs of being low integrity” is a bad one, as:
it relies on ability to compare person to idealised person rather than behaviour to rule, and the former is much more difficult to reason about
it’s moving the problem elsewhere not solving it
it’s likely to reduce diversity and upside potential of the community
it doesn’t mitigate the risk when a bad actor passes the filter
I’d favour starting from the premise that everyone has the potential to act without integrity and trying to design systems than mitigate this risk.
What systems could be designed to mitigate risks of people taking advantage of others? What about spreading the knowledge of how we are influenced? With this knowledge, we can recognize these behaviors and turn off our auto pilots so we can defend ourselves from bad actors. Or will that knowledge being widespread lead to some people using this knowledge to do more damage?