Really great post, agree with almost everything, thanks for writing!
(More speculatively, it seems plausible to me that many EAs have worse judgement of character than average, because e.g. they project their good intentions onto others.)
Agreed. Another plausible reason is that system 1 / gut instincts play an important role in character judgment but many EAs dismiss their system 1 intuitions more or experience them less strongly than the average human. This is partly due to selection effects (EA appeals more to analytical people) but perhaps also because several EA principles emphasize putting more weight on reflective, analytical reasoning than on instincts and emotions (e.g., the heuristics and biases literature, several top cause areas (like AI) aren’t intuitive at all, and so on). [1]
That’s at least what I experienced first hand when interacting with a dangerous EA several years ago. I met a few people who had negative impressions of this person’s character but couldn’t really back them up with any concrete evidence or reasoning, and this EA continued to successfully deceive me for more than a year.[2] Personally, I didn’t have a negative impression in the first place (partly because the concept of a non-trustworthy EA was completely out of my hypothesis space back then) so other people were clearly able to pick up on something that I couldn’t.
To be clear, I’m not saying that reflective reasoning is bad (it’s awesome) or that we now should all trust our gut instincts when it comes to character judgment. Gut instincts are clearly fallible. The average human certainly isn’t amazing at character judgment, for example, ~50% of US Americans have voted for clearly dangerous people like Trump.
Really great post, agree with almost everything, thanks for writing!
Agreed. Another plausible reason is that system 1 / gut instincts play an important role in character judgment but many EAs dismiss their system 1 intuitions more or experience them less strongly than the average human. This is partly due to selection effects (EA appeals more to analytical people) but perhaps also because several EA principles emphasize putting more weight on reflective, analytical reasoning than on instincts and emotions (e.g., the heuristics and biases literature, several top cause areas (like AI) aren’t intuitive at all, and so on). [1]
That’s at least what I experienced first hand when interacting with a dangerous EA several years ago. I met a few people who had negative impressions of this person’s character but couldn’t really back them up with any concrete evidence or reasoning, and this EA continued to successfully deceive me for more than a year.[2] Personally, I didn’t have a negative impression in the first place (partly because the concept of a non-trustworthy EA was completely out of my hypothesis space back then) so other people were clearly able to pick up on something that I couldn’t.
To be clear, I’m not saying that reflective reasoning is bad (it’s awesome) or that we now should all trust our gut instincts when it comes to character judgment. Gut instincts are clearly fallible. The average human certainly isn’t amazing at character judgment, for example, ~50% of US Americans have voted for clearly dangerous people like Trump.
FWIW, my experiences with this person were a major inspiration for this post.