Reminder that split-brain experiments indicate that the part of the brain that makes decisions is not the part of the brain that explains decisions. The evolutionary purpose of the brain’s explaining-module is to generate plausible-sounding rationalizations for the brain’s decision-modules’ actions. These explanations also have to adhere to the social norms of the tribe, in order to avoid being shunned and starving.
Humans are literally built to generate prosocial-sounding rationalizations for their behavior. They rationalize things to themselves even when they are not being interrogated, possibly because it’s best to pre-compute and cache rationalizations that one is likely to need later. It has been postulated that this is the reason that people have internal monologues, or indeed, the reason that humans evolved big brains in the first place.
We were built to do motivated reasoning, so it’s not a bad habit that you can simply drop after reading the right blog post. Instead, it’s a fundamental flaw in our thought-processes, and must always be consciously corrected. Anytime you say “I did X because Y” without thinking about it, you are likely dead wrong.
The only way to figure out why you did anything is through empirical investigation of your past behavior (revealed preferences). This is not easy, it risks exposing your less-virtuous motivations, and almost nobody does it, so you will seem weird and untrustworthy if you always respond to “Why did you do X?” with “I don’t know, let me think”. People will instinctively want to trust and befriend the guy who always has a prosocial rationalization on the tip of his tongue. Honesty is hard.
Yes, people will always have motivated reasoning, for essentially every explanation of their actions they give. That being said, I expect it to be weaker for the small set of things people actually think about deeply, rather than things they’re asked to explain after the fact that they didn’t think about at all. Though I could be wrong about this expectation.
If you spend a lot of time in deep thought trying to reconcile “I did X, and I want to do Y” with the implicit assumption “I am a virtuous and pure-hearted person”, then you’re going to end up getting way better at generating prosocial excuses via motivated reasoning.
If, instead, you’re willing to consider less-virtuous hypotheses, you might get a better model of your own actions. Such a hypothesis would be “I did X in order to impress my friends, and I chose career path Y in order to make my internal model of my parents proud”.
Realizing such uncomfortable truths bruises the ego, but can also bear fruit. For example: If a lot of EAs’ real reason for working on what they do is to impress others, then this fact can be leveraged to generate more utility. A leaderboard on the forum, ranking users by (some EA organization’s estimate of) their personal impact could give rise to a whole bunch of QALYs.
This is a good point which I don’t think I considered enough. This post describes this somewhat.
I do think the signal for which actions are best to take has to come from somewhere. You seem to be suggesting the signal can’t come from the decisionmaker at all since people make decisions before thinking about them. I think that’s possible, but I still think there’s at least some component of people thinking clearly about their decision, even if what they’re actually doing is trying to emulate what those around them would think.
We do want to generate actual signal for what is best, and maybe we can do this somewhat by seriously thinking about things, even if there is certainly a component of motivated reasoning no matter what.
A leaderboard on the forum, ranking users by (some EA organization’s estimate of) their personal impact could give rise to a whole bunch of QALYs.
If this estimate is based on social evaluations, won’t the people making those evaluations have the same problem with motivated reasoning? It’s not clear this is a better source of signal for which actions are best for individuals.
If signal can never truly come from subjective evaluation, it seems like it wouldn’t be solved by moving to social evaluation. One thing that would seem difficult would be concrete, measurable metrics, but this seems way harder in some fields than others.
(Intersubjective evaluation—the combination of multiple people’s subjective evaluations—could plausibly be better than one person’s subjective evaluation, especially if of themselves, assuming ‘errors’ are somewhat uncorrelated.)
Reminder that split-brain experiments indicate that the part of the brain that makes decisions is not the part of the brain that explains decisions. The evolutionary purpose of the brain’s explaining-module is to generate plausible-sounding rationalizations for the brain’s decision-modules’ actions. These explanations also have to adhere to the social norms of the tribe, in order to avoid being shunned and starving.
Humans are literally built to generate prosocial-sounding rationalizations for their behavior. They rationalize things to themselves even when they are not being interrogated, possibly because it’s best to pre-compute and cache rationalizations that one is likely to need later. It has been postulated that this is the reason that people have internal monologues, or indeed, the reason that humans evolved big brains in the first place.
We were built to do motivated reasoning, so it’s not a bad habit that you can simply drop after reading the right blog post. Instead, it’s a fundamental flaw in our thought-processes, and must always be consciously corrected. Anytime you say “I did X because Y” without thinking about it, you are likely dead wrong.
The only way to figure out why you did anything is through empirical investigation of your past behavior (revealed preferences). This is not easy, it risks exposing your less-virtuous motivations, and almost nobody does it, so you will seem weird and untrustworthy if you always respond to “Why did you do X?” with “I don’t know, let me think”. People will instinctively want to trust and befriend the guy who always has a prosocial rationalization on the tip of his tongue. Honesty is hard.
Yes, people will always have motivated reasoning, for essentially every explanation of their actions they give. That being said, I expect it to be weaker for the small set of things people actually think about deeply, rather than things they’re asked to explain after the fact that they didn’t think about at all. Though I could be wrong about this expectation.
If you spend a lot of time in deep thought trying to reconcile “I did X, and I want to do Y” with the implicit assumption “I am a virtuous and pure-hearted person”, then you’re going to end up getting way better at generating prosocial excuses via motivated reasoning.
If, instead, you’re willing to consider less-virtuous hypotheses, you might get a better model of your own actions. Such a hypothesis would be “I did X in order to impress my friends, and I chose career path Y in order to make my internal model of my parents proud”.
Realizing such uncomfortable truths bruises the ego, but can also bear fruit. For example: If a lot of EAs’ real reason for working on what they do is to impress others, then this fact can be leveraged to generate more utility. A leaderboard on the forum, ranking users by (some EA organization’s estimate of) their personal impact could give rise to a whole bunch of QALYs.
This is a good point which I don’t think I considered enough. This post describes this somewhat.
I do think the signal for which actions are best to take has to come from somewhere. You seem to be suggesting the signal can’t come from the decisionmaker at all since people make decisions before thinking about them. I think that’s possible, but I still think there’s at least some component of people thinking clearly about their decision, even if what they’re actually doing is trying to emulate what those around them would think.
We do want to generate actual signal for what is best, and maybe we can do this somewhat by seriously thinking about things, even if there is certainly a component of motivated reasoning no matter what.
If this estimate is based on social evaluations, won’t the people making those evaluations have the same problem with motivated reasoning? It’s not clear this is a better source of signal for which actions are best for individuals.
If signal can never truly come from subjective evaluation, it seems like it wouldn’t be solved by moving to social evaluation. One thing that would seem difficult would be concrete, measurable metrics, but this seems way harder in some fields than others.
(Intersubjective evaluation—the combination of multiple people’s subjective evaluations—could plausibly be better than one person’s subjective evaluation, especially if of themselves, assuming ‘errors’ are somewhat uncorrelated.)