I was a bit worried about some possible methodological issues with the GiveWell measures of life satisfaction. I looked into the data, and the issue doesn’t completely undermine the result, but I think having looked closely I am now moderately less convinced that the negative spillover effect observed is a real problem.
Some measures of “life evaluation” use a technique like the Cantril Ladder; this is often used as a measure of happiness or subjective well-being (e.g., in the World Happiness Report 2021). In the words of that report, the Cantril ladder question “asks respondents to think of a ladder, with the best possible life for them being a 10, and the worst possible life being a 0. They are then asked to rate their own current lives on that 0 to 10 scale”.
When measuring the “negative spillover” effects discussed above in Section 6, the Cantril would be an inappropriate measure to use. That’s because, when respondents think of the ladder and imagine the “best possible life” and the “worst possible life”, many are likely to anchor on to exemplars that are salient/at hand, like, for instance, the distribution of people in their local community.
Imagine two people in a community, Andrew and Bob. You give Andrew $100 and Bob $0, and then ask Bob to rate his own life on a 0 to 10 scale. Bob might think of Andrew, who just got $100 for nothing, and think of that as particularly good. His idea of how good life can get just got a little bit higher. (If you think this is silly, rather than imagining, I don’t know, Elon Musk or some other fantastically privileged person, keep in mind that respondents probably have a few seconds to answer this questionnaire and what is salient is very important. for more discussion on the importance of salience in questionnaire respondents, consult the work of psychologist Norbert Schwarz at USC.) So Bob would be relatively lower on the ladder and he’d give himself a lower life evaluation score. But it isn’t clear this really reflects any of Bob’s subjective experience of happiness from day to day, other than times when he’s asked to rate himself on Cantril ladders. That would depend on him actually making those subjective comparisons and feeling bad about them on a regular basis.
the “happiness” and “life satisfaction” questions from the World Values Survey; the total score on the Center for Epidemiologic Studies Depression scale (CESD) (Radloff 1977); and total score on the Perceived Stress Scale (PSS)(Cohen, Kamarck, and Mermelstein 1983).
The happiness and life satisfaction from the World Values Survey can be viewed here and are:
“Taking all things together, would you say you are very happy, rather happy, not very happy, or not at all happy?”
“All things considered, how satisfied are you with your life as a whole these days? Using this card on which 1 means you are “completely dissatisfied” and 10 means you are “completely satisfied” where would you put your satisfaction with your life as a whole?”
Of the four well-being measures, the negative spillover result relates to the Life Satisfaction question specifically. It is possible that when answering (2) a respondent might use the same kind of social comparison processes that Bob used in the example above. But importantly, these weren’t suggested to the respondent by the surveyer. If the respondent had used social comparison, they did so unprompted, and it’s not so hard to imagine they do that often in their life in a way that affects their emotional affect.
Nevertheless, I’d be even more convinced if an effect on the happiness question had been observed, and the fact that the study observed a negative spillover result on Life Satisfaction and not Happiness does suggest that perhaps social comparison is occurring for Life Satisfaction in a way that doesn’t seem to impact on the Happiness measure. If you think that Life Satisfaction tells us something additional to the Happiness measure about basic intrinsic hedonic utils, you should probably still be concerned about the negative spillover. If you think that Life Satisfaction is only important to the extent that it affects self-reported happiness, you should be cautious about interpreting the result of the negative spillover.
What the survey didn’t do, because it’s very expensive and hard, and require respondents to at least be able to text on a cellphone, is to measure basic momentary positive and negative affect through a method like Ecological Momentary Assessment. I’d be interested in a future study looking at that and seeing whether we observe (1) any effect of the GiveWell intervention on momentary positive and negative affect and (2) whether there are negative spillover effects there.
I was a bit worried about some possible methodological issues with the GiveWell measures of life satisfaction. I looked into the data, and the issue doesn’t completely undermine the result, but I think having looked closely I am now moderately less convinced that the negative spillover effect observed is a real problem.
Some measures of “life evaluation” use a technique like the Cantril Ladder; this is often used as a measure of happiness or subjective well-being (e.g., in the World Happiness Report 2021). In the words of that report, the Cantril ladder question “asks respondents to think of a ladder, with the best possible life for them being a 10, and the worst possible life being a 0. They are then asked to rate their own current lives on that 0 to 10 scale”.
When measuring the “negative spillover” effects discussed above in Section 6, the Cantril would be an inappropriate measure to use. That’s because, when respondents think of the ladder and imagine the “best possible life” and the “worst possible life”, many are likely to anchor on to exemplars that are salient/at hand, like, for instance, the distribution of people in their local community.
Imagine two people in a community, Andrew and Bob. You give Andrew $100 and Bob $0, and then ask Bob to rate his own life on a 0 to 10 scale. Bob might think of Andrew, who just got $100 for nothing, and think of that as particularly good. His idea of how good life can get just got a little bit higher. (If you think this is silly, rather than imagining, I don’t know, Elon Musk or some other fantastically privileged person, keep in mind that respondents probably have a few seconds to answer this questionnaire and what is salient is very important. for more discussion on the importance of salience in questionnaire respondents, consult the work of psychologist Norbert Schwarz at USC.) So Bob would be relatively lower on the ladder and he’d give himself a lower life evaluation score. But it isn’t clear this really reflects any of Bob’s subjective experience of happiness from day to day, other than times when he’s asked to rate himself on Cantril ladders. That would depend on him actually making those subjective comparisons and feeling bad about them on a regular basis.
Fortunately, the GiveWell study did not use a Cantril Ladder. As explained by Haushofer, Reisinger, and Shapiro (2015), they used four well-being measures:
The happiness and life satisfaction from the World Values Survey can be viewed here and are:
Of the four well-being measures, the negative spillover result relates to the Life Satisfaction question specifically. It is possible that when answering (2) a respondent might use the same kind of social comparison processes that Bob used in the example above. But importantly, these weren’t suggested to the respondent by the surveyer. If the respondent had used social comparison, they did so unprompted, and it’s not so hard to imagine they do that often in their life in a way that affects their emotional affect.
Nevertheless, I’d be even more convinced if an effect on the happiness question had been observed, and the fact that the study observed a negative spillover result on Life Satisfaction and not Happiness does suggest that perhaps social comparison is occurring for Life Satisfaction in a way that doesn’t seem to impact on the Happiness measure. If you think that Life Satisfaction tells us something additional to the Happiness measure about basic intrinsic hedonic utils, you should probably still be concerned about the negative spillover. If you think that Life Satisfaction is only important to the extent that it affects self-reported happiness, you should be cautious about interpreting the result of the negative spillover.
What the survey didn’t do, because it’s very expensive and hard, and require respondents to at least be able to text on a cellphone, is to measure basic momentary positive and negative affect through a method like Ecological Momentary Assessment. I’d be interested in a future study looking at that and seeing whether we observe (1) any effect of the GiveWell intervention on momentary positive and negative affect and (2) whether there are negative spillover effects there.