Haushofer et al., (2020), a trial of both psychotherapy and cash transfers in a LMIC, perform a test ‘experimenter demand effect’, where they explicitly state to the participants whether they expect the research to have a positive or negative effect on the outcome in question. We take it this would generate the maximum effect, as participants would know (rather than have to guess) what the experimenter would like to hear. Haushofer et al., (2020), found no impact of explicitly stating that they expected the intervention to increase (or decrease) self-reports of depression. The results were non-significant and close to zero (n = 1,545). We take this research to suggest social desirability bias is not a major issue with psychotherapy. Moreover, it’s unclear why, if there were a social desirability bias, it would be proportionally more acute for psychotherapy than other interventions. Further tests of experimenter demand effects would be welcome.
Other less relevant evidence of experimenter demand effects finds that it results in effects that are small or close to zero. Bandiera et al., (n =5966; 2020) studied a trial that attempted to improve the human capital of women in Uganda. They found that experimenter demand effects were close to zero. In an online experiment Mummolo & Peterson, (2019) found that “Even financial incentives to respond in line with researcher expectations fail to consistently induce demand effects.” Finally, in de Quidt et al., (2018) while they find experimenter demand effects they conclude by saying “Across eleven canonical experimental tasks we … find modest responses to demand manipulations that explicitly signal the researcher’s hypothesis… We argue that these treatments reasonably bound the magnitude of demand in typical experiments, so our … findings give cause for optimism.”
The experimenter demand test is quite reassuring! Although I disagree with the rest of that section, since I didn’t have in mind a conventional social desirability bias. I disagree with the idea that psychotherapy is no different from other interventions in this regard—anecdotally, depressed people are much more sensitive than average to feeling like a burden and not wanting other people to worry about their problems.
On (2): Here’s the section on social desirability bias from HLI’s cost-effectiveness analysis.
The experimenter demand test is quite reassuring! Although I disagree with the rest of that section, since I didn’t have in mind a conventional social desirability bias. I disagree with the idea that psychotherapy is no different from other interventions in this regard—anecdotally, depressed people are much more sensitive than average to feeling like a burden and not wanting other people to worry about their problems.