Hey Nick, thanks for this very valuable experience-informed comment. I’m curious what you make of the original 2002 RCT that first tested IPT-G in Uganda. When we (at Founders Pledge) looked at StrongMinds (which we currently recommend, in large part on the back of HLI’s research), I was surprised to see that the results from the original RCT lined up closely with the pre/post scores reported by recent program participants.
Would your take on this result be that participants in the treated group were still basically giving what they saw as socially desirable answers, irrespective of the efficacy of the intervention? It’s true that the control arm in the 2002 RCT did not receive a comparable placebo treatment, so that does seem a reasonable criticism. But if the socially desirability bias is so strong as to account for the massive effect size reported in the 2002 paper, I’d expect it to appear in the NBER paper you cite, which also featured a pure control group. But that paper seems to find no effect of psychotherapy alone.
Matt these are fantastic questions that I definitely don’t have great answers to, but here are a few thoughts.
First I’m not saying at all that the Strong minds intervention is likely useless—I think it is likely very useful. Just that the positive effects may well be grossly overstated for the reasons outlined above.
My take on the result of that original 2002 RCT and Strong Minds. Yes like you say in both cases it could well be that the treatment group are giving positive answers both to appease the interviewer (Incredibly the before and after interviews were done by the same researcher in that study which is deeply problematic!) and because they may have been hoping positive responses might provide them with further future help.
Also in most of these studies, participants are given something physical for being part of the intervention groups. Perhaps small allowances for completing interviews, or tea and biscuits during the sessions. These tiny physical incentives can be more appreciated than the actual intervention. Knowing World Vision this would almost certainly be the case
I have an immense mistrust of World vision for a whole range of reasons, who were heavily involved in that famous 2002 RCT. This is due to their misleading advertising and a number of shocking experiences of their work here in Northern Uganda which I won’t expand on. I even wrote a blog about this a few years ago, encouraging people not to give them money. I know this may be a poor reason to mistrust a study but my previous experience heavily biases me all the same.
Great point about the NBER paper which featured a pure control group. First it was a different intervention—individual CBT not group therapy.
Second it feels like the Kenyan study was more dispassionate than some of the other big ones. I might be wrong but a bunch of the other RCTs are partly led and operated by organisations with something to prove. I did like that the Kenyan RCT felt less likely to be biased as there didn’t seem to be as much of an agenda as with some other studies.
Third, the Kenyan study didn’t pre-select people with depression, the intervention was performed on people randomly selected from the population. Obviously this means you are comparing different situations when comparing this to the studies with group psychotherapy for people with depression.
Finally allow me to speculate with enormous uncertainty. I suspect having the huge 1000 dollar cash transfers involved really changed the game here. ALL participants would have known for sure that some people people were getting the cash and this would have changed dynamics a lot. One outcome could have been that other people getting a wad of cash might have devalued the psychotherapy in participants eyes. Smart participants may even have decided the were more likely to get cash in future if they played down the effect of the therapy. Or even more extreme the confounding could go in the opposite direction of other studies, if participants assigned to psychotherapy undervalued a potentially positive intervention, out of disappointment at not getting the cash in hand. Again really just summising, but never underrate the connectivity and intelligence of people in villages in ths region!
Hey Nick, thanks for this very valuable experience-informed comment. I’m curious what you make of the original 2002 RCT that first tested IPT-G in Uganda. When we (at Founders Pledge) looked at StrongMinds (which we currently recommend, in large part on the back of HLI’s research), I was surprised to see that the results from the original RCT lined up closely with the pre/post scores reported by recent program participants.
Would your take on this result be that participants in the treated group were still basically giving what they saw as socially desirable answers, irrespective of the efficacy of the intervention? It’s true that the control arm in the 2002 RCT did not receive a comparable placebo treatment, so that does seem a reasonable criticism. But if the socially desirability bias is so strong as to account for the massive effect size reported in the 2002 paper, I’d expect it to appear in the NBER paper you cite, which also featured a pure control group. But that paper seems to find no effect of psychotherapy alone.
Matt these are fantastic questions that I definitely don’t have great answers to, but here are a few thoughts.
First I’m not saying at all that the Strong minds intervention is likely useless—I think it is likely very useful. Just that the positive effects may well be grossly overstated for the reasons outlined above.
My take on the result of that original 2002 RCT and Strong Minds. Yes like you say in both cases it could well be that the treatment group are giving positive answers both to appease the interviewer (Incredibly the before and after interviews were done by the same researcher in that study which is deeply problematic!) and because they may have been hoping positive responses might provide them with further future help.
Also in most of these studies, participants are given something physical for being part of the intervention groups. Perhaps small allowances for completing interviews, or tea and biscuits during the sessions. These tiny physical incentives can be more appreciated than the actual intervention. Knowing World Vision this would almost certainly be the case
I have an immense mistrust of World vision for a whole range of reasons, who were heavily involved in that famous 2002 RCT. This is due to their misleading advertising and a number of shocking experiences of their work here in Northern Uganda which I won’t expand on. I even wrote a blog about this a few years ago, encouraging people not to give them money. I know this may be a poor reason to mistrust a study but my previous experience heavily biases me all the same.
Great point about the NBER paper which featured a pure control group. First it was a different intervention—individual CBT not group therapy.
Second it feels like the Kenyan study was more dispassionate than some of the other big ones. I might be wrong but a bunch of the other RCTs are partly led and operated by organisations with something to prove. I did like that the Kenyan RCT felt less likely to be biased as there didn’t seem to be as much of an agenda as with some other studies.
Third, the Kenyan study didn’t pre-select people with depression, the intervention was performed on people randomly selected from the population. Obviously this means you are comparing different situations when comparing this to the studies with group psychotherapy for people with depression.
Finally allow me to speculate with enormous uncertainty. I suspect having the huge 1000 dollar cash transfers involved really changed the game here. ALL participants would have known for sure that some people people were getting the cash and this would have changed dynamics a lot. One outcome could have been that other people getting a wad of cash might have devalued the psychotherapy in participants eyes. Smart participants may even have decided the were more likely to get cash in future if they played down the effect of the therapy. Or even more extreme the confounding could go in the opposite direction of other studies, if participants assigned to psychotherapy undervalued a potentially positive intervention, out of disappointment at not getting the cash in hand. Again really just summising, but never underrate the connectivity and intelligence of people in villages in ths region!