[Disclaimer: I worked at HLI until March 2023. I now work at the International Alliance of Mental Health Research Funders]
Gregory says
these problems are sufficiently major I think potential donors are ill-advised to follow the recommendations and analysis in this report.
That is a strong claim to make and it requires him to present a convincing case that GiveDirectly is more cost-effective than StrongMinds. I’ve found his previous methodological critiques to be constructive and well-explained. To their credit, HLI has incorporated many of them in the updated analysis. However, in my opinion, the critiques he presents here do not make a convincing case.
Taking his summary points in turn...
The literature on PT in LMICs is a complete mess. Insofar as more sense can be made from it, the most important factors appear to belong to the studies investigating it (e.g. their size) rather than qualities of the PT interventions themselves.
I think this is much too strong. The three meta-analyses (and Gregory’s own calculations) give me confidence that psychotherapy in LMICs is effective, although the effects are likely to be small.
2. Trying to correct the results of a compromised literature is known to be a nightmare. Here, the qualitative evidence for publication bias is compelling. But quantifying what particular value of ‘a lot?’ the correction should be is fraught: numerically, methods here disagree with one another dramatically, and prove highly sensitive to choices on data exclusion.
There is no consensus on the appropriate methodology for adjusting publication bias. I don’t have an informed opinion on this, but HLI’s approach seems reasonable to me and I think it’s reasonable for Greg to take a different view. From my limited understanding, neither approach makes GiveDirectly more cost-effective.
3. Regardless of how PT looks in general, StrongMinds, in particular, is looking less and less promising. Although initial studies looked good, they had various methodological weaknesses, and a forthcoming RCT with much higher methodological quality is expected to deliver disappointing results.
We don’t have any new data on StrongMinds so I’m confused why Greg thinks it’s “less and less promising”. HLI’s Bayesian approach is a big improvement on the subjective weightings they used in the first cost-effectiveness analysis. As with publication bias, it’s reasonable to hold different views on how to construct the prior, but personally, I do believe that any psychotherapy intervention in LMICs, so long as cost per patient is <$100, is a ~certain bet to beat cash transfers. There are no specific models of psychotherapy that perform better than the others, so I don’t find it surprising that training people to talk to other people about their problems is a more cost-effective way to improve wellbeing in LMICs. Cash transfers are much more expensive and the effects on subjective wellbeing are also small.
4. The evidential trajectory here is all to common, and the outlook typically bleak. It is dubious StrongMinds is a good pick even among psychotherapy interventions (picking one at random which doesn’t have a likely-bad-news RCT imminent seems a better bet). Although pricing different interventions is hard, it is even more dubious SM is close to the frontier of “very well evidenced” vs. “has very promising results” plotted out by things like AMF, GD, etc. HLI’s choice to nonetheless recommend SM again this giving season is very surprising. I doubt it will weather hindsight well.
HLI had to start somewhere and I think we should give credit to StrongMinds for being brave enough to open themselves up to the scrutiny they’ve faced. The three meta-analyses and the tentative analysis of Friendship Bench suggest there is ‘altruistic gold’ to be found here and HLI has only just started to dig. The field is growing quickly and I’m optimistic about the trajectories of CE-incubated charities like Vida Plena and Kaya Guides.
In the meantime, although the gap between GiveDirectly and StrongMinds has clearly narrowed, I remain unconvinced that cash is clearly the better option (but I do remain open-minded and open to pushback).
[Disclaimer: I worked at HLI until March 2023. I now work at the International Alliance of Mental Health Research Funders]
Gregory says
That is a strong claim to make and it requires him to present a convincing case that GiveDirectly is more cost-effective than StrongMinds. I’ve found his previous methodological critiques to be constructive and well-explained. To their credit, HLI has incorporated many of them in the updated analysis. However, in my opinion, the critiques he presents here do not make a convincing case.
Taking his summary points in turn...
I think this is much too strong. The three meta-analyses (and Gregory’s own calculations) give me confidence that psychotherapy in LMICs is effective, although the effects are likely to be small.
There is no consensus on the appropriate methodology for adjusting publication bias. I don’t have an informed opinion on this, but HLI’s approach seems reasonable to me and I think it’s reasonable for Greg to take a different view. From my limited understanding, neither approach makes GiveDirectly more cost-effective.
We don’t have any new data on StrongMinds so I’m confused why Greg thinks it’s “less and less promising”. HLI’s Bayesian approach is a big improvement on the subjective weightings they used in the first cost-effectiveness analysis. As with publication bias, it’s reasonable to hold different views on how to construct the prior, but personally, I do believe that any psychotherapy intervention in LMICs, so long as cost per patient is <$100, is a ~certain bet to beat cash transfers. There are no specific models of psychotherapy that perform better than the others, so I don’t find it surprising that training people to talk to other people about their problems is a more cost-effective way to improve wellbeing in LMICs. Cash transfers are much more expensive and the effects on subjective wellbeing are also small.
HLI had to start somewhere and I think we should give credit to StrongMinds for being brave enough to open themselves up to the scrutiny they’ve faced. The three meta-analyses and the tentative analysis of Friendship Bench suggest there is ‘altruistic gold’ to be found here and HLI has only just started to dig. The field is growing quickly and I’m optimistic about the trajectories of CE-incubated charities like Vida Plena and Kaya Guides.
In the meantime, although the gap between GiveDirectly and StrongMinds has clearly narrowed, I remain unconvinced that cash is clearly the better option (but I do remain open-minded and open to pushback).