The Happier Lives Institute connects donors, researchers, and policymakers with the most cost-effective opportunities to increase global wellbeing.
Using the latest subjective wellbeing data, we identify the problems that matter most to people and find evidence-based ways to solve them.
Thank you for your comments, Gregory. We’re aware you have strong views on the subject and we appreciate your conscientious contributions. We discussed your previous comments internally but largely concluded revisions weren’t necessary as we (a) had already considered them in the report and appendix, (b) will return to them in later versions and didn’t expect they would materially affect the results, or (c) simply don’t agree with these views. To unpack:
Study quality. We conclude the data set does contain bias, but we account for it (sections 3.2 and 5; it’s an open question among academics how best to do this). We don’t believe that the entire field of LMIC psychotherapy should be considered bunk, compromised, or uninformative. Our results are in line with existing meta-analyses of psychotherapy considered to have low risk of bias (see footnote).[1]
Evidentiary standards. We drew on a large number of RCTs for our systematic reviews and meta-analyses of cash transfers and psychotherapy (42 and 74, respectively). If one holds that the evidence for something as well-studied as psychotherapy is too weak to justify any recommendations, charity evaluators could recommend very little.
Outlier exclusion. The issues regarding outlier exclusion were discussed in some depth (3.2 in the main report and in Appendix B). Excluding outliers is thought sensible practice here; two related meta-analyses, Cuijpers et al., 2020c; Tong et al., 2023, used a similar approach. It’s consistent with not taking the entire literature at face value but also not taking guilt by association too far. If one excludes outliers, the specific way one does this has a minor effect (e.g., a 10% decline in effectiveness, see appendix). Our analysis necessarily makes analytic choices: some were pre-registered, some made on reflection, many were discussed in our sensitivity analysis. If one insisted only on using charity evaluations that had every choice pre-registered, there would be none to choose from.
Bayesian analysis: The method we use (‘grid approximation’, see 8.3 and 9.3) avoids subjective inputs. It is not this Bayesian analysis itself that ‘stacks the deck’ in favour of psychotherapy, but the evidence. Given that over 70 studies form the prior, it would be surprising if adding one study, as we did for StrongMinds, would radically alter the conclusions. [Edit 5/12/2023: on the point that StrongMinds could be more cost-effective than GiveDirectly, even if StrongMinds only has the small effect we assume it does in our hypothetical placeholder studies, it doesn’t seem inconceivable that a small, less effective intervention can still be more cost-effective than a big, expensive one. For context, we estimate it costs StrongMinds $63 per intervention—providing one person with a course of therapy—whereas it costs GiveDirectly $1221 per intervention—an $1000 cash transfer which costs $221 in overheads. As the therapy is about 20x cheaper, it can be far less effective yet still more cost-effective.]
Making recommendations: we aim to recommend the most cost-effective ways of increasing WELLBYs we’ve found so far. While we have intuitions about how good different interventions are our perspective as an organisation is that conclusions about what’s cost-effective should be led heavily by the evidence rather than by our pre-evidential beliefs (‘priors’). Given the evidence we’ve considered, we don’t see a strong case for recommending cash transfers over psychotherapy.
This is a working report, and we’ll be reflecting on how to incorporate the above, similarly psychotherapy-sceptical perspectives, and other views in the process of preparing it for academic review. In the interests of transparency, we don’t plan to engage beyond our comments above so as to preserve team resources.
We find an initial effect is 0.70 SDs, reduced to 0.46 SDs after publication bias adjustments. Cuijpers et al. 2023 find an effect of psychotherapy of 0.49 SDs for studies with low risk of bias (RoB) in low, middle, and high income countries (comparisons = 218), which reduces to between 0.27 and 0.57 after publication adjustment. Tong et al. 2023 find an effect of 0.69 SDs for studies with low RoB in non-western countries (primarily low and middle income; comparisons = 36), which adjust to between 0.42 and 0.60 after publication correction. Hence, our initial and adjusted numbers are similar.
We’d like to thank Aidan and the Giving What We Can (GWWC) team for a careful, constructive, and genuinely collaborative evaluation. They were open with their reasoning, generous with their time, and responsive throughout.
We naturally appreciate the positive things they highlighted: the transparency and rigour of our work, the promise of our recommended charities for donors with strong life-improving preferences, and our contribution to foundational wellbeing research and to the broader ecosystem.
GWWC described the decision to “not (yet)” rely on our recommendations as “unusually difficult” and “a close call,” with reasonable disagreement among their evaluators. We take this as a sign that our work is being taken seriously; we aim to make it an unusually easy call for GWWC next time.
1) Why we’re not surprised or discouraged by the evaluation results
For the purposes of GWWC’s assessment, the charity evaluator they are considering relying on needs to be reasonably competitive with the existing field leader in their cause area. In our case, as we fall under the “global health and wellbeing” category, this means HLI was directly compared to GiveWell on the cost-effectiveness of its charity recommendations and on process reliability. This is a high hurdle to pass. For context, sometimes people are surprised to hear that:
Our budget is about 5% of GiveWell’s (under $1m vs over $20m)
Our research staff is ~3 FTE compared to around 50.
We’ve only been making charity recommendations since 2022, whereas GiveWell has been doing so since 2008.
In addition, HLI has a dual mandate. While we place a high value on charity recommendations, we also work to pioneer wellbeing impact methodology and conduct a range of applied and theoretical work outside direct charity evaluation.
Given this, we aren’t necessarily surprised that GWWC describes our processes as immature, given the comparison. In fact, we did better than we expected – and for this reason, we feel energised and encouraged to do and become better.
2) How we’re planning to improve our research
Many improvements GWWC proposed were already on our roadmap; their evaluation is a welcome push to do more on these issues. Some examples of things we plan to work on include:
Applying consistent assumptions across evaluations (e.g., revisiting AMF using Taimaka-style assumptions, forming a view on time-discounting, and reviewing our income to wellbeing conversion rate)
Developing a structured “charity quality” rubric that captures things like implementation strength, monitoring and evaluation practices, organisational responsiveness.
Striving for even greater transparency, including publishing code, all relevant spreadsheets and articulating a clearer “bar” for recommendations
3) The challenges to understanding AMF’s life-improving effects
A major part of GWWC’s analysis involved applying the assumptions from our 2024 Taimaka evaluation to our 2022 AMF model. Under those revised assumptions, the life-improving benefits of AMF increase substantially, resulting in a (highly uncertain) cost-effectiveness estimate for AMF in the same ballpark as HLI’s top recommendations.
We appreciated GWWC drawing attention to inconsistencies across our analyses. We agree these should be resolved and plan to revisit AMF more systematically. However, we think it is too early to conclude that AMF has large life-improving benefits. Three evidentiary challenges make us cautious:
Limited and low-relevance evidence. There is no published direct subjective wellbeing evidence on the long-run effects of early-life malaria exposure. Instead, both HLI and GiveWell rely on just two income-based studies from the mid-20th century, DDT-based malaria eradication campaigns in the USA and India (Bleakley 2010, Cutler et al., 2010). These show small, long-lasting income increases of roughly ~2% per year of malaria averted before adjustments. These are quite different contexts from modern bednet distribution and malaria prevention in sub-Saharan Africa.
Mechanistic mismatch. These historical studies require us to convert small, diffuse, long-term income changes into WELLBYs. But this also poses issues given that cash transfers, our best evidence for income → wellbeing, involve large, relatively concentrated, highly salient income changes (10–100% increases). In contrast, malaria-reduction income gains are small, spread out, and unlikely to be psychologically salient. Because of this difference, it’s unclear to us whether a 1% income increase for 40 years should produce the same wellbeing benefit as a 40% increase for one year.
The case of deworming. In our analysis of deworming, we found that small long-run income effects do not correspond to detectable long-run wellbeing effects. This tempers our expectations against assuming that very small, long-run income changes reliably translate into meaningful improvements in wellbeing in other cases.
Given these issues, we think comparisons between AMF and our top charities—which treat depression via therapy and have large amounts of relevant, direct wellbeing data—remain “apples to oranges” until stronger, more relevant data become available. That said, we plan to revisit our AMF analysis, harmonise assumptions, re-examine the income-to-WELLBY relationship, and explore opportunities for generating direct wellbeing evidence on malaria. But we think it’s quite plausible that, in the long run, the case for the life-improving effects of malaria reduction will remain speculative until and unless more direct wellbeing evidence becomes available.
4) Closing thoughts
Overall, we appreciate the serious thought and care that went into GWWC’s evaluation. We agree with many of their points, have constructive disagreements on others, and are grateful for the clear, actionable feedback. This evaluation is an important milestone in our growth, and we look forward to strengthening our research, improving our processes, and continuing to build a world where evidence-based decision-making takes happiness seriously.
— The Happier Lives Institute