Founders Pledge recently published a report recommending Action for Happiness, a UK-based charity that aims to improve people’s subjective well-being and to help people live happy and fulfilling lives. There are two reasons members of the Effective Altruism community might find this report especially interesting:
We believe that Action for Happiness presents an unusually cost-effective giving opportunity for a charity that operates in high-income countries, comparable to StrongMinds,our mental health recommendation.
We carried out a Bayesian analysis to estimate the effectiveness of Action for Happiness’s intervention. This was our first attempt at evaluating a funding opportunity using formal Bayesian analysis, so it can no doubt be improved going forwards. However, it offers a concrete example of applying Bayesian inference to cost-effectiveness evaluation.
Below is the executive summary. The full report can be found here.
Executive Summary
1. Money, health, and subjective well-being
Almost everyone cares about experiencing positive well-being: to be happy and satisfied with life, and free from negative emotions and depression. Often, when we try to improve the world, we try to increase people’s economic status or their health, but it is often unclear how well these things translate into subjective well-being.
For instance, as Figure 1 shows, increasing income only has a weak effect on increasing subjective well-being.1 Here, household income on the x-axis is shown on a logarithmic scale: the gap between $1,000 and $2,000 is the same as the gap between $32,000 and $64,000. The data suggest that income has a rapidly declining effect on subjective well-being the richer you get: increasing your income from $1,000 to $2,000 has roughly the same effect as increasing your income from $32,000 to $64,000. There is evidence that the effect declines to zero once equivalised household income reaches around $95,000.2
Figure 1: Self-reported life satisfaction and self-reported annual household Income.
Source: “Money Can Buy Happiness, Money Can Buy Happiness,” The Economist, accessed January 27, 2020, https://www.economist.com/graphic-detail/2013/05/02/money-can-buy-happiness.
There is also evidence that health problems have a much smaller effect on subjective well-being than one might imagine.3
2. The case for focusing on subjective well-being
Because income fails to track subjective well-being accurately in some cases, it is important to look outside the typical realm of economic analysis when identifying the best opportunities for improving people’s lives. By failing to consider subjective well-being directly, it is possible that many philanthropists and governments miss out on some outstanding opportunities to do good.
Over the last year, the Founders Pledge research team has explored ways to increase subjective well-being directly. During the course of this research, we came across the charity Action for Happiness. Their programme seemed promising in improving participants’ subjective well-being, and their scale-up seemed like a highly leveraged funding opportunity. This prompted us to carry out a more in-depth evaluation, resulting in our recommendation and this report. We plan to expand our work on how best to improve subjective well-being in the future.
3. Charity recommendation: Action for Happiness
Action for Happiness (AfH) is a UK-based charity that brings people together in small, face-to-face groups to explore what really matters for a happy and meaningful life. AfH is trying to build a community of people transforming their own lives to be happier and to help those around them. AfH provides 8-week courses, called Exploring What Matters (EWM), run by volunteers in their local community. The course aims to help people to tune in to what really matters for a happy and meaningful life, connect with others in meaningful face-to-face conversations, and to take action to boost happiness for participants and for others. Most courses to date have taken place in the UK, but courses have also been run in 20 countries around the world, including the US, Australia, Germany and Italy.
AfH is planning a five-fold scale-up over the next three years. In 2018, AfH provided 108 courses for a total of 1,537 attendees, and provided 148 courses for 2,198 attendees in 2019. The scale-up aims to reach 600 courses for 10,200 attendees per year from 2023 onwards, with a projected cost of £1 million ($1.3 million). The majority of scale-up funding would be spent hiring additional staff to facilitate the scale-up.
On average, interventions in high-income countries are less cost-effective than interventions in low- and middle-income countries. This is because high-income countries have more resources to spend on improving lives than low- and middle-income countries, so many of the best opportunities have already been taken. In part as a consequence of this, people in high-income countries also tend to be happier and healthier, so it is more difficult to improve their lives. However, we believe that facilitating AfH’s scale-up is an unusually cost-effective donation opportunity for a high-income country intervention. This is because (i) participants make voluntary donations, which are estimated to provide more than 50% of revenue after scale-up and (ii) AfH generates revenue from some of its other activities, such as its educational services and events. As a result, donations are not used to directly pay for EWM courses but rather cover the scale-up costs that would enable AfH to sustainably reach far more people through EWM courses.
Summary
What do they do? Action for Happiness helps people to live a happy and meaningful life, predominantly through its 8-week Exploring What Matters course, which is run by volunteers in their local communities. Action for Happiness is seeking funding to scale-up to reach 5 times more people through the EWM course and to run sustainably at this larger scale.
Is there evidence the intervention works? The main evidence for the efficacy of the Exploring What Matters course comes from a recent randomised controlled trial (RCT) of the programme. We also considered data routinely collected by AfH to measure the effect of the course, as well as less direct evidence, in the form of another RCT of a similar course designed to improve subjective well-being.
Is the intervention cost-effective? We estimated the cost-effectiveness of Action for Happiness in terms of reduction in depression and gains in happiness and life satisfaction. Reductions in depression are given in terms of DALY-equivalents averted and years of severe major depressive disorder prevented. DALYs measure the burden of disease by accounting for the premature death (mortality) that it causes and for the years lived with illness (morbidity) it causes: a DALY burden can stem from premature death or from short-term or long-term ill health. The disability weights of different diseases range from 0 to 1 (no disability to 100% disabled). One DALY can be thought of as one lost year of healthy life.
Happiness and life satisfaction points are measured on a 0-10 scale. One happiness point year gain is one year of life with an additional happiness point on the 0-10 scale. Life satisfaction point year gains can be understood similarly.
We estimated cost-effectiveness as follows:
What are the wider benefits? The Exploring What Matters course also improves other subjective well-being measures, such as compassion, worthwhileness and anxiety, and increases in self-reported measures of social trust and pro-social behaviour. By running these courses and other related activities, Action for Happiness is building a movement for happiness and prosociality, the benefits of which could be large but are not taken into account in our cost-effectiveness model.
Is it a strong organisation? Action for Happiness has a good track record and takes a keen interest in measuring its effects on participants through a recent RCT and ongoing measurements of its effects on course participants. The organisation has been transparent in its communication with us.
Is there room for funding? Action for Happiness is seeking £1 million ($1.3 million) over the next three years to facilitate its scale-up.
Sources
1. Stevenson, Betsey, and Justin Wolfers. ‘Subjective Well-Being and Income: Is There Any Evidence of Satiation?’ American Economic Review 103, no. 3 (May 2013): 598–604. https://doi.org/10.1257/aer.103.3.598
2. Jebb, Andrew T., Louis Tay, Ed Diener, and Shigehiro Oishi. ‘Happiness, Income Satiation and Turning Points around the World’. Nature Human Behaviour 2, no. 1 (January 2018): 33–38. https://doi.org/10.1038/s41562-017-0277-0.
3. Dolan, Paul, and Daniel Kahneman. ‘Interpretations Of Utility And Their Implications For The Valuation Of Health*’. The Economic Journal 118, no. 525 (2008): 215–34. https://doi.org/10.1111/j.1468-0297.2007.02110.x.
I like your general approach to this evaluation, especially:
the use of formal Bayesian updating from a prior derived in part from evidence for related programmes
transparent manual discounting of the effect size based on particular concerns about the direct study
acknowledgement of most of the important limitations of your analysis and of the RCT on which it was based
careful consideration of factors beyond the cost-effectiveness estimate.
I’d like to see more of this kind of medium-depth evaluation in EA.
I don’t have time at the moment for a close look at the CEA, but aside from limitations acknowledged in your text, 3 aspects stand out as potential concerns:
1. The “conservative” and “optimistic” results are quite extreme. This seems to be in part because “conservative” and “optimistic” values for several parameters are multiplied together (e.g. DALYs gained, yearly retention rate of benefits, % completing the course, discount rates...). As you’ll know, it is highly improbable that even, say, three independent parameters would simultaneously obtain at, say, the 10th percentile: 0.1*0.1*0.1 = 0.001. Did you consider making a probabilistic model in Guesstimate, Causal, Excel (with macros for Monte Carlo simulation), R, etc in order to generate confidence intervals around the final results? (I appreciate there are major advantages to using Sheets, but it should be fairly straightforward to reproduce at least the “Main CEA” and “Subjective CEA inputs” tabs in, for example, Guesstimate. This would also enable a rudimentary sensitivity analysis.)
2. The inputs for “Yearly retention rate of benefits” (row 10) seem pretty high (0.30, 0.50, and 0.73 for conservative, best guess, and optimistic, respectively) and the results seem fairly sensitive to this parameter. IIRC the study this was based on only had an 8-week follow-up, which would be about half your “conservative” figure (8/52 = 0.15). Even their “extended” follow-up (without a control group) was only for another 2 months. It is certainly plausible that the benefits endure for several months, but I would say that estimates of about 0.1, 0.3, and 0.7 are more reasonable. With those inputs, the cost per DALY increases to about $47,000, $4,500, or $196. That central figure is roughly on a par with CBT for depression in high-income countries, i.e. pretty good but not comparable with developing-country interventions. (And I wouldn’t take the “optimistic” figure seriously for the reasons given in (1) above.)
3. I haven’t seen the “growth model” on which the cost estimates are based, but my guess is that it doesn’t account for the opportunity cost of facilitators’ (or participants’) time. IIRC each course is led by two “skilled” volunteers who may otherwise do another pro-social activity.
Thanks very much for this thoughtful comment and for taking the time to read and provide feedback on the report. Sorry about the delay in replying—I was ill for most of last week.
1. Yes, you’re absolutely right. The current bounds are very wide and they represent extreme, unlikely scenarios. We’re keen to develop probabilistic models in future cost-effectiveness analyses to produce e.g. 90% confidence intervals and carry out sensitivity analyses, probably using Guesstimate or R. We didn’t have time to do so for this project but this is high on our list of methodological improvements.
2. Estimating the retention rates is challenging so it’s helpful for us to know that you think our values are too high. We based this primarily on our retention rate for StrongMinds, but adjusted downwards. It’s possible we anchored on this too much. However, it’s not clear to me that our values are too high. In particular, if our best-guess retention rate for AfH is too high, then this is probably also true for StrongMinds. Since we’re using StrongMinds as a benchmark, this might not change our conclusions very much.
The total benefits are calculated somewhat confusingly and I appreciate you haven’t had the chance to look at the CEA in detail. If x is the effect directly post-treatment and r is the retention rate, we calculated the total benefits as
That is, we assume half a year of full effect, and then discount each year that follows by r each time. We calculated it in this way because for StrongMinds, we had 6 month follow-up data. However, it’s not clear that this approach is best in this case. It might have been better to:
Assume 0.15 years at full effect
Since the study has only an 8 week follow-up, as you mention
Assume somewhere in between 0.15 and 0.5 years at full effect
Since the effects still looked very good at 8 week follow-up (albeit with no control) and evidence from interventions such as StrongMinds that suggest longer-lasting effects still seems somewhat relevant
Finally, I think there are good reasons to prefer AfH over CBT in high-income countries, even if our CEA suggests they are similarly cost-effectiveness in terms of depression. (Though they might not be strong enough to convince you that AfH and e.g. StrongMinds are similarly cost-effective.)
AfH aims to improve well-being broadly, not just by treating mental health problems.
Although much—perhaps most—of the benefits of AfH’s courses come from reduction in depression, some of the benefits to e.g. happiness, life satisfaction and pro-social behaviour aren’t captured by measuring depression
Our CEA is very conservative in some respects
The effect sizes we used (after our Bayesian analysis) are about 30% as large as reported in the study
If CBT effects aren’t held to similar levels of scrutiny, then we can’t compare cost-effectiveness fairly
We think that the wider benefits of AfH’s scale-up could be very large
We focused just on the scale-up of the Exploring What Matters courses because this is easiest to measure
The happiness movement that AfH is leading and growing could be very beneficial, e.g. widely sharing materials on AfH’s website, bringing (relatively small) benefits to a large number of people
That said, I think it’s worth reconsidering our retention rates when we review this funding opportunity. Thanks for your input.
3. This is correct. We did not account for the opportunity cost of facilitators’ or participants’ time. As always, there are many factors and given time constraints, we couldn’t account for all of them. We thought that these costs would be small compared to the benefits of the course so we didn’t prioritise their inclusion. I don’t think we explicitly mentioned the opportunity cost of time in the report though, so thanks for pointing this out.
Thanks Aidan! Hope you’re feeling better now.
Most of your comments sound about right.
On retention rates: Your general methods seem to make sense, since one would expect gradual tapering off of benefits, but your inputs seem even more optimistic than I originally thought.
I’m not sure Strong Minds is a great benchmark for retention rates, partly because of the stark differences in context (rural Uganda vs UK cities), and partly because IIRC there were a number of issues with SM’s study, e.g. a non-randomised allocation and evidence of social desirability bias in outcome measurement, plus of course general concerns related to the fact it was a non-peer-reviewed self-evaluation. Perhaps retention rates of effects from UK psychotherapy courses of similar duration/intensity would be more relevant? But I haven’t looked at the SM study for about a year, and I haven’t looked into other potential benchmarks, so perhaps yours was a sensible choice.
Also not a great benchmark in a UK context, but Haushofer and colleagues recently did a study* of Problem Management+ in Uganda that found no benefits at the end of a year (paper forthcoming), even though it showed effectiveness at the 3 month mark in a previous study in Kenya.
*Haushofer, J., Mudida, R., & Shapiro, J. (2019). The Comparative Impact of Cash Transfers and Psychotherapy on Psychological and Economic Well-being. Working Paper. Available upon request.
Yes, feeling much better now fortunately! Thanks for these thoughts and studies, Derek.
Given our time constraints, we did make some judgements relatively quickly but in a way that seemed reasonable for the purposes of deciding whether to recommend AfH. So this can certainly be improved and I expect your suggestions to be helpful in doing so. This conversation has also made me think it would be good to explore six monthly/quarterly/monthly retention rates rather than annual ones—thanks for that. :)
Our retention rates for StrongMinds were also based partly on this study, but I wasn’t involved in that analysis so I’m not sure on the details of the retention rates there.
This is only the case for (some) physical health problems, especially those associated with reduced mobility. People tend to underestimate the SWB impact of (at least some) mental health problems. See e.g. Gilbert & Wilson, 2000; De Wit et al., 2000; Dolan & Kahneman, 2007; Dolan 2008; Pyne et al., 2009; Karimi et al., 2017
Yes, we had physical health problems in mind here. I appreciate this isn’t clear though—thanks for pointing out. Indeed, we are aware of the underestimation of the badness of mental health problems and aim to take this into account in future research in the subjective well-being space.