Really interesting post, but I do want to flag a big concern I have in the comparative calculation. Broadly, estimated effects are almost always just going to be way more positive than well studied effects. For example if you estimated GDâs impact using standard income vs happiness adjustment measures (e.g. the value of double someoneâs income on their happiness) you end up at a much higher number than the RCT results. I think this sort of thing happens pretty consistently and predictability. For example, it would be really easy to imagine Strong Minds treatments are different enough from the most studied ways of doing CBT for the treatment effects to only persist 1 year (which would reduce the cost-effectiveness to about equal), and itâs easy to imagine several such changes (almost all going in a more pessimistic direction).
On the flip side, there has been extensive research, evaluation and huge numbers of charities founded in the global health space leading to a comparatively very small number of super strong charities, many of which are explicitly focused on cost-effectiveness/âimpact, etc. This same work (as far as I know) has not been done in the mental health area. In many ways, you are comparing a very strong global poverty charity to a much more average mental health charity. Thus personally I would not necessarily need to see a current mental health charity beating GiveWellâs best to be convinced the area as a whole could be very effective (if some strong research, evaluations and impact focused charities) were founded or identified in the area. Given my current work with Charity Entrepreneurship, the main case I am considering is if a new well researched and impact focused charity in mental health could be competitive with GiveWell top charities in effectiveness. I feel like the posts you have made over time have made this claim seem pretty plausible.
I may have misunderstood your first comment, but if I had estimated the effects for GiveDirectly it would have been (on my best guess) less effective than the study showed. From the 2016 paper I inferred GD increased life satisfaction (LS) by 0.3/â10 per person. In the Origins of Happiness, Clark et al find a doubling of income increase LS 0.12/â10 by. IIRC (and I may not), the $750 transfer from GD is less than a doubling of household income. So the estimated effects would have been approx. 3 times smaller for GD.
Regarding StrongMindsâ treatment, Reay et al. (2012) have a 2 year study of how much of the benefits are retained for interpersonal group therapy (which is what StrongMinds delivers). I agree it is more appropriate to use this than using the Wiles et. al (2016) modelâwhich I interpret as a constant effect for 4 years and then nothing thereafterâas Wiles et al. is based on UK CBT, I think delivered individually. To account for this, in my spreadsheet, I do two estimates, one where I assume the treatment effect is constant as lasts only 4 years, another where 75% of the benefits are retained annually. This latter estimation method is taken from Halstead and Snowdenâs Founderâs Pledge report on mental health where they also assess StrongMinds. It turns out the estimates give practically identical results so, in this case, the cost-effectiveness is not sensitive to how duration of effect is modeled.
I agree with you that the best current mental health charity is probably far less cost-effective, relative to whatever the best possible intervention is, than the best current development or physical health charities, on the grounds more effort has been put into the latter. (As you and I have discussed) I am optimistic about finding/âdeveloping even better ways to do provide mental health treatments. I didnât stress this point on the grounds the reader was probably more interested in current interventions than hypothetical interventions, but that could have been an error on my part.
Really interesting post, but I do want to flag a big concern I have in the comparative calculation. Broadly, estimated effects are almost always just going to be way more positive than well studied effects. For example if you estimated GDâs impact using standard income vs happiness adjustment measures (e.g. the value of double someoneâs income on their happiness) you end up at a much higher number than the RCT results. I think this sort of thing happens pretty consistently and predictability. For example, it would be really easy to imagine Strong Minds treatments are different enough from the most studied ways of doing CBT for the treatment effects to only persist 1 year (which would reduce the cost-effectiveness to about equal), and itâs easy to imagine several such changes (almost all going in a more pessimistic direction).
On the flip side, there has been extensive research, evaluation and huge numbers of charities founded in the global health space leading to a comparatively very small number of super strong charities, many of which are explicitly focused on cost-effectiveness/âimpact, etc. This same work (as far as I know) has not been done in the mental health area. In many ways, you are comparing a very strong global poverty charity to a much more average mental health charity. Thus personally I would not necessarily need to see a current mental health charity beating GiveWellâs best to be convinced the area as a whole could be very effective (if some strong research, evaluations and impact focused charities) were founded or identified in the area. Given my current work with Charity Entrepreneurship, the main case I am considering is if a new well researched and impact focused charity in mental health could be competitive with GiveWell top charities in effectiveness. I feel like the posts you have made over time have made this claim seem pretty plausible.
Hello Joey,
I may have misunderstood your first comment, but if I had estimated the effects for GiveDirectly it would have been (on my best guess) less effective than the study showed. From the 2016 paper I inferred GD increased life satisfaction (LS) by 0.3/â10 per person. In the Origins of Happiness, Clark et al find a doubling of income increase LS 0.12/â10 by. IIRC (and I may not), the $750 transfer from GD is less than a doubling of household income. So the estimated effects would have been approx. 3 times smaller for GD.
Regarding StrongMindsâ treatment, Reay et al. (2012) have a 2 year study of how much of the benefits are retained for interpersonal group therapy (which is what StrongMinds delivers). I agree it is more appropriate to use this than using the Wiles et. al (2016) modelâwhich I interpret as a constant effect for 4 years and then nothing thereafterâas Wiles et al. is based on UK CBT, I think delivered individually. To account for this, in my spreadsheet, I do two estimates, one where I assume the treatment effect is constant as lasts only 4 years, another where 75% of the benefits are retained annually. This latter estimation method is taken from Halstead and Snowdenâs Founderâs Pledge report on mental health where they also assess StrongMinds. It turns out the estimates give practically identical results so, in this case, the cost-effectiveness is not sensitive to how duration of effect is modeled.
I agree with you that the best current mental health charity is probably far less cost-effective, relative to whatever the best possible intervention is, than the best current development or physical health charities, on the grounds more effort has been put into the latter. (As you and I have discussed) I am optimistic about finding/âdeveloping even better ways to do provide mental health treatments. I didnât stress this point on the grounds the reader was probably more interested in current interventions than hypothetical interventions, but that could have been an error on my part.