“The longtermist case for animal welfare”
Have you seen this? https://forum.effectivealtruism.org/posts/W5AGTHm4pTd6TeEP3/should-longtermists-mostly-think-about-animals
Derek
Karma: 421
Suicide is a very poor indicator of the dead/neutral point, for a host of reasons.
A few small, preliminary surveys I’ve seen place it around 2⁄10, though it ranges from about 0.5 to 6 depending on whom and how you ask.
(I share your concerns in parentheses, and am doing some work along these lines—it’s been sidelined in part due to covid projects.)
Hah! I was working on them before getting sidelined with covid stuff.
I can send you the drafts if you send me a PM. The content is >80% done (I’ve decided to add more, so the % complete has dropped) but they need reorganising into ~10 manageable posts rather than 3 massive ones.
Thanks Aidan! Hope you’re feeling better now.
Most of your comments sound about right.
On retention rates: Your general methods seem to make sense, since one would expect gradual tapering off of benefits, but your inputs seem even more optimistic than I originally thought.
I’m not sure Strong Minds is a great benchmark for retention rates, partly because of the stark differences in context (rural Uganda vs UK cities), and partly because IIRC there were a number of issues with SM’s study, e.g. a non-randomised allocation and evidence of social desirability bias in outcome measurement, plus of course general concerns related to the fact it was a non-peer-reviewed self-evaluation. Perhaps retention rates of effects from UK psychotherapy courses of similar duration/intensity would be more relevant? But I haven’t looked at the SM study for about a year, and I haven’t looked into other potential benchmarks, so perhaps yours was a sensible choice.
Also not a great benchmark in a UK context, but Haushofer and colleagues recently did a study* of Problem Management+ in Uganda that found no benefits at the end of a year (paper forthcoming), even though it showed effectiveness at the 3 month mark in a previous study in Kenya.
*Haushofer, J., Mudida, R., & Shapiro, J. (2019). The Comparative Impact of Cash Transfers and Psychotherapy on Psychological and Economic Well-being. Working Paper. Available upon request.
Do you think GiveWell top charities are the best of all current giving opportunities? If so, what is the next best opportunity?
Do you think adopting subjective wellbeing as your primary focus would materially affect your recommendations?
In particular:
(a) Would using SWB as the primary outcome measure in your cost-effectiveness analysis change the rank ordering of your current top charities in terms of estimated cost-effectiveness?
(b) If it did, would that affect the ranking of your recommendations?
(c) Would it likely cause any of your current top charities to no longer be recommended?
(d) Would it likely cause the introduction of other charities (such as ones focused on mental health) into your top charity list?
How likely is it that GiveWell will ultimately (e.g. over a 100-year or 10,000-year period) do more harm than good? If that happens, what is the most likely explanation?
A recent post on this forum (one of the most upvoted of all time) argued that “randomista” development projects like GiveWell’s top charities are probably less cost-effective than projects to promote economic growth. Do you have any thoughts on this?
I like your general approach to this evaluation, especially:
the use of formal Bayesian updating from a prior derived in part from evidence for related programmes
transparent manual discounting of the effect size based on particular concerns about the direct study
acknowledgement of most of the important limitations of your analysis and of the RCT on which it was based
careful consideration of factors beyond the cost-effectiveness estimate.
I’d like to see more of this kind of medium-depth evaluation in EA.
I don’t have time at the moment for a close look at the CEA, but aside from limitations acknowledged in your text, 3 aspects stand out as potential concerns:
1. The “conservative” and “optimistic” results are quite extreme. This seems to be in part because “conservative” and “optimistic” values for several parameters are multiplied together (e.g. DALYs gained, yearly retention rate of benefits, % completing the course, discount rates...). As you’ll know, it is highly improbable that even, say, three independent parameters would simultaneously obtain at, say, the 10th percentile: 0.1*0.1*0.1 = 0.001. Did you consider making a probabilistic model in Guesstimate, Causal, Excel (with macros for Monte Carlo simulation), R, etc in order to generate confidence intervals around the final results? (I appreciate there are major advantages to using Sheets, but it should be fairly straightforward to reproduce at least the “Main CEA” and “Subjective CEA inputs” tabs in, for example, Guesstimate. This would also enable a rudimentary sensitivity analysis.)
2. The inputs for “Yearly retention rate of benefits” (row 10) seem pretty high (0.30, 0.50, and 0.73 for conservative, best guess, and optimistic, respectively) and the results seem fairly sensitive to this parameter. IIRC the study this was based on only had an 8-week follow-up, which would be about half your “conservative” figure (8/52 = 0.15). Even their “extended” follow-up (without a control group) was only for another 2 months. It is certainly plausible that the benefits endure for several months, but I would say that estimates of about 0.1, 0.3, and 0.7 are more reasonable. With those inputs, the cost per DALY increases to about $47,000, $4,500, or $196. That central figure is roughly on a par with CBT for depression in high-income countries, i.e. pretty good but not comparable with developing-country interventions. (And I wouldn’t take the “optimistic” figure seriously for the reasons given in (1) above.)
3. I haven’t seen the “growth model” on which the cost estimates are based, but my guess is that it doesn’t account for the opportunity cost of facilitators’ (or participants’) time. IIRC each course is led by two “skilled” volunteers who may otherwise do another pro-social activity.
There is also evidence that health problems have a much smaller effect on subjective well-being than one might imagine.
This is only the case for (some) physical health problems, especially those associated with reduced mobility. People tend to underestimate the SWB impact of (at least some) mental health problems. See e.g. Gilbert & Wilson, 2000; De Wit et al., 2000; Dolan & Kahneman, 2007; Dolan 2008; Pyne et al., 2009; Karimi et al., 2017
You might want to mention the publication date (1937)
Thanks—I missed that on my skim. But the “extended” follow-up is only for another two months. It does seem to indicate that effects persist for at least that period, without any trend towards baseline, which is promising (though without a control group the counterfactual is impossible to establish with confidence). I wonder why they didn’t continue to collect data beyond this period.
Thanks—“trained facilitator” might be a bit misleading. Still, it looks like there were two volunteer course leaders for each course, selected in part for their unspecified “skills”, who were given “on-going guidance and support” to facilitate the sessions, and who have to arrange a venue etc themselves, then go through a follow-up process when it’s over. So it’s not a trivial amount of overhead for an average of 13 participants.
I don’t have much time to spend on this, but here are a few thoughts based on a quick skim of the paper.
The study was done by some of the world’s leading experts in wellbeing and the study design seems okay-ish (‘waitlist randomisation’). The main concern with internal validity, which the authors acknowledge, is that changes in the biomarkers, while mostly heading in the right direction, were far from statistically significant. This could indicate that the effects reported on other measures were due to some factor other than actual SWB improvement, e.g. social desirability bias. But biomarkers are not a great metric, and measures were taken to address these concerns, so I find it plausible that the effects in the study population were (nearly) as large as reported.
However:
- The participants were self-selected, largely from people who were already involved with Action for Happiness (“The charity aims to help people take action to create more happiness, with a focus on pro-social behaviour to bring happiness to others around them”), and largely in the UK. They also had to register online. It’s unclear how useful it would be for other populations.
- It’s quite an intensive program, involving weekly 2–2.5 hour group meetings with a trained facilitator two volunteer facilitators. (“Each of these sessions builds on a thematic question, for example, what matters in life, how to find meaning at work, or how to build happier communities.”) This may limit its scalability and accessibility to certain groups.
- Follow-up was only for 2 months, the duration of the course itself. (This limitation seems to be due to the study design: the control group was people taking the course 8 weeks later.)
- The effect sizes for depression and anxiety were smaller than for CBT, so it may still not be the best option for mental health treatment (though the CBT studies were done in populations with a diagnosed mental disorder, so direct comparison is hard; and subgroup analyses showed that people with lower baseline wellbeing benefited most from the program).
- For clarity, the average effect size for life satisfaction was about 1 point on a 10-point scale. This is good compared to most wellbeing interventions, but that might say more about how ineffective most other interventions are than about how good this one is.So at the risk of sounding too negative: it’s hardly surprising that people who are motivated enough to sign up for and attend a course designed to make them happier do in fact feel a bit happier while taking the course. It seems important to find out how long these effects endure, and whether the course is suitable for a broader range of people.
But I really think the whole name should be reconsidered.
You could keep the name but drop the first ‘A’: CEELAR. Excluding the ‘A’ of Altruism isn’t great, but I think you’re allowed to take major liberties with acronyms. And really, almost anything is better than CEEALAR.
Thanks Rob!
As you’ve said, in addition to averting deaths it looks like AMF considerably improves lives, e.g. by improving economic outcomes and reducing episodes of illness. Have you considered collecting data on subjective wellbeing in order to help quantify these improvements? Could that be integrated into your program without too much expense/difficulty?
On the other side of the coin, one possible negative impact of programs that increase wealth and/or population size is the suffering of animals farmed for food (since better-off people tend to eat more meat). Do you have any data on dietary changes resulting from bed net distribution (or similar programs)? Would it be feasible to collect that data in future?
A recent post on this forum (the fourth most popular ever, at the time of writing) argued that “randomista” development projects like AMF are probably less cost-effective than projects to promote economic growth. Do you have any thoughts on this?
What are your thoughts on the indirect (“flow-through”) effects of AMF? For example:
1. What do you think are the main positive and negative indirect impacts of the program, both long- and short-term? (E.g. increasing productivity and economic growth, increasing/decreasing total population, strengthening health systems, greenhouse gas emissions, consumption of factory-farmed meat...) Do you have any data on these? Are you planning to gather data on any of them?
2. What proportion of the long-term benefit from the program is due to short-term direct effects such as saving lives and averting unpleasant episodes of malaria, relative to indirect benefits?
3. Do you hold a particular view of population ethics (totalism, averagism, person-affecting, etc)?
4. What is your response to critics who claim we are ultimately “clueless” about the long-run magnitude or even sign of interventions like this? (I think the basic argument is that e.g. averting deaths has a wide range of knock-on effects, both good and bad, and that we may not be justified in being confident that ultimately – say, over the next few hundred years—the impact will be net positive. See e.g. here, here, and here for a better explanation)
Should Covid-19 be a priority for EAs?
A scale-neglectedness-tractability assessment, or even a full cost-effectiveness analysis, of Covid as a cause area (compared to other EA causes) could be useful. I’m starting to look into this now – please let me know if it’s already been done.