I am a researcher at the Happier Lives Institute. In my work, I assess the cost-effectiveness of interventions in terms of subjective wellbeing.
JoelMcGuire
Karma: 1,375
A quick response—I’ll respond in more depth later,
I love the idea of your RCT cash transfers vs. psychotherapy but I’m confused about a number of parts of the design and have a few questions if you will humour me .
To be clear, the planned RCT has nothing to do with HLI, so I would forward all questions to the authors! :)
And to be clear when you say “before and after measures”, do you think this applies to RCTs where the measurement of an effect is comparing a control to a treatment group instead of before and after a group receives an intervention?
Hi Jack,
Right. The psychedelic work will probably be a more speculative and a lower-bound estimate. I expect we’ll take the opportunity to cut our teeth on estimating the cost-effectiveness of research.
Hi Rosie,
Thanks for your comment! Grief is a thorny issue, and we have different priors about it as a team.
The reason I’m a little skeptical of this is first that it seems likely to me (disclaimer that I have not done a deep dive) that losing a child would increase the likelihood of depression and other mental illnesses, alongside other things like marriage disruption (e.g. Rogers et al. 2008, which highlights effects lasting to the 18 year mark). I don’t think these effects will be accounted for by pulling out the estimate coming from Oswald & Powdthavee according to the Clark paper.
Could you clarify why you think the effects aren’t accounted for? Is it because we aren’t specifically looking at evidence that captures the long-term effects of grief for the loss of a child?
If so I’m not sure that child-loss will have a different temporal dynamic than spouse-loss. I’d assume that spouse-loss, if anything, would persist longer. However, this was a shallow estimate that could be improved upon with more research—but we haven’t prioritized further research because we don’t expect it to make much of a difference.
Rather than underestimate grief, I’m inclined to think our grief estimate is relatively generous for several reasons.
We assume grief is equal for each household member. But do we expect a 2 year old child, if their sibling dies after childbirth, to be as sad as being depressed for 5 years? The effects on parents are probably larger.
We use evidence from high income countries. Will the effect on grief be large or smaller in low income countries? I’d guess that the effect of grief is less intense when death is less surprising and death is less surprising where it’s more common which is the case in low income countries. To be clear, I’m not claiming how much less intense it is.
The team differs on this, but some of us think that if we account for the counterfactual value of grief, we’d decrease its effect. The reasoning from the pro-counterfactualists is that everyone dies at some point, so presumably one will still be grieved for if they die at 50 instead of 5-years old. This is could get complex, because presumably the intensity of grief and number of people experiencing grief changes with the age of the person dying.
All of this being said, I don’t think it would change our priorities even if we had a sizeable change to the grief numbers, eg doubled or halved them, so we aren’t sure how much effort it’s worth to do more work.
Hi David,
You’re right that we should be concerned with the quality of published evidence. I discounted psychotherapy’s effect by 17% for having a higher risk of effect inflation than cash transfers, see Appendix C of McGuire & Plant (2021). However, this was the first pass at a fundamental problem in science, and I recognize we could do better here.
We’re planning on revisiting this analysis and improving our methods – but we’re currently prioritizing finding new interventions more than improving our analyses of old ones. Unfortunately, we currently don’t have the research capacity to do both well!
Hi Nick,
Thank you for sharing! This is valuable to hear. The issue of being primed to respond in a certain way has surprisingly not been explored widely in low-income countries.
We’re concerned about this, but to our knowledge, the existing evidence suggests this isn’t a large concern. The only study we’ve seen that explicitly tries to measure the impact of this was a module in Haushofer et al., (2020 section III.I) – where they find about a zero effect. If there’s more evidence we don’t know about, please share!
Here’s the excerpt (they call these effects “experimenter demand effects”).
You also said “Also, we’re comparing self-reports to other self-reports”, which doesn’t help the matter, because those who don’t get help are likely to keep scoring the survey lowly because they feel like they didn’t get help
Allow me to disagree—I think this could help the matter when we’re comparing between interventions. If this effect was substantial, the big concern would be if this effect differs dramatically between interventions. E.g., the priming effect is larger for psychotherapy than for cash transfers. I.e., it’s okay if the same bias is balanced across interventions. Of course, I’m not sure if it’s plausible for this to be the case—my prior is pretty uniform here.
All that being said, I totally endorse more research on this topic!
We can’t ignore how people feel, but we need to try and find objective ways of assessing it, especially in contexts like here in Uganda where NGOs have wrecked any chance of self reporting being very accurate.
I think this is probably a point where we disagree. A point we’ve expanded upon previously (see McGuire et al., 2022) is that we seriously doubt that we can measure feelings objectively. The only way we can know if that objective measure is capturing a feeling is by asking someone if it does so—so why not just do that?
I am much more pro “find the flaws and fix them” then “abandon ship when it leaks” regarding measuring subjective wellbeing.
Alternatively I’d be VERY interested in a head to head Cash transfer vs Strongminds RCT—should be pretty straightforward , even potentially using your same subjective before and after scores. Surely this would answer some important questions.
This may be in the works!
This is implicitly 0.42 WELLBYs per $1k if we go with the HLI adjusted figures, or it’s 1.67 WELLBYs per $1k if you take GiveWell’s income figures at face value.
Again, GiveWell doesn’t explicitly model the morbidity effects other than by inflating the value of malaria prevention’s life-saving and income-increasing effect by 9%. We didn’t tinker with the supplemental charity-level adjustments, supplemental intervention-level adjustments or their leverage/funging adjustments because that is, I expect, a whole can of worms. Because we kept these adjustments that GiveWell uses to tweak the value of income and life -- the morbidity effects that GiveWell implicitly incorporates, we implicitly include it as well.
Basically, if you think that the morbidity effects should merit a different adjustment than 9%, we don’t account for that. If you’re satisfied with 9%, then it’s already accounted for, just in a weird opaque way as part of GiveWell’s suite of subjective adjustments.
What worries me here is that you don’t need to be a totalist to have these concerns.
Right, I should have clarified that the gnarly thing with totalism is considering the effect on all future 14k+ generations and the likelihood they exist, not just the higher-order effects on the presently existing population.
However, I’m not the philosopher, so Michael may disagree with my sketch of the situation.
Hi Jack, thanks for the feedback! I think your suggested intuitions are about right for the first two points.
Regarding your third point, I’m inclined to phrase it slightly differently. People aren’t generally pretty sad on a life-satisfaction view, it’s more that the people AMF saves will live hard lives. That’s why we’re trying to help them. If we expected them to be considerably more satisfied with their lives, then I think the cost-effectiveness comparison would look very different.
Unfortunately, no. I don’t know of a website or spreadsheet that does this. I made a toy version of this just for the AMF to GiveDirectly and StrongMinds comparison, but it only includes the topics discussed in this report.
This is on HLI’s wishlist of things to do, but we have a very small team, so I’m not sure when we’ll get around to it. I’m personally very into this idea.
We already include GiveWell’s adjustments, including their supplemental intervention adjustments in the lives saved and income-generated figures we use (and then adjust in the case of the income effects).
Hi MHR,
I also wish that these choices need not be made.
I find myself still trying to form a view on how much we can and should outsource our moral reasoning by surveying people in general and beneficiaries in particular. I think this is a tricky question, as there are many technical questions we don’t and probably shouldn’t survey the affected people to decide on the best course of action (i.e., setting interest rates). That being said, I would welcome more work understanding people’s views on these tough philosophical questions. I think this could be a promising line of research[1].
Answering your specific comments.
Yes, this is after correcting the previous error.
I would hesitate to ever put “simple” next to “total utilitarian”! But more seriously, we considered adding this perspective, but we found it difficult to implement and more difficult to communicate. The difficulty with incorporating fertility is when to stop counting. Take the immediate “replacement” effect. Roodman (2014) estimates for every 4 children who die, this leads to between 1 and 2 extra children being born. So if we just stop there, adding fertility in a totalist perspective makes AMF look relatively worse. But why stop there? Why not count the next generation? If we save 4 people, but only 2 are counterfactually created, then there are still 2 people who could have 3 children of their own. So instead of just saving 4 people (which would have been the result in the non-totalist analysis), you’ve actually saved 8! Double the value! But would it make sense to add the generation after that? What about the 4th, 5th or 6th generations? At what point are we just making up numbers? A further issue is that this whole discussion doesn’t incorporate a whole lot of more speculative effects that could potentially swamp the totalists calculus. These are infamously tough problems like: is it better or worse for the rest of the population for more people to exist? AKA is the world under or overpopulated? Does adding new people increase or decrease existential risk? I don’t have good, confident answers to these questions, and I’d be sceptical of anyone who claims to! My hunch is that if you’re a totalist, your view on saving lives will probably not be driven by crunching the numbers about fertility but about your speculation on how adding people affects the wellbeing of the population and existential security.
Yes, we try to account for the benefit that’d accrue to everyone who doesn’t get sick. We mention this at the beginning of Section 3 and discuss the calculations in more detail in the appendix. Overall we think these “life improving” (as opposed to “life extending”) effects are relatively small. And if malaria prevention is cost-effective (which we argue depends on your philosophy), then I think it’ll be the life-extending benefits that drive that.
Currently, we’re primarily relying on evidence about how malaria prevention increases incomes—and then converting that into estimates of how it’d affect people’s subjective wellbeing. We’re currently missing good data on how malaria prevention affects people’s feelings.
The elephant in the bednet: the importance of philosophy when choosing between extending and improving lives
I’d prefer more investment in exploration and accuracy[1]. I wouldn’t be surprised if GiveWell pursues but doesn’t publish more speculative research of the type you’re mentioning.
I found most of these issues in a couple of hours. I also didn’t have time to publish critiques on all of the issues I found. So I am not quite as reassured as you are, particularly because relatively small changes in these figures can still lead to changes in the allocation of millions of dollars.
To be clear, I primarily cite natural experiment which I’d argue are not evidentially equivalent to observational studies.
Aside: I only skimmed Croke (2019), but are the null effects surprising given that control children are already given a big boost? -- Plausibly including other GiveWell top charity interventions such as vitamin-A supplementation and vaccines?
The intervention was delivered via a Child Health Day (CHD). Child Health Days are public events, held biannually, in which all parents in a given catchment area are requested to bring pre-school age children to a treatment site to receive a set of basic health services such as Vitamin A supplementation, growth monitoring, and vaccines. At the five Child Health Days held over the course of the project, children in the control parishes who attended the event were offered the standard CHD interventions, while those in the treatment group were also given deworming treatment (400 mg albendazole).
Yeah, I think increasing labor participation rates by women would mitigate this effect seen in the historical data. However, in some places like South Asia it’s remained low (and even declined) since the 90s.
Hi MHR,
Thanks for this! You’re definitely right about the text, and I double checked the numbers and also think you’re right there.
We’re also uncertain how much we should update on Liu & Liu. But one reason it’d be strange for GiveWell to assign zero weight to Bleakley (2007) or Liu and Liu (2019) is that this is the only type of evidence (historical natural experiments) that they use for estimating the income effects of malaria prevention.
I can maybe help with question 5, since the $170 figure originates from my analysis.
I finalized the cost figures during COVID when their cost figures were very high ($400 per person). I tried to project what they’d be over the next 3 years (starting in 2020) and assumed it’d come down, but the costs have come down faster than I imagined. They now say they expect 2022 to cost 105 USD per person treated.
They regularly update their cost and expense figures in their quarterly reports.
And here’s the general breakdown of their expenses according to their 2021 tax returns (page 10).