This is amazing work! I have a bunch of thoughts, which I’ll number so it’s easier for you or others to respond to. Sorry that this comment is a bit long; you can respond to the numbers one-by-one instead of all at once if you’d like:
I would love to hear what GiveWell’s response is to your findings here. As you show, I think there’s a strong case to be made for why StrongMinds should be a GiveWell top charity. I’m definitely not an expert in this though, and maybe there are good reasons GiveWell or others have for why StrongMinds shouldn’t be a GiveWell top charity.
It will likely take some time before GiveWell would be able to make StrongMinds a top charity, but it would be exciting if StrongMinds (or any mental health charity) could make it to GiveWell’s list of recommended charities as early as the end of 2022. It would be nice to hear from GiveWell about the following too if they:
do push through with doing moral weights work related to SWB (which I think would be valuable and important to do)
will assess the cost-effectiveness of task-shifted psychotherapy in low- and middle-income countries, i.e. by creating a program review on it
plan on doing anything else with regards to assessing interventions in terms of SWB
I’m looking forward to HLI’s report on the cost-effectiveness of deworming in terms of SWB! I’m also looking forward to your reports on other interventions you think might be just as cost-effective (or even better) than StrongMinds.
Just thought I’d share this here: I and Shen Javier from EA Philippines recently got a grant from the EA Infrastructure Fund to lead a 6-month part-time research project to find the top mental health charity ideas in the Philippines. This is our follow-up project as part of our participation in Charity Entrepreneurship’s 2021 incubation program, specifically their region-specific research track. We have recently hired 3 part-time research analysts (all with more background in mental health and research than Shen and I) to help us with this project.
Our goal is to find 1-2 charity ideas that are highly cost-effective to implement in the Philippines (and competitive with StrongMinds), and that Charity Entrepreneurship will be willing to incubate in 2022. These reports of yours will likely be very useful for us, which is why I took the time to read this report and browse through some of the others linked here. And I can see how we can build off and learn from this research in various ways. We’ll probably email you within the next couple of weeks to schedule a call with you and/or Joel, with more specific questions about HLI’s research and to get advice about our project!
You talk about how you estimate the long-term effects on SWB of StrongMinds and psychotherapy via modeling it as exponentially decaying. I want to understand this better—does this mean you are saying that improvements to subjective wellbeing are high post-treatment, and then basically decrease a lot (eventually to zero) over time? Is this saying that these beneficiaries become depressed again, or is this saying that their continuous lack of depression is more going to be due to other factors (and not due to the initial StrongMinds program)? If none of what I wrote above is correct, feel free to explain it better.
Hope it’s okay I ask this publicly, but feel free to respond privately if you’d like: How much time did each of these reports take (on cash transfers, psychotherapy, StrongMinds, and this summary report)?
(Optional) Do you see HLI possibly becoming like GiveWell in the future, but with charity recommendations based on which ones improve subjective wellbeing the most? Although maybe you’d want another organization to do the charity analysis, while you focus on the intervention analysis or bigger-picture research questions.
This might make sense as a vision if GiveWell doesn’t plan on recommending some charities that do well on improving SWB (i.e. StrongMinds). Hopefully GiveWell does though.
(Optional) I’m not sure if you’ll answer this, since you probably are quite uncertain, but do you tentatively think that donating to StrongMinds would be better than donating to any of the life-saving GiveWell top charities (i.e. AMF)? I know you’ve written about this before here, but I wonder if you have updated views on this.
Brian, I am glad to see your interest in our work!
1.) We have discussed our work with GiveWell. But we will let them respond :).
2.) We’re also excited to wade deeper into deworming. The analysis has opened up a lot of interesting questions.
3.) I’m excited about your search for new charities! Very cool. I would be interested to discuss this further and learn more about this project.
4.) You’re right that in both the case of CTs and psychotherapy we estimate that the effects eventually become zero. We show the trajectory of StrongMinds effects over time in Figure 5. I think you’re asking if we could interpret this as an eventual tendency towards depression relapse. If so, I think you’re correct since most individuals in the studies we summarize are depressed, and relapse seems very common in longitudinal studies. However, it’s worth noting that this is an average. Some people may never relapse after treatment and some may simply receive no effect.
5.) I’ll message you privately about this for the time being.
6.) In general we hope to get more people to make decisions using SWB.
7.) I am going to pass the buck on making a comment on this :P. This decision will depend heavily on your view of the badness of death for the person dying and if the world is over or underpopulated. We discuss this a bit more in our moral weights piece. In my (admittedly limited) understanding, the goodness of improving the wellbeing of presently existing people is less sensitive to the philosophical view you take.
Brian, our team is working on our own reports on how we view interpersonal group therapy interventions and subjective well-being measures more generally. We expect to publish our reports within the next 3-6 months.
We have spoken to HLI about their work, and HLI has given us feedback on our reports. It’s been really helpful to discuss this topic with Michael, Joel, and the team at HLI. Their work has provided some updates to how we view this topic, even if we do not ultimately end up reaching the same conclusions.
We’re still looking into this area and some of the important questions HLI has raised. While we plan to provide a more detailed view once our reports are published, a few areas where we differ from HLI are below:
The most concrete methodological difference between our approach and HLI’s is that we have a stronger view that there are larger intra-household spillovers for cash transfers than for therapy. HLI assumes any spillover effects to other household members are proportional across interventions—i.e., if a cash transfer benefits other household members’ subjective well-being x% as much as it benefits the recipient, the same is true for therapy. We have a strong prior that, since income is shared within households, other household members would benefit much more from a cash transfer than from therapy delivered to one of its members. If there are 4 to 5 members per household (roughly what we estimate for participants in GiveDirectly’s program) and there were no household multipliers from therapy, psychotherapy would be 2x-3x as cost-effective as cash transfers, taking HLI’s other assumptions as given. HLI has flagged this as an uncertainty and something they may look further into, and we would be interested to see what they find. It seems possible there are significant spillovers from therapy, but our current best guess is these would be much smaller than for cash.
Another concrete difference is that we’re not sure comparing standard deviation effects across interventions is appropriate. HLI compares the effect on well-being in standard deviations (SDs) across cash transfer and therapy interventions. Our impression is that therapy interventions like StrongMinds target individuals with depression, who might be concentrated at the lower end of subjective well-being scales, to a greater extent than cash transfer interventions do. If this is the case, then the measure of well-being per SD may be lower for therapy than for cash transfer interventions. For example, a SD for psychotherapy may be a 0.5 on a 10-point scale of well-being, while the SD for cash transfers may be 1. HLI notes this as an uncertainty, and we’d be interested in more evidence on this, too.
We put less weight on subjective well-being as a measure to compare different interventions. As HLI notes, subjective well-being measures could provide a common currency for comparing interventions like cash and therapy. We agree this is a useful perspective. However, we think there are some limitations to these measures, and give weight to other factors. I expect that GiveWell would find interventions that have benefits beyond subjective well-being more cost-effective than HLI would. As a result, even if a subjective well-being approach showed a program was more cost-effective than programs we’d expect to recommend marginal funding to, we wouldn’t necessarily make a recommendation based on that alone (though we would give it some weight).
When we’ve looked at group therapy from perspectives other than comparing different interventions’ effects on subjective well-being measures, we find much lower cost-effectiveness than HLI is finding. We’ve considered a few different angles. First, we’ve looked at the effect of therapy under our current moral weights, which we use to trade off outcomes like increasing consumption, averting deaths, and averting morbidity. Under this approach and using the effect of depression on DALYs, we find similar cost-effectiveness between therapy and cash transfers. Second, we’ve taken a very shallow look at a recent trial that compares cash transfers with therapy head-to-head. This trial finds a much smaller effect of therapy vs. cash. Third, it seems intuitively surprising that a $1,000 cash transfer for a household in a low-income country would have a substantially smaller effect than 3-4 months of therapy (0.5 SD for cash transfers vs. 1.6 SD for therapy in HLI’s study). If the average GiveDirectly household has consumption of $2,000-$3,000, a $1,000 transfer amounts to a 30%-50% increase in baseline consumption for an entire household. We would guess that if we, for example, gave households the choice between 3-4 months of therapy and a $1,000 transfer, many more would choose the transfer. While the above approaches definitely have drawbacks and we don’t think we should put 100% weight on them either, the implications seem substantially different enough from HLI’s estimates that they give us some additional skepticism.
We still have a lot of uncertainty about how to compare different interventions like cash transfers and therapy, and making these comparisons is crucial to our decisions on what funding opportunities to recommend to our donors. As a result, we hope to continue to discuss this topic with individuals who have a differing view than us on our moral weights so that we can continue to refine our approach.
We look forward to engaging once we publish a fully vettable report. Until then, I hope this answers the immediate questions you have about where the views of GiveWell and HLI differ.
Before I respond to the details, I’d like to thank GiveWell for engaging with these questions. I’m delighted our research has led to them producing their own reports into group psychotherapy and using subjective wellbeing to determine one’s moral weights.
GiveWell kindly shared a draft of their reply with us in advance and we made several comments clarifying our position. However, they decided to publish their original draft unchanged (without offering a further explanation) so we’re restating our comments here so that readers can build a better understanding of where our positions differ. I’ll split these up so it’s easier to follow, first quoting the response from GiveWell, then providing our reply.
We would guess that if we, for example, gave households the choice between 3-4 months of therapy and a $1,000 transfer, many more would choose the transfer.
To clarify, the intervention is not to provide therapy to anyone, it’s just to provide it to those who are depressed. I expect that even some depressed people would choose cash over therapy. But it’s reasonable to assume people don’t always know what’s best for them and under/overconsume on certain goods due to lack of information, etc. That’s why we need studies to see what truly improves people’s subjective well-being.
If one was serious about always giving people what they choose, then you would just give people cash and let them decide. Given that GiveWell claims that bednets and deworming are better than cash, it seems they already accept cash is not necessarily best. Hence, it’s unclear how they could raise this as a problem for therapy without being inconsistent.
Third, it seems intuitively surprising that a $1,000 cash transfer for a household in a low-income country would have a substantially smaller effect than 3-4 months of therapy (0.5 SD for cash transfers vs. 1.6 SD for therapy in HLI’s study).
What I think might have been overlooked here is that therapy is only being given to people diagnosed with mental illnesses, but the cash transfers go to poor people in general (only some of whom will be depressed). Hence, it’s perhaps not so surprising that directly treating the depression of depressed people is more impactful than giving out money (even if those people are poor). If you were in pain but also poor, no one would assume that giving you money would do more for your happiness than morphine would.
Second, we’ve taken a very shallow look at a recent trial that compares cash transfers with therapy head-to-head. This trial finds a much smaller effect of therapy vs. cash.
We account for this trial in our meta-analysis—if we hadn’t incorporated it, therapy would look even a bit more cost-effective. Of course, the point of meta-analyses is to look at the whole evidence base, rather than just selecting one or two pieces of evidence; one could discount any meta-analysis this way by pointing to the trial with the lowest effect.
We don’t think one study should overshadow the results of a meta-analysis, which aggregates a much wider set of data (“beware the man of only one study” etc). If there was one study finding no impact of bednets, I doubt GiveWell would conclude it would be reasonable to discount all the previous data on bednets.
First, we’ve looked at the effect of therapy under our current moral weights, which we use to trade off outcomes like increasing consumption, averting deaths, and averting morbidity. Under this approach and using the effect of depression on DALYs, we find similar cost-effectiveness between therapy and cash transfers.
What is the conversion rate here between DALYs and income increases, and on what is it based? I’m not sure what method could be being used here except by inputting one’s intuitions. In which case, it would be good to make that clear, as people may think the conversation rate is an authoritative fact, rather than (just) an opinion. It would be interesting to state how much readers’ opinions would need to differ from GiveWell’s to reach alternative conclusions!
To bang a familiar drum, the reason to use subjective wellbeing measures is that we can observe how much health and income changes improve wellbeing, rather than having to guess.
However, we think there are some limitations to these measures, and give weight to other factors. I expect that GiveWell would find interventions that have benefits beyond subjective well-being more cost-effective than HLI would.
It’s not easy to respond to this—it’s not stated what the limitations and other factors are.
More generally, there’s no reason to think in the abstract that, if you’re pluralist rather than monist about value, this changes the relative cost-effectiveness ranking of different actions. You’d need to provide a specific argument about what the different values are, how each intervention relatively does on this, and how the units of value are commensurate.
For example, imagine a scenario where intervention X provides 15 units of happiness/$ but does nothing for autonomy and intervention Y provides 10 units of happiness/$ and 10 units of autonomy/$. If we take one unit of happiness as being as valuable as one unit of autonomy, then Y is better than X. However, someone who only valued happiness would think X is better.
It seems possible there are significant spillovers from therapy, but our current best guess is these would be much smaller than for cash.
It would be helpful if GiveWell could share what their current best guess is. Even if spillovers are 30% for therapy and 100% for cash, assuming the original 12x multiple and 3 other household members, then the multiple would still be 5.7.
HLI assumes any spillover effects to other household members are proportional across interventions—i.e., if a cash transfer benefits other household members’ subjective well-being x% as much as it benefits the recipient, the same is true for therapy.
If there are 4 to 5 members per household (roughly what we estimate for participants in GiveDirectly’s program) and there were no household multipliers from therapy, psychotherapy would be 2x-3x as cost-effective as cash transfers, taking HLI’s other assumptions as given.
This is definitely not what we think, particularly the assumption it will be proportional across ‘any’ intervention! I’m sure why someone would believe that.
Our position, as outlined in this twitter thread, is quite a bit more nuanced. There wasn’t much evidence we could find on household spillovers—five studies for cash, one for mental health—and in each case it indicated very large spillover effects, i.e. in the range that other household members got 70-100% of the benefitted the recipient did. We didn’t include that in the final estimate because there was so little evidence and, if we’d taken it at face value, it would only have modestly changed the results (making therapy 8-10x better). Even in the extreme, and implausible, case where therapy has no household spillovers, it wouldn’t have yielded the result that psychotherapy is more cost-effective than cash transfers. We discussed this in the individual cost-effectiveness analysis reports and flagged it as something to come back to for further research.
We agree that the effects of household spillovers from cash are large. Where our priors may diverge is that HLI (and others) think that the spillovers from therapy are also large, whereas GiveWell is very sceptical about this. We are now conducting a thorough search for more evidence on household spillovers, so we are not just swapping priors.
We have published an updated cost-effectiveness comparison of psychotherapy and cash transfers to include an estimate of the effects on other household members. You can read a summary here.
For cash transfers, we estimate that each household member experiences 86% of the benefits experienced by the recipient. For psychotherapy, we estimate the spillover ratio to be 53%.
After including the household spillover effects, we estimate that StrongMinds is 9 times more cost-effective than GiveDirectly (a slight reduction from 12 times in our previous analysis).
This is amazing work! I have a bunch of thoughts, which I’ll number so it’s easier for you or others to respond to. Sorry that this comment is a bit long; you can respond to the numbers one-by-one instead of all at once if you’d like:
I would love to hear what GiveWell’s response is to your findings here. As you show, I think there’s a strong case to be made for why StrongMinds should be a GiveWell top charity. I’m definitely not an expert in this though, and maybe there are good reasons GiveWell or others have for why StrongMinds shouldn’t be a GiveWell top charity.
It will likely take some time before GiveWell would be able to make StrongMinds a top charity, but it would be exciting if StrongMinds (or any mental health charity) could make it to GiveWell’s list of recommended charities as early as the end of 2022. It would be nice to hear from GiveWell about the following too if they:
do push through with doing moral weights work related to SWB (which I think would be valuable and important to do)
will assess the cost-effectiveness of task-shifted psychotherapy in low- and middle-income countries, i.e. by creating a program review on it
plan on doing anything else with regards to assessing interventions in terms of SWB
I’m looking forward to HLI’s report on the cost-effectiveness of deworming in terms of SWB! I’m also looking forward to your reports on other interventions you think might be just as cost-effective (or even better) than StrongMinds.
Just thought I’d share this here: I and Shen Javier from EA Philippines recently got a grant from the EA Infrastructure Fund to lead a 6-month part-time research project to find the top mental health charity ideas in the Philippines. This is our follow-up project as part of our participation in Charity Entrepreneurship’s 2021 incubation program, specifically their region-specific research track. We have recently hired 3 part-time research analysts (all with more background in mental health and research than Shen and I) to help us with this project.
Our goal is to find 1-2 charity ideas that are highly cost-effective to implement in the Philippines (and competitive with StrongMinds), and that Charity Entrepreneurship will be willing to incubate in 2022. These reports of yours will likely be very useful for us, which is why I took the time to read this report and browse through some of the others linked here. And I can see how we can build off and learn from this research in various ways. We’ll probably email you within the next couple of weeks to schedule a call with you and/or Joel, with more specific questions about HLI’s research and to get advice about our project!
You talk about how you estimate the long-term effects on SWB of StrongMinds and psychotherapy via modeling it as exponentially decaying. I want to understand this better—does this mean you are saying that improvements to subjective wellbeing are high post-treatment, and then basically decrease a lot (eventually to zero) over time? Is this saying that these beneficiaries become depressed again, or is this saying that their continuous lack of depression is more going to be due to other factors (and not due to the initial StrongMinds program)? If none of what I wrote above is correct, feel free to explain it better.
Hope it’s okay I ask this publicly, but feel free to respond privately if you’d like: How much time did each of these reports take (on cash transfers, psychotherapy, StrongMinds, and this summary report)?
(Optional) Do you see HLI possibly becoming like GiveWell in the future, but with charity recommendations based on which ones improve subjective wellbeing the most? Although maybe you’d want another organization to do the charity analysis, while you focus on the intervention analysis or bigger-picture research questions.
This might make sense as a vision if GiveWell doesn’t plan on recommending some charities that do well on improving SWB (i.e. StrongMinds). Hopefully GiveWell does though.
(Optional) I’m not sure if you’ll answer this, since you probably are quite uncertain, but do you tentatively think that donating to StrongMinds would be better than donating to any of the life-saving GiveWell top charities (i.e. AMF)? I know you’ve written about this before here, but I wonder if you have updated views on this.
Brian, I am glad to see your interest in our work!
1.) We have discussed our work with GiveWell. But we will let them respond :).
2.) We’re also excited to wade deeper into deworming. The analysis has opened up a lot of interesting questions.
3.) I’m excited about your search for new charities! Very cool. I would be interested to discuss this further and learn more about this project.
4.) You’re right that in both the case of CTs and psychotherapy we estimate that the effects eventually become zero. We show the trajectory of StrongMinds effects over time in Figure 5. I think you’re asking if we could interpret this as an eventual tendency towards depression relapse. If so, I think you’re correct since most individuals in the studies we summarize are depressed, and relapse seems very common in longitudinal studies. However, it’s worth noting that this is an average. Some people may never relapse after treatment and some may simply receive no effect.
5.) I’ll message you privately about this for the time being.
6.) In general we hope to get more people to make decisions using SWB.
7.) I am going to pass the buck on making a comment on this :P. This decision will depend heavily on your view of the badness of death for the person dying and if the world is over or underpopulated. We discuss this a bit more in our moral weights piece. In my (admittedly limited) understanding, the goodness of improving the wellbeing of presently existing people is less sensitive to the philosophical view you take.
Thanks for the flag, Joel.
Brian, our team is working on our own reports on how we view interpersonal group therapy interventions and subjective well-being measures more generally. We expect to publish our reports within the next 3-6 months.
We have spoken to HLI about their work, and HLI has given us feedback on our reports. It’s been really helpful to discuss this topic with Michael, Joel, and the team at HLI. Their work has provided some updates to how we view this topic, even if we do not ultimately end up reaching the same conclusions.
We’re still looking into this area and some of the important questions HLI has raised. While we plan to provide a more detailed view once our reports are published, a few areas where we differ from HLI are below:
The most concrete methodological difference between our approach and HLI’s is that we have a stronger view that there are larger intra-household spillovers for cash transfers than for therapy. HLI assumes any spillover effects to other household members are proportional across interventions—i.e., if a cash transfer benefits other household members’ subjective well-being x% as much as it benefits the recipient, the same is true for therapy. We have a strong prior that, since income is shared within households, other household members would benefit much more from a cash transfer than from therapy delivered to one of its members. If there are 4 to 5 members per household (roughly what we estimate for participants in GiveDirectly’s program) and there were no household multipliers from therapy, psychotherapy would be 2x-3x as cost-effective as cash transfers, taking HLI’s other assumptions as given. HLI has flagged this as an uncertainty and something they may look further into, and we would be interested to see what they find. It seems possible there are significant spillovers from therapy, but our current best guess is these would be much smaller than for cash.
Another concrete difference is that we’re not sure comparing standard deviation effects across interventions is appropriate. HLI compares the effect on well-being in standard deviations (SDs) across cash transfer and therapy interventions. Our impression is that therapy interventions like StrongMinds target individuals with depression, who might be concentrated at the lower end of subjective well-being scales, to a greater extent than cash transfer interventions do. If this is the case, then the measure of well-being per SD may be lower for therapy than for cash transfer interventions. For example, a SD for psychotherapy may be a 0.5 on a 10-point scale of well-being, while the SD for cash transfers may be 1. HLI notes this as an uncertainty, and we’d be interested in more evidence on this, too.
We put less weight on subjective well-being as a measure to compare different interventions. As HLI notes, subjective well-being measures could provide a common currency for comparing interventions like cash and therapy. We agree this is a useful perspective. However, we think there are some limitations to these measures, and give weight to other factors. I expect that GiveWell would find interventions that have benefits beyond subjective well-being more cost-effective than HLI would. As a result, even if a subjective well-being approach showed a program was more cost-effective than programs we’d expect to recommend marginal funding to, we wouldn’t necessarily make a recommendation based on that alone (though we would give it some weight).
When we’ve looked at group therapy from perspectives other than comparing different interventions’ effects on subjective well-being measures, we find much lower cost-effectiveness than HLI is finding. We’ve considered a few different angles. First, we’ve looked at the effect of therapy under our current moral weights, which we use to trade off outcomes like increasing consumption, averting deaths, and averting morbidity. Under this approach and using the effect of depression on DALYs, we find similar cost-effectiveness between therapy and cash transfers. Second, we’ve taken a very shallow look at a recent trial that compares cash transfers with therapy head-to-head. This trial finds a much smaller effect of therapy vs. cash. Third, it seems intuitively surprising that a $1,000 cash transfer for a household in a low-income country would have a substantially smaller effect than 3-4 months of therapy (0.5 SD for cash transfers vs. 1.6 SD for therapy in HLI’s study). If the average GiveDirectly household has consumption of $2,000-$3,000, a $1,000 transfer amounts to a 30%-50% increase in baseline consumption for an entire household. We would guess that if we, for example, gave households the choice between 3-4 months of therapy and a $1,000 transfer, many more would choose the transfer. While the above approaches definitely have drawbacks and we don’t think we should put 100% weight on them either, the implications seem substantially different enough from HLI’s estimates that they give us some additional skepticism.
We still have a lot of uncertainty about how to compare different interventions like cash transfers and therapy, and making these comparisons is crucial to our decisions on what funding opportunities to recommend to our donors. As a result, we hope to continue to discuss this topic with individuals who have a differing view than us on our moral weights so that we can continue to refine our approach.
We look forward to engaging once we publish a fully vettable report. Until then, I hope this answers the immediate questions you have about where the views of GiveWell and HLI differ.
Before I respond to the details, I’d like to thank GiveWell for engaging with these questions. I’m delighted our research has led to them producing their own reports into group psychotherapy and using subjective wellbeing to determine one’s moral weights.
GiveWell kindly shared a draft of their reply with us in advance and we made several comments clarifying our position. However, they decided to publish their original draft unchanged (without offering a further explanation) so we’re restating our comments here so that readers can build a better understanding of where our positions differ. I’ll split these up so it’s easier to follow, first quoting the response from GiveWell, then providing our reply.
To clarify, the intervention is not to provide therapy to anyone, it’s just to provide it to those who are depressed. I expect that even some depressed people would choose cash over therapy. But it’s reasonable to assume people don’t always know what’s best for them and under/overconsume on certain goods due to lack of information, etc. That’s why we need studies to see what truly improves people’s subjective well-being.
If one was serious about always giving people what they choose, then you would just give people cash and let them decide. Given that GiveWell claims that bednets and deworming are better than cash, it seems they already accept cash is not necessarily best. Hence, it’s unclear how they could raise this as a problem for therapy without being inconsistent.
What I think might have been overlooked here is that therapy is only being given to people diagnosed with mental illnesses, but the cash transfers go to poor people in general (only some of whom will be depressed). Hence, it’s perhaps not so surprising that directly treating the depression of depressed people is more impactful than giving out money (even if those people are poor). If you were in pain but also poor, no one would assume that giving you money would do more for your happiness than morphine would.
We account for this trial in our meta-analysis—if we hadn’t incorporated it, therapy would look even a bit more cost-effective. Of course, the point of meta-analyses is to look at the whole evidence base, rather than just selecting one or two pieces of evidence; one could discount any meta-analysis this way by pointing to the trial with the lowest effect.
We don’t think one study should overshadow the results of a meta-analysis, which aggregates a much wider set of data (“beware the man of only one study” etc). If there was one study finding no impact of bednets, I doubt GiveWell would conclude it would be reasonable to discount all the previous data on bednets.
What is the conversion rate here between DALYs and income increases, and on what is it based? I’m not sure what method could be being used here except by inputting one’s intuitions. In which case, it would be good to make that clear, as people may think the conversation rate is an authoritative fact, rather than (just) an opinion. It would be interesting to state how much readers’ opinions would need to differ from GiveWell’s to reach alternative conclusions!
To bang a familiar drum, the reason to use subjective wellbeing measures is that we can observe how much health and income changes improve wellbeing, rather than having to guess.
It’s not easy to respond to this—it’s not stated what the limitations and other factors are.
More generally, there’s no reason to think in the abstract that, if you’re pluralist rather than monist about value, this changes the relative cost-effectiveness ranking of different actions. You’d need to provide a specific argument about what the different values are, how each intervention relatively does on this, and how the units of value are commensurate.
For example, imagine a scenario where intervention X provides 15 units of happiness/$ but does nothing for autonomy and intervention Y provides 10 units of happiness/$ and 10 units of autonomy/$. If we take one unit of happiness as being as valuable as one unit of autonomy, then Y is better than X. However, someone who only valued happiness would think X is better.
It would be helpful if GiveWell could share what their current best guess is. Even if spillovers are 30% for therapy and 100% for cash, assuming the original 12x multiple and 3 other household members, then the multiple would still be 5.7.
Cash = 1 + (1*3) = 4
Psychotherapy = 12 + ((0.3*12)*3) = 12 + 10.8 = 22.8
Hence, therapy still looks quite a bit better even if the spillover effects are small.
This is definitely not what we think, particularly the assumption it will be proportional across ‘any’ intervention! I’m sure why someone would believe that.
Our position, as outlined in this twitter thread, is quite a bit more nuanced. There wasn’t much evidence we could find on household spillovers—five studies for cash, one for mental health—and in each case it indicated very large spillover effects, i.e. in the range that other household members got 70-100% of the benefitted the recipient did. We didn’t include that in the final estimate because there was so little evidence and, if we’d taken it at face value, it would only have modestly changed the results (making therapy 8-10x better). Even in the extreme, and implausible, case where therapy has no household spillovers, it wouldn’t have yielded the result that psychotherapy is more cost-effective than cash transfers. We discussed this in the individual cost-effectiveness analysis reports and flagged it as something to come back to for further research.
We agree that the effects of household spillovers from cash are large. Where our priors may diverge is that HLI (and others) think that the spillovers from therapy are also large, whereas GiveWell is very sceptical about this. We are now conducting a thorough search for more evidence on household spillovers, so we are not just swapping priors.
We have published an updated cost-effectiveness comparison of psychotherapy and cash transfers to include an estimate of the effects on other household members. You can read a summary here.
For cash transfers, we estimate that each household member experiences 86% of the benefits experienced by the recipient. For psychotherapy, we estimate the spillover ratio to be 53%.
After including the household spillover effects, we estimate that StrongMinds is 9 times more cost-effective than GiveDirectly (a slight reduction from 12 times in our previous analysis).