First a minor note here on this reponse here “We take these data to be our best guess because there are no prior studies of the effect of deworming on SWB, and the evidence of impact on other outcomes is very uncertain. However, all the effects are non-significant. We don’t have a theory of action because we think the overall evidence points to there being no effect (or at least just a very small one).”
Where data is as bad as this, it’s better to say that there is not enough meaningful data to draw a conclusion, rather than saying “with the best evidence we have (which is bad), conclusion is X”. It’s better to not use bad evidence at all, and say there is no meaningful evidence available, than to try and draw weak conclusions from it like you do here.
My major point—as the effect on each individual from deworming is so tiny, I don’t think you’re ever going to find a significant increase in self reported happiness on a blunt 3 point scale (like ths one) , even if deworming does cause people to be a little happier
Imagine that deworming really does increase 1000 recipient’s happiness by 2% each. This won’t tip them over the line from “happy” to “very happy” so people will report the same level of happiness. Wheras the same amount of money for Strongminds MIGHT increase 10 people’s happiness by 30% each so they would all report a higher score. So even though deworming’s small increase in happiness for more people would make deworming more WELLBY effective than Strongminds, the blunt 3 point scale can’t possibly show this. Sorry if you discussed this already and I missed it.
On the bigger WELLBY picture, the more I think about it, I think the Happier Lives institute should prioritise funding a WELLBY measured RCT of StrongMinds vs. cash transfers A.S.A.P performed by an independent organisation, to answer 2 questions. 1. If StrongMinds really are superior to cash on the WEllBY front? 2. If before and after wellbeing scores can be valid, and not spoiled by the effects I’ve discussed earlier
I was super disappointed that the current StongMinds trial doesn’t test this. I don’t like to be cynical, but I wonder if there might be fear around what the potential result of that trial might be, leading to that research not happening. At this stage StrongMinds have FAR more to lose than to gain from a head to head trial vs. cash...
About your major point first. If it was up to us (we didn’t collect this data), we would use a nicer 0-10 scale. However, this is the only SWB data we are aware of. There are other measures of wellbeing in the data (including a 1-10 scale, some 1-6 frequency scales, and some binary scales) but the 3-point scale is the only measure that was collected across all three KLPS rounds. None of the other measures are significant. Some are negative, some are positive. In Appendix A3.1 we conduct an analysis where we use the effect sizes of all the other measures, and we obtain very similar results, which gives us more confidence about this measure.
Imagine that deworming really does increase 1000 recipient’s happiness by 2% each. This won’t tip them over the line from “happy” to “very happy” so people will report the same level of happiness
I’m not sure this is completely true. Some people will be tipped over the line; namely, all the people that are 2% away from changing between categories (the people who answer ‘not happy’ but will be close to answering ‘happy’).
Where data is as bad as this, it’s better to say that there is not enough meaningful data to draw a conclusion, rather than saying “with the best evidence we have (which is bad), conclusion is X”. It’s better to not use bad evidence at all, and say there is no meaningful evidence available, than to try and draw weak conclusions from it like you do here.
Both the literature (that does not contain SWB data; Section 1.3) and this SWB data (the only we could find) do not give us good grounds to recommend deworming. For completeness, we go through many considerations that we present in Section 2.3 (alternative analyses, Bayes factors, cost-effectiveness, converting GiveWell’s analysis to WELLBYs, etc.). As we note in the report, this evidence does not prove the effect is zero, but these converging lines of evidence support the conclusion that the effect is zero or very small. If strong SWB evidence that deworming is more cost-effective than StrongMinds is produced, then we would change our minds. Yes, collecting this data will be expensive, but we’d prefer some of the money going to deworming serves to nail down the effect of deworming (on SWB because we think this is what ultimately matters).
On the bigger WELLBY picture, the more I think about it, I think the Happier Lives institute should prioritise funding a WELLBY measured RCT of StrongMinds vs. cash transfers A.S.A.P performed by an independent organisation, to answer 2 questions.
1. If StrongMinds really are superior to cash on the WEllBY front?
2. If before and after wellbeing scores can be valid, and not spoiled by the effects I’ve discussed earlier
Just to clarify for people reading this (I know you are not saying this, it is just that I know that people too often misunderstand this): StrongMinds gives depression treatment for people in LMIC (who happen to also be poor). GiveDirectly transfers cash to people in poverty (some of which might be depressed). If you want to increase WELLBYs, StrongMinds will produce more WELLBYs per dollar than GiveDirectly. We are not suggesting that psychotherapy treats poverty nor that psychotherapy should be given to non-depressed poor people. The reasons why psychotherapy can help in LMICs are (1) the lower costs and (2) the counterfactual that there are many people there who need help with mental health and that LMICs rarely have any infrastructure to help (and sometimes, when they do have some, it involves actively hurting people—like chaining people up). There is a bidirectional relationship between poverty and mental health that is complex and fascinating (Ridley et al., 2020).
With that out of the way (sorry, I know this wasn’t your point), the RCT you propose would be testing whether giving the StrongMinds sessions helps their depressed patients more than giving them the cash equivalent of that session. This is, admittedly, very interesting and can provide extra data about the effect of cash on people with depression and using a very special kind of active control. We do not have the funding to do something like that for now. Additionally, I think there might be other areas that are more in need of investigation (e.g., more research on household spillovers). Resources permitting we would be interested in conducting more RCTs with partners
Thank you for engaging with our work, I hope our answers help.
Thanks for the post
First a minor note here on this reponse here “We take these data to be our best guess because there are no prior studies of the effect of deworming on SWB, and the evidence of impact on other outcomes is very uncertain. However, all the effects are non-significant. We don’t have a theory of action because we think the overall evidence points to there being no effect (or at least just a very small one).”
Where data is as bad as this, it’s better to say that there is not enough meaningful data to draw a conclusion, rather than saying “with the best evidence we have (which is bad), conclusion is X”. It’s better to not use bad evidence at all, and say there is no meaningful evidence available, than to try and draw weak conclusions from it like you do here.
My major point—as the effect on each individual from deworming is so tiny, I don’t think you’re ever going to find a significant increase in self reported happiness on a blunt 3 point scale (like ths one) , even if deworming does cause people to be a little happier
Imagine that deworming really does increase 1000 recipient’s happiness by 2% each. This won’t tip them over the line from “happy” to “very happy” so people will report the same level of happiness. Wheras the same amount of money for Strongminds MIGHT increase 10 people’s happiness by 30% each so they would all report a higher score. So even though deworming’s small increase in happiness for more people would make deworming more WELLBY effective than Strongminds, the blunt 3 point scale can’t possibly show this. Sorry if you discussed this already and I missed it.
On the bigger WELLBY picture, the more I think about it, I think the Happier Lives institute should prioritise funding a WELLBY measured RCT of StrongMinds vs. cash transfers A.S.A.P performed by an independent organisation, to answer 2 questions.
1. If StrongMinds really are superior to cash on the WEllBY front?
2. If before and after wellbeing scores can be valid, and not spoiled by the effects I’ve discussed earlier
I was super disappointed that the current StongMinds trial doesn’t test this. I don’t like to be cynical, but I wonder if there might be fear around what the potential result of that trial might be, leading to that research not happening. At this stage StrongMinds have FAR more to lose than to gain from a head to head trial vs. cash...
Thanks team Happier Lives ;).
Hi Nick,
Thank you for your comment.
About your major point first.
If it was up to us (we didn’t collect this data), we would use a nicer 0-10 scale. However, this is the only SWB data we are aware of. There are other measures of wellbeing in the data (including a 1-10 scale, some 1-6 frequency scales, and some binary scales) but the 3-point scale is the only measure that was collected across all three KLPS rounds. None of the other measures are significant. Some are negative, some are positive. In Appendix A3.1 we conduct an analysis where we use the effect sizes of all the other measures, and we obtain very similar results, which gives us more confidence about this measure.
I’m not sure this is completely true. Some people will be tipped over the line; namely, all the people that are 2% away from changing between categories (the people who answer ‘not happy’ but will be close to answering ‘happy’).
Both the literature (that does not contain SWB data; Section 1.3) and this SWB data (the only we could find) do not give us good grounds to recommend deworming. For completeness, we go through many considerations that we present in Section 2.3 (alternative analyses, Bayes factors, cost-effectiveness, converting GiveWell’s analysis to WELLBYs, etc.). As we note in the report, this evidence does not prove the effect is zero, but these converging lines of evidence support the conclusion that the effect is zero or very small. If strong SWB evidence that deworming is more cost-effective than StrongMinds is produced, then we would change our minds. Yes, collecting this data will be expensive, but we’d prefer some of the money going to deworming serves to nail down the effect of deworming (on SWB because we think this is what ultimately matters).
Just to clarify for people reading this (I know you are not saying this, it is just that I know that people too often misunderstand this): StrongMinds gives depression treatment for people in LMIC (who happen to also be poor). GiveDirectly transfers cash to people in poverty (some of which might be depressed). If you want to increase WELLBYs, StrongMinds will produce more WELLBYs per dollar than GiveDirectly. We are not suggesting that psychotherapy treats poverty nor that psychotherapy should be given to non-depressed poor people. The reasons why psychotherapy can help in LMICs are (1) the lower costs and (2) the counterfactual that there are many people there who need help with mental health and that LMICs rarely have any infrastructure to help (and sometimes, when they do have some, it involves actively hurting people—like chaining people up). There is a bidirectional relationship between poverty and mental health that is complex and fascinating (Ridley et al., 2020).
With that out of the way (sorry, I know this wasn’t your point), the RCT you propose would be testing whether giving the StrongMinds sessions helps their depressed patients more than giving them the cash equivalent of that session. This is, admittedly, very interesting and can provide extra data about the effect of cash on people with depression and using a very special kind of active control. We do not have the funding to do something like that for now. Additionally, I think there might be other areas that are more in need of investigation (e.g., more research on household spillovers). Resources permitting we would be interested in conducting more RCTs with partners
Thank you for engaging with our work, I hope our answers help.