For Kaya Guides, this has got us thinking much more explicitly about what we’re comparing to. GiveWell and GiveDirectly have a lot more resources, so they can do things like go out to communities and measure second order and spillover effects.
On the one hand, this has got us thinking about other impacts we can incorporate into our analyses. Like GiveDirectly, we probably also have community spillover effects, we probably also avert deaths, and we probably also increase our beneficiaries’ incomes by improving productivity. I suspect this is true for many GHD charities!
On the other, it doesn’t seem fair to compare our analysis on individual subjective wellbeing to GiveDirectly’s analysis that incorporates many more things. Unless we believed that GiveDirectly is likely to be systematically better, it’s not the case that many GHD charities got 3–4× less cost-effective relative to cash transfers overnight, they may just count 3–4× less things! So I wonder if the standard cash transfers benchmark might have to include more nuance in the near-term. Kaya Guides already only makes claims about cost-effectiveness ‘at improving subjective wellbeing’ to try and cover for this.
Are other GHD charities starting to think the same way? Do people have other angles on this?
I wouldn’t update too strongly on this single comparison, and I don’t know if there are better analyses of spillover effects for different kinds of interventions, but it seems that there are reasons to believe that spillover effects from cash transfers are relatively greater than for other interventions.
I agree to your general analysis. Just some quick thoughts (if you have other ideas, I’d be excited to hear them!) For us at ACTRA, communicating our benefits in terms of x times cash has been very helpful, because we do not have the typical “one outcome measure makes up for 80% of the effect” situation: Crime reduction leads to reduced interpersonal violence like homicides (DALYs), averts economic damage ($) and has wellbeing benefits (WELLBYs). Our current analysis suggests, that each of those three is about 1⁄3 of our total effect. So having something, where we cann sum these benefits up and then compare to other charities that might focus on only one of these is very helpful for us. If we want to keep using cash estimates I see us either 1) incorporating best guesses for our spillover effects etc. in our CEA based on the scientific literature available, discounting them somewhat but not too heavily, to be comparable to the Give Directly estimate 2) Consciously deciding not to include spillover effects etc. and then comparing “apples to apples”. So if there was a simple table saying “this is the direct Give Directly effect, this is the additional spillover effect and this is X, Y, and Z”, then we can basically produce the same table for our charity with some fields saying n/a and that would make it comparable somehow…
2 weeks out from the new GiveWell/GiveDirectly analysis, I was wondering how GHD charities are evaluating the impact of these results.
For Kaya Guides, this has got us thinking much more explicitly about what we’re comparing to. GiveWell and GiveDirectly have a lot more resources, so they can do things like go out to communities and measure second order and spillover effects.
On the one hand, this has got us thinking about other impacts we can incorporate into our analyses. Like GiveDirectly, we probably also have community spillover effects, we probably also avert deaths, and we probably also increase our beneficiaries’ incomes by improving productivity. I suspect this is true for many GHD charities!
On the other, it doesn’t seem fair to compare our analysis on individual subjective wellbeing to GiveDirectly’s analysis that incorporates many more things. Unless we believed that GiveDirectly is likely to be systematically better, it’s not the case that many GHD charities got 3–4× less cost-effective relative to cash transfers overnight, they may just count 3–4× less things! So I wonder if the standard cash transfers benchmark might have to include more nuance in the near-term. Kaya Guides already only makes claims about cost-effectiveness ‘at improving subjective wellbeing’ to try and cover for this.
Are other GHD charities starting to think the same way? Do people have other angles on this?
Don’t know if this is useful, but years ago HLI tried to estimate spillover effects from therapy in Happiness for the whole household: accounting for household spillovers when comparing the cost-effectiveness of psychotherapy to cash transfers, and already found that spillover effects were likely significantly higher for cash transfers compared to therapy.
In 2023 in Talking through depression: The cost-effectiveness of psychotherapy in LMICs, revised and expanded they estimated that the difference is even greater in favour of cash transfers. (after feedback like Why I don’t agree with HLI’s estimate of household spillovers from therapy and Assessment of Happier Lives Institute’s Cost-Effectiveness Analysis of StrongMinds)
I wouldn’t update too strongly on this single comparison, and I don’t know if there are better analyses of spillover effects for different kinds of interventions, but it seems that there are reasons to believe that spillover effects from cash transfers are relatively greater than for other interventions.
I agree to your general analysis. Just some quick thoughts (if you have other ideas, I’d be excited to hear them!)
For us at ACTRA, communicating our benefits in terms of x times cash has been very helpful, because we do not have the typical “one outcome measure makes up for 80% of the effect” situation: Crime reduction leads to reduced interpersonal violence like homicides (DALYs), averts economic damage ($) and has wellbeing benefits (WELLBYs). Our current analysis suggests, that each of those three is about 1⁄3 of our total effect. So having something, where we cann sum these benefits up and then compare to other charities that might focus on only one of these is very helpful for us.
If we want to keep using cash estimates I see us either
1) incorporating best guesses for our spillover effects etc. in our CEA based on the scientific literature available, discounting them somewhat but not too heavily, to be comparable to the Give Directly estimate
2) Consciously deciding not to include spillover effects etc. and then comparing “apples to apples”. So if there was a simple table saying “this is the direct Give Directly effect, this is the additional spillover effect and this is X, Y, and Z”, then we can basically produce the same table for our charity with some fields saying n/a and that would make it comparable somehow…