Thanks for putting this together, I think this is an exciting report and project.
I mostly agree with Habryka’s points.
I have another minor point:
We think there is a small-to-moderate chance that CAP would generate several very impactful indirect benefits. For example, the additional donations going to animal-focused charities may reduce the risk of global pandemics caused by antibiotic resistance, and the program may help create a broader culture of effective giving at US workplaces.
I feel like it’s odd to categorize the former example as “indirect benefits”. I think a cost-effectiveness model should aim to capture the overall expected impact of all the charities by applying some “impact-adjusted money moved” metric. (If you’re evaluating from a long-termist perspective, this would mean a long-termist perspective on all supported charities.) Otherwise, any project that involves some amount of leverage on various other organizations will always have high indirect benefits and harms, which makes the overall rating non-informative.
I agree that “help create a broader culture of effective giving at US workplaces” is a good example of an indirect benefit.
For instance, charities that reduce poverty and disease may cause economic growth, which is likely to increase the number of animals raised in factory farms and could contribute to climate change and existential risks.
Again, the same points seem to hold here; I think this should already be factored into the cost-effectiveness estimates.
(I realize my explanation of my view is a bit vague; I have a pretty strong intuition here and it would take me more time to think about it more and really explain it in depth.)
Thanks Jonas. Sorry for the slow reply—I took some time off and am now travelling.
I was also uneasy about counting indirect effects of the beneficiary charities rather than CAP itself, though I’m not sure it’s for exactly the same reasons as you. The measure of benefit in the CEA was dollars counterfactually moved to the recommended charities (or equivalent), which seems to implicitly cover all consequences of those dollars. Considering indirect effects separately, with the implication that these consequences are in addition to the dollars moved, risks double-counting.
I’m not sure if that’s what you’re getting at, or if you also think we should be explicitly modelling the indirect effects of all the beneficiary charities in the CEA. I don’t think the latter is really feasible, for several reasons. For a start, we would need some universal metric for capturing both the direct and indirect consequences. The best one would probably be subjective wellbeing, but figuring out the SWB associated with a wide range of circumstances, across a large number of species, is not within the scope of an evaluation like this. Another issue is that indirect effects would likely overwhelm the direct ones, and it isn’t clear that this is appropriate given their far more speculative nature. There would probably have to be some kind of weighting system that discounts the more uncertain predictions (could be a fully Bayesian approach, or a simpler option like this), but implementing such a system itself would be very hard to do well, and perhaps less useful than highlighting them and scoring them much more subjectively as we’ve done here.
I would add that one advantage of explicitly considering the indirect effects of charities is that it makes them more salient. I’d imagine most donors/funders would only really think of the direct benefits when deciding whether a project like CAP (or the charities themselves) is worthwhile, so it helps to highlight the other issues at stake. This consideration may outweigh the methodological concerns with ‘double-counting’ etc.
Thanks for putting this together, I think this is an exciting report and project.
I mostly agree with Habryka’s points.
I have another minor point:
I feel like it’s odd to categorize the former example as “indirect benefits”. I think a cost-effectiveness model should aim to capture the overall expected impact of all the charities by applying some “impact-adjusted money moved” metric. (If you’re evaluating from a long-termist perspective, this would mean a long-termist perspective on all supported charities.) Otherwise, any project that involves some amount of leverage on various other organizations will always have high indirect benefits and harms, which makes the overall rating non-informative.
I agree that “help create a broader culture of effective giving at US workplaces” is a good example of an indirect benefit.
Again, the same points seem to hold here; I think this should already be factored into the cost-effectiveness estimates.
(I realize my explanation of my view is a bit vague; I have a pretty strong intuition here and it would take me more time to think about it more and really explain it in depth.)
Thanks Jonas. Sorry for the slow reply—I took some time off and am now travelling.
I was also uneasy about counting indirect effects of the beneficiary charities rather than CAP itself, though I’m not sure it’s for exactly the same reasons as you. The measure of benefit in the CEA was dollars counterfactually moved to the recommended charities (or equivalent), which seems to implicitly cover all consequences of those dollars. Considering indirect effects separately, with the implication that these consequences are in addition to the dollars moved, risks double-counting.
I’m not sure if that’s what you’re getting at, or if you also think we should be explicitly modelling the indirect effects of all the beneficiary charities in the CEA. I don’t think the latter is really feasible, for several reasons. For a start, we would need some universal metric for capturing both the direct and indirect consequences. The best one would probably be subjective wellbeing, but figuring out the SWB associated with a wide range of circumstances, across a large number of species, is not within the scope of an evaluation like this. Another issue is that indirect effects would likely overwhelm the direct ones, and it isn’t clear that this is appropriate given their far more speculative nature. There would probably have to be some kind of weighting system that discounts the more uncertain predictions (could be a fully Bayesian approach, or a simpler option like this), but implementing such a system itself would be very hard to do well, and perhaps less useful than highlighting them and scoring them much more subjectively as we’ve done here.
I would add that one advantage of explicitly considering the indirect effects of charities is that it makes them more salient. I’d imagine most donors/funders would only really think of the direct benefits when deciding whether a project like CAP (or the charities themselves) is worthwhile, so it helps to highlight the other issues at stake. This consideration may outweigh the methodological concerns with ‘double-counting’ etc.