Hi Toni, thanks for posting. My apologies if the questions below have been answered elsewhere; I have not engaged very much with your research over the past year or so.
I’m wondering if you could provide a bit of clarification regarding the role of CEEs in the formation of your overall view of cost effectiveness. You describe CEEs and (e.g.) leadership quality as being independent features in determining the marginal effectiveness of donations. I understand “independent” here as meaning that each feature can vary as the other(s) are held constant (roughly, of course there will be some correlation between all of the criteria).
This makes sense, but I think it isn’t hitting the core of Halstead’s argument. The criteria seem to comprise a multiplicative model, where setting any of the variables to a sufficiently low value is enough to bring the estimated marginal impact close to zero. If donations don’t cash out in the implementation of cost-effective interventions, then the rest of the features don’t matter; likewise if the leadership is so poor that the organization disbands. In this view, something like CEEs are critical even if they are not “core” in a sense of being much more important than the other criteria, such that a compelling argument against the evidence of cost effectiveness does feel per se like a compelling argument against the evidence for marginal cost effectiveness.
While the criteria may vary independently, it doesn’t seem that they independently contribute to animal welfare (i.e. contribute additively rather than multiplicatively).
Also relevant are one’s prior on the cost effectiveness of animal welfare charities, and one’s confidence in arguments in the style of CEEs. If I think that most animal welfare interventions have little impact, then I need compelling evidence that a particular charity is doing better in order to form a positive overall view of the org. If my prior is more optimistic, then other considerations pointing to organizational quality will be sufficient to think the org is a worthwhile target for donations (especially if I think my ability to form accurate CEEs is weak).
Reviewing your paraphrase of Halstead’s argument:
ACE’s corporate outreach intervention report is flawed.
Corporate outreach accounts for 90% of THL’s and Animal Equality’s CEEs.
Each charity’s CEE is the primary piece of evidence that the charity improves animal welfare.
Therefore,
ACE does not provide much reason to believe that their recommended charities improve animal welfare.
It seems to me that a compellingly positive CEE, primary evidence or no, is nonetheless a necessary component in the belief that an organization will improve animal welfare, particularly if one has a pessimistic prior. As such, effectively attacking the CEE is basically decisive. I’ll note that my argument somewhat rests on a conceptual confusion: the CEE as you use it isn’t actually your estimate of cost-effectiveness, just the subset that is easily quantifiable. The argument still seems to carry given a pessimistic prior and a lack of justification for the subset of impact that is hard to quantify.
I expect you have a more optimistic prior on the benefits of animal welfare charities, and I take your point (as I understand it) that there are benefits that are hard to capture in a CEE, such as movement capacity should we find some great intervention later on, and the maintenance of enthusiastic grassroots support for use as leverage in corporate campaigns. Does ACE have material explicating and justifying its understanding of this systemic/hard-to-quantify value? Is there something else I’m missing about your argument for independence, such that arguments for the value of something like strong leadership don’t rely on/flow through arguments for the cost-effectiveness of programs?
Thanks for your post, and no worries about asking questions we’ve answered elsewhere; we have a lot of research on our website, so we don’t expect anyone to know about all of it!
When I said that we consider each criterion to be an indication of a charity’s marginal cost-effectiveness “independently” of the charity’s average cost-effectiveness, I meant that—regardless of whether the charity has a high average cost-effectiveness or not—we still consider our six other criteria to be indications of marginal cost-effectiveness. There’s no one or two (or three, or four…) criteria that we think are perfect indications of marginal cost-effectiveness, though we think that all seven of them together are a very good indication. We discuss this a bit in our page on cost-effectiveness estimates, here: https://animalcharityevaluators.org/research/methodology/our-use-of-cost-effectiveness-estimates/
I won’t write more about this right now because we actually have a forthcoming blog post about how we weigh our criteria against each other to make our recommendation decisions. It’s being edited now and then we’ll likely seek external feedback before publishing, so I’d expect it in a month or so.
“It seems to me that a compellingly positive CEE, primary evidence or no, is nonetheless a necessary component in the belief that an organization will improve welfare, particularly if one has a pessimistic prior.”
We think it’s totally possible to make well-reasoned, evidence-based decisions about how to help animals, even in the absence of quantitative CEEs. After all, we don’t even publish quantitative CEEs for some charities that we review (especially if they are working towards long-term or difficult-to-measure outcomes). Take The Good Food Institute, for example. They are one of our Top Charities, but we have not published a quantitative CEE for them. It would be very difficult for us to quantitatively estimate the good they have done so far, since they are working to change the food system in a way that could take years or even decades. Still, we think they have excellent leadership, strong strategy, a healthy culture, we think their programs are likely to have a high long-term impact, and so on. We explain why in their review, and we think we’ve provided a compelling case for donating to them based on their marginal cost-effectiveness.
Regarding your question about “material explicating and justifying [ACE’s] understanding of this systemic/hard-to-quantify value,” we explain some of our thinking about long-term outcomes on the page about our cost-effectiveness estimates, linked above. If you’re asking for explanations of our assessment of the long-term value of particular charities or interventions, that would be in each charity review (mostly discussed in the “high-impact” section with the theories of change) and in our specific intervention reports. For instance, our protest report discusses the importance of movement building.
Hope that helps to answer some of your questions, and watch our blog for the post on our weighing of each criterion!
Hi Toni, thanks for posting. My apologies if the questions below have been answered elsewhere; I have not engaged very much with your research over the past year or so.
I’m wondering if you could provide a bit of clarification regarding the role of CEEs in the formation of your overall view of cost effectiveness. You describe CEEs and (e.g.) leadership quality as being independent features in determining the marginal effectiveness of donations. I understand “independent” here as meaning that each feature can vary as the other(s) are held constant (roughly, of course there will be some correlation between all of the criteria).
This makes sense, but I think it isn’t hitting the core of Halstead’s argument. The criteria seem to comprise a multiplicative model, where setting any of the variables to a sufficiently low value is enough to bring the estimated marginal impact close to zero. If donations don’t cash out in the implementation of cost-effective interventions, then the rest of the features don’t matter; likewise if the leadership is so poor that the organization disbands. In this view, something like CEEs are critical even if they are not “core” in a sense of being much more important than the other criteria, such that a compelling argument against the evidence of cost effectiveness does feel per se like a compelling argument against the evidence for marginal cost effectiveness.
While the criteria may vary independently, it doesn’t seem that they independently contribute to animal welfare (i.e. contribute additively rather than multiplicatively).
Also relevant are one’s prior on the cost effectiveness of animal welfare charities, and one’s confidence in arguments in the style of CEEs. If I think that most animal welfare interventions have little impact, then I need compelling evidence that a particular charity is doing better in order to form a positive overall view of the org. If my prior is more optimistic, then other considerations pointing to organizational quality will be sufficient to think the org is a worthwhile target for donations (especially if I think my ability to form accurate CEEs is weak).
Reviewing your paraphrase of Halstead’s argument:
It seems to me that a compellingly positive CEE, primary evidence or no, is nonetheless a necessary component in the belief that an organization will improve animal welfare, particularly if one has a pessimistic prior. As such, effectively attacking the CEE is basically decisive. I’ll note that my argument somewhat rests on a conceptual confusion: the CEE as you use it isn’t actually your estimate of cost-effectiveness, just the subset that is easily quantifiable. The argument still seems to carry given a pessimistic prior and a lack of justification for the subset of impact that is hard to quantify.
I expect you have a more optimistic prior on the benefits of animal welfare charities, and I take your point (as I understand it) that there are benefits that are hard to capture in a CEE, such as movement capacity should we find some great intervention later on, and the maintenance of enthusiastic grassroots support for use as leverage in corporate campaigns. Does ACE have material explicating and justifying its understanding of this systemic/hard-to-quantify value? Is there something else I’m missing about your argument for independence, such that arguments for the value of something like strong leadership don’t rely on/flow through arguments for the cost-effectiveness of programs?
Hi Adom,
Thanks for your post, and no worries about asking questions we’ve answered elsewhere; we have a lot of research on our website, so we don’t expect anyone to know about all of it!
When I said that we consider each criterion to be an indication of a charity’s marginal cost-effectiveness “independently” of the charity’s average cost-effectiveness, I meant that—regardless of whether the charity has a high average cost-effectiveness or not—we still consider our six other criteria to be indications of marginal cost-effectiveness. There’s no one or two (or three, or four…) criteria that we think are perfect indications of marginal cost-effectiveness, though we think that all seven of them together are a very good indication. We discuss this a bit in our page on cost-effectiveness estimates, here: https://animalcharityevaluators.org/research/methodology/our-use-of-cost-effectiveness-estimates/
I won’t write more about this right now because we actually have a forthcoming blog post about how we weigh our criteria against each other to make our recommendation decisions. It’s being edited now and then we’ll likely seek external feedback before publishing, so I’d expect it in a month or so.
We think it’s totally possible to make well-reasoned, evidence-based decisions about how to help animals, even in the absence of quantitative CEEs. After all, we don’t even publish quantitative CEEs for some charities that we review (especially if they are working towards long-term or difficult-to-measure outcomes). Take The Good Food Institute, for example. They are one of our Top Charities, but we have not published a quantitative CEE for them. It would be very difficult for us to quantitatively estimate the good they have done so far, since they are working to change the food system in a way that could take years or even decades. Still, we think they have excellent leadership, strong strategy, a healthy culture, we think their programs are likely to have a high long-term impact, and so on. We explain why in their review, and we think we’ve provided a compelling case for donating to them based on their marginal cost-effectiveness.
Regarding your question about “material explicating and justifying [ACE’s] understanding of this systemic/hard-to-quantify value,” we explain some of our thinking about long-term outcomes on the page about our cost-effectiveness estimates, linked above. If you’re asking for explanations of our assessment of the long-term value of particular charities or interventions, that would be in each charity review (mostly discussed in the “high-impact” section with the theories of change) and in our specific intervention reports. For instance, our protest report discusses the importance of movement building.
Hope that helps to answer some of your questions, and watch our blog for the post on our weighing of each criterion!
Best, Toni