First, we want to sincerely thank Giving What We Can for running this “Evaluating the Evaluators” exercise. We recognize that ACE has set ourselves a difficult task, compounded by the fact that we’re the only organization doing what we do. Therefore, receiving this kind of feedback is both very rare and very welcome. There’s a great deal in GWWC’s report that will help us improve our processes for 2024, which ultimately means more animals will be helped and spared. While we were disappointed that GWWC has decided not to defer to our recommendations this year or recommend our Movement Grants program as a top-rated fund, we were heartened by the positive points in their report and their optimism about ACE’s future, and look forward to receiving further helpful feedback in a future evaluation from them. We were also delighted that GWWC recommended the EA Animal Welfare Fund as an effective giving opportunity.
Second, as an organization that values transparency and seeks to be open about our own limitations, we appreciated GWWC’s same openness about the limitations to, and uncertainties around, their evaluation. As they noted, this included limitations to their animal welfare expertise, the early stage of the charity evaluation space in animal welfare, and the time constraints forcing them to take a minimum viable product approach to this evaluation. The bulk of this year’s process also coincided with the culmination of our charity recommendation decisions—which, combined with GWWC’s demanding deadlines, made some aspects of the process challenging for us. GWWC fully recognized this, and we are confident that any future evaluation exercise will be even more helpful than this year’s.
Third, we were reassured that much of GWWC’s constructive feedback aligns with ACE’s own self-identified areas for improvement. For example, we agree that we need to continuously assess whether we want to give out fewer, larger Movement Grants than we do currently. We also agree that we should be more strategic in using the valuable feedback we get from our Movement Grants grantees to inform our own views on priority tactics and translate this into useful information for the broader animal advocacy movement.
We are also continuously working toward improvements to our charity evaluation methods, such as how to more sensitively capture differences in scope. As in previous years, we will be conducting a thorough review of the top-priority improvements to make to next year’s Movement Grants and Charity Evaluations programs. We will certainly draw on GWWC’s feedback for this while also acknowledging that capacity constraints will inevitably make it impossible to make all of the improvements we would like.
Fourth, there are some elements of GWWC’s report that we did not fully agree with. For example, while we agree that there’s plenty of opportunity for improvements to our Cost Effectiveness model to ensure that it reflects the cost effectiveness of charities’ achievements as accurately as possible, we would like to highlight that this year’s model is the result of considerable research, external guidance, and exploration of alternatives. We built this year’s model systematically to try to capture the most important aspects of what makes achievements impactful, based on empirical evidence wherever possible, and the quantitative metric closest to impact on animals that we could access for all charities’ achievements (e.g., the number of people reached per dollar for an educational campaign). We consulted with several external experts on how to best combine these scores into a single score and went through several iterations to ensure that the scores held up in confidence checks. We also think it’s likely that GWWC is overestimating how easy it is to deliver on their recommendation of reliably estimating the “marginal cost effectiveness of a dollar spent on the charity, based on the charity’s specific context.” In some past rounds of our Charity Evaluations program, ACE carried out back-of-the-envelope-calculation (BOTEC)-type cost-effectiveness modeling using Guesstimate, which aligns with GWWC’s recommendation. However, ACE then decided to change this approach for the reasons outlined here. We continuously review this decision and are open to reintroducing elements of our past approach if we determine it would be valuable for more effectively advancing our theory of change.
As another example of disagreement, GWWC noted that they would prefer our Movement Grants program to focus exclusively on national or international projects rather than the types of regional projects we have sometimes funded. However, given that one of the aims of the Movement Grants program is to build up the movement in priority regions with a relatively small animal advocacy movement, and that for some regions, we receive significantly more applications for regional projects than for high-quality, tractable, national-level applications, we expect that we will continue funding some regional projects that we consider particularly promising. Relatedly, GWWC disagrees with our view that funding projects in countries with very little animal advocacy representation should be a key part of ensuring the animal advocacy movement’s long-term success, noting that others may reasonably be more sympathetic to this view.
Fifth, we share GWWC’s commitment to prioritizing marginal funding to the projects where it will be the most cost effective. However, elements of ACE’s work—none of which are unique to us—make this particularly complex to achieve in practice. With our charity evaluations, for example, we only re-evaluate charities every two years, and we do not directly disburse most of the funding that we influence to our Recommended Charities. As such, ACE does not regularly calculate cost effectiveness over a range of possible allocations and distributes funding only to those above a particular effectiveness bar; instead, we recommend a set of charities that we are confident will put additional funding toward effective use to help animals over a longer time horizon without our oversight. We make this decision based on all of our evaluation criteria, with a strong focus on Cost Effectiveness (which examines the effectiveness of past work) and Room For More Funding (which assesses whether a charity’s planned uses for funding over the next two years will be roughly similar to their past work).
It is also worth noting that because we have shifted to one recommendation level for our charities this year (as opposed to the previous Top Charities/Standout Charities distinction), we plan to develop a new decision-making process that better accounts for the marginal cost effectiveness of funding disbursed from ACE’s Recommended Charity Fund. This will better enable us to leverage our grantmaking role in addition to our recommendation role.
For our Movement Grants—especially smaller projects, projects benefiting species for whom few interventions have been tried, and projects in regions where the movement is particularly small, so there’s particularly little evidence—it is not currently possible to sufficiently investigate each project application to make meaningful cost-effectiveness estimates. We, therefore, rely on proxies such as the coherence of an applicant’s theory of change, the priority of their focus animal groups, and the neglectedness of the region in which they operate.
Additionally, because we believe that supporting a range of different approaches is essential for an effective animal advocacy ecosystem (and because our recommendations influence donor and public opinion), we want to feature a plurality of approaches with a strong potential for impact. Because of the high uncertainty about the most effective ways to help animals, and because the different interventions reinforce and facilitate each other, we think supporting a range of approaches is both necessary and beneficial for the animal advocacy ecosystem as a whole. We view this as preferable to diverting funding near-exclusively to charities and programs with the most convincing shorter-term theories of change, which runs the risk of dismissing potentially pivotal interventions due to measurability bias.
Sixth, we are glad that GWWC chose to recommend The Humane League (THL) based in part on our evaluation. As we note in our 2023 review, we view giving to THL as an excellent opportunity to support initiatives that create the most positive change for animals. At the same time, we are disappointed that GWWC chose to only recommend one of our Recommended Charities and to restrict grants for their corporate campaign work. After months of evaluation, we are confident that all of our Recommended Charities represent highly promising giving opportunities. We are also convinced of the need for a pluralistic and resilient movement incorporating a range of effective tactics toward different outcomes to achieve wellbeing for as many animals as soon as possible globally.
We also want to make clear that while GWWC’s decision might imply that THL represents a superior giving opportunity compared to our other recommended charities, this is not a view that ACE shares. We view all of our recommended charities, including THL, as highly impactful giving opportunities. Following GWWC’s initial conclusion that they weren’t going to defer to our overall charity recommendations this year, we would have welcomed the opportunity to provide supporting materials for more than three of our Recommended Charities and to have had more time to do so.
Lastly, and most importantly, we congratulate, once more, the latest additions to our list of Recommended Charities. We will diligently continue working to ensure that our approach to evaluation and our methods capture the full extent of charities’ work as accurately as possible, and we expect these improvements to be an ongoing endeavor that will continue for as long as ACE exists. At the same time, following months of research, preparation, and evaluation, we are confident that Kafessiz Türkiye, Dansk Vegetarisk Forening, Faunalytics, Fish Welfare Initiative, Good Food Institute, Legal Impact for Chickens, New Roots Institute, Shrimp Welfare Project, Sinergia Animal, The Humane League, and Wild Animal Initiative all do incredible work and represent extremely promising giving opportunities. We are pleased that GWWC’s report recognizes this, both through their direct recommendation of our Recommended Charity The Humane League, and through their strong recommendation to impact-maximizing donors to give to ACE’s recommended charities over the average animal welfare charity.
It’s worth pointing out that ACE’s estimates/models (mostly weighted factor models, including ACE’s versions of Scale-Tractability-Neglectedness, or STN) are often already pretty close to being BOTECs, but aren’t quite BOTECs. I’d guess the smallest fixes to make them more scope-sensitive are to just turn them into BOTECs, or whatever parts of them you can into BOTECs[1], whenever not too much extra work. BOTECs and other quantitative models force you to pick factors, and scale and combine them in ways that are more scope-sensitive.
For the cost-effective criterion, ACE makes judgements about the quality of charities’ achievements with Achievement Quality Scores. For corporate outreach and producer outreach, ACE already scores factors from which direct average impact BOTECs could pretty easily be done with some small changes, which I’d recommend:
Score “Scale (1-7)” = “How many locations and animals are estimated to be affected by the commitments/campaign, if successful?” in terms of the number of animals (or animal life-years) per year of counterfactual impact instead of 1-7.
Ideally, “Impact on animals (1-7)” should be scored quantitatively using Welfare Footprint Project’s approach (some rougher estimates here and here) instead of 1-7, but this is a lower priority than other changes. Welfare improvements per animal or per year of animal life can probably vary much more than 7 times, though, and can end up negative instead, so I’d probably at least adjust the range to be symmetric around 0 and let researchers select 0 or values very close to it.
The BOTEC is then just the product of “Impact on animals (1-7)” (the average[2] welfare improvement with successful implementation), “Scale”, “Likelihood of implementation (%)”, expected welfare range and the number of years of counterfactual impact (until similar welfare improvements for the animals would have happened anyway and made these redundant). Similar BOTECs could be done for the direct impacts of other interventions.
For groups aiming to impact decision-making or funding in the near term with research like Faunalytics, ACE could also highlight some of the most important decisions that have been (or are not too unlikely to be) informed by their research so that we can independently judge how they compare to corporate outreach or other interventions. ACE could also use RP’s model or something similar to get impact BOTECs to make comparisons with more direct work.
For other charities, ACE could also think about how to turn the models into BOTECs or quantitative models of important outcomes. These can be intermediate outcomes or outputs that aren’t necessarily comparable across all interventions, if impact for animals is too speculative, but the potential upside is high enough and the potential downside small enough.[1]
For the Impact Potential criterion, ACE uses STN a lot and cites the 80,000 Hours’ article where 80,000 Hours explains how to get a BOTEC by interpreting and scoring the factors in specific ways. ACE could just follow that procedure and then the STN estimates would be BOTECs.
That being said, STN is really easy to misapply generally (e.g. various critiques here), and I’d be careful about relying on it even if you were to follow 80,000 Hours’ procedure to get BOTECs. For example, only a tiny share of a huge but relatively intractable problem, like wild animal welfare/suffering, may be at all tractable, so it’s easy to overestimate the combination of Scale and Tractability in those cases. See also Joey’s Why we look at the limiting factor instead of the problem scale and Saulius’s Why I No Longer Prioritize Wild Animal Welfare. STN can be useful for guiding what to investigate further and filtering charities for review, but I’d probably go for BOTECs done other ways, like above to replace Achievement Quality Scores, and with more detailed theories of change.
For example, you could do a BOTEC of the number of additional engagement-weighted animal advocates, which could be part of a BOTEC for impact on animals, but going from engagement-weighted animal advocates to animals could be too speculative, so you stop at engagement-weighted animal advocates. This could be refined further, weighing by country scores.
Hi Michael, thanks a lot for the helpful comments, and for taking the time to be so thorough in your feedback. We’ve been thinking a lot about how to produce proxies for impact that can be meaningfully compared with one another, with BOTECs being one possible way to help achieve that, so it’s really useful to get your views. We’ll talk these through as a team as we consider improvements to our process for the coming years.
Thank you! As we mention in the report, we’re grateful for how you’ve engaged with our evaluations process, and I think this comment is a good illustration of the open, constructive and collaborative attitude you’ve had throughout it. We look forward to re-evaluating ACE’s work next year, and in the meantime remain excited to host many of ACE’s funds and recommendations on our donation platform as promising opportunities for donors to consider.
First, we want to sincerely thank Giving What We Can for running this “Evaluating the Evaluators” exercise. We recognize that ACE has set ourselves a difficult task, compounded by the fact that we’re the only organization doing what we do. Therefore, receiving this kind of feedback is both very rare and very welcome. There’s a great deal in GWWC’s report that will help us improve our processes for 2024, which ultimately means more animals will be helped and spared. While we were disappointed that GWWC has decided not to defer to our recommendations this year or recommend our Movement Grants program as a top-rated fund, we were heartened by the positive points in their report and their optimism about ACE’s future, and look forward to receiving further helpful feedback in a future evaluation from them. We were also delighted that GWWC recommended the EA Animal Welfare Fund as an effective giving opportunity.
Second, as an organization that values transparency and seeks to be open about our own limitations, we appreciated GWWC’s same openness about the limitations to, and uncertainties around, their evaluation. As they noted, this included limitations to their animal welfare expertise, the early stage of the charity evaluation space in animal welfare, and the time constraints forcing them to take a minimum viable product approach to this evaluation. The bulk of this year’s process also coincided with the culmination of our charity recommendation decisions—which, combined with GWWC’s demanding deadlines, made some aspects of the process challenging for us. GWWC fully recognized this, and we are confident that any future evaluation exercise will be even more helpful than this year’s.
Third, we were reassured that much of GWWC’s constructive feedback aligns with ACE’s own self-identified areas for improvement. For example, we agree that we need to continuously assess whether we want to give out fewer, larger Movement Grants than we do currently. We also agree that we should be more strategic in using the valuable feedback we get from our Movement Grants grantees to inform our own views on priority tactics and translate this into useful information for the broader animal advocacy movement.
We are also continuously working toward improvements to our charity evaluation methods, such as how to more sensitively capture differences in scope. As in previous years, we will be conducting a thorough review of the top-priority improvements to make to next year’s Movement Grants and Charity Evaluations programs. We will certainly draw on GWWC’s feedback for this while also acknowledging that capacity constraints will inevitably make it impossible to make all of the improvements we would like.
Fourth, there are some elements of GWWC’s report that we did not fully agree with. For example, while we agree that there’s plenty of opportunity for improvements to our Cost Effectiveness model to ensure that it reflects the cost effectiveness of charities’ achievements as accurately as possible, we would like to highlight that this year’s model is the result of considerable research, external guidance, and exploration of alternatives. We built this year’s model systematically to try to capture the most important aspects of what makes achievements impactful, based on empirical evidence wherever possible, and the quantitative metric closest to impact on animals that we could access for all charities’ achievements (e.g., the number of people reached per dollar for an educational campaign). We consulted with several external experts on how to best combine these scores into a single score and went through several iterations to ensure that the scores held up in confidence checks. We also think it’s likely that GWWC is overestimating how easy it is to deliver on their recommendation of reliably estimating the “marginal cost effectiveness of a dollar spent on the charity, based on the charity’s specific context.” In some past rounds of our Charity Evaluations program, ACE carried out back-of-the-envelope-calculation (BOTEC)-type cost-effectiveness modeling using Guesstimate, which aligns with GWWC’s recommendation. However, ACE then decided to change this approach for the reasons outlined here. We continuously review this decision and are open to reintroducing elements of our past approach if we determine it would be valuable for more effectively advancing our theory of change.
As another example of disagreement, GWWC noted that they would prefer our Movement Grants program to focus exclusively on national or international projects rather than the types of regional projects we have sometimes funded. However, given that one of the aims of the Movement Grants program is to build up the movement in priority regions with a relatively small animal advocacy movement, and that for some regions, we receive significantly more applications for regional projects than for high-quality, tractable, national-level applications, we expect that we will continue funding some regional projects that we consider particularly promising. Relatedly, GWWC disagrees with our view that funding projects in countries with very little animal advocacy representation should be a key part of ensuring the animal advocacy movement’s long-term success, noting that others may reasonably be more sympathetic to this view.
Fifth, we share GWWC’s commitment to prioritizing marginal funding to the projects where it will be the most cost effective. However, elements of ACE’s work—none of which are unique to us—make this particularly complex to achieve in practice. With our charity evaluations, for example, we only re-evaluate charities every two years, and we do not directly disburse most of the funding that we influence to our Recommended Charities. As such, ACE does not regularly calculate cost effectiveness over a range of possible allocations and distributes funding only to those above a particular effectiveness bar; instead, we recommend a set of charities that we are confident will put additional funding toward effective use to help animals over a longer time horizon without our oversight. We make this decision based on all of our evaluation criteria, with a strong focus on Cost Effectiveness (which examines the effectiveness of past work) and Room For More Funding (which assesses whether a charity’s planned uses for funding over the next two years will be roughly similar to their past work).
It is also worth noting that because we have shifted to one recommendation level for our charities this year (as opposed to the previous Top Charities/Standout Charities distinction), we plan to develop a new decision-making process that better accounts for the marginal cost effectiveness of funding disbursed from ACE’s Recommended Charity Fund. This will better enable us to leverage our grantmaking role in addition to our recommendation role.
For our Movement Grants—especially smaller projects, projects benefiting species for whom few interventions have been tried, and projects in regions where the movement is particularly small, so there’s particularly little evidence—it is not currently possible to sufficiently investigate each project application to make meaningful cost-effectiveness estimates. We, therefore, rely on proxies such as the coherence of an applicant’s theory of change, the priority of their focus animal groups, and the neglectedness of the region in which they operate.
Additionally, because we believe that supporting a range of different approaches is essential for an effective animal advocacy ecosystem (and because our recommendations influence donor and public opinion), we want to feature a plurality of approaches with a strong potential for impact. Because of the high uncertainty about the most effective ways to help animals, and because the different interventions reinforce and facilitate each other, we think supporting a range of approaches is both necessary and beneficial for the animal advocacy ecosystem as a whole. We view this as preferable to diverting funding near-exclusively to charities and programs with the most convincing shorter-term theories of change, which runs the risk of dismissing potentially pivotal interventions due to measurability bias.
Sixth, we are glad that GWWC chose to recommend The Humane League (THL) based in part on our evaluation. As we note in our 2023 review, we view giving to THL as an excellent opportunity to support initiatives that create the most positive change for animals. At the same time, we are disappointed that GWWC chose to only recommend one of our Recommended Charities and to restrict grants for their corporate campaign work. After months of evaluation, we are confident that all of our Recommended Charities represent highly promising giving opportunities. We are also convinced of the need for a pluralistic and resilient movement incorporating a range of effective tactics toward different outcomes to achieve wellbeing for as many animals as soon as possible globally.
We also want to make clear that while GWWC’s decision might imply that THL represents a superior giving opportunity compared to our other recommended charities, this is not a view that ACE shares. We view all of our recommended charities, including THL, as highly impactful giving opportunities. Following GWWC’s initial conclusion that they weren’t going to defer to our overall charity recommendations this year, we would have welcomed the opportunity to provide supporting materials for more than three of our Recommended Charities and to have had more time to do so.
Lastly, and most importantly, we congratulate, once more, the latest additions to our list of Recommended Charities. We will diligently continue working to ensure that our approach to evaluation and our methods capture the full extent of charities’ work as accurately as possible, and we expect these improvements to be an ongoing endeavor that will continue for as long as ACE exists. At the same time, following months of research, preparation, and evaluation, we are confident that Kafessiz Türkiye, Dansk Vegetarisk Forening, Faunalytics, Fish Welfare Initiative, Good Food Institute, Legal Impact for Chickens, New Roots Institute, Shrimp Welfare Project, Sinergia Animal, The Humane League, and Wild Animal Initiative all do incredible work and represent extremely promising giving opportunities. We are pleased that GWWC’s report recognizes this, both through their direct recommendation of our Recommended Charity The Humane League, and through their strong recommendation to impact-maximizing donors to give to ACE’s recommended charities over the average animal welfare charity.
-ACE Team
It’s worth pointing out that ACE’s estimates/models (mostly weighted factor models, including ACE’s versions of Scale-Tractability-Neglectedness, or STN) are often already pretty close to being BOTECs, but aren’t quite BOTECs. I’d guess the smallest fixes to make them more scope-sensitive are to just turn them into BOTECs, or whatever parts of them you can into BOTECs[1], whenever not too much extra work. BOTECs and other quantitative models force you to pick factors, and scale and combine them in ways that are more scope-sensitive.
For the cost-effective criterion, ACE makes judgements about the quality of charities’ achievements with Achievement Quality Scores. For corporate outreach and producer outreach, ACE already scores factors from which direct average impact BOTECs could pretty easily be done with some small changes, which I’d recommend:
Score “Scale (1-7)” = “How many locations and animals are estimated to be affected by the commitments/campaign, if successful?” in terms of the number of animals (or animal life-years) per year of counterfactual impact instead of 1-7.
Ideally, “Impact on animals (1-7)” should be scored quantitatively using Welfare Footprint Project’s approach (some rougher estimates here and here) instead of 1-7, but this is a lower priority than other changes. Welfare improvements per animal or per year of animal life can probably vary much more than 7 times, though, and can end up negative instead, so I’d probably at least adjust the range to be symmetric around 0 and let researchers select 0 or values very close to it.
The BOTEC is then just the product of “Impact on animals (1-7)” (the average[2] welfare improvement with successful implementation), “Scale”, “Likelihood of implementation (%)”, expected welfare range and the number of years of counterfactual impact (until similar welfare improvements for the animals would have happened anyway and made these redundant). Similar BOTECs could be done for the direct impacts of other interventions.
For groups aiming to impact decision-making or funding in the near term with research like Faunalytics, ACE could also highlight some of the most important decisions that have been (or are not too unlikely to be) informed by their research so that we can independently judge how they compare to corporate outreach or other interventions. ACE could also use RP’s model or something similar to get impact BOTECs to make comparisons with more direct work.
For other charities, ACE could also think about how to turn the models into BOTECs or quantitative models of important outcomes. These can be intermediate outcomes or outputs that aren’t necessarily comparable across all interventions, if impact for animals is too speculative, but the potential upside is high enough and the potential downside small enough.[1]
For the Impact Potential criterion, ACE uses STN a lot and cites the 80,000 Hours’ article where 80,000 Hours explains how to get a BOTEC by interpreting and scoring the factors in specific ways. ACE could just follow that procedure and then the STN estimates would be BOTECs.
That being said, STN is really easy to misapply generally (e.g. various critiques here), and I’d be careful about relying on it even if you were to follow 80,000 Hours’ procedure to get BOTECs. For example, only a tiny share of a huge but relatively intractable problem, like wild animal welfare/suffering, may be at all tractable, so it’s easy to overestimate the combination of Scale and Tractability in those cases. See also Joey’s Why we look at the limiting factor instead of the problem scale and Saulius’s Why I No Longer Prioritize Wild Animal Welfare. STN can be useful for guiding what to investigate further and filtering charities for review, but I’d probably go for BOTECs done other ways, like above to replace Achievement Quality Scores, and with more detailed theories of change.
For example, you could do a BOTEC of the number of additional engagement-weighted animal advocates, which could be part of a BOTEC for impact on animals, but going from engagement-weighted animal advocates to animals could be too speculative, so you stop at engagement-weighted animal advocates. This could be refined further, weighing by country scores.
Per animal or per animal life-year, to match Scale.
It seems ACE did so for the Scale factor, but no specific quantitative interpretation for the others.
Hi Michael, thanks a lot for the helpful comments, and for taking the time to be so thorough in your feedback. We’ve been thinking a lot about how to produce proxies for impact that can be meaningfully compared with one another, with BOTECs being one possible way to help achieve that, so it’s really useful to get your views. We’ll talk these through as a team as we consider improvements to our process for the coming years.
- Max
Thank you! As we mention in the report, we’re grateful for how you’ve engaged with our evaluations process, and I think this comment is a good illustration of the open, constructive and collaborative attitude you’ve had throughout it. We look forward to re-evaluating ACE’s work next year, and in the meantime remain excited to host many of ACE’s funds and recommendations on our donation platform as promising opportunities for donors to consider.