At Animal Charity Evaluators, we find and promote the most effective ways to help animals. We use effective altruism principles to evaluate causes and research.
Animal Charity Evaluators
Karma: 1,385
Note: this comment has been posted in response to both Vetted Causes’ first and second posts about Sinergia, as it addresses points raised in each.
ACE welcomes rigorous external evaluations of our work, as such feedback strengthens our ability to provide high-quality recommendations to donors and, ultimately, helps us maximize our impact for animals. To this end, we engage with external experts on a) our methodology for evaluations and grantmaking and b) on evaluations and grant reviews themselves. For instance, we interact with the EA and FAST forums and the Hive Slack channel, taking advantage of both solicited and unsolicited feedback. The strongest example of our commitment to engaging with feedback lies in GWWC’s evaluation of our programs.
The recent post by Vetted Causes is no exception. However, to balance the opportunity cost of engaging with this feedback with its potential to increase the quality of our work, we will first address the content of the concerns raised. We will then offer some suggestions for improvement of future reviews by Vetted Causes.
Regarding Vetted Causes’ specific claims about Sinergia, most of those claims have been addressed by Sinergia directly. We’d also like to note that while Vetted Causes refers to “ACE/Sinergia”, we’d prefer that we be treated as separate entities. Therefore, there are several notes we’d like to add that refer to ACE’s work specifically:
Issue 1
Alleged Commitment 1: Vetted Causes states, “in 2024, JBS stated that they still use ear notching due to ‘Difficulty in finding alternatives that ensure process traceability.’”
This report came out after ACE’s evaluation of Sinergia was completed, so it did not inform any part of our evaluation. Sinergia addressed the specifics of this claim in their response.
Issue 2
Alleged Commitment 2: Vetted Causes states, “Sinergia claims that in 2023, JBS published a commitment to not use gestation crates in all new projects, with a ‘Transition deadline’ of 2023.
As evidence for this claim, Sinergia provided a link to one of the JBS’s animal welfare pages. However, the gestation crate policy that the alleged commitment references was already listed on JBS’s website in 2020, and has been in effect since that point.”
In their second post about Sinergia, Vetted Causes also states, “Although Sinergia downplays this as a ‘minor mistake,’ it results in Sinergia receiving credit for helping millions of JBS’s pigs who were not impacted. This is not a ‘minor mistake.’ Further, Sinergia claims that this mistake has been corrected, but all that was fixed was changing the phrase ‘by 2023’ to ‘in 2023’ in Cell K10. The impact calculations were not fixed, and still incorrectly credit Sinergia for helping millions of JBS’s pigs who were not impacted.”
As Sinergia notes in their response, the number of sows reported for JBS includes those whose welfare may be impacted by the enforcement of the previous commitment.
In the process of double-checking our calculations with Sinergia, we noticed the same miscommunication that would have caused changes to our calculations. Although it would have been ideal to apply discounts at both the “number of animals affected” and the “SADs averted per dollar” levels, our SADs estimates were already conservative, and the magnitude of the change would not affect our decision to recommend Sinergia. Therefore, we are currently opting not to spend the time refreshing these calculations, as the changes could be quite involved.
Issue 3
Sinergia Claims That With $1 They can “Liberat[e]” 354 Piglets from “Brutal Confinement”
Vetted Causes states that, “ACE gave Sinergia credit for helping over 30 million female piglets through surgical castration commitments that Sinergia allegedly secured.”
Out of the two figures that ACE publishes for Sinergia’s pig welfare program (SADs averted per dollar and piglets affected per dollar), the former is correct because Ambitious Impact’s SADs calculations already take into consideration the “prevalence” of castration (which only occurs in male piglets). However, the latter indeed requires the same prevalence discount.
We have adjusted the estimates in the CEA spreadsheet for Sinergia’s pig welfare program. As a result, the impact estimate has been reduced from 354 piglets affected per dollar to 285 piglets affected per dollar. We have updated our review of Sinergia to reflect this change.
We would like to acknowledge that the difference in the cost-effectiveness of one of Sinergia’s programs (representing ~9% of expenditures) would not have had a determinative impact on our decision to recommend Sinergia. In fact, we believe we were conservative elsewhere in our estimates. For example, because Sinergia wasn’t able to provide estimates for the number of animals affected by three major commitments from food retailers and processors (Ceratti, Dia, and Habib’s in rows 7–9), we conservatively assumed that the number was zero. We remain proud to recommend Sinergia to donors and thank them for their excellent work.
Deletion of Column W
In their second post about Sinergia, Vetted Causes states, “Unfortunately, Sinergia/ACE deleted all of Column W right before Sinergia posted their response, and did not add any note stating that this column was deleted.”
During ACE’s evaluations of charities, we often redact information that charities provide us before publication on our website. In this particular instance, Sinergia had requested that we remove this column last year during their evaluation for confidentiality reasons related to their work, but we missed it in this particular case. This was not related to Vetted Causes’ review of Sinergia, but it was something that we noticed recently when double-checking our spreadsheet. We are not trying to hide any information; cell K10 states the change we made related to issue 2 above.
Improving Collaboration for Better Collective Results
We would like to take this opportunity to raise some questions about Vetted Causes’ approach to evaluations and the publication of their reviews, as well as provide our perspective on them.
Approach and intent: ACE aims to identify where additional donations are likely to reduce animal suffering as much as possible and drive counterfactual funding toward effective animal advocacy. Understanding Vetted Causes’ theory of change and mission would help clarify their aims and how to best collaborate moving forward. Are Vetted Causes focusing their evaluations solely on animal charities and/or ACE Recommended Charities? It’d be valuable to understand how they prioritize causes and organizations, and why.
Avoiding undue harm: Evaluation relies on nuances about program implementation (which, as noted by numerous commenters on this post, are notoriously challenging in animal advocacy, especially for those unfamiliar with the space). Furthermore, some types of work require confidentiality, so an evaluator would not get all the relevant context solely from looking at a charity’s website or external reviews. As such, we believe that it is important that Vetted Causes shares their reviews with charities before publishing them so that any factual errors can be corrected and knowledge gaps can be filled before making claims that can lead to reputational harm (e.g., Sinergia Makes False Claims About Helping Millions of Animals, Sinergia Continues to Make False Claims, etc.). This is not just ACE’s opinion—many others share our perspective. Furthermore, publicly responding point-by-point requires significantly more time than privately sharing context or clarifying misunderstandings directly. As such, our willingness to fully engage with future posts will depend on how Vetted Causes chooses to engage moving forward, to minimize further disruption to our work.
We believe that if Vetted Causes were to take a more collaborative approach, this would better serve our shared goal of reducing animal suffering as effectively as possible. We welcome continued discussion about how we can collectively improve our evaluation methods and maintain high standards of evidence while also acknowledging the complex realities of animal advocacy work.
Applications Open For Animal Charity Evaluators’ 2025 Charity Evaluations
Thanks for this interesting perspective on how to balance different values within the work of evaluations, Devin. Considering you drafted this in 2022, we do want to note that a lot has changed at ACE in the last three years, not least of which has been a shift to new leadership. Since early 2022, ACE has transitioned to a new Executive Director, Programs Director, Charity Evaluations Manager, Movement Grants Manager, Operations Director, and Communications Director.
That said, ACE continues to assess organizational health as part of our charity evaluations—we assess whether any aspects of an organization’s governance or work environment pose a risk to its effectiveness or stability, thereby reducing its potential to help animals. Furthermore, bad actors and toxic practices could negatively affect the reputation of the broader animal advocacy movement, which is highly relevant for a growing social movement, as well as advocates’ wellbeing and their willingness to remain in the movement. You can read more about our reasoning here and about our current evaluation criteria here.
Thanks for your thought-provoking piece. We are continually refining our evaluation methods so we will consider your points further about the kinds of instrumental information we might want to gather and how we could do so in a pragmatic way.
Thanks, Elisabeth
Hi Vasco,
We agree that the majority of our analysis should focus on the future work that would be enabled by ACE’s recommendation. However, forward-looking CEAs are inherently more subjective because they rely on projected metrics rather than actual past results. For this reason, we tend to create backward-looking CEAs and then assess whether there are any reasons to expect diminishing returns in the next two years (the duration of an ACE recommendation). When GWWC shared with us anonymized comments from the experts they consulted on this topic, the comments acknowledged these limitations of forward-looking CEAs. However, we also think there are cases where forward-looking CEAs can be helpful despite these limitations, for example when charities are planning new programs that are not currently funded.
We do not recommend charities if there is a large enough gap between their expected marginal cost-effectiveness and that of our other charities, and we do use the framing that you suggest when considering adding the next marginal charity. However, since we are unable to always fully quantify the impact on animals of charities’ work, this is partially based on qualitative arguments and judgments, so our decisions may not always appear consistent with the results of our CEAs.
In general, we quantify uncertainty within our CEA assessments and we also qualitatively assess the risk of each program. Additionally, we screen out applicants whose work is “too” uncertain based on their track record and whether or not the charities themselves are uncertain about where future funding would go. Our Movement Grants program does not have these bars and is willing to fund newer and more exploratory programs. However, we do agree that it’d be worthwhile to be clearer about how we weigh different types of risk in our decision-making, and we’ll consider adding this to our communications.
Thanks, Vince
Thanks for the questions!
As noted in GWWC’s report, our reasoning for recommending ÇHKD is that we think they’re very plausibly competitive with our other recommended charities, such as Sinergia. Sinergia’s CEA rested on more high uncertainty assumptions than ÇHKD’s did, and their CEA covered a smaller percentage of their work. We think it’s reasonable to support both a charity that we are more certain is highly cost-effective (such as ÇHKD) as well as one that we are more uncertain is extremely cost-effective (such as Sinergia). We also think ÇHKD may have more potential to have increased cost-effectiveness in the future, based on their recent shift to focus attention on winning commitments from larger retailers.
There are a few things we’d like to note when it comes to SWP and ALI:
They were evaluated in different years (SWP in 2023 and ALI in 2024) with different methodologies for assessing cost-effectiveness. In 2023, we assessed cost-effectiveness using weighted factor models that consider achievement quantity and quality, whereas in 2024 we switched to back-of-the-envelope calculations of impact per dollar. Because of this, there was no direct comparison between the shrimp stunning programs at SWP and ALI. However, the next time we evaluate SWP we expect to create an impact per dollar estimate, in which case the estimates you’ve created (including differentiating slaughter via ice slurry vs asphyxiation) will come in handy.
ALI’s shrimp work only accounts for ~38% of their overall expenditure, and we had strong reasons to recommend them for their other work (policy outreach, the Aquatic Animal Alliance, etc.).
While ACE values plurality, we don’t take a “best-in-class” approach and wouldn’t rule out recommending multiple charities doing similar work.
Thanks, Vince
Starting Now: Animal Charity Evaluators is hosting a live AMA
Hi Steven! That’s fantastic that you’re planning to donate to cost-effective animal charities. Thanks for thinking of ACE’s Recommended Charities and engaging with our work. When people ask us about the most impactful animal charities to donate to, we typically recommend donating to our Recommended Charity Fund. Our team of researchers decides how best to allocate this money among our Recommended Charities based on their current funding needs and the latest information on which activities this money would fund so that we can be confident that donations are being used as cost-effectively as possible. That said, here are some things we think are worth considering if you prefer to select charities yourself:
At ACE, we agree with the other commenters regarding the importance of wild animal welfare, given the sheer number of animals living in the wild, the likelihood that many of these animals suffer intensely, and the fact that the wild animal welfare field is currently so neglected. It’s great that you’re planning to look into this in more detail. If you haven’t seen them already, some helpful resources include this EA Forum post and a recent 80,000 Hours interview with Cameron Meyer Shorb (head of Wild Animal Initiative). ACE’s Why Wild Animals? page also cites some useful sources.
As for fishes, there is strong and growing evidence for their sentience and capacity for pain and pleasure. This briefing gives a helpful overview; you could also check out this EA Forum post or this recent interview with Doug Waley on the How I Learned to Love Shrimp podcast.
While there’s little current evidence for shrimp sentience, this is because there has been very little research in general rather than because people have found evidence against shrimp sentience. Shrimp Welfare Project has published an overview of the existing evidence, and this report by Rethink Priorities gives a great rundown of the potential welfare threats of shrimp farming. When you consider these threats, the huge numbers farmed, and the reasonable probability of shrimp sentience, it seems likely that current shrimp farming practices could be a major source of suffering.
Finally, while the kinds of work done by Faunalytics and New Roots Institute are less direct than some of our other charities, we think they’re likely to relieve a great deal of suffering by empowering other advocates and organizations to help many more animals than they otherwise would. Effective animal advocacy strategies to build a world where animals experience well-being has a relatively long time horizon. We expect that the work these organizations do speeds up the timeline significantly. The faster we can bring about a future where animals are protected, the more suffering is abated.
Hope that’s helpful and wherever you end up donating, thank you for thoughtfully trying to create a better world for animals!
- Max
Thank you, Vasco! Yes, that’s what I will be working toward next and assessing over time. I will be sure to share my findings :)
Holly
Reaching a New Audience: Insights From a Market Research Survey
Announcement: Animal Charity Evaluators is hosting an AMA on February 17 at 8-10am PT
Please find more information on animal suffering below:
Farmed animal suffering
Trillions of farmed animals (including fishes) experience cruel treatment and brutal deaths on factory farms. Farmed animal advocacy presents one of the most significant opportunities to reduce animal suffering on a large scale.
Chickens
Chickens are among the most abundant farmed animals, outranked only by farmed fishes and certain invertebrates, with an estimated 75 billion killed annually. When given enough space and an appropriate environment, chickens will forage for food, form nests and roosts, preen their feathers, and socialize with their flock. In contrast, chickens on factory farms live their entire lives in cramped, barren battery cages or crowded together in a shed with thousands of others, with little to no room to turn around or stretch their wings.
Fishes
As detailed in our Farmed Fish Welfare Report, existing research on fish welfare is limited, and the research that has been done tends to focus more on maximizing benefits for farms rather than fishes. While we believe that more research into promising ways to help farmed fish species is needed, we still believe that fish welfare should be prioritized within the broader effective animal advocacy movement. This is due to the current neglectedness of farmed fish animal welfare, the viability of potential strategies to improve farmed fish welfare, and the immense scale of fish farming—roughly 124 billion fishes are farmed in the world annually, with an additional 1.56 trillion caught (mid point) in the wild each year, not including bycatch and discards.
Shrimps
It is estimated that up to 518 billion (upper bound) farmed crustaceans are killed for food each year worldwide, yet less than 1% of animal welfare funding helps them. Unlike other invertebrates, it is more widely accepted that decapod crustaceans (e.g., shrimp, crabs, lobsters) and cephalopod mollusks (e.g., octopus, squid) are sentient creatures. In 2022, the sentience of both animal groups was officially recognized in U.K. law—an extremely promising development for organizations working to improve welfare conditions in aquatic animal farms. However, there is still significant room for more funding in this highly neglected area.
Pigs
Pigs are highly sentient animals capable of experiencing complex emotions, yet they are subjected to severe welfare issues, including confinement, tail mutilations, and the inability to exhibit natural behaviors such as playing and socializing. With nearly 1.5 billion pigs slaughtered for meat each year worldwide, addressing their welfare is both impactful and viable, given the availability of practical strategies to alleviate suffering and the current neglect of this issue.
Cows
Life for farmed cows is far from the idyllic, green pastures we see on milk cartons and meat packaging. Many of these curious animals spend their days in a crowded, confined space, with little chance to roam or enjoy a quiet rest under shady trees. Like humans, mother cows must give birth to produce milk. On farms, they are artificially inseminated every single year to become pregnant. In most cases, mother cows are separated from their calves after giving birth, denying them the chance to bond and nurture their young—a deeply distressing experience for both mothers and calves. Male calves are deemed “no use” by the dairy industry, so they are sold off for veal or beef or sometimes killed immediately after birth. It is estimated that around 308 million cows are killed for food each year worldwide.
Wild animal suffering
The number of wild animals far exceeds the combined number of humans, farmed animals, companion animals, and animals in laboratories. Unfortunately, many wild animals—possibly the vast majority—live very short lives and experience painful deaths.
Most organizations that focus on wild animals seem to be concerned with biodiversity conservation rather than the welfare of individual wild animals. We know of only a handful of charities that primarily focus on improving wild animal welfare. Their total global funding is estimated to be less than $5 million per year—a tiny amount relative to other cause areas; e.g., $290 million goes toward farmed animal welfare each year. We hope to see more charities working to improve wild animal welfare in the near future.
The lack of evidence for the effectiveness of strategies to improve wild animal welfare is partially due to the complexity of natural ecosystems and the difficulty of measuring the impacts of interventions in the wild. Due to this limitation, we believe that: (i) research is a promising way to help wild animals because it can inform decisions about which strategies to pursue. We acknowledge that improving wild animal welfare involves a high level of uncertainty, but its potential positive impact is also high.
Why We Should Fund Animal Charities
We would like to extend our gratitude to Giving What We Can (GWWC) for conducting the “Evaluating the Evaluators” exercise for a second consecutive year. We value the constructive dialogue with GWWC and their insights into our work. While we are disappointed that GWWC has decided not to defer to our charity recommendations this year, we are thrilled that they have recognized our Movement Grants program as an effective giving opportunity alongside the EA Animal Welfare Fund.
Movement Grants
After reflecting on GWWC’s 2023 evaluation of our Movement Grants (MG) program we made several adjustments, all of which are noted in GWWC’s 2024 report. We’re delighted to see that the refinements we made to our program this year have led to grantmaking decisions that meet GWWC’s bar for marginal cost-effectiveness and that they will recommend our MG program on their platform and allocate half of their Effective Animal Advocacy Fund to Movement Grants.
As noted by GWWC, ACE’s MG program is unique in its aims to fund underserved segments of the global animal advocacy movement and address two key limitations to effectiveness within the movement:
Limited evidence about which interventions are effective and in which contexts
Disproportionate attention devoted to some regions and animal groups
For impact-focused donors seeking opportunities to build an evidence-based and resilient global animal advocacy movement, our Movement Grants program is an effective giving opportunity which supports brilliant animal advocates all over the world.
Alongside their recommendation of our MG program, GWWC has outlined several areas for improvement that we are grateful for and will reflect on.
We agree with these suggestions by GWWC:
Improving the MG model to reflect our movement-building strategy—we have already started to revise our theory of change for the MG program to explicitly separate the pathways to impact for grants that are made primarily on the basis of movement-building and which are made on a more ‘direct impact’ basis. This will help us better account for movement building in our model and make future assessments about whether we want to maintain this approach to our grantmaking.
Revising our use of Impact Potential (IP) scores in the MG model—the IP scores played a minimal role in our final grant decisions this year. Before the GWWC evaluation, we had already decided to revise our use of these scores because they do not model scope in a sufficiently useful way and they run the risk of combining parameters in our model in such a way that clouds the decision-relevant information.
Better integrating scope comparisons—while our current MG model has scope baked into some of the factors (e.g. theory of change, long-term impact), we agree with GWWC that a more robust approach would be to make scope a factor that our grant reviewers score separately. We intend to make this adjustment in our next round of Movement Grants and we appreciate the specific logistical suggestions from GWWC on how we might do this.
Improving the documentation of our reasoning for making grant decisions—this is mainly related to our internal processes that don’t have any bearing on our grant decisions; however, we agree with GWWC that despite our diligent record keeping for how our thinking evolves through the grant review process, we need a better record that summarizes, in one place, the main rationale and cruxes for each grant decision. This is something we intend to implement in our next granting round.
We also want to note the following challenge:
GWWC recommends that we introduce a clearer framework for prioritizing between interventions. We agree with this recommendation—of the possible interventions available to help animals, some are already excluded from applying or rejected at an early stage from our grantmaking based on the scope of impact. However, while we intend to make further improvements in scope comparison between interventions, there remain challenges due to the many externalities that affect intervention effectiveness and our ability to estimate them. GWWC notes in the report that we appear resistant to doing this because it would be unhelpfully speculative. We want to clarify that we are willing to make speculations where we think they will be useful while highlighting the challenges. There is a difference between the more speculative forward-looking cost-effectiveness analyses (CEAs) we would be undertaking as a grantmaker compared with the CEAs the Charity Evaluations team undertakes on completed work. To overcome this, we may also consider comparing the known cost-effectiveness of the most similar organizations’ previous work or leveraging CEAs that have used the total available information on an intervention (e.g. this estimate). However, we remain cautious about spending time and resources on trying to find comparable cost estimates between interventions when doing so might not sufficiently increase the overall marginal cost-effectiveness of our grant decisions. We also want to note that this is a challenge for any animal advocacy funder, not just ACE. This is an area where we expect to continue to try different approaches and improve year-on-year, balancing available information and our team’s capacity.
We are grateful for the rare opportunity to reflect deeply on our work and to learn from GWWC’s perspectives so that we can award grants that are the most impactful for animals. We are especially thankful too for the larger GWWC and EA community that is willing to support highly promising projects to help some of the most neglected individuals who suffer greatly.
Charity Evaluations
On the other hand, we are disappointed that GWWC does not find our Charity Evaluations program justifiably competitive with MG (and the EA Animal Welfare Fund) and believe that donors might miss out on some of the most impactful donation opportunities because of GWWC’s decision. We will elaborate on the relationship between our Charity Evaluations and Movement Grants below, but first address some points specific to Charity Evaluations.
We agree with some of GWWC’s conclusions and suggestions for improvement, which appear in their 2024 report. We think that focusing on these will improve the quality of our recommendations moving forward:
Devote more resources to quantitatively modeling cost-effectiveness, including better defining upper and lower bounds and exploring new ways to assess speculative programs (such as using a mix of qualitative and quantitative arguments).
Explicitly assess charities’ strategic prioritization, such as their own focus on cost-effectiveness. While we develop a sense of this internally as we evaluate charities, we think it could be helpful to take it into account more systematically.
Explore other methods for more comprehensively capturing the magnitude of impact and limiting factors. This might mean adapting the theory of change analysis to include systematically assessing the weakest-seeming and most fundamental assumptions.
Reconsider the minimum and maximum sizes of grants we’re willing to make through the Recommended Charity Fund (RCF), for example by potentially reconsidering the safeguards in the disbursement model that lead to more consistent grant amounts over time.
However, there are also areas where we disagree with GWWC’s conclusions. While we acknowledge these parts of our methods have room for improvement, we think the changes that they suggest may not make a meaningful difference to the quality of our recommendations:
Include all decision-relevant factors in the published charity reviews. GWWC has highlighted their concerns based on our public reviews, such as recommendation decisions that seemed to insufficiently take into account scope (e.g., comparing ÇHKD and Sinergia’s cost-effectiveness results, arguments for longer-term paths to impact, etc.). While we agree that it’d be ideal to more clearly publicly demonstrate and formalize cross-charity comparisons and the other thinking that leads to our decisions, we also want to note that we already consider much of what GWWC suggests in our internal decision-making. We’ve made significant adjustments to our charity reviews in the past year and will continue to refine them to ensure they are transparent and informative while remaining accessible and encouraging effective giving. We think it’s possible that the Charity Evaluations program may have been disadvantaged in this evaluation by our in-depth charity reviews since GWWC’s comments seem to be informed more by what we published rather than the quality of our internal decisions.
Update the decision-making process so that it directly compares all recommended charities on marginal cost-effectiveness. Our basis for deciding whether to add a Recommended Charity is whether we think it would lead to more animals being helped on the margin (compared to having a smaller number of Recommended Charities), which is conceptually different from ranking charities. Given the types of uncertainty currently faced by the animal advocacy movement when it comes to calculating cost-effectiveness, we decide whether a charity should be recommended based on a range of decision criteria rather than scoring and ranking charities based on our sense of their relative marginal cost-effectiveness. In the future, if we had sufficiently robust evidence to form reliable cost-effectiveness estimates, including evidence or good proxies for speculative work with complex long-term theories of change, it’s possible we would move more toward the kind of ranking approach that GWWC suggests. Additionally, we consider relative cost-effectiveness during each Recommended Charity Fund distribution, where we adjust the size of each grant depending on the most up-to-date plans that charities share with us.
Increase emphasis on the marginal dollar. We think our current methods adequately track where influenced funding would go. With the exception of granting restricted funding, we already implement GWWC’s suggestions. For example, we consider how charities’ programs are likely to change with gaining or losing an ACE recommendation and we also take into account charities’ strategic prioritization (although we don’t assess it systematically, as noted above, and some of this information is kept confidential so it doesn’t appear in our charity reviews). Additionally, this consideration only comes into play relatively infrequently; in most cases, ACE-influenced funding does not go toward novel programs but toward expansions of current programs, investments in the charity as a whole, or funding gaps.
Give more consideration to cost-effectiveness analyses (CEAs) in decisions not to recommend charities. Similar to GWWC’s evaluations of evaluators, ACE looks for evidence to justifiably conclude that a charity should be recommended, and we do not recommend charities in the absence of that evidence. When we say that a CEA did not play a meaningful role in our decision not to recommend a charity, we mean that it was insufficient to justify a recommendation. From that perspective, CEAs play just as much of a role to recommend as to not recommend a particular charity. While there are improvements we can make to our CEAs so that they are more indicative of cost-effectiveness (or lack thereof), we don’t see this as an indicator of a lack of scope sensitivity.
GWWC also suggests some broader strategic shifts in our programs that we plan to consider as a part of upcoming strategic planning. While these changes would help align our Charity Evaluations program more closely with GWWC’s criteria, we’re currently unsure if they would do the most good for the animal advocacy movement and for animals. These include:
Making restricted recommendations to specific programs, granting restricted funding, and removing the constraint that the fund grants to all current Recommended Charities each round. We need to carefully consider any trade-offs between the benefits of this idea and downstream considerations such as the needs of our donor base and losing the benefits of allowing Recommended Charities the flexibility to pursue what they need the most to maximize their impact. Also, restricting funding likely introduces some degree of funging, reducing any cost-effectiveness gains.
Allowing Recommended Charities to apply for Movement Grants. Theoretically, this would mean Recommended Charities could apply for targeted Movement Grants funding for their marginal programs (which we don’t currently permit). However, this might risk negatively impacting the movement-building goals of the Movement Grants program.
Changing the evaluation cycle from two years to one year. In theory, this would allow for a more precise assessment of charities’ upcoming programs, which is why ACE used a one-year cycle until 2021. However, we changed to a two-year cycle largely based on feedback from the charities we evaluated that annual evaluations were overly demanding/time-consuming. We’ll carefully consider the balance between timely evaluations, the burden on charities that we evaluate, and our team capacity (for comparison, ACE has an evaluations team of 5.5 FTE compared to GiveWell’s ~39 researchers).
Moving forward, we will continue evolving the Charity Evaluations program to find the organizations that can do the most good with additional donations and we thank GWWC for critically engaging with our work. We also appreciate that they acknowledge the difficulties of our work and the inherent differences between evaluators and funders in the animal advocacy space. However, given those difficulties, we’re currently not sure whether ACE nor any other evaluator that recommends whole charities in this space would be seen by GWWC as competitive with charitable funds that give restricted grants. Because of this, we’re not sure whether we think that GWWC’s current approach leads to the best outcomes for animals.
How ACE views Movement Grants vs. Charity Recommendations
We want to acknowledge that the language GWWC uses implies that they see our Movement Grants (MGs) and Charity Evaluations programs as “competitive” with each other. This is not a view we share—we see them as complementary.
Although there’s some overlap between charities that are a fit for each program, they serve different purposes:
The Recommended Charity Fund (RCF)
provides high-impact evidence-based donation opportunities, and
ensures critical organizations can do their underfunded and neglected work.
The Movement Grants program
funds highly promising projects,
is more likely to fund hits-based opportunities, and
focuses on movement-building.
The programs are complementary and supplement each other:
MGs support charities that may go on to become Recommended Charities (RCs)—this is evidenced by the fact that the majority of our current RCs are former MG recipients.
Some charities are ineligible for our Charity Evaluations because RCs need to be able to absorb more funding and have more of a track record, and ACE needs to have confidence in them over a longer time horizon (the entire two-year recommendation cycle). This means that the RCF misses some of the highest-impact funding opportunities because it doesn’t fund independent projects, brand-new charities, or only specific projects at a charity that might do a mixture of more and less cost-effective work.
We don’t give MGs to RCs for the duration of their recommendation, so the MG program also misses some of the highest-impact funding opportunities.
MGs can support former RCs if they re-enter an exploratory phase and need project- or program-specific funding to test out their ideas.
They also serve different donors:
In our view neither clearly has a higher expected value than the other. However, since the MG program funds projects with a smaller track record or projects that can have a longer-term impact in building the movement, which leads to more uncertainty and variance in outcomes, it is more suited to donors with a higher risk tolerance. On the other hand, the RCF includes more established, highly cost-effective organizations with a strong track record and a clear path to impact and appeals to more risk-averse donors.
Funds work well for individual donors but some users of ACE’s and GWWC’s services, such as certain Effective Giving Initiatives and other third parties, cannot collect donations for regranting funds. They must either set up their own fund composed of established, well-researched charities that can take in large amounts of unrestricted additional donations, and/or direct donations to specific charities.
The effective animal advocacy space is still deeply neglected and mostly funded by existing animal advocates. Therefore, we see ACE’s unique role as critical in securing counterfactually new donations to high-impact charities. We aim to engage audiences who are currently unfamiliar with both effective giving and EAA, and who might never change their individual behavior (for instance by changing their diet). To convert them to support farmed and wild animal welfare requires more accessible and simple solutions. Individual Recommended Charities are more intuitive and easy to understand and may be more appealing to this audience.
We are proud to support all of our current Recommended Charities and Movement Grantees, and would like to take this opportunity to celebrate the impactful work they do to help make the world a kinder place for animals.
- ACE Team
Hi Isaac! Now that we’ve announced our 2024 Recommended Charities, we’ve had more time to process your feedback. Thanks again for engaging with our work.
As mentioned before, we’ve substantively updated our evaluation methods this year. This was informed in part by detailed feedback we received as part of Giving What We Can’s 2023 ‘Evaluating the Evaluators’ project, some of which aligns with your feedback.
One of these changes is that we now seek to conduct more direct cost-effectiveness analyses, rather than the 1-7 scoring method that we used last year. This more direct approach is possible in part thanks to Ambitious Impact’s recent work to allow quantification of animal suffering averted per dollar. Of course, these kinds of calculations are still extremely challenging, limited, and subject to significant uncertainties; we describe our methods and their limitations on our website. For example, while cost-effectiveness = impact divided by cost, it can be difficult to measure impact meaningfully in a way that is also quantifiable, so we rely on other criteria to help us make our assessments.
Another major change was introducing a formal Theory of Change assessment to understand the reasoning, evidence base, and limitations around each charity’s main programs. In our 2023 Evaluations, we discussed these considerations in our Recommendations Decisions meetings but did not systematically incorporate them into our public reviews. Together, we think these changes allow for a more nuanced assessment of charities’ work and (we hope) more informative and accessible reviews.
Regarding the impact of our recommendations, this year, we conducted an assessment of ACE’s programs and our counterfactual influence on funding. As part of this work, we surveyed donors to our Recommended Charity Fund (RCF) and asked them where they’d donate if ACE didn’t exist. This indicated that over 60% of our RCF donors would donate less to animal charities if ACE were not to exist, of whom around 12% would not donate to animal charities at all. We aim to publish these influenced-giving reports on November 29th. We hope this reassures you that animals are not worse off because of ACE’s charity recommendations.
In terms of your specific feedback on last year’s methodology:
‘Charities can receive a worse Cost-Effectiveness Score by spending less money to achieve the exact same results’ / ‘Charities can receive a better Cost-Effectiveness Score by spending more money to achieve the exact same results’ / ‘Charities can rearrange their budget and achieve the exact same results (with the exact same total expenditures), but their Cost-Effectiveness Score can significantly change.’
Your findings here are correct. Because the weighted averages in this model depended on the percentage of expenditure for each factor, they sometimes produced unintended and unhelpful results. In part due to this, we interrogated the outputs of our models in our Recommendation Decisions meetings at the time and considered cost-effectiveness scores alongside other decision-relevant factors (such as their Impact Potential and Room For More Funding), rather than taking cost-effectiveness as the only relevant factor to consider when evaluating charities or prioritizing giving opportunities. This was informed in part by each charity’s uncertainty scores, which helped inform how much weight to assign to cost-effectiveness and other criteria in our final recommendations decisions. As you would expect given their work, Legal Impact for Chickens’ uncertainty scores were among the highest of our 2023 evaluated charities. Of all our 2023 Recommended Charities, our Recommendations Decisions discussions played the biggest role for Legal Impact for Chickens given that our models were not as well-suited to their work compared to those for other charities.
How we addressed this in our 2024 Evaluations: As noted above, we now do direct cost-effectiveness analysis rather than using a weighted factor model. We think the role of our Theory of Change-focused discussions in our 2023 Recommendation Decisions meetings should have been more systematic and more clearly communicated in our 2023 reviews, which is one reason why we introduced the new Theory of Change assessment this year.
‘Charities can have 1,000,000 times the impact at the exact same price, and their Normalized Achievement Scores and Cost-Effectiveness Score can remain the same.’
This isn’t the case, but we didn’t publish the full details about our method for assessing the impact of books, podcasts, and other interventions, so we see why this wasn’t clear. Essentially for each intervention in our Menu of Interventions we identified proxies for its likely impact. For books, we had intended to include sales/views as well as a rating of the overall audience response/reviews. In practice, this wasn’t possible for various reasons given the wide variation in types of publication (e.g., some publications had not been released yet, or had been provided directly to the audience with no feedback collected), so we had to factor in such considerations on a more case-by-case basis in our Recommendations Decisions discussions. Issues such as this highlighted to us the inherent limitations of seeking to distill a charity’s work in a weighted factor model, given e.g. the large variation in tactics used by animal advocacy charities and the challenges involved in obtaining the necessary data to score their achievements based on pre-set criteria.
We used additive rather than multiplicative scoring because our objective was to create weighted factor models that reflect the quality of achievements (rather than, e.g., estimating the number of animals helped by books being written). Since we’ve transitioned to directly estimating cost-effectiveness, we now use a straightforward multiplication for factors like “likelihood of implementation.”
How we addressed this in our 2024 Evaluations: Same as the point above: we have now updated to a more direct cost-effectiveness analysis rather than using this weighted factor model, and have also introduced a new Theory of Change assessment.
‘Charities can increase their Normalized Achievement Scores and Cost-Effectiveness Score by breaking down actions into smaller steps, even if the overall results remain unchanged.’
This actually isn’t the case (sorry if this wasn’t clear). Breaking down an achievement into smaller steps would drive up the ‘Achievement quantity’ score, but would be offset by lower ‘Achievement quality’ scores for each achievement. However, there was still a risk of this introducing inconsistency into the model, which is another reason why we updated our methods this year.
‘The most important factor in determining the Normalized Achievement Score of an intervention (Impact Potential Score) is decided before the intervention even begins. This makes the maximum Normalized Achievement Score for certain interventions relatively low, even if they have extremely high impact.’
We developed this model because past evaluations have shown that the intervention type drives much of the impact of a charity’s achievements. Starting with a baseline intervention score and adjusting it still allows for particularly strong implementations to at least partially make up for a lower intervention score. That said, we agree with you on this model’s shortcomings. As with the cost-effectiveness model, we interrogated the model’s outputs in our Recommendation Decisions meetings and had a mechanism to weight Impact Potential lower in our decision-making when we were less certain about its relevance.
How we addressed this in our 2024 Evaluations: We have updated this model and now only use it in a very limited way to supplement a qualitative assessment of charities’ work during the charity selection phase, rather than during the Evaluations themselves.
‘Legal Impact for Chickens did not achieve any favorable legal outcomes, yet ACE rated them a Recommended Charity.’
When ACE considers impact for animals, we consider all the ways that animal suffering might be reduced when interventions are implemented. While they did not secure a litigation win, Legal Impact for Chickens’ Costco lawsuit garnered significant media attention that put pressure on the companies being litigated. While their work is more ‘hits-based’ than some of our other Recommended Charities, we think the considerable impact of any future legal wins means high expected value for this work overall, especially now that funding from ACE, Open Philanthropy, and the EA Animal Welfare Fund has allowed them to hire more litigators. Check out Alene Anello’s recent EA Forum post for an update on Legal Impact for Chickens’ latest achievements.
Thanks again for your engagement with our evaluations. We hope you get in touch with us directly if you come across new evidence-based methods to meaningfully capture cost-effectiveness or to improve the evaluation of animal charities. We might also reach out to you via email in the coming weeks as we go through retrospectives and plan for next year’s evaluation. Because of the complexity of the animal welfare cause area, the many uncertainties and knowledge gaps in the field of charity evaluation, and the urgency and scope of suffering, we embrace productive collaboration.
Thank you.
- The ACE team
Starting Now: Animal Charity Evaluations is hosting a live AMA
Hi Pablo, important question! GFI decided to postpone re-evaluation to a future year to allow their teams more time to focus on new opportunities and challenges in the alternative proteins sector. Our researchers will be available to answer more questions about our 2024 charity recommendations in our AMA next week (Nov 19, 8-10am PT) on the FAST Forum. We hope to catch you there! Thanks, Holly
Hi Michael, I’m glad you’re happy to see the cost-effectiveness models! And thank you for letting us know; there’s been much more traffic than we’re used which has made our website slow. We’re actively trying to resolve this. Please hold on a bit while we get our site back up. Clearing your cache may also help :) Thanks for your patience everyone!
Announcing our 2024 Charity Recommendations
Thank you for spending time analyzing our methods. We appreciate those who are willing to engage with our work and help us improve the accuracy of our recommendations and reduce animal suffering as much as possible.
Based on previously received feedback and internal reflection, we have significantly updated our evaluation methods in the past year and will be publishing the details next Tuesday when we release our charity recommendations for 2024. From what we can tell from a quick skim, we think that our changes largely address Vetted Causes’ concerns here, as well as the detailed feedback we received last year from Giving What We Can (see also our response at the time) as part of their program that evaluates evaluators. Our cost-effectiveness analyses no longer use achievement or intervention scores, but rather directly calculate cost-effectiveness by dividing impact by cost, as you suggest. That being said, our work will never be perfect so we invite anyone reading this with the expertise to improve the rigor of our work to reach out, now or in the future.
Although your comments are related to methods that we no longer use, we’d like to spend more time understanding and engaging with them, learning from them, and potentially correcting any misconceptions. Unfortunately, we won’t have the opportunity to do so until after our charity recommendations are released next week. Additionally, it might be a comfort to know that for the past few months, Giving What We Can has been assessing ACE’s new evaluation methods along with a panel of other experts and that they intend to publish the results later this month.
Thank you.
- The ACE team
Note: this comment has been posted in response to both Vetted Causes’ first and second posts about Sinergia, as it addresses points raised in each.
ACE welcomes rigorous external evaluations of our work, as such feedback strengthens our ability to provide high-quality recommendations to donors and, ultimately, helps us maximize our impact for animals. To this end, we engage with external experts on a) our methodology for evaluations and grantmaking and b) on evaluations and grant reviews themselves. For instance, we interact with the EA and FAST forums and the Hive Slack channel, taking advantage of both solicited and unsolicited feedback. The strongest example of our commitment to engaging with feedback lies in GWWC’s evaluation of our programs.
The recent post by Vetted Causes is no exception. However, to balance the opportunity cost of engaging with this feedback with its potential to increase the quality of our work, we will first address the content of the concerns raised. We will then offer some suggestions for improvement of future reviews by Vetted Causes.
Regarding Vetted Causes’ specific claims about Sinergia, most of those claims have been addressed by Sinergia directly. We’d also like to note that while Vetted Causes refers to “ACE/Sinergia”, we’d prefer that we be treated as separate entities. Therefore, there are several notes we’d like to add that refer to ACE’s work specifically:
Issue 1
Alleged Commitment 1: Vetted Causes states, “in 2024, JBS stated that they still use ear notching due to ‘Difficulty in finding alternatives that ensure process traceability.’”
This report came out after ACE’s evaluation of Sinergia was completed, so it did not inform any part of our evaluation. Sinergia addressed the specifics of this claim in their response.
Issue 2
Alleged Commitment 2: Vetted Causes states, “Sinergia claims that in 2023, JBS published a commitment to not use gestation crates in all new projects, with a ‘Transition deadline’ of 2023.
As evidence for this claim, Sinergia provided a link to one of the JBS’s animal welfare pages. However, the gestation crate policy that the alleged commitment references was already listed on JBS’s website in 2020, and has been in effect since that point.”
In their second post about Sinergia, Vetted Causes also states, “Although Sinergia downplays this as a ‘minor mistake,’ it results in Sinergia receiving credit for helping millions of JBS’s pigs who were not impacted. This is not a ‘minor mistake.’ Further, Sinergia claims that this mistake has been corrected, but all that was fixed was changing the phrase ‘by 2023’ to ‘in 2023’ in Cell K10. The impact calculations were not fixed, and still incorrectly credit Sinergia for helping millions of JBS’s pigs who were not impacted.”
As Sinergia notes in their response, the number of sows reported for JBS includes those whose welfare may be impacted by the enforcement of the previous commitment.
In the process of double-checking our calculations with Sinergia, we noticed the same miscommunication that would have caused changes to our calculations. Although it would have been ideal to apply discounts at both the “number of animals affected” and the “SADs averted per dollar” levels, our SADs estimates were already conservative, and the magnitude of the change would not affect our decision to recommend Sinergia. Therefore, we are currently opting not to spend the time refreshing these calculations, as the changes could be quite involved.
Issue 3
Sinergia Claims That With $1 They can “Liberat[e]” 354 Piglets from “Brutal Confinement”
Vetted Causes states that, “ACE gave Sinergia credit for helping over 30 million female piglets through surgical castration commitments that Sinergia allegedly secured.”
Out of the two figures that ACE publishes for Sinergia’s pig welfare program (SADs averted per dollar and piglets affected per dollar), the former is correct because Ambitious Impact’s SADs calculations already take into consideration the “prevalence” of castration (which only occurs in male piglets). However, the latter indeed requires the same prevalence discount.
We have adjusted the estimates in the CEA spreadsheet for Sinergia’s pig welfare program. As a result, the impact estimate has been reduced from 354 piglets affected per dollar to 285 piglets affected per dollar. We have updated our review of Sinergia to reflect this change.
We would like to acknowledge that the difference in the cost-effectiveness of one of Sinergia’s programs (representing ~9% of expenditures) would not have had a determinative impact on our decision to recommend Sinergia. In fact, we believe we were conservative elsewhere in our estimates. For example, because Sinergia wasn’t able to provide estimates for the number of animals affected by three major commitments from food retailers and processors (Ceratti, Dia, and Habib’s in rows 7–9), we conservatively assumed that the number was zero. We remain proud to recommend Sinergia to donors and thank them for their excellent work.
Deletion of Column W
In their second post about Sinergia, Vetted Causes states, “Unfortunately, Sinergia/ACE deleted all of Column W right before Sinergia posted their response, and did not add any note stating that this column was deleted.”
During ACE’s evaluations of charities, we often redact information that charities provide us before publication on our website. In this particular instance, Sinergia had requested that we remove this column last year during their evaluation for confidentiality reasons related to their work, but we missed it in this particular case. This was not related to Vetted Causes’ review of Sinergia, but it was something that we noticed recently when double-checking our spreadsheet. We are not trying to hide any information; cell K10 states the change we made related to issue 2 above.
Improving Collaboration for Better Collective Results
We would like to take this opportunity to raise some questions about Vetted Causes’ approach to evaluations and the publication of their reviews, as well as provide our perspective on them.
Approach and intent: ACE aims to identify where additional donations are likely to reduce animal suffering as much as possible and drive counterfactual funding toward effective animal advocacy. Understanding Vetted Causes’ theory of change and mission would help clarify their aims and how to best collaborate moving forward. Are Vetted Causes focusing their evaluations solely on animal charities and/or ACE Recommended Charities? It’d be valuable to understand how they prioritize causes and organizations, and why.
Avoiding undue harm: Evaluation relies on nuances about program implementation (which, as noted by numerous commenters on this post, are notoriously challenging in animal advocacy, especially for those unfamiliar with the space). Furthermore, some types of work require confidentiality, so an evaluator would not get all the relevant context solely from looking at a charity’s website or external reviews. As such, we believe that it is important that Vetted Causes shares their reviews with charities before publishing them so that any factual errors can be corrected and knowledge gaps can be filled before making claims that can lead to reputational harm (e.g., Sinergia Makes False Claims About Helping Millions of Animals, Sinergia Continues to Make False Claims, etc.). This is not just ACE’s opinion—many others share our perspective. Furthermore, publicly responding point-by-point requires significantly more time than privately sharing context or clarifying misunderstandings directly. As such, our willingness to fully engage with future posts will depend on how Vetted Causes chooses to engage moving forward, to minimize further disruption to our work.
We believe that if Vetted Causes were to take a more collaborative approach, this would better serve our shared goal of reducing animal suffering as effectively as possible. We welcome continued discussion about how we can collectively improve our evaluation methods and maintain high standards of evidence while also acknowledging the complex realities of animal advocacy work.