VettedCauses

Karma: 74

VettedCauses 11 Nov 2024 14:13 UTC
−6 points
0 ∶ 0
in reply to: abrahamrowe’s comment on: A Critique of Animal Charity Evaluators (ACE)
You straw manned us, and now you claim that “This is starting to feel pretty bad faith”.
Here is the quote of what we said:
we have reviewed 5 of ACE’s “Top 11 Animal Charities to Donate to in 2024” and only one of them (Shrimp Welfare Project) appears to be an effective charity for helping animals.
Here is the quote of what you said we claimed:
5 / 11 (or more) ACE top charities are not effective
Notice that we said that only one of the 5 appears to be effective (meaning 4 did not appear to be effective), and you changed this claim to 5 are not effective.
Is the claim “4 did not appear to be effective” the same as “5 are not effective”?

VettedCauses 10 Nov 2024 20:59 UTC
−4 points
0 ∶ 2
in reply to: abrahamrowe’s comment on: A Critique of Animal Charity Evaluators (ACE)
Wait, those are related to each other though—if we haven’t seen the full impact of their previous actions, we haven’t yet seen their historical cost-effectiveness in full!
No, they are not. Historical cost-effectiveness refers to past actions and outcomes—what has already occurred.
All of LIC’s legal actions have already been either dismissed or rejected. What are you suggesting we need to wait for before we can analyze LIC’s historical cost-effectiveness in full?
You are conflating the issue of past cost-effectiveness with future potential.
Also, you cite these as reasons the project should be dismissed in your post—you have a section literally called “Legal Impact for Chickens Did Not Achieve Any Favorable Legal Outcomes, Yet ACE Rated Them a Top Charity” which reads to me that you believe that it is bad they were rated a Top Charity, and make these same arguments (and no others) in the section, suggesting that you think this evidence means they should be dismissed.
Did I claim that I don’t think LIC “should be dismissed”?

VettedCauses 10 Nov 2024 19:38 UTC
−2 points
0 ∶ 1
in reply to: abrahamrowe’s comment on: A Critique of Animal Charity Evaluators (ACE)
Responding because this is inaccurate)
I don’t know what you’re saying is inaccurate. My reply addressed every single word from the section you claimed I didn’t provide evidence for.
My claim in the comment above was that you haven’t provided any evidence that:
- 5 / 11 (or more) ACE top charities are not effective
We never made this claim.
- That animals are suffering as a result of ACE recommendations
I’ll ask again. Our review details how ACE is rewarding charities for inefficiency (and punishing them for efficiency), and how LIC was rewarded for their inefficiency with the designation “Top 11 Animal Charities to Donate to in 2024.” Our review also details how ACE’s recommendations direct the flow of millions of dollars. Are you asking for evidence that directing millions of dollars toward ineffective animal charities, rather than effective ones, leads to animal suffering?

VettedCauses 10 Nov 2024 17:42 UTC
5 points
0 ∶ 0
in reply to: abrahamrowe’s comment on: A Critique of Animal Charity Evaluators (ACE)
Appreciate you going through everything with me!
No problem. Thank you for your replies!

VettedCauses 10 Nov 2024 17:39 UTC
1 point
0 ∶ 0
in reply to: abrahamrowe’s comment on: A Critique of Animal Charity Evaluators (ACE)
I don’t find that evidence particularly compelling on its own, no. Lots of projects cost more than 1M or take more than a few years to have success. I don’t see why those things would be cause to dismiss a project out of hand.
The question I asked was: “You don’t find these facts particularly compelling evidence that LIC is not historically cost-effective?”
The question was not about whether these facts are compelling evidence that LIC won’t be successful in the future, or if the project should be dismissed.

VettedCauses 10 Nov 2024 17:34 UTC
−4 points
0 ∶ 0
in reply to: abrahamrowe’s comment on: A Critique of Animal Charity Evaluators (ACE)
Note: this reply addresses everything Abraham claims we did not provide evidence for.
“ACE’s poor evaluation process leads to ineffective charities receiving recommendations”
Our review covered how under ACE’s evaluation process:
1. Charities can receive a worse Cost-Effectiveness Score by spending less money to achieve the exact same results.
2. Charities can have 1,000,000 times the impact at the exact same price, and their Cost-Effectiveness Score can remain the same.
3. The most important factor in determining the impact of an intervention is decided before the intervention even begins.
This is clear evidence that ACE uses a poor evaluation process. Is the fact that ACE’s evaluation process rewards inefficiency, and punishes efficiency, “no evidence” for ACE recommending ineffective charities?
If you’d like me to get even more specific, let’s look at Problem 1 of our review:
We go on to detail how if LIC had spent less than $2,000 on the lawsuit (saving over $200,000) and achieved the exact same outcome, ACE would have assigned LIC a Cost-Effectiveness Score of 1.8. The lowest Cost-Effectiveness Score ACE assigned to any charity in 2023 was 3.3. This means if LIC had spent less than $2,000 on the lawsuit, LIC’s Cost-Effectiveness Score would have been significantly worse than any charity ACE evaluated in 2023.
Instead, LIC spent over $200,000 on the lawsuit, and LIC rewarded them for this inefficiency by giving them a Cost-Effectiveness Score of 3.7, and deeming LIC a top 11 animal charity.
As we noted in our review, these Cost-Effectiveness Scores are defined by ACE as “how cost effective we think the charity has been”. LIC achieved no favorable legal outcomes despite receiving over a million dollars in funding. As we also noted in our review, every lawsuit LIC filed was dismissed for failing to state a valid legal claim.
If I provided evidence that a Law Firm Rating Organization rewards law firms for losing lawsuits and wasting money, and punishes law firms for winning lawsuits and saving money, would this be no evidence that the Law Firm Rating Organization is recommending ineffective law firms?
ACE’s poor evaluation process leads to ineffective charities receiving recommendations, and many animals are suffering as a result.
Our review details how ACE’s recommendations direct the flow of millions of dollars. Are you asking for evidence that directing millions of dollars toward ineffective animal charities, rather than effective ones, leads to animal suffering?
we have reviewed 5 of ACE’s “Top 11 Animal Charities to Donate to in 2024” and only one of them (Shrimp Welfare Project) appears to be an effective charity for helping animals.
Imagine a film critic watches 5 of the 11 films that received a ‘Best Films’ award and writes, “Of the five films I’ve seen, only one appears to deserve the award. I plan to release my reviews of the films shortly.” Does this statement by the film critic require evidence?

VettedCauses 10 Nov 2024 15:41 UTC
1 point
0 ∶ 0
in reply to: abrahamrowe’s comment on: A Critique of Animal Charity Evaluators (ACE)
But I don’t find this level of evidence particularly compelling on its own.
You don’t find these facts particularly compelling evidence that LIC is not historically cost-effective?
1. LIC’s most cost-effective intervention was one in which they spent over $200,000, and the lawsuit was dismissed for failing to state a valid legal claim.
2. LIC received over a million dollars in funding prior to being reviewed
3. LIC existed for multiple years prior to being reviewed
4. LIC failed to secure any favorable legal outcomes, or file any lawsuit that stated a valid legal claim?
What would be compelling evidence for LIC not being historically cost-effective?
I think I feel confused about the example you’re giving because it isn’t about hypothetical cost-effectiveness, it’s about historic cost-effectiveness, where what matters are the counterfactuals.
ACE does 2 separate analyses for past cost-effectiveness, and room for future funding. For example, those two sections in ACE’s review of LIC are:
- Cost Effectiveness: How much has Legal Impact for Chickens achieved through their programs?
- Room For More Funding: How much additional money can Legal Impact for Chickens effectively use in the next two years?
Our review focuses on ACE’s Cost-Effectiveness analysis, not on their Room For More Funding analysis. In the future, we may evaluate ACE’s Room For More Funding Analysis, but that is not what our review focused on.
However, I would like to pose a question to you: Given the ACE often gives charities a worse historic cost-effectiveness rating for spending less money to achieve the exact same outcomes (see Problem 1), how confident do you feel in ACE’s ability to analyze future cost-effectiveness (which is inherently more difficult to analyze)?

VettedCauses 10 Nov 2024 15:21 UTC
1 point
0 ∶ 0
in reply to: abrahamrowe’s comment on: A Critique of Animal Charity Evaluators (ACE)
Thank you for your response!
From this post, it seems like you’re trying to calculate historic cost-effectiveness and rate charities exclusively on that (since you haven’t published an evaluation of an animal charity yet I could be wrong here though)
This is not what we are trying to do. We simply critiqued the way that ACE calculated historic cost-effectiveness, and how ACE gave Legal Impact for Chickens a relatively high historic cost-effectiveness rating despite have no historic success.
My understanding of what ACE is trying to do with its evaluations as a whole is identify where marginal dollars might be most useful for animal advocacy, and move money from less effective opportunities to those.
ACE does 2 separate analyses for past cost-effectiveness, and room for future funding. For example, those two sections in ACE’s review of LIC are:
- Cost Effectiveness: How much has Legal Impact for Chickens achieved through their programs?
- Room For More Funding: How much additional money can Legal Impact for Chickens effectively use in the next two years?
Our review focuses on ACE’s Cost-Effectiveness analysis, not on their Room For More Funding analysis. In the future, we may evaluate ACE’s Room For More Funding Analysis, but that is not what our review focused on. We wanted to keep our review short enough that people could read it without a huge time investment, so we could not include an assessment of every single part of ACE’s evaluation process in our review.
It is also less reasonable to hold ACE accountable for their Room For More Funding analysis, since this is inherently more subjective and difficult to do. It is far easier for ACE (or any charity evaluator) to analyze historic cost-effectiveness than to analyze future cost-effectiveness. However, I would like to pose a question to you: Given the ACE often gives charities a worse historic cost-effectiveness rating for spending less money to achieve the exact same outcomes (see Problem 1), how confident do you feel in ACE’s ability to analyze future cost-effectiveness?
My understanding is ACE has tried to do something that’s just cost-effectiveness analysis in the past (they used to give probability distributions for how many animals were helped, for example).
ACE responded to this thread acknowledging that the problems listed in our review needed to be addressed, and that they changed their methodology (to a cost-effectiveness calculation of simply impact divided by cost) to do so:

VettedCauses 10 Nov 2024 5:09 UTC
3 points
0 ∶ 0
in reply to: abrahamrowe’s comment on: A Critique of Animal Charity Evaluators (ACE)
I also felt like this felt pretty politically motivated. Not sure if that is your intention, but paragraphs like this:
ACE’s recommendations determine which animal charities receive millions of dollars in donations.^[1] Thus far, we have reviewed 5 of ACE’s “Top 11 Animal Charities to Donate to in 2024” and only one of them (Shrimp Welfare Project) appears to be an effective charity for helping animals. ACE’s poor evaluation process leads to ineffective charities receiving recommendations, and many animals are suffering as a result.
Without any evidence feels pretty intense. ACE is kind of low hanging fruit to pick on in the EA space, so this read to me like more of that, without necessarily the evidence base to back it. Reading your report, I felt kind of like “oh, there are interesting assumptions here, would be interested to learn more”, and not “ACE is doing an extremely bad job.”
What claims did we make that we did not provide evidence for?

VettedCauses 9 Nov 2024 21:31 UTC
2 points
0 ∶ 0
in reply to: abrahamrowe’s comment on: A Critique of Animal Charity Evaluators (ACE)
both you and ACE seem to have different goals (calculating historic cost-effectiveness vs marginal impact of future dollars)
ACE states (under Criterion 2) that a charity’s Cost-Effectiveness Score “indicates, on a 1-7 scale, how cost effective we think the charity has been [...] with higher scores indicating higher cost effectiveness.”
Would you mind clarifying what you believe ACE’s goal is, and what you believe my goal is?
The analysis in my review is entirely about calculating historic cost-effectiveness. ACE’s Cost-Effectiveness Scores are also entirely about calculating historic cost-effectiveness.

VettedCauses 9 Nov 2024 21:14 UTC
5 points
0 ∶ 0
in reply to: tobycrisford 🔸’s comment on: A Critique of Animal Charity Evaluators (ACE)
Hi Toby,
Thank you for your reply!
Of course, the other thing that could be going on here, is that average cost-effectiveness is not the same as cost-effectiveness on the margin, which is presumably what ACE should care about.
I’m not certain if by cost-effectiveness on the margin, you meant cost-effectiveness in the future if additional funding is obtained. If that’s the case, the following information could be helpful.
ACE does 2 separate analyses for past cost-effectiveness, and room for future funding. For example, those two sections in ACE’s review of LIC are:
- Cost Effectiveness: How much has Legal Impact for Chickens achieved through their programs?
- Room For More Funding: How much additional money can Legal Impact for Chickens effectively use in the next two years?
Our review focuses on ACE’s Cost-Effectiveness analysis. Additionally, ACE states (under Criterion 2) that a charity’s Cost-Effectiveness Score “indicates, on a 1-7 scale, how cost effective we think the charity has been [...] with higher scores indicating higher cost effectiveness.”

VettedCauses 9 Nov 2024 20:50 UTC
1 point
0 ∶ 0
in reply to: MathiasKB🔸’s comment on: A Critique of Animal Charity Evaluators (ACE)
Hi Mathias,
Thank you for your comment!
I can spend a weekend day doing this and writing up my own thoughts in a new post if there is interest
We would definitely be interested in hearing your thoughts. We’ve set post notifications on for your profile, and look forward to seeing your post!

VettedCauses 9 Nov 2024 5:07 UTC
2 points
1 ∶ 0
in reply to: Ben_West🔸’s comment on: A Critique of Animal Charity Evaluators (ACE)
Hi Ben,
Thank you for your response!
I will address your points, but first I would like to clarify what we believe the crux of the problem is with LIC being deemed a top 11 animal charity by ACE.
In Problem 1 of our review, we state the following:
We go on how to detail how if LIC had spent less than $2,000 on the lawsuit (saving over $200,000) and achieved the exact same outcome, ACE would have assigned LIC a Cost-Effectiveness Score of 1.8. The lowest Cost-Effectiveness Score ACE assigned to any charity in 2023 was 3.3. This means if LIC had spent less than $2,000 on the lawsuit, LIC’s Cost-Effectiveness Score would have been significantly worse than any charity ACE evaluated in 2023.
Instead, LIC spent over $200,000 on the lawsuit, and LIC rewarded them for this inefficiency by giving them a Cost-Effectiveness Score of 3.7, and deeming LIC a top 11 animal charity.
This is the crux of the problem, and it is really an issue with ACE deeming LIC a top 11 animal charity, not with LIC itself. ACE elected to give LIC this distinction, and LIC merely accepted it.
I would also like to note encouraging or valuing lawsuits that fail to state valid legal claims (but burden defendants/garner publicity) risks causing the legal system to take animal rights/welfare cases less seriously. If courts observe a pattern of weak or legally insufficient cases being filed for publicity/to burden the defendant, they will become skeptical of all animal rights/welfare lawsuits—even those with strong legal merit. Prior to being deemed a top 11 animal charity by ACE, every single lawsuit filed by LIC failed to state a valid legal claim.
I note that you excluded the some key things when quoting ACE: LIC’s first lawsuit, a shareholder derivative case against Costco’s executives for chicken neglect, was featured on TikTok and in multiple media outlets, including CNN Business, Fox Business, The Washington Post, and Meatingplace… We thought the achievement has strong potential for indirect impact, and it received a high amount of media attention. - ACE
ACE’s review of LIC contains a section titled “Our Assessment of Legal Impact for Chickens’ Cost Effectiveness”, and the quote you have provided is not part of this section. Our entire review of ACE is about ACE incorrectly calculating cost-effectiveness; consequently, this is the section we decided to focus on. ACE’s review of LIC is over 5,000 words, and we cannot include every quote from ACE’s review of LIC.
Additionally, the quote you’ve provided gives no metrics to gauge how much media attention was received. If media attention is a strong justification for stating a $200,000 lawsuit that failed to state a valid legal claim is “particularly cost-effective” (as ACE put it), ACE should provide metrics regarding how much media attention was received. Ironically, the Facebook post you mentioned appears to have more metrics than ACE’s review of LIC regarding amount of media attention caused by the Costco lawsuit, since the Facebook post lists the numbers of likes and comments the post received.
The Facebook fan page still as of this writing has a post about the lawsuit pinned to the top because apparently the owner decided to boycott after learning about the cruelty.
The Facebook post you referred to received 56 likes and 83 comments. To my understanding, the post is also not pinned to the top, it is simply the last post the Facebook page has made (it appears that the page has not posted in over 2 years). I do not think this is very strong evidence that LIC’s $200,000 lawsuit that was dismissed for failing to state a valid legal claim was “particularly cost-effective” (as ACE put it).
It sounds like the Costco board also had to take official action
Correct, the Costco board took official action by rejecting LIC’s demands.
Is it worth $200k to get a bunch of bad publicity for Costco [...]?
Could you please define what “a bunch of bad publicity for Costco” means? And could you provide evidence that this level of publicity was caused by LIC’s lawsuit?
Is it worth $200k to [...] force the board to form a committee and hire an investigator, etc.?
Costco’s board formed a committee to review and investigate LIC’s demands. The committee then recommended that the board reject the demand, which they did. This does not appear to be a very good outcome.
Is it worth $200k to get a bunch of bad publicity for Costco, force the board to form a committee and hire an investigator, etc.?
I don’t know, I’m pretty willing to believe that the answer is “no”, but it doesn’t seem obvious to me. I could pretty easily believe that the CEO of the next company they sue would to change their policies instead of having to deal with the embarrassment of asking the board to form a committee to investigate.
It is ACE’s job to write charity reviews that provide the empirics necessary to answer questions like the one you’ve asked. From your own statement, it seems like ACE has failed to do this. ACE did not provide metrics on how much media attention the Costco lawsuit caused, and did not provide any insight into how much of a burden it was to form a committee to review and investigate LIC’s demands (I don’t recall ACE’s review even mentioning this).

VettedCauses 9 Nov 2024 3:55 UTC
15 points
0 ∶ 0
in reply to: Animal Charity Evaluators’s comment on: A Critique of Animal Charity Evaluators (ACE)
Hi,
Thank you for your response!
we have significantly updated our evaluation methods in the past year and will be publishing the details next Tuesday when we release our charity recommendations for 2024. From what we can tell from a quick skim, we think that our changes largely address Vetted Causes’ concerns here, as well as the detailed feedback we received last year from Giving What We Can (see also our response at the time) as part of their program that evaluates evaluators. Our cost-effectiveness analyses no longer use achievement or intervention scores, but rather directly calculate cost-effectiveness by dividing impact by cost, as you suggest.
We are glad to hear that ACE has changed their evaluation methods, and we hope that the changes effectively address the concerns listed in our review.
We look forward to seeing ACE’s new charity recommendations when they are released next week.

VettedCauses 8 Nov 2024 22:30 UTC
4 points
0 ∶ 0
in reply to: MichaelStJules’s comment on: A Critique of Animal Charity Evaluators (ACE)
Hi Michael,
ACE re-evaluates their Recommended Charities every two years. In our review of ACE, all charities mentioned were evaluated in 2023 (the most recent published review cycle). Therefore, every charity mentioned in our review will still be recommended in ACE’s upcoming list of Recommended Charities.
When the new reviews come out, we will be sure to read them though!

VettedCauses 8 Nov 2024 20:58 UTC
29 points
1 ∶ 0
in reply to: Sarah Cheng’s comment on: A Critique of Animal Charity Evaluators (ACE)
Thank you for your feedback. We will view the tips, and keep them in mind during our future reviews!
Edit: I have also changed the title of the post. For transparency, the original title was: Animal Charity Evaluators (ACE) is Extremely Bad at Evaluating Charities.

VettedCauses 8 Nov 2024 20:32 UTC
3 points
0 ∶ 0
in reply to: abrahamrowe’s comment on: A Critique of Animal Charity Evaluators (ACE)
Yep, this seems right, but it’s also the case that if they did something else with that funding, the effectiveness of that action would be rated much more highly, which also seems correct.
Our hypothetical in Problem 1 of the review is about two scenarios:
1. LIC spending $204,428 on the Costco lawsuit and achieving outcome X. (this is what actually happened)
2. LIC spending $1,566 on the Costco lawsuit and achievement outcome X. (this is what happened in the hypothetical)
Note that both scenarios achieve the exact same result, which is outcome X. ACE would rate LIC less cost-effective for spending $1,566 (saving $202,862) to achieve outcome X.
The hypothetical is not about something else they could do with the funding. The hypothetical is about assessing what happens to a charity’s Cost-Effectiveness Score if they save money to achieve the same outcome.
it’s also the case that if they did something else with that funding, the effectiveness of that action would be rated much more highly, which also seems correct.
How do you know that if they did something else with that funding the effectiveness of that action would be rated much more highly? According to ACE, the Costco lawsuit was a particularly cost effective intervention in spite of the fact that the lawsuit was dismissed for failing to state a valid legal claim:
- “We think that out of all of Legal Impact for Chickens’ achievements, the Costco shareholder derivative case is particularly cost effective because it scored high on achievement quality.”
I’m also not sure how it would even be possible to evaluate a hypothetical in which LIC does something else with the funding. Could you explain how this hypothetical would work, and how it would be evaluated? All of the examples in our review are 100% objective and based on ACE’s own methodology. There is no subjectivity in our examples, and this was done intentionally.
You are correct about a mathematical fact, but both you and ACE seem to have different goals (calculating historic cost-effectiveness vs marginal impact of future dollars)
I’m not sure how ACE’s goals could align with the principles of effective altruism if they intentionally created a methodology that contains the problem described above.
Again, I don’t really see good evidence for this—what is the typical track record for legal campaigns? How much do they cost? How long do they take to work?
Imagine there is a law firm that has received over a million of dollars in funding, existed for multiple years, and failed to secure any favorable legal outcomes. Also imagine that their most cost-effective lawsuit was one in which they spent over $200,000, and the lawsuit was dismissed for failing to state a valid legal claim.
If this law firm were rated one of the top 100 law firms in the world, what would you think of the organization that assigned this rating? Would you say there is not good evidence for this being an incorrect rating?
ACE rated LIC as one of the Top 11 Animal Charities to Donate to in 2024. Prior to being reviewed, LIC received over a million dollars in funding, existed for multiple years, and failed to secure any favorable legal outcomes. According to ACE, LIC’s most cost-effective intervention was one in which they spent over $200,000, and the lawsuit was dismissed for failing to state a valid legal claim.
Are these factors poor evidence for LIC not being one of the 11 best animal charities to donate to?

VettedCauses 8 Nov 2024 19:48 UTC
1 point
0 ∶ 0
in reply to: David T’s comment on: A Critique of Animal Charity Evaluators (ACE)
Hi David,
Thank you for your reply!
I think it is possible for an charity focusing on taking legal action to be impactful without [consistent] legal success, which the review doesn’t really acknowledge. A large part of the theory of change around suing corporate bad behaviour is the idea that it will deter bad behaviour in future, by making standards compliance more cost effective than defending lawsuits
I definitely agree that this is possible! However, as you said
it may be very difficult to have a deterrent effect if cases are typically dismissed.
ACE evaluated 3 “legal actions” in their review of LIC:
- 2 of the legal actions were dismissed under Rule 12(b)(6) for failing to state a valid legal claim. 12(b)(6) dismissals occur very early on in the legal process, making any legal expenses incurred by the Defendants relatively low. Additionally, encouraging or valuing lawsuits that fail to state valid legal claims but cost the defendant money risks causing the legal system to take animal rights/welfare cases less seriously. If courts observe a pattern of weak or legally insufficient cases being filed to burden defendants, they will become skeptical of all animal rights/welfare lawsuits—even those with strong legal merit.
- The 3rd legal action ACE evaluated was not actually a legal action, but rather a public comment submission (ACE still classified it as a legal action). The public comment was rejected, and it is difficult to see how this would have a positive impact.
I don’t think this necessarily reflects badly on the organization, which is a young charity focused on a legal process which inevitably takes time.
ACE endorses LIC as a top charity. Currently, I don’t think this endorsement is justified given LIC’s track record, and I don’t think ACE provided a very strong justification for it. Here is a quote from ACE’s review of LIC:
- “We think that out of all of Legal Impact for Chickens’ achievements, the Costco shareholder derivative case is particularly cost effective because it scored high on achievement quality.”
The Costco shareholder derivative case cost LIC over $200,000 and was dismissed for failing to state a valid legal claim. It is difficult to understand why ACE thinks this is a particularly cost effective achievement.
Some of the questions raised about ACE’s weightings are quite independent from the example given.
Could you elaborate on what you mean by this?
I think this would have been an interesting contribution to the Animal Welfare debate week.
I wasn’t aware of that week. Maybe we’ll be able to prepare something for it next year!
Thank you for your feedback!

VettedCauses 8 Nov 2024 19:02 UTC
1 point
2 ∶ 3
in reply to: Joey 🔸’s comment on: A Critique of Animal Charity Evaluators (ACE)
Hi Joey,
Thank you for taking the time to read our review!
(a) ACE uses a methodology that yields quite different results than a raw cost-effectiveness analysis; (b) this methodology seems to have major flaws, as it can lead to clearly incoherent conclusions and recommendations easily; and (c) thus, it is better to use a more straightforward, direct CEA.
I agree with points A and B, but I am much less convinced about point C.
I would like to point to Problem 1 and Problem 4 from the review:
1. Charities can receive a worse Cost-Effectiveness Score by spending less money to achieve the exact same results.
2. Charities can have 1,000,000 times the impact at the exact same price, and their Cost-Effectiveness Score can remain the same.
Effective giving is all about achieving the greatest impact at the lowest cost. ACE’s methodology is not properly accounting for impact, or for cost.
Using the equation impact / cost at least results in impact being in the numerator, and cost being in the denominator. To me, this alone makes a straightforward, direct CEA a better methodology than the one used by ACE.
To me, this feels a bit like an isolated demand for methodological rigor. Every methodology has flaws, and it’s easy to find situations that lead to clearly incoherent conclusions.
I absolutely agree that every methodology has flaws, and we did not mean to imply otherwise. However, the incoherent conclusions described in our review of ACE’s methodology are not one off instances. hey are pervasive problems that impact all of ACE’s reviews.
Thank you for your feedback!

VettedCauses 8 Nov 2024 18:41 UTC
1 point
3 ∶ 0
in reply to: abrahamrowe’s comment on: A Critique of Animal Charity Evaluators (ACE)
Hi Abraham,
I didn’t look at this for more than a couple minutes
Thank you for reading some of the article. I hope that you find some time to read the rest.
Did you ask ACE to review this before publishing?
No, I did not ask ACE. I hope that this article inspires a public discussion.
I noticed that in your first example, you assume in your example (I believe), that if LIC didn’t spend 200k on the lawsuit against Costco, they wouldn’t spend it on anything else. It’s unclear to me that this is the counterfactual, or how ACE is conceptualizing those funds.
What do you mean by conceptualizing funds? In this hypothetical, they simply spend $200k less on the lawsuit. LIC did not spend their entire budget, and charities oftentimes do not. Under ACE’s methodology, LIC’s cost-effectiveness would worsen if they spent $200k less and achieved the exact same total outcomes as a charity. The calculations we’ve done are 100% objective, and if you can find an error that we made, please let us know. You can find those calculations here:
- Original: https://docs.google.com/spreadsheets/d/1BzLUSefsd5K2uhGw81UPAe_v0E_JJR8Kdgh8zsBtxsU/edit?gid=0#gid=0
- Hypothetical: https://docs.google.com/spreadsheets/d/1IZbSfk4eNukmUU7ntRT0wCgvrLG6_ZcyXt3uANmFYbE/edit?gid=1427660869#gid=1427660869
Reading your report, I felt kind of like “oh, there are interesting assumptions here, would be interested to learn more”
What assumptions are you referring to?
If ACE didn’t exist, how would the funds the direct be spent otherwise? Would that be better or worse for animals?
If ACE didn’t exist, I would hope more funds would go to effective charities instead of ineffective ones. I hope to take part in this positive change.
- Is historical track record / cost-effectiveness the only lens on which to evaluate charities?
  If the answer is yes, seems very hard to start new things!
  I don’t know if the LIC legal case is this, but celebrating the potential impact of promising bets that didn’t pan out seems good to me.
LIC has a historical track record, and it is a bad one. People should have the opportunity to start something new. However, they shouldn’t be rated a top 11 animal charity after receiving over a million dollars in funding and failing to achieve any positive legal outcomes.
I also felt like this felt pretty politically motivated.
I saw that you call into question the integrity of the article. I want to be clear in saying that we have no relationship with any charity. However, I noticed that you co-founded the Wild Animal Initiative, which is a charity endorsed by ACE. Still, I don’t question your feedback on the article. I hope that going forward you will evaluate our reviews for what they are, rather than suggest something “political” is going on.
That being said, critique of critical organizations is great in my opinion, so appreciate you putting this out there!
Thank you, and I appreciate your feedback!