Thank you for reading some of the article. I hope that you find some time to read the rest.
To be clear, I read the whole thing—I meant that I think the fact that a pretty important issue jumped out to me within a few minutes of starting reading struck me as a reason that getting feedback from ACE seems really important.
Did you ask ACE to review this before publishing?
I really think you should! I also really think you should ask for feedback from other people who have done charity evaluations, and the charities you evaluate. You should definitely still publish them, but they’ll be better critiques for having engaged with the best case for the thing you’re critiquing!
What do you mean by conceptualizing funds? In this hypothetical, they simply spend $200k less on the lawsuit. LIC did not spend their entire budget, and charities oftentimes do not. Under ACE’s methodology, LIC’s cost-effectiveness would improve if they spent $200k less and achieved the exact same total outcomes as a charity. The calculations we’ve done are 100% objective, and if you can find an error that we made, please let us know. You can find those calculations here:
Yep, this seems right, but it’s also the case that if they did something else with that funding, the effectiveness of that action would be rated much more highly, which also seems correct. I think the issues you point to are interesting, but they strike me as intentional decisions, which ACE may have internal views on, and for which I think getting their feedback might be really important. You are correct about a mathematical fact, but both you and ACE seem to have different goals (calculating historic cost-effectiveness vs marginal impact of future dollars), and there are assumptions underlying your analysis that if changed, might change the output.
What assumptions are you referring to?
I meant ACE’s assumptions—I thought your post raised some really good questions. They are issues that if I saw, I’d email to ACE and ask why they made the choices they made, then choose whether or not to publicly publish them based on their response. Maybe these choices are reasonable, and maybe they aren’t—you raised some really good points I think. But it just seems hard to evaluate in a vacuum.
LIC has a historical track record, and it is a bad one. People should have the opportunity to start something new. However, they shouldn’t be rated a top charity after receiving over a million dollars in funding and failing to achieve any positive legal outcomes.
Again, I don’t really see good evidence for this—what is the typical track record for legal campaigns? How much do they cost? How long do they take to work? These all would be important questions to answer before claiming cost-effectiveness or lack thereof. In this case, I could easily be persuaded to agree with you, but not for any of the reasons in your analysis — the fact that they spent some money some lawsuits and it didn’t work isn’t the only evidence I’d want to think about whether or not donations to them will be useful.
Yep, this seems right, but it’s also the case that if they did something else with that funding, the effectiveness of that action would be rated much more highly, which also seems correct.
Our hypothetical in Problem 1 of the review is about two scenarios:
LIC spending $204,428 on the Costco lawsuit and achieving outcome X. (this is what actually happened)
LIC spending $1,566 on the Costco lawsuit and achievement outcome X. (this is what happened in the hypothetical)
Note that both scenarios achieve the exact same result, which is outcome X. ACE would rate LIC less cost-effective for spending $1,566 (saving $202,862) to achieve outcome X.
The hypothetical is not about something else they could do with the funding. The hypothetical is about assessing what happens to a charity’s Cost-Effectiveness Score if they save money to achieve the same outcome.
it’s also the case that if they did something else with that funding, the effectiveness of that action would be rated much more highly, which also seems correct.
How do you know that if they did something else with that funding the effectiveness of that action would be rated much more highly? According to ACE, the Costco lawsuit was a particularly cost effective intervention in spite of the fact that the lawsuit was dismissed for failing to state a valid legal claim:
“We think that out of all of Legal Impact for Chickens’ achievements, the Costco shareholder derivative case is particularly cost effective because it scored high on achievement quality.”
I’m also not sure how it would even be possible to evaluate a hypothetical in which LIC does something else with the funding. Could you explain how this hypothetical would work, and how it would be evaluated? All of the examples in our review are 100% objective and based on ACE’s own methodology. There is no subjectivity in our examples, and this was done intentionally.
You are correct about a mathematical fact, but both you and ACE seem to have different goals (calculating historic cost-effectiveness vs marginal impact of future dollars)
I’m not sure how ACE’s goals could align with the principles of effective altruism if they intentionally created a methodology that contains the problem described above.
Again, I don’t really see good evidence for this—what is the typical track record for legal campaigns? How much do they cost? How long do they take to work?
Imagine there is a law firm that has received over a million of dollars in funding, existed for multiple years, and failed to secure any favorable legal outcomes. Also imagine that their most cost-effective lawsuit was one in which they spent over $200,000, and the lawsuit was dismissed for failing to state a valid legal claim.
If this law firm were rated one of the top 100 law firms in the world, what would you think of the organization that assigned this rating? Would you say there is not good evidence for this being an incorrect rating?
ACE rated LIC as one of the Top 11 Animal Charities to Donate to in 2024. Prior to being reviewed, LIC received over a million dollars in funding, existed for multiple years, and failed to secure any favorable legal outcomes. According to ACE, LIC’s most cost-effective intervention was one in which they spent over $200,000, and the lawsuit was dismissed for failing to state a valid legal claim.
Are these factors poor evidence for LIC not being one of the 11 best animal charities to donate to?
I don’t really have a strong view about LIC—as I’ve mentioned elsewhere in the comments, I’m skeptical in general that very EA donors should give to farmed vertebrate welfare issues in the near future. But I don’t find this level of evidence particularly compelling on its own. I think I feel confused about the example you’re giving because it isn’t about hypothetical cost-effectiveness, it’s about historic cost-effectiveness, where what matters are the counterfactuals.
I broadly think the critique is interesting, and again, seems like probably an issue with the methodology, but on its own doesn’t seem like reason to think that ACE isn’t identifying good donation opportunities, because things besides cost-effectiveness also matter here.
But I don’t find this level of evidence particularly compelling on its own.
You don’t find these facts particularly compelling evidence that LIC is not historically cost-effective?
LIC’s most cost-effective intervention was one in which they spent over $200,000, and the lawsuit was dismissed for failing to state a valid legal claim.
LIC received over a million dollars in funding prior to being reviewed
LIC existed for multiple years prior to being reviewed
LIC failed to secure any favorable legal outcomes, or file any lawsuit that stated a valid legal claim?
What would be compelling evidence for LIC not being historically cost-effective?
I think I feel confused about the example you’re giving because it isn’t about hypothetical cost-effectiveness, it’s about historic cost-effectiveness, where what matters are the counterfactuals.
ACE does 2 separate analyses for past cost-effectiveness, and room for future funding. For example, those two sections in ACE’s review of LIC are:
Cost Effectiveness: How much has Legal Impact for Chickens achieved through their programs?
Room For More Funding: How much additional money can Legal Impact for Chickens effectively use in the next two years?
Our review focuses on ACE’s Cost-Effectiveness analysis, not on their Room For More Funding analysis. In the future, we may evaluate ACE’s Room For More Funding Analysis, but that is not what our review focused on.
However, I would like to pose a question to you: Given the ACE often gives charities a worse historic cost-effectiveness rating for spending less money to achieve the exact same outcomes (see Problem 1), how confident do you feel in ACE’s ability to analyze future cost-effectiveness (which is inherently more difficult to analyze)?
I don’t find that evidence particularly compelling on its own, no. Lots of projects cost more than 1M or take more than a few years to have success. I don’t see why those things would be cause to dismiss a project out of hand. I don’t really buy social movement theories of change for animal advocacy, but many people do, and it just seems like many social movement-y things take a long time to build momentum, and legal and research-focused projects take forever to play out. Things I’d want to look at to form a view on this (though to be clear, I plausibly agree with you!):
How much lawsuits of this type typically cost
What the base rate for success is for this kind of work
How long this kind of work typically takes to get traction
Has anyone else tried similar work on misleading labelling or whatever? Was it effective or not?
Has LIC’s work inspired other lawsuits, as ACE reported might be a positive side effect?
I don’t think we disagree that much here, except how much these things matter — I don’t really care about ACE’s ability to analyze cost-effectiveness outside broad strokes because I think the primary benefits of organizations like ACE is shifting money to more cost-effective things within the animal space, which I do believe ACE does. I also don’t mind ACE endorsing speculative bets that don’t pay off — I think there are many things that were worth paying for in expectation that don’t end up helping any animals, and will continue to be, because we don’t really know very many effective ways to help animals so the information value of trying new things is high.
But to answer your question specifically, I’d be very skeptical of anyone’s numbers on future cost-effectiveness, ACE’s or yours or my own, because I think this is an issue that has historically been extremely difficult to estimate cost-effectiveness for. I’m not convinced that’s the right way to approach identifying effective animal interventions, in part because it is so hard to do well. I don’t really think ACE is making cost-effectiveness estimates here though—it seems much more like trying to get a rough sense of relative cost-effectiveness, which, putting aside the methodological issues you’ve raised, seems like the right approach to me, but only a small part of the information I’d want to know where money should move in animal advocacy.
What the base rate for success is for this kind of work
How long this kind of work typically takes to get traction
The Nonhuman Rights Project provides a possible point of comparison. From 2013 to 2023 they raised $13.2 Million. As far as I know, they have never won a case.
I don’t find that evidence particularly compelling on its own, no. Lots of projects cost more than 1M or take more than a few years to have success. I don’t see why those things would be cause to dismiss a project out of hand.
The question I asked was: “You don’t find these facts particularly compelling evidence that LIC is not historically cost-effective?”
The question was not about whether these facts are compelling evidence that LIC won’t be successful in the future, or if the project should be dismissed.
Wait, those are related to each other though—if we haven’t seen the full impact of their previous actions, we haven’t yet seen their historical cost-effectiveness in full! Also, you cite these as reasons the project should be dismissed in your post—you have a section literally called “Legal Impact for Chickens Did Not Achieve Any Favorable Legal Outcomes, Yet ACE Rated Them a Top Charity” which reads to me that you believe that it is bad they were rated a Top Charity, and make these same arguments (and no others) in the section, suggesting that you think this evidence means they should be dismissed.
Wait, those are related to each other though—if we haven’t seen the full impact of their previous actions, we haven’t yet seen their historical cost-effectiveness in full!
No, they are not. Historical cost-effectiveness refers to past actions and outcomes—what has already occurred.
All of LIC’s legal actions have already been either dismissed or rejected. What are you suggesting we need to wait for before we can analyze LIC’s historical cost-effectiveness in full?
You are conflating the issue of past cost-effectiveness with future potential.
Also, you cite these as reasons the project should be dismissed in your post—you have a section literally called “Legal Impact for Chickens Did Not Achieve Any Favorable Legal Outcomes, Yet ACE Rated Them a Top Charity” which reads to me that you believe that it is bad they were rated a Top Charity, and make these same arguments (and no others) in the section, suggesting that you think this evidence means they should be dismissed.
Did I claim that I don’t think LIC “should be dismissed”?
both you and ACE seem to have different goals (calculating historic cost-effectiveness vs marginal impact of future dollars)
ACE states (under Criterion 2) that a charity’s Cost-Effectiveness Score “indicates, on a 1-7 scale, how cost effective we think the charity has been [...] with higher scores indicating higher cost effectiveness.”
Would you mind clarifying what you believe ACE’s goal is, and what you believe my goal is?
The analysis in my review is entirely about calculating historic cost-effectiveness. ACE’s Cost-Effectiveness Scores are also entirely about calculating historic cost-effectiveness.
From this post, it seems like you’re trying to calculate historic cost-effectiveness and rate charities exclusively on that (since you haven’t published an evaluation of an animal charity yet I could be wrong here though). My understanding of what ACE is trying to do with its evaluations as a whole is identify where marginal dollars might be most useful for animal advocacy, and move money from less effective opportunities to those. Cost-effectiveness might be one component of that, but is far from the only one (e.g. intervention scalability might matter, having a diversity of types of opportunities to appeal to different donors, etc.). It’s pretty easy to imagine scenarios where you wouldn’t prefer to only look at cost-effectiveness of individual charities when making recommendation, even if that’s what matters in the end. It’s also easy to imagine scenarios where recommending less effective opportunities leads to better outcomes to animals—maybe installing shrimp stunners is super effective, but only some donors will give to it. Maybe it can only scale to a few M per year but you influence more money than that. Depending on your circumstances, a lot more than cost-effectiveness of specific interventions matters for making the most effective recommendations.
My understanding is also that ACE doesn’t see EAs as its primary audience (but I’m less certain about this). This is a reason I’m excited about your project—seems nice to have “very EA” evaluations of charities in addition to ACE’s. But, I also imagine it would be hard to get charities to participate in your evaluation process if you don’t run the evaluations by them in advance, which could make it hard for you to get information to do what you’re trying to do, unless you rely on the information ACE collects, which then puts you in an awkward position of making a strong argument against an organization you might need to conduct evaluations.
My understanding is ACE has tried to do something that’s just cost-effectiveness analysis in the past (they used to give probability distributions for how many animals were helped, for example). But it’s really difficult to do confidently for animal issues, and that’s part of the reason it’s only a portion of the whole picture (along with other factors like I mention above).
From this post, it seems like you’re trying to calculate historic cost-effectiveness and rate charities exclusively on that (since you haven’t published an evaluation of an animal charity yet I could be wrong here though)
This is not what we are trying to do. We simply critiqued the way that ACE calculated historic cost-effectiveness, and how ACE gave Legal Impact for Chickens a relatively high historic cost-effectiveness rating despite have no historic success.
My understanding of what ACE is trying to do with its evaluations as a whole is identify where marginal dollars might be most useful for animal advocacy, and move money from less effective opportunities to those.
ACE does 2 separate analyses for past cost-effectiveness, and room for future funding. For example, those two sections in ACE’s review of LIC are:
Cost Effectiveness: How much has Legal Impact for Chickens achieved through their programs?
Room For More Funding: How much additional money can Legal Impact for Chickens effectively use in the next two years?
Our review focuses on ACE’s Cost-Effectiveness analysis, not on their Room For More Funding analysis. In the future, we may evaluate ACE’s Room For More Funding Analysis, but that is not what our review focused on. We wanted to keep our review short enough that people could read it without a huge time investment, so we could not include an assessment of every single part of ACE’s evaluation process in our review.
It is also less reasonable to hold ACE accountable for their Room For More Funding analysis, since this is inherently more subjective and difficult to do. It is far easier for ACE (or any charity evaluator) to analyze historic cost-effectiveness than to analyze future cost-effectiveness. However, I would like to pose a question to you: Given the ACE often gives charities a worse historic cost-effectiveness rating for spending less money to achieve the exact same outcomes (see Problem 1), how confident do you feel in ACE’s ability to analyze future cost-effectiveness?
My understanding is ACE has tried to do something that’s just cost-effectiveness analysis in the past (they used to give probability distributions for how many animals were helped, for example).
ACE responded to this thread acknowledging that the problems listed in our review needed to be addressed, and that they changed their methodology (to a cost-effectiveness calculation of simply impact divided by cost) to do so:
This is not what we are trying to do. We simply critiqued the way that ACE calculated historic cost-effectiveness, and how ACE gave Legal Impact for Chickens a relatively high historic cost-effectiveness rating despite have no historic success.
FWIW this seems great—excited to see more comprehensive evaluations. Yeah, I agree with many of your comments here on the granular level — it seems you found something that is a potential issue for how ACE does (or did) some aspects of their evaluations, and publishing that is great! I think we just disagree on how important it is?
By the way, I’m ending further engagement on this (though feel free to leave a response if useful!) just because I already find the EA Forum distracting from other work, and don’t have time this week to think about this more. Appreciate you going through everything with me!
To be clear, I read the whole thing—I meant that I think the fact that a pretty important issue jumped out to me within a few minutes of starting reading struck me as a reason that getting feedback from ACE seems really important.
I really think you should! I also really think you should ask for feedback from other people who have done charity evaluations, and the charities you evaluate. You should definitely still publish them, but they’ll be better critiques for having engaged with the best case for the thing you’re critiquing!
Yep, this seems right, but it’s also the case that if they did something else with that funding, the effectiveness of that action would be rated much more highly, which also seems correct. I think the issues you point to are interesting, but they strike me as intentional decisions, which ACE may have internal views on, and for which I think getting their feedback might be really important. You are correct about a mathematical fact, but both you and ACE seem to have different goals (calculating historic cost-effectiveness vs marginal impact of future dollars), and there are assumptions underlying your analysis that if changed, might change the output.
I meant ACE’s assumptions—I thought your post raised some really good questions. They are issues that if I saw, I’d email to ACE and ask why they made the choices they made, then choose whether or not to publicly publish them based on their response. Maybe these choices are reasonable, and maybe they aren’t—you raised some really good points I think. But it just seems hard to evaluate in a vacuum.
Again, I don’t really see good evidence for this—what is the typical track record for legal campaigns? How much do they cost? How long do they take to work? These all would be important questions to answer before claiming cost-effectiveness or lack thereof. In this case, I could easily be persuaded to agree with you, but not for any of the reasons in your analysis — the fact that they spent some money some lawsuits and it didn’t work isn’t the only evidence I’d want to think about whether or not donations to them will be useful.
Our hypothetical in Problem 1 of the review is about two scenarios:
LIC spending $204,428 on the Costco lawsuit and achieving outcome X. (this is what actually happened)
LIC spending $1,566 on the Costco lawsuit and achievement outcome X. (this is what happened in the hypothetical)
Note that both scenarios achieve the exact same result, which is outcome X. ACE would rate LIC less cost-effective for spending $1,566 (saving $202,862) to achieve outcome X.
The hypothetical is not about something else they could do with the funding. The hypothetical is about assessing what happens to a charity’s Cost-Effectiveness Score if they save money to achieve the same outcome.
How do you know that if they did something else with that funding the effectiveness of that action would be rated much more highly? According to ACE, the Costco lawsuit was a particularly cost effective intervention in spite of the fact that the lawsuit was dismissed for failing to state a valid legal claim:
“We think that out of all of Legal Impact for Chickens’ achievements, the Costco shareholder derivative case is particularly cost effective because it scored high on achievement quality.”
I’m also not sure how it would even be possible to evaluate a hypothetical in which LIC does something else with the funding. Could you explain how this hypothetical would work, and how it would be evaluated? All of the examples in our review are 100% objective and based on ACE’s own methodology. There is no subjectivity in our examples, and this was done intentionally.
I’m not sure how ACE’s goals could align with the principles of effective altruism if they intentionally created a methodology that contains the problem described above.
Imagine there is a law firm that has received over a million of dollars in funding, existed for multiple years, and failed to secure any favorable legal outcomes. Also imagine that their most cost-effective lawsuit was one in which they spent over $200,000, and the lawsuit was dismissed for failing to state a valid legal claim.
If this law firm were rated one of the top 100 law firms in the world, what would you think of the organization that assigned this rating? Would you say there is not good evidence for this being an incorrect rating?
ACE rated LIC as one of the Top 11 Animal Charities to Donate to in 2024. Prior to being reviewed, LIC received over a million dollars in funding, existed for multiple years, and failed to secure any favorable legal outcomes. According to ACE, LIC’s most cost-effective intervention was one in which they spent over $200,000, and the lawsuit was dismissed for failing to state a valid legal claim.
Are these factors poor evidence for LIC not being one of the 11 best animal charities to donate to?
I don’t really have a strong view about LIC—as I’ve mentioned elsewhere in the comments, I’m skeptical in general that very EA donors should give to farmed vertebrate welfare issues in the near future. But I don’t find this level of evidence particularly compelling on its own. I think I feel confused about the example you’re giving because it isn’t about hypothetical cost-effectiveness, it’s about historic cost-effectiveness, where what matters are the counterfactuals.
I broadly think the critique is interesting, and again, seems like probably an issue with the methodology, but on its own doesn’t seem like reason to think that ACE isn’t identifying good donation opportunities, because things besides cost-effectiveness also matter here.
You don’t find these facts particularly compelling evidence that LIC is not historically cost-effective?
LIC’s most cost-effective intervention was one in which they spent over $200,000, and the lawsuit was dismissed for failing to state a valid legal claim.
LIC received over a million dollars in funding prior to being reviewed
LIC existed for multiple years prior to being reviewed
LIC failed to secure any favorable legal outcomes, or file any lawsuit that stated a valid legal claim?
What would be compelling evidence for LIC not being historically cost-effective?
ACE does 2 separate analyses for past cost-effectiveness, and room for future funding. For example, those two sections in ACE’s review of LIC are:
Cost Effectiveness: How much has Legal Impact for Chickens achieved through their programs?
Room For More Funding: How much additional money can Legal Impact for Chickens effectively use in the next two years?
Our review focuses on ACE’s Cost-Effectiveness analysis, not on their Room For More Funding analysis. In the future, we may evaluate ACE’s Room For More Funding Analysis, but that is not what our review focused on.
However, I would like to pose a question to you: Given the ACE often gives charities a worse historic cost-effectiveness rating for spending less money to achieve the exact same outcomes (see Problem 1), how confident do you feel in ACE’s ability to analyze future cost-effectiveness (which is inherently more difficult to analyze)?
I don’t find that evidence particularly compelling on its own, no. Lots of projects cost more than 1M or take more than a few years to have success. I don’t see why those things would be cause to dismiss a project out of hand. I don’t really buy social movement theories of change for animal advocacy, but many people do, and it just seems like many social movement-y things take a long time to build momentum, and legal and research-focused projects take forever to play out. Things I’d want to look at to form a view on this (though to be clear, I plausibly agree with you!):
How much lawsuits of this type typically cost
What the base rate for success is for this kind of work
How long this kind of work typically takes to get traction
Has anyone else tried similar work on misleading labelling or whatever? Was it effective or not?
Has LIC’s work inspired other lawsuits, as ACE reported might be a positive side effect?
I don’t think we disagree that much here, except how much these things matter — I don’t really care about ACE’s ability to analyze cost-effectiveness outside broad strokes because I think the primary benefits of organizations like ACE is shifting money to more cost-effective things within the animal space, which I do believe ACE does. I also don’t mind ACE endorsing speculative bets that don’t pay off — I think there are many things that were worth paying for in expectation that don’t end up helping any animals, and will continue to be, because we don’t really know very many effective ways to help animals so the information value of trying new things is high.
But to answer your question specifically, I’d be very skeptical of anyone’s numbers on future cost-effectiveness, ACE’s or yours or my own, because I think this is an issue that has historically been extremely difficult to estimate cost-effectiveness for. I’m not convinced that’s the right way to approach identifying effective animal interventions, in part because it is so hard to do well. I don’t really think ACE is making cost-effectiveness estimates here though—it seems much more like trying to get a rough sense of relative cost-effectiveness, which, putting aside the methodological issues you’ve raised, seems like the right approach to me, but only a small part of the information I’d want to know where money should move in animal advocacy.
How much lawsuits of this type typically cost
What the base rate for success is for this kind of work
How long this kind of work typically takes to get traction
The Nonhuman Rights Project provides a possible point of comparison. From 2013 to 2023 they raised $13.2 Million. As far as I know, they have never won a case.
The question I asked was: “You don’t find these facts particularly compelling evidence that LIC is not historically cost-effective?”
The question was not about whether these facts are compelling evidence that LIC won’t be successful in the future, or if the project should be dismissed.
Wait, those are related to each other though—if we haven’t seen the full impact of their previous actions, we haven’t yet seen their historical cost-effectiveness in full! Also, you cite these as reasons the project should be dismissed in your post—you have a section literally called “Legal Impact for Chickens Did Not Achieve Any Favorable Legal Outcomes, Yet ACE Rated Them a Top Charity” which reads to me that you believe that it is bad they were rated a Top Charity, and make these same arguments (and no others) in the section, suggesting that you think this evidence means they should be dismissed.
No, they are not. Historical cost-effectiveness refers to past actions and outcomes—what has already occurred.
All of LIC’s legal actions have already been either dismissed or rejected. What are you suggesting we need to wait for before we can analyze LIC’s historical cost-effectiveness in full?
You are conflating the issue of past cost-effectiveness with future potential.
Did I claim that I don’t think LIC “should be dismissed”?
ACE states (under Criterion 2) that a charity’s Cost-Effectiveness Score “indicates, on a 1-7 scale, how cost effective we think the charity has been [...] with higher scores indicating higher cost effectiveness.”
Would you mind clarifying what you believe ACE’s goal is, and what you believe my goal is?
The analysis in my review is entirely about calculating historic cost-effectiveness. ACE’s Cost-Effectiveness Scores are also entirely about calculating historic cost-effectiveness.
From this post, it seems like you’re trying to calculate historic cost-effectiveness and rate charities exclusively on that (since you haven’t published an evaluation of an animal charity yet I could be wrong here though). My understanding of what ACE is trying to do with its evaluations as a whole is identify where marginal dollars might be most useful for animal advocacy, and move money from less effective opportunities to those. Cost-effectiveness might be one component of that, but is far from the only one (e.g. intervention scalability might matter, having a diversity of types of opportunities to appeal to different donors, etc.). It’s pretty easy to imagine scenarios where you wouldn’t prefer to only look at cost-effectiveness of individual charities when making recommendation, even if that’s what matters in the end. It’s also easy to imagine scenarios where recommending less effective opportunities leads to better outcomes to animals—maybe installing shrimp stunners is super effective, but only some donors will give to it. Maybe it can only scale to a few M per year but you influence more money than that. Depending on your circumstances, a lot more than cost-effectiveness of specific interventions matters for making the most effective recommendations.
My understanding is also that ACE doesn’t see EAs as its primary audience (but I’m less certain about this). This is a reason I’m excited about your project—seems nice to have “very EA” evaluations of charities in addition to ACE’s. But, I also imagine it would be hard to get charities to participate in your evaluation process if you don’t run the evaluations by them in advance, which could make it hard for you to get information to do what you’re trying to do, unless you rely on the information ACE collects, which then puts you in an awkward position of making a strong argument against an organization you might need to conduct evaluations.
My understanding is ACE has tried to do something that’s just cost-effectiveness analysis in the past (they used to give probability distributions for how many animals were helped, for example). But it’s really difficult to do confidently for animal issues, and that’s part of the reason it’s only a portion of the whole picture (along with other factors like I mention above).
Thank you for your response!
This is not what we are trying to do. We simply critiqued the way that ACE calculated historic cost-effectiveness, and how ACE gave Legal Impact for Chickens a relatively high historic cost-effectiveness rating despite have no historic success.
ACE does 2 separate analyses for past cost-effectiveness, and room for future funding. For example, those two sections in ACE’s review of LIC are:
Cost Effectiveness: How much has Legal Impact for Chickens achieved through their programs?
Room For More Funding: How much additional money can Legal Impact for Chickens effectively use in the next two years?
Our review focuses on ACE’s Cost-Effectiveness analysis, not on their Room For More Funding analysis. In the future, we may evaluate ACE’s Room For More Funding Analysis, but that is not what our review focused on. We wanted to keep our review short enough that people could read it without a huge time investment, so we could not include an assessment of every single part of ACE’s evaluation process in our review.
It is also less reasonable to hold ACE accountable for their Room For More Funding analysis, since this is inherently more subjective and difficult to do. It is far easier for ACE (or any charity evaluator) to analyze historic cost-effectiveness than to analyze future cost-effectiveness. However, I would like to pose a question to you: Given the ACE often gives charities a worse historic cost-effectiveness rating for spending less money to achieve the exact same outcomes (see Problem 1), how confident do you feel in ACE’s ability to analyze future cost-effectiveness?
ACE responded to this thread acknowledging that the problems listed in our review needed to be addressed, and that they changed their methodology (to a cost-effectiveness calculation of simply impact divided by cost) to do so:
FWIW this seems great—excited to see more comprehensive evaluations. Yeah, I agree with many of your comments here on the granular level — it seems you found something that is a potential issue for how ACE does (or did) some aspects of their evaluations, and publishing that is great! I think we just disagree on how important it is?
By the way, I’m ending further engagement on this (though feel free to leave a response if useful!) just because I already find the EA Forum distracting from other work, and don’t have time this week to think about this more. Appreciate you going through everything with me!
No problem. Thank you for your replies!