So, I have some mixed views about this post. Letâs start with the positive.
In terms of agreement: I do think organizational critics are valuable, and specifically, critics of ACE in the past have been helpful in improving their direction and impact. I also love the idea of having more charity evaluators (even in the same cause area) with slightly different methods or approaches to determining how to do good, so Iâm excited to see this initiative. I also have quite a bit of sympathy for giving higher weight to explicit cost-effectiveness models when it comes to animal welfare evaluations.
I can personally relate to the feeling of being disappointed after digging deeper into the numbers of well-respected EA meta organizations, so I understand the tone and frustration. However, I suspect your arguments may get a lot of pushback on tone alone, which could distract from the more important substance of the post and concepts (Iâll leave that for others to address, as it feels less important, in my opinion).
In terms of disagreement: I will focus on what I think is the crux of the issue, which I would summarize as: (a) ACE uses a methodology that yields quite different results than a raw cost-effectiveness analysis; (b) this methodology seems to have major flaws, as it can lead to clearly incoherent conclusions and recommendations easily; and (c) thus, it is better to use a more straightforward, direct CEA.
I agree with points A and B, but I am much less convinced about point C. To me, this feels a bit like an isolated demand for methodological rigor. Every methodology has flaws, and itâs easy to find situations that lead to clearly incoherent conclusions. Expected value theory itself, using pure EV terms, has well-known issues like St. Petersburg Paradox, optimizerâs curse, and general model mistakes. CEAs in general share these issues and have additional flaws (see more on this here). I think CEAs are a super useful tool, but they are ultimately a model of reality, not reality itself, and I think EA can sometimes get too caught up in them (whereas the rest of the world probably doesnât use them nearly enough). GW, which has ~20x the budget of ACE, still finds model errors and openly discusses how softer judgments on ethics and discount factors influence outcomes (and they consider more than just a pureCEA calculation when recommending a charity).
Overall, being pretty familiar with ACEâs methodology and CEAs, I would expect, for example, that a 10-hour CEA of the same organizations would be quite a bit further from the truth of the actual impact or effectiveness of an organization. Itâs not clear to me that spending equal time on pure CEAs versus a mix of evaluative techniques (as ACE currently does) would lead to more accurate results (I would probably weakly bet against it). I think this post overstates the importance of discarding a model due to a flaw that can be exploited.
A softer argument, such as âACE should spend double the percentage of time it currently spends on CEAs relative to other methodsâ or âACE should ensure that intervention weightings do not overshadow program-level execution data,â is something I have a lot of sympathy for.
(a) ACE uses a methodology that yields quite different results than a raw cost-effectiveness analysis; (b) this methodology seems to have major flaws, as it can lead to clearly incoherent conclusions and recommendations easily; and (c) thus, it is better to use a more straightforward, direct CEA.
I agree with points A and B, but I am much less convinced about point C.
I would like to point to Problem 1 and Problem 4 from the review:
Charities can receive a worse Cost-Effectiveness Score by spending less money to achieve the exact same results.
Charities can have 1,000,000 times the impact at the exact same price, and their Cost-Effectiveness Score can remain the same.
Effective giving is all about achieving the greatest impact at the lowest cost. ACEâs methodology is not properly accounting for impact, or for cost.
Using the equation impact /â cost at least results in impact being in the numerator, and cost being in the denominator. To me, this alone makes a straightforward, direct CEA a better methodology than the one used by ACE.
To me, this feels a bit like an isolated demand for methodological rigor. Every methodology has flaws, and itâs easy to find situations that lead to clearly incoherent conclusions.
I absolutely agree that every methodology has flaws, and we did not mean to imply otherwise. However, the incoherent conclusions described in our review of ACEâs methodology are not one off instances. hey are pervasive problems that impact all of ACEâs reviews.
So, I have some mixed views about this post. Letâs start with the positive.
In terms of agreement: I do think organizational critics are valuable, and specifically, critics of ACE in the past have been helpful in improving their direction and impact. I also love the idea of having more charity evaluators (even in the same cause area) with slightly different methods or approaches to determining how to do good, so Iâm excited to see this initiative. I also have quite a bit of sympathy for giving higher weight to explicit cost-effectiveness models when it comes to animal welfare evaluations.
I can personally relate to the feeling of being disappointed after digging deeper into the numbers of well-respected EA meta organizations, so I understand the tone and frustration. However, I suspect your arguments may get a lot of pushback on tone alone, which could distract from the more important substance of the post and concepts (Iâll leave that for others to address, as it feels less important, in my opinion).
In terms of disagreement: I will focus on what I think is the crux of the issue, which I would summarize as: (a) ACE uses a methodology that yields quite different results than a raw cost-effectiveness analysis; (b) this methodology seems to have major flaws, as it can lead to clearly incoherent conclusions and recommendations easily; and (c) thus, it is better to use a more straightforward, direct CEA.
I agree with points A and B, but I am much less convinced about point C. To me, this feels a bit like an isolated demand for methodological rigor. Every methodology has flaws, and itâs easy to find situations that lead to clearly incoherent conclusions. Expected value theory itself, using pure EV terms, has well-known issues like St. Petersburg Paradox, optimizerâs curse, and general model mistakes. CEAs in general share these issues and have additional flaws (see more on this here). I think CEAs are a super useful tool, but they are ultimately a model of reality, not reality itself, and I think EA can sometimes get too caught up in them (whereas the rest of the world probably doesnât use them nearly enough). GW, which has ~20x the budget of ACE, still finds model errors and openly discusses how softer judgments on ethics and discount factors influence outcomes (and they consider more than just a pure CEA calculation when recommending a charity).
Overall, being pretty familiar with ACEâs methodology and CEAs, I would expect, for example, that a 10-hour CEA of the same organizations would be quite a bit further from the truth of the actual impact or effectiveness of an organization. Itâs not clear to me that spending equal time on pure CEAs versus a mix of evaluative techniques (as ACE currently does) would lead to more accurate results (I would probably weakly bet against it). I think this post overstates the importance of discarding a model due to a flaw that can be exploited.
A softer argument, such as âACE should spend double the percentage of time it currently spends on CEAs relative to other methodsâ or âACE should ensure that intervention weightings do not overshadow program-level execution data,â is something I have a lot of sympathy for.
Hi Joey,
Thank you for taking the time to read our review!
I would like to point to Problem 1 and Problem 4 from the review:
Charities can receive a worse Cost-Effectiveness Score by spending less money to achieve the exact same results.
Charities can have 1,000,000 times the impact at the exact same price, and their Cost-Effectiveness Score can remain the same.
Effective giving is all about achieving the greatest impact at the lowest cost. ACEâs methodology is not properly accounting for impact, or for cost.
Using the equation impact /â cost at least results in impact being in the numerator, and cost being in the denominator. To me, this alone makes a straightforward, direct CEA a better methodology than the one used by ACE.
I absolutely agree that every methodology has flaws, and we did not mean to imply otherwise. However, the incoherent conclusions described in our review of ACEâs methodology are not one off instances. hey are pervasive problems that impact all of ACEâs reviews.
Thank you for your feedback!