If you’re correct in the linked analysis, this sounds like a really important limitation in ACE’s methodology, and I’m very glad you’ve shared this!
In case anyone else has the same confusion as me when reading your summary: I think there is nothing wrong with calculating a charity’s cost effectiveness by taking the weighted sum of the cost-effectiveness of all of their interventions (weighted by share of total funding that intervention receives). This should mathematically be the same as (Total Impact / Total cost), and so should indeed go up if their spending on a particular intervention goes down (while achieving the same impact).
The (claimed) cause of the problem is just that ACE’s cost-effectiveness estimate does not go up by anywhere near as much as it should when the cost of an intervention is reduced, leading the cost-effectiveness of the charity as a whole to actually change in the wrong direction when doing the above weighted sum!
If this is true it sounds pretty bad. Would be interested to read a response from them.
Of course, the other thing that could be going on here, is that average cost-effectiveness is not the same as cost-effectiveness on the margin, which is presumably what ACE should care about. Though I don’t see why an intervention representing a smaller share of a charity’s expenditure should automatically mean that this is not where extra dollars would be allocated. The two things seem independent to me.
Of course, the other thing that could be going on here, is that average cost-effectiveness is not the same as cost-effectiveness on the margin, which is presumably what ACE should care about.
I’m not certain if by cost-effectiveness on the margin, you meant cost-effectiveness in the future if additional funding is obtained. If that’s the case, the following information could be helpful.
ACE does 2 separate analyses for past cost-effectiveness, and room for future funding. For example, those two sections in ACE’s review of LIC are:
Cost Effectiveness: How much has Legal Impact for Chickens achieved through their programs?
Room For More Funding: How much additional money can Legal Impact for Chickens effectively use in the next two years?
Our review focuses on ACE’s Cost-Effectiveness analysis. Additionally, ACE states (under Criterion 2) that a charity’s Cost-Effectiveness Score “indicates, on a 1-7 scale, how cost effective we think the charity has been [...] with higher scores indicating higher cost effectiveness.”
If you’re correct in the linked analysis, this sounds like a really important limitation in ACE’s methodology, and I’m very glad you’ve shared this!
In case anyone else has the same confusion as me when reading your summary: I think there is nothing wrong with calculating a charity’s cost effectiveness by taking the weighted sum of the cost-effectiveness of all of their interventions (weighted by share of total funding that intervention receives). This should mathematically be the same as (Total Impact / Total cost), and so should indeed go up if their spending on a particular intervention goes down (while achieving the same impact).
The (claimed) cause of the problem is just that ACE’s cost-effectiveness estimate does not go up by anywhere near as much as it should when the cost of an intervention is reduced, leading the cost-effectiveness of the charity as a whole to actually change in the wrong direction when doing the above weighted sum!
If this is true it sounds pretty bad. Would be interested to read a response from them.
Of course, the other thing that could be going on here, is that average cost-effectiveness is not the same as cost-effectiveness on the margin, which is presumably what ACE should care about. Though I don’t see why an intervention representing a smaller share of a charity’s expenditure should automatically mean that this is not where extra dollars would be allocated. The two things seem independent to me.
Hi Toby,
Thank you for your reply!
I’m not certain if by cost-effectiveness on the margin, you meant cost-effectiveness in the future if additional funding is obtained. If that’s the case, the following information could be helpful.
ACE does 2 separate analyses for past cost-effectiveness, and room for future funding. For example, those two sections in ACE’s review of LIC are:
Cost Effectiveness: How much has Legal Impact for Chickens achieved through their programs?
Room For More Funding: How much additional money can Legal Impact for Chickens effectively use in the next two years?
Our review focuses on ACE’s Cost-Effectiveness analysis. Additionally, ACE states (under Criterion 2) that a charity’s Cost-Effectiveness Score “indicates, on a 1-7 scale, how cost effective we think the charity has been [...] with higher scores indicating higher cost effectiveness.”
This is very helpful, thanks!