Thank you for this detailed reply! This agrees with my intuition that the numbers the volunteer told me were not at all realistic. My main goal with writing this post was not to question the effectiveness, but to bring it to the attention of the community that there is value in making this kind of information (such as what you write in this comment) available publicly, instead of the current silence on mega-charities.
I completely agree there would be enormous value in making this kind of info public. The BINGOS (Big NGOs) know that too, but they also know that if they publically estimated cost effectiveness of their interventions, it might make for pretty sombre reading as ther cost effectiveness might be orders of magnitude less than many other orgs—and I’m not just talking the creme de-la creme Givewell ones.
There’s also the problem that they would probably expose that some of their most public “Marquee” interventions might be less effecitve than some of their lesser known interventions, even within the same BINGO. For example if they found out that the small amount of money they spent deworming to prevent river blindness was far more cost-effective than their more heart-throb malnutrition intervention they are advertising, how would they respond to that?
Unfortunately it is most likelyIn the BINGOs best interests NOT to do (or at least not to publish) cost-effectiveness analysis on their interventions. It is a rational decision on their part. All they might achieve is more scrutiny on how low-impact their interventions were and perhaps even get less funding as a result
That’s why they don’t publish these kinds of analysis, not because its that hard to do—they have the staffing and expertise. Small orgs like ours and Lafiya Nigeria have had a go even without specific expertise. I’m somewhat surprised UNICEF dipped their toe into publicly sharing any kind of cost-effectiveness here, which we possibly should be giving them some credit for. At least they kind of tried?
I realise it sounds like this is dripping in cynicism, but I genuinely think this explains to a large part why the BINGOs don’t publish this kind of info and instead go full ostrich.
I really appreciate your insight here. I’ve been thinking lately about lobbying the IRS in the US to require cost effectiveness disclosures in yearly reports. There are a couple concerns I have...
1: if it is even possible to convince the IRS to add cost effectiveness disclosures to the 990
2: if orgs have the expertise/capacity to evaluate their programs
3: I think the disclosure requirement must be VERY vague to allow orgs to disclose what they think is most appropriate
4: lack of oversight means these disclosure could be easily exaggerated
Benefits:
Within a similar outcome, effectiveness could be compared across orgs. Something we have never been able to do without folks like Givewell, so things like the program ratio (ew) are used by many donors.
What gets measured gets managed, so this will likely lead to orgs operating more effectively overall.
IRS and US charity regulators do a lousy job of preventing charitable scams and near-scams. I don’t see any likely universe in which they have bandwidth to decide on a cost-effectiveness methodology that works for 1.5 MM US charities doing a extremely wide range of activities.
Very likely it would be so loose all charities could give themselves high marks.… and if it were somehow not loose, there’s no clear reason to think the resultant methodology would be anything like EA.
After all, it’s a political decision (my guess is that it would require legislation by Congress, not just Treasury regs: medium confidence as a legal matter without wading through the tax code, high confidence that IRS would never do this w/o Congressional directive).
Fair points, and the idea is certainly a massive longshot.
It’s unclear to me exactly how the decision for the big redesign in 2007 was made, and if congress was involved at all. My guess is that a change like this could be made just by the IRS.
I am FAR from an expert on evaluation but even if orgs just reported outputs (X people helped, Y houses built, etc.), that information would be immensely useful. Guidestar is allowing orgs to disclose output metrics to get a platinum seal, but this is a voluntary disclosure. Form 990 could use a similar library of metrics for orgs to select from and make disclosure mandatory. I don’t think the form would be like EA at all—but the data would be extremely useful for donors/orgs trying to actually evaluate charity effectiveness.
At the end of the day, orgs may just completely BS the outputs so the idea certainly needs some work. Maybe encourage audits of outcomes?
I read an org’s 990 before making a non-trivial donation, and my guess from your bio is that you do too. But I wonder how many people (1) would take the time to carefully read the 990, (2) have enough methodological sophistication to see through at least moderate levels of obfuscation, but (3) are not significant enough donors that they feel (or are) empowered to call someone and ask for someone to provide relevant information.
Thanks Kyle, I like the idea of the IRS cost effectiveness thing but agree with Jason that practially it would never work. Even if it came in, it would be so loose as to be meaningless.
Thank you for this detailed reply! This agrees with my intuition that the numbers the volunteer told me were not at all realistic. My main goal with writing this post was not to question the effectiveness, but to bring it to the attention of the community that there is value in making this kind of information (such as what you write in this comment) available publicly, instead of the current silence on mega-charities.
Nice one Jesper!
I completely agree there would be enormous value in making this kind of info public. The BINGOS (Big NGOs) know that too, but they also know that if they publically estimated cost effectiveness of their interventions, it might make for pretty sombre reading as ther cost effectiveness might be orders of magnitude less than many other orgs—and I’m not just talking the creme de-la creme Givewell ones.
There’s also the problem that they would probably expose that some of their most public “Marquee” interventions might be less effecitve than some of their lesser known interventions, even within the same BINGO. For example if they found out that the small amount of money they spent deworming to prevent river blindness was far more cost-effective than their more heart-throb malnutrition intervention they are advertising, how would they respond to that?
Unfortunately it is most likely In the BINGOs best interests NOT to do (or at least not to publish) cost-effectiveness analysis on their interventions. It is a rational decision on their part. All they might achieve is more scrutiny on how low-impact their interventions were and perhaps even get less funding as a result
That’s why they don’t publish these kinds of analysis, not because its that hard to do—they have the staffing and expertise. Small orgs like ours and Lafiya Nigeria have had a go even without specific expertise. I’m somewhat surprised UNICEF dipped their toe into publicly sharing any kind of cost-effectiveness here, which we possibly should be giving them some credit for. At least they kind of tried?
I realise it sounds like this is dripping in cynicism, but I genuinely think this explains to a large part why the BINGOs don’t publish this kind of info and instead go full ostrich.
Hi Nick,
I really appreciate your insight here. I’ve been thinking lately about lobbying the IRS in the US to require cost effectiveness disclosures in yearly reports. There are a couple concerns I have...
1: if it is even possible to convince the IRS to add cost effectiveness disclosures to the 990
2: if orgs have the expertise/capacity to evaluate their programs
3: I think the disclosure requirement must be VERY vague to allow orgs to disclose what they think is most appropriate
4: lack of oversight means these disclosure could be easily exaggerated
Benefits:
Within a similar outcome, effectiveness could be compared across orgs. Something we have never been able to do without folks like Givewell, so things like the program ratio (ew) are used by many donors.
What gets measured gets managed, so this will likely lead to orgs operating more effectively overall.
IRS and US charity regulators do a lousy job of preventing charitable scams and near-scams. I don’t see any likely universe in which they have bandwidth to decide on a cost-effectiveness methodology that works for 1.5 MM US charities doing a extremely wide range of activities.
Very likely it would be so loose all charities could give themselves high marks.… and if it were somehow not loose, there’s no clear reason to think the resultant methodology would be anything like EA.
After all, it’s a political decision (my guess is that it would require legislation by Congress, not just Treasury regs: medium confidence as a legal matter without wading through the tax code, high confidence that IRS would never do this w/o Congressional directive).
Fair points, and the idea is certainly a massive longshot.
It’s unclear to me exactly how the decision for the big redesign in 2007 was made, and if congress was involved at all. My guess is that a change like this could be made just by the IRS.
I am FAR from an expert on evaluation but even if orgs just reported outputs (X people helped, Y houses built, etc.), that information would be immensely useful. Guidestar is allowing orgs to disclose output metrics to get a platinum seal, but this is a voluntary disclosure. Form 990 could use a similar library of metrics for orgs to select from and make disclosure mandatory. I don’t think the form would be like EA at all—but the data would be extremely useful for donors/orgs trying to actually evaluate charity effectiveness.
At the end of the day, orgs may just completely BS the outputs so the idea certainly needs some work. Maybe encourage audits of outcomes?
I read an org’s 990 before making a non-trivial donation, and my guess from your bio is that you do too. But I wonder how many people (1) would take the time to carefully read the 990, (2) have enough methodological sophistication to see through at least moderate levels of obfuscation, but (3) are not significant enough donors that they feel (or are) empowered to call someone and ask for someone to provide relevant information.
Thanks Kyle, I like the idea of the IRS cost effectiveness thing but agree with Jason that practially it would never work. Even if it came in, it would be so loose as to be meaningless.