I agree with all the comments below, but just to add further clarity from my perspective, a doctor who has worked a little with malnutrition in Uganda.
This claim will be wrong. Any reasonable cost estimate for malnutrition treatment will not achieve that degree of cost effectiveness, unless perhaps if they had a new technology or product outside of the current treatment paradigm. Its just as likely the individual got it wrong as they were told to say that by UNICEF.
To explain (using basic hack estimates) why this is probably so wrong, here’s some basic math. They will likely have underestimated the cost of treatment and overestimated the effect
I’m guessing they claim to treat one child with malnutrition for $1 a day. But this would barely cover the cost of the food, let alone the cost of medical staff and medications needed. Perhaps they only include the cost of UNICEF’S contribution to the food, not the cost of medical care, tests and medications which is borne by the public health system and/or the patients’ family
I’m going to pretty wildly estimate the real cost of treating malnutrition as $4-10 daily (I’m sure givewell and others have more accurate estimates).
I’m guessing they are assuming they and they alone save every child they treat, not accounting for at least 3 factors that some children treated will die, that some would have been treated to varying extents without UNICEF intervention and that some would have survived without the treatment. I’m going to very generously assume that UNICEF’s intervention saves the lives of between 1 and 5 and 1 and 20 of the kids that they support
These factors alone could make an order of magnitude difference in their cost effectiveness estimate—which to be honest I doubt they have really done. More likely someone probably had basic flawed logic like
”We contribute $1 worth of food for each kid in this program which must save that child’s life”
UNICEF in my personal experience here in Gulu, Northern Uganda is a classic “White Landcruiser” NGO and shows the classic signs being an inefficient charity, paying a lot of office workers huge salaries while claiming to “strengthen government programmes” in ways that can’t really be measured and which I usually don’t believe. I’m actually a big fan of activism and lobbying as a potentially cost-effective method but they don’t do activism and lobbying, they just cosy alongside.
Thank you for this detailed reply! This agrees with my intuition that the numbers the volunteer told me were not at all realistic. My main goal with writing this post was not to question the effectiveness, but to bring it to the attention of the community that there is value in making this kind of information (such as what you write in this comment) available publicly, instead of the current silence on mega-charities.
I completely agree there would be enormous value in making this kind of info public. The BINGOS (Big NGOs) know that too, but they also know that if they publically estimated cost effectiveness of their interventions, it might make for pretty sombre reading as ther cost effectiveness might be orders of magnitude less than many other orgs—and I’m not just talking the creme de-la creme Givewell ones.
There’s also the problem that they would probably expose that some of their most public “Marquee” interventions might be less effecitve than some of their lesser known interventions, even within the same BINGO. For example if they found out that the small amount of money they spent deworming to prevent river blindness was far more cost-effective than their more heart-throb malnutrition intervention they are advertising, how would they respond to that?
Unfortunately it is most likelyIn the BINGOs best interests NOT to do (or at least not to publish) cost-effectiveness analysis on their interventions. It is a rational decision on their part. All they might achieve is more scrutiny on how low-impact their interventions were and perhaps even get less funding as a result
That’s why they don’t publish these kinds of analysis, not because its that hard to do—they have the staffing and expertise. Small orgs like ours and Lafiya Nigeria have had a go even without specific expertise. I’m somewhat surprised UNICEF dipped their toe into publicly sharing any kind of cost-effectiveness here, which we possibly should be giving them some credit for. At least they kind of tried?
I realise it sounds like this is dripping in cynicism, but I genuinely think this explains to a large part why the BINGOs don’t publish this kind of info and instead go full ostrich.
I really appreciate your insight here. I’ve been thinking lately about lobbying the IRS in the US to require cost effectiveness disclosures in yearly reports. There are a couple concerns I have...
1: if it is even possible to convince the IRS to add cost effectiveness disclosures to the 990
2: if orgs have the expertise/capacity to evaluate their programs
3: I think the disclosure requirement must be VERY vague to allow orgs to disclose what they think is most appropriate
4: lack of oversight means these disclosure could be easily exaggerated
Benefits:
Within a similar outcome, effectiveness could be compared across orgs. Something we have never been able to do without folks like Givewell, so things like the program ratio (ew) are used by many donors.
What gets measured gets managed, so this will likely lead to orgs operating more effectively overall.
IRS and US charity regulators do a lousy job of preventing charitable scams and near-scams. I don’t see any likely universe in which they have bandwidth to decide on a cost-effectiveness methodology that works for 1.5 MM US charities doing a extremely wide range of activities.
Very likely it would be so loose all charities could give themselves high marks.… and if it were somehow not loose, there’s no clear reason to think the resultant methodology would be anything like EA.
After all, it’s a political decision (my guess is that it would require legislation by Congress, not just Treasury regs: medium confidence as a legal matter without wading through the tax code, high confidence that IRS would never do this w/o Congressional directive).
Fair points, and the idea is certainly a massive longshot.
It’s unclear to me exactly how the decision for the big redesign in 2007 was made, and if congress was involved at all. My guess is that a change like this could be made just by the IRS.
I am FAR from an expert on evaluation but even if orgs just reported outputs (X people helped, Y houses built, etc.), that information would be immensely useful. Guidestar is allowing orgs to disclose output metrics to get a platinum seal, but this is a voluntary disclosure. Form 990 could use a similar library of metrics for orgs to select from and make disclosure mandatory. I don’t think the form would be like EA at all—but the data would be extremely useful for donors/orgs trying to actually evaluate charity effectiveness.
At the end of the day, orgs may just completely BS the outputs so the idea certainly needs some work. Maybe encourage audits of outcomes?
I read an org’s 990 before making a non-trivial donation, and my guess from your bio is that you do too. But I wonder how many people (1) would take the time to carefully read the 990, (2) have enough methodological sophistication to see through at least moderate levels of obfuscation, but (3) are not significant enough donors that they feel (or are) empowered to call someone and ask for someone to provide relevant information.
Thanks Kyle, I like the idea of the IRS cost effectiveness thing but agree with Jason that practially it would never work. Even if it came in, it would be so loose as to be meaningless.
I agree with all the comments below, but just to add further clarity from my perspective, a doctor who has worked a little with malnutrition in Uganda.
This claim will be wrong. Any reasonable cost estimate for malnutrition treatment will not achieve that degree of cost effectiveness, unless perhaps if they had a new technology or product outside of the current treatment paradigm. Its just as likely the individual got it wrong as they were told to say that by UNICEF.
To explain (using basic hack estimates) why this is probably so wrong, here’s some basic math. They will likely have underestimated the cost of treatment and overestimated the effect
I’m guessing they claim to treat one child with malnutrition for $1 a day. But this would barely cover the cost of the food, let alone the cost of medical staff and medications needed. Perhaps they only include the cost of UNICEF’S contribution to the food, not the cost of medical care, tests and medications which is borne by the public health system and/or the patients’ family
I’m going to pretty wildly estimate the real cost of treating malnutrition as $4-10 daily (I’m sure givewell and others have more accurate estimates).
I’m guessing they are assuming they and they alone save every child they treat, not accounting for at least 3 factors that some children treated will die, that some would have been treated to varying extents without UNICEF intervention and that some would have survived without the treatment. I’m going to very generously assume that UNICEF’s intervention saves the lives of between 1 and 5 and 1 and 20 of the kids that they support
These factors alone could make an order of magnitude difference in their cost effectiveness estimate—which to be honest I doubt they have really done. More likely someone probably had basic flawed logic like
”We contribute $1 worth of food for each kid in this program which must save that child’s life”
UNICEF in my personal experience here in Gulu, Northern Uganda is a classic “White Landcruiser” NGO and shows the classic signs being an inefficient charity, paying a lot of office workers huge salaries while claiming to “strengthen government programmes” in ways that can’t really be measured and which I usually don’t believe. I’m actually a big fan of activism and lobbying as a potentially cost-effective method but they don’t do activism and lobbying, they just cosy alongside.
Thank you for this detailed reply! This agrees with my intuition that the numbers the volunteer told me were not at all realistic. My main goal with writing this post was not to question the effectiveness, but to bring it to the attention of the community that there is value in making this kind of information (such as what you write in this comment) available publicly, instead of the current silence on mega-charities.
Nice one Jesper!
I completely agree there would be enormous value in making this kind of info public. The BINGOS (Big NGOs) know that too, but they also know that if they publically estimated cost effectiveness of their interventions, it might make for pretty sombre reading as ther cost effectiveness might be orders of magnitude less than many other orgs—and I’m not just talking the creme de-la creme Givewell ones.
There’s also the problem that they would probably expose that some of their most public “Marquee” interventions might be less effecitve than some of their lesser known interventions, even within the same BINGO. For example if they found out that the small amount of money they spent deworming to prevent river blindness was far more cost-effective than their more heart-throb malnutrition intervention they are advertising, how would they respond to that?
Unfortunately it is most likely In the BINGOs best interests NOT to do (or at least not to publish) cost-effectiveness analysis on their interventions. It is a rational decision on their part. All they might achieve is more scrutiny on how low-impact their interventions were and perhaps even get less funding as a result
That’s why they don’t publish these kinds of analysis, not because its that hard to do—they have the staffing and expertise. Small orgs like ours and Lafiya Nigeria have had a go even without specific expertise. I’m somewhat surprised UNICEF dipped their toe into publicly sharing any kind of cost-effectiveness here, which we possibly should be giving them some credit for. At least they kind of tried?
I realise it sounds like this is dripping in cynicism, but I genuinely think this explains to a large part why the BINGOs don’t publish this kind of info and instead go full ostrich.
Hi Nick,
I really appreciate your insight here. I’ve been thinking lately about lobbying the IRS in the US to require cost effectiveness disclosures in yearly reports. There are a couple concerns I have...
1: if it is even possible to convince the IRS to add cost effectiveness disclosures to the 990
2: if orgs have the expertise/capacity to evaluate their programs
3: I think the disclosure requirement must be VERY vague to allow orgs to disclose what they think is most appropriate
4: lack of oversight means these disclosure could be easily exaggerated
Benefits:
Within a similar outcome, effectiveness could be compared across orgs. Something we have never been able to do without folks like Givewell, so things like the program ratio (ew) are used by many donors.
What gets measured gets managed, so this will likely lead to orgs operating more effectively overall.
IRS and US charity regulators do a lousy job of preventing charitable scams and near-scams. I don’t see any likely universe in which they have bandwidth to decide on a cost-effectiveness methodology that works for 1.5 MM US charities doing a extremely wide range of activities.
Very likely it would be so loose all charities could give themselves high marks.… and if it were somehow not loose, there’s no clear reason to think the resultant methodology would be anything like EA.
After all, it’s a political decision (my guess is that it would require legislation by Congress, not just Treasury regs: medium confidence as a legal matter without wading through the tax code, high confidence that IRS would never do this w/o Congressional directive).
Fair points, and the idea is certainly a massive longshot.
It’s unclear to me exactly how the decision for the big redesign in 2007 was made, and if congress was involved at all. My guess is that a change like this could be made just by the IRS.
I am FAR from an expert on evaluation but even if orgs just reported outputs (X people helped, Y houses built, etc.), that information would be immensely useful. Guidestar is allowing orgs to disclose output metrics to get a platinum seal, but this is a voluntary disclosure. Form 990 could use a similar library of metrics for orgs to select from and make disclosure mandatory. I don’t think the form would be like EA at all—but the data would be extremely useful for donors/orgs trying to actually evaluate charity effectiveness.
At the end of the day, orgs may just completely BS the outputs so the idea certainly needs some work. Maybe encourage audits of outcomes?
I read an org’s 990 before making a non-trivial donation, and my guess from your bio is that you do too. But I wonder how many people (1) would take the time to carefully read the 990, (2) have enough methodological sophistication to see through at least moderate levels of obfuscation, but (3) are not significant enough donors that they feel (or are) empowered to call someone and ask for someone to provide relevant information.
Thanks Kyle, I like the idea of the IRS cost effectiveness thing but agree with Jason that practially it would never work. Even if it came in, it would be so loose as to be meaningless.