[Question] Looking for more ‘PlayPumps’ like examples

[calebp]

Hello, my organisation is looking for case studies of ineffective interventions (similar to PlayPumps). I have found tracking down these case studies a little tricky so far but I am sure that there are many similar programs that I am yet to stumble across. This is for an intro to EA text that we are writing for a Christian audience.

Do you have any suggestions that are ideally

• Implemented by charities

• Working in global health/​poverty alleviation

• Ended up being harmful (or at least negligible impact) Bonus points for

• Well known/​easily googleable

• Clearly well-intentioned and programme implemented successfully (lack of impact was not due to corruption or similar)

Examples so far that we have thought about

• Microcredit (not harmful, just less impact than expected by the global health community, but still a pretty good example)

• Tom’s shoes (this is the most similar to what we are looking for, thanks to Jack Lewars for suggesting this)

• Guide dogs vs cataract surgery (not really a failure of a programme just a difference in priorities)

• Scared straight (not in global health/​poverty alleviation)

If there is already a post like that please do link it below, I did have a quick look and expect that a list like that has been written somewhere already.

Thanks for your help!

[Stephen Clare]

For data on employment programs in poor countries, check out section 2 of this very good review by Blattman and Ralston. They review evaluations of job training programs, a very popular development intervention, and generally find very small or null effects:

“Training” is probably one of the most ubiquitous employment interventions. What is striking, however, is that there are very few examples of evaluated programs that have had positive effects, at least on men. It is even more difficult to find any that pass a cost-benefit test, for men or women. [p. 8]

You could probably look through some of the citations there to find specific examples of programs that didn’t have an impact, e.g.:

at least 4.5 million people in 100 countries have taken part in the ILO’s Start & Improve Your Business (SIYB) programme alone. Unfortunately there is little evidence these programs have any effect where they matter most: on sales or profits. [p. 9]

or

The fact that so many people decline opportunities to participate in these programs, or dropout after starting, is especially concerning. An interesting example comes from Pakistan’sSkills for Employability program. Even among poor households who expressed interest invocational skills, more than 95% did not enroll when given a voucher [p. 10]

I’ll just note that it’s not all bad news: some of the specific programs they review had positive effects. In particular, programs that provide beneficiaries with capital goods, either instead of or in addition to training, seem to be more positive.

In contrast to skills training programs, such capital-centric programs are relatively rare—so rare, in fact, that none appeared in a recent text analysis of all employment interventions in the World Bank. Yet the evidence from more and more program evaluations is that capital-centric programs can stimulate self-employment cost-effectively. [p. 13]

This is a big reason why Founders Pledge recommends the Graduation approach, a capital-centric job training program, in our Women’s Empowerment report!

[dspeyer]

I remember a program specifically for young women (possibly in Bangladesh, possibly linked from slatestarcodex) that specifically listed “ambition” as one of the things it wanted to foster. But participants went in wanting to be doctors and left wanting to be administrative assistants. They did not show improvement on any measured axis. Can’t seem to find the link

[calebp]

Thanks for this, I’ll be sure to take a look!

[saulius]

I don’t know much about it, but this talk mentions how sending free second-hand clothing as aid has damaged local textile industry in some countries. Quick google reveals some articles like this that should talk about it in more depth (I haven’t read them though). Also, this article came to my mind but it seems that no charity was involved so it probably doesn’t fit your purpose.

[Matt_Sharp]

Homeopaths Without Borders

[calebp]

I thought this was a joke at first but I assume this is the org, thanks for the suggestion!

[Matt_Sharp]

I thought it was a joke at first, too! Maybe they will inadvertently do some good in the world if their example helps recruit future EAs

[Pablo]

[ADDED: Please note the anonymous reply to this comment, in light of which I no longer think this study should be used as an example.]

The Make-A-Wish Foundation (though it doesn’t satisfy the ‘Working in global health/​poverty alleviation’ desideratum, and depending on your audience it may not be rhetorically the most effective example). Here’s Peter Singer (The Most Good You Can Do, pp. 5–6):

In 2013, as the Christmas giving season approached, twenty thousand people gathered in San Francisco to watch a five-year-old boy dressed as Batkid ride around the city in a Batmobile with an actor dressed as Batman by his side. The pair rescued a damsel in distress and captured the Riddler, for which they received the keys of “Gotham City” from the mayor—not an actor, he really was the mayor of San Francisco—for their role in fighting crime. The boy, Miles Scott, had been through three years of chemotherapy for leukemia, and when asked for his greatest wish, he replied, “To be Batkid.” The Make-A-Wish Foundation had made his wish come true.

Does that give you a warm glow? It gives me one, even though I know there is another side to this feel-good story. Make-A-Wish would not say how much it cost to fulfill Miles’s wish, but it did say that the average cost of making a child’s wish come true is $7,500. Effective altruists would, like anyone else, feel emotionally drawn toward making the wishes of sick children come true, but they would also know that $7,500 could, by protecting families from malaria, save the lives of at least three children and maybe many more. Saving a child’s life has to be better than fulfilling a child’s wish to be Batkid.

[anonymous_ea]

I’m not sure Make a Wish is a good example given the existence of this study. Quoting Dylan Matthews from Future Perfect on it (emphasis added):

The average wish costs $10,130 to fulfill. Given that Malaria Consortium can save the life of a child under 5 for roughly $2,000 (getting a precise figure is, of course, tough, but it’s around that), you could probably save four or five children’s lives in sub-Saharan Africa for the cost of providing a nice experience for a single child in the US. For the cost of the heartwarming Batkid stunt — $105,000 — you could save the lives of some 50-odd kids.

So that’s why I’ve been hard on Make-A-Wish in the past, and why effective altruists like Peter Singer have criticized the group as well.

But now I’m reconsidering. A new study in the journal Pediatric Research, comparing 496 patients at the Nationwide Children’s Hospital in Columbus, Ohio, who got their wishes granted to 496 “control” patients with similar ages, gender, and diseases, found that the patients who got their wishes granted went to the emergency room less, and were less likely to be readmitted to the hospital (outside of planned readmissions).

In a number of cases, this reduction in hospital admissions and emergency room visits resulted in a cost savings in excess of $10,130, the cost of the average wish. In other words, Make-A-Wish helped, and helped in a cost-effective way.

[Pablo]

Thank you for bringing this study to my attention!

My subjective impression, without having spent more than five minutes looking at the paper, is that the findings are unlikely to replicate. (If anyone disagrees, feel free to challenge me to a bet and I commit to either accept or revise my estimates.) Still, this seems enough to show that the study shouldn’t be used as an example, and I have updated my comment to note this.

[Matt_Sharp]

Agree that it seems unlikely to replicate. It would be interesting to see if e.g. hospitals are now funding Make a Wish on the grounds of it saving them future costs

[MichaelDickens]

In a number of cases, this reduction in hospital admissions and emergency room visits resulted in a cost savings in excess of $10,130, the cost of the average wish. In other words, Make-A-Wish helped, and helped in a cost-effective way.

This doesn’t follow. The $10,130 cost savings went into hospital budgets, not into buying bednets, so it doesn’t particularly matter that this money was saved.

Also, it seems implausible that Make-A-Wish could meaningfully reduce hospital admissions, so I’m inclined to disbelieve this study.

[BenSchifman]

Hi Caleb—this is probably not quite what you’re looking for, but an example I sometimes use of the importance of looking into the effectiveness of a charity before you donate is this

—People donated millions to the “Black Lives Matter Foundation” but it turns out that is a charity that isn’t associated with the Black Lives Matter movement, but rather is one random guy who plans to use the charity’s funds to “bring police officers and members of certain neighborhoods together for an annual buffet dinner and other gatherings” and “a program that would distribute bulletins featuring positive news about police for display at local businesses.” I doubt either of those have been demonstrated to be effective interventions!

See https://​​www.buzzfeednews.com/​​article/​​ryanmac/​​black-lives-matter-foundation-unrelated-blm-donations

[dspeyer]

DARE belongs in the same paragraph as Scared Straight, FWIW

[Emily Grundy]

Did you end up writing that intro to EA and would you be able to share it? I’m currently looking for a similar list of examples to use in a talk I’m giving on EA, and it would be useful to read what you ended up with.