Concerns about AMF from GiveWell reading—Part 2
I’ve now read everything on the GiveWell website about the Against Malaria Foundation, a top rated charity since 2011. This has helped me increase my understanding of the work they do and the challenges involved. This is the second in a series of posts summarising my outstanding questions from this reading.
It may be that I’ll find the answers to some of these questions by looking elsewhere, for example reading the AMF website or getting in touch with them directly. That means this is not the final word on my view of the Against Malaria Foundation. However, I’m capturing my progress at this stage so that I have a clear basis to build on for further work.
Concern #2: Future bednet distributions are expected to be less effective than other top charities and less effective than previous bednet distributions.
Virtually all donations today to the Against Malaria Foundation will go towards long lasting insecticidal nets for the Democratic Republic of Congo. GiveWell’s 2021 v2 cost effectiveness calculation for this distribution can be summarised as follows:
| (A) Notional amount spent on nets | $215,812 |
| (B) Total deaths averted—base estimate | 58 |
| (C) Total deaths averted—excluded effects | 56% |
| (D) Funging adjustment | 90% |
| (E) Cost per life saved: A/(BxCxD) | $7,426 |
The first thing to note is this $7,400 cost-per-life-saved is outside the range typically quoted by GiveWell for the Against Malaria Foundation. GiveWell’s Top Charities page (https://www.givewell.org/charities/top-charities) states an estimate of $3,000-$5,000 to save a life by funding bednets. Based on that range, someone donating $100,000 may expect to save 20-33 lives with their donation but GiveWell’s own best estimate is that this donation would only save 13 lives. This is a significant difference and it is unclear why the Top Charities page maintains this inconsistency.
The same $3,000-$5,000 range is used for the other top charities where impact is measured in terms of cost per life saved. Taken at face value, this means the proposed bednet donations are expected to have less impact than similar top charities. In particular, funding malaria medicine via the Malaria Consortium would be nearly twice as effective, since the estimated cost-per-life-saved is only $4,100 for this intervention. While GiveWell says charity rankings do not depend on cost effectiveness estimates alone, this does suggest that the Against Malaria Foundation is currently less effective than other top charities.
The key difference for the latest net distributions is that they will not be used to increase net coverage. Rather they will be used to reduce the amount of time between net distributions. This is why the excluded effects adjustment (C) is so large. The base estimate for total deaths averted (B) measures the impact of providing nets to people that currently have no nets. However the Democratic Republic of Congo distributions will target people that have old nets. As the old nets still provide some protection, the impact of this distribution is reduced. GiveWell estimate this approach to be 41% less effective than giving nets to those who have none. Consequently, without this change in approach and everything else being equal, the excluded effects adjustment would be 96% and the cost per life saved would fall to $3,900. (Technically everything else is not equal, in particular the funging adjustment (D) would also update, but this remains an indicative measure of the importance of this assumption.)
GiveWell estimates that treated bednets have a 17% shorter life in the Democratic Republic of Congo than elsewhere. This helps justify the campaign to reduce the time between distributions, particular when some provinces already wait longer between distributions than the usual 3 year target. Even so, this implies that the best distributions donors can fund are in areas where (a) people already have some bednets, and (b) the new bednets will last less time than if they were distributed elsewhere.
Finally, the cost effectiveness calculations for other regions with recent distributions all have a lower cost-per-life-saved. Guinea is $4,300, Uganda is $5,700 and Togo is $6,600. Another upcoming distribution in Nigeria also has a lower cost per life saved of only $4,100. Taken together, this evidence reinforces that the Democratic Republic of Congo distributions are expected to be less effective than other recent distributions, and likely less effective than other distributions made in the past.
I spent time looking for this table below including directly Googling some of the numbers, which typically turn up results, but this didn’t work and I couldn’t find this information:
I think it is very important that you provide sources for all figures discussion (which is as easy as copying and pasting the hyperlink) because the amount of “surface area” to draw content from is large and the context, age, purpose of sources matters.
The numbers seem to come from the latest GiveWell’s Cost-Effectiveness model published here:
https://www.givewell.org/how-we-work/our-criteria/cost-effectiveness/cost-effectiveness-models
Direct link: https://docs.google.com/spreadsheets/d/11HsJLpq0Suf3SK_PmzzWpK1tr_BTd364j0l3xVvSCQw/edit#gid=1364064522
Lorenzo, thanks for diving in here to help out. I can confirm this is indeed the source I used.
Charles, I didn’t expect anybody to bother checking my source values. I described them as “GiveWell’s 2021 cost effectiveness calculation” which I think satisfies your criteria on context, age & purpose of source. Googling that gives me Lorenzo’s first link above.
Inspired by your comment I’ve now figured out how to add in line links for sources so that will perhaps help address this type of comment in future.
Hi,
By the way, sorry for the lack of response to your thoughtful reply in your previous post. I think the perspective in your comment is valid but I can’t easily write a complete response that would be fully useful. (I think your perspectives in your post and comments are right, and AMF/GiveWell and EA is right at the same time, and explaining this seems hard.)
Edit: The content below is wrong, I was confused and thought the OP was literally discussing a ~$216k program. AMF aims to spend tens of millions of dollars annually for distribution to the Congo (DRC).
Personally, as an outsider doing desk analysis, I think I risk ending up only superficially understanding the work involved delivering effective aid to sub-Saharan Africa.I would be modest when making simple comparisons, for example, I would make considerations for logistics and execution:It’s not like we are regularly given container ships full of nets and each month we decide which country we sail into and unload net deliveries. For each net distribution program, it takes years to plan, build partnerships and setup logistics. Maybe calculations can change by 50% in some cases.Another consideration is that, based on your numbers, it’s about $216,000 spent on the Congo program. That seems much smaller than the others. Maybe they scaled it back for the very reasons you mentioned, because it’s less cost effective. Note that even if you want to pull back, you may want to keep a small operation to keep up relationships, infrastructure, permits, politics, and other knowledge.Another reason is that maybe AMF spent years and hundreds of thousands investigating the Congo, which is exactly what you should do when deploying tens of millions of dollars. To do this research, maybe it was necessary for GiveWell or AMF to promise some program at the end of it, and this $216K program was the smallest viable such program.Charles
No need to apologise for a lack of response. Thanks for sharing your high-level view here even with the limitation that explaining that view is hard. It’s much better for me to have that comment than none at all, and I know the feeling of struggling to explain a view.
For info, I agree with the thrust of what you were saying about real-world constraints on distributions, though agree this is not what the post was actually about.
Hi JPHoughton,
“Virtually all donations today to the Against Malaria Foundation will go towards long lasting insecticidal nets for the Democratic Republic of Congo.”
As far as I can tell from GiveWell’s current cost-effectiveness analysis, only 19% of donations will go to DRC? https://docs.google.com/spreadsheets/d/1B1fODKVbnGP4fejsZCVNvBm5zvI1jC7DhkaJpFk6zfo/edit#gid=1364064522 (row 19)
Taking a weighted average of each country’s cost-effectiveness by the % of donations, I get a cost-effectiveness of $5,636. https://docs.google.com/spreadsheets/d/1GdCKfSNJkp8aGDxjKlnuRook_LgkKJOJvQvzzZ5KqTw/edit#gid=1364064522&range=B223 (cell B223)
This is, of course, still outside the $3000-$5000 range (although not far off), so it would be good to hear from GiveWell whether they expect future donations opportunities to drop back into that range.
Note also that this cost per life figure does not include the development effects of preventing malaria, which are potentially quite material (these are included in the “multiples of cash transfers” calculations).
Cheers,
Lucas
I get “Access Denied” trying to access your spreadsheets.
In any case the most recent released model is here: https://docs.google.com/spreadsheets/d/11HsJLpq0Suf3SK_PmzzWpK1tr_BTd364j0l3xVvSCQw/edit#gid=1364064522 .
The “Percentage of funding to be allocated to each country with marginal donations”, described as “percentage of funding we expect the charity would allocate to each country if it were to receive additional funding.” seems to be 100% to DRC.
I assume that’s what JPHoughton is referring to, I think it might be helpful to clarify it in the main post
Sorry, access provided now! I believe there is a newer version (version 3) of GiveWell’s spreadsheet: https://docs.google.com/spreadsheets/d/1B1fODKVbnGP4fejsZCVNvBm5zvI1jC7DhkaJpFk6zfo/edit#gid=1364064522
Thanks Lucas. You are right that there is a version 3 of this spreadsheet and my post is based on version 2. I originally took a copy of the spreadsheet in late August and v3 appeared in September.
The only difference I can see between the two is the allocation of funding across countries (row 7), with the rest of the values agreeing once this is adjusted. Still, this is a vital difference for my essay, and agree v3 gets the average cost-per-life-saved pretty close to the stated range.
GiveWell don’t justify this change in their changelog and it seems a little finger-in-the-air to me. Some quick research shows AMF’s $108m of recently committed funding almost exclusively went to DRC. That said, it’s not clear where the next $80m will go as AMF do not publicly identify countries involved until an Agreement is signed.
Loreno, thanks again for your efforts here. I agree with what you said, and have tweaked the post to say “2021 v2” for clarity.
Hi JPHoughton,
After looking at this a bit more closely, it appears that the % of funding to each country (rows 7,19) is actually purely arbitary GiveWell’s most recent cost-effectiveness analysis (CEA). Hence, the 19% figure I quoted above is not meaningful. Apologies for my misleading comment.
I suspect that this new approach of using arbitrary percentages reflects the complex question of “room for more funding” outlined in GiveWell’s recent blog post. Nonetheless, my understanding is that the funding GiveWell actually allocated to AMF in 2020 was well within the $5000 cost per life saved range.
Note also that DRC’s program remains 12.7x cash in the most recent CEA (once development effects are included).
Cheers,
Lucas