Concerns about AMF from GiveWell reading—Part 3
I’ve now read everything on the GiveWell website about the Against Malaria Foundation, a top rated charity since 2011. This has helped me increase my understanding of the work they do and the challenges involved. This is the third in a series of posts summarising my outstanding questions from this reading.
It may be that I’ll find the answers to some of these questions by looking elsewhere, for example reading the AMF website or getting in touch with them directly. That means this is not the final word on my view of the Against Malaria Foundation. However, I’m capturing my progress at this stage so that I have a clear basis to build on for further work.
Concern #3: Significant uncertainty in cost-per-life-saved estimates means the true cost could be $20,000 or more.
Digging into GiveWell’s 2021 v2 cost effectiveness calculation shows around 80% of the expected deaths averted are in children under 5 years old. The expectation is that protecting 1,000 under 5s for a year will save 2.72 lives on average. This is derived as follows:
| (A) Reduction in child mortality due to having bednet | 17% |
| (B) Mortality if no bednet distribution, per 1,000 child years | 11.9 |
| (C) Mortality attributed to malaria vs. 2018 meta-analysis | 180% |
| (D) Adjustment for net use, insecticide resistance, etc | 75% |
| (E) Deaths averted per 1,000 years of coverage: AxBxCxD | 2.72 |
For (A), the reduction in child mortality due to having a bednet comes straight from the 2018 Cochrane review, covering data collected between 1987 and 2001. The 95% confidence interval is 11%-23%, implying an interval of 1.76-3.67 lives saved in under 5s assuming nothing else changes. Put another way, while the central estimate by GiveWell is $7,400 per life saved, the uncertainty around this input alone suggests a confidence interval between $5,500 and $11,500.
For (B), the mortality if there is no bednet distribution is estimated based on the reduction in deaths between 2004 and 2019 and assuming 25% of the reduction was due to mass distributions of bednets. Firstly, as part (A) of the calculation says bednets cause a 17% reduction, using a higher rate of 25% here seems a little odd. Secondly, as this calculation is based on a counterfactual world that cannot be observed it is obviously quite uncertain, the true confidence interval must be wider than that suggested above. More importantly however, it is not clear to me why any adjustment based on estimated reductions in deaths is required at all.
When considering the impact of a donation to AMF, we should compare the expected mortality benefit if AMF distributes bednets compared to if they do not. According to their website, AMF did not make any significant bednet distributions before 2019, with just 1.4m nets across 2014-2016 for a population of around 75m. This means the counterfactual for AMF not making distributions in future is the same as the past, and that the current mortality rate of 7.7 per 1,000 child years is maintained. There is no reason to consider an increase to 11.9 or any other number if there are no future AMF distributions since there have been almost no past AMF distributions in this country. It may well be that child mortality has decreased in the past due to net distributions from other sources, but those distributions are not affected by any donation to AMF and so are not relevant here. Changing this assumption from 11.9 to 7.7 increases the cost-per-life-saved estimate from $7,400 to $11,400, with a confidence interval of $8,400-$17,600.
For (C), the 180% difference in malaria mortality between potential AMF distributions and the 2018 meta-analysis highlights the difference between the world where RCT effectiveness was measured and the world we live in today. Taken as a whole, the meta-study applies to a total population where the average child mortality rate was 37.8, whereas an equivalent value for the proposed distribution is a rate of 9.8. (As above, I don’t see a strong reason for not using the actual current child mortality rate of 7.7 but that’s not material here.) Similarly, the meta-study applies to a total population where the mortality rate due to malaria was 9.5 (i.e. 25% of all deaths) but the current malaria death rate is 4.4 (i.e. 45% of all deaths). If the world worked in a simple linear fashion, it would be reasonable to scale up the RCT effectiveness by 45%/25% = 180%. I don’t think there is any evidence for a linear relationship here, and it is reasonable to believe the scaling factor could be much smaller.
The central question is this: Given you know about a world where for each 1,000 children 37.8 will die a year and 9.5 of those will be due to malaria, how much do you know about a world where for each 1,000 children only 9.8 children die a year and only 4.4 of those are due to malaria? The 75% reduction in all-cause mortality suggests a wide range of health improvements. The 50% reduction in malaria mortality further suggests that the impact of further net distributions would be reduced. In my view it would be quite an achievement if the current population still saw a 17% reduction in mortality when protected by bednets. Rather than assuming the 17% impact will increase to 17% x 180% = 30% impact, I would argue a more reasonable central estimate would be a fall in effectiveness. Even if we (optimistically?) assume no change, this increases the cost-per-life-saved from $11,400 to $20,500. Given this uncertainty, the confidence interval is now wide enough to include the possibility that extra bednets won’t save any lives at all. This possibility is acknowledged by GiveWell themselves:
“Perhaps other improvements in general health are independently averting all the deaths that ITNs could avert in their absence. Under this model it’s possible ITNs don’t avert any deaths at all.”
For (D), a range of adjustments are made for other potential differences to the meta-study. Nets use may be lower since there is less follow-up in real world conditions than in the randomised controlled trials, and there is some evidence that mosquitoes have adapted to the insecticide used in nets. The 25% reduction here is mostly a matter of judgment, and given the lack of evidence it seems easy to believe the actual factor could be anywhere between 0% and 50%, reinforcing once again the uncertainty in the cost-per-life-saved estimate.
In summary, looking at the GiveWell cost-effectiveness calculation I think it is hard to justify any increases in key parameters beyond those levels we observe in the real world. Removing these uplifts reduces the expected impact by a factor of three, meaning the cost-per-life saved trebles to over $20,000. Examining the detail also shows the uncertainty within key elements of the calculation, meaning it may well be that the real cost-per-life saved is even higher than that.
| GW | Here | Bound | |
| (A) Reduction in child mortality due to having bednet | 17% | 17% | 11% |
| (B) Mortality if no bednet distribution, per 1,000 child years | 11.9 | 7.7 | 7.7 |
| (C) Mortality attributed to malaria vs. 2018 meta-analysis | 180% | 100% | 0% |
| (D) Adjustment for net use, insecticide resistance, etc | 75% | 75% | 50% |
| (E) Deaths averted per 1,000 years of coverage: AxBxCxD | 2.72 | 0.98 | 0 |
For completeness, I will point out that I wouldn’t be able to have any particular view if GiveWell were not transparent about their calculation process. I am very grateful for this transparency and the rigour GiveWell has applied in understanding the impact of each donation. Also, in their 2012 review of the Against Malaria Foundation GiveWell acknowledges the significant uncertainty in such calculations:
“Cost-effectiveness estimates such as these should not be taken literally, due to the significant uncertainty around them. We provide these estimates (a) for comparative purposes and (b) because working on them helps us ensure that we are thinking through as many of the relevant issues as possible.”
In practice I am sure many readers do take these estimates literally, but that is a point for another day.
If AMF distributed 1.4m nets across 2014-16, then that’s a lot of children with nets. Say 2.8m, if it’s 2 children per net. If nets work, then these children will be protected to some extent, and have reduced mortality from malaria. An absence of future AMF bednet distributions (and an absence of an alternative) would result in increased mortality for these children.
Now, there’s the question of whether Givewell are right to indicate mortality would increase from 7.7 to 11.9. If these are country-level figures, in a country which mostly doesn’t have bednets, then plausibly mortality for those who do have bednets is actually lower than the country-level average of 7.7. Then, if the AMF bednets are stopped, we might expect an increase in mortality back up to the country average of 7.7. However, it may be that Givewell have already adjusted for this (I haven’t looked into it), and actually the 11.9 is indeed the country-level figure that the mortality rate would be expected to increase back up to.
(a minor point—it would be helpful if you edited this to indicate you’re discussing the Democratic Republic of Congo; I initially thought you were making claims about AMF’s total distributions)
Thanks for your comments. Agree with your suggested edit—there’s now two references to the Democratic Republic of Congo. Note that for the 2021 v2 model this is AMF’s total distributions as no other countries were expected to get distributions but regardless it’s worth stating explicitly.
Net distributions cover the whole community, they are not targetted at just under-5s. Using GiveWell’s figures, 16% of the population is under 5. Scaled up 1.8 people per net this suggests coverage for 1.4m * 16% * 1.8 = 0.4m young children. That’s not going to materially change the mortality rate in a country with c.12m under 5s.
Also any protection from 2014-2016 distributions would be massively diluted by the time you measure the mortality rate in 2019, which is when the 7.7 number comes from. GiveWell estimates that DRC bednets effectively last 1.74 years on average.
I can confirm that the 7.7 mortality rate is an unadjusted country-wide mortality rate and 11.9 is the rate GiveWell estimates would occur with no distributions from AMF.
Thanks for your response—and more generally, thanks for putting time and effort into scrutinising GiveWell’s analysis, and sharing your views here.
You’re of course right. I originally wrote ‘people’ rather than ‘children’, but changed it because the discussion was focused on under 5 mortality.
Sure—but the question is whether it changes the mortality rate of those receiving the bednets.
I think you may be right, and it seems like GiveWell may have made a mistake here. But that doesn’t mean the mortality rate would be unchanged for those who receive (or would receive) bed nets. Rather, as I suggested before:
mortality for those who do have bednets might be lower than the country-wide mortality rate. e.g. if bednets reduce mortality by 17%, then we might assume the mortality rate with bednets goes from 7.7 to ~6.4.
then, if the AMF bednets are stopped, we might expect an increase in mortality back up to around the country average (which we presume indicates the mortality rate of those without bednets). So the increase would be 6.4 back to 7.7.
I agree for those that get a net it’s a good thing and mortality is reduced, and also agree mortality would increase again if no further nets appeared. However this point isn’t material given the volumes involved. 0.4m children at a rate of 6.4 and 11.6m at a rate of 7.7 is an average rate of 7.66 for the 12m children in total.
Even this small effect is diluted further when you consider the 6.4 rate only applies for 1. 74 years after the 2014-2016 distributions, so much lower when you measure mortality in 2019.
Previous AMF distributions fall into the level of rounding error in this counterfactual, which is why I’m saying an uplift from 7.7 to 11.9 is unreasonable.
I’m not sure I follow your point about volumes. The cost-effectiveness model is for those who receive the net. There’s no need to dilute the impact on these people merely because other people don’t experience the same impact. You just say ‘this is the benefit to these people, achieved at this cost’.
I think we both agree that bednets give a 17% reduction in mortality. The question is what mortality rate to apply this 17% to.
GiveWell say 11.9.
I say 7.7.
Based on your points I thought you were either saying (a) 7.7, agreeing with me, (b) an adjusted version of 7.7, which I calculate to be 7.66. Either way we’re agreeing here.