Concerns about AMF from GiveWell reading—Part 1

I’ve now read everything on the GiveWell website about the Against Malaria Foundation, a top rated charity since 2011. This has helped me increase my understanding of the work they do and the challenges involved. This is the first in a series of posts summarising my outstanding concerns from this reading.

It may be that I’ll find the answers to some of these points by looking elsewhere, for example reading the AMF website or getting in touch with them directly. That means this is not the final word on my view of the Against Malaria Foundation. However, I’m capturing my progress at this stage so that I have a clear basis to build on for further work.

Concern #1: When you try to measure outputs (malaria case rates/​deaths) rather than inputs (bed nets distributed) for non-RCT/​”real world” distributions, there is no evidence of impact.

One of the big lessons I took from William Easterly’s work was to focus on outputs, not inputs. In general, charities and NGOs like to boast about how much effort they’ve put in but what we really care about is the impact they’ve had. A lot of the argument for bed net distributions is about inputs. The cost-effectiveness calculations are a prediction of how many lives should be saved, and how many malaria cases should be avoided, if our assumptions hold. What we really care about is how many lives are saved and how many many malaria cases are avoided, something that in principle can be measured by counting malaria deaths, or by counting malaria cases before and after our distributions.

In principle I think GiveWell and AMF would agree with this. Indeed, AMF had a plan to monitor malaria case rates before and after distributions to prove their effectiveness. However, when they actually collected the data they concluded the data was of poor quality and so abandoned this plan. From the GiveWell website:

“Malaria case rate data: Previously, AMF expected to collect data on malaria case rates from the regions in which it funded LLIN distributions…In 2016, AMF shared malaria case rate data from Malawi…but we have not prioritized analyzing it closely. AMF believes that this data is not high quality enough to reliably indicate actual trends in malaria case rates, so we do not believe that the fact that AMF collects malaria case rate data is a consideration in AMF’s favor, and do not plan to continue to track AMF’s progress in collecting malaria case rate data.”

I find this very worrying. Maybe the data was of poor quality, but that is a reason for working harder in this area rather than abandoning it altogether. In general, if we only have poor quality data about malaria in a region, doesn’t that mean we do not know how effective a bednet distribution will be? More cynically, maybe the data did not demonstrate a significant reduction in malaria and that in itself was taken as evidence that the data was low quality. If that is the case then we may be ignoring evidence that the world is more complex than we thought, something which effective altruists ignore at their peril.

Elsewhere, I have read that AMF requires its distribution partners to collect monthly malaria case rate data from all health centers in the distribution zone for 12 months preceding and 4 years following the distribution. I don’t think this requirement is actually enforced.

Whatever the reasons, it seems that the only time AMF tried to evidence their impact by collecting data they were unable to do so. This is a very bad sign. The fact that GiveWell is not concerned with this is also confusing, though not my primary issue in this review.

Taking a step back from the Against Malaria Foundation to look at the malaria problem more generally, there is mixed evidence that bed net distributions reduce malaria case rates. GiveWell has a macro review of the evidence which shows at the nation-level you cannot demonstrate any impact from all malaria control initiatives. Highlights include:

“On the whole, continent-level data do not convincingly show a relationship between the scale-up of malaria control and a fall in malaria mortality, or even a clear trend in malaria mortality.”

“In most cases, funding allocated to ITNs is significant, but many malaria control measures at once are occurring and malaria data quality is unclear (more below), so it is difficult to say much about the relative contribution of ITNs.”

“There are also 15 countries where it appears that malaria control efforts have been strong, yet there is …”Limited evidence of decrease” in malaria burden.”

“GiveWell charted ITN coverage and malaria deaths 2000-2009. Some countries look like they could be cases where a rapid scale-up in ITN coverage failed to result in a drop in malaria deaths”

“Available data and studies appear to show some cases of apparent malaria control success, and also seem to indicate that the overall burden of malaria in Africa is more likely to be falling than rising. However, in most cases it is difficult to link changes in the burden of malaria to particular malaria control measures, or to malaria control in general, and the data remains quite limited and incomplete, such that we cannot confidently say that the burden of malaria has been falling on average.”

Digging around on the GiveWell website finds more details worth highlighting. Malaria rates in Benin, DRC, Ghana, Mali & Sierra Leone increased as net coverage increased, which is more evidence that the malaria data being used is not great. In central Africa malaria was trending downwards before bednet coverage was scaled up, further muddying the waters when trying to measure impact.

GiveWell’s response to all of these points seems to boil down to “we don’t use this data as a basis for our recommendations so these issues are irrelevant”. GiveWell’s recommendation is based on the evidence from Randomised Controlled Trials that using bednets does reduce malaria cases and deaths. Maybe the RCT evidence is so convincing that the noise of country-level data doesn’t matter. However, the point remains that there are multiple attempts at evidencing impact of bednet distributions and none of these attempts are convincing. A lack of evidence of impact for real-world distributions should be a concern to any donors to this cause.