I’ve now read everything on the GiveWell website about the Against Malaria Foundation, a top rated charity since 2011. This has helped me increase my understanding of the work they do and the challenges involved. This is the third in a series of posts summarising my outstanding questions from this reading.
It may be that I’ll find the answers to some of these questions by looking elsewhere, for example reading the AMF website or getting in touch with them directly. That means this is not the final word on my view of the Against Malaria Foundation. However, I’m capturing my progress at this stage so that I have a clear basis to build on for further work.
Concern #3: Significant uncertainty in cost-per-life-saved estimates means the true cost could be $20,000 or more.
Digging into GiveWell’s 2021 v2 cost effectiveness calculation shows around 80% of the expected deaths averted are in children under 5 years old. The expectation is that protecting 1,000 under 5s for a year will save 2.72 lives on average. This is derived as follows:
(A) Reduction in child mortality due to having bednet
17%
(B) Mortality if no bednet distribution, per 1,000 child years
11.9
(C) Mortality attributed to malaria vs. 2018 meta-analysis
180%
(D) Adjustment for net use, insecticide resistance, etc
75%
(E) Deaths averted per 1,000 years of coverage: AxBxCxD
2.72
For (A), the reduction in child mortality due to having a bednet comes straight from the 2018 Cochrane review, covering data collected between 1987 and 2001. The 95% confidence interval is 11%-23%, implying an interval of 1.76-3.67 lives saved in under 5s assuming nothing else changes. Put another way, while the central estimate by GiveWell is $7,400 per life saved, the uncertainty around this input alone suggests a confidence interval between $5,500 and $11,500.
For (B), the mortality if there is no bednet distribution is estimated based on the reduction in deaths between 2004 and 2019 and assuming 25% of the reduction was due to mass distributions of bednets. Firstly, as part (A) of the calculation says bednets cause a 17% reduction, using a higher rate of 25% here seems a little odd. Secondly, as this calculation is based on a counterfactual world that cannot be observed it is obviously quite uncertain, the true confidence interval must be wider than that suggested above. More importantly however, it is not clear to me why any adjustment based on estimated reductions in deaths is required at all.
When considering the impact of a donation to AMF, we should compare the expected mortality benefit if AMF distributes bednets compared to if they do not. According to their website, AMF did not make any significant bednet distributions before 2019, with just 1.4m nets across 2014-2016 for a population of around 75m. This means the counterfactual for AMF not making distributions in future is the same as the past, and that the current mortality rate of 7.7 per 1,000 child years is maintained. There is no reason to consider an increase to 11.9 or any other number if there are no future AMF distributions since there have been almost no past AMF distributions in this country. It may well be that child mortality has decreased in the past due to net distributions from other sources, but those distributions are not affected by any donation to AMF and so are not relevant here. Changing this assumption from 11.9 to 7.7 increases the cost-per-life-saved estimate from $7,400 to $11,400, with a confidence interval of $8,400-$17,600.
For (C), the 180% difference in malaria mortality between potential AMF distributions and the 2018 meta-analysis highlights the difference between the world where RCT effectiveness was measured and the world we live in today. Taken as a whole, the meta-study applies to a total population where the average child mortality rate was 37.8, whereas an equivalent value for the proposed distribution is a rate of 9.8. (As above, I don’t see a strong reason for not using the actual current child mortality rate of 7.7 but that’s not material here.) Similarly, the meta-study applies to a total population where the mortality rate due to malaria was 9.5 (i.e. 25% of all deaths) but the current malaria death rate is 4.4 (i.e. 45% of all deaths). If the world worked in a simple linear fashion, it would be reasonable to scale up the RCT effectiveness by 45%/25% = 180%. I don’t think there is any evidence for a linear relationship here, and it is reasonable to believe the scaling factor could be much smaller.
The central question is this: Given you know about a world where for each 1,000 children 37.8 will die a year and 9.5 of those will be due to malaria, how much do you know about a world where for each 1,000 children only 9.8 children die a year and only 4.4 of those are due to malaria? The 75% reduction in all-cause mortality suggests a wide range of health improvements. The 50% reduction in malaria mortality further suggests that the impact of further net distributions would be reduced. In my view it would be quite an achievement if the current population still saw a 17% reduction in mortality when protected by bednets. Rather than assuming the 17% impact will increase to 17% x 180% = 30% impact, I would argue a more reasonable central estimate would be a fall in effectiveness. Even if we (optimistically?) assume no change, this increases the cost-per-life-saved from $11,400 to $20,500. Given this uncertainty, the confidence interval is now wide enough to include the possibility that extra bednets won’t save any lives at all. This possibility is acknowledged by GiveWell themselves:
“Perhaps other improvements in general health are independently averting all the deaths that ITNs could avert in their absence. Under this model it’s possible ITNs don’t avert any deaths at all.”
For (D), a range of adjustments are made for other potential differences to the meta-study. Nets use may be lower since there is less follow-up in real world conditions than in the randomised controlled trials, and there is some evidence that mosquitoes have adapted to the insecticide used in nets. The 25% reduction here is mostly a matter of judgment, and given the lack of evidence it seems easy to believe the actual factor could be anywhere between 0% and 50%, reinforcing once again the uncertainty in the cost-per-life-saved estimate.
In summary, looking at the GiveWell cost-effectiveness calculation I think it is hard to justify any increases in key parameters beyond those levels we observe in the real world. Removing these uplifts reduces the expected impact by a factor of three, meaning the cost-per-life saved trebles to over $20,000. Examining the detail also shows the uncertainty within key elements of the calculation, meaning it may well be that the real cost-per-life saved is even higher than that.
GW
Here
Bound
(A) Reduction in child mortality due to having bednet
17%
17%
11%
(B) Mortality if no bednet distribution, per 1,000 child years
11.9
7.7
7.7
(C) Mortality attributed to malaria vs. 2018 meta-analysis
180%
100%
0%
(D) Adjustment for net use, insecticide resistance, etc
75%
75%
50%
(E) Deaths averted per 1,000 years of coverage: AxBxCxD
2.72
0.98
0
For completeness, I will point out that I wouldn’t be able to have any particular view if GiveWell were not transparent about their calculation process. I am very grateful for this transparency and the rigour GiveWell has applied in understanding the impact of each donation. Also, in their 2012 review of the Against Malaria Foundation GiveWell acknowledges the significant uncertainty in such calculations:
“Cost-effectiveness estimates such as these should not be taken literally, due to the significant uncertainty around them. We provide these estimates (a) for comparative purposes and (b) because working on them helps us ensure that we are thinking through as many of the relevant issues as possible.”
In practice I am sure many readers do take these estimates literally, but that is a point for another day.
Concerns about AMF from GiveWell reading—Part 3
I’ve now read everything on the GiveWell website about the Against Malaria Foundation, a top rated charity since 2011. This has helped me increase my understanding of the work they do and the challenges involved. This is the third in a series of posts summarising my outstanding questions from this reading.
It may be that I’ll find the answers to some of these questions by looking elsewhere, for example reading the AMF website or getting in touch with them directly. That means this is not the final word on my view of the Against Malaria Foundation. However, I’m capturing my progress at this stage so that I have a clear basis to build on for further work.
Concern #3: Significant uncertainty in cost-per-life-saved estimates means the true cost could be $20,000 or more.
Digging into GiveWell’s 2021 v2 cost effectiveness calculation shows around 80% of the expected deaths averted are in children under 5 years old. The expectation is that protecting 1,000 under 5s for a year will save 2.72 lives on average. This is derived as follows:
For (A), the reduction in child mortality due to having a bednet comes straight from the 2018 Cochrane review, covering data collected between 1987 and 2001. The 95% confidence interval is 11%-23%, implying an interval of 1.76-3.67 lives saved in under 5s assuming nothing else changes. Put another way, while the central estimate by GiveWell is $7,400 per life saved, the uncertainty around this input alone suggests a confidence interval between $5,500 and $11,500.
For (B), the mortality if there is no bednet distribution is estimated based on the reduction in deaths between 2004 and 2019 and assuming 25% of the reduction was due to mass distributions of bednets. Firstly, as part (A) of the calculation says bednets cause a 17% reduction, using a higher rate of 25% here seems a little odd. Secondly, as this calculation is based on a counterfactual world that cannot be observed it is obviously quite uncertain, the true confidence interval must be wider than that suggested above. More importantly however, it is not clear to me why any adjustment based on estimated reductions in deaths is required at all.
When considering the impact of a donation to AMF, we should compare the expected mortality benefit if AMF distributes bednets compared to if they do not. According to their website, AMF did not make any significant bednet distributions before 2019, with just 1.4m nets across 2014-2016 for a population of around 75m. This means the counterfactual for AMF not making distributions in future is the same as the past, and that the current mortality rate of 7.7 per 1,000 child years is maintained. There is no reason to consider an increase to 11.9 or any other number if there are no future AMF distributions since there have been almost no past AMF distributions in this country. It may well be that child mortality has decreased in the past due to net distributions from other sources, but those distributions are not affected by any donation to AMF and so are not relevant here. Changing this assumption from 11.9 to 7.7 increases the cost-per-life-saved estimate from $7,400 to $11,400, with a confidence interval of $8,400-$17,600.
For (C), the 180% difference in malaria mortality between potential AMF distributions and the 2018 meta-analysis highlights the difference between the world where RCT effectiveness was measured and the world we live in today. Taken as a whole, the meta-study applies to a total population where the average child mortality rate was 37.8, whereas an equivalent value for the proposed distribution is a rate of 9.8. (As above, I don’t see a strong reason for not using the actual current child mortality rate of 7.7 but that’s not material here.) Similarly, the meta-study applies to a total population where the mortality rate due to malaria was 9.5 (i.e. 25% of all deaths) but the current malaria death rate is 4.4 (i.e. 45% of all deaths). If the world worked in a simple linear fashion, it would be reasonable to scale up the RCT effectiveness by 45%/25% = 180%. I don’t think there is any evidence for a linear relationship here, and it is reasonable to believe the scaling factor could be much smaller.
The central question is this: Given you know about a world where for each 1,000 children 37.8 will die a year and 9.5 of those will be due to malaria, how much do you know about a world where for each 1,000 children only 9.8 children die a year and only 4.4 of those are due to malaria? The 75% reduction in all-cause mortality suggests a wide range of health improvements. The 50% reduction in malaria mortality further suggests that the impact of further net distributions would be reduced. In my view it would be quite an achievement if the current population still saw a 17% reduction in mortality when protected by bednets. Rather than assuming the 17% impact will increase to 17% x 180% = 30% impact, I would argue a more reasonable central estimate would be a fall in effectiveness. Even if we (optimistically?) assume no change, this increases the cost-per-life-saved from $11,400 to $20,500. Given this uncertainty, the confidence interval is now wide enough to include the possibility that extra bednets won’t save any lives at all. This possibility is acknowledged by GiveWell themselves:
For (D), a range of adjustments are made for other potential differences to the meta-study. Nets use may be lower since there is less follow-up in real world conditions than in the randomised controlled trials, and there is some evidence that mosquitoes have adapted to the insecticide used in nets. The 25% reduction here is mostly a matter of judgment, and given the lack of evidence it seems easy to believe the actual factor could be anywhere between 0% and 50%, reinforcing once again the uncertainty in the cost-per-life-saved estimate.
In summary, looking at the GiveWell cost-effectiveness calculation I think it is hard to justify any increases in key parameters beyond those levels we observe in the real world. Removing these uplifts reduces the expected impact by a factor of three, meaning the cost-per-life saved trebles to over $20,000. Examining the detail also shows the uncertainty within key elements of the calculation, meaning it may well be that the real cost-per-life saved is even higher than that.
For completeness, I will point out that I wouldn’t be able to have any particular view if GiveWell were not transparent about their calculation process. I am very grateful for this transparency and the rigour GiveWell has applied in understanding the impact of each donation. Also, in their 2012 review of the Against Malaria Foundation GiveWell acknowledges the significant uncertainty in such calculations:
In practice I am sure many readers do take these estimates literally, but that is a point for another day.