Summary

The Against Malaria Foundation (AMF) or Malaria Consortium (MC) both work to prevent malaria, often in the same countries. Mismodelling how malaria bednet use and chemoprevention overlap in a population or misattributing credit for lives saved when both AMF and MC are targeting the same population can lead to either overestimating or underestimating the number of lives saved by each charity, and therefore their cost-effectiveness. I illustrate how. My recommendations are:

Be explicit about how the intervention will apply across a target population, and consider other possibilities. The intervention might not apply statistically independently of individuals’ risks or how much each individual would benefit from your intervention.
When estimating the impacts of multiple organizations working towards a common or mutually dependent outcome that you and those you funge with are funding or otherwise supporting, keep in mind the possibility of double-counting or undercounting impact. Credited impacts, including for cost-effectiveness estimates, should sum to 100%. You could use Shapley values explicitly, but even if you don’t, thinking about them can be helpful.

To be clear, I just happened to be thinking of malaria when I came up with these. These types of problems certainly aren’t unique to malaria or global health and development, and I don’t think they’re uncommon in impact and cost-effectiveness estimation in EA. However, they could plausibly make the most difference for global health and development, because of very similar estimated cost-effectiveness across GiveWell’s top charities, currently each $4,000 to $5,500 per life saved.

Background

The Against Malaria Foundation (AMF) and Malaria Consortium (MC) both work on preventing malaria and mortality by malaria. Both are GiveWell top charities, AMF for its long-lasting insecticide-treated net (LLIN or bednet) program, and MC for its seasonal malaria chemoprevention (SMC or medicine) program. When I refer to AMF and MC, I mean these respective programs.

GiveWell assumes a 79% relative reduction in risk of death for treated children from malaria through SMC, through a 79% reduction in malaria cases, and a reduction in malaria deaths proportional to the reduction in incidence. This is effectively both relative to bednet use and relative to no bednet use. Similarly, GiveWell assumes a 39% to 56% (24% in Chad) reduction in malaria mortality for children under 5 from bednet use, again effectively both relative to SMC treatment and no SMC treatment. (GiveWell also accounts for reduction in malaria in the untreated.)

If someone has already received either a bednet (and uses it) or antimalarial medicine, then they are at much lower risk of contracting malaria and dying from malaria, so the extra impact of giving them the other should be lower than if they didn’t already receive the first (assuming they would actually use the bednet). To account for this, we can use malaria incidence and mortality estimates that already reflect bednet use or SMC treatment (other than what extra the charity under consideration will do), or adjust our estimates for them.

Then, ignoring effects on the untreated for simplicity, if malaria incidence and mortality rate estimates already reflect expected bednet use (but not SMC, for simplicity) in the target population, then the most straightforward way of estimating the impact of SMC for MC in the population is by multiplying by the 79% reduction from SMC and by the share of the target population who would receive SMC. Similarly, if malaria incidence and mortality rate estimates already reflect expected SMC treatment (but not bednet use, for simplicity) in the target population, then the most straightforward way of estimating the impact of delivering bednets for AMF in the population is by multiplying by the % reduction from bednet use and by the share of the target population who would use bednets. This is, as far as I can tell, equivalent to how GiveWell estimates the impact on malaria incidence and mortality of AMF and MC, aside from their further adjustments for counterfactual bednet use for AMF and counterfactual SMC treatment for MC, as well as effects on the untreated (and other non-mortality effects).

However, those estimates and their underlying assumptions can be wrong, and this can lead to double-counting and overestimating some lives saved or underestimating lives saved. There are two main ways I can imagine misestimating:

Mismodelling the overlap between bednet use and SMC treatment among a target population.
Attributing credit for a life saved to AMF and MC in a way that doesn’t add up to 100%.

Each can lead to either overestimating or underestimating lives saved. I illustrate these possibilities below.

Mismodelling the overlap

If you mismodel the overlap in populations getting SMC and those using bednets even if you get the totals/shares for each right, you can double-count or undercount. For example, if you assume children get bednets and SMC independently of each other, as I think GiveWell has done, but children who get one are disproportionately likely to get the other (maybe their parents are more motivated to prevent malaria, or face fewer barriers in getting both), then you’ll double-count some potential averted deaths. On the other hand, if children getting bednets are less likely to get SMC than children not getting bednets (or vice versa), in case the parents or distributors believe one is enough, then the assumption of independence will cause you to miss some potential averted deaths. If receiving either a bednet or SMC makes someone less likely to get the other, this would tend to displace the other to someone who was less likely to be protected by either, which would decrease overlap and increase reduction in deaths. On the other hand, they may be delivered jointly within a population, as MC does deliver both SMC and bednets, and if they do so together, they would plausibly increase overlap and decrease the reduction in deaths.^[1]

To illustrate more concretely with mostly made-up numbers, suppose SMC reduces the average recipient’s risk by 80% (retaining 20% of the risk), both with and without bednets, and bednets reduce the average recipient’s risk by 30% (retaining 70% of the risk), both with and without SMC, and we ignore protection for non-recipients. If half of the (target) population gets bednets and half gets SMC, then this is compatible with, comparing to neither bednets nor SMC:

49% overall reduction in risk, with half of the population getting bednets and half getting SMC, independently of one another, so we have 25% unprotected, 25% doubly protected, 25% only protected by bednets and 25% only protected by SMC.^[2]
55% overall reduction in risk, with bednets for one half, and SMC for the other half, completely disjoint subpopulations.^[3]
47.3% overall reduction in risk, with bednets and SMC for one half of the population, and the other half completely unprotected.^[4]

49% is very close to both 55% and 47.3%, so it seems like it shouldn’t make much difference how we assume bednets and SMC are distributed throughout the population, at least for this hypothetical.

However, if SMC and bednets are being distributed to the same place, we’re also interested in the marginal impact compared to just one of them. Just one half of the population covered by SMC and no one covered by bednets would be a 40% reduction in risk compared to no protection.^[5] Just one half of the population covered by bednets and no one covered by SMC would be a 15% reduction in risk compared to no protection.^[6]

If mortality is proportional to incidence, as GiveWell assumes, then the additional percentage point (ppt) reduction in mortality from both bednets and SMC compared to one would be between, by taking differences:

7.3 ppts to 15 ppts, for bednets and SMC compared to just SMC, with 9 ppts for the independence assumption, and
32.3 ppts to 40 ppts, for bednets and SMC compared to just bednets, with 35 ppts for the independence assumption.

If 1000 children would have died with one of bednets or SMC covering half of the population and no one with both, then, compared to just SMC, using both together would save another 73 to 150 children, or 90 with the independence assumption, or compared to just bednets, another 323 to 400 children, or 350 with the independence assumption.

Then, in this hypothetical, the true value for AMF could be up to 67% higher or 19% lower than with the independence assumption. On the other hand, the true value for SMC would only be up to 14% higher or 8% lower.

Note that this is just an illustration, and ignores protection for those not directly treated or using bednets. The actual values could go even further in either direction, or be more constrained near the independence assumption.

A further possibility is that MC causes people to be more or less likely to use a bednet (as often or properly) than otherwise, without displacing that bednet for use to others. They get a bednet, but use it more or less than otherwise. Then, we would underestimate or overestimate bednet use in the counterfactual of MC’s intervention relative to MC not delivering SMC, and so underestimate or overestimate MC’s impact. Either seems plausible. MC could further promote or instruct on bednet use to people along with SMC delivery, potentially making bednet use more likely. Or, people receiving SMC may think SMC already offers relatively good protection, licensing them to use their bednets less (or parents to protect their children with bednets less). In this case, MC should be seen as having a causal effect on bednet use independently of the number of distributed bednets.

Credit assignment

AMF and MC work in some of the same countries, like Togo and Nigeria, and some of the same regions in these countries (AMF sheet, MC sheet). I don’t know whether or not or to what extent AMF and MC work in the same cities, towns or villages, though, or otherwise target some of the same people. MC apparently takes bednet coverage into account in deciding where to work (AMF might take SMC work into account, too, but I didn’t check), and GiveWell makes adjustments in its cost-effectiveness estimates for AMF based on SMC coverage, and MC for otherwise underestimated net coverage in Nigeria. However, it’s not clear to me to what extent this data reflects the immediate plans of the other charity or their most recent distributions, nor what assumptions GiveWell is making in its cost-effectiveness models.

Both charities may start making plans to work in the same place, because of high malaria burden and low coverage, but once the first of the two charities distributes, the second’s estimates of coverage will be too low and malaria incidence too high, if they aren’t updated. Or, they may both assume the other will distribute in the region, and only claim the difference conditional on the other, but then the first’s estimates of coverage will be too high and malaria incidence too low. I illustrate both possibilities.

Let’s start with an oversimplified illustration. Suppose a specific child definitely would have died without protection from either SMC or a bednet, and definitely wouldn’t have died with protection from either. Suppose AMF provides that child a bednet and MC provides that child SMC. Each charity might claim to have saved the child, ignoring that the other would have in their absence, double-counting the life saved. Or, they might both recognize that the other would have saved the child and each refuse to claim credit (neither was counterfactual!), so the life saved doesn’t get counted at all. Both approaches are wrong!

Let’s make this a bit more realistic, but with some made-up numbers. Suppose that each of a bednet and SMC independently reduce a protected child’s risk of death by 75%, whether or not the other is used. If a child uses both, their risk will be reduced to 6.25%=25%*25% compared to using neither, or a 93.75% reduction in risk. The reduction in risks attributed to AMF and MC together should add up to 93.75% for the sake of estimating their cost-effectiveness together, but they may not:

If AMF assumes that child would not have received SMC (or we assume so on their behalf), they would claim a reduction in risk of 75% compared to no coverage and multiply by the no coverage risk. MC, then, to add up to 93.75%, can only claim the 18.75%=25%-6.25% difference in risk compared to neither, or a 75% reduction in risk compared to a bednet. Similarly if we swap the roles of AMF and MC. If they each claim 75% compared to no coverage or we do so on their behalves, this would sum to a 150% reduction in risk, which is too high (and impossible).
If AMF assumes that the child would have received SMC (or we assume so on their behalf), they would claim the difference in risk of 18.75%=25%-6.25%, and impact proportional to that by multiplying by the probability of death with neither. Or, equivalently, AMF could claim a 75% reduction in risk compared to the remaining risk with SMC, and multiply the latter by 75%. Then, to add up to 93.75%, MC can’t only claim the difference over bednets — also 18.75% — MC must claim the full initial 75% reduction compared to no coverage (neither bednet nor SMC). Similarly if we swap the roles of AMF and MC. If they each claim 18.75% compared to no coverage or we do so on their behalves, this would sum to 37.5% reduction in risk, which is too low.

This means that the two obvious ways of estimating impact relative to no coverage can’t be applied symmetrically for both AMF and MC. One way will overestimate if used for both, summing to a 150% reduction in risk, and the other will underestimate, summing to a 37.5% reduction in risk. The overestimate is 4x greater than the underestimate, and their average is exactly 93.75%, the correct number.^[7]

If we prefer only one of the two charities to work in a given place, it would be the first to announce its plan to work there that we should treat as first, as long as it follows through soon enough. When we prefer both to work in a given place, the order plausibly doesn’t matter. When the order does matter, that may incentivize each charity to get there first or make the first commitment to get there, because they could claim greater marginal impact than otherwise. To ignore the order, for each charity, we can just average over both orders, giving the Shapley value. For discussion of Shapley values on the Effective Altruism Forum, see Sempere, 2019.

I was informed by GiveWell staff that they do attempt to account for the near-term plans of other malaria-targeting charities for each of AMF and MC, including AMF for MC and MC for AMF, and if they do so like I illustrated above for both, holding MC’s plans constant for AMF and AMF’s plans constant for MC, then this would cause a downward bias in their cost-effectiveness estimates. So, if the estimates are otherwise unbiased (i.e. except for this credit assignment issue), then AMF and MC would actually be more cost-effective than GiveWell’s estimates suggest.

Acknowledgements

This article was inspired by discussion with VictorW in How do inspired contributions and externalized costs factor in cost-effectiveness calculations? I’d also like to thank GiveWell staff for clarification on their models.

^
That being said, then we need to consider whether or not MC’s bednet distributions are already included in bednet coverage estimates for where MC works.
^
0.2*0.7*0.25 + 0.2*0.25 + 0.7*0.25 + 0.25 = 0.51, and 1-0.51 = 0.49
^
0.2*0.5 + 0.7*0.5 = 0.45, and 1-0.45 = 0.55
^
0.2*0.27*0.5 + 0.5 = 0.527, and 1-0.527 = 0.473
^
0.2*0.5 + 0.5 = 0.6, and 1-0.6 = 0.4
^
0.7*0.5 + 0.5 = 0.85, and. 1-0.85 = 0.15
^
The average of the two will be correct in general, because summing the high estimate for AMF, $H_{A M F}$ , with the low estimate for MC, $L_{M C}$ , gives the correct total reduction, $R$ , as does summing the low estimate for AMF, $L_{A M F}$ , with the high estimate for MC, $H_{M C}$ . Then,
$(H_{A M F} + H_{M C}) / 2 + (L_{A M F} + L_{M C}) / 2 = (H_{A M F} + L_{M C}) / 2 + (L_{A M F} + H_{M C}) / 2$
$= R / 2 + R / 2 = R$

Malaria bednet and medicine overlap

Summary

Background

Mismodelling the overlap

Credit assignment

Acknowledgements