Hey Nick, thanks for raising this question about the plausibility of chlorination’s effect of mortality and the need for more research to understand why. I’m a senior researcher at GiveWell and wanted to chime in with a little more context.
When we did our analysis, we agreed that the ~25%-30% headline figure from the Kremer et al. meta-analysis felt implausibly high. We end up with estimates of ~5%-15% depending on the country and program (e.g., ~6% for the Dispensers for Safe Water Program in Uganda and ~12% for the in-line chlorination program in Malawi).
A bit more on what we did:
We were reluctant to take the Kremer et al. results at face value for the reasons you listed — the reduction in mortality is higher than we’d expect, even if we make pretty generous assumptions on the Mills-Reinke effect, and higher than experts we spoke to would’ve expected, too.
Instead, we first did our own meta-analysis. We focus only on RCTs that study chlorination (as opposed to other ways to increase water quality) and have follow-up lengths of at least a year. We also exclude one RCT with an implausibly high effect. Based on this, we estimate an effect of ~12%.
We adjust this downward to account for some of the RCTs including additional interventions beyond chlorination like hygiene programs (which could overstate the effect of chlorination alone). Then we adjust for differences across countries in the share of deaths we think are attributable to chlorination, compared to the RCTs (e.g., in countries where enteric infection is a smaller share of deaths than in trials, we estimate a smaller effect) and differences in amount of chlorination provided by programs, compared to the RCTs (e.g., we think in-line chlorination provides more chlorination than chlorination programs studied in trials and so has a larger effect).
We also compare our best guess to a “plausibility cap” — an upper bound on what we think the effect of chlorination on mortality could be. This is (I think) the 17% figure the post mentions. In countries where we considered funding in-line chlorination, for example, we guess that if chlorination reduces diarrhea by 25% and infectious diseases account for 70% of mortality in under-5s in these countries, a plausible maximum reduction of mortality is ~17% (25% x 70%). This plausibility cap requires some really uncertain assumptions (e.g., what share of deaths could plausibly be affected by chlorination and by how much?), and we don’t have a lot of confidence in it. We explain more of our rationale for this cap here. In countries we’ve looked at, though, the plausibility cap tends to exceed our initial best guess so it doesn’t end up making a difference in our bottom line estimates.
These mortality effect estimates are really uncertain, though, so we’ve funded follow-up research (as Dan alluded to in his comment).
Our best guess is still much larger than we’d expect based on the effect of chlorination on diarrhea alone (implying chlorination averts ~3 deaths from non-diarrhea causes for each death averted due to diarrhea) and relies on a lot of judgment calls.
Because of that, we recently made a grant of $1.8 to Michael Kremer and colleagues at the Development Innovation Lab at the University of Chicago to launch an additional RCT in Kenya and scope larger trials in Nigeria and India.
More on why we made the grant is on our grant page.
Thanks again for boosting this question — it’s something we’ve been thinking about a lot, and I’m glad it’s getting some more attention. I think we’d be open to hearing more thoughts about how we could learn more about the extent to which chlorination affects mortality and why, since we’re continuing to explore more grants to chlorination.
I didn’t explicitly say it, but I think GiveWell did a fantastic job of re-analysing and adjusting here, your range and final estimate for estimating the mortality of the intervention are pretty similar to what my intuition would have moved to, and I agree with your methods of getting there—especially doing your own meta-analysis only using the RCTs.
A couple of other comments
1. I agree with you on excluding the Haushofer/Kremer study, but not for the reasons you state. I dont’ really understand why exclude a study just because “we believe the effect size it reports is implausibly large, and it has a substantial impact on the pooled estimate” . This seems unnecessarily subjective to meI’m not sure why 30% might be plausible but 60%ish is not? I know 60% does seem subjectively implausible but I’m not sure that’s enough of a reason to exclude a study. If a study is methodologically sound, then why not include it.
BUT I think its reasonable to exclude the study because it was done retrospectively, so was not an RCT at all—You say on your website that is is an RCT, but it is not. They retrospectively gather data on mortality post-hoc well after the original RCT was done. The study itself doesn’t claim to be an RCT. As you only want to consider RCTs, this would exclude it automatically as it is not one. Its a small thing but maybe you could consider correcting this on the website?
2. (This is very minor) I like your plausibility “cap”, but feel like it might be a little low given that 5 meta-analysed RCTs did show a 25-30% mortality reduction. Would it not perhaps be more logical to use the research figure as the cap? Its hard to “reason” our way to plausible percentages here, precisely because we have little idea why the mortality reduction is happening.
The question I’m very interested in of course is why, and I really hope some of the grants you are making will go towards , rather than purely investigating the magnitude of mortality question—which is obviously still the primary purpose of the studies. The most important way to make progress on the answer might be to ascertain the causes of death in those who die in the treatment vs. control groups, so I hope they are planning to collect that data as best they can at the very least. Your study page doesn’t mention that any of the RCT work is geared towards the why—but it’s not too late ;).
Again amazing job on analysing this, and funding more research on the topic. I don’t think it would have happened without you!
Hey Nick, thanks for raising this question about the plausibility of chlorination’s effect of mortality and the need for more research to understand why. I’m a senior researcher at GiveWell and wanted to chime in with a little more context.
When we did our analysis, we agreed that the ~25%-30% headline figure from the Kremer et al. meta-analysis felt implausibly high. We end up with estimates of ~5%-15% depending on the country and program (e.g., ~6% for the Dispensers for Safe Water Program in Uganda and ~12% for the in-line chlorination program in Malawi).
A bit more on what we did:
We were reluctant to take the Kremer et al. results at face value for the reasons you listed — the reduction in mortality is higher than we’d expect, even if we make pretty generous assumptions on the Mills-Reinke effect, and higher than experts we spoke to would’ve expected, too.
Instead, we first did our own meta-analysis. We focus only on RCTs that study chlorination (as opposed to other ways to increase water quality) and have follow-up lengths of at least a year. We also exclude one RCT with an implausibly high effect. Based on this, we estimate an effect of ~12%.
We adjust this downward to account for some of the RCTs including additional interventions beyond chlorination like hygiene programs (which could overstate the effect of chlorination alone). Then we adjust for differences across countries in the share of deaths we think are attributable to chlorination, compared to the RCTs (e.g., in countries where enteric infection is a smaller share of deaths than in trials, we estimate a smaller effect) and differences in amount of chlorination provided by programs, compared to the RCTs (e.g., we think in-line chlorination provides more chlorination than chlorination programs studied in trials and so has a larger effect).
We also compare our best guess to a “plausibility cap” — an upper bound on what we think the effect of chlorination on mortality could be. This is (I think) the 17% figure the post mentions. In countries where we considered funding in-line chlorination, for example, we guess that if chlorination reduces diarrhea by 25% and infectious diseases account for 70% of mortality in under-5s in these countries, a plausible maximum reduction of mortality is ~17% (25% x 70%). This plausibility cap requires some really uncertain assumptions (e.g., what share of deaths could plausibly be affected by chlorination and by how much?), and we don’t have a lot of confidence in it. We explain more of our rationale for this cap here. In countries we’ve looked at, though, the plausibility cap tends to exceed our initial best guess so it doesn’t end up making a difference in our bottom line estimates.
There’s more detail in our intervention report on water quality and this blog post on why we changed our mind on chlorination programs.
These mortality effect estimates are really uncertain, though, so we’ve funded follow-up research (as Dan alluded to in his comment).
Our best guess is still much larger than we’d expect based on the effect of chlorination on diarrhea alone (implying chlorination averts ~3 deaths from non-diarrhea causes for each death averted due to diarrhea) and relies on a lot of judgment calls.
Because of that, we recently made a grant of $1.8 to Michael Kremer and colleagues at the Development Innovation Lab at the University of Chicago to launch an additional RCT in Kenya and scope larger trials in Nigeria and India.
More on why we made the grant is on our grant page.
Thanks again for boosting this question — it’s something we’ve been thinking about a lot, and I’m glad it’s getting some more attention. I think we’d be open to hearing more thoughts about how we could learn more about the extent to which chlorination affects mortality and why, since we’re continuing to explore more grants to chlorination.
Thanks so much for engaging Alex.
I didn’t explicitly say it, but I think GiveWell did a fantastic job of re-analysing and adjusting here, your range and final estimate for estimating the mortality of the intervention are pretty similar to what my intuition would have moved to, and I agree with your methods of getting there—especially doing your own meta-analysis only using the RCTs.
A couple of other comments
1. I agree with you on excluding the Haushofer/Kremer study, but not for the reasons you state. I dont’ really understand why exclude a study just because “we believe the effect size it reports is implausibly large, and it has a substantial impact on the pooled estimate” . This seems unnecessarily subjective to me I’m not sure why 30% might be plausible but 60%ish is not? I know 60% does seem subjectively implausible but I’m not sure that’s enough of a reason to exclude a study. If a study is methodologically sound, then why not include it.
BUT I think its reasonable to exclude the study because it was done retrospectively, so was not an RCT at all—You say on your website that is is an RCT, but it is not. They retrospectively gather data on mortality post-hoc well after the original RCT was done. The study itself doesn’t claim to be an RCT. As you only want to consider RCTs, this would exclude it automatically as it is not one. Its a small thing but maybe you could consider correcting this on the website?
2. (This is very minor) I like your plausibility “cap”, but feel like it might be a little low given that 5 meta-analysed RCTs did show a 25-30% mortality reduction. Would it not perhaps be more logical to use the research figure as the cap? Its hard to “reason” our way to plausible percentages here, precisely because we have little idea why the mortality reduction is happening.
The question I’m very interested in of course is why, and I really hope some of the grants you are making will go towards , rather than purely investigating the magnitude of mortality question—which is obviously still the primary purpose of the studies. The most important way to make progress on the answer might be to ascertain the causes of death in those who die in the treatment vs. control groups, so I hope they are planning to collect that data as best they can at the very least. Your study page doesn’t mention that any of the RCT work is geared towards the why—but it’s not too late ;).
Again amazing job on analysing this, and funding more research on the topic. I don’t think it would have happened without you!