I believe the paper you’re referring to is “Water Treatment And Child Mortality: A Meta-Analysis And Cost-effectiveness Analysis” by Kremer, Luby, Maertens, Tan, & Więcek (2023).
The abstract of this version of the paper (which I found online) says:
We estimated a mean cross-study reduction in the odds of all-cause under-5 mortality of about 30% (Peto odds ratio, OR, 0.72; 95% CI 0.55 to 0.92; Bayes OR 0.70; 95% CrI 0.49 to 0.93). The results were qualitatively similar under alternative modeling and data inclusion choices. Taking into account heterogeneity across studies, the expected reduction in a new implementation is 25%.
That’s a point estimate of a 25-30% reduction in mortality (across 3 methods of estimating that number), with a confidence/credible interval that has a lower bound of a 7-8% reduction in mortality. So, it’s a fairly noisy estimate, due to some combination of the noisiness of individual studies and the heterogeneity across different studies.
That interval for the reduction in mortality just barely overlaps with your number that “Sub-saharan Africa diarrhoea causes 5-10% of child mortality.” (The overlap might be larger if that rate was higher than 5-10% in the years & locations where the studies were conducted.)
So it could be that the clean water interventions prevent most children’s deaths from diarrhoea and few other deaths, if the mortality reduction is near the bottom of the range that Kremer & colleagues estimate. Or they might prevent a decent chunk of other deaths, but not nearly as many as your Part 4 chart & list suggest, if the true mortality reduction is something like 15%.
There is generally also a possibility of a meta-analysis giving inflated results, due to factors like publication bias affecting which studies they include or other methodological issues in the original studies, which could mean that the true effect is smaller than the lower bound of their interval. I don’t know how likely that is in this case.
Here’s a more detailed look at their meta-analysis results:
Does it make sense to pool the effect of chlorine interventions with filtration interventions, when these are two different types of interventions? I don’t think it does and notably the Cochrane review on this topic that looks at diorrhoea rather than mortality doesn’t pool these effects—it doesn’t even pool cholirnation products and flocculation sachets together, or different types of filtration together - https://www.cochrane.org/CD004794/INFECTN_interventions-improve-water-quality-and-prevent-diarrhoea—it’s hard not to notice that neither of these sub-group effects were statistically insignificant until they were pooled together, which makes me worry about p-hacking.
These interventions obviously have spillover benefits to other individuals in the household, so I suspect that focusing on mortality in under-5s significantly underestimates the DALYs averted by point-of-care chlorine dispenser and water filtration interventions.
it’s hard not to notice that neither of these sub-group effects were statistically insignificant until they were pooled together, which makes me worry about p-hacking.
But that’s the whole purpose of a meta analysis like this. All of the individual studies are under-powered to detect an effect on mortality; even if there was a real effect there, mortality is too rare of an event to reliably detect in a small sample.
Right, but pooling or not pooling effects of different interventions relies on a subjective assessment of whether the interventions (chlorine, filtration, spring protection) are similar enough. Kremer et al have made different assessments to the Cochrane review authors, which I think needs justification. The subjectivity in this part of any meta-analysis is very susceptible to p-hacking.
It looks to me like the Kremer paper and the Cochrane review authors have both different methodology and ask different questions—the Cochrane review analysis RCTs as they stand and asks if clean water reduces diarrhoea (which it did), while Kremer mines extra mortality data from previous RCTs then meta-analysis it to look for mortality reduction.
I completely agree the Kremer paper is far more ambitious, and has potential for p-hacking. One of my points in the article though is that Kremer’s mortality reduction finding is eerily similar to what Mills and Reinke found 100 years ago which adds a little more credence I think. Also I like Givewell’s approach of agreeing that there is likely to be a significant mortality benefit, but being more conservative in their approach than the results of Kremer’s study.
What different assessments did you think Kremer made from the Cochrane review authors?
I think pooling different methods is probably fair enough, although like you and Dan point out, p hacking is a possibility in retrospective studies like this with no pre printed protocol.
Yes there are many other benefits, and Givewell accounts for some of these in their analysis. This article was focusing though on the mortality overhang, as it were.
Hey yes I somehow failed to reference the most important paper I was referring to my bad!
Thanks so much for the in depth look here. I agree with all of your points. I was debating writing a list of these issues with the study, but decided not to for simplicity and instead just wrote
“Kremer looks retrospectively at data not gathered for-purpose, which is in epidemiological speak a little dodgy.” And yeah, potential p hacking and noisiness are aspects of that dodginess
A couple of small notes
I think even the 8 percent mortality reduction lower bound wouldn’t completely wipe out the question. Clean water reduces diarrhoea by 30 to 50 percent, leaving a highest plausible mortality reduction of about 5 percent (I think Kremer listed it as 4 in the study?), so even at the lower bound of mortality reduction and higher bound of diarrhea reduction, there is still a discrepancy.
On publication bias, the kind of big studies they are looking at are likely to get published even with negative results, and their funnel plot looking for the bias looked pretty good.
In general I think a huge RCT (potentially even multi county) is still needed which can look at mortality, and can also explore potential reasons for the large overall mortality reduction.
I believe the paper you’re referring to is “Water Treatment And Child Mortality: A Meta-Analysis And Cost-effectiveness Analysis” by Kremer, Luby, Maertens, Tan, & Więcek (2023).
The abstract of this version of the paper (which I found online) says:
That’s a point estimate of a 25-30% reduction in mortality (across 3 methods of estimating that number), with a confidence/credible interval that has a lower bound of a 7-8% reduction in mortality. So, it’s a fairly noisy estimate, due to some combination of the noisiness of individual studies and the heterogeneity across different studies.
That interval for the reduction in mortality just barely overlaps with your number that “Sub-saharan Africa diarrhoea causes 5-10% of child mortality.” (The overlap might be larger if that rate was higher than 5-10% in the years & locations where the studies were conducted.)
So it could be that the clean water interventions prevent most children’s deaths from diarrhoea and few other deaths, if the mortality reduction is near the bottom of the range that Kremer & colleagues estimate. Or they might prevent a decent chunk of other deaths, but not nearly as many as your Part 4 chart & list suggest, if the true mortality reduction is something like 15%.
There is generally also a possibility of a meta-analysis giving inflated results, due to factors like publication bias affecting which studies they include or other methodological issues in the original studies, which could mean that the true effect is smaller than the lower bound of their interval. I don’t know how likely that is in this case.
Here’s a more detailed look at their meta-analysis results:
Two thoughts on this paper:
Does it make sense to pool the effect of chlorine interventions with filtration interventions, when these are two different types of interventions? I don’t think it does and notably the Cochrane review on this topic that looks at diorrhoea rather than mortality doesn’t pool these effects—it doesn’t even pool cholirnation products and flocculation sachets together, or different types of filtration together - https://www.cochrane.org/CD004794/INFECTN_interventions-improve-water-quality-and-prevent-diarrhoea—it’s hard not to notice that neither of these sub-group effects were statistically insignificant until they were pooled together, which makes me worry about p-hacking.
These interventions obviously have spillover benefits to other individuals in the household, so I suspect that focusing on mortality in under-5s significantly underestimates the DALYs averted by point-of-care chlorine dispenser and water filtration interventions.
But that’s the whole purpose of a meta analysis like this. All of the individual studies are under-powered to detect an effect on mortality; even if there was a real effect there, mortality is too rare of an event to reliably detect in a small sample.
Right, but pooling or not pooling effects of different interventions relies on a subjective assessment of whether the interventions (chlorine, filtration, spring protection) are similar enough. Kremer et al have made different assessments to the Cochrane review authors, which I think needs justification. The subjectivity in this part of any meta-analysis is very susceptible to p-hacking.
It looks to me like the Kremer paper and the Cochrane review authors have both different methodology and ask different questions—the Cochrane review analysis RCTs as they stand and asks if clean water reduces diarrhoea (which it did), while Kremer mines extra mortality data from previous RCTs then meta-analysis it to look for mortality reduction.
I completely agree the Kremer paper is far more ambitious, and has potential for p-hacking. One of my points in the article though is that Kremer’s mortality reduction finding is eerily similar to what Mills and Reinke found 100 years ago which adds a little more credence I think. Also I like Givewell’s approach of agreeing that there is likely to be a significant mortality benefit, but being more conservative in their approach than the results of Kremer’s study.
What different assessments did you think Kremer made from the Cochrane review authors?
Nice one.
Thanks freedom interesting questions.
I think pooling different methods is probably fair enough, although like you and Dan point out, p hacking is a possibility in retrospective studies like this with no pre printed protocol.
Yes there are many other benefits, and Givewell accounts for some of these in their analysis. This article was focusing though on the mortality overhang, as it were.
Nice one.
Hey yes I somehow failed to reference the most important paper I was referring to my bad!
Thanks so much for the in depth look here. I agree with all of your points. I was debating writing a list of these issues with the study, but decided not to for simplicity and instead just wrote
“Kremer looks retrospectively at data not gathered for-purpose, which is in epidemiological speak a little dodgy.” And yeah, potential p hacking and noisiness are aspects of that dodginess
A couple of small notes
I think even the 8 percent mortality reduction lower bound wouldn’t completely wipe out the question. Clean water reduces diarrhoea by 30 to 50 percent, leaving a highest plausible mortality reduction of about 5 percent (I think Kremer listed it as 4 in the study?), so even at the lower bound of mortality reduction and higher bound of diarrhea reduction, there is still a discrepancy.
On publication bias, the kind of big studies they are looking at are likely to get published even with negative results, and their funnel plot looking for the bias looked pretty good.
In general I think a huge RCT (potentially even multi county) is still needed which can look at mortality, and can also explore potential reasons for the large overall mortality reduction.