Does it make sense to pool the effect of chlorine interventions with filtration interventions, when these are two different types of interventions? I don’t think it does and notably the Cochrane review on this topic that looks at diorrhoea rather than mortality doesn’t pool these effects—it doesn’t even pool cholirnation products and flocculation sachets together, or different types of filtration together - https://www.cochrane.org/CD004794/INFECTN_interventions-improve-water-quality-and-prevent-diarrhoea—it’s hard not to notice that neither of these sub-group effects were statistically insignificant until they were pooled together, which makes me worry about p-hacking.
These interventions obviously have spillover benefits to other individuals in the household, so I suspect that focusing on mortality in under-5s significantly underestimates the DALYs averted by point-of-care chlorine dispenser and water filtration interventions.
it’s hard not to notice that neither of these sub-group effects were statistically insignificant until they were pooled together, which makes me worry about p-hacking.
But that’s the whole purpose of a meta analysis like this. All of the individual studies are under-powered to detect an effect on mortality; even if there was a real effect there, mortality is too rare of an event to reliably detect in a small sample.
Right, but pooling or not pooling effects of different interventions relies on a subjective assessment of whether the interventions (chlorine, filtration, spring protection) are similar enough. Kremer et al have made different assessments to the Cochrane review authors, which I think needs justification. The subjectivity in this part of any meta-analysis is very susceptible to p-hacking.
It looks to me like the Kremer paper and the Cochrane review authors have both different methodology and ask different questions—the Cochrane review analysis RCTs as they stand and asks if clean water reduces diarrhoea (which it did), while Kremer mines extra mortality data from previous RCTs then meta-analysis it to look for mortality reduction.
I completely agree the Kremer paper is far more ambitious, and has potential for p-hacking. One of my points in the article though is that Kremer’s mortality reduction finding is eerily similar to what Mills and Reinke found 100 years ago which adds a little more credence I think. Also I like Givewell’s approach of agreeing that there is likely to be a significant mortality benefit, but being more conservative in their approach than the results of Kremer’s study.
What different assessments did you think Kremer made from the Cochrane review authors?
I think pooling different methods is probably fair enough, although like you and Dan point out, p hacking is a possibility in retrospective studies like this with no pre printed protocol.
Yes there are many other benefits, and Givewell accounts for some of these in their analysis. This article was focusing though on the mortality overhang, as it were.
Two thoughts on this paper:
Does it make sense to pool the effect of chlorine interventions with filtration interventions, when these are two different types of interventions? I don’t think it does and notably the Cochrane review on this topic that looks at diorrhoea rather than mortality doesn’t pool these effects—it doesn’t even pool cholirnation products and flocculation sachets together, or different types of filtration together - https://www.cochrane.org/CD004794/INFECTN_interventions-improve-water-quality-and-prevent-diarrhoea—it’s hard not to notice that neither of these sub-group effects were statistically insignificant until they were pooled together, which makes me worry about p-hacking.
These interventions obviously have spillover benefits to other individuals in the household, so I suspect that focusing on mortality in under-5s significantly underestimates the DALYs averted by point-of-care chlorine dispenser and water filtration interventions.
But that’s the whole purpose of a meta analysis like this. All of the individual studies are under-powered to detect an effect on mortality; even if there was a real effect there, mortality is too rare of an event to reliably detect in a small sample.
Right, but pooling or not pooling effects of different interventions relies on a subjective assessment of whether the interventions (chlorine, filtration, spring protection) are similar enough. Kremer et al have made different assessments to the Cochrane review authors, which I think needs justification. The subjectivity in this part of any meta-analysis is very susceptible to p-hacking.
It looks to me like the Kremer paper and the Cochrane review authors have both different methodology and ask different questions—the Cochrane review analysis RCTs as they stand and asks if clean water reduces diarrhoea (which it did), while Kremer mines extra mortality data from previous RCTs then meta-analysis it to look for mortality reduction.
I completely agree the Kremer paper is far more ambitious, and has potential for p-hacking. One of my points in the article though is that Kremer’s mortality reduction finding is eerily similar to what Mills and Reinke found 100 years ago which adds a little more credence I think. Also I like Givewell’s approach of agreeing that there is likely to be a significant mortality benefit, but being more conservative in their approach than the results of Kremer’s study.
What different assessments did you think Kremer made from the Cochrane review authors?
Nice one.
Thanks freedom interesting questions.
I think pooling different methods is probably fair enough, although like you and Dan point out, p hacking is a possibility in retrospective studies like this with no pre printed protocol.
Yes there are many other benefits, and Givewell accounts for some of these in their analysis. This article was focusing though on the mortality overhang, as it were.
Nice one.