it’s hard not to notice that neither of these sub-group effects were statistically insignificant until they were pooled together, which makes me worry about p-hacking.
But that’s the whole purpose of a meta analysis like this. All of the individual studies are under-powered to detect an effect on mortality; even if there was a real effect there, mortality is too rare of an event to reliably detect in a small sample.
Right, but pooling or not pooling effects of different interventions relies on a subjective assessment of whether the interventions (chlorine, filtration, spring protection) are similar enough. Kremer et al have made different assessments to the Cochrane review authors, which I think needs justification. The subjectivity in this part of any meta-analysis is very susceptible to p-hacking.
It looks to me like the Kremer paper and the Cochrane review authors have both different methodology and ask different questions—the Cochrane review analysis RCTs as they stand and asks if clean water reduces diarrhoea (which it did), while Kremer mines extra mortality data from previous RCTs then meta-analysis it to look for mortality reduction.
I completely agree the Kremer paper is far more ambitious, and has potential for p-hacking. One of my points in the article though is that Kremer’s mortality reduction finding is eerily similar to what Mills and Reinke found 100 years ago which adds a little more credence I think. Also I like Givewell’s approach of agreeing that there is likely to be a significant mortality benefit, but being more conservative in their approach than the results of Kremer’s study.
What different assessments did you think Kremer made from the Cochrane review authors?
But that’s the whole purpose of a meta analysis like this. All of the individual studies are under-powered to detect an effect on mortality; even if there was a real effect there, mortality is too rare of an event to reliably detect in a small sample.
Right, but pooling or not pooling effects of different interventions relies on a subjective assessment of whether the interventions (chlorine, filtration, spring protection) are similar enough. Kremer et al have made different assessments to the Cochrane review authors, which I think needs justification. The subjectivity in this part of any meta-analysis is very susceptible to p-hacking.
It looks to me like the Kremer paper and the Cochrane review authors have both different methodology and ask different questions—the Cochrane review analysis RCTs as they stand and asks if clean water reduces diarrhoea (which it did), while Kremer mines extra mortality data from previous RCTs then meta-analysis it to look for mortality reduction.
I completely agree the Kremer paper is far more ambitious, and has potential for p-hacking. One of my points in the article though is that Kremer’s mortality reduction finding is eerily similar to what Mills and Reinke found 100 years ago which adds a little more credence I think. Also I like Givewell’s approach of agreeing that there is likely to be a significant mortality benefit, but being more conservative in their approach than the results of Kremer’s study.
What different assessments did you think Kremer made from the Cochrane review authors?
Nice one.