Hey yes I somehow failed to reference the most important paper I was referring to my bad!
Thanks so much for the in depth look here. I agree with all of your points. I was debating writing a list of these issues with the study, but decided not to for simplicity and instead just wrote
“Kremer looks retrospectively at data not gathered for-purpose, which is in epidemiological speak a little dodgy.” And yeah, potential p hacking and noisiness are aspects of that dodginess
A couple of small notes
I think even the 8 percent mortality reduction lower bound wouldn’t completely wipe out the question. Clean water reduces diarrhoea by 30 to 50 percent, leaving a highest plausible mortality reduction of about 5 percent (I think Kremer listed it as 4 in the study?), so even at the lower bound of mortality reduction and higher bound of diarrhea reduction, there is still a discrepancy.
On publication bias, the kind of big studies they are looking at are likely to get published even with negative results, and their funnel plot looking for the bias looked pretty good.
In general I think a huge RCT (potentially even multi county) is still needed which can look at mortality, and can also explore potential reasons for the large overall mortality reduction.
Hey yes I somehow failed to reference the most important paper I was referring to my bad!
Thanks so much for the in depth look here. I agree with all of your points. I was debating writing a list of these issues with the study, but decided not to for simplicity and instead just wrote
“Kremer looks retrospectively at data not gathered for-purpose, which is in epidemiological speak a little dodgy.” And yeah, potential p hacking and noisiness are aspects of that dodginess
A couple of small notes
I think even the 8 percent mortality reduction lower bound wouldn’t completely wipe out the question. Clean water reduces diarrhoea by 30 to 50 percent, leaving a highest plausible mortality reduction of about 5 percent (I think Kremer listed it as 4 in the study?), so even at the lower bound of mortality reduction and higher bound of diarrhea reduction, there is still a discrepancy.
On publication bias, the kind of big studies they are looking at are likely to get published even with negative results, and their funnel plot looking for the bias looked pretty good.
In general I think a huge RCT (potentially even multi county) is still needed which can look at mortality, and can also explore potential reasons for the large overall mortality reduction.