Hi again, Nick! Sorry it took so long to answer your remaining points!
So, first about this one:
(Potentially shaky assumption shared by GiveWell). The 2.35 Mills Reincke multiplier for over 5s. I could be wrong but I’m not sure there’s any good data at all on mortality reductions for clean water in over 5s. The Kremer study Is the only one I know which looks directly at mortality at all, and then only in under 5s. Where does this number come from? My intuition would that the multiplier should be far smaller than this (very very uncertain).
Taking a more careful look into the footnotes and supplemental materials for GiveWell’s discussion of the Mills-Reincke effect, it appears that they have gathered mixed evidence on its existence for the over-5 population:
This supplementary spreadsheet lists six econometric studies including overall all-cause mortality effects from water quality improvements. These are mostly natural experiments regarding the implementation of municipal water quality improvements in now-developed countries (Germany, USA, Sweden, Japan), during the 20th and early 21st centuries (a summary of all papers is available here; we have only briefly reviewed the relevant ones). Both the mean and the median effect sizes on the population-wide all-cause mortality were −19%, ranging from +0.9 to −58%. Surprisingly, this is an even larger effect size than the one observed when considering under-5 mortality alone (median: −11%; mean: −10 to −13%). This can be taken at least as some rough evidence that the MR phenomenon can be widespread across age groups. [1]
On the other hand, a long note to the “Adjustment for smaller Mills-Reincke effect in over-5s” cell of their CEA, authors of the GW report on water quality interventions mention one study (Newman et al. 2020) that looked into one of the plausible mechanisms for explaining the MR phenomenon and found no evidence of such effects on adults. However, the report’s authors argue that it may still be true for older children and adolescents:
[...] However, we do not have much evidence on whether it operates in people over 5. Based on conversations with experts[...], we believe a likely mechanism by which the Mills-Reincke phenomenon operates is that enteric diseases impair nutritional status and body energy reserves, increasing risk from subsequent infectious diseases. This mechanism is expected to affect young children the most since they have smaller body reserves of energy, protein, and other nutrients. We identified one study that addresses this hypothesis empirically in under-5s and over-5s side-by-side. Newman et al. 2020 measured the risk of respiratory infection following a bout of diarrhea in infants and mothers. Infants had a substantially elevated risk of respiratory infection following diarrhea, while mothers did not. This provides some evidence that the Mills-Reincke phenomenon may not operate in adults. It is worth noting that the “over 5” category we use in this CEA is not composed solely of adults, and children over 5 may still be susceptible to the Mills-Reincke phenomenon. To derive this adjustment, we assume that Mills-Reincke has an impact half as large in over-5s as in under-5s. This is a very uncertain guess. We also assume that water quality interventions avert 2.7 non-enteric deaths for each enteric death averted (see water quality report for calculation, link below), and that the 50% downward adjustment only applies to non-enteric deaths.
In any case, we added a sensitivity analysis to our Effectiveness Supplement where we consider no Mills-Reincke effect for age groups over 5 y.o. And it does make a great difference. On average, it reduces the estimated (cost-)effectiveness of filtration interventions by 33%, and by 21% for chlorination interventions across all countries(simple average). For our top 5 prioritized countries, the difference is even bigger: a 43% reduction for filtration, and 36% for chlorination [2]. For Nigeria, for instance, the effectiveness lowers from ~24k DALYs/100,000 people served, to ~13k DALYs/100,000 people served.
Given the magnitude of the difference, we suggest further research would greatly benefit from trying to better understand the evidence base for or against the existence of the Mills-Reincke effect among older children, adolescents, and adults.
Note that GiveWell itself mentioned reasons to take these estimates with a grain of salt. Quoting them (end of the “Studies of historical water quality improvements” section of their report):
Several factors lead us to be uncertain about the relevance of these studies to modern interventions that target waterborne disease in low-income settings:
- Most were conducted in contexts that differ substantially from the contexts in which water quality interventions would be applied today. This includes differences in location, infectious disease profile, baseline mortality rates, and other factors.
- The water quality interventions represented in these studies vary, and none is identical to the interventions to which we might direct funding.
- These studies are observational and use different strategies to attempt to isolate the causal impact of water quality on mortality. We believe some of these strategies are more convincing than others, but all of them may be susceptible to some degree of confounding from other variables that changed alongside water quality.
- We do not know whether unsupportive studies went unpublished, potentially creating publication bias. This would make the overall literature appear more supportive than it would otherwise be.
- None of these studies were preregistered, increasing the risk of bias.
For these reasons, we view these studies as providing rough triangulation for our mortality reduction estimate and a plausible mechanism to explain its unexpectedly large effect size. We do not see them as providing precise effect size estimates for the impact of water quality interventions on mortality.
Differences across countries and intervention types are due to the different YLL and YLD proportions coming from each pathogen and each age group in each country (according to data from GBD 2019).
From my experience it is borderline implausible that 5.8% of diarrhoeal episodes lead to hospitalisation. I’m not sure if you referenced where this number came from? I’ve worked in Northern Uganda for 10 years and diarrhoea is a far less common than you would expect reason for presentation to primary care (Under 3% of our OneDay Health Patients present with diarrhoea) and at best an uncommon reason for hospitalisation. I even removed treating diarrhoea from our OneDay Health cost-effectiveness analysis because it appeared insignificant (both because the effect of treating one patient is so small, and because we treat so few patients for it)
It took a while, but we finally found some good empirical data regarding hospitalization rates due to diarrhea in LMICs, beyond the four smaller studies GiveWell references for justifying their 5.8% estimate [1].
So, the Global Enteric Multicenter Study of Diarrheal Disease in Infants and Young Children in Developing Countries (GEMS) was a 3-year, multi-site case-control study focusing on diarrhea in children under 5 years old living in seven low-income countries in sub-Saharan Africa and South Asia (source).
Even though the main GEMS study had a focus on moderate and severe cases, it was preceded by a preparatory survey called “Health Care Utilization and Attitudes Survey (HUAS)”. According to their “Establishing a Sampling Frame for the Case/Control Study and Selecting Health Centers for Case Recruitment” section:
In preparation for the case/control study, we performed a Health Care Utilization and Attitudes Survey (HUAS). An age-stratified sample of approximately 1000 children aged 0–59 months per site randomly selected from each updated DSS [demographic surveillance system] dataset was visited at home, and parents/primary caretakers were asked whether their child had experienced diarrhea during the previous 14 days. If so, the presence of findings suggestive of MSD [moderate-to-severe diarrhea] was solicited (sunken eyes, wrinkled skin, hospitalization, receipt of intravenous hydration, or dysentery), and source(s) of healthcare were recorded. These data were used to adjust the size of the DSS population at each site as necessary to contribute the requisite number of cases of MSD to each age stratum, and to select 1 or more “sentinel” health centers (SHCs) serving the DSS population at each site (Table 1) as venues for the case/control study based on their potential to capture MSD cases from the DSS.
It turns out that the resulting data from the HUAS study is easily accessible for anyone to consult. And it recorded the number of children under 5 who reportedly had diarrhea during the 14 days previous to the interview, and whether they were admitted to a hospital.
According to this data, out of 5,171 who reportedly suffered from diarrhea in the previous 14 days, and for whom data for this variable was available, 270 children were admitted to hospitals. This gives a 5.22% hospitalization rate (95% CI: 4.65-5.86%), which is roughly what GiveWell estimated from a set of smaller studies.
Therefore, we think this part of the CEA is also roughly aligned with evidence.
Finally, about the other points you raised, we recommend you (and everyone interested in the theme) check the new version of the report published by Ambitious Impact/Charity Entrepreneurship when recommending this intervention for CE’s next incubation round[1].
AIM/CE researchers dedicated a few hours to polishing the remaining loose ends in our previous version. It includes a model for calculating the mortality reductions based on diarrhea case-fatality rates. This is a slightly enhanced version of our model and represents an interesting proposition on how to extrapolate the effect of the intervention from cases to mortality—given more time, it could even be expanded to consider pathogen-specific case-fatality rates, which should be even better.
Hi again, Nick! Sorry it took so long to answer your remaining points!
So, first about this one:
Taking a more careful look into the footnotes and supplemental materials for GiveWell’s discussion of the Mills-Reincke effect, it appears that they have gathered mixed evidence on its existence for the over-5 population:
This supplementary spreadsheet lists six econometric studies including overall all-cause mortality effects from water quality improvements. These are mostly natural experiments regarding the implementation of municipal water quality improvements in now-developed countries (Germany, USA, Sweden, Japan), during the 20th and early 21st centuries (a summary of all papers is available here; we have only briefly reviewed the relevant ones). Both the mean and the median effect sizes on the population-wide all-cause mortality were −19%, ranging from +0.9 to −58%. Surprisingly, this is an even larger effect size than the one observed when considering under-5 mortality alone (median: −11%; mean: −10 to −13%). This can be taken at least as some rough evidence that the MR phenomenon can be widespread across age groups. [1]
On the other hand, a long note to the “Adjustment for smaller Mills-Reincke effect in over-5s” cell of their CEA, authors of the GW report on water quality interventions mention one study (Newman et al. 2020) that looked into one of the plausible mechanisms for explaining the MR phenomenon and found no evidence of such effects on adults. However, the report’s authors argue that it may still be true for older children and adolescents:
In any case, we added a sensitivity analysis to our Effectiveness Supplement where we consider no Mills-Reincke effect for age groups over 5 y.o. And it does make a great difference. On average, it reduces the estimated (cost-)effectiveness of filtration interventions by 33%, and by 21% for chlorination interventions across all countries (simple average). For our top 5 prioritized countries, the difference is even bigger: a 43% reduction for filtration, and 36% for chlorination [2]. For Nigeria, for instance, the effectiveness lowers from ~24k DALYs/100,000 people served, to ~13k DALYs/100,000 people served.
Given the magnitude of the difference, we suggest further research would greatly benefit from trying to better understand the evidence base for or against the existence of the Mills-Reincke effect among older children, adolescents, and adults.
Note that GiveWell itself mentioned reasons to take these estimates with a grain of salt. Quoting them (end of the “Studies of historical water quality improvements” section of their report):
Differences across countries and intervention types are due to the different YLL and YLD proportions coming from each pathogen and each age group in each country (according to data from GBD 2019).
About this other point:
It took a while, but we finally found some good empirical data regarding hospitalization rates due to diarrhea in LMICs, beyond the four smaller studies GiveWell references for justifying their 5.8% estimate [1].
So, the Global Enteric Multicenter Study of Diarrheal Disease in Infants and Young Children in Developing Countries (GEMS) was a 3-year, multi-site case-control study focusing on diarrhea in children under 5 years old living in seven low-income countries in sub-Saharan Africa and South Asia (source).
Even though the main GEMS study had a focus on moderate and severe cases, it was preceded by a preparatory survey called “Health Care Utilization and Attitudes Survey (HUAS)”. According to their “Establishing a Sampling Frame for the Case/Control Study and Selecting Health Centers for Case Recruitment” section:
It turns out that the resulting data from the HUAS study is easily accessible for anyone to consult. And it recorded the number of children under 5 who reportedly had diarrhea during the 14 days previous to the interview, and whether they were admitted to a hospital.
According to this data, out of 5,171 who reportedly suffered from diarrhea in the previous 14 days, and for whom data for this variable was available, 270 children were admitted to hospitals. This gives a 5.22% hospitalization rate (95% CI: 4.65-5.86%), which is roughly what GiveWell estimated from a set of smaller studies.
Therefore, we think this part of the CEA is also roughly aligned with evidence.
The four studies are Burton et al. 2011, Page et al. 2011, Breiman et al. 2011, and Omore et al. 2013. See footnote #92 of GW’s report.
Finally, about the other points you raised, we recommend you (and everyone interested in the theme) check the new version of the report published by Ambitious Impact/Charity Entrepreneurship when recommending this intervention for CE’s next incubation round [1].
AIM/CE researchers dedicated a few hours to polishing the remaining loose ends in our previous version. It includes a model for calculating the mortality reductions based on diarrhea case-fatality rates. This is a slightly enhanced version of our model and represents an interesting proposition on how to extrapolate the effect of the intervention from cases to mortality—given more time, it could even be expanded to consider pathogen-specific case-fatality rates, which should be even better.
We, the authors of the originial report, had only very limited participation in the changes made between the two versions.