Thanks so much for this. I love that you have done this initial CEA and then revised it. For what its worth even after my comments below, I think its entirely possible that an efficient well drilling organisation could be more cost-effective than in-line chlorination, especially in a place like Niger with the experience this org has. Wells4Wellness seem impressive in their efficiency and transparency.
First, @GiveWell does a lot of research and I think (could be wrong) have basically decided that for the moment they are doubling down on in-line chlorination as a cost-effective, scalable clean water option. Part of this might be that the research on wells is pretty poor, there just isn’t the RCT bank we have for chlorination. Also scaling wells cost effectively and quickly isn’t easy.
One thing I’m super impressed with about Wells4Wellness about is that they include all their expenses in their “cost of a well drilled” calculation. Its simply “Total org money spent this year / No. of wells drilled”. Even many GiveWell orgs fudge around the edges here, and don’t include management, marketing costs etc.in their cost of operations.
I think you’ve done a good job on your CEA, but I still feel like you’re a bit optimistic on your DALY calculation, mainly due to unrealistic time horizons, and lack of discounting. Here are a few reasons why…
(My biggest issue) Your 50 year estimate for well lifespan seems both unrealistic and doesn’t match other cost-effectiveness time horizons. Here in Northern Uganda I haven’t seen boreholes last more than 30ish years (many last less long). When I did our initial CEA for our OneDay Health centers I capped it at 5 years even though they will likely persist much longer. I think maybe 10-15 years makes more sense as a time horizon? More than that is tough to justify given how hard the future is to predict. A lot of things could change in the meantime. Political upheaval can make wells redundant, urbanisation could pull populations away from the borehole (likely), piped water might reach more places making them redundant (see India), wells might break faster than predicted…
Costs of repair and replacement parts might skyrocket as wells get more than 10 years old. Boreholes drilled by Wells4Wellness are still very new. I can’t see only $200 a year being realistic for more than 5 years or so. I’d be interested to know how much is being spent on maintaining their oldest boreholes?
Diarrhoea morbidity has improved over time and that will likely continue.. I don’t think its reasonable to take a 2018 DALY estimate and project it into the future (especially not 50 years), then attribute it all to wells.The 2023 Global burden of disease estimate was 330 DALYs per 1000, down from 360ish in 2018 (already a bit lower than your estimate).
(Small) I got 18% not 20% of Niger’s population under 5 through world bank/ chat GPT but no big deal
The wells serving 1,200 people seems on the high end to me (but uncertain) - population would need to be fairly high density for that to be realistic. Most of Niger is pretty low population density. Unfortunately people will usually opt for the closest water source, even if the clean one is only a little further away. You wouldn’t believe the heavily used dirty pond near my house loads of people use when there are clean options a bit further
(On the positive side) I think you are pretty conservative on your mortality reduction figures which is nice, as you don’t include the mills-reinke effect here.
As a side note (not so important) I don’t think those studies you cited add too much to the discussion, their quality isn’t great. They are broad cross-sectional studies which can’t (and don’t claim to) even hint much at causality. The first study is a slight on BMC global health and their system. How a peer reviewed study gets away with the first sentence which says…
“Diarrhea, the second leading cause of child morbidity and mortality worldwide, is responsible for more than 90% of deaths in children under 5 years of age in low and middle-income countries (LMICs).”
I would love to know what percent the leading cause of mortality makes up then. I mean, I can’t even….. I hope its just a typo…
But as you mention, there are still other uncertainties, and discounting would be the easiest way to deal with those.
To avoid doing harder math, if we assume a 20 year lifespan and no discounting, we’d be at around $16 per DALY (instead of $10 per DALY).
Willie would probably have a better answer for you here and I think he’s going to take a look at this! My understanding is that there are only a couple parts of the well that are at much risk of needing maintenance, and those parts are replaced roughly every 10 years. But there isn’t an expectation that maintenance costs will increase over time
Good point! It looks like the U5 mortality rate in Niger has gone down ~ 10% in the past decade, so diarrhea morbidity also has probably improved. Maybe the easiest way to deal with this is through discounting? Or we can just assume the diarrhea burden goes down ~1% per year
Thanks. I can’t figure out what our source was for 20%, but changing to 18% wouldn’t make a big difference
No idea on this one! Hopefully Willie can talk a bit about that when he checks this out. 1,200 does seem high when I look up “how many people share a water well in africa?”
Thanks for sharing this! We hadn’t seen your post but its very interesting!
We agree the cross-sectional studies we cited are super limited, which is why we relied on GiveWell’s chlorination mortality estimates in our initial draft.
We moved away from GiveWell’s numbers after we got a comment suggesting wells are a bit less effective than chlorination, because water can get contaminated between when its taken from the well and when its used.
We should probably just use GiveWell’s numbers and apply a small penalty for wells (can’t imagine wells are more than 10-15% less effective?). That way we aren’t relying on these meh papers, and can instead rely on a team that’s spent a lot of time thinking about this.
We get very similar estimates when we use GiveWell’s numbers, so I don’t think this would change much. Though if you’re right that GiveWell might be underestimating the effect, that would obviously improve things.
My guess is that making some of the changes mentioned above might 2x-3x the cost per DALY, to something like $20-$30, which would still be great!
Nice one love it, all sounds very reasonable. With CEAs there are always so many tricky decisions to make. Keep in mind this isn’t my specific area of expertise so don’t over index on my suggestions too much 😊.
I’m not sure why there hasn’t been more high quality research on wells given how common an intervention it is. The one big RCT in Ghana i could find showed a reduction in diarrhea if stuff 15 percent, similar to other water cleaning interventions.
Thanks so much for this. I love that you have done this initial CEA and then revised it. For what its worth even after my comments below, I think its entirely possible that an efficient well drilling organisation could be more cost-effective than in-line chlorination, especially in a place like Niger with the experience this org has. Wells4Wellness seem impressive in their efficiency and transparency.
First, @GiveWell does a lot of research and I think (could be wrong) have basically decided that for the moment they are doubling down on in-line chlorination as a cost-effective, scalable clean water option. Part of this might be that the research on wells is pretty poor, there just isn’t the RCT bank we have for chlorination. Also scaling wells cost effectively and quickly isn’t easy.
One thing I’m super impressed with about Wells4Wellness about is that they include all their expenses in their “cost of a well drilled” calculation. Its simply “Total org money spent this year / No. of wells drilled”. Even many GiveWell orgs fudge around the edges here, and don’t include management, marketing costs etc.in their cost of operations.
I think you’ve done a good job on your CEA, but I still feel like you’re a bit optimistic on your DALY calculation, mainly due to unrealistic time horizons, and lack of discounting. Here are a few reasons why…
(My biggest issue) Your 50 year estimate for well lifespan seems both unrealistic and doesn’t match other cost-effectiveness time horizons. Here in Northern Uganda I haven’t seen boreholes last more than 30ish years (many last less long). When I did our initial CEA for our OneDay Health centers I capped it at 5 years even though they will likely persist much longer. I think maybe 10-15 years makes more sense as a time horizon? More than that is tough to justify given how hard the future is to predict. A lot of things could change in the meantime. Political upheaval can make wells redundant, urbanisation could pull populations away from the borehole (likely), piped water might reach more places making them redundant (see India), wells might break faster than predicted…
Costs of repair and replacement parts might skyrocket as wells get more than 10 years old. Boreholes drilled by Wells4Wellness are still very new. I can’t see only $200 a year being realistic for more than 5 years or so. I’d be interested to know how much is being spent on maintaining their oldest boreholes?
Diarrhoea morbidity has improved over time and that will likely continue.. I don’t think its reasonable to take a 2018 DALY estimate and project it into the future (especially not 50 years), then attribute it all to wells.The 2023 Global burden of disease estimate was 330 DALYs per 1000, down from 360ish in 2018 (already a bit lower than your estimate).
(Small) I got 18% not 20% of Niger’s population under 5 through world bank/ chat GPT but no big deal
The wells serving 1,200 people seems on the high end to me (but uncertain) - population would need to be fairly high density for that to be realistic. Most of Niger is pretty low population density. Unfortunately people will usually opt for the closest water source, even if the clean one is only a little further away. You wouldn’t believe the heavily used dirty pond near my house loads of people use when there are clean options a bit further
(On the positive side) I think you are pretty conservative on your mortality reduction figures which is nice, as you don’t include the mills-reinke effect here.
As a side note (not so important) I don’t think those studies you cited add too much to the discussion, their quality isn’t great. They are broad cross-sectional studies which can’t (and don’t claim to) even hint much at causality. The first study is a slight on BMC global health and their system. How a peer reviewed study gets away with the first sentence which says…
“Diarrhea, the second leading cause of child morbidity and mortality worldwide, is responsible for more than 90% of deaths in children under 5 years of age in low and middle-income countries (LMICs).”
I would love to know what percent the leading cause of mortality makes up then. I mean, I can’t even….. I hope its just a typo…
Thanks for such a thoughtful reply Nick!
Fair points. We think the 50 year expected lifespan is reasonable, but we should probably account for uncertainty.
Water-wise, it sounds like the basin under Niger is much larger than in other parts of Africa, so there isn’t a risk of the water running out soon.
But as you mention, there are still other uncertainties, and discounting would be the easiest way to deal with those.
To avoid doing harder math, if we assume a 20 year lifespan and no discounting, we’d be at around $16 per DALY (instead of $10 per DALY).
Willie would probably have a better answer for you here and I think he’s going to take a look at this! My understanding is that there are only a couple parts of the well that are at much risk of needing maintenance, and those parts are replaced roughly every 10 years. But there isn’t an expectation that maintenance costs will increase over time
Good point! It looks like the U5 mortality rate in Niger has gone down ~ 10% in the past decade, so diarrhea morbidity also has probably improved. Maybe the easiest way to deal with this is through discounting? Or we can just assume the diarrhea burden goes down ~1% per year
Thanks. I can’t figure out what our source was for 20%, but changing to 18% wouldn’t make a big difference
No idea on this one! Hopefully Willie can talk a bit about that when he checks this out. 1,200 does seem high when I look up “how many people share a water well in africa?”
Thanks for sharing this! We hadn’t seen your post but its very interesting!
We agree the cross-sectional studies we cited are super limited, which is why we relied on GiveWell’s chlorination mortality estimates in our initial draft.
We moved away from GiveWell’s numbers after we got a comment suggesting wells are a bit less effective than chlorination, because water can get contaminated between when its taken from the well and when its used.
We should probably just use GiveWell’s numbers and apply a small penalty for wells (can’t imagine wells are more than 10-15% less effective?). That way we aren’t relying on these meh papers, and can instead rely on a team that’s spent a lot of time thinking about this.
We get very similar estimates when we use GiveWell’s numbers, so I don’t think this would change much. Though if you’re right that GiveWell might be underestimating the effect, that would obviously improve things.
My guess is that making some of the changes mentioned above might 2x-3x the cost per DALY, to something like $20-$30, which would still be great!
Nice one love it, all sounds very reasonable. With CEAs there are always so many tricky decisions to make. Keep in mind this isn’t my specific area of expertise so don’t over index on my suggestions too much 😊.
I’m not sure why there hasn’t been more high quality research on wells given how common an intervention it is. The one big RCT in Ghana i could find showed a reduction in diarrhea if stuff 15 percent, similar to other water cleaning interventions.
tohttps://pmc.ncbi.nlm.nih.gov/articles/PMC4626959/