Thanks so much for this. I love that you have done this initial CEA and then revised it. For what its worth even after my comments below, I think its entirely possible that an efficient well drilling organisation could be more cost-effective than in-line chlorination, especially in a place like Niger with the experience this org has. Wells4Wellness seem impressive in their efficiency and transparency.
First, @GiveWell does a lot of research and I think (could be wrong) have basically decided that for the moment they are doubling down on in-line chlorination as a cost-effective, scalable clean water option. Part of this might be that the research on wells is pretty poor, there just isn’t the RCT bank we have for chlorination. Also scaling wells cost effectively and quickly isn’t easy.
One thing I’m super impressed with about Wells4Wellness about is that they include all their expenses in their “cost of a well drilled” calculation. Its simply “Total org money spent this year / No. of wells drilled”. Even many GiveWell orgs fudge around the edges here, and don’t include management, marketing costs etc.in their cost of operations.
I think you’ve done a good job on your CEA, but I still feel like you’re a bit optimistic on your DALY calculation, mainly due to unrealistic time horizons, and lack of discounting. Here are a few reasons why…
(My biggest issue) Your 50 year estimate for well lifespan seems both unrealistic and doesn’t match other cost-effectiveness time horizons. Here in Northern Uganda I haven’t seen boreholes last more than 30ish years (many last less long). When I did our initial CEA for our OneDay Health centers I capped it at 5 years even though they will likely persist much longer. I think maybe 10-15 years makes more sense as a time horizon? More than that is tough to justify given how hard the future is to predict. A lot of things could change in the meantime. Political upheaval can make wells redundant, urbanisation could pull populations away from the borehole (likely), piped water might reach more places making them redundant (see India), wells might break faster than predicted…
Costs of repair and replacement parts might skyrocket as wells get more than 10 years old. Boreholes drilled by Wells4Wellness are still very new. I can’t see only $200 a year being realistic for more than 5 years or so. I’d be interested to know how much is being spent on maintaining their oldest boreholes?
Diarrhoea morbidity has improved over time and that will likely continue.. I don’t think its reasonable to take a 2018 DALY estimate and project it into the future (especially not 50 years), then attribute it all to wells.The 2023 Global burden of disease estimate was 330 DALYs per 1000, down from 360ish in 2018 (already a bit lower than your estimate).
(Small) I got 18% not 20% of Niger’s population under 5 through world bank/ chat GPT but no big deal
The wells serving 1,200 people seems on the high end to me (but uncertain) - population would need to be fairly high density for that to be realistic. Most of Niger is pretty low population density. Unfortunately people will usually opt for the closest water source, even if the clean one is only a little further away. You wouldn’t believe the heavily used dirty pond near my house loads of people use when there are clean options a bit further
(On the positive side) I think you are pretty conservative on your mortality reduction figures which is nice, as you don’t include the mills-reinke effect here.
As a side note (not so important) I don’t think those studies you cited add too much to the discussion, their quality isn’t great. They are broad cross-sectional studies which can’t (and don’t claim to) even hint much at causality. The first study is a slight on BMC global health and their system. How a peer reviewed study gets away with the first sentence which says…
“Diarrhea, the second leading cause of child morbidity and mortality worldwide, is responsible for more than 90% of deaths in children under 5 years of age in low and middle-income countries (LMICs).”
I would love to know what percent the leading cause of mortality makes up then. I mean, I can’t even….. I hope its just a typo…
But as you mention, there are still other uncertainties, and discounting would be the easiest way to deal with those.
To avoid doing harder math, if we assume a 20 year lifespan and no discounting, we’d be at around $16 per DALY (instead of $10 per DALY).
Willie would probably have a better answer for you here and I think he’s going to take a look at this! My understanding is that there are only a couple parts of the well that are at much risk of needing maintenance, and those parts are replaced roughly every 10 years. But there isn’t an expectation that maintenance costs will increase over time
Good point! It looks like the U5 mortality rate in Niger has gone down ~ 10% in the past decade, so diarrhea morbidity also has probably improved. Maybe the easiest way to deal with this is through discounting? Or we can just assume the diarrhea burden goes down ~1% per year
Thanks. I can’t figure out what our source was for 20%, but changing to 18% wouldn’t make a big difference
No idea on this one! Hopefully Willie can talk a bit about that when he checks this out. 1,200 does seem high when I look up “how many people share a water well in africa?”
Thanks for sharing this! We hadn’t seen your post but its very interesting!
We agree the cross-sectional studies we cited are super limited, which is why we relied on GiveWell’s chlorination mortality estimates in our initial draft.
We moved away from GiveWell’s numbers after we got a comment suggesting wells are a bit less effective than chlorination, because water can get contaminated between when its taken from the well and when its used.
We should probably just use GiveWell’s numbers and apply a small penalty for wells (can’t imagine wells are more than 10-15% less effective?). That way we aren’t relying on these meh papers, and can instead rely on a team that’s spent a lot of time thinking about this.
We get very similar estimates when we use GiveWell’s numbers, so I don’t think this would change much. Though if you’re right that GiveWell might be underestimating the effect, that would obviously improve things.
My guess is that making some of the changes mentioned above might 2x-3x the cost per DALY, to something like $20-$30, which would still be great!
Nice one love it, all sounds very reasonable. With CEAs there are always so many tricky decisions to make. Keep in mind this isn’t my specific area of expertise so don’t over index on my suggestions too much 😊.
I’m not sure why there hasn’t been more high quality research on wells given how common an intervention it is. The one big RCT in Ghana i could find showed a reduction in diarrhea if stuff 15 percent, similar to other water cleaning interventions.
Thank you @Zach Gilfix for all of the hard work and insight to this review. My team is very appreciative to have been chosen for this project!
Last night, I attended the local EA meeting and this thread was brought up. My apologies for being late to the game.
Here is my response to a couple of points made by @NickLaing
2. Maintenance is a major priority for Wells 4 Wellness, we believe it is one of the aspects that separates us from other water charities. I am on WhatsApp with my lead drillers and they receive updates from the point person in each village regarding well maintenance. We only drill where a request for a well has been made, so the interest in maintaining the well is already in place before the locals see the benefits of clean water and the need for maintenance. A “purse” system is established at the time the well becomes functional and those funds are used for regular/affordable maintenance and repairs. Locals that see value in the well voluntarily give to the “purse”, but it is not required. Wells 4 Wellness does not keep ownership of the wells, we encourage the village to take ownership and be pro-active about their own water supply. However, if there is a cost that exceeds their purse, I am alerted and I will make a transfer to one of our lead drillers and a tech person/team will make the repair.
On our wells page, I add a list to the bottom line of each year for any repair related activities. This cost is absorbed into the annual average cost of well drilling/production.
We cover large costs due to the fact that the average income is less than one thousand US dollars per year and some repairs would just be completely unattainable for a remote village to cover.
5. When it comes to the number of people these wells provide water to is based on village and surrounding village populations drawing water. There are never 1,200 people waiting for water at the same time. Most family members send one or two people to fetch water in a five gallon jerry can that they bring back to share.
The aquifer is quite large, estimated to be close to four times larger than Lake Erie. When we hit water, our protocol is to continue 15 meters further in order to have a tall water column in the borehole to draw from. The aquifer is quite deep, in some places up to 200 meters. So, we install a water tower with a thousand gallon tank that is constantly being filled by a solar powered immersion pump that is run on solar panels.
This tank is a water battery that allows the jerry cans to be filled quickly without manual labor. As the water is being dispensed, the tank is being replenished at a rate of approximately 2,000 Litres per hour (L/h) equals approximately 528.34 US Gallons per hour (GPH) during the day.
Hope this helps.
I am happy to answer any other questions if they arise! :) - Willie
Thanks so much for this. I love that you have done this initial CEA and then revised it. For what its worth even after my comments below, I think its entirely possible that an efficient well drilling organisation could be more cost-effective than in-line chlorination, especially in a place like Niger with the experience this org has. Wells4Wellness seem impressive in their efficiency and transparency.
First, @GiveWell does a lot of research and I think (could be wrong) have basically decided that for the moment they are doubling down on in-line chlorination as a cost-effective, scalable clean water option. Part of this might be that the research on wells is pretty poor, there just isn’t the RCT bank we have for chlorination. Also scaling wells cost effectively and quickly isn’t easy.
One thing I’m super impressed with about Wells4Wellness about is that they include all their expenses in their “cost of a well drilled” calculation. Its simply “Total org money spent this year / No. of wells drilled”. Even many GiveWell orgs fudge around the edges here, and don’t include management, marketing costs etc.in their cost of operations.
I think you’ve done a good job on your CEA, but I still feel like you’re a bit optimistic on your DALY calculation, mainly due to unrealistic time horizons, and lack of discounting. Here are a few reasons why…
(My biggest issue) Your 50 year estimate for well lifespan seems both unrealistic and doesn’t match other cost-effectiveness time horizons. Here in Northern Uganda I haven’t seen boreholes last more than 30ish years (many last less long). When I did our initial CEA for our OneDay Health centers I capped it at 5 years even though they will likely persist much longer. I think maybe 10-15 years makes more sense as a time horizon? More than that is tough to justify given how hard the future is to predict. A lot of things could change in the meantime. Political upheaval can make wells redundant, urbanisation could pull populations away from the borehole (likely), piped water might reach more places making them redundant (see India), wells might break faster than predicted…
Costs of repair and replacement parts might skyrocket as wells get more than 10 years old. Boreholes drilled by Wells4Wellness are still very new. I can’t see only $200 a year being realistic for more than 5 years or so. I’d be interested to know how much is being spent on maintaining their oldest boreholes?
Diarrhoea morbidity has improved over time and that will likely continue.. I don’t think its reasonable to take a 2018 DALY estimate and project it into the future (especially not 50 years), then attribute it all to wells.The 2023 Global burden of disease estimate was 330 DALYs per 1000, down from 360ish in 2018 (already a bit lower than your estimate).
(Small) I got 18% not 20% of Niger’s population under 5 through world bank/ chat GPT but no big deal
The wells serving 1,200 people seems on the high end to me (but uncertain) - population would need to be fairly high density for that to be realistic. Most of Niger is pretty low population density. Unfortunately people will usually opt for the closest water source, even if the clean one is only a little further away. You wouldn’t believe the heavily used dirty pond near my house loads of people use when there are clean options a bit further
(On the positive side) I think you are pretty conservative on your mortality reduction figures which is nice, as you don’t include the mills-reinke effect here.
As a side note (not so important) I don’t think those studies you cited add too much to the discussion, their quality isn’t great. They are broad cross-sectional studies which can’t (and don’t claim to) even hint much at causality. The first study is a slight on BMC global health and their system. How a peer reviewed study gets away with the first sentence which says…
“Diarrhea, the second leading cause of child morbidity and mortality worldwide, is responsible for more than 90% of deaths in children under 5 years of age in low and middle-income countries (LMICs).”
I would love to know what percent the leading cause of mortality makes up then. I mean, I can’t even….. I hope its just a typo…
Thanks for such a thoughtful reply Nick!
Fair points. We think the 50 year expected lifespan is reasonable, but we should probably account for uncertainty.
Water-wise, it sounds like the basin under Niger is much larger than in other parts of Africa, so there isn’t a risk of the water running out soon.
But as you mention, there are still other uncertainties, and discounting would be the easiest way to deal with those.
To avoid doing harder math, if we assume a 20 year lifespan and no discounting, we’d be at around $16 per DALY (instead of $10 per DALY).
Willie would probably have a better answer for you here and I think he’s going to take a look at this! My understanding is that there are only a couple parts of the well that are at much risk of needing maintenance, and those parts are replaced roughly every 10 years. But there isn’t an expectation that maintenance costs will increase over time
Good point! It looks like the U5 mortality rate in Niger has gone down ~ 10% in the past decade, so diarrhea morbidity also has probably improved. Maybe the easiest way to deal with this is through discounting? Or we can just assume the diarrhea burden goes down ~1% per year
Thanks. I can’t figure out what our source was for 20%, but changing to 18% wouldn’t make a big difference
No idea on this one! Hopefully Willie can talk a bit about that when he checks this out. 1,200 does seem high when I look up “how many people share a water well in africa?”
Thanks for sharing this! We hadn’t seen your post but its very interesting!
We agree the cross-sectional studies we cited are super limited, which is why we relied on GiveWell’s chlorination mortality estimates in our initial draft.
We moved away from GiveWell’s numbers after we got a comment suggesting wells are a bit less effective than chlorination, because water can get contaminated between when its taken from the well and when its used.
We should probably just use GiveWell’s numbers and apply a small penalty for wells (can’t imagine wells are more than 10-15% less effective?). That way we aren’t relying on these meh papers, and can instead rely on a team that’s spent a lot of time thinking about this.
We get very similar estimates when we use GiveWell’s numbers, so I don’t think this would change much. Though if you’re right that GiveWell might be underestimating the effect, that would obviously improve things.
My guess is that making some of the changes mentioned above might 2x-3x the cost per DALY, to something like $20-$30, which would still be great!
Nice one love it, all sounds very reasonable. With CEAs there are always so many tricky decisions to make. Keep in mind this isn’t my specific area of expertise so don’t over index on my suggestions too much 😊.
I’m not sure why there hasn’t been more high quality research on wells given how common an intervention it is. The one big RCT in Ghana i could find showed a reduction in diarrhea if stuff 15 percent, similar to other water cleaning interventions.
tohttps://pmc.ncbi.nlm.nih.gov/articles/PMC4626959/
Thank you @Zach Gilfix for all of the hard work and insight to this review. My team is very appreciative to have been chosen for this project!
Last night, I attended the local EA meeting and this thread was brought up. My apologies for being late to the game.
Here is my response to a couple of points made by @NickLaing
2. Maintenance is a major priority for Wells 4 Wellness, we believe it is one of the aspects that separates us from other water charities. I am on WhatsApp with my lead drillers and they receive updates from the point person in each village regarding well maintenance. We only drill where a request for a well has been made, so the interest in maintaining the well is already in place before the locals see the benefits of clean water and the need for maintenance. A “purse” system is established at the time the well becomes functional and those funds are used for regular/affordable maintenance and repairs. Locals that see value in the well voluntarily give to the “purse”, but it is not required. Wells 4 Wellness does not keep ownership of the wells, we encourage the village to take ownership and be pro-active about their own water supply. However, if there is a cost that exceeds their purse, I am alerted and I will make a transfer to one of our lead drillers and a tech person/team will make the repair.
On our wells page, I add a list to the bottom line of each year for any repair related activities. This cost is absorbed into the annual average cost of well drilling/production.
https://www.wells4wellness.com/wells.html
We cover large costs due to the fact that the average income is less than one thousand US dollars per year and some repairs would just be completely unattainable for a remote village to cover.
-----------------------------------------------------------------
5. When it comes to the number of people these wells provide water to is based on village and surrounding village populations drawing water. There are never 1,200 people waiting for water at the same time. Most family members send one or two people to fetch water in a five gallon jerry can that they bring back to share.
The aquifer is quite large, estimated to be close to four times larger than Lake Erie. When we hit water, our protocol is to continue 15 meters further in order to have a tall water column in the borehole to draw from. The aquifer is quite deep, in some places up to 200 meters. So, we install a water tower with a thousand gallon tank that is constantly being filled by a solar powered immersion pump that is run on solar panels.
https://www.wells4wellness.com/hidden-beneath-the-drifting-sand-is-a-freshwater-ocean-ready-to-make-waves-in-the-sahara-desert.html
This tank is a water battery that allows the jerry cans to be filled quickly without manual labor. As the water is being dispensed, the tank is being replenished at a rate of approximately 2,000 Litres per hour (L/h) equals approximately 528.34 US Gallons per hour (GPH) during the day.
Hope this helps.
I am happy to answer any other questions if they arise! :) - Willie