Nigeria pilot report: Reducing child mortality from diarrhoea with ORS & zinc, Clear Solutions
Summary
We introduce Clear Solutions, a Charity Entrepreneurship (now AIM) incubated charity founded in September 2023. Our focus is the prevention of deaths of young children from diarrhoea, an illness that kills approximately 444,000 children under-5 every year.
From December 2023 to February 2024, we ran a pilot distribution of low-cost, highly effective treatments for diarrhoea, oral rehydration solution and zinc (ORSZ) in Kano, Nigeria, with implementation partner iDevPro Africa. We estimate having reached ~6900 children under-5. The intervention, based upon a randomised controlled trial in Uganda (Wagner et al, 2019), provides free co-packaged ORS and zinc (“co-packs”) door-to-door to all households with children under 5 years old. The distribution is performed by local Community Health Workers (CHWs), who provide guidance and printed instructions on ORSZ usage during the visit.
We surveyed communities pre- and post-intervention, allowing 6 weeks between ORSZ distribution and follow-up surveys for diarrhoea cases to accumulate. At these survey rounds, we recorded the timing of the child’s last diarrhoea episode (if applicable) and how they were treated (if at all). Our primary outcome measure is the change in ORSZ usage rates pre-to-post intervention, though also we collected extensive contextual data to monitor operations and guide program improvements.
This post summarises our preliminary analysis and conclusions. A more detailed report is available on our website here. We were kindly supported by knowledgeable advisors, but did not have an academic partnership, nor has this analysis been peer-reviewed. Nonetheless, we believe there is value in sharing our results and learnings with this community.
Results in brief: Across 4 wards (geographic areas) of differing rurality, baseline usage rates for under-5s’ last diarrhoea episode in the preceding 4 weeks were reported at a range across wards of 44.7% − 50.9% for ORS and 11.1% − 26.7% for ORS+zinc when asked directly. At follow-up post-intervention, the usage rate for the preceding 4-weeks was reported at a range across wards of 90.0 − 97.7% for ORS and 88.2% − 94.1% for ORSZ. (95% margins of error up to 10pp and are not shown here for readability; see Results for details.)
Superficially, this indicates a change of 42.0 − 52.8 percentage points (pp) in ORS use and 61.5 − 83.0pp for ORSZ. However, we treat this result with caution, with specific concerns such as social desirability bias in survey responses inflating true values. We discuss more in Limitations below.
Conclusions in brief: We consider this to be a solid result in favour of the intervention having a strong potential to prevent deaths in a cost-effective manner in the Nigerian context. (We do not estimate cost-effectiveness in this report, but will be working on a follow-up with that). There are, however, clear limitations in the pilot that warrant considerable down-weighting of our results, though we do not expect this to change the conclusions qualitatively.
Introducing Clear Solutions
Clear Solutions was founded in September 2023 with the support of Charity Entrepreneurship (now AIM). Our mission is to prevent deaths of young children from diarrhoea, a leading cause of death for under-5s globally, in a cost-effective and evidence-based manner.
The 1970s medical breakthrough, Oral Rehydration Solution (ORS), a dosed mixture of sugar, salts and water, unlocked the possibility of preventing >90% of diarrhoeal deaths at full coverage. The addition of zinc can reduce diarrhoea duration and recurrence, and the World Health Organisation recognised this in 2019 by adding co-packaged ORS and Zinc (together, ORSZ) to their model National Essential Medicines list.
The Global Health community has invested heavily in reducing diarrhoeal mortality with ORS for several decades, and significant progress has been made in many countries. For example, efforts have been made to improve the market dynamics for ORS and zinc, enable health workers, and increase demand from caregivers (Goh & Yobo, 2016). However, about 444,000 children under-5 still die from diarrhoea each year, with sub-Saharan Africa suffering a disproportionate burden.
A 2019 randomised controlled trial in Uganda provided inspiration for our approach as a new organisation in this historied domain. Wagner et al. 2019 found a >30% (19 percentage points) increase in ORSZ usage in communities where ORSZ was provided free door-to-door by Community Health Workers (CHWs), compared to a control of CHWs’ normal mode which included selling ORSZ. Importantly, the ORSZ was provided free and preemptively (ie, for caregivers to retain for immediate availability when a child suffers diarrhoea). Some earlier academic literature (eg. Zwisler et al, 2013, Digre et al, 2016) indicated caregivers were sceptical towards ORS, perceiving it as less like “real medicine” than injections or tablets.
Cost-effectiveness analyses by Charity Entrepreneurship, and later GiveWell in their August 2023 ORSZ Program report, modelled free pre-emptive community distribution. Givewell estimated $2.6K per death averted in Nigeria, with Charity Entrepreneurship ranging as low as $2.1K in Chad and $6.9K average over 8 high burden SSA countries.
Geographic selection and contextual learning
Diarrhoeal mortality and existing ORSZ usage rates are major determinants of our impact potential. 5-10 countries in sub-Saharan Africa have the combination of high mortality rates, low ORSZ usage, and large enough population to make sense for our first intervention. After initial exploration of implementing in Ethiopia, Madagascar and Nigeria, we ultimately decided to focus first on Nigeria for reasons of scale, language and partner networks.
Within Nigeria, we were again guided by mortality and ORSZ usage data at a state level, and applied more practical considerations, with an initial plan to execute ‘micropilots’ in up to 3 of our shortlisted states, Enugu, Federal Capital Territory, Gombe, and Kano. Our November 2023 visits involved meeting potential partner teams and Ministry of Health representatives to secure support for the intervention and plan for implementation.
During visits to Gombe and Kano, we performed focus groups and individual interviews with key stakeholders including caregivers, community leaders, community health workers, and facility health workers. These were valuable in understanding the local context: awareness of, and demand for, ORS was relatively high (contrary to our initial impression from some literature, but possibly as a result of subsequent global health initiatives), but the cost of the ORS itself was cited as significant barriers to regular use. Some participants also indicated that ORS distributed for free may be ‘diverted’ outside of our intended under-5s usage groups, with demand existing across the full range of age groups.
We were pleasantly surprised to find that the partner in Kano, iDevPro Africa, had a window to secure governmental approvals and execute very quickly at a larger scale during December 2023. Thus our plan pivoted to capitalise on this opportunity and we proceeded with the pilot as outlined below.
Pilot overview
Six months into this journey, we have preliminary results from our pilot of community distribution of co-packaged oral rehydration solution and zinc (ORSZ) diarrhoea treatments in Kano, Nigeria.
The pilot ran from December 2023 to February 2024, with ORSZ distributions in 20 communities across 4 wards in 2 Local Government Areas. We estimate having reached ~6900 children under-5, based on distribution and procurement records. Survey staff were deployed before, immediately after, and 6 weeks after distribution to gather operational learnings and enable evaluation of the intervention impact.
Community Health Workers (CHWs) from the National Primary Healthcare Development Agency CHIPS (Community Health Influencers, Promoters & Services) program—and Volunteer Community Mobilisers, one of its predecessors—performed the ORSZ distribution. The CHIPS agents and VCMs are a volunteer cadre within the overall primary healthcare system and are well known within their communities.
Over a period of 3-5 days, each CHW visited households in their community and, if they had children under-5 and a caregiver present, the CHW talked them through an ORSZ instructions sheet and gave one ORSZ co-pack (2 ORS sachets, 10 zinc tablets) per child under-5. Households were marked with chalk to help with later identification by survey staff. Paper distribution tracking sheets were used to monitor progress.
Survey staff visited a random sample of households at Baseline (before CHW visits), without prior knowledge of whether under-5s were present, and later used the CHIPS/VCMs’ chalk markings to identify eligible households for Spotcheck and Follow-Up surveys. A smartphone survey app (SurveyCTO Collect) was used to capture survey results.
Preliminary analysis and results
We performed the following analysis on operational data from the intervention and surveys. Given the absence of peer review or academic partnership, we present just summary-level highlights to avoid implying levels of precision or rigour beyond the practical reality. We calculated our 95% confidence intervals using z-scores (with combined variances for the difference in proportions calculations).
Operationally, surveys and distribution were divided into 2 stages by location, with work in Ward 1 (Urban Slum) preceding Wards 2-4 (Urban, Peri-Urban, Rural) by approximately 2 weeks to enable operational learnings to be incorporated.
Data collection was focussed on a primary outcome of change in ORSZ usage rate. We also collected other contextual data, inspired by comparable datasets such as the Demographic Health Survey (DHS) and our benchmark study, Wagner et al. 2019 in Uganda.
Diarrhoea Prevalence
One of the first data points we ascertained was the timing of the last episode of diarrhoea suffered by each child under 5 as reported by the caregiver. From this, we calculate the prevalence over defined periods preceding the survey.
| 2-week Prevalence | 4-week Prevalence | |||||||
| Ward | Baseline (PRE) | Follow-Up (POST) | Baseline (PRE) | Follow-Up (POST) | ||||
| Urban Slum (nPRE 787, nPOST 1164) | 30.1% | ±3.2pp | 18.4% | ±2.2pp | 41.8% | ±3.4pp | 32.7% | ±2.7pp |
| Urban (nPRE 280, nPOST 504) | 18.2% | ±4.5pp | 22.4% | ±3.6pp | 38.2% | ±5.7pp | 36.3% | ±4.2pp |
| Peri-Urban (nPRE 807, nPOST 816) | 20.7% | ±2.8pp | 15.4% | ±2.5pp | 30.5% | ±3.2pp | 22.7% | ±2.9pp |
| Rural (nPRE 997, nPOST 876) | 24.1% | ±2.7pp | 18.7% | ±2.6pp | 33.3% | ±2.9pp | 35.7% | ±3.2pp |
Note: Diarrhoea prevalence measures the percentage of children under the age of 5 who had diarrhoea in the defined weeks preceding the survey. All margins of error shown are for a 95% confidence interval. nPRE and nPOST indicate the number of under-5s reflected in the PRE and POST results.
Comparison to external data sources
DHS & MICS
To sense-check our data, we compared our data points to well-recognised external surveys such as The Demographic Health Survey (DHS) and the Multiple Indicator Cluster Surveys (MICS). These surveys contain many key child health metrics including diarrhoea prevalence and ORSZ usage rates. Both of these surveys measure a 2-week prevalence for diarrhoea, with the timing of these surveys in Nigeria being DHS 2018 and MICS 2021. We reviewed the data specific to Kano state.
| Treatment | DHS 2018 | MICS 2021 | Pilot data range |
| 2-week Prevalence | 17.7% | 22.5% | 18.2 − 30.1% |
These data points give us some confidence in the accuracy of the data we collected, while acknowledging the limitations in generalisability of our data (refer to limitations section). We note that the urban slum ward had a disproportionately high prevalence at baseline at 30.1%; we consider the possibility that this may be due to a local outbreak (anecdotally provided).
ORS/Zinc/Both Usage rates, pre- and post-intervention
Next, we investigated our primary question, the impact of the intervention on ORSZ usage for diarrhoea in under-5s. The table below shows the proportion of children with diarrhoea in the 4 weeks preceding the survey who were treated with ORS, zinc, or both. Rate change measures the percentage point (as opposed to percentage) increase from the PRE-intervention group to the POST-intervention group.
In designing our questionnaire, we considered how different biases may play a role in ascertaining accurate information. Thus, “Unprompted’ (“How did you treat the diarrhoea?”) and “Prompted” (“Did you use ORS?”) question styles were used in this sequence to elicit treatment information. We hypothesised potential under-reporting bias from failure to recall or recognise ORS as a treatment for Unprompted, and over-reporting social desirability bias for Prompted.
| Ward | Question style | % ORS/Z usage to treat last diarrhoea in 4-week window | RATE CHANGE pp (PRE to POST ) | |||||
| PRE | POST | |||||||
Urban Slum nPRE 329 nPOST 357 | Unprompted | ORS | 36.2% | ±5.2pp | 91.3% | ±2.9pp | 55.1 | ±6pp |
| Zinc | 31.3% | ±5pp | 93.3% | ±2.6pp | 62.0 | ±5.6pp | ||
| Both | 18.5% | ±4.2pp | 89.4% | ±3.2pp | 70.8 | ±5.3pp | ||
| Prompted | ORS | 44.7% | ±5.4pp | 95.0% | ±2.3pp | 50.3 | ±5.8pp | |
| Zinc | 35.6% | ±5.2pp | 96.1% | ±2pp | 60.5 | ±5.6pp | ||
| Both | 23.4% | ±4.6pp | 92.7% | ±2.7pp | 69.3 | ±5.3pp | ||
Urban nPRE 106 nPOST 181 | Unprompted | ORS | 49.1% | ±9.5pp | 92.3% | ±3.9pp | 43.2 | ±10.3pp |
| Zinc | 32.1% | ±8.9pp | 92.8% | ±3.8pp | 60.7 | ±9.6pp | ||
| Both | 16.0% | ±7pp | 90.1% | ±4.4pp | 74.0 | ±8.2pp | ||
| Prompted | ORS | 50.9% | ±9.5pp | 93.9% | ±3.5pp | 43.0 | ±10.1pp | |
| Zinc | 35.8% | ±9.1pp | 94.5% | ±3.3pp | 58.6 | ±9.7pp | ||
| Both | 19.8% | ±7.6pp | 91.7% | ±4pp | 71.9 | ±8.6pp | ||
Peri- Urban nPRE 247 nPOST 170 | Unprompted | ORS | 24.7% | ±5.4pp | 90.0% | ±4.5pp | 65.3 | ±7pp |
| Zinc | 26.3% | ±5.5pp | 83.5% | ±5.6pp | 57.2 | ±7.8pp | ||
| Both | 15.0% | ±4.5pp | 81.2% | ±5.9pp | 66.2 | ±7.4pp | ||
| Prompted | ORS | 44.9% | ±6.2pp | 96.5% | ±2.8pp | 51.5 | ±6.8pp | |
| Zinc | 40.1% | ±6.1pp | 90.6% | ±4.4pp | 50.5 | ±7.5pp | ||
| Both | 26.7% | ±5.5pp | 88.2% | ±4.8pp | 61.5 | ±7.3pp | ||
Rural nPRE 334 nPOST 305 | Unprompted | ORS | 21.3% | ±4.4pp | 93.8% | ±2.7pp | 72.5 | ±5.2pp |
| Zinc | 8.7% | ±3pp | 91.5% | ±3.1pp | 82.8 | ±4.4pp | ||
| Both | 5.1% | ±2.4pp | 89.5% | ±3.4pp | 84.4 | ±4.2pp | ||
| Prompted | ORS | 44.9% | ±5.3pp | 97.7% | ±1.7pp | 52.8 | ±5.6pp | |
| Zinc | 15.9% | ±3.9pp | 96.4% | ±2.1pp | 80.5 | ±4.4pp | ||
| Both | 11.1% | ±3.4pp | 94.1% | ±2.6pp | 83.0 | ±4.3pp | ||
Note: All margins of error shown are for a 95% confidence interval. These are unadjusted data. nPRE and nPOST indicate the number of under-5s’ last diarrhoea episode treatment reflected in the PRE and POST results.
Comparison to external data sources
We consider three comparable datasets: DHS 2018 (Nigeria), MICS 2021 (Nigeria), and Wagner et al 2019 study (Uganda). We use DHS and MICS to sense-check our baseline ORSZ usage rates, and the Uganda study to compare our effect sizes.
DHS & MICS
The Kano-specific numbers for ORS/Zinc usage rates are as follows. The comparable figures would be the PRE-intervention data for the prompted questions. Note that both these surveys measured in the 2-week time block in keeping with their diarrhoea prevalence data.
| Treatment | DHS 2018 | MICS 2021 | Pilot data range |
| ORS | 52.8% | 42.4% | 44.7 − 50.9% |
| Zinc | 45.6% | 43.9% | 15.9 − 40.1% |
| ORS and Zinc | 37.3% | 25.7% | 11.1 − 26.7% |
Our pilot data looks moderately comparable for the ORS usage rate but tends to be lower for the Zinc and ORSZ usage rates. Our pilot data seems consistent with our community members’ focus groups, in which awareness and usage of zinc were markedly low. It is plausible that differences in the chance of sampling explain the slightly lower zinc usage rates seen in the urban, urban-slum, and peri-urban wards. The significant drop in this rate for the rural ward may be indicative of the impact of rurality on zinc usage.
Wagner et al 2019 (Uganda RCT 2019)
The Wagner et al. 2019 study in Uganda captured data for a time block of 4 weeks and is the main benchmark study for our analysis. This study compared four different ORS distribution methods: our comparisons of interest are the “free and convenient ” and control groups.
The free and convenient group closely reflects our pilot intervention group, with CHWs delivering ORSZ co-packs door-to-door for free prior to a child being ill. The control group differs slightly in that in the Wagner et al 2019 study, this was the ‘status quo’ where the CHWs sell ORS, zinc and other health products door-to-door once per month. In practice, only a minority of homes were visited each month. Similar services are not present in our pilot setting to the best of our knowledge, though there was some overlap with another free distribution program, ANRiN, which is currently winding down. We consider the implication of this in the next section.
The relevant Wagner results are as follows (copied from the abstract):
“During follow-up, … share of cases treated with ORS was… 77% (448/584) in the free and convenient group,… 56% (335/597) in the control group… After adjusting for potential confounders, instructing CHWs to provide free and convenient distribution increased ORS coverage by 19 percentage points relative to the control group (95% CI 13–26; P < 0.001)...”
In our pilot data, the most conservative increase in ORS usage rate was 43 pp (unadjusted), a remarkably high value compared to the Wagner et al. 2019 study of 19 pp (21 pp unadjusted). The Wagner study noted that “the intervention had a stronger effect among villages with particularly low ORS use at baseline”, and our pilot had a lower baseline (47% − 51%) compared to the Wagner study (56%). However, the magnitude of the difference is too large to be explained by this alone.
If our results approximate the realities in Nigeria, this is a significant opportunity with the potential for a very cost-effective intervention. However, we carefully consider the limitations of our pilot in a later section and how ‘discounts’ might be applied to the apparent effect size.
Overlap with another ORSZ distribution program
A nutrition project, ANRiN, also distributed ORSZ within Kano in a period overlapping with our pilot. Upon further analysis, 6.5% of the households in our pilot had interaction with both interventions. The urban ward only had one household reporting an ANRiN visit so we excluded this ward from the following analysis. We examined the difference in the proportion of ORS usage (for direct questions) grouped by whether the caregiver reported ANRiN interaction during the 6-week follow-up period.
| Used ORS | Did not use ORS | Proportion using ORS | |
| Did not interact with ANRiN | 922 | 69 | 93.04% |
| Interacted with ANRiN | 90 | 1 | 98.90% |
We performed a chi-squared test on this contingency table which returned a p-value of 0.05075. We do not want to assert any strong statistical conclusions from this given the numerous limitations described in a later section. However, this does nudge us to speculate that having additional co-packs and/or multiple exposures to staff promoting ORSZ could have a positive impact.
Reasons for non-usage of ORS & zinc
We also asked the caregivers about the reasons for not using ORSZ for their child’s diarrhoea in order to understand potential barriers. At Baseline the top cited reasons for this were as follows.
| ORS | Zinc | ||
| Did not have it at home when diarrhoea started | 35% | Did not know the benefit of it for diarrhoea | 38% |
| Did not know the benefit of it for diarrhoea | 14% | Did not have it at home when diarrhoea started | 21% |
| Too expensive | 9% | Too expensive | 9% |
| Child did not seem ill enough to need it | 3% | Not prescribed or not available at the clinic / hospital / chemist | 3% |
| Not prescribed or not available at the clinic / hospital / chemist | 3% | Gave a different medicine | 1% |
| Child won’t drink it | 2% | ||
At Follow-Up, the high ORSZ usage rate and limited 6-week timeframe for eligible diarrhoea cases meant a much smaller sample, within which “Child refuses” was the main reason provided for ORS (22%) and zinc (24%).
These are interesting findings in light of the caregiver focus group noting ‘price’ as the main barrier. Though qualitative results need to be looked at more carefully (e.g. are there factors that people would be less inclined to mention directly), we can gather that all three elements of price, education, and pre-emptive ownership of ORSZ seem important.
Diversion
Since the intervention is based upon home administration of ORSZ when a child is ill, the caregiver has an opportunity to ‘divert’ ORSZ to users who are less vulnerable to diarrhoeal mortality, ie. older children or adults. Having learned of diversion as a potentially meaningful side-effect of positive awareness of ORS in the local context during focus groups, we sought to quantify it and explore mitigations.
We collected relevant data at two different points in the survey as outlined below. The following data are for “eligible” households in our intervention; ie. they have 1+ child under-5 and a primary caregiver home at the time of the visit. The data reflects a 6-week window from distribution to follow-up surveys.
% Eligible Households who had used ORS (any source) for a given age category
| Ward | Elderly | Non-Elderly Adult | Child 5+ | Child Under-5 |
| Urban Slum | 8% | 27% | 21% | 70% |
| Urban | 8% | 20% | 13% | 56% |
| Peri-Urban | 3% | 24% | 10% | 50% |
| Rural | 4% | 16% | 13% | 63% |
Note: This usage is for ORS from any source, with small chemist shops/stalls (“PPMVs”) being a common additional source. This data was collected at the beginning of the survey to determine the general pattern of ORS usage in a household regardless of its source.
This usage pattern (50-70% of eligible households using ORS for children under-5) is not concerning in and of itself, but raises the question of whether the freely distributed ORS was ‘diverted’. As a signal for this, we examine the following data.
% Clear Solutions ORS sachets used, % by age group
| Ward | Elderly | Non-Elderly Adult | Child 5+ | Child Under-5 |
| Urban Slum | 2% | 3% | 8% | 87% |
| Urban | 1% | 7% | 9% | 82% |
| Peri-Urban | 2% | 12% | 7% | 79% |
| Rural | 1% | 7% | 8% | 84% |
Note: The ORS sachets here refer specifically to those provided as part of the co-packs from Clear Solutions. This data was collected at the end of the survey. Approximately 2000 ORS sachets were reported used during the 6-week period.
The proportion of ‘intended’ usage for children under 5 was 79-87%, whilst the ‘diversion’ rate was 13-21%.
An ORS sachet or co-pack used for an older child or adult is one that is not available to treat an ill under-5, our target population. The primary basis for our impact evaluation is mortality aversion in the under-5 population. Thus, it is important to understand what proportion of our ORS is reaching that population, the distribution size and frequency necessary to provide good under-5 coverage, and the associated cost-effectiveness impact of the diversion.
We suspect that the 13-21% diversion rate observed here is likely to be an underestimation of the true rate because the importance of retaining and using the co-packs specifically for children under 5 was emphasised during distribution. Thus, we believe that caregivers were subject to social desirability bias (see “Limitations” section) which may yield an inflated proportion of the socially ‘correct’ answer, ie. under 5s, not other groups.
We think it is possible to influence this diversion rate with targeted behaviour change methods, perhaps modifying the CHW messaging or product packaging. This is something we will explore in our future programs.
Limitations
There are several limitations, beyond those already described in the body, to the pilot program that are important to keep in mind as we interpret these data.
Generalisability
Though we examined wards of differing ruralities, this represents a small geographic sample of the state of Kano. Thus, generalisability at a wider level (state, federal) could be limited. Furthermore, we only present summary-level data that are not adjusted for potential confounders.
Social Desirability Bias
A challenge in evaluating the ORSZ distribution intervention is that we are not able to directly observe treatment taking place, instead providing instruction and later capturing respondents’ treatment reports.
During follow-up surveys, when a caregiver is asked whether ORSZ was used for the child’s diarrhoea, the response can be subject to social desirability bias as caregivers may be tempted to provide answers they believe will be viewed favourably by others. Furthermore, caregivers may also believe that saying they have used our ORSZ will lead to more being given, or if not used, it may be taken away. We tried to mitigate this through a clear and repeated disclaimer that their response would not change how they are treated in the intervention.
Repeated sampling of the population
Survey respondents experienced between 1 and 4 interactions with staff from our pilot program (1 CHW and sampling in up to 3 survey rounds), which may have accentuated the social desirability bias. We have tried to mitigate this by not emphasising our origin and purpose of visits, but our data enumeration team found that many caregivers seemed to recognise the enumerators during follow-ups. We believe that this could bias the results more towards giving a “yes, I used ORS” response as caregivers may want to appear appreciative of the organisation that distributed and educated them on the benefits of this product.
This challenge can be meaningfully mitigated as our scale of operation becomes larger allowing us to sample a smaller number of participants from each community. Ideally, we would run a quasi-randomised design which can provide a more confident result than a pre-post comparison.
Statistical Assumptions Non-adherence
Our statistical tests are limited by the fact that the survey data does not adhere to the assumption of independent samples. We did not have distinct households in both pre and post-surveys, rather they overlap meaningfully given our high sampling rates. Furthermore, there are inherent correlations of population within the same communities. These factors could lead us to underestimate the true variance of our effect size estimates, resulting in unduly narrow confidence intervals.
Assumptions of pre/post comparison
The nature of the pre/post-test we conducted comes with many underlying assumptions. We elaborate on two assumptions that may be particularly applicable in our context.
The first assumption is that by attributing all the changes between our pre-group and post-group solely to our intervention, we are implying that no other factors played a role that may have influenced the ORSZ usage rate. The 6-week interval between the two surveys could be considered a relatively short time for major external changes that may impact our outcome. However, one clear example where this assumption does not hold is the overlap with the ANRiN program as mentioned earlier.
The second assumption is that the activity of the pre-survey would not have impacted the outcome of the post-survey. However, in our context, it seems plausible that more interaction with the program staff (including data enumerators) may act as a behaviour-changing activity. For example, by helping caregivers remember and believe the importance of ORSZ usage for children’s diarrhoea, we might assert additional influence upon the post-survey usage rates for those who were exposed to the pre-survey.
Factors supporting a large effect size
We outline above why our results may overestimate the intervention impact and it will be important to generously discount our effect size as a result. However, we also want to provide an alternative perspective that may support the genuineness of our large effect size.
It was often thought that one of the challenges of improving ORS uptake is people often do not see ORS as of ‘medicinal value’ and thus limiting its uptake. However, our formative research indicated that the communities we interviewed had positive attitudes towards ORS—so much so that adults would also use this product for various reasons beyond diarrhoea such as for strength, weakness, and dehydration. However, it was also noted that price was the main barrier to using ORS.
Whilst the above finding raised concerns regarding the potential diversion of our freely provided ORSZ co-packs, this also points to the fact that once the price barrier is removed, a very high usage rate of these products may not seem too far-fetched.
Learning and Conclusions
We are positively surprised by the apparent effect size of the intervention and the operational success that delivered it. We are very aware of the risks that come with surveys as a measure of reality, given recall bias, social desirability bias, and other errors that are impossible to fully remove. Nonetheless, we conclude that the intervention was successful in increasing ORSZ treatment of children under-5, with a “least favourable” result of 43pp increase (urban ward, high baseline ORS usage), and larger apparent effects in the other wards, for zinc, and for ORS and zinc in combination.
Diversion of co-packs to other age groups was reported and we consider this an area warranting further analysis and possibly dedicated research. At a reported 79+% of our ORS sachets used for under-5s, the loss would not impact cost-effectiveness unduly; but we must take care to establish a reliable signal for this, given the social factors at play in caregivers admitting usage outside of the target group. We explore possible options in the next section.
It is also likely that the initially high effect size we observe at the beginning of the program will diminish over time. This could be due to recall, as the novelty wears off, or just a natural phenomenon of behaviour change interventions. Thus, it will be important to understand the frequency and the number of co-packs that need to be distributed to sustain the program.
We have not yet performed a detailed cost-effectiveness analysis of the pilot: we will cover that in a later post. In the meantime, our general sense is that the pilot was lower cost per co-pack delivered than initially expected given our limited economies of scale. The total program cost (ie. excluding “HQ costs”, mostly co-founder salaries) of the pilot was approximately US$10,000. This includes a disproportionate spend (~50%) on surveying at much higher sampling rates than would be likely at scale.
Next steps
With such promising pilot results, we plan to take measured steps to scale-up, and build upon the learnings to date with operational variants aimed at improving the core intervention.
An important consideration for our 12-18 month plan is the GiveWell grant to CHAI for ORSZ distribution and RCT in Bauchi state. With coincidental proximity in timing and geography (neither we nor CHAI knew of each other’s plans until late 2023), we see a great opportunity to bring agility and innovation to optimise intervention cost-effectiveness (we discuss potential avenues below), in parallel to CHAI’s work on proving the foundation with the large-scale RCT. We are optimistic for a fruitful collaboration that increases our collective impact significantly.
Operational variants and studies under consideration
As we plan for the remainder of 2024, we have ideas for operational variants or studies to improve cost-effectiveness or strengthen impact evaluation. These go beyond the extensive set of tactical improvements that we will take forward to the next intervention stage, and are presented here to give a flavour of the opportunities we are considering in parallel to the core.
Studying “worst case” diversion rates by providing co-packs without under-5 specific messaging, then surveying usage without a clear ‘correct answer’
Enabling caregivers to request ORSZ ‘top-ups’ via SMS, with turnarounds such that CHWs may verify the reported child’s illness.
Testing co-pack labelling variants (eg. colour, images) impact upon diversion rates.
Trialling community-centre distribution points for “top-ups” after the first round of door-to-door outreach. Eg. “Quarterly ORSZ Day”.
Short-term next steps
While we evaluate the ideas above, we are also planning for a refined execution of the core implementation at 2-3x scale in the next 3-4 months.
Our focus will be:
Improving recruitment, training, and supervision approaches for the quality and scalability of CHWs.
Improving diversion rate survey estimates with more coherent cross-checks and consistency checking between questions.
Automating the processing of paper distribution tracking sheets (used by CHWs who do not have consistent access to or familiarity with smartphones) with photos and image processing.
Interested in learning more or supporting us?
Thank you for your interest!
Get in touch or follow our journey with our newsletter and social media.
We would love to hear from you with suggestions from related work, improvement ideas, or with introductions to folks who may be interested in collaborating or supporting us.
Volunteering & Paid Positions
We are not yet hiring for permanent positions, but will update our career notifications group when we do so. We especially welcome interest from, or introductions to, potential candidates for a role overseeing on-the-ground and partner operations at a state/regional level in northern Nigeria.
We have some capacity to work with volunteers who can support with outreach & social media, M&E methods and design, data analysis, and charity registration in UK or US. If you are interested in volunteering, please review the volunteer guide on our careers page.
We also have a need for specialist experience within Nigeria primary healthcare, community health intervention design, and build-out of SMS-based and voucher interventions.
Financial support
We project that we are funded until September 2024, but we have ambitious plans in mind for the final quarter, with a funding gap of approximately $100,000.
You can donate here any time, but for materials to support a larger funding decision, please do get in touch.
A huge thank you to everyone who has supported us to date, with partnership, funding, advice, and encouragement!
More detailed report, updates until June′24
A slightly more detailed version of this report is posted at clearsolutions.global/pilot. We expect to make additions to it until June′24 as time allows further analysis and reflection.
Thanks for sharing this report, and for all the work that went into this program so far.
Regarding the social desirability bias, and survey problems generally, there may be a few tweaks that would help with the situation.
Social desirability bias in surveys can be significantly reduced by using the “list experiment” technique.
There might we a way to phrase the question so that the social desirability bias goes the other way. For example, instead of asking “did you use the products”, you could ask “do you still have the products?”
If you ask people to keep the packaging after use, then you could ask to see it (and observe if it has been used, or not). This might also help estimate diversion.
Regarding the overlap with ANRiN, have you estimated the prior probability of that happening, given the size of the programs? It makes me wonder if there is a bias in the selection of treatment locations that makes this more likely, and which might also affect results in other ways. For example, maybe both organizations are selecting treatment locations with better transportation infrastructure, in which case the program might prove harder to scale in the future.
This point about altering the questioning is brilliant. Personally (many disagree with me) I think that social desirability and “future hope,” bias are so overwhelming when free stuff has been given out that there’s almost no point in asking someone if they used the ors correctly.
I like the “do you still have the products” as a betterquestion to partly mitigate that or you could take that further and ask “can you show me the ORS”.
I also find the drop in “2 week prevalence” suspicious pre and post, as is the pre prevalence of 38 percent in the slum. This high rate seems implausible, unless there’s a cholera epidemic going on or similar (and even then it’s probably still almost impossible).
I also had a question about what percent of the ORS given was actually used during the 4 week period. Not what percent of diarrhea was treated with ORS, but what percent of what was overall given out was reportedly used. This is an important figure to help you check potential reporting bias as well. I might be missing it but I couldn’t see it there.
Interested to hear your thoughts on this. Thanks for all the amazing work
Hi Nick, thanks for sharing your thoughts and excellent points.
Regarding the urban slum rates, thank you for calling this out! On digging back into it, we realise we unfortunately missed copy-pasting corrected prevalence data into the report when we fixed a code bug for urban slum Baseline (which initially included diarrhoea instances for an additional week for both timeframes). The other wards used a later survey version with separate logic and are not impacted. The pre-post results data in the report were updated after the fix, so no change needed there.
That all said, the corrected urban slum Baseline 2-week and 4-week prevalences of 30.1% and 41.8% (will update original post) are still comparatively high.
Regarding the proportion of all distributed ORS that was used, we asked households how many ORS sachets they used by age-group for the 6-week follow-up period and counted how many sachets they had left, so we do have those extra signals with which to scrutinise their claimed ORS treatment rates. A complicator here is the unknown volume of ORS used in a given treatment; 2 x one litre sachets are provided per co-pack, but one litre can be sufficient depending on the diarrhoea duration and whether the caregiver abides by the instruction to discard prepared ORS after 24 hours. Nonetheless, this is certainly something for us to look into further.
Thanks again for your thoughtful comments and for helping improve our program!
Nice one great reply!
Just to double check (still not completely clear) did you distribute to about 1000 households and about 930 of those used the ORS over a 6 week follow up period? Or did you distribute to more than that, but about 1000 households said their kid had diarrhoea in that time?
A bit confused whether 93% - ish of all families given ORS used it over 6 weeks or something different? Obviously it would be impossible for that higher proportion of the families to have kids with Diarrhoea in only a 6 week period—still trying to get my head around this.
Nick.
Something different :) But I think I see what you’re getting at.
Total distribution was to ~4000 households (families) across the 4 wards. The question on usage by age group of ‘our’ ORS was asked at follow-up, with approx. 2400 HHs surveyed. Of the ORS sachets used by HHs in that sample, the data in the table “% Clear Solutions ORS sachets used, % by age group” expresses who (by age group) used them.
I think you’re asking what proportion of all the ORS provided was used (1) by anyone, and (2) by under-5s. This is a good point, as it gives an idea of what revisit cadence, or potential increase in packets given-per-child, would be needed for continuous coverage.
Said sample of HHs reported 3317 co-packs received (ie. 6634 ORS sachets, 2 per co-pack), with 2046 sachets reported used (with ~80%, varying by ward, used for under-5s). So 2046⁄6634 = 30.8% of the ORS sachets distributed were reported used within the 6-week follow-up period. We’ll prep this also for addition to the report—thank you!
Thanks that makes sense. I was actually trying to ask what proportion of the households reported using ORS, not what percent of sachets were used. I think I get most of it it now nice one, still one thing I’m not clear on is...
Of the 2400 Households surveyed, how many of them reported using any ORS at all after 6 weeks? That’s a crucial number for me both as a sanity check and uptake check. I’m not sure if that’s in this report here or not
Just my 2 cents, but I think its helpful to start a report with the really basic design stuff, ie We gave out xxxx ORS sachets to XXXX Families while doing XXXXX education, then followed them up after XXXX weeks—I struggled a little bit to follow the process here. Not a big deal
That’s good feedback, thanks: we’ve perhaps leapt too directly from the conceptual description to the results, without properly quantifying the basic operation. Noted for improvement.
To (finally, I hope!) answer your question, of the ~2400 (2381) households surveyed—actually 2163 once we filter-out the non-consents:
1518 (70.1%) reported any occupant using ORS from any source; and
924 (42.7%) reported any occupant using ORS from our distribution.
in the 6 weeks of our follow-up period.
This is “all wards” data, so may skew somewhat according to exact response numbers per ward. Please take these numbers as provisional / subject-to-error in the name of a timely response.
We have not looked at them in more depth yet, but I see the value in this perspective and we’ll think more about what we might learn from them. I’m also interested in your take on what we might infer from these, Nick (and others).
Hi Ian, thank you for your comment!
Thank you for your suggestions on the social desirability front. Do you have specific resources you could suggest regarding this? We have tried incorporating some of your points such as packet counting in the pilot but are always looking for other methods like you listed.
We unfortunately did not have an in-depth view of the ANRiN program during our pilot implementation. However, we have since gotten in touch and are aiming to understand some of your questions retrospectively if possible. Going forward, it is likely that the overlap will not be of major concern as this is the final year of funding for ANRiN. However, we will aim to monitor this status as well.
Regardless, your point on the treatment location bias is an important consideration that we have not paid as much attention to at the pilot stage. Instead, we focused more on quickly learning the operational feasibility and were not as meticulous in treatment location selection beyond having diverse rurality representation. We will certainly pay more attention to this as we plan for the next phase.
Hi Charlie, thanks for your reply.
I am a dilettante and don’t have much further to offer on social desirability bias, unfortunately. You might try connecting with a social scientist, development economist, or staff at one of the EA or EA-adjacent global health and development charities operating at the frontier of evidence for their respective interventions, such as GiveWell, GiveDirectly, Living Goods, IDinsight, DMI, Evidence Action, etc.
Executive summary: Clear Solutions’ pilot program in Kano, Nigeria showed promising results in increasing usage of oral rehydration solution and zinc (ORSZ) to treat diarrhea in children under 5 through free, preemptive community distribution by community health workers.
Key points:
Clear Solutions aims to reduce child mortality from diarrhea by increasing usage of ORSZ in a cost-effective way, inspired by a 2019 Uganda RCT.
A pilot in Kano, Nigeria reached ~6,900 under-5 children. Surveys measured diarrhea prevalence and ORSZ usage rates before and after the intervention.
ORSZ usage rates for under-5s increased substantially, by 43-84 percentage points depending on the ward. Some diversion of ORSZ to older age groups was observed.
Results are promising but have limitations. Social desirability bias and overlap with another ORSZ program may have inflated the effect size.
Next steps include scaling up the intervention, collaborating with CHAI on their GiveWell-funded RCT, and testing operational variants to improve cost-effectiveness.
Clear Solutions seeks additional funding of ~$100K to support ambitious plans through 2024.
This comment was auto-generated by the EA Forum Team. Feel free to point out issues with this summary by replying to the comment, and contact us if you have feedback.