A simplified cost-effectiveness estimation methodology for use in Solar4Africa’s Fall-23 student projects
In this post, I present in general terms the simplified, cost-effectiveness estimation methodology that we plan to use in student projects that are evaluating a variety off-grid solar-electric technology access interventions in rural Africa.
Each of the interventions are focused on the Gobal Health and Welfare (GHW) cause area and either (A) Use funds contributed by philanthropic donors to subsidize access to a particular off-grid solar-electric product or service, or (B) Evaluate new, more efficient ways of administering philanthropic subsidies to potentially a large variety of distributed development activities.
Generally, the question that we want to answer is: “What is the probability that a particular intervention can be sufficiently cost-effective to compete with the most cost-effective GHW interventions currently supported by effective altruists (EAs)?”
Our focus is on calculating the MARGINAL cost-effectiveness (MCE) of an intervention under the key assumption that if the EA movement maximizes the MCE of donations for charities working in the GHW cause area, then this will maximize the good that EA donors can do for GHW given limited available resources.
We define the marginal cost-effectiveness to be the inverse of the marginal cost per unit of impact (MCI) which is:
MCI = Cd / NBs
Where Cd is the cost to donors per unit of intervention, and NBs is the Net benefit of the subsidized intervention that can be attributable to the impact of donor financing.
Because MCI and MCE are inversely related minimizing MCI, maximizes MCE.
Note that when donations enable access to a new or higher quality product or service to become available for a population, this is solving a very common market failure. Typically access is enabled by a partial subsidy of a higher quality version of a product or service and without the subsidy the particular product or service may not be available at all in a particular market. This can happen when there are “asymmetric information” market failures in a market, which are very common in lower income countries. See for example: “Market for Lemons” …
When a market-for-lemons market failure occurs, low-benefit products “crowd out” higher quality, high-benefit products. Targeted subsidies lower the price of higher-quality products and the lower price for higher quality products allows higher-benefit products to enter the market and compete vis-a-vis cheaper, lower-benefit alternatives.
In this context, Cd in the equation for MCI represents the cost of subsidizing higher-quality products and services, while NBs represents the net increase in benefit of the higher quality product or service relative to the lower-quality alternative.
Target thresholds for MCI and MCE:
Different EA meta-charities define a variety of CE thresholds when they evaluate charities and interventions to determine of they are cost-effective.
Some of the different CE thresholds within the EA GHW community are as follows:
~$5000 per life saved
~$125/DALY, (Assuming an average of 40 DALY averted per life saved)
6X to 10X as cost effective as cash transfers
$50/DALY
“increasing income for 4 people by ~1% for a year for $1”
Center for Exploratory Altruism Research (CEARCH):
10X GiveWell CE of ~700 DALY/$100k
i.e ~$15/DALY
Details re: the Marginal Cost of Impact estimation
The two inputs to MCI, cost to donors (Cd) and the net benefit of a subsidized product (NBs), are conceptually simple quantities. But there are several factors that can influence their values in practice. Here we discuss some of the details of estimating Cd and NBs in practice.
Estimating Cd
Cd can be somewhat difficult to calculate for the subsidized distribution of products and services because the amount of subsidy versus the amount of customer payment can vary substantially. The beneficiaries of a new product or service will of course want to get that product or service for a price that is as low as possible: preferably for free. But that will not be the scenario in which the intervention is most cost-effective. So a key factor in estimating Cd is estimating the portion of product or service cost that is paid by donations.
In addition, there are administrative, management, and data collection costs associated with administering and implementing philanthropic subsidies. These costs will tend to be additional to the cost of the product or service subsidy.
One way of decomposing Cd into component factors is through the following equation:
Cd = Cprod × Psub × Md
Where Cprod is the cost of the product or service, Psub is the percent subsidy that is necessary to make the product affordable and desirable for the beneficiaries, and Md is the markup factor that represent the costs of administration and data collection for the subsidy donation. For example, if for every $1 of subsidy, there are $0.30 of administrative, data collection and donation marketing costs, then Md = 1.3.
Estimating NBs
Similarly, estimating the net benefits of a new product or service is conceptually simple, but can be somewhat more complicated in practice.
The first key complication is that benefits need to be measured relative to what would have occurred without the subsidy program. While a new and better product may provide clearly measurable benefits, what is harder to measure is what the product users might have done without the subsidized availability of the product.
The second factor that needs to be considered is that a higher-quality product may last many more years than a lower quality product. In this case, both the benefits and the costs of both the intervention and the no-intervention alternative need to be annualized so that they can be compared over a standard time interval of impact. (For example see: https://www.epa.gov/sites/default/files/2017-09/documents/ee-0568-06.pdf for details of how to do this). The annual net benefit of the intervention NBann then becomes the difference between the annualized benefit of the intervention minus the annualized benefit of the alternative in case of no intervention.
A third factor, is that even though a new product or service may provide a net benefit (such as pumping water for a garden, clean water, or household lighting), the beneficiaries may not utilize the full benefits. We characterize this with a utilization factor (UF) that is generally between 0 and 1, depending on how much the average program beneficiary utilizes the full benefits of a subsidized product or service.
And last but not least, the value of the annualized benefits may increase or decrease over time. Thus a discounted sum of a net value trend over the expected lifetime or duration of net benefits can be characterized with the net present value (i.e. discounted sum) of relative annual values.
NBs = [NBann × Fattrib × UF × NPVsum - (1-Psub) × Cprod]
In this equation, NBann is the annualized benefit of the intervention minus the benefit produced by the alternative without the intervention. Fattrib represent the fraction of intervention adoption that can be attributed to donations that are supporting the intervention. UF is the fraction by which beneficiaries actually use the new product or service and actually reap its benefits. And NPVsum is the discounted present value of relative annual benefit factors over the lifetime of the product or serve. If the value of NBann is constant over time, then the relative annual benefit factors are all 1. If the value of the benefit increases 10%/year then [Annual Benefit Factor](year+1) = (1 + 10%) × [Annual Benefit Factor](year), etc. The term (1-Psub) × Cprod represents the portion of the initial product price paid by the beneficiary.
And finally, NBs needs to be converted to standardized EA units such as DALYs, or people-percent-years of income increase in order to make the resulting CE estimates comparable to typical minimum CE donation criteria. Another useful cost-effectiveness metric that we like to use is “dollars of net beneficial impact per dollar donated.”
Accounting for uncertainty and variability
Because the inputs that influence impact cost-effectiveness of an intervention can be both uncertain and variable, the results of a CE calculation is most appropriately provided as a probability distribution.
The standard approach to performing a benefit-cost calculation with variable or uncertain inputs is to perform a Monte Carlo simulation (The Wikipedia page on this topic is quite good: https://en.wikipedia.org/wiki/Monte_Carlo_method). In our CE estimation with uncertain inputs, we implement a highly simplified Monte Carlo method that we call a simplified Monte Carlo or “poor man’s” Monte Carlo calculation.
In our simplified Monte Carlo calculation, we initially estimate ranges for all or most of the input parameters, and represent these ranges by low, median, and high values. Given a probability distribution of what values a parameter may take, the low value represents the average value of the lowest 1⁄3 of possibilities, the median value represents the average of the middle 1⁄3 of probable values and the high value represents the average of the largest 1⁄3 of probably values. This approximates a probability distribution of possible input parameter values by three discrete values of equal probability.
Once all of the input parameters are represented by three values of equal probability, then the CE result is calculated for all combinations of input parameters. If each of the input parameters are independent and uncorrelated, then the set of CE values that result from all combinations of inputs all have equal probability. A histogram of the full set of CE results is then constructed to illustrate the full range of possible CE values and their respective approximate probabilities.
We illustrate this simplified Monte Carlo calculation below.
Example calculations of CE distributions
Example calculation for long-lasting solar lights
To illustrate the CE estimation methodology above, we perform example calculations for nascent interventions that Solar4Africa.org implements in rural Malawi. One such intervention is the subsidized distribution of small solar lighting systems with cell phone charging to households in rural villages. The solar lighting system uses a newer, slightly more expensive battery chemistry (i.e., lithium titanate) which has cycle-life that is approximately ten times the cycle life of standard lithium ion batteries which are commonly used in solar lights in throughout rural Africa
Here we calculate the MCE for subsidized long-lasting solar lights, using the equations specified above:
MCE = NBs/Cd
= [NBann×Fattrib×UF×NPVsum-(1-Psum)×Cprod]/[Cprod×Psub×Md]
The ranges of parameter values for the calculation are specified in Table 1. The system costs approximately $30 installed though depending on assembly and distribution efficiency, the installed cost can range from $20 to $40. Customers pay for the lighting system, but pay a subsidized price. The annual benefits of the solar lighting system consists of not having to pay for disposable batteries for a battery-powered light and not having to pay for cell phone charging at a local trading center. These savings can range from $2/month to $5/month or $24 to $60 per year of savings. Impact attribution is less than 100% because it is possible for some households to buy alternative solar systems from other providers, and some households may not use the system for its entire lifetime. Because of the very long battery life, the system can last from 5 to 15 years so the discounted NPVsum ranges from 4.5 to 11 for a discount rate that may vary between 2%/year to 6%/year.
Table 1: Input parameters for subsidized long-lasting solar lights cost-effectiveness
| Variable | Low Value | Medium Value | High Value |
| Cprod | $20 | $30 | $40 |
| Psub | 20% | 35% | 50% |
| Md | 1.2 | 1.35 | 1.5 |
| NBann | $24 | $40 | $60 |
| Fattrib | 30% | 60% | 90% |
| UF | 40% | 60% | 80% |
| NPVsum | 4.5 | 8 | 11 |
The results of the simplified Monte Carlo calculation is shown in Table 2 for the full set of 37 = 2187 input parameter combinations.
For this intervention, the most probable cost-effectiveness is approximately $8 of benefit per $1 donated. If these benefits accrue to people who have a per-capita income of $200/year, then this corresponds to the poverty-reduction cost-effectiveness that is equal to the threshold stated by Open Philanthropy of “increasing income for 4 people by ~1% for a year for $1.”
Table 2: Results of Monte Carlo calculation for subsidized long-lasting solar lights
| $ Benefit/$ Donated | Frequency |
| <0.7 | 178 |
| 1 | 156 |
| 2 | 306 |
| 4 | 472 |
| 8 | 545 |
| 16 | 348 |
| 32 | 156 |
| 64 | 25 |
| 128 | 1 |
MCI estimate for subsidized mosquito trap
A similar calculation can be performed for subsidizing solar powered mosquito traps to reduce the incidence of malaria in rural Malawi. Mass mosquito trapping is a malaria vector control method that has shown some promise, but which is not yet widely deployed. Solar4Africa.org has found that small DC fans can be used to make a very simple, low-power mosquito trap than can kill hundreds of mosquitos per night, even without using odor bait.
Here we use our CE estimation methodology to estimate the marginal cost of impact of this intervention in units of $/DALY, assuming that the simple traps cost about $20 each, that the subsidy ranges from 50% to 100%, and that a baseline Malaria burden of disease of a family of 4 of 0.2 DALY/year can be reduced between 10% and 50%.
The equation for the marginal cost of impact is:
MCI = Cd / NBs
= [Cprod×Psub×Md ] / [NBann×Fattrib×UF×NPVsum]
Because the benefits are measured in units of DALY of disease impact avoided, we do not subtract the consumer cost of the mosquito trap. This is equivalent to saying that if the user is paying for a portion of the trap cost, then they will get other benefits from the trap such as greater comfort from fewer mosquito bites.
Table 3 provides the input parameter values used in constructing a simplified Monte Carlo calculation of MCI.
Table 3: Input parameters for subsidized solar-powered mosquito traps
| Variable | Low Value | Medium Value | High Value |
| Cprod | $15 | $20 | $25 |
| Psub | 50% | 75% | 100% |
| Md | 1.3 | 1.6 | 2.0 |
| NBann (DALY/yr) | 0.02 | 0.05 | 0.10 |
| Fattrib | 60% | 80% | 100% |
| UF | 50% | 75% | 90% |
| NPVsum | 2 | 4 | 6 |
Table 4: Results of Monte Carlo calculation of marginal cost of impact for subsidized malaria mosquito traps
| $/DALY | Frequency |
| 20 | 11 |
| 40 | 124 |
| 80 | 371 |
| 160 | 553 |
| 320 | 550 |
| 640 | 377 |
| 1280 | 164 |
| 2560 | 37 |
Table 4 shows the distribution of results arising from the simplified Monte Carlo calculation of MCI. About half of the time the intervention is not cost effective relative to the GiveWell threshold of $150/DALY averted. Compared to the Open Philanthropy and CEARCH criteria of $50/DALY and $15/DALY, the intervention is not cost-effective most of the time. But we note that the intervention might be cost-effective relative to the more stringent criteria if the mosquito trapping device can be made to last a long time, if it can have a relatively large impact on vector populations, and if it is deployed in areas with a high burden of Malaria disease.
Summary/Conclusion
In the Fall-23 EA student projects being conducted by Solar4Africa.org we are searching for new solar-powered interventions that can be cost-competitive with some of the most cost-effective GHW interventions that are supported by EAs.
We have developed generic marginal cost-effectiveness equations and a simplified Monte Carlo technique which allows the calculations of a probability distribution of results for either marginal cost-effectiveness or the marginal cost of impact. These methods allow the estimation of the probability that new interventions with uncertain inputs can reach different cost-effectiveness threshold criteria.
Our hope is that this will aid in the development of new, highly cost-effective and a greater variety of easy-to-implement interventions that can help the EA movement produce greater GHW progress given the limited EA donation resources that are available now and in the near-term future.
- Solar4Africa Project 2: Small solar cars for non-emergency health services access — Preliminary cost effectiveness analysis by (24 Dec 2023 17:06 UTC; 13 points)
- Solar4Africa Project #4 - General Intervention Refinement and Cost-Effectiveness (CE) Analysis of Solar Electric Cooking Systems: Final Analysis and Justification of Intervention by (20 Jan 2024 5:46 UTC; 10 points)
- Solar4Africa Project 3: Solar pumps for income generation–Project overview and methodology by (19 Nov 2023 14:07 UTC; 5 points)
Thanks for sharing this work! It’s very interesting to see how you do this and good to see Monte Carlo in action.
The cost-effectiveness of subsidised solar lights comes out as very high, which invites scrutiny. If solar lamps are such a good deal, why don’t people buy them at full price (in which case cash transfers would be better)? Lack of cash on hand could be a barrier, but a 35% subsidy would presumably only make a difference for a small subset of people.
You clearly signpost this as a “simplified” calculation, so this is slightly nitpicky. A few things that may be overly optimistic in the model:
NBaum is based on the amount households would spend on batteries etc. Is this really what people would do? Seems crazy that people would pay $40/year or 20% of one person’s income on batteries. Also, it seems likely that there’s a strong correlation between income and amount spent on batteries and other lighting sources. The poorest people may be more likely to burn wood for light or spend longer in the dark, and hence would save less money by having a solar lamp
This $200/year income is pivotal to your cost-effectiveness, so ideally this should be modelled with uncertainty in the Monte Carlo. There may be correlational issues here also, where the people who can afford the subsidised lamp are less likely to be among the poorest.
Fattrib seems high. Could the subsidies really increase the chance of uptake by 30-90%?
You don’t model the out-of-pocket cost to the recipient (as far as I can tell)
NPVsum of 4.5 − 11 may also be high. Do the lamps have a track record of lasting a long time, especially in low-tech areas? In a more complex model I would also seek to account for the chance that the lamp becomes obsolete, for example if the area gets access to electricity. Electrification in Africa has been slow, but there is at least a push to expand access.
You’re welcome to interpret my questions as hypothetical—no need to fully justify everything about your organisation!
I’d be willing to help you build this into a slightly more complex model in Dagger if you like.
Thanks again for sharing.
Hi Stan: Thanks for the great comments and questions.
You are completely correct on noting that I need to subtract the cost that the customer pays from the present value net benefits. I have edited the post to incorporate that correction.
BUT the benefits of the solar lights is still very high, and therefore it continues to invite scrutiny.
Trying to address your other points:
THE key reason, the solar lights from our intervention produce benefits that are so much higher for other solar lights, is because, we use a battery technology that is currently not used at all in solar lighting systems because it is quite a bit more expensive than other battery technologies. The technology is lithium titanate battery chemistry. And the cycle-life of that battery tech is about 10 TIMES longer than lithium ion batteries and lead-acid batteries which dominate the market.
If you look at the “Rechargeable characteristics” at:
https://en.wikipedia.org/wiki/Comparison_of_commercial_battery_types
You will see that regular lithium ion which is listed as lithium cobalt-oxide has a cycle life of 500 to 1000 cycles, while lithium titanate has a cycle life of 6,000 to 10,000 cycles.
BTW, we are currently installing subsidized small solar lighting systems to about 1000 to 2000 household per year.
The much much longer cycle life allows for a much more long-lasting solar light, which is what produces the huge benefit. One expects the market to normally equilibrate where the benefits approximately equal the cost. So perhaps it is not surprising if you introduce a technology into the market which lasts 10 times longer than the current market equilibrium, and then subsidize the cost so that the price is about the same as the low-quality technology, then the net benefit is 8 times the cost of the old technology, which is what our analysis indicates.
As for people getting access to electricity and making the lights obsolete. In Malawi, where we work, this is pretty unlikely. Many people do have power lines nearby, but people use so little electricity, and the cost of connecting is so high, that very few people connect to the grid. And the electric company does not like connecting folks and may take over a year to satisfy a request for a connection even when someone pushes hard to have one. That is because the cost of connecting and serving low-use customers to the grid is subsidized, and the national electric company is often having budget problems.
Looking at World Bank statistics, perhaps this dynamic is working at a larger scale in Africa. For example, if we look at the World Bank data for electricity access in rural SubSaharan Africa (SSA):
https://data.worldbank.org/indicator/EG.ELC.ACCS.RU.ZS?locations=ZG
We see that access has increased from 12% to 30% from the year 2000 to 2021 in rural SSA. I think it is safe to say that more than half of rural SSA will still be largely without electricity access by 2030.
This is evidence that it is fairly likely that hundreds of millions of rural Africans will be able to benefit from more beneficial off-grid solar lighting for several decades to come.
Addressing some of the other points.
Re: the amount people spend on batteries and cell phone charging. We have done surveys. See:
https://www.researchgate.net/publication/325131559_Preliminary_impact_analysis_Kuyere_solar_system_distribution_MChinji_Malawi
I have discussed with people in village in Malawi how crazy it is that they spend so much on batteries. This argument makes it pretty easy to market the subsidized solar systems. But people live very hand-to-mouth in rural Malawi. So they will spend to buy something every week rather than invest to reduce a cost if the investment takes more than six months or a year to pay back.
I agree that the $200 income should be modeled with the Monte Carlo. But we can also target the subsidies to areas where incomes are lower. So the analysis indicates in my mind that the subsidies should be targeted to households that have incomes of $200 or less.
We do need to do more work assuring that the lighting systems are guaranteed to last 5 to 15 years, but from an engineering perspective, there is no reason the system cannot last that long given the battery we are using. The remaining engineering issue is going to be how to make sure all of the other parts are very durable, or very cheap and easy to replace.
Thanks again. I hope that answers most of your questions and concerns.
Thank you for the very thorough reply!! Really interesting.