Critiquing GiveWell's Model of Economic Effects from Health Interventions

I would like to thank Isobel Phillips for providing helpful comments and feedback on an earlier draft. All mistakes that remain are my own.

In GiveWell’s cost-effectiveness models for health interventions, an important factor in the analysis is how much the intervention increases beneficiaries’ incomes later in life, likely through increased ability to work or productivity. In some analyses, this can be a considerable factor with it accounting for around 30% of the Against Malaria Foundation’s impact, as well as being the crucial factor in their past recommendations of deworming charities. While this is both reasonable to account for and based upon solid evidence, the way these effects are currently assessed for new interventions is likely to systematically underestimate the developmental benefits from understudied early life health interventions due to their model having inappropriate priors. Specifically, their model assumes a null effect, or something close to it, for effective early life health interventions on later life incomes and proxy indicators in the absence of direct academic evidence, despite there being a wealth of evidence that effective health interventions almost always affect income later in life.

Additionally, GiveWell’s modelling could also be improved by allowing for interventions affecting health capital to have positive and negative economic spill-overs to untreated populations. As spill-overs from health capital at the macro-level are often estimated to be at least as big as the micro-level direct effect at the macro level (Bloom et al 2019), the direction and magnitude of these spill-overs could affect the extent to which GiveWell chooses to recommend interventions on the basis of developmental effects. Indeed, GiveWell recognises this as an issue by applying an ad hoc general equilibrium adjustment to down-weight the effects of deworming and it is unclear why this is not applied to the development effects of other interventions, as similar mechanisms are likely to be at play.

GiveWell’s Development Effects Model

The current GiveWell development effects model assigns each intervention a score of 0 to 4, based on the strength of the evidence and the relevant effect sizes, to several categories that are correlated with or directly underlie consumption later in life and then constructs a weighted average of these scores. Then, the score is compared to the average for malaria treatment (2.2 in figure 1) for which there are two well-identified econometric studies providing direct evidence (Bleakley 2010; Cutler et al 2010). In figure 1, you can see this model being implemented, with New Incentives’ vaccinations—an intervention for which they have less direct evidence—being estimated to be half as impactful as seasonal malaria chemoprevention (SMC). The full explanation of their model is available here.

Figure 1

Source: GiveWell Development Effects Model—New Incentives

This model’s fundamental weakness is that it treats the strength of the evidence on multiple benchmarks as independently valuable pieces of evidence to be combined via calculating a weighted average. However, these assumptions are highly unlikely to be correct as the variables themselves are not independent. This is most obvious for interventions with a large body of convincing evidence on the direct effects, the outcome of interest, in which case the other information is of no additional use. Another potential failure mode, where GiveWell is more likely to apply this method, is that lesser studied health interventions are penalised for having no studies of decent quality within specific categories. In practice, this means that an early life health intervention – a category all GiveWell’s top charities lie within –with reliable effects on mortality is unlikely to be considered more than 27.5% as effective (through the mortality, morbidity, and plausibility categories) as malaria treatment at achieving developmental effects. As a wide range of factors that affect early life mortality also affect later life incomes in the same direction as malaria treatment, implicitly assuming that the developmental effects will be significantly smaller is a strong assumption that requires investigation. Some examples of early life health shocks that both cause early life mortality and reduce later life incomes include pneumonia (Bhalotra and Venkataramani 2015), waterborne disease (Beach et al 2016), and air pollution (Isen et al 2019). Furthermore, if we extend the search to allow for proxies for later life incomes such as height, weight, and cognitive performance as well as shocks that occur in-utero the number of relevant health shocks expands rapidly (see Almond, Currie, and Duque 2018 for a review). Consequently, it is no surprise that quasi-experimental evidence, using difference-in-differences, has since emerged suggesting that receiving the same set of vaccinations encouraged by New Incentives increased the incomes of recipients in India by 14% (Summan, Nandi, and Bloom 2022) and that the introduction of the measles vaccine alone had an effect of 1% in the US (Atwood 2022).

Considering this evidence, I believe that the best way forward would be for GiveWell to examine the relevant economic literature to form a prior probability on how early life health interventions’ effects on mortality are likely to translate to developmental benefits. Then, movements can be made away from this prior in a Bayesian manner if specific unexpected evidence, direct or indirect, emerges that warrants it.

General Equilibrium

Studies identifying the effect of an early life health shock typically do so relative to a control group of individuals who were not exposed to the health shock within the same region a few years beforehand. Therefore, if the shock has spill-over effects onto the earnings of those unexposed to the shock within the same region these studies will not accurately capture the full developmental effects of that intervention. These spill-overs could even affect the policy recommendations from studies such as Bleakley (2010) that conducts its analysis at the ten year cohort level if spill-overs occur between cohorts^[1]. As some of the income effects these health improvements come from increased cognitive ability and education (Atwood 2022) , many of these mechanisms will be the same as for human capital Commonly proposed mechanisms for spill-over effects from human capital include knowledge and idea spill-overs, the signalling model of education, changes in relative factor prices, shifts in investment behaviour, and political economy effects^[2]. In addition, better health is likely to affect population size and fertility rates which will have their own effects on incomes.

Indeed, these effects could be practically relevant, with macroeconomic attempts to identify the effects of health capital on growth in GDP per capita frequently finding effects over twice as large as the corresponding microeconomic estimates or spill-overs that reduce them to zero depending on the context and assumptions (Bloom et al 2019). Although I do not currently have the time or training to properly assess this macroeconomic literature, large spill-overs are very plausible in developing countries, especially given what we already know about human capital spillovers that share some common mechanisms. The most direct evidence of human capital spill-overs in the developing world thatI am aware of is Khanna (forthcoming)’s assessment of an Indian school building program exploiting sharp cut-offs in a district’s eligibility for the program to achieve quasi-experimental variation. Using this cut-off to learn about the program’s effect on the proportion of workers achieving more education as well as its effect on the wages of more and less educated workers separately, Khanna (forthcoming) is able to estimate that general equilibrium spill-overs reduced the program’s impacts on income by over 30%, primarily by reducing the wages of workers who would have received an education regardless. These effects were partially, but not fully, offset by an increase in wages for previously unskilled workers. Although it is impossible to identify the exact mix of different spill-overs present here, these results are certainly consistent with a relative factor prices explanation where the skill premium decreases as human capital increases. Similarly, Angrist (1995) found that the skill premium in Palestine decreased after a large school-building program was undertaken but, unlike Khanna (forthcoming), did not develop a model to separate out the different welfare effects.

Pertinent evidence of potential negative spill-overs also comes from Barton et al (2017) identifying that public sector teaching jobs in Kenya come with rents equivalent to a doubling of income using a cut-off in the hiring algorithm as a regression discontinuity. As Kenya is not an outlier within Africa for its permanent teachers being well-paid relative to its economy (Bold et al 2017) and achieving an education is necessary for acquiring this job, it is plausible that some of the productivity gains from health improvements are directed towards zero-sum competition for rents from rationed civil service jobs. A final source for which there is some evidence of negative spill-overs from health capital is that saving lives can depress wages by reducing the capital stock per worker, with Cervellati and Sunde (2011) finding that decreases in mortality are associated with reduced economic growth within countries that have not undergone the demographic transition.

Alongside potential negative effects, positive spill-over effects could also occur from investments in health capital. For example, they could free up time for mothers who would have to spend less time pregnant to reach their desired family size and also have to spend less time caring for the sick. Additionally, positive spill-overs could also occur through a human capital channel as ideas are non-rival and spreading them can enhance productivity. Some examples of knowledge spill-overs include the wages of native workers increasing in cities that receive highly skilled STEM workers (Peri, Shih, and Sparber 2013) and inventors increasing their productivity when they move to cities with many other specialists in their field (Moretti 2021). Although some of this is unlikely to transfer to developing countries, it is still possible that some spill-overs could occur as important knowledge may not be held by others within a firm, or other workers may induce more effort to keep up with more productive peers. Some examples of these factors being relevant include a randomised field experiment showing that job training increases productivity among the worker’s peers (De Grip and Sauermann 2012) and supermarket checkout workers increasing their output when more productive workers come on shift (Mas and Moretti 2009).

As discussed above, there are several ways in which increases in one person’s health capital could affect another’s earnings, with these effects potentially being large relative to the individual-level microeconomic effects. Thus, I believe it would be beneficial for GiveWell to perform a more comprehensive survey of the evidence to reach an informed conclusion on this topic, given its importance to their recommendations. Personally, my suspicion is that this would likely cause developmental effects to be down-weighted by their models, as potential sources of negative spill-overs seem particularly acute within developing countries and positive ones seem likely to be attenuated. Nevertheless, a firm conclusion would require greater examination of the relevant macroeconomic evidence.

Conclusion

In sum, I believe that GiveWell’s development effects models contain two implicit assumptions that materially affect their recommendations and may not be fully accurate. The first assumption is that understudied early life interventions effective in reducing mortality will have weaker developmental effects than a malaria treatment saving the same number of lives, which may prove untrue upon a systematic analysis of the relationship between early life health shocks and later life earnings. This assumption could cause GiveWell to recommend more heavily studied early life interventions over lesser studied ones, even when a rational bayesian would expect the lesser studied one to be more cost-effective. The second assumption is that higher earnings from increased health capital for one individual does not affect those of others. Assuming a lack of spill-overs could cause GiveWell to over or underestimate the cost-effectiveness of interventions where a significant portion of their impact comes through development effects, depending on whether the spill-overs are negative or positive respectively.

Both of the issues outlined are solvable. The first can be solved by forming an appropriate prior about the development effects of an effective early life health intervention from the literature and then adjusting from it when more direct information on the intervention becomes available. Meanwhile, the second can be solved by assessing how likely spill-overs are to occur in a given context by weighing a mixture of micro and macroeconomic evidence.

References

Acemoglu, D., 2010. Theory, general equilibrium, and political economy in development economics. Journal of Economic Perspectives, 24(3), pp.17-32.

Almond, D., Currie, J. and Duque, V., 2018. Childhood circumstances and adult outcomes: Act II. Journal of Economic Literature, 56(4), pp.1360-1446.

Angrist, J.D., 1995. The economic returns to schooling in the West Bank and Gaza Strip. The American Economic Review, pp.1065-1087.

Atwood, A., 2022. The Long-Term Effects of Measles Vaccination on Earnings and Employment. American Economic Journal: Economic Policy, 14(2), pp.34-60.

Barton, N., Bold, T. and Sandefur, J., 2017. Measuring Rents from Public Employment: Regression Discontinuity Evidence from Kenya. Center for Global Development Working Paper 457.

Beach, B., Ferrie, J., Saavedra, M. and Troesken, W., 2016. Typhoid fever, water quality, and human capital formation. The Journal of Economic History, 76(1), pp.41-75.

Bhalotra, S.R. and Venkataramani, A., 2015. Shadows of the captain of the men of death: Early life health interventions, human capital investments, and institutions. Human Capital Investments, and Institutions (August 8, 2015).

Bleakley, H., 2010. Malaria eradication in the Americas: A retrospective analysis of childhood exposure. American Economic Journal: Applied Economics, 2(2), pp.1-45.

Bloom, D.E., Canning, D., Kotschy, R., Prettner, K. and Schünemann, J.J., 2019. Health and economic growth: reconciling the micro and macro evidence (No. w26003). National Bureau of Economic Research Working Paper 26003.

Bold, T., Filmer, D., Martin, G., Molina, E., Stacy, B., Rockmore, C., Svensson, J. and Wane, W., 2017. Enrollment without learning: Teacher effort, knowledge, and skill in primary schools in Africa. Journal of Economic Perspectives, 31(4), pp.185-204.

Cervellati, M. and Sunde, U., 2011. Life expectancy and economic growth: the role of the demographic transition. Journal of Economic Growth, 16(2), pp.99-133.

Cutler, D., Fung, W., Kremer, M., Singhal, M. and Vogl, T., 2010. Early-life malaria exposure and adult outcomes: Evidence from malaria eradication in India. American Economic Journal: Applied Economics, 2(2), pp.72-94.

De Grip, A. and Sauermann, J., 2012. The effects of training on own and co‐worker productivity: Evidence from a field experiment. The Economic Journal, 122(560), pp.376-399.

Isen, A., Rossin-Slater, M. and Walker, W.R., 2017. Every breath you take—every dollar you’ll make: The long-term consequences of the clean air act of 1970. Journal of Political Economy, 125(3), pp.848-902.

Khanna, G., forthcoming. Large-scale education reform in general equilibrium: Regression discontinuity evidence from India. Journal of Political Economy.

Mas, A. and Moretti, E., 2009. Peers at work. American Economic Review, 99(1), pp.112-45.

Moretti, E., 2021. The effect of high-tech clusters on the productivity of top inventors. American Economic Review, 111(10), pp.3328-75.

Peri, G., Shih, K. and Sparber, C., 2015. STEM workers, H-1B visas, and productivity in US cities. Journal of Labor Economics, 33(S1), pp.S225-S255.

Romer, P.M., 1994. The origins of endogenous growth. Journal of Economic Perspectives, 8(1), pp.3-22.

Summan, A., Nandi, A., Bloom, D.E., 2022. A shot at economic prosperity: Long-term effects of India’s childhood immunization program on earnings and consumption expenditure. National Bureau of Economic Research Working Paper 30173.

^
I am aware that Bleakley (2010) argues spill-overs do not appear to have a large effect on his results for early 20^th Century Latin American economies in the early 20^th Century, but modern developing economies are substantially different and the effects may differ.
^
See Acemoglu (2010) for a review of these general equilibrium concepts and their importance. Meanwhile, the role of idea and knowledge spill-overs in economic growth is reviewed by Romer (1994).

Critiquing GiveWell’s Model of Economic Effects from Health Interventions