The epistemic challenge: This raises a real question about EA’s framework. Should we only fund interventions we can measure with near-certainty? Or should we have some capacity for high-uncertainty, high-potential-impact interventions during acute emergencies?
I think most people would say that the analysis should be close to risk-neutral. However, global-health donors seem more risk-averse in practice.
That being said, I would submit that we probably should penalize early-stage research and cost-effectiveness analysis, not based on risk tolerance per se but because experience teaches that effectiveness often goes down as analytical rigor goes up. To analogize to a different domain, lots of drugs look great in early trials but fall apart in late-stage trials. So I think that the necessary showing is probably this: is there a substantial probability that the cost-effectiveness significantly exceeds the counterfactual use of the money (which I will assume to be GiveWell All Grants?)
GiveWell has made malnutrition grants, such as this one. The estimated cost-effectiveness was somewhat less than its usual bar (8x, as opposed to 10x, would have been 10x absent funging adjustment). This appears to be a program for extremely malnourished young children, as evidenced by a cost of $215 per child. I’m not qualified to say what the sweet spot for combating malnutrition is (e.g., whether a program for somewhat less malnourished young children might be more cost-effective because it could use less specialized foods, or whether the extra costs of feeding a larger population predominate.) On the other hand, if our starting point is that young-child extreme malnutrition programs are close to the bar, then it seems likely that programs for mild-to-moderate malnutrition and adult malnutrition probably wouldn’t clear the bar. All that is very shallow and out of my domain expertise. But that’s my initial stab at how we might start to bridge the evidentiary deficit here.
We also have pre-existing work (e.g., by GiveDirectly) on the effects of just giving cash to people in poverty, and emerging work that suggests giving the cash at the right time (e.g., shortly before childbirth) has a multiplier effect. Although selecting a multiplier would be dicey here, I would be willing to accept that there’s a multiplier here (and that provision of food and basic medicine is close enough to providing cash in these circumstances to use the cash data). You’d need a large multiple to get to 10x, though.
In any event, I think trying to adapt existing cost-effectiveness estimates to project results for a different context is a reasonable first step here. The projections are going to be error-prone, but I think they could inform whether to invest in more specific work.
There’s another awkward issue here. It’s more likely that ERRs engage in some programs that are more effective than the marginal GiveWell dollar than it is that the marginal dollar given to an ERR outperforms the marginal dollar given to GiveWell All Grants. While recognizing the diversity of ERRs, could we give (e.g.) $1MM dedicated to young-child malnutrition work and predict that ~$1MM more in that work will get done? Or will money get shifted around such that we get (e.g.) $250K more of that, of communal kitchens, of paying those who currently volunteer, and of something else? If the latter, we would need to base the cost-effectiveness estimate off the true marginal effect of the donation.
But there’s a chicken-and-egg problem: we can’t get that evidence without some initial funding, but we can’t get funding without that evidence.
I fear it’s even worse than that. The classic EA global-health model assumes a fairly stable situation: malaria is ~malaria, and usually the world hasn’t changed that much in 5-10 years (and isn’t so different from country to country in a similar area) to render reliance on prior work dicey. By the time you were able to get high-quality results on ERRs, would the situation have changed enough to undermine reliance on that data? How much confidence could we justifiably have that results on ERRs obtained during one crisis would hold for a different crisis in a different country?
In the end, you have to do the best you can with the information you have. But if evidence will become stale quickly and is very context-dependent, that would make me somewhat less excited about spending a lot of resources to gather it.
I think most people would say that the analysis should be close to risk-neutral. However, global-health donors seem more risk-averse in practice.
That being said, I would submit that we probably should penalize early-stage research and cost-effectiveness analysis, not based on risk tolerance per se but because experience teaches that effectiveness often goes down as analytical rigor goes up. To analogize to a different domain, lots of drugs look great in early trials but fall apart in late-stage trials. So I think that the necessary showing is probably this: is there a substantial probability that the cost-effectiveness significantly exceeds the counterfactual use of the money (which I will assume to be GiveWell All Grants?)
GiveWell has made malnutrition grants, such as this one. The estimated cost-effectiveness was somewhat less than its usual bar (8x, as opposed to 10x, would have been 10x absent funging adjustment). This appears to be a program for extremely malnourished young children, as evidenced by a cost of $215 per child. I’m not qualified to say what the sweet spot for combating malnutrition is (e.g., whether a program for somewhat less malnourished young children might be more cost-effective because it could use less specialized foods, or whether the extra costs of feeding a larger population predominate.) On the other hand, if our starting point is that young-child extreme malnutrition programs are close to the bar, then it seems likely that programs for mild-to-moderate malnutrition and adult malnutrition probably wouldn’t clear the bar. All that is very shallow and out of my domain expertise. But that’s my initial stab at how we might start to bridge the evidentiary deficit here.
We also have pre-existing work (e.g., by GiveDirectly) on the effects of just giving cash to people in poverty, and emerging work that suggests giving the cash at the right time (e.g., shortly before childbirth) has a multiplier effect. Although selecting a multiplier would be dicey here, I would be willing to accept that there’s a multiplier here (and that provision of food and basic medicine is close enough to providing cash in these circumstances to use the cash data). You’d need a large multiple to get to 10x, though.
In any event, I think trying to adapt existing cost-effectiveness estimates to project results for a different context is a reasonable first step here. The projections are going to be error-prone, but I think they could inform whether to invest in more specific work.
There’s another awkward issue here. It’s more likely that ERRs engage in some programs that are more effective than the marginal GiveWell dollar than it is that the marginal dollar given to an ERR outperforms the marginal dollar given to GiveWell All Grants. While recognizing the diversity of ERRs, could we give (e.g.) $1MM dedicated to young-child malnutrition work and predict that ~$1MM more in that work will get done? Or will money get shifted around such that we get (e.g.) $250K more of that, of communal kitchens, of paying those who currently volunteer, and of something else? If the latter, we would need to base the cost-effectiveness estimate off the true marginal effect of the donation.
I fear it’s even worse than that. The classic EA global-health model assumes a fairly stable situation: malaria is ~malaria, and usually the world hasn’t changed that much in 5-10 years (and isn’t so different from country to country in a similar area) to render reliance on prior work dicey. By the time you were able to get high-quality results on ERRs, would the situation have changed enough to undermine reliance on that data? How much confidence could we justifiably have that results on ERRs obtained during one crisis would hold for a different crisis in a different country?
In the end, you have to do the best you can with the information you have. But if evidence will become stale quickly and is very context-dependent, that would make me somewhat less excited about spending a lot of resources to gather it.