Extraordinary cost-effectiveness analyses call for supporting theory
What needs to be true for an estimate to be reasonable?
Pure Earth is a GiveWell grantee that works to reduce lead and mercury exposure. In an August 2023 post, they provided a “preliminary analysis” suggesting that their lead reduction program in Bangladesh “can avert an equivalent DALY for just under $1.” By contrast, they estimate that GiveDirectly “has a cost-effectiveness of approximately $836 per DALY-equivalent averted.”
About 86% of the money GiveDirectly spends goes directly to recipients, so an $836 donation to GiveDirectly results in about $719 going directly to a very poor person. In effect, Pure Earth is claiming that either GiveDirectly can give a person in poverty $719, or Pure Earth can spend $1 helping people in Bangladesh, and these would be about equally good for human welfare.
Pure Earth calls this an “extraordinary result” and forthrightly identifies places where their analysis might go wrong. But what I’m missing is a sense of why their analysis might be right—a story about why this extraordinary opportunity exists. In other words, theory.
I find this to be a pretty common lacuna in effective altruist CEAs.[1] Our explanations tend to be technical but not contextual. They enumerate our assumptions about the world at large but lack granularity about social and political conditions. But for my tastes, any claim that that boils down to ‘we can help poor people more effectively than they can help themselves’ requires theoretical buttressing in the form of a plausible and specific story.
What kinds of theories might suffice?
In the case of lead reduction in spices, six come to mind.
Experts have a comparative advantage at helping poor people manage their affairs. (Here is some evidence against this theory.[1])
There is extraordinary political gridlock that an external organization is especially well-suited to solving. (Pure Earth says that the “project’s impact lies not in identifying and enforcing food safety regulations, but rather in expediting its implementation by several years.”)
There is a narrow opportunity for positive impact stemming from deep insight about a particular context.
There is a market failure, e.g. a collective action problem, where a push from an NGO can create a self-sustaining equilibrium.
There is a cognitive bias and/or cultural failure that leads people to undervalue something that is good, or overvalue something that is bad, and they need a push in the right direction.
Unconditional cash transfers engender negative spillovers, whereas public health interventions typically have positive spillovers.
The story will vary from case to case. For New Incentives, I assume it’s some version of theories 4, 5, and 6. For anti-malarial interventions, I am not sure (and am generally a skeptic). For Pure Earth, I am also not sure, because I am not an expert. The supporting story would need to be extraordinary, because the claim is. I can potentially back that story out of a spreadsheet. But I’d prefer to hear it directly.
^ I use Pure Earth to illustrate because their CEA was on the far tail of the distribution, so it stuck in my memory, but this point could be made about many CEAs on the forum. As far as I can tell, Pure Earth is doing great work on an important, neglected, and tractable issue.
^ Framed differently, what would it take for you to accept the same thing for yourself—that an organization is hundreds of times better at helping you than you are at helping yourself?
Most of the interventions from Pure Earth is political, right? So I’d guess that #4 is correct. I think it should be relatively straightforward why rational selfish actors are not incentivized to make political interventions for their own interests:
Suppose you have V voters. As V increases, the total benefit of a good political intervention grows proportionally.
However, the benefit to each voter for the intervention succeeding remains constant, regardless of V.
The probability of an individual vote changing the outcome decreases as V increases (roughly 1/V).
From a selfish standpoint, as V grows larger, the incentive for an individual to be well-informed decreases.
Sure, there is an intuitive plausibility to this. But how extraordinary must the political dysfunction be for no one within Bangladesh to be capable of solving this themselves through political agitation without NGO support? If the DALY calculations are to be believed, the potential gains are enormous and comparatively cheap. As an outsider to the situation, I am looking for context on why this hasn’t happened yet. A good theory in the social sciences will abstract from specifics of the situation, or map a theory onto specific things that have happened (or that haven’t), rather than making the general observation that collective action problems cause some public goods to be underprovisioned.
You might find this article helpful for context: https://undark.org/2023/07/19/the-vice-of-spice-confronting-lead-tainted-turmeric/
Fwiw I’m sympathetic to your general point
Thanks, I think this was featured on Marginal Revolution last year — definitely good background.
Hmm I guess I wouldn’t be that surprised if we observed similar levels of what you call “dysfunction” in the US. Earlier you asked:
I guess the intuitive plausibility of this is rather low (or perhaps I have an overly high opinion of myself) if the problem is framed as one of my own rationality, but I can much more easily buy that there are collective action problems that benefit “people like me” at >100x the costs.
That seems like a particularly cynical way of describing this argument. Another description might be: Individuals are on average fine at identifying ways to improve their lives, and if you think life improvements are heavy tailed, this implies that individual will perform much less well than experts who aim to find the positive tail interventions.
Here’s a similar situation: A high school student is given 2 hours with no distractions and told they should study for a test. How do you think their study method of choice would compare to if a professional tutor designs a studying curriculum for them to follow? My guess is that the tutor designed curriculum is somewhere between 20% and 200% better, depending on the student. Now that’s still really far from 719x, but I think it’s fine for building the intuition. I wouldn’t necessarily say the student is “bad at managing their own affairs”, in fact they might be solidly average for students, but I would say they’re not an expert at studying, and like other domains, studying benefits from expertise.
Sure, there are more reasonable ways to express the argument, all of which boil down to “experts have a comparative advantage at producing welfare gains.” And the people don’t need to be poor for this theory to be in vogue, e.g. American public health officials giving very nuanced guidance about masks in March-April 2020 because they were looking to achieve a second-order effect (proper distribution). I think my broader point, however, about the necessity for such a theory, regardless of how it is expressed, holds. I went with a cynical version in part because I was trying to make a point that a theory can be a ‘folk theory’ and therefore impolite, but elucidating that was out of scope for the text.
I amended the text to be less inflammatory
Hi Seth,
GiveWell’s 2021 analysis of Pure Earth links to a sheet according to which its cost-effectiveveness is 18 times that of unconditional cash transfers.
I estimated corporate campaigns for chicken welfare have a cost-effectiveness of 15.0 DALY/$.
Thank you for the additional context!
re: Pure Earth: GiveWell notes that its Pure Earth estimates are “substantially less rigorous than both our top charity cost-effectiveness estimates,” so I don’t want to read too much into it. However, a claim that an intervention is merely 18X better at helping poor people than they are at helping themselves still strikes me as extraordinary, albeit in a way that we become acclimated to over time.
As to what good social theory would look like here, there is some nice work in sociology on the causes and consequences of lead exposure in America (see Muller, Sampson, and Winter 2018 for a review). I don’t expect EA orgs to produce this level of granularity when justifying their work, but some theory about why an opportunity exist would be very much appreciated, at least by me.
I’ve followed your work a bit w.r.t. animal welfare. That’s 15 chicken DALYs right? That seems plausible to me. The theory I would construct for this would start with the fact that there are probably more chickens living on factory farms at this moment than there are humans alive. Costco alone facilitates the slaughter of ~100M chickens/year. If you improve the welfare of just the Costco chickens by just 1% of a DALY per chicken, that’s 1M DALYs. I could very much believe that a corporate campaign of that magnitude might cost about $66K (approximately 1M/15). So I find this claim much less extraordinary.
You are welcome!
For reference, Pure Earth’s estimate of 1 DALY/$ is 5.59 (= 1/(18*0.00994)) times GiveWell’s estimate.
Thanks!
No, it is 15.0 “normal” DALY/$, i.e. 1.51 k times (= 15.0/0.00994) as cost-effective as GiveWell’s top charities.
Thanks for clarifying. That inevitably rests on a strong assumption about the relative importance of chicken welfare to human welfare, and it looks like your work builds on Bob Fischer’s estimates for conversion. That’s a fine starting point but for my tastes, this is a truly hard problem where the right answer is probably not knowable even in theory. When I’m discussing this, I’ll probably stick to purely empirical claims, e.g., “we can make X chickens’ lives better in Y ways” or “we can reduce meat consumption by Z pounds” and be hand-wavy about the comparison between species. YMMV.
Yupe, I relied on Rethink Priorities’ (RP’s) median welfare range of chickens of 0.332. However, even for their 5th percentile of 0.002, which is 0.602 % (= 0.002/0.332) of their median, corporate campaigns for chicken welfare would be 9.09 (= 0.00602*1.51*10^3) times as cost-effective as GiveWell’s top charities. Uncertainty in other variables besides the welfare range means there might be something like a 5 % chance of corporate campaigns for chicken welfare being less cost-effective than GiveWell’s top charities, but I believe we should be comparing the expected cost-effectiveness of both interventions, not a worst-case scenario of corporate campaigns with the expected scenario of GiveWell’s top charities.
Even if it is not knowable in theory[1], trade-offs are inevitable, so our actions implicitly attribute a given welfare range to chickens. So I would say we might as well rely on the best empirical estimate we have from RP instead of our vague intuitions.
I think it is, as I strongly endorse expected total hedonistic utilitarianism.