I research a wide variety of issues relevant to global health and development. I’m always happy to chat—if you think we have similar interests and would like to talk, send me a calendar invite at karthikt@berkeley.edu!
Karthik Tadepalli
The ones that walk away
What do we really know about growth in LMICs? (Part 1: sectoral transformation)
[edited] Inequality is a (small) problem for EA and economic growth
New intervention: paying farmers to not burn crops
Nice to see a rare substantive post in the sea of meta ;)
This is a serious transparency issue
Weighting a strong prior over evidence is a serious decision that GiveWell absolutely should have justified in public documents. I actually think that is a really important point to highlight above and beyond the specifics of this particular CEA. Grantmakers like GiveWell enjoy a really large influence over EA giving and activities, and without strong transparency norms over how they make their decisions, we risk human error amplifying into very large misallocations of resources.
The only reason I could imagine not to share this justification is because it would increase researcher burden from an already-extensive CEA. In the tradeoff between researcher burden and transparency, I think there’s a pretty high bar to prioritize researcher burden, and this does not seem like it meets that bar. I would strongly support GiveWell publishing this justification and making it a norm for all future CEAs to justify exclusion of evidence in a model.
Use prior distributions to shrink estimates, not non-specific discount factors
CEAs commonly use Bayesian hierarchical modelling to estimate true treatment effects from a small number of studies. To get these estimates requires plugging in a prior. If GiveWell thinks that 0.885 log units is unreasonable and 0.113 is more reasonable, then they should formalize that upfront in a prior distribution over the true treatment effect.
Yes, you can still tinker with a prior ex-post so it doesn’t solve the issue completely. Yes, a shrinking of the estimate is technically equivalent to applying a prior distribution that places low weight on 0.885 and higher weights on lower values. That doesn’t mean these are practically equivalent procedures. It’s much easier for motivated reasoning to slip into ex-post adjustments. Moreover, ex-post adjustments are impossible to interpret except in how they change the final estimate, whereas people can interpret priors before the analysis is done and debate whether they are reasonable. So I don’t think non-specific discounting of estimates is a good practice.
However, I think your claim that discounting should only be based on specific factors is too strong. Any reasonable prior over the true treatment effect is discounting some treatment estimates just because of their “unreasonable” magnitude. CEAs aren’t mechanical and shouldn’t be treated as such in the name of following the evidence.
Both your calculations and GiveWell’s should count consumption benefits over the lifetime, not 40 years
40 years may be the working lives of the subjects, but benefits accrued while working will almost certainly increase consumption beyond working age. First, savings accumulated while working will be consumed after retirement. Moreover, higher income while working likely leads children to earn more and thus support their parents better. So consumption benefits in the working life are likely substantially understate the real consumption benefits of any treatment.
I would understand ignoring or substantially discounting consumption benefits to the future children of deworming recipients because of decay—but consumption benefits should still be counted over people’s whole lifetime, say 55 years. I could not find a spreadsheet with your calculations to plug a longer time horizon in, but I would be very curious to see how your results change with a longer window of effects.
GiveWell have validated whether they are imposing a Western perspective when it comes to this moral weight judgement—they have surveyed people in Africa on this question.
People bring this up a lot, and I think it’s inaccurate. Although GiveWell has discussed the IDinsight survey as factoring into their moral weights, they do not actually incorporate its results, and the vast majority of their moral weights comes from surveys of donor preferences. From their 2020 update:
60% weight on donor responses
10% weight on James Snowden’s 2018 weights (as a proxy for 2018 GiveWell staff)
30% weight on YLLs (both as a commonly-used metric itself and as a proxy for the IDinsight survey)
They discuss their issues with the IDinsight survey:
Lots of respondents placed extremely high value (over $10 million) on life. That doesn’t necessarily imply lack of engagement with the question – it might be a true preference – but it’s not helpful for resource allocation. It’s an extraordinary result without extraordinary evidence to support it. If we believed this was an “accurate” representation of how the world should work, we expect that it would imply that low- and middle-income country governments and other institutions should be acting very differently than they currently are.
And conclude:
we don’t really trust the results as providing a coherent and accurate picture of people’s preferences.
So it’s not accurate to say that GiveWell’s moral weights have been “validated” by beneficiary preferences. GiveWell claims to be quite skeptical of the beneficiary preferences survey, so they just added a bit more weight on years of life lost (YLLs) to capture the rough idea of people valuing life more than they expected. In practice, this amounts to valuing the lives of under-5 children a bit more than valuing the lives of other people (since they have more YLLs):
our new weights value young children slightly more highly relative to people over the age of five, and essentially leave the relationship between averting a death and increasing consumption unchanged. (emphasis mine)
Given how much more people valued averting death over increasing consumption in the IDinsight survey, the fact that this relationship was unchanged makes it difficult to argue that GiveWell has really incorporated beneficiary preferences into their moral weights.
I know this wasn’t the main point of your comment, but it’s important to clarify because it comes up a lot as a defense when EA is criticized for being paternalistic, and I just don’t think that represents reality. GiveWell is absolutely imposing a Western perspective by having 70% of their weights be from donors and staff.
Often people post cost-effectiveness analyses of potential interventions, which invariably conclude that the intervention could rival GiveWell’s top charities. (I’m guilty of this too!) But this happens with such frequency, and I am basically never convinced that the intervention is actually competitive with GWTC. The reason is that they are comparing ex-ante cost-effectiveness (where you make a bunch of assumptions about costs, program delivery mechanisms, etc) with GiveWell’s calculated ex-post cost-effectiveness (where the intervention is already delivered, so there are much fewer assumptions).
Usually, people acknowledge that ex-ante cost-effectiveness is less reliable than ex-post cost-effectiveness. But I haven’t seen any acknowledgement that this systematically overestimates cost-effectiveness, because people who are motivated to try and pursue an intervention are going to be optimistic about unknown factors. Also, many costs are “unknown unknowns” that you might only discover after implementing the project, so leaving them out underestimates costs. (Also, the planning fallacy in general.) And I haven’t seen any discussion of how large the gap between these estimates could be. I think it could be orders of magnitude, just because costs are in the denominator of a benefit-cost ratio, so uncertainty in costs can have huge effects on cost-effectiveness.
One straightforward way to estimate this gap is to redo a GiveWell CEA, but assuming that you were setting up a charity to deliver that intervention for the first time. If GiveWell’s ex-post estimate is X and your ex-ante estimate is K*X for the same intervention, then we would conclude that ex-ante cost-effectiveness is K times too optimistic, and deflate ex-ante estimates by a factor of K.
I might try to do this myself, but I don’t have any experience with CEAs, and would welcome someone else doing it.
Labor markets in LMICs (what we know about growth, part 3)
As I write this, the commenting guidelines say “Aim to explain, not persuade” and “Approach disagreements with curiosity”. It doesn’t feel like the media policy embodies those values! Whenever I’ve seen media/outsiders criticize EA, EAs react defensively—which is a very normal human reaction, but hardly the kind of thing that should be coded into CEA policy.
My two cents is that if anyone is contacted by the media to discuss EA, they have no obligation whatsoever to follow CEA’s media policy. This isn’t a political party.
Could we have a higher karma discount for community posts? Almost every post on the frontpage for the past few weeks has been a community post, and I think karma inflation has outpaced the effect of the current discount. It would be nice for the EA forum frontpage to not be an endless pit of meta...
Firm growth in LMICs (what we know about growth, part 2)
I think reasonable people can disagree about the norms for EA Forum posts. Personally, I’m very happy that posts like this exist amidst the sea of hyperrational evaluation-focused posts. As said in one of my favorite posts:
Sidenote for the rigorous readers of this forum: I ask you to read this essay somewhat impressionistically. “The logic is incomplete, but let’s see if he’s still onto something,” is the attitude I’d suggest.
This post does not have complete logic by any means, but it’s onto something—in my view, it’s onto a moral intuition that we should pay more attention to than we often do. That’s not at odds with rationality, because rationality can only offer limited guidance on what morality should be, it can’t free us from relying on moral intuitions. So posts that are purely evaluative will miss a lot of ground on which people can (and do) make decisions. Put differently, moral intuitions and rational evaluation are complements, not substitutes.
I think the high karma of this post reflects the fact that for many people, including myself, it’s a reminder to keep alive the flame of trying to live ethically.
This is really strongly giving off suspicious convergence vibes, especially “AI safety means you don’t have to choose a cause”.
Also, “AI is better than us” is kind of scary religious talk. It sounds like we are worshipping a god and trying to summon it :)
Cause area: climate adaptation in low-income countries
Could you say more about the other side of the tradeoff? As in, what’s the affirmative case for having this policy? So far in this thread the main reason has been “we don’t want people to get the impression that X statement by a junior researcher represents RP’s views”. I see a very simple alternative as “if individuals make statements that don’t represent RP’s views they should always make that clear up front”. So is there more reason to have this policy?
In your position, my response to the question “why spend so much on this one dog?” would be “because I wanted to lol”. You don’t have to justify yourself to anyone, and you don’t have to reconstruct some post-hoc justification as to why you did it.
I understand that’s not a satisfying solution, in that it doesn’t preclude a slippery slope into “all of my money goes into arbitrary things that tug my heartstrings and none of it goes to the most effective causes”. You may be seeking guardrails against that possibility. There are none. Which is okay, because you probably don’t need them! You identify as EA for a reason. I’m going to guess it’s because the suffering of people and animals tugs at your heartstrings, even when you don’t see them. As long as that’s true, it seems extremely unlikely that you will fall off this slippery slope.
Moreover, I don’t think it’s healthy to try to justify all your life choices on EA grounds, a point made best here.
Thanks for writing this. I have to admit to confirmation bias here, but SM’s effects are so stupidly large that I just don’t believe they are possible. I hadn’t seen the control group also having a sharp decline but that raises even more alarm bells.
This is also very important for organizations trying to follow SM’s footsteps, like the recently incubated Vida Plena.
I anticipate that SM could enter a similar space as deworming now, where the evidence is shaky but the potential impacts are so high and the cost of delivery so low that it might be recommended/worth doing anyway.
EA is not ethically agnostic. It is unquestionably utilitarian (although I feel there is still some debate over the “total” part). Is this a problem for people of other ethical viewpoints? I don’t know, I can’t speak for other people. But I think there’s a lot of value to where the utilitarian rubber meets the road even if you have other moral considerations, in the ruthless focus on what works. For example, I still maintain a monthly donation to GiveDirectly even though I know it is much less cost effective than other givewell top charities. Why? Because I care about the dignity afforded by cash transfers in a non-consequentialist way, and I will comfortably make that choice instead of having a 10x larger impact from AMF or something like that. So I follow the utilitarian train up to a certain point (cash transfers work, we measure the impact with evidence) and then get off the train.
In this metaphor, EA keeps the train going until the end of the line (total utilitarianism?) But you don’t need to stay on until the end of the line. You can get off whenever you want. That makes it pretty freeing. The problem comes only when you feel like you need to stay on the train until the end, because of social pressure or because you feel clueless and want to be in with the smart people.
The eternal mantra: EA is a question, not an answer. And even if the majority of people believe in an answer that I don’t, I don’t care—it’s my question as much as it is theirs.
I think this post is very accurate, but I worry that people will agree with it in a vacuous way of “yes, there is a problem, we should do something about it, learning from others is good”. So I want to make a more pointed claim: I think that the single biggest barrier to interfacing between EAs and non-EAs is the current structure of community building. Community-building is largely structured around creating highly-engaged EAs, usually through recruiting college students or even high-school students. These students are not necessarily in the best position to interface between EA and other ways of doing good, precisely because they are so early into their careers and don’t necessarily have other competencies or viewpoints. So EA ends up as their primary lens for the world, and in my view that explains a sizable part of EA’s quasi-isolationist thinking on doing good.
This doesn’t mean all EAs who joined as college students (like me) end up as totally insular—life puts you into environments where you can learn from non-EAs. But that isn’t the default, and especially outside of global health and development, it is very easy for a young highly-engaged EA to avoid learning about doing good from non-EAs.