This line of reasoning seems sensible to me. However, it does raise the following question: will GiveWell also stop recommending GiveDirectly, given that, by your own cost-effectiveness numbers, it’s 10-20x less cost-effective than basically all your other recommendations? And, if not, why not?
I can understand the importance of having some variety of options to recommend donors, which necessitates recommending some things that are worse than others, but 10x worse seems to leave quite a lot of value on the table. Hence, I’d be curious to hear the rationale.
I’ll post Catherine’s reply and then raise a couple of issues:
Thanks for your question. You’re right that we model GiveDirectly as the least cost-effective top charity on our list, and we prioritize directing funds to other top charities (e.g. through the Maximum Impact Fund). GiveDirectly is the benchmark against which we compare the cost-effectiveness of other opportunities we might fund.
As we write in the post above, standout charities were defined as those that “support programs that may be extremely cost-effective and are evidence-backed” but “we do not feel as confident in the impact of these organizations as we do in our top charities.”
Our level of confidence, rather than their estimated cost-effectiveness, is the key difference between our recommendation of GiveDirectly and standout charities.
We consider the evidence of GiveDirectly’s impact to be exceptionally strong. We’re not sure that our standout charities were less cost-effective than GiveDirectly (in fact, as we wrote, some may be extremely cost-effective), but we felt less confident in making that assessment, based on the more limited evidence in support of their impact, as well as our more limited engagement with them.
I don’t see a justification here for keeping GiveDirectly in the list. Okay, there are charities GiveWell is ‘confident’ in, and those that they aren’t, and GiveDirectly, like the other top picks, is in the first category. But this still raises the question of why to recommend GiveDirectly at all. Indeed, it’s arguably more puzzling: if you think there’s basically no chance A is better than B, why advocate for A? At least if you think A might be better than B, then you might defend recommending A on the grounds there’s a chance, that is, if someone believes X, Y, Z they might sensibly believe it’s better.
The other thing that puzzles me about this response is its seemingly non-standard approach to expected value reasoning. Suppose you can do G, which has a 100% chance of doing one ‘unit’ of good, or H, which has a 50% chance of doing 3 ‘units’ of good. I say you should pick H because, in expectation, it’s better, even though you’re not sure it will be better.
Where might having less evidence fit into this?
One approach to dealing with different levels of evidence is to discount the ‘naive’ expected value of the intervention, that is, the one you get from taking the evidence at face value. Why and by how much should you discount your ‘naive’ estimate? Well, you reduce it to what you expect you would conclude its actual expected value was if you had better information. For instance, suppose one intervention has RCTs with much smaller samples, and you know that effect sizes tend to go down when interventions use larger samples (they are harder to implement at scale, etc.). Hence, you’re justified in discounting it because and to that extent. Once you’ve done this, you have the ‘sophisticated’ expected values. Then you do the thing with the higher ‘sophisticated’ expected value.
Hence, I don’t see why lower (‘naive’) cost-effectiveness should stop someone from recommending something.
This line of reasoning seems sensible to me. However, it does raise the following question: will GiveWell also stop recommending GiveDirectly, given that, by your own cost-effectiveness numbers, it’s 10-20x less cost-effective than basically all your other recommendations? And, if not, why not?
I can understand the importance of having some variety of options to recommend donors, which necessitates recommending some things that are worse than others, but 10x worse seems to leave quite a lot of value on the table. Hence, I’d be curious to hear the rationale.
They answered this in their own comments section.
I’ll post Catherine’s reply and then raise a couple of issues:
I don’t see a justification here for keeping GiveDirectly in the list. Okay, there are charities GiveWell is ‘confident’ in, and those that they aren’t, and GiveDirectly, like the other top picks, is in the first category. But this still raises the question of why to recommend GiveDirectly at all. Indeed, it’s arguably more puzzling: if you think there’s basically no chance A is better than B, why advocate for A? At least if you think A might be better than B, then you might defend recommending A on the grounds there’s a chance, that is, if someone believes X, Y, Z they might sensibly believe it’s better.
The other thing that puzzles me about this response is its seemingly non-standard approach to expected value reasoning. Suppose you can do G, which has a 100% chance of doing one ‘unit’ of good, or H, which has a 50% chance of doing 3 ‘units’ of good. I say you should pick H because, in expectation, it’s better, even though you’re not sure it will be better.
Where might having less evidence fit into this?
One approach to dealing with different levels of evidence is to discount the ‘naive’ expected value of the intervention, that is, the one you get from taking the evidence at face value. Why and by how much should you discount your ‘naive’ estimate? Well, you reduce it to what you expect you would conclude its actual expected value was if you had better information. For instance, suppose one intervention has RCTs with much smaller samples, and you know that effect sizes tend to go down when interventions use larger samples (they are harder to implement at scale, etc.). Hence, you’re justified in discounting it because and to that extent. Once you’ve done this, you have the ‘sophisticated’ expected values. Then you do the thing with the higher ‘sophisticated’ expected value.
Hence, I don’t see why lower (‘naive’) cost-effectiveness should stop someone from recommending something.