We appreciate the thoughtful comments by Eric Friedman. At the same time, we feel that they misrepresent GiveWell’s criteria and emphasis, and we feel it is important to clarify matters.
This post portrays GiveWell as placing strong weight on strength of evidence and as being highly averse to taking risks and chasing “upside,” i.e., potential for greater impact. I don’t think that’s accurate, but I do think the kind of “upside” we’re looking for is different from the kind Eric is looking for.
The “upside” Eric is seeking seems to be in the form of funding interventions with higher direct impact per dollar. It’s worth noting that the variance here—within the range of interventions under consideration—is not particularly large. We estimate that deworming does 2-3x more direct good per dollar spent than cash transfers; given the low robustness of the calculation, I don’t think it’s straightforward to say which actually has higher expected direct good accomplished per dollar outright. Bednets appear have similar cost-effectiveness to deworming and stronger evidence, and my sense is that the best interventions carried out by UNICEF are in the same ballpark as well. The gains here would be around a factor of 3 if one believed our estimates are highly robust. I believe that the right number to use when accounting for low robustness is below a factor of 3, though how much below is a matter for debate.
The “upside” we’re seeking, by contrast, is in the form of creating public goods and incentives that lead to better giving over time. Transparency is important because it allows more accurate cost-effectiveness analysis, assessment of room for more funding, and general learning that makes us better than evaluations. Generation of new evidence (something we believe GiveDirectly is more likely to provide than Evidence Action) is important for the same reasons. Rewarding charities that communicate clearly about their work and facilitate more learning is also important. I believe our emphasis on this sort of “upside” rather than on maximal (non-robustly) estimated cost-effectiveness has already led us to a much better position from which to be making informed giving decisions. I expect the vast majority of the good we and our donors do to lie well in the future, and so prioritizing factors relevant to learning and quality of information seems appropriate when the sacrifices in estimated-direct-impact terms are relatively small.
Another sort of “upside” we are interested in is that of scaling relatively small but high-potential organizations, especially innovative ones.
The contrast is particularly apparent in the case of GiveDirectly (strongly recommended by GiveWell, not supported by Eric) and UNICEF (supported by Eric, not recommended by GiveWell). Over the last year, GiveDirectly has attracted major attention in the media and in the development community. This attention may lead to dramatic rethinking of what aid should look like and how it should be evaluated. GiveDirectly is also pioneering unusual methods of aid delivery (mobile money) and evaluation/data collection (for example, using Google Earth to verify recipient eligibility). GiveDirectly has run an RCT of its own program and plans further experiments on both operational and programmatic components of cash transfers, which will produce new public goods. There are a lot of questions about GiveDirectly’s future; we expect to follow along with it, learn a lot, and share our learning publicly. By contrast, it is probably not possible to learn much from following along with the impact of one’s donations to UNICEF (it would not be possible even if GiveWell were to devote significant capacity to trying to do so), and my sense is that UNICEF’s contribution to global public goods and rethinking aid is dramatically lower than GiveDirectly’s when taking budget size into account. (Based on budget size and place in the aid community alone, it seems unlikely on its face that an $X donation to UNICEF has higher upside than an $X donation to GiveDirectly.) With all of this in mind, it doesn’t strike me as correct to say that the case for GiveDirectly over UNICEF is all about evidence and confidence, or to say that UNICEF has greater upside.
The contrast between GiveDirectly and Evidence Action is also an important one to note. Evidence Action is affiliated with IPA, but it does not itself carry out rigorous studies. While I believe Evidence Action is an excellent organization and strongly support donations to it, I don’t believe that donations to Evidence Action will lead to much more public knowledge about the effectiveness of chlorine dispensers or deworming, in the way that donations to GiveDirectly will lead to much more public knowledge about the impact of cash transfers.
I respect Eric’s preference for high estimated cost-effectiveness over strong evidence; this is a judgment call that I disagree with but am not confident about. However, I think it is incorrect to say that GiveWell is neglecting upside entirely in its recommendations. I see GiveDirectly as the most innovative and high-upside of the charities discussed in this post.
We appreciate the thoughtful comments by Eric Friedman. At the same time, we feel that they misrepresent GiveWell’s criteria and emphasis, and we feel it is important to clarify matters.
This post portrays GiveWell as placing strong weight on strength of evidence and as being highly averse to taking risks and chasing “upside,” i.e., potential for greater impact. I don’t think that’s accurate, but I do think the kind of “upside” we’re looking for is different from the kind Eric is looking for.
The “upside” Eric is seeking seems to be in the form of funding interventions with higher direct impact per dollar. It’s worth noting that the variance here—within the range of interventions under consideration—is not particularly large. We estimate that deworming does 2-3x more direct good per dollar spent than cash transfers; given the low robustness of the calculation, I don’t think it’s straightforward to say which actually has higher expected direct good accomplished per dollar outright. Bednets appear have similar cost-effectiveness to deworming and stronger evidence, and my sense is that the best interventions carried out by UNICEF are in the same ballpark as well. The gains here would be around a factor of 3 if one believed our estimates are highly robust. I believe that the right number to use when accounting for low robustness is below a factor of 3, though how much below is a matter for debate.
The “upside” we’re seeking, by contrast, is in the form of creating public goods and incentives that lead to better giving over time. Transparency is important because it allows more accurate cost-effectiveness analysis, assessment of room for more funding, and general learning that makes us better than evaluations. Generation of new evidence (something we believe GiveDirectly is more likely to provide than Evidence Action) is important for the same reasons. Rewarding charities that communicate clearly about their work and facilitate more learning is also important. I believe our emphasis on this sort of “upside” rather than on maximal (non-robustly) estimated cost-effectiveness has already led us to a much better position from which to be making informed giving decisions. I expect the vast majority of the good we and our donors do to lie well in the future, and so prioritizing factors relevant to learning and quality of information seems appropriate when the sacrifices in estimated-direct-impact terms are relatively small.
Another sort of “upside” we are interested in is that of scaling relatively small but high-potential organizations, especially innovative ones.
The contrast is particularly apparent in the case of GiveDirectly (strongly recommended by GiveWell, not supported by Eric) and UNICEF (supported by Eric, not recommended by GiveWell). Over the last year, GiveDirectly has attracted major attention in the media and in the development community. This attention may lead to dramatic rethinking of what aid should look like and how it should be evaluated. GiveDirectly is also pioneering unusual methods of aid delivery (mobile money) and evaluation/data collection (for example, using Google Earth to verify recipient eligibility). GiveDirectly has run an RCT of its own program and plans further experiments on both operational and programmatic components of cash transfers, which will produce new public goods. There are a lot of questions about GiveDirectly’s future; we expect to follow along with it, learn a lot, and share our learning publicly. By contrast, it is probably not possible to learn much from following along with the impact of one’s donations to UNICEF (it would not be possible even if GiveWell were to devote significant capacity to trying to do so), and my sense is that UNICEF’s contribution to global public goods and rethinking aid is dramatically lower than GiveDirectly’s when taking budget size into account. (Based on budget size and place in the aid community alone, it seems unlikely on its face that an $X donation to UNICEF has higher upside than an $X donation to GiveDirectly.) With all of this in mind, it doesn’t strike me as correct to say that the case for GiveDirectly over UNICEF is all about evidence and confidence, or to say that UNICEF has greater upside.
The contrast between GiveDirectly and Evidence Action is also an important one to note. Evidence Action is affiliated with IPA, but it does not itself carry out rigorous studies. While I believe Evidence Action is an excellent organization and strongly support donations to it, I don’t believe that donations to Evidence Action will lead to much more public knowledge about the effectiveness of chlorine dispensers or deworming, in the way that donations to GiveDirectly will lead to much more public knowledge about the impact of cash transfers.
I respect Eric’s preference for high estimated cost-effectiveness over strong evidence; this is a judgment call that I disagree with but am not confident about. However, I think it is incorrect to say that GiveWell is neglecting upside entirely in its recommendations. I see GiveDirectly as the most innovative and high-upside of the charities discussed in this post.