Thanks for posting this, Jonathan! I was going to share it on the EA Forum too but just haven’t gotten around to it.
I think GIF’s impact methodology is not comparable to GiveWell’s. My (limited) understanding is that their Practical Impact approach is quite similar to USAID’s Development Innovation Ventures’ impact methodology. DIV’s approach was co-authored by Michael Kremer so it has solid academic credentials. But importantly, the method takes credit for the funded NGO’s impact over the next 10 years, without sharing that impact with subsequent funders. The idea is that the innovation would fail without their support so they can claim all future impact if the NGO survives (the total sum of counterfactual impact need not add to 100%). This is not what GiveWell does. GiveWell takes credit for the long-term impact of the beneficiaries it helps but not for the NGOs themselves. So this is comparing apples to oranges. It’s true that GiveWell Top Charities are much more likely to survive without GiveWell’s help but this leads to my next point.
GiveWell also provides innovation grants through their All Grants Fund (formerly called Incubation Grants). They’ve been funding a range of interventions that aren’t Top Charities and in many cases, are very early, with GiveWell support being critical to the NGO’s survival. According to GiveWell’s All Grants Fund page, “As of July 2022, we expect to direct about three-quarters of our grants to top charity programs and one-quarter to other programs, so there’s a high likelihood that donations to the All Grants Fund will support a top charity grant.” This suggests that in GiveWell’s own calculus, innovation grants as a whole cannot be overwhelmingly better than Top Charities. Otherwise, Top Charities wouldn’t account for the majority of the fund.
When thinking about counterfactual impact, the credit one gets for funding innovation should depend on the type of future donors the NGO ends up attracting. If these future donors would have given with low cost-effectiveness otherwise (or not at all), then you deserve much credit. But if they would have given to equally (or even more) cost-effective projects, then you deserve zero (or even negative) credit. So if GIF is funding NGOs that draw money from outside EA (whereas GiveWell isn’t), it’s plausible their innovations have more impact and thus are more ‘cost-effective’. But we are talking about leverage now, so again, I don’t think the methodologies are directly comparable.
Finally, I do think GIF should be more transparent about their impact calculations when making such a claim. It would very much benefit other donors and the broader ecosystem if they can make public their 3x calculation (just share the spreadsheet please!). Without such transparency, we should be skeptical and not take their claim too seriously. Extraordinary claims require extraordinary evidence.
Thanks for posting this, Jonathan! I was going to share it on the EA Forum too but just haven’t gotten around to it.
I think GIF’s impact methodology is not comparable to GiveWell’s. My (limited) understanding is that their Practical Impact approach is quite similar to USAID’s Development Innovation Ventures’ impact methodology. DIV’s approach was co-authored by Michael Kremer so it has solid academic credentials. But importantly, the method takes credit for the funded NGO’s impact over the next 10 years, without sharing that impact with subsequent funders. The idea is that the innovation would fail without their support so they can claim all future impact if the NGO survives (the total sum of counterfactual impact need not add to 100%). This is not what GiveWell does. GiveWell takes credit for the long-term impact of the beneficiaries it helps but not for the NGOs themselves. So this is comparing apples to oranges. It’s true that GiveWell Top Charities are much more likely to survive without GiveWell’s help but this leads to my next point.
GiveWell also provides innovation grants through their All Grants Fund (formerly called Incubation Grants). They’ve been funding a range of interventions that aren’t Top Charities and in many cases, are very early, with GiveWell support being critical to the NGO’s survival. According to GiveWell’s All Grants Fund page, “As of July 2022, we expect to direct about three-quarters of our grants to top charity programs and one-quarter to other programs, so there’s a high likelihood that donations to the All Grants Fund will support a top charity grant.” This suggests that in GiveWell’s own calculus, innovation grants as a whole cannot be overwhelmingly better than Top Charities. Otherwise, Top Charities wouldn’t account for the majority of the fund.
When thinking about counterfactual impact, the credit one gets for funding innovation should depend on the type of future donors the NGO ends up attracting. If these future donors would have given with low cost-effectiveness otherwise (or not at all), then you deserve much credit. But if they would have given to equally (or even more) cost-effective projects, then you deserve zero (or even negative) credit. So if GIF is funding NGOs that draw money from outside EA (whereas GiveWell isn’t), it’s plausible their innovations have more impact and thus are more ‘cost-effective’. But we are talking about leverage now, so again, I don’t think the methodologies are directly comparable.
Finally, I do think GIF should be more transparent about their impact calculations when making such a claim. It would very much benefit other donors and the broader ecosystem if they can make public their 3x calculation (just share the spreadsheet please!). Without such transparency, we should be skeptical and not take their claim too seriously. Extraordinary claims require extraordinary evidence.