Thanks for posting this, Jonathan! I was going to share it on the EA Forum too but just havenât gotten around to it.
I think GIFâs impact methodology is not comparable to GiveWellâs. My (limited) understanding is that their Practical Impact approach is quite similar to USAIDâs Development Innovation Venturesâ impact methodology. DIVâs approach was co-authored by Michael Kremer so it has solid academic credentials. But importantly, the method takes credit for the funded NGOâs impact over the next 10 years, without sharing that impact with subsequent funders. The idea is that the innovation would fail without their support so they can claim all future impact if the NGO survives (the total sum of counterfactual impact need not add to 100%). This is not what GiveWell does. GiveWell takes credit for the long-term impact of the beneficiaries it helps but not for the NGOs themselves. So this is comparing apples to oranges. Itâs true that GiveWell Top Charities are much more likely to survive without GiveWellâs help but this leads to my next point.
GiveWell also provides innovation grants through their All Grants Fund (formerly called Incubation Grants). Theyâve been funding a range of interventions that arenât Top Charities and in many cases, are very early, with GiveWell support being critical to the NGOâs survival. According to GiveWellâs All Grants Fund page, âAs of July 2022, we expect to direct about three-quarters of our grants to top charity programs and one-quarter to other programs, so thereâs a high likelihood that donations to the All Grants Fund will support a top charity grant.â This suggests that in GiveWellâs own calculus, innovation grants as a whole cannot be overwhelmingly better than Top Charities. Otherwise, Top Charities wouldnât account for the majority of the fund.
When thinking about counterfactual impact, the credit one gets for funding innovation should depend on the type of future donors the NGO ends up attracting. If these future donors would have given with low cost-effectiveness otherwise (or not at all), then you deserve much credit. But if they would have given to equally (or even more) cost-effective projects, then you deserve zero (or even negative) credit. So if GIF is funding NGOs that draw money from outside EA (whereas GiveWell isnât), itâs plausible their innovations have more impact and thus are more âcost-effectiveâ. But we are talking about leverage now, so again, I donât think the methodologies are directly comparable.
Finally, I do think GIF should be more transparent about their impact calculations when making such a claim. It would very much benefit other donors and the broader ecosystem if they can make public their 3x calculation (just share the spreadsheet please!). Without such transparency, we should be skeptical and not take their claim too seriously. Extraordinary claims require extraordinary evidence.
Thanks for posting this, Jonathan! I was going to share it on the EA Forum too but just havenât gotten around to it.
I think GIFâs impact methodology is not comparable to GiveWellâs. My (limited) understanding is that their Practical Impact approach is quite similar to USAIDâs Development Innovation Venturesâ impact methodology. DIVâs approach was co-authored by Michael Kremer so it has solid academic credentials. But importantly, the method takes credit for the funded NGOâs impact over the next 10 years, without sharing that impact with subsequent funders. The idea is that the innovation would fail without their support so they can claim all future impact if the NGO survives (the total sum of counterfactual impact need not add to 100%). This is not what GiveWell does. GiveWell takes credit for the long-term impact of the beneficiaries it helps but not for the NGOs themselves. So this is comparing apples to oranges. Itâs true that GiveWell Top Charities are much more likely to survive without GiveWellâs help but this leads to my next point.
GiveWell also provides innovation grants through their All Grants Fund (formerly called Incubation Grants). Theyâve been funding a range of interventions that arenât Top Charities and in many cases, are very early, with GiveWell support being critical to the NGOâs survival. According to GiveWellâs All Grants Fund page, âAs of July 2022, we expect to direct about three-quarters of our grants to top charity programs and one-quarter to other programs, so thereâs a high likelihood that donations to the All Grants Fund will support a top charity grant.â This suggests that in GiveWellâs own calculus, innovation grants as a whole cannot be overwhelmingly better than Top Charities. Otherwise, Top Charities wouldnât account for the majority of the fund.
When thinking about counterfactual impact, the credit one gets for funding innovation should depend on the type of future donors the NGO ends up attracting. If these future donors would have given with low cost-effectiveness otherwise (or not at all), then you deserve much credit. But if they would have given to equally (or even more) cost-effective projects, then you deserve zero (or even negative) credit. So if GIF is funding NGOs that draw money from outside EA (whereas GiveWell isnât), itâs plausible their innovations have more impact and thus are more âcost-effectiveâ. But we are talking about leverage now, so again, I donât think the methodologies are directly comparable.
Finally, I do think GIF should be more transparent about their impact calculations when making such a claim. It would very much benefit other donors and the broader ecosystem if they can make public their 3x calculation (just share the spreadsheet please!). Without such transparency, we should be skeptical and not take their claim too seriously. Extraordinary claims require extraordinary evidence.