How many hits does hits-based giving get? A concrete study idea to find out (and a $1500 offer for implementation)

I have a pro­ject that I want to run by the com­mu­nity.

A while ago, Holden Karnofsky de­clared that the Open Philan­thropy Pro­ject is ded­i­cated to “hits-based giv­ing”, a frame­work that ac­cepts “philan­thropic risk” where 90% of grants have zero im­pact but the num­ber of grants that do have im­pact have suffi­cient im­pact to make the en­tire pro­ject worth­while. This could be com­pared to the more tra­di­tional ap­proach of GiveWell, where all grants made are to or­ga­ni­za­tions where the ex­pected value is rel­a­tively well known (even if the or­ga­ni­za­tions may still have zero im­pact).

While I’m quite sym­pa­thetic to the clas­sic GiveWell ap­proach, this kind of “hits-based” in­vest­ment policy sounds quite plau­si­bly effec­tive to me. When we’re in a world with many differ­ent pro­jects, with only a limited amount of time to get to know them, and with way too many un­re­solv­able un­knowns, we have to try to get some hits. This is quite analo­gous to what I think pretty much ev­ery ma­jor for-profit ven­ture cap­i­tal firm does with their for-profit in­vest­ments.

How­ever, I do have some room to doubt about the “hits-based” ap­proach. With poor se­lec­tion, it could re­solve to be “ran­dom giv­ing”, which I would ex­pect to hit at ap­prox­i­mately the mean in­ter­ven­tion—even if it were in a cause area with a high top 1%, we may not be able to find that top 1%, and the mean in­ter­ven­tion may be worse than the top global poverty in­ter­ven­tion we already know about.

I also don’t re­ally know if all ma­jor in­vest­ing can be de­scribed as “hits-based”. Per­haps sto­ries we hear about this “hits-based” strat­egy be­ing suc­cess­ful are mere sur­vivor­ship bias. I imag­ine many VC and non-VC in­vest­ing firms do fre­quently re­search their in­vest­ments sig­nifi­cantly, per­haps more sig­nifi­cantly than the Open Philan­thropy Pro­ject does. And even if the strat­egy does work well for for-profit VCs, the strat­egy may not be eas­ily ap­plied to the non-profit world, where in­cen­tives are no­tice­ably worse.

But fear not, for I think these ques­tions can be an­swered em­piri­cally. All we would have to do is run Open Phil for long enough and try to track down, best we can, how well the grants perform com­pared to AMF. For ex­am­ple, ar­guably Open Phil’s com­mit­ment to cage-free cor­po­rate cam­paign­ing could qual­ify as a “hit” that po­ten­tially sur­passes AMF (as­sum­ing pledge promises are suc­cess­fully kept with­out sig­nifi­cantly more spend­ing and that the fu­ture in­vest­ments in cam­paign­ing get com­pa­rable re­turns) and does ac­count for 12.5% of OpenPhil’s non-GiveWell grants to date[1].

Given that a sub­stan­tial com­par­i­son over time would still take a few years, if not decades, to fully re­solve (plus the value of ex­is­ten­tial risk miti­ga­tion may never be known), we might in­stead want to turn to peo­ple who have already done this for a long time and see how they have done.

A de­cent refer­ence class that came to mind was com­par­ing some his­tor­i­cal big foun­da­tions’ hits and misses (that tend to take more of a hits-based giv­ing ap­proach) with some com­pa­rable gov­ern­ment pro­grams that do similar sorts of pro­jects but with a more ev­i­dence-based low-var­i­ance strat­egy. I think it would take some re­search to find the right and a large enough sam­ple of foun­da­tions and gov­ern­ment agen­cies to com­pare but they seem to of­ten differ in this way, so it seems like it could be pos­si­ble. For ex­am­ple, the Gates Foun­da­tion seems to pur­sue hits-based giv­ing while the DFID does not seem to… is this char­ac­ter­i­za­tion true? If so, which one seems to be more cost-effec­tive on av­er­age?

As an­other ex­am­ple, if you took an ob­jec­tive crite­rion like “top 10 biggest foun­da­tions 1975-2000” and looked at all the biggest hits over those 25 years and di­vided it by all the money over those 25 years, would the cost-effec­tive­ness jus­tify all that spend­ing? If it turned out to be around the same as GiveDirectly, I’d be pretty con­vinced by the model of “hits-based giv­ing”, though we would have to ad­just for the fact that many ma­jor foun­da­tions are non-util­i­tar­ian and don’t aim to bring about the great­est pos­si­ble good.

And, of course, this whole idea will not be perfect. It will vary a lot in qual­ity based on the time and effort put into it, but it would be a huge step for­ward from the pretty soft in­tu­itions I have seen on this ques­tion so far. But I could see 40 hours of re­search mak­ing a good deal of progress on this prob­lem and I’m sur­prised that GiveWell, de­spite com­mit­ting to study­ing the his­tory of philan­thropy, has not pro­duced some­thing com­pre­hen­sive like this in defend­ing their wor­ld­view.

Re­solv­ing this ques­tion would be pretty ac­tion-rele­vant for my­self and a few other peo­ple, as we may per­son­ally be more in­clined to try to take big risks on big bets with our own pro­jects, rather than rely­ing on high-qual­ity ev­i­dence or work­ing to cre­ate more high-qual­ity ev­i­dence.

Pre­vi­ously I paid $100 to com­mis­sion a pro­ject that I sug­gested on the EA fo­rum and that went pretty well. I think this one is im­por­tant enough that I’d be will­ing to wa­ger money on this too. I’d pay $1500 for the first per­son that an­swers the ques­tion to my satis­fac­tion. Please con­tact me at pe­ter@pe­ter­hur­ prior to un­der­tak­ing this so I can help guide you and to avoid du­pli­ca­tion of work.


Up­date − 2 March 2017: See here for a more de­tailed elab­o­ra­tion of the pro­ject.

Up­date − 23 Aug 2017: It ended up be­ing the case that the data on grants from the top ten biggest foun­da­tions is sim­ply not available enough to make this pro­ject fea­si­ble in its cur­rent form. Most foun­da­tions do not have pub­lic digi­tal grant records and those that do typ­i­cally start af­ter 2000.


[1]: $154,008,339 to­tal grants given, minus $95,885,518 to GiveWell top char­i­ties = $58,122,821 non-GiveWell grants. Cage-free cam­paigns equal $7,239,392 of grant­ing, which is 12.5% of $58,122,821.