Should the Oxford Prioritisation Project look for lego bricks?

I’ll reg­u­larly be cross-post­ing things from the Oxford Pri­ori­ti­sa­tion Pro­ject blog to the EA fo­rum. This serves partly as a com­mit­ment mechanism for us, whereby we would be em­bar­rassed if we stopped post­ing. More im­por­tantly, we value the EA com­mu­nity’s feed­back on what we are do­ing. If you point out mis­takes in our work or make well-con­sid­ered sug­ges­tions, you’d stand a good chance of in­fluenc­ing our fi­nal de­ci­sion.

Here’s a re­cent post by Kon­stantin Sietzy, Ja­cob Lager­ros, and my­self.

Read and com­ment on the Google Doc­u­ment ver­sion of this post here.

By Tom Sit­tler, Kon­stantin Sietzy, Ja­cob Lagerros

Sum­mary:

Should the Oxford Pri­ori­ti­sa­tion Pro­ject fo­cus on dona­tion op­por­tu­ni­ties that are ‘the right size’? Is it im­por­tant to find a £10,000 fund­ing gap for a spe­cific pur­chase, (or by way of anal­ogy, £10,000-shaped lego bricks)? This ques­tion can quickly be­come very con­fus­ing. Over a se­ries of con­ver­sa­tions, we have man­aged to move to­wards greater con­cep­tual clar­ity around this is­sue, and we at­tempt to lay out our think­ing here.

Our con­clu­sion is that lego bricks are un­likely to be rele­vant to the Oxford Pri­ori­ti­sa­tion Pro­ject, for two main rea­sons:

1) What ap­pear to be gen­uine lego brick op­por­tu­ni­ties of­ten turn out to only shift benefits for­ward in time, and these shifts in time are con­tin­u­ous rather than dis­crete.

2) In or­der to iden­tify lego bricks, one needs very de­tailed in­for­ma­tion about or­gani­sa­tions’ fund­ing situ­a­tions.

Another brick in the wall?

As we move along with the Oxford Pri­ori­ti­sa­tion Pro­ject, we con­tinue to en­counter ques­tions of a more strate­gic na­ture, in­ter­sect­ing our ob­ject-level in­ves­ti­ga­tion at var­i­ous points. We sum this up as “meta-pri­ori­ti­sa­tion”, or per­haps more sim­ply, pri­ori­ti­sa­tion strat­egy. Ques­tions of meta-pri­ori­ti­sa­tion may in­clude con­sid­er­a­tions such as: how to re­solve value dis­agree­ments re­sult­ing from plu­ral­ist moral at­ti­tudes among the team; de­cid­ing be­tween build­ing ex­per­tise in a fo­cus area and keep­ing a bird’s eye view; or aiming at gen­er­at­ing max­i­mum di­rect im­pact vs. gen­er­at­ing in­for­ma­tion use­ful to fu­ture donors.

We ex­plore our first such ques­tion in this blog post: can, and should, the Oxford Pri­ori­ti­sa­tion Pro­ject team tar­get ‘lego bricks’ - giv­ing op­por­tu­ni­ties where we could effect a step change rather than a mere con­tin­u­ous in­crease in the util­ity of the tar­get or­gani­sa­tion with our dona­tion of £10,000?

There has been much dis­cus­sion of this topic, ow­ing to the fact that differ­ent team mem­bers had very differ­ent in­tu­itions about it. Some thought there surely must be lego bricks, and the world would be a very strange place if there weren’t, while oth­ers have the ex­act op­po­site in­tu­ition.

What are lego bricks?

We can con­trast two ap­proaches to find­ing dona­tion op­por­tu­ni­ties. Un­der a ‘marginal’ ap­proach, we would seek to find the or­gani­sa­tion with the high­est ex­pected cost-effec­tive­ness at the mar­gin. Bar­ring diminish­ing re­turns (un­likely at the scale of £10,000), we would be­have in roughly the same way as if we were look­ing to donate £5,000 or £20,000. Un­der a ‘lego brick’ ap­proach, in­stead, we would seek an op­por­tu­nity tai­lored to the ex­act size of the dona­tion. We would be look­ing for a highly spe­cific £10,000 fund­ing gap, like a hole in a lego brick tower that our dona­tion could click into.

The lego brick ap­proach has some in­tu­itive pull. We want to solve prob­lems with our dona­tion, and it seems rea­son­able in gen­eral to choose a prob­lem that’s the right size for one’s re­sources. We also have the in­tu­ition that donat­ing >£100,000 and donat­ing £10,000 should look very differ­ent and re­quire the an­swer to differ­ent ques­tions, mean­ing that the mar­ket for our fund­ing op­por­tu­ni­ties is suffi­ciently differ­ent from than that ex­plored by larger pri­ori­ti­za­tion or­ga­ni­za­tions such as GiveWell and the Open Philan­thropy Pro­ject.

How­ever, we be­lieve this is to a large ex­tent an illu­sion, and that search­ing for lego bricks would be a mis­take for the Oxford Pri­ori­ti­sa­tion Pro­ject. After much dis­cus­sion, there re­mains some dis­agree­ment as to whether lego bricks are merely ir­rele­vant for our pro­ject, be­cause they are the wrong size, and are too hard to find (Kon­stantin, Ja­cob), or whether they do not mean­ingfully ex­ist em­piri­cally (Tom).

This topic has been the source of much con­fu­sion among the three of us, and on the Oxford Pri­ori­ti­sa­tion Pro­ject team. Many similar, but in fact sub­tly differ­ent ideas play into whether there are lego bricks, and it can be hard to get one’s con­cepts clear and avoid talk­ing past each other. We hope this post can help us and oth­ers think more clearly in the fu­ture.

The lego brick ap­proach as­sumes that at least some giv­ing op­por­tu­ni­ties are non­con­tin­u­ous. In other words, filling a fund­ing gap with a £10,000 lego brick has not twice as much im­pact as giv­ing this or­gani­sa­tion £5,000, but per­haps five or ten times as much. Similarly, donat­ing £20,000 has less than twice as much im­pact as £10,000. A lego brick there­fore amounts to a step change in util­ity func­tions.


Figure 1: A nor­mal step function

Look­ing at Figure 1, some moves along the hori­zon­tal axis (dona­tions y) have no im­pact at all on the value u cre­ated by the or­gani­sa­tion, while some much smaller moves cre­ate a step-in­crease in value.

The in­tu­itive ra­tio­nale for step-func­tions is that the pur­chases driv­ing an or­gani­sa­tion’s im­pact are dis­crete. The Against Malaria Foun­da­tion may only con­duct an ad­di­tional bed­net dis­tri­bu­tion if it ac­quires an ex­tra $3 mil­lion, and do noth­ing if it gets $2.99 mil­lion. An or­gani­sa­tion may be un­able to hire a new staff mem­ber if their funds fall just short of a thresh­old.

Does this mean it’s cru­cial to find such steps in the util­ity func­tion and fill them? If the step-func­tion view of the world is cor­rect, then filling a cor­rectly-sized gap mat­ters vastly more than the effec­tive­ness of the or­gani­sa­tion one is choos­ing. Even if we found the most effec­tive or­gani­sa­tion in the world, but donated on a flat part of its step func­tion, we would pro­duce zero im­pact.

Keep in mind that ‘filling’ in our use means to ‘push the util­ity func­tion over the edge’. It mustn’t nec­es­sar­ily amount to filling the en­tire hori­zon­tal length of a step as in­di­cated by the di­a­grams; rather, what mat­ters is the fi­nal, value-gen­er­at­ing step.

This whole post as­sumes that no-one else will fill the var­i­ous fund­ing gaps. If they were go­ing to be filled any­way, our im­pact would come en­tirely from shift­ing some­one else’s dona­tion, re­gard­less of whether or not there are lego bricks. We think that the var­i­ous is­sues around donor co­or­di­na­tion, are or­thog­o­nal to the is­sue we dis­cuss here.

Do lego bricks ex­ist (even when or­gani­sa­tions can save)?

The Against Malaria Foun­da­tion can trace each bed­net to a dis­tance of 6 me­ters, be­cause each one has a GPS tracker. Imag­ine a coun­ter­fac­tual world in which AMF had just dis­tributed bed nets ev­ery year, with­out us­ing track­ers. Per­haps at some point they would have no­ticed that in­stal­ling track­ers on bed nets could in­crease their effec­tive­ness, due to higher re­ten­tion rates, etc. – but the one-off costs of de­vel­op­ing a hard­ware and soft­ware solu­tion for equip­ping nets with track­ers would have been to high. So they keep donat­ing bed nets ev­ery year, and while their over­all pos­i­tive im­pact con­tinues to in­crease, an op­por­tu­nity for a step change in util­ity is missed. (This is a fully hy­po­thet­i­cal ex­am­ple, and we are sure it can be picked apart on its nar­ra­tive de­tails; what mat­ters, though, is whether it illus­trates the point).

A counter-ar­gu­ment is that, in this case, AMF should have just stopped spend­ing money on bed nets the year they re­al­ised this op­por­tu­nity, saved up money, and even­tu­ally restarted dis­tri­bu­tion with track­ers in­stalled, per­haps two years later. This would have ended up mul­ti­ply­ing the value of each bed net con­sid­er­ably such that the pos­i­tive im­pact of fu­ture net dis­tri­bu­tions would out­weigh the dis­value of the two-year de­lay in net dis­tri­bu­tion.

In prac­tice, or­gani­sa­tions are un­likely to re­duce their costs so dras­ti­cally. Or­gani­sa­tions gen­er­ally have a high share of fixed costs in their to­tal ex­pen­di­ture, which are difficult to sub­stan­tially re­duce tem­porar­ily. Th­ese fixed costs in­clude office rental, staff salaries, etc.

We need a more com­plex model. Sup­pose a char­ity has yearly rev­enue R, and fixed costs F, such that its dis­cre­tionary in­come is R-F=D.

We are in­ter­ested in the size of D rel­a­tive to po­ten­tial lego bricks. For a lego brick op­por­tu­nity of size L, a char­ity would need to de­lay its pur­chase by L/​D years.

This con­clu­sion sig­nifi­cantly weak­ens lego brick ar­gu­ments. For if we donated a pro­por­tion pL of L, we would re­duce the de­lay by a pro­por­tion p. For in­stance, if an or­gani­sa­tion only had D=£2,000/​year, and could make a high-value pur­chase worth £10,000, it would take them 5 years to save the money if they re­ceived no dona­tion, 4 years if they re­ceived a £2,000 dona­tion, 2 years if they re­ceived a £6,000 dona­tion, etc. In other words, a step­wise func­tion u(y) turns into a smoother func­tion, of ‘de­lay in util­ity’ as a func­tion of y. (We call this the de­lay ar­gu­ment)


Figure 2: the de­lay ar­gu­ment.

Time-limited opportunities

Ja­cob pointed out that the above as­sumes that op­por­tu­ni­ties are not time-limited, so an or­gani­sa­tion can always de­lay pur­chases. Hence we see an op­por­tu­nity that is ac­tu­ally strongly time-limited, on the scale of months or a year, as a case of a gen­uine lego brick. (But we think these are very difficult to find for the Oxford Pri­ori­ti­sa­tion Pro­ject. See be­low, ‘Time-de­pen­dent op­por­tu­ni­ties’).

How­ever, all op­por­tu­ni­ties are time-limited to some ex­tent. If L/​D=5, it seems un­rea­son­able to ex­pect or­gani­sa­tions will save for 5 years, given the risk that the op­por­tu­nity may dis­ap­pear in the mean­time.

If D was very small, there would be many lego brick op­por­tu­ni­ties, even once we con­sider the de­lay ar­gu­ment.

The size of D is an em­piri­cal ques­tion. There is some dis­agree­ment be­tween Tom and Kon­stantin about the size of D we should ex­pect to en­counter.

One ar­gu­ment for think­ing D is un­likely to be small (say, smaller than £50,000) is that or­gani­sa­tions want to hedge against the pos­si­bil­ity of los­ing fund­ing. Say an or­gani­sa­tion had R=£1,000,000. They would be un­likely to set F=£990,000 be­cause they would en­counter ma­jor prob­lems if they lost only one £10,000 donor. One rough guess is that we would ex­pect D to be at least 10% of R.

How­ever, Kon­stantin ar­gued that F might be larger than we think, and thus D smaller. This is be­cause or­gani­sa­tions face a va­ri­ety of ‘soft con­straints’ (e.g. ir­ra­tional in­er­tia, mis­al­igned in­cen­tives, ig­no­rant donors).

For ex­am­ple, if or­gani­sa­tions scale back some of their work in or­der to save, they may lose im­por­tant fun­ders or part­ners on the ground. To take an ex­treme ex­am­ple, F can­not be 0 for the Against Malaria foun­da­tion: if they sim­ply stopped dis­tribut­ing bed­nets for mul­ti­ple years they would lose the trust of their part­ners.

F might be higher than we think be­cause of var­i­ous prin­ci­pal-agent prob­lems. Even char­i­ta­ble or­gani­sa­tions are un­likely to find that their staff will hap­pily ac­cept salary freezes to spon­sor a fu­ture benefi­cial but non-cru­cial util­ity in­crease. Scal­ing back some pro­grammes may be per­ceived as hav­ing failed as an or­gani­sa­tion or as a leader, so man­agers might be re­luc­tant to do so to cap­ture fu­ture benefits.

As sav­ings build up, or­gani­sa­tions might be­come tempted to use them to in­crease F, for ex­am­ple by hiring more staff, rather than to keep sav­ing for a fu­ture pur­chase. Kon­stantin thinks this is es­pe­cially true in cases where the pur­chase would be nonessen­tial to the con­tinued func­tion­ing of the or­gani­sa­tions—such as in the GPS tracker ex­am­ple (AMF would still be dis­tribut­ing highly cost-effec­tive bed­nets if they had no track­ers). There may be in­ter­nal dis­agree­ment about pri­ori­ties and the ne­ces­sity of in­vest­ment D within the or­gani­sa­tion or be­tween the or­gani­sa­tion and the donor com­mu­nity. Paus­ing dis­cre­tionary spend­ing in or­der to make a fu­ture pur­chase or hire is likely to be un­pop­u­lar in­ter­nally or ex­ter­nally . We would ex­pect this in­er­tia-effect to scale with L/​D, i.e. as the op­por­tu­nity be­comes more costly rel­a­tive to ex­ist­ing in­come and moves fur­ther into the fu­ture (see be­low). In this case, a lego-brick ori­ented donor might achieve sig­nifi­cant im­pact if they act as an ex­ter­nal shock. This be­lief is con­sis­tent with ini­tial psy­cholog­i­cal re­search that sug­gests peo­ple fol­low men­tal ac­count­ing pro­ce­dures to be more will­ing to spend un­ex­pected wind­falls than ob­jec­tively iden­ti­cal amounts out of pre-planned bud­gets.

The ex­tent to which these con­straints ex­ist, and whether they are a ma­jor source of lego bricks, re­mains an un­re­solved em­piri­cal dis­agree­ment be­tween Kon­stantin and Tom.

Are lego bricks rele­vant for the Oxford Pri­ori­ti­sa­tion Pro­ject?

Do we have a high enough chance of find­ing lego bricks?

Knowl­edge of fund­ing situation

Even if one were con­vinced that lego bricks did ex­ist in a mean­ingful sense, find­ing lego bricks re­quires be­ing knowl­edge­able in de­tail about an or­ga­ni­za­tion’s’ fund­ing situ­a­tion, and the op­por­tu­ni­ties they are fac­ing. But if we can­not know this, we are in effect fac­ing many differ­ent step func­tions. Aver­ag­ing out across them brings us back to a con­tin­u­ous func­tion.

Figure 3: Un­der im­perfect in­for­ma­tion, from the donor per­spec­tive step func­tions av­er­age out to be­ing continuous

Time-de­pen­dent opportunities

As de­scribed above, we see a strongly time-limited op­por­tu­nity, on the scale of months or a year, as a case of a gen­uine lego brick. (“we could hire this amaz­ing em­ployee this month who will oth­er­wise do a PhD and take up a teach­ing ca­reer”). How­ever, given the time con­straints of the Oxford Pri­ori­ti­sa­tion Pro­ject, and the limi­ta­tions of our net­works and de­tailed knowl­edge of par­tic­u­lar or­gani­sa­tions, we think we’d be very un­likely to find a time-limited op­por­tu­nity that would not oth­er­wise have been filled, be­fore the dead­line for our de­ci­sion.

£10,000-sized lego bricks?

If there were lego bricks to be found, what size would they be? A rea­son­able heuris­tic could be the cost to hire an em­ployee for a year, let’s say £50,000. We strug­gle to think of ex­am­ples of £10,000 lego bricks.

Fur­ther­more, Kon­stantin gave an­other rea­son to ex­pect lego bricks to be in the £100,000s of pounds rather than the low £10,000s. Soft con­straints stem­ming from ‘ir­ra­tional’ or­gani­sa­tional be­havi­our are more likely to prove pro­hibitive where the amounts con­cerned are larger; both in­ter­nal and ex­ter­nal stake­hold­ers are more likely to ac­cept re­duc­ing dis­cre­tionary spend­ing for a cause they don’t con­sider nec­es­sary if this is a small amount rel­a­tive to the or­gani­sa­tion’s over­all rev­enue than a large one. For ex­am­ple, AMF may plau­si­bly con­vince team mem­bers and donors to save on sta­tion­ery and dis­tribute fewer bed­nets to the value of £9,000 for a year, but may not be able to ar­gue the same with £500,000.

Conclusion

Over­all, we agree that the Oxford Pri­ori­ti­sa­tion Pro­ject should not at­tempt to find lego bricks.