List of ways in which cost-effectiveness estimates can be misleading

In my cost-effec­tive­ness es­ti­mate of cor­po­rate cam­paigns, I wrote a list of all the ways in which my es­ti­mate could be mis­lead­ing. I thought it could be use­ful to have a more broadly-ap­pli­ca­ble ver­sion of that list for cost-effec­tive­ness es­ti­mates in gen­eral. It could maybe be used as a check­list to see if no im­por­tant con­sid­er­a­tions were missed when cost-effec­tive­ness es­ti­mates are made or in­ter­preted.

The list be­low is prob­a­bly very in­com­plete. If you know of more items that should be added, please com­ment. I tried to op­ti­mize the list for skim­ming.

How cost es­ti­mates can be misleading

  • Costs of work of oth­ers. Sup­pose a char­ity pur­chases a vac­cine. This causes the gov­ern­ment to spend money dis­tribut­ing that vac­cine. It’s un­clear whether the costs of the gov­ern­ment should be taken into ac­count. Similarly, it can be un­clear whether to take into ac­count the costs that pa­tients have to spend to travel to a hos­pi­tal to get vac­ci­nated. This is closely re­lated to con­cepts of lev­er­age and per­spec­tive. More on it can be read in Byford and Raftery (1998), Karnofsky (2011), Snow­den (2018), and Sethu (2018).

  • It can be un­clear whether to take into ac­count the fixed costs from the past that will not have to be spent again. E.g., costs as­so­ci­ated with set­ting up a char­ity that are already spent and are not di­rectly rele­vant when con­sid­er­ing whether to fund that char­ity go­ing for­ward. How­ever, such costs can be rele­vant when con­sid­er­ing whether to found a similar char­ity in an­other coun­try. Some guidelines sug­gest an­nu­al­iz­ing fixed costs. When fixed costs are taken into ac­count, it’s of­ten un­clear how far to go. E.g., when es­ti­mat­ing the cost of dis­tribut­ing a vac­cine, even the costs of roads that were built partly to make the dis­tri­bu­tion eas­ier could be taken into ac­count.

  • Not tak­ing fu­ture costs into ac­count. E.g., an es­ti­mate of cor­po­rate cam­paigns may take into ac­count the costs of win­ning cor­po­rate com­mit­ments, but not fu­ture costs of en­sur­ing that cor­po­ra­tions will com­ply with these com­mit­ments. Fu­ture costs and effects may have to be ad­justed for the pos­si­bil­ity that they don’t oc­cur.

  • Not tak­ing past costs into ac­count. In the first year, a home­less­ness char­ity builds many houses. In the sec­ond year, it finds home­less peo­ple to live in those houses. In the first year, the im­pact of the char­ity could be calcu­lated as zero. In the sec­ond year, it could be calcu­lated to be un­rea­son­ably high. But the char­ity wouldn’t be able to sus­tain the cost-effec­tive­ness of the sec­ond year.

  • Not ad­just­ing past or fu­ture costs for in­fla­tion.

  • Not tak­ing over­head costs into ac­count. Th­ese are costs as­so­ci­ated with ac­tivi­ties that sup­port the work of a char­ity. It can in­clude op­er­a­tional, office rental, util­ities, travel, in­surance, ac­count­ing, ad­minis­tra­tive, train­ing, hiring, plan­ning, man­age­rial, and fundrais­ing costs.

  • Not tak­ing costs that don’t pay off into ac­count. Noth­ing But Nets is a char­ity that dis­tributes bed­nets that pre­vent mosquito-bites and con­se­quently malaria. One of their old blog posts, Sauber (2008), used to claim that “If you give $100 of your check to Noth­ing But Nets, you’ve saved 10 lives.” While it may be true that it costs around $10 or less[1] to provide a bed­net, and some bed­nets save lives, costs of bed­nets that did not save lives should be taken into ac­count as well. Ac­cord­ing to GiveWell’s es­ti­mates, it cur­rently costs roughly $3,500 for a similar char­ity (Against Malaria Foun­da­tion) to save one life by dis­tribut­ing bed­nets.

    Wiblin (2017) de­scribes a sur­vey in which re­spon­dents were asked “How much do you think it would cost a typ­i­cal char­ity work­ing in this area on av­er­age to pre­vent one child in a poor coun­try from dy­ing un­nec­es­sar­ily, by im­prov­ing ac­cess to med­i­cal care?” The me­dian an­swer was $40. There­fore, it seems that many peo­ple are mis­led by claims like the one by Noth­ing But Nets.

  • Failing to take into ac­count vol­un­teer time as costs. Imag­ine many vol­un­teers col­lab­o­rat­ing to do a lot of good, and hav­ing a small bud­get for snacks. Their cost-effec­tive­ness es­ti­mate could be very high, but it would be a mis­take to ex­pect their im­pact to dou­ble if we dou­ble their fund­ing for snacks. Such prob­lems would not hap­pen if we val­ued one hour of vol­un­teer time at say $10 when es­ti­mat­ing costs. The more a char­ity de­pends on vol­un­teers, the more this con­sid­er­a­tion is rele­vant.

  • Failing to take into ac­count the coun­ter­fac­tual im­pact of al­tru­is­tic em­ploy­ees (op­por­tu­nity cost). There are hid­den costs of em­ploy­ing peo­ple who would be do­ing good even if they weren’t em­ployed by a char­ity. For ex­am­ple:

    • A per­son who used to do earn­ing-to-give is em­ployed by a char­ity, takes a low salary, and stops donat­ing money. The im­pact of their lost dona­tions should ideally be some­how added to the cost es­ti­mate, but it’s very difficult to do it in prac­tice.

    • A char­ity hires the most tal­ented EAs and makes them work on things that are not top pri­or­ity. De­spite amaz­ing re­sults, the char­ity could be do­ing harm be­cause the tal­ented EAs could have made more im­pact by work­ing el­se­where.

  • Ease of fundrais­ing /​ coun­ter­fac­tual im­pact of dona­tions. Let’s say you are de­cid­ing which char­ity you should start. Char­ity A could do a very cost-effec­tive in­ter­ven­tion but only peo­ple who already donate to cost-effec­tive char­i­ties would be in­ter­ested in sup­port­ing it. Char­ity B could do a slightly less cost-effec­tive in­ter­ven­tion but would have a main­stream ap­peal and could fundraise from peo­ple who don’t donate to any char­i­ties or only donate to in­effec­tive char­i­ties. Other things be­ing equal, you would do more good by start­ing Char­ity B, even though it would be less cost-effec­tive. Firstly, Char­ity B wouldn’t take fund­ing away from other effec­tive char­i­ties.. Se­condly, Char­ity B could grow to be much larger and hence do more good (pro­vided that its in­ter­ven­tion is scal­able).

  • Cost of eval­u­a­tion. Imag­ine want­ing to start a small pro­ject and ask­ing for fund­ing from many differ­ent EA donors and funds. The main cost of the pro­ject might be the time it takes for these EA donors and funds to eval­u­ate your pro­posal and de­cide whether to fund it.

How effec­tive­ness es­ti­mates can be misleading

  • Indi­rect effects. For ex­am­ple, send­ing clothes to Africa can hurt the lo­cal tex­tile in­dus­try and cause peo­ple to lose their jobs. Sav­ing hu­man lives can in­crease the hu­man pop­u­la­tion, which can in­crease pol­lu­tion and an­i­mal product con­sump­tion. Some ways to han­dle in­di­rect effects are dis­cussed in Hur­ford (2016).

    • Effects on the long-term fu­ture are es­pe­cially difficult to pre­dict, but in many cases they could po­ten­tially be more im­por­tant than di­rect effects.

    • The value of in­for­ma­tion/​learn­ing from pur­su­ing an in­ter­ven­tion is usu­ally not taken into ac­count be­cause it’s difficult to quan­tify. Meth­ods of an­a­lyz­ing it are re­viewed in Wil­son (2015).

  • Limited scope. Nor­mally only the out­comes for in­di­vi­d­u­als di­rectly af­fected are mea­sured, whereas the wellbe­ing of oth­ers (fam­ily, friends, car­ers, broader so­ciety, and differ­ent species) also mat­ters.

  • Over-op­ti­miz­ing for a suc­cess met­ric rather than real im­pact. Sup­pose a home­less­ness char­ity has a suc­cess met­ric of re­duc­ing the num­ber of home­less peo­ple in an area. It could sim­ply trans­port lo­cal home­less peo­ple into an­other city where they are still left home­less. De­spite the fact that the char­ity would have no pos­i­tive im­pact, it would ap­pear to be very cost-effec­tive ac­cord­ing to its suc­cess met­ric.

  • Coun­ter­fac­tu­als. Some of the im­pacts would have hap­pened any­way. E.g., sup­pose a char­ity dis­tributes medicine that peo­ple would have bought for them­selves if they weren’t given it for free. While the effect of the medicine might be large, the real coun­ter­fac­tual im­pact of the char­ity is sav­ing the peo­ple the money that they would have used to buy that medicine.[2]

    • Another pos­si­bil­ity is that an­other char­ity would have dis­tributed the same medicine to the same peo­ple, and now that char­ity uses its re­sources for some­thing less effec­tive.

  • Con­flat­ing ex­pected value es­ti­mates with effec­tive­ness es­ti­mates. There is a differ­ence be­tween a 50% chance to save 10 chil­dren, and a 100% chance to save 5 chil­dren. Es­ti­mates some­times don’t make a clear dis­tinc­tion.

  • Diminish­ing/​ac­cel­er­at­ing re­turns, room for more fund­ing. If you es­ti­mate the im­pact of the char­ity and di­vide it by its bud­get, you get the cost-effec­tive­ness of an av­er­age dol­lar spent by the char­ity. It shouldn’t be con­fused with the marginal cost-effec­tive­ness of an ad­di­tional donated dol­lar. They can differ for a va­ri­ety of rea­sons. For ex­am­ple:

    • A limited num­ber of good op­por­tu­ni­ties. A char­ity that dis­tributes medicine might be cost-effec­tive on av­er­age be­cause it does most of the dis­tri­bu­tions in ar­eas with a high prevalence of the tar­get dis­ease. How­ever, it doesn’t fol­low that an ad­di­tional dona­tion to the char­ity will be cost-effec­tive be­cause it might fund a dis­tri­bu­tion in an area with a lower prevalence rate.

    • A char­ity is tal­ent-con­strained (rather than fund­ing-con­strained). That is, a char­ity may be un­able to find peo­ple to hire for po­si­tions that would al­low it to use more money effec­tively.

  • Mo­ral is­sues.

    • Fair­ness and health equity. Cost-effec­tive­ness es­ti­mates typ­i­cally treat all health gains as equal. How­ever, many think that pri­or­ity should be given to those with se­vere health con­di­tions and in dis­ad­van­taged com­mu­ni­ties, even if it leads to less over­all de­cline in suffer­ing or ill­ness (Nord, 2005, Cook­son et al. (2017), Kamm (2015)).

    • Mo­rally ques­tion­able means. E.g., a cor­po­rate cam­paign or a lob­by­ing effort could be more effec­tive if it em­ploys tac­tics that in­volve ly­ing, black­mail, or bribing. How­ever, many (if not most) peo­ple find such ac­tions un­ac­cept­able, even if they lead to pos­i­tive con­se­quences. Since cost-effec­tive­ness es­ti­mates only in­form us about the con­se­quences, they may provide in­com­plete in­for­ma­tion for such peo­ple.

    • Sub­jec­tive moral as­sump­tions in met­rics. To com­pare char­i­ties that pur­sue differ­ent in­ter­ven­tions, some char­ity eval­u­a­tors as­sign sub­jec­tive moral weights to var­i­ous out­comes. E.g., GiveWell as­sumes that the “value of avert­ing the death of an in­di­vi­d­ual un­der 5” is 47 times larger than the value of “dou­bling con­sump­tion for one per­son for one year.” Read­ers who would use differ­ent moral weights may be mis­lead by re­sults of such es­ti­mates if they don’t ex­am­ine such sub­jec­tive as­sump­tions and only look at the re­sults. GiveWell ex­plains their ap­proaches to moral weights in GiveWell (2017).

  • Health in­ter­ven­tions are of­ten mea­sured in dis­abil­ity-ad­justed life-years (DALYs) or qual­ity-ad­justed life-years (QALYs). Th­ese can make analy­ses mis­lead­ing, es­pe­cially when peo­ple treat them as if they mea­sure all that mat­ters. For ex­am­ple:

    • DALYs and QALYs give no weight to hap­piness be­yond re­lief from ill­ness or dis­abil­ity. E.g., an in­ter­ven­tion that in­creases the hap­piness of men­tally healthy peo­ple would reg­ister zero benefit.

    • The ‘bad­ness’ of each health state is nor­mally mea­sured by ask­ing mem­bers of the gen­eral pub­lic how bad they imag­ine them to be, not us­ing the ex­pe­rience of peo­ple with the rele­vant con­di­tions. Con­se­quently, mis­per­cep­tions of the gen­eral pub­lic can skew the re­sults. E.g., some schol­ars claim that peo­ple tend to over­es­ti­mate the suffer­ing caused by most phys­i­cal health con­di­tions, while un­der­es­ti­mat­ing some men­tal di­s­or­ders (Dolan & Kah­ne­man, 2007; Pyne et al., 2009; Karimi et al., 2017).

    • DALYs and QALYs trade off length and qual­ity of life. This al­lows com­par­i­sons of differ­ent kinds of in­ter­ven­tions, but can ob­scure im­por­tant differ­ences (Far­quhar and Owen Cot­ton-Bar­ratt (2015)).

Other

  • An es­ti­mate is for a spe­cific situ­a­tion and is not gen­er­al­iz­able to other con­texts. E.g., just be­cause an in­ter­ven­tion was cost-effec­tive in one coun­try, doesn’t mean it will be cost-effec­tive in an­other. See more on this in Vi­valt (2019) and an 80,000 Hours Pod­cast with the au­thor. Ac­cord­ing to her find­ings, this is a big­ger is­sue than one might ex­pect.

  • Es­ti­mates based on past data might not be in­dica­tive of the cost-effec­tive­ness in the fu­ture:

    • This can be par­tic­u­larly mis­lead­ing if you only es­ti­mate the cost-effec­tive­ness of one par­tic­u­lar pe­riod which is atyp­i­cal. For ex­am­ple, you es­ti­mate the cost-effec­tive­ness of giv­ing medicine to ev­ery­one dur­ing an epi­demic. Once, the epi­demic passes, the cost-effec­tive­ness will be differ­ent. This may have hap­pened to a de­gree with effec­tive­ness es­ti­mates of de­worm­ing.

    • If the past cost-effec­tive­ness is un­ex­pected (e.g., very high), we may ex­pect re­gres­sion to the mean.

  • Bi­ased cre­ators. It can be use­ful to think about the ways in which the cre­ator(s) of an es­ti­mate might have been bi­ased and how it could have im­pacted the re­sults. For ex­am­ple:

    • A char­ity might (in­ten­tion­ally or not) over­es­ti­mate its own im­pact out of the de­sire to get more fund­ing. This is even more likely when you con­sider that em­ploy­ees of a char­ity might be work­ing for it be­cause they are un­usu­ally ex­cited about the char­ity’s in­ter­ven­tions. Even if the es­ti­mate is done by a third party, it is usu­ally based on the in­for­ma­tion that a char­ity pro­vides, and char­i­ties are more likely to pre­sent in­for­ma­tion that shows them in a pos­i­tive light.

    • A re­searcher cre­at­ing the es­ti­mate may want to find that the in­ter­ven­tion is effec­tive be­cause that would lead to their work be­ing cel­e­brated more.

  • Publi­ca­tion bias. Es­ti­ma­tions that find that some in­ter­ven­tion is cost-effec­tive are more likely to be pub­lished and cited. This can lead to situ­a­tions where in­ter­ven­tions seem to have more ev­i­dence in fa­vor of them than they should be­cause only the es­ti­ma­tions that found it to be im­pact­ful were pub­lished.

  • Bias to­wards mea­surable re­sults. If a char­ity’s im­pact is difficult to mea­sure, it may have a mis­lead­ingly low es­ti­mated cost-effec­tive­ness, or there may be no es­ti­mate of its effects at all. Hence, if we choose a char­ity that has the high­est es­ti­mated cost-effec­tive­ness, our se­lec­tion is bi­ased to­wards char­i­ties whose effects are eas­ier to mea­sure.

  • Op­ti­mizer’s Curse. Sup­pose you weigh ten iden­ti­cal items with very in­ac­cu­rate scales. The item that is the heav­iest ac­cord­ing to your re­sults is sim­ply the item whose weight was the most over­es­ti­mated by the scales. Now sup­pose the items are similar but not iden­ti­cal. The item that is the heav­iest ac­cord­ing to the scales is also the item whose weight is most likely an over­es­ti­mate.

    Similarly, sup­pose that you make very ap­prox­i­mate cost-effec­tive­ness es­ti­mates of ten differ­ent char­i­ties. The char­ity that seems the most cost-effec­tive ac­cord­ing to your es­ti­mates could seem that way only be­cause you over­es­ti­mated its cost-effec­tive­ness, not be­cause it is ac­tu­ally more cost-effec­tive than oth­ers.

    Con­se­quently, even if we are un­bi­ased in our es­ti­mates, we might be too op­ti­mistic about char­i­ties or ac­tivi­ties that seem the most cost-effec­tive. I think this is part of the rea­son why some peo­ple find that “re­gard­less of the cause within which one in­ves­ti­gates giv­ing op­por­tu­ni­ties, there’s a strong ten­dency for giv­ing op­por­tu­ni­ties to ap­pear pro­gres­sively less promis­ing as one learns more.” The more un­cer­tain cost-effec­tive­ness es­ti­mates are, the stronger the effect of op­ti­mizer’s curse is. Hence we should pre­fer in­ter­ven­tions whose cost-effec­tive­ness es­ti­mates are more ro­bust. More on this can be read in Karnofsky (2016).

  • Some re­sults can be very sen­si­tive to one or more un­cer­tain pa­ram­e­ters and con­se­quently, seem more ro­bust than they are. To un­cover this, a sen­si­tivity anal­y­sis or un­cer­tainty anal­y­sis can be performed.

  • To cor­rectly in­ter­pret cost-effec­tive­ness es­ti­mates, it’s im­por­tant to know whether time dis­count­ing was ap­plied. Time dis­count­ing makes cur­rent costs and benefits worth more than those oc­cur­ring in the fu­ture be­cause:

    • There is a de­sire to en­joy the benefits now rather than in the fu­ture.

    • There are op­por­tu­nity costs of spend­ing money now. E.g., if the money was in­vested rather than spent, it would likely be worth more in a cou­ple of years.

  • Model un­cer­tainty. That is, un­cer­tainty due to nec­es­sary sim­plifi­ca­tion of real-world pro­cesses, mis­speci­fi­ca­tion of the model struc­ture, model mi­suse, etc. There are prob­a­bly some more for­mal meth­ods to re­duce model un­cer­tainty, but per­son­ally, I find it use­ful to cre­ate sev­eral differ­ent mod­els and com­pare their re­sults. If they all ar­rive at a similar re­sult in differ­ent ways, you can be more con­fi­dent about the re­sult. The more differ­ent the mod­els are, the bet­ter.

  • Wrong fac­tual as­sump­tions. E.g., when es­ti­mat­ing the cost-effec­tive­ness of dis­tribut­ing bed­nets, it would be a mis­take to as­sume that all the peo­ple who re­ceive them would use them cor­rectly.

  • Mis­takes in calcu­la­tions. This in­cludes mis­takes in stud­ies that an es­ti­mate de­pends on. As ex­plained in Les­son #1 in Hur­ford and Davis (2018), such mis­takes hap­pen more of­ten than one might think.

Com­pli­ca­tions of es­ti­mat­ing the im­pact of donated money

  • Fun­gi­bil­ity. If a char­ity does mul­ti­ple pro­grams, donat­ing to it could fail to in­crease spend­ing on the pro­gram you want to sup­port, even if you re­strict your dona­tion. Sup­pose a char­ity was plan­ning to spend $1 mil­lion of its un­re­stricted funds on a pro­gram. If you donate $1,000 and re­strict it to that pro­gram, the char­ity could still spend ex­actly $1 mil­lion on the pro­gram and use an ad­di­tional $1,000 of un­re­stricted funds on other pro­grams.

  • Re­place­abil­ity of dona­tions. It can some­times be use­ful to ask your­self: “Would some­one else have fulfilled char­ity X’s fund­ing gap if I hadn’t?” Note that if some­one else would have donated to char­ity X, they may not have donated money to char­ity Y (their sec­ond op­tion). That said, I think it’s easy to think about this too much. Imag­ine if all donors in EA were only look­ing for op­por­tu­ni­ties that no one else would fund. When an ob­vi­ously promis­ing EA char­ity asks for money, all donors might wait un­til the last minute, think­ing that some other donor might fund it in­stead of them. That would cost more time and effort for both the char­ity and the po­ten­tial donors. To avoid this, donors need to co­or­di­nate.

  • Tak­ing dona­tion match­ing liter­ally. A lot of the time, when some­one claims they would match dona­tions to some char­ity, they would have donated the money that would be used for match­ing any­way, pos­si­bly even to the same char­ity. This is not always the case though (e.g., em­ploy­ers match­ing dona­tions to any char­ity).

  • In­fluenc­ing other donors. For ex­am­ple:

    • Re­ceiv­ing a grant from a re­spected foun­da­tion can in­crease the le­gi­t­i­macy and the pro­file of a pro­ject and make other fun­ders more will­ing to donate to it.

    • Shar­ing news about in­di­vi­d­ual dona­tions can in­fluence friends to donate as well (Hur­ford (2014)). Note that the strength of this effect partly de­pends on the char­ity you donate to.

    • Donors can make moral trades to achieve out­comes that are bet­ter for ev­ery­one in­volved. E.g., sup­pose one donor wants to donate $1,000 to a gun con­trol char­ity, and an­other donor wants to donate $1,000 to a gun rights char­ity. Th­ese dona­tions may can­cel each other out in terms of ex­pected im­pact. Donors could agree to donate to a char­ity they both find valuable (e.g., anti-poverty), on the con­di­tion that the other one does the same.

  • In­fluenc­ing the char­ity. For ex­am­ple:

    • Char­i­ties may try to do more ac­tivi­ties that ap­peal to their ex­ist­ing and po­ten­tial fun­ders to se­cure ad­di­tional fund­ing.

    • Let­ting a char­ity eval­u­a­tor (e.g., GiveWell, An­i­mal Char­ity Eval­u­a­tors) or a fund (e.g., EA Funds) to di­rect your dona­tion sig­nals to char­i­ties the im­por­tance of these eval­u­a­tors. It can in­cen­tivize char­i­ties to co­op­er­ate with eval­u­a­tors dur­ing eval­u­a­tions and try to be bet­ter ac­cord­ing to the met­rics that eval­u­a­tors mea­sure.

    • Fun­ders can con­sciously in­fluence the di­rec­tion of a char­ity they fund. See more on this in Karnofsky (2015).

  • There are many im­por­tant con­sid­er­a­tions about whether to donate now rather than later. See Wise (2013) for a sum­mary. For ex­am­ple, it’s im­por­tant to re­mem­ber that if the money was in­vested, it would likely have more value in the fu­ture.

  • Tax de­ductibil­ity. If you give while you are earn­ing money, in some coun­tries (e.g. U.S., UK, Canada) your dona­tions to char­i­ties that are reg­istered in your coun­try are tax de­ductible. This means that the gov­ern­ment effec­tively gives more to the same char­ity. E.g. see de­ductibil­ity of ACE-recom­mended char­i­ties here. If you are donat­ing money to a char­ity reg­istered in an­other coun­try, there might still be ways to make it tax de­ductible. E.g., by donat­ing through or­ga­ni­za­tions like RC For­ward (which is made for Cana­dian donors), or us­ing Dona­tion Swap.

I’m a re­search an­a­lyst at Re­think Pri­ori­ties. The views ex­pressed here are my own and do not nec­es­sar­ily re­flect the views of Re­think Pri­ori­ties.

Author: Saulius Šimčikas. Thanks to Ash Had­jon-Whit­mey, Derek Foster, and Peter Hur­ford for re­view­ing drafts of this post. Also, thanks to Derek Foster for con­tribut­ing to some parts of the text.

References

Byford, S., & Raftery, J. (1998). Per­spec­tives in eco­nomic eval­u­a­tion. Bmj, 316(7143), 1529-1530.

Cook­son, R., Mirel­man, A. J., Griffin, S., Asaria, M., Dawk­ins, B., Norheim, O. F., … & Culyer, A. J. (2017). Us­ing cost-effec­tive­ness anal­y­sis to ad­dress health equity con­cerns. Value in Health, 20(2), 206-212.

Dolan, P., & Kah­ne­man, D. (2008). In­ter­pre­ta­tions of util­ity and their im­pli­ca­tions for the val­u­a­tion of health. The eco­nomic jour­nal, 118(525), 215-234.

Far­quhar, S., Cot­ton-Bar­ratt, O. (2015). Break­ing DALYs down into YLDs and YLLs for in­ter­ven­tion comparison

GiveWell. (2017). Ap­proaches to Mo­ral Weights: How GiveWell Com­pares to Other Actors

Hur­ford, P. (2014). To in­spire peo­ple to give, be pub­lic about your giv­ing.

Hur­ford, P. (2016). Five Ways to Han­dle Flow-Through Effects

Hur­ford, P., Davis, M. A. (2018). What did we take away from our work on vaccines

Kamm, F. (2015). Cost effec­tive­ness anal­y­sis and fair­ness. Jour­nal of Prac­ti­cal Ethics, 3(1).

Karimi, M., Bra­zier, J., & Paisley, S. (2017). Are prefer­ences over health states in­formed?. Health and qual­ity of life out­comes, 15(1), 105.

Karnofsky, H. (2011). Lev­er­age in charity

Karnofsky, H. (2015). Key Ques­tions about Philan­thropy, Part 1: What is the Role of a Fun­der?

Karnofsky, H. (2016). Why we can’t take ex­pected value es­ti­mates liter­ally (even when they’re un­bi­ased)

Nord, E. (2005). Con­cerns for the worse off: fair in­nings ver­sus sever­ity. So­cial sci­ence & medicine, 60(2), 257-263.

Pyne, J. M., Fort­ney, J. C., Tri­pathi, S., Feeny, D., Ubel, P., & Bra­zier, J. (2009). How bad is de­pres­sion? Prefer­ence score es­ti­mates from de­pressed pa­tients and the gen­eral pop­u­la­tion. Health Ser­vices Re­search, 44(4), 1406-1423.

Sauber, J. (2008). Put your money where your heart is

Sethu, H. (2018). How rank­ing of ad­vo­cacy strate­gies can mislead

Snow­den, J. (2018). Re­vis­it­ing leverage

Vi­valt, E. (2019). How Much Can We Gen­er­al­ize from Im­pact Eval­u­a­tions?

Wiblin, R. (2017). Most peo­ple re­port be­liev­ing it’s in­cred­ibly cheap to save lives in the de­vel­op­ing world

Wil­son, E. C. (2015). A prac­ti­cal guide to value of in­for­ma­tion anal­y­sis. Phar­ma­coeco­nomics, 33(2), 105-121.

Wise, J. (2013). Giv­ing now vs. later: a sum­mary.


  1. I’ve heard the claim that Noth­ing But Nets used to say that it costs $10 to provide a bed­net be­cause it’s an easy num­ber to re­mem­ber and think about, de­spite the fact that it costs less. Ac­cord­ing to GiveWell, on av­er­age the to­tal cost to pur­chase, dis­tribute, and fol­low up on the dis­tri­bu­tion of a bed­net funded by Against Malaria Foun­da­tion is $4.53. ↩︎

  2. Another ex­am­ple of coun­ter­fac­tu­als: sup­pose there is a very cost-effec­tive stall that gives peo­ple ve­gan leaflets. Some­one opens an­other iden­ti­cal stall right next to it. Half of the peo­ple who would have gone to the old stall now go to the new one. The new stall doesn’t at­tract any peo­ple who wouldn’t have been at­tracted any­way so it has zero im­pact. But if you es­ti­mate its effec­tive­ness ig­nor­ing this cir­cum­stance, it can still be high ↩︎