GiveWell money moved forecasts and implications

vipulnaik19 Dec 2015 20:22 UTC
8 points
8 comments23 min readEA link

Note: See UPDATES 1, 2, 3, 4, and 5 at the bottom of the post.

This post assumes some familiarity with the work of charity evaluator GiveWell, but here’s a quick backgrounder for those not familiar: GiveWell lists a small set of top charities to donate to, and publishes detailed reviews of each. The list for 2015 comprises Against Malaria Foundation (malaria bednet distribution), GiveDirectly (unconditional cash transfers), Deworm the World Initiative (deworming), and Schistosomiasis Control Initiative (deworming). GiveWell’s recommendations have acquired the status of a gold standard for conventional, unattached donors, and GiveWell has a particularly strong reputation among many of the people who read this site and identify as effective altruists. GiveWell also works closely with Good Ventures, the private foundation of Facebook co-founder Dustin Moskovitz and his wife Cari Tuna. Good Ventures has made significant donations to GiveWell’s top charities, mostly at GiveWell’s suggestion, with the amount reaching $44.4 million (plus $1 million to standout charities, plus a separate three-year $25 million grant to GiveDirectly) in 2015.

In a recent post on the Effective Altruists Facebook group critiquing what I saw as claims of take-off exponential growth of the EA movement, I noted some estimates of what I believe GiveWell’s money moved will be for 2015 and the years to come. In this post, I want to develop and refine those estimates more.

TL;DR

[Added at request of commenters, even though I’m skeptical that this TL;DR can accurately capture the entirety of the post].

For simplicity, this TL;DR does not describe the intervals of uncertainty. For those you’ll have to read the whole post (or the appropriate section thereof).

  • Forecasting the amount of money moved by GiveWell helps donors understand their marginal impact. It also helps recipient organizations and fundraisers.

  • GiveWell and its recommended charities are currently a very small fraction of the sectors within which they operate. There are no hard ceilings being hit just yet.

  • I estimate GiveWell’s non-Good Ventures money moved to grow by a little over 40% from 2014 to 2015, though there is considerable uncertainty. I believe that Elie’s estimate of well over 20 million dollars in money moved is a tad overconfident.

  • I believe that GiveWell’s growth in money moved this year will come partly from an increase in the number of donors (about 20%) and partly from an increase in the amount of money donated per donor (about 20%). In future years, growth will be concentrated more in terms of money moved per donor rather than number of donors, though both will grow modestly.

  • I see three key determinants of GiveWell’s money moved: discovery/​brand awareness, conversion, and retention/​upsell. The biggest constraint on GiveWell’s money moved is poor conversion. The biggest source of near-term growth is retention/​upsell. While discovery/​brand awareness is low, increasing it significantly won’t help much since conversion is a bigger bottleneck and will be an even bigger bottleneck if brand awareness were suddenly increased.

  • GiveWell’s choice of top charities itself affects its money moved, and the 2015 list of top charities is most conducive to moving more money.

  • The long-run solution to the conversion problem is expanding the range of recommended charities, but this is unlikely to happen in a meaningful way for at least the next two years.

Why is this important?

GiveWell was an early proponent of the idea of room for more funding, and of the closely related concept of marginal impact, i.e., what effect does your marginal contribution have? In order to best estimate an organization’s room for more funding, GiveWell tries to estimate how much the organization will raise from non-GiveWell donors, and also takes into account the amount it is recommending that Good Ventures donate. However, GiveWell doesn’t provide official guidance on how much money it expects GiveWell-influenced donors other than Good Ventures to give. I believe that carrying out this estimation is important for a few reasons:

  • It tells donors the relevant margin at which their donation is actually happening: One of the biggest determinants of marginal impact is how much money would be moved in the absence of your donation. [Mathematically, if we think of marginal impact in terms of the derivative of the impact function, then knowing how much money would be moved otherwise tells us at what point we need to take the derivative.]

  • It helps recipient organizations plan better for the future: Knowing how much money they expect to raise can help the organizations budget appropriately for staff and programs. Overestimates could lead to taking up too many programs that they then cannot follow through, whereas underestimates can lead to significant opportunities being underfilled.

  • It helps fundraising organizations and individuals understand the margins at which they are operating

It could still be the case that while accurate answers are helpful, wild speculation isn’t. But getting to accurate answers itself requires practice at estimation and forecasting, even if initial forecasts are of low quality. This post is an attempt at initial forecasts, along with a walkthrough of the reasoning used to generate the forecasts. The forecasts might well be of low quality, but I hope people find the exercise illuminative.

In addition to the value of the final forecasts, the reasoning used in arriving at the forecasts helps shed light on other facets important to donors, recipient organizations, charity evaluators, and fundraising organizations.

The amounts of money moved are now sufficiently significant that this sort of analysis is worthwhile

As GiveWell’s donor base has increased in size and its annual money moved has hit eight digits, estimation makes more and more sense. A few reasons:

  • With more donors and larger amounts, more accurate forecasting becomes more possible (with the caveat that a billionaire here or there could tip the scales). Explicitly, with a larger number of donors, the law of large numbers starts kicking in. Moreover, if donors are donating larger amounts, the proportional variation in the amounts they donate is likely to be less (because the difference between donating $10,000 and $20,000 for a donor is more significant than the difference between donating $10 and $20).

  • The recipient organizations have now demonstrated room for more funding according to GiveWell, and are planning their expansion operations specifically with the expectation of raising money from GiveWell. In particular, Against Malaria Foundation and GiveDirectly seem to be expanding rapidly with GiveWell playing a key role in their growth. While a lot of GiveWell’s contribution comes through Good Ventures grants, individual donors play a nontrivial part for both organizations.

  • GiveWell itself holds considerable promise as a pioneer of the idea of effective altruism (broadly construed, not necessarily to be equated with the eponymous social movement or this website). Understanding the growth of its money moved can shed insight into just how far the idea of effective altruism can go in the world of philanthropy.

An overall sense of magnitude

To give some sense of the numbers:

  • Money donated to charity in the United States is about 350 billion dollars a year, which is a little over 2% of US GDP, which in turn is around 15-20% of Gross World Product.

  • GiveWell’s non-Good Ventures money moved to top charity was $12.7 million-$13.0 million in 2014, which is around 0.004% of total money moved to charity in the United States. GiveWell doesn’t officially publish distribution of donors by country (I did write to them and they said they currently don’t have that data) but my guess is that over half of its money moved comes from US donors, and possibly a lot more. So, excluding Good Ventures, GiveWell is responsible for moving somewhere between 0.002% and 0.004% of money moved to charity in the United States. Or viewed another way, total money moved to GiveWell-recommended charities in 2014, excluding money from Good Ventures, represented 0.00001% of gross world product.

  • Based on various surveys of EA organizations and data published by EA organizations, my guess is that somewhere between 10% and 30% of money moved by GiveWell was through people who got to GiveWell through the EA movement. I won’t elaborate on this point much here because it’s not the focus of my estimation exercise, but you can check out the 2014 EA Survey results for some example data.

What about GiveWell-recommended charities? Here are some rough estimates I got with a few minutes of Googling; numbers may be far from accurate.

  • Against Malaria Foundation works in the domain of bednet distribution. Prior to 2015, it had raised less than 20 million dollars in total (a large part of it from Good Ventures and others influenced by GiveWell’s recommendations, plus some influenced by Giving What We Can recommendations). With the new influx of Good Ventures and GiveWell-influenced money, its annual revenue could be in the tens of millions of dollars. For comparison, the Global Fund to Fight AIDS, Tuberculosis, and Malaria, a Gates Foundation-backed and internationally supported grantmaker, spends between 2 and 5 billion a year, and devotes about 20% of its expenses to fighting malaria (according to GiveWell’s page on them). The estimate would therefore be that it spends between 400 million and 1 billion dollars on anti-malaria programs. Thus, prior to 2015, AMF accounted for less than 1% of global philanthropic expenditure against malaria, whereas it could now go up to anywhere between 2% and 10% (depending on how the other numbers shake out).

  • GiveDirectly is the only non-governmental organization devoted to large-scale unconditional cash transfers, and it’s not immediately clear what to compare it against. As a fraction of the economy of the countries it operates in, it’s small but not as insignificant as it is for the sending economies. For instance, its annual expenses in Kenya are under 0.1% of Kenya’s nominal GDP, but more than 0.01%.

  • I wasn’t able to get good estimates of the total money spent on deworming, but since SCI and DtWI (and in particular DtWI) work with governments, the money leveraged by their efforts is significantly more than their own money spent.

These broad estimates show that in terms of money moved, fraction of the philanthropic sector, fraction of the wealth in donor groups, and fraction of the resources available to recipient groups, GiveWell and its top charities are still quite small. This suggests that there is considerable upward room to grow without hitting hard limits. Barriers to growth can of course arise in a variety of ways, but even so, there is no strong evidence against the proposition that, over the next few decades, GiveWell’s annual non-Good Ventures money moved can grow by 1-2 orders of magnitude, while still remaining a small fraction of the quantities involved. For instance, GiveWell cited the Money for Good study (funded to help rethink the Nonprofit Marketplace Initiative of the Hewlett Foundation) and noted that, based on interpretation, somewhere between $550 million and $4 billion in annual giving in the United States could be influenced by GiveWell-like resources. This would be somewhere between 0.2% and a little over 1% of annual giving in the United States, and represents a 40-200X increase in the non-Good Ventures money moved by GiveWell.

Money moved estimates

A few relevant pieces of information:

My estimates for the non-Good Ventures money moved by GiveWell have changed slightly since my Effective Altruists Facebook group post; here are the revised estimates.

  • Estimate for 2015: Median estimate of $18.7 million, 50% probability estimate of $16.2 million to $24.5 million.

  • Estimate for 2016: Median estimate of $25.3 million, 50% probability estimate of $17.5 million to $37.5 million.

  • Estimate for 2017: Median estimate of $30.1 million, 50% probability estimate of $19.5 million to $58 million.

In contrast, according to an October 2015 board meeting, Elie Hassenfeld of GiveWell was confident that it would move over 20 million dollars in 2015. I believe this confidence is misplaced, for reasons I discuss below (with that said, the 20 million+ scenario is about 40% likely, so it’s not way off from my estimate). But it is also possible that they have insider information that I lack access to. That is part of the reason for my upward revision of my estimates since the Facebook post.

For the 2015 money moved estimate, here are some more details before I delve into my reasons.

Here is the distribution by percentile:

  • 1st percentile: $8.8 million

  • 2.5th percentile: $11.6 million

  • 10th percentile: $14.7 million

  • 25th percentile: $16.2 million

  • 50th percentile: $18.7 million

  • 75th percentile: $24.5 million

  • 90th percentile: $28.6 million

  • 97.5th percentile: $33.1 million

  • 99th percentile: $45 million

Here is the distribution for top charities (50% probability for each):

  • Against Malaria Foundation: $8 million - $17 million

  • GiveDirectly: $1 million - $4 million

  • Schistosomiasis Control Initiative: $1 million - $4 million

  • Deworm the World Initiative: $250,000 - $4 million

The low lower-end estimate for Deworm the World Initiative arises from the fact that it raised less money last year, and the increased endorsement of the organization happened at the same time that GiveWell doubled down on AMF as a top pick. Moreover, Good Ventures has agreed to fill its entire Level 1 funding gap, making it less attractive for individual donors who take GiveWell’s analysis serious.

Here’s the size distribution of donors for 2012, 2013, and 2014 (from the same GiveWell blog post).

GiveWell money moved by size bucket

We see that 2013 was a significant increase year for number of donors, whereas 2014 was an year when the number of donors didn’t increase much (there was a lot of churn) but the average money moved per donor increased significantly. I believe that the situation for 2015 will look positive on both fronts: an increase in the number of donors, and an increase in the money moved per donor. My median estimates for the growth on both fronts is around 20%. Explicitly, my 50% interval for the number of donors in 2015 is 9,500-12,500. My 50% interval estimate for the money moved per donor is $1,500-2,000.

Explanation of the lower and upper end estimates

Explanation of the lower end: Since GiveWell has already moved $5.56 million in money in the first two quarters of 2015, and Elie was reasonably confident of the 20 million figure for the whole year, I expect that donations did not completely stop in Q3 of 2015. Therefore, I expect that just the first three quarters alone would have accounted for at least $6.5 million in donations. That leaves about $2.3 million in potential donations for the end of year, which is fairly easy to achieve with existing pledges. Therefore, I expect a probability of about 1% of missing the $8.8 million mark. That 1% covers major scandals at GiveWell and its recommended charities, or other unforeseen problems.

Explanation of the upper end: The upper end of estimates is mainly governed by factors such as wealthy donors, highly successful fundraising groups or celebrity endorsements or media publicity, and other unforeseen factors.

With the lower and upper end taken care of, I’ll focus on explaining the middle. Specifically, I want to explain why my median estimate is lower than the “well over 20 million” that Elie and Holden have put out.

A quick theoretical model for how GiveWell moves money

One thing seems reasonably clear to me from many different sources of anecdotal evidence:

Here’s my description of the most typical pathway for a GiveWell donor:

  • The person discovers GiveWell through some method such as search, friend referral, social media, or a website or person they follow online.

  • The person checks out GiveWell’s site, and hears positive things about GiveWell.

  • When Giving Season (concentrated around December) arrives, the person checks out GiveWell’s recommendations, and in some cases makes a donation to a GiveWell-recommended charity on GiveWell’s recommendation.

Based on this model, a few key determinants of GiveWell’s money moved stand out:

  • Discovery and brand awareness: How many new people are discovering GiveWell? What is the total number of people who know about GiveWell and what it stands for? I did a Facebook post on a SurveyMonkey Audience survey of brand awareness of various philanthropic organizations. GiveWell did decently okay but way below the Gates Foundation, and also below the relatively recent Chan Zuckerberg Initiative as well as Charity Navigator.

  • Conversion: For the people who have heard of GiveWell, how likely is the person to go through GiveWell’s recommendations when considering a donation, and how likely are they to actually follow through? This to me seems like the biggest filter, since for many people, GiveWell’s philosophy and areas of focus just don’t resonate. They just aren’t interested in the causes GiveWell’s top charities are in. Interestingly, this includes many in the EA community (such as those who find animal welfare or existential risk way more important) plus more conventional donors with their own causes of interest (environmentalists, donors who prefer keeping money local, etc.) GiveWell is not an impulse buy. GiveWell doesn’t move much money from people who casually Google for where to donate money, land on GiveWell’s site, and immediately buy. Fundamentally, since GiveWell is asking people to donate to charities that they probably hadn’t heard of and weren’t considering explicitly, it takes some time to get them to actually donate. It also takes time to understand the broad philosophy motivating GiveWell’s decisions. This is in contrast with, say, Charity Navigator, that you might go to in order to check out a charity you are already interested in, and then, after reading the review, decide to donate or not donate (it’s hard to say whether Charity Navigator caused the donation, but it did influence the donation decision somewhat). For more on a comparison between the immediate appeal of GiveWell and Charity Navigator, see my Facebook thread discussing the results of a SurveyMonkey Audience survey.

  • Retention and upsell: For the people who are already donating to GiveWell, how likely are they to return next year, and will the amount they donate keep going up? While this partly depends on how good they find GiveWell, it also depends on how much money they’re making. If GiveWell captures potential donors when they are young and starting their careers, its donor base will see an increase in per capita income and therefore will probably move larger sums of money per donor.

At the beginning of my investigation, I believed that the focus should be on discovery and conversion. This partly stems from my pro-web bias, where short-run pageviews are both the most important and fairly decent proxies for long-run pageviews. However, after I dug deeper into the data, I now believe that the main driver of GiveWell’s growth in the near term is retention and upsell. My upshot:

  • Discovery and brand awareness: GiveWell still has a long ways to go before people have vaguely heard of it and are vaguely aware of it. However, progress in recent years has been lukewarm at best. A lot of GiveWell’s media coverage to date was back in 2008 in the wake of the MetaFilter scandal (!), and even in recent years, GiveWell received more attention on some metrics in 2011 and 2012 than it does now. With that said, there seems to have been one discrete but permanent bump in July 2014. 2015 appears to be mostly flat along some metrics, but rising along others, painting a decidedly mixed picture.

  • Conversion: I believe 2015 will see higher conversion than previous years because of more careful coordination among GiveWell allies to move money to GiveWell-recommended charities, plus other reasons I discuss below.

  • Retention and upsell: GiveWell seems to have succeeded at the upsell in 2014. This seems to me to be largely a result of GiveWell donors getting richer (and therefore moving up the size buckets), but it could also be due to other factors such as increased confidence in GiveWell recommendations. Partly, this is also a signal of GiveWell not doing that well at attracting newer donors.

Discovery and brand awareness

While we don’t have money moved or GiveWell web traffic data from August 2015 onward, we do have other pieces of data that we can use to understand GiveWell’s growth.

I combine pageview counts for the Wikipedia page about GiveWell with GiveWell’s spreadsheet with data on the number of unique visitors. Due to month-to-month variation in traffic I look at year-over-year growth for a three-month total ending at the current month. So the first row is the percentage increase between the September to November 2014 period and the September to November 2015 period.

Month Views of page GiveWell Year-over-year growth (three-month total ending at the current month) Unique visitors (GA) Year-over-year growth (three month total ending at the current month)
201511 2126 −3.14%
201510 1783 −5.75%
201509 1735 3.40%
201508 1840 31.09% 58,371 14.51%
201507 1929 53.40% 58,997 66.47%
201506 1966 60.89% 69,496 58.21%
201505 1695 44.61% 91,280 41.86%
201504 1893 49.77% 69,479 12.29%
201503 1722 60.99% 60,762 6.29%
201502 1660 37.24% 51,995 10.26%
201501 1884 25.99% 67,305 −5.05%
201412 2316 18.10% 145,927 −11.95%
201411 1928 41.12% 68,231 −24.74%
201410 1910 45.40% 64,799 −8.76%
201409 1989 46.04% 56,306 −9.58%
201408 1786 8.53% 76,064 −15.90%
201407 1548 −12.95% 39,238 −27.20%
201406 1041 −20.33% 47,890 −16.99%
201405 1055 −16.96% 44,894 −3.81%
201404 1356 −26.09% 52,754 −1.19%
201403 1261 −38.41% 58,512 −4.87%
201402 905 −32.33% 51,031 18.58%
201401 1105 −25.41% 59,859 49.38%
201312 2260 −17.50% 129,662 75.63%
201311 1499 −13.02% 106,909 89.65%
201310 1452 −2.23% 80,231 101.46%
201309 1178 2.79% 64,443 104.32%
201308 1280 10.59% 71,414 118.98%
201307 1187 22.54% 53,943 92.31%
201306 1564 56.99% 68,686 82.39%
201305 1435 100.64% 58,711 33.86%
201304 1334 169.36% 47,934 32.52%
201303 1653 188.64% 55,700 36.17%
201302 1778 208.71% 60,624 35.46%
201301 1880 179.39% 61,754 4.82%
201212 2652 167.06% 80,480 −8.85%
201211 1989 55.28% 56,205 −21.56%
201210 1675 45.68% 43,695 −17.82%
201209 1083 40.44% 32,758 −20.52%
201208 1241 156.33% 30,810 −13.96%
201207 1222 154.74% 29,325 −3.45%
201206 1182 82.06% 28,476 −7.37%
201205 1012 23.20% 36,494 −4.26%
201204 566 3.39% 31,160 13.80%
201203 626 17.42% 53,627 44.43%
201202 577 47.26% 39,163 110.29%
201201 637 64.83% 37,986 144.78%
201112 830 84.48% 72,603 180.97%
201111 867 167.22% 78,724 249.98%
201110 668 137.05% 46,566 277.10%
201109 1522 133.58% 43,824 308.39%
201108 555 45.10% 40,133 300.78%
201107 448 43.58% 32,925 235.28%
201106 419 66.59% 29,934 206.37%
201105 474 101.24% 34,801 231.79%
201104 623 165.27% 39,044 186.15%
201103 692 58.76% 52,837 82.94%
201102 396 42.95% 17,039 18.61%
201101 479 47.04% 20,671 24.87%
201012 513 124.13% 33,503 47.92%
201011 424 134.43% 23,167 58.11%
201010 345 119.73% 13,763 53.78%
201009 375 89.32% 11,391 50.04%
201008 438 64.15% 9,458 49.54%
201007 268 62.43% 7,771 85.33%
201006 274 58.26% 8,469 118.57%
201005 392 24.16% 12,888 68.58%
201004 244 −12.84% 12,517 69.78%
201003 253 −37.61% 12,776 136.14%
201002 148 −62.03% 12,771 300.57%
201001 586 −74.00% 23,948 435.80%
200912 237 −79.56% 23,319 442.92%
200911 140 −70.74% 14,672 592.54%
200910 195 6.90% 9,623 479.03%
200909 153 39.27% 6,267 489.83%
200908 179 36.61% 6,617 442.80%
200907 239 10.36% 6,191 264.66%
200906 179 −4.96% 4,377 198.79%
200905 157 28.09% 5,149 205.76%
200904 239 50.41% 5,972 112.69%
200903 320 98.74% 11,527 −8.44%
200902 181 292.18% 4,921 −78.52%
200901 1081 4,512
200812 1295 5,555
200811 1328 1,493
200810 175 1,722
200809 165 1,198
200808 153 967
200807 92 1,069
200806 192 1,130
200805 237 2,111
200804 176 1,946
200803 146 3,350
200802 170 5,245
200801 480 14,298
200712 2 50,240

If you dig into the Wikipedia pageviews more at the link, you’ll see there was a sharp increase in the latter half of July 2014. In fact, the date range for the increase can be narrowed to the period between July 11 and August 4. You can also check against stats.grok.se data.

For this reason, we see that three-month periods whose end dates are after August 2014 but before August 2015 show robust year-over-year growth, whereas three-month periods September 2015 onward show anemic growth in Wikipedia views.

On these indicators, 2015 seems to be a year of flatlining. However, Google Trends data suggests robust year-on-year growth from 2014 to 2015, up to and including November and December. I’m not really sure how much relative weight to place on Wikipedia view data versus Google Trends data. Since Google Trends methodology is less transparent, I’m a little more suspicious of it. But on the other hand, Google clearly has much more money and brains behind the quality of the data. Overall I would say the evidence for growth in 2015 (over and above whatever happened in the middle of 2014) is decidedly mixed. Either way, it is unlikely to be the driver of an increase in donations. If discovery and brand awareness were the only thing that I were looking at, I wouldn’t estimate more than 10% increase in annual money moved. But I actually estimate a much greater increase (over 40%, with 20% being due to more donors and 20% being due to more money per donor). Why? Basically, conversion and upsell. Since these two are somewhat hard to separate, I’ll discuss some changes in 2015 that I believe will positively affect both conversion and upsell, without separating conversion from upsell.

I should also mention here some currently fledgeling efforts being made to increase brand awareness of GiveWell and the ideas surrounding it. Gleb Tsipursky wrote a Huffington Post listicle on the 8 Secrets of Savvy Donors, and he reported in an EA Forum post that it increased awareness and also converted at least one moderately influential person. It’s too early to evaluate the significance of these for the 2015 Giving Season.

The role played by the specifics of GiveWell’s recommendations and the rhetoric they use

There is a reasonable argument that the actual list of top charities, and the commentary that GiveWell makes when recommending them, itself affects how much money is moved to them. The top charity list hasn’t changed since the 2014 giving season. But there are a few important differences:

  • For February-November 2014 (the first ten months of the year), the top charity list didn’t include the Against Malaria Foundation. (GiveWell announced in October 2014 that AMF had established room for more funding again, however it officially updated its list only at the end of November). Ceteris paribus, it seems that giving AMF more prominence in recommendations increases money moved, because AMF seems naturally more appealing to GiveWell donors (it’s doing something that saves lives applying a proven intervention). AMF has been in GiveWell’s list for all of 2015, so it’s had more time in the limelight.

  • In its top charity announcement in 2015, GiveWell has explicitly doubled down on AMF, asking donors to direct all their money to AMF. This clarity of message is likely to inspire unattached donors to give more. Note that GiveWell finds room for more funding in all its top picks, so those determined to give to another charity will not be dissuaded from giving entirely.

  • GiveWell has identified larger amounts of room for more funding in all its top charities relative to previous years, and Good Ventures has made substantially larger grants this year than in previous years. This recommendation anchors individual donors to give more and makes them more likely to donate, as long as they feel somewhat positively about at least one of the top charities [On the flip side, it may make individual donors more reluctant to fund something that already has so much money behind it. However, as an empirical matter, this seems a relatively minor concern among GiveWell donors]. This will most significantly affect upsell rather than conversion, but as noted earlier, upsell is a bigger short-run source of increase in money moved than conversion of new donors.

Indeed, as described in the 2015 staff members’ personal donations blog post, most of GiveWell’s staff are donating all or most of their own annual money to AMF, with Holden and Elie in particular donating all their money to AMF. In contrast, in previous years, GiveWell’s own staff members were more skeptical of the ranking or relative allocation suggested by GiveWell, and opted for a different mix. GiveWell’s staff also expressed more confidence in 2015 (justified or not) in the rigor of the process used to evaluate top charities and their room for more funding. For instance, co-founder and co-executive director Elie wrote:

Last year, we gave significantly more to GiveDirectly than GiveWell recommended because I put less weight on cost-effectiveness analysis than other staff. I remain ambivalent about putting so much weight on cost-effectiveness estimates because they are subject to numerous debatable assumptions, but I’ve changed my mind based on work we did over the past few months. We put significant effort into debating our cost-effectiveness model (.xls) and encouraged all GiveWell staff to input their own values into the model. The resulting strong consensus that cash is significantly less cost-effective than bednets/​deworming persuaded me to put more weight on cost-effectiveness analysis in my giving this year.

Similarly, Alexander Berger notes his increased confidence in the recommendations and how much AMF strikes him as a great opportunity:

What I wrote last year is even more true this year: I think our recommendations this year are stronger than they were last year, and I’m excited to support them. And since I’ve been (even) less involved in our top charities research this year than in previous years, I’m also inclined to just follow the recommended GiveWell allocation (i.e. 100% to AMF).

I’m particularly struck this giving season by just how strong a recommendation AMF is. We believe that AMF can fairly reliably fill a gap in global bednet coverage and avert child deaths for something on the order of every $3,000 donated. I’m much more skeptical than some of my colleagues about how much we can learn from our cost-effectiveness models, particularly for deworming, but I see the case for bednets as being quite strong. And even if we are off by a factor of several times, which I think is absolutely possible, I see that as remarkably cost-effective: for comparison, common estimates put the value of a statistical life in the United States at around $5 million.

Increased coordinated doubling down on GiveWell recommendations by its allies

One major thing I see being done at a larger scale in 2015 is coordination by organizations that seek to move money to GiveWell top charities through careful marketing and messaging. GiveWell, which has historically remained very detached from efforts to spread the word about its top charities, has given a more explicit nod to fundraising efforts explicitly aligned with the goal of raising money for its top recommended charity. For instance, their Outreach Associate Catherine Hollander (the position didn’t even exist last year) wrote a blog post recently on fundraising for AMF, the top charity. Charity Science’s fundraising operation also appears to have matured this year, as exemplified in their post providing guidance for Christmas fundraisers. Giving What We Can also seems to have upped its social media presence and the publicity surrounding its Pledges. While not all the money moved through this will be attributed to GiveWell, it may raise the profile of GiveWell and move some money explicitly attributed to GiveWell.

More on the distribution of money moved to top charities

In 2014, AMF was not in the top charity list till the end of the year, but still raised $4.4 million from non-GV donors, higher than any other charity. This year, AMF was a top charity throughout the year, and GiveWell identified as the sole charity that it recommended GiveWell-aligned donors to give to. GiveWell itself, and its allies, are picking on AMF as the focal point of fundraising campaigns. AMF is also intrinsically appealing to GiveWell donors, since it’s a scalable implementation of a proven intervention that saves lives. For all these reasons, I believe that AMF will raise the most money among GiveWell-recommended charities by a huge margin. Moreover, the uncertainty around money moved to AMF accounts for most of the uncertainty in total money moved.

I believe the other charities will generally receive money from people who have strong prior reasons for wanting to donate to the charities, with the GiveWell recommendation playing a Charity Navigator-like role of validating the organization’s legitimacy. For instance, GiveDirectly excites a large number of effective altruists, due to the simplicity, non-paternalism, strong organizational growth, and significant scalability. People with whom these values resonate will likely continue to donate to GiveDirectly and use the huge funding gap GiveWell has found for the organization to bolster their case. Similarly for SCI and DtWI. It’s also important to note that since the doubling down on AMF happened in November, the first few months probably saw a larger share of donations to these other charities.

Growth in coming years

Most of my discussion so far has been focused either on general models for money moved or what’ll happen in 2015. However, my primary interest is in understanding growth over the next 5 or more years. The reason I devoted so many pixels to 2015 is that the statements I make about 2015 will be tested against reality in fairly short order, and I believe this calibration against reality will be crucial to evaluating my credibility.

Let’s get back to three determinants of GiveWell’s money moved: discovery and brand awareness, conversion, and retention/​upsell.

  • Discovery and brand awareness: I believe that this is low, but I don’t believe that increasing it right now will do much. I think increasing brand awareness will do very little till the deeper problem of low conversion is solved. As the surveys I did showed, GiveWell brand awareness is low but not that low. It’s particularly high among people who already use Charity Navigator, and yet, most of the people who are aware of GiveWell don’t actively use it.

  • Conversion: 2015 might already be a peak in terms of the extent to which the charities convert people. I think that, over the timeframe of the next two years, there’s a greater chance that AMF, SCI, DtWI, or GiveDirectly would run out of room for more funding as judged by GiveWell than that GiveWell will find another promising charity to which it recommends moving a comparable amount of money. I also believe that fundraiser coordination is also reaching close to the best it can achieve in squeezing out money. I think we are already nearing the long-run bottleneck on conversion: GiveWell charities just don’t resonate with enough people.

  • Retention and upsell: There is a momentum effect here arising from the fact that GiveWell got a large number of young donors in its early years. The income and giving potential of GiveWell donors will continue to increase over the next few years, so that alone will be a driver of growth. I don’t believe this will run out for the next 2-3 years, but it could run out after that if GiveWell doesn’t add enough new donors with high earning potential in the pipeline.

The upshot: most of the growth in money moved to GiveWell top charities in the next two years will come from retention and upsell. More concretely, this means that the number of donors probably won’t increase that much from 2015 to 2016 and 2017, but the money moved per donor will continue to increase, largely because of an increase in donors in high size buckets (and mostly this will be donors who donated smaller amounts in earlier years). More concretely, I expect the number of donors to increase by 10% per year for the next two years (2015 to 2016 and 2016 to 2017) and I expect the money moved per donor to increase by about 20% and 15% in the coming two years. A lot of this will come from the highest donation size buckets, via phenomena such as Founders Pledge donations.

I don’t think the time is right to spell out the model in detail. I’d like to see how close my 2015 predictions are to reality and uncover any biases in how I’m thinking about things before elucidating a plausible story for the next few years.

Implications

To the extent that I believe the median case of my forecasts, it leaves me with mixed feelings. I believe that there is wisdom in having larger numbers of eyeballs engaging with the data and recommendations. The idea of growth being propelled more by retention than by user growth is a negative as far as quality is concerned. On the other hand, assuming some rationality on the part of donors, larger sums of money moved per donor incentivizes each donor to care more about their donations, and could in principle put more scrutiny on GiveWell. I believe that the former effect significantly outweighs the latter at the present margin, particularly because a lot of the growth in money moved per donor comes from donors just getting richer, rather than deciding to donate a larger fraction of their income to charity.

I also think the continued short-run increase in money moved to GiveWell top charities reduces the incentives for GiveWell to expand to more domains and undertake the hard task of finding new areas to donate to. This limits their long-term appeal (in particular, it puts a ceiling on how much conversion they can accomplish), but more importantly from a social welfare perspective, it limits the fraction of the donor population and the fraction of donor money that will be influenced by the kind of deep analysis that GiveWell strives to do. (With that said, Charity Navigator-level or GuideStar-level analysis of charities has taken off in a decent way and is probably informing over 1% of money moved already, but that’s not the same type of analysis since it’s based on a much smaller, though still important, set of metrics).

UPDATE 1: In a blog post comment on December 23, Elie Hassenfeld reported that as of December 22, GiveWell had moved $15.7 million in non-Good Ventures donations to its top charities. My original estimates did not include this information, but accounting for it would move the lower end of my estimates upwards somewhat while not significantly affecting the higher ends.

UPDATE 2: In a blog post on January 8, 2016, Tyler Heishman reports GiveWell’s Q3 2015 money moved and also provides a preliminary estimate of the money moved between February and December 2015. This doesn’t include money that might be moved in January 2016, though that is unlikely to materially affect the estimates much since the bulk of money moved is in December. Heishman reports that $28 million in money moved from sources other than Good Ventures is attributable to GiveWell. This is close to the 90th percentile in my estimate range ($28.6 million). So overall, my estimates were quite conservative, and it may have been possible to more accurately make higher estimates. With that said, my identification of the potential reasons for the money moved being a lot higher than my median estimate was broadly correct. As I wrote above:

The upper end of estimates is mainly governed by factors such as wealthy donors, highly successful fundraising groups or celebrity endorsements or media publicity, and other unforeseen factors.

The GiveWell blog post says:

Excluding Good Ventures, we have tracked about $28 million (of which, roughly half has come from donors giving $1 million or more).

Hence, at least the “wealthy donors” part seems right.

Moreover, I expect that the other predictions I made should roughly hold up with appropriate rescaling to the greater amount of money moved, but also accounting for the increased variability due to a small number of large donors.

UPDATE 3: Catherine Hollander linked to almost-final estimates of money moved by charity and size bucket. The total non-Good Ventures money moved was about $33.37 million (versus about 13 or 14 million last year, and a 50% confidence interval of $16.2-24.5 million in my post). Within the distribution specified in my post, the amount actually moved fell at the 97.5th percentile.

Size buckets: Money from those giving over $100,000 (including both the $100,000-$999,999 bucket and the $1,000,000+ bucket) was about $20.29 million, versus $4.42 million last year. Excluding these heavy donors, the money moved was $13.4 million, compared to about $8.4-10 million last year. The money moved per donor was $2,157.

I believe my primary source of error in estimates was at the high end. I correctly predicted that the money moved per donor would increase, but I underestimated the extent to which the total money moved by donors giving in excess of $100,000 would increase. If the giving in these buckets had increased in the same proportion as the giving in the sub-$100,000 buckets, the money moved would have been within my 50% interval. With that said, I also underestimated somewhat the sheer volume of donors: my 50% estimate for the number of donors was 9,500-12,500, whereas the actual number was 15,468.

Donations by recipient charity: Here, too, my estimates were off, but not systematically so. Three of the four top charities (Against Malaria Foundation, Schistosomiasis Control Initiative, and Deworm the World Initiative) fell in my 50% intervals for those charities (with AMF, at almost 15 million, being at the higher end of its range but still within it). The one outlier was GiveDirectly. My 50% estimate for GiveDirectly was between 1 and 4 million dollars, but actual money moved appears to have been over 13 million dollars. This is interesting in that it was unexpected not only to me but also (probably) to many other GiveWell watchers. It was also contrary to the message that GiveWell was focused on spreading, namely, that donating to AMF should be the top priority.

UPDATE 4: Natalie Stone-Crispin of GiveWell has just published a blog post on the GiveWell blog providing insight into GiveWell’s research plans for 2016. Some of these updates are related to predictions I made, and make me more confident that my thinking was in the right direction. In my Facebook post that had been the original impetus for this Effective Altruists group post, I had said that GiveWell’s top charities are unlikely to change in 2016, and Natalie’s just-published post strengthens my view that the top charity list is likely to stay the same (with possible temporary removal of some top charities because of exhaustion of room for more funding or concerns about scaling challenges). GiveWell is also looking into switching focus from identifying top charities to outreach, something that I had not made a prediction on in either direction.

UPDATE 5: On Friday, May 13, 2016, GiveWell finally released the final report on money moved in 2015. I have published a new post going over my forecasts, how they compared with reality, and implications.

Disclosure: I am not financially affiliated with GiveWell, but have some past and ongoing association with them. You can get a detailed account of my past interaction and affiliation with GiveWell here.

vipulnaik19 Dec 2015 20:22 UTC
8 points
8 comments23 min readEA link