Thank you so much for answering questions like this. I think itâs really worthwhile :). When you have a free moment it would be great if you could answer these questions. Unlike Peter I have only 6 questions:
1.) Should GWWCâs realistic impact estimate include the % of people each year who fulfill their pledge? For instance, the summary of the 2014 Giving Review states that 20-55% of people who pledged from 2009-2013 didnât fulfill their pledge in 2014.
2.) On what information is the ratio of actual donations to pledged donations used in GWWCâs most recent realistic impact estimate based upon?
3.) Is the current technique that GWWC uses to calculate counterfactuals more likely to overestimate or underestimate GWWCâs impact given that the counterfactual percentage of all future donations is estimated when people initially take the pledge rather than when they make their donations in subsequent years?
6.) It seems that GWWCâs comparative advantage is in generating and maintaining pledges for effective charities. Given that there is a large overlap between GWWCâs and GiveWellâs charity recommendations and GiveWell being in a better position to continue charity evaluations, why is it worth GWWC continuing charity evaluations?
Also on Nick Becksteadâs review, the reason he made that comment is that in 2013, it looked like growth was linear (2011 and 2012 both added 100 members), which made it more unclear whether marginal activities were helping.
The situation is different todayâgrowth is definitely faster than linear.
Thereâs now been two new impact evaluations after Nick wrote his, with much more in-depth and up-to-date figures.
Michelle is in a better position to answer some of these, but Iâll answer the ones I can. Iâd also suggest having a look at the comments section of our last fundraising prospectus, which covered some similar ground and which may provide more detail to some of your questions.
1) This is largely covered in the step Accounting for members donating a different amount than they pledged, which uses data from members who have reported their donations in My Giving, and comparing their actual donations with their pledges. The upper bound estimate in the Giving Review (80% of people keeping their pledge) uses the same dataset, but only takes into account a binary âpledge metâ vs. âpledge not metâ. The ratio of pledges to donations (117%) has more bearing our calculations because it captures both people who donate less than they pledge, and people who donate substantially more. Overall, due to people on average donating more than their pledge, the ratio is actually larger than 1:1 (so, even if only 80% of members hit their pledge amount, the number of people donating more than their pledge means the cohort as a whole donates more than it pledges).
The only quibble that you might have here is whether this cohort (people who report donations in My Giving) donates at a substantially different rate than people who donât report their donations (after weâve factored out people who have both gone silent, and stopped donating, as per the earlier Accounting for membership attrition step). We have good reason to think this isnât the case (we know a lot of people personally who choose not to use My Giving, but who keep their pledges), but if you were more pessimistic about this, you could downweight the Ratio of Actual Donations to Pledged Donations in the spreadsheet (currently 1.17).
2) As above, this was calculated using data from members who have reported their donations in My Giving, and taking the ratio between their pledged amount and their reported donations, averaging over all members. See this section of the impact page for more info.
3) To clarify, are you talking about the impact of changes to membersâ income over time, or asking whether weâre accounting for potential changes to donation patterns over time which affect the counterfactual ratio?
Weâre currently calculating our counterfactuals based on the amounts members say they would have donated without us â we havenât modelled behaviour changes into the future, and Iâm not sure what weâd base such a model on. Whether itâs conservative or optimistic is unclear, but Iâd say that this is probably a wash â itâs hard to know whether peopleâs predictions of what they would have donated are overall optimistic or pessimistic. In our conversations with members, many people who say that they would have donated 10% without us also tell us that weâre a useful commitment device (indicating that perhaps they wouldnât stick to their 10% without us, and that our counterfactual impact is actually greater than what weâve accounted for in the model).
If youâre just talking about the effect of membersâ income on the counterfactuals (because the calculations assume they will be static, when in reality income is likely to rise) then we think the calculation is fairly conservative. See the Donations Pledged By Members section of the impact page:
This methodology relies on the accuracy of membersâ predictions about their future income. In general we have found that these predictions seem conservative, as most people underestimate their future earning potential[7] . If members do not estimate their future salary, we use the median salary for their country. We think that is a fairly pessimistic assumption, as our median member has an expected earning potential higher than the median wage[8] .
Footnotes 7 and 8 expand on this:
For example, many members estimate their future income will be the same as their current income, even though they are at the beginning of their careersâin reality, income typically increases throughout a personâs career
For example, many members attend prestigious universities and/âor are pursuing careers that have an average salary much higher than the median wage
See also this comment made on the last prospectus for some discussion of the effect of modelling changing income over time. Using the same model with the updated figures yields a ratio of between 69:1 (using a fairly arbitrary starting pledged amount of $4,200,000 which produces donations over membersâ careers equivalent to the $344 million pledged amount, but accounts for income growth) and 157:1 (assuming that the current pledges correspond to current income levels, and that all members are at the beginning of their careers). You can play with this assumption by editing the figure in cell C2 on the âCalculationsâ sheet of this spreadsheet, and the income growth rates at the bottom of the column.
5) Weâve taken into account an additional year (2014), where we had strong growth, but where our costs were not significantly higher. Our membership more than doubled (386 in 2009-13 vs 792 in 2009-14) but our costs only went up by around 40% (ÂŁ238k in 2009-13 vs ÂŁ332k in 2009-14). The assumptions have remained essentially the same, so the difference in those ratios accounts for most of the difference (with a less significant amount being due to small changes in membership attrition and counterfactual pledge ratios).
As we note in the caveats, we do expect this amount to go down in future as our staff costs increase, and we donât want people to fixate on it as a predictor of our impact. We see it more as a sanity check of whether weâre a good bet, vs giving money to other effective causes.
The degree to which this will change significantly in future really depends on how strong our member growth vs. costs growth is. If the cost of creating a new member hits diminishing marginal returns soon (not at all unlikely), then itâs likely to drop back fairly quickly. We donât see this as particularly troubling â so long as our absolute number of members keeps increasing and the ratio is positive, then weâre still a good bet.
We doubled staff numbers over 2015 (3 > 6) and weâre hiring again now (6 > 8 or 9, depending on fundraising), so itâs likely that this will push it back down. We think that maximising future membership growth will be contingent on broadening the skillset of our team (and just having more hands on deck to do outreach work would be a huge help!). Expanding the team, strengthening our organisation, and increasing our growth rate seems very important right now (and much more important than maintaining this ratio at the current level). My guess is that it will settle somewhere between 20:1 and 60:1 â not quite as impressive as 104:1, but still suggestive that weâre making a big difference!
(This difference is similar to the reason that we donât think that using âoverheadâ is a good measure of a charityâs effectiveness. In effect, this is our overheads increasing, but so long as this leads to greater (counterfactual) overall member growth and donations to top charities, then we should be happy for the ratio to drop.)
Samâs covered most of what I would say, but on 4): I think heâs right that itâs difficult to know to what extent growth is down to particular actions of our staff versus other growth factors (although we do have a bunch more information than we did back then). There are a lot of different factors at play, and often there are a lot of necessary conditions for someone becoming a member. Often itâs difficult even to know the difference between âorganic growthâ and âstaff activitiesâ â does someone joining due in part to an action of one of our staff 3 years ago count as organic or staff? A couple of concrete examples: 1) the Cambridge pledge event last year seemed to get around 80 people to join who wouldnât otherwise have joined. The pledge event wouldnât have happened without the Cambridge chapter, which wouldnât have started up without GWWC central. The central team also did various things to support the chapter, and built a website where itâs easy to join etc. But initiative was taken and work was done by a group of students acting mostly on their own. 2) Marinella has been writing to people who attended EA Global who werenât yet members of GWWC â learning about how they found out about us, answering questions they have etc. 9 of the people she contact joined GWWC directly after /â during their conversations with her. On the one hand, that seems concretely down to her reaching out. On the other hand, EA Global was also a necessary part. And on the other hand again, the GWWC staff were at EA Global and actively making people aware of GWWC and effective giving there.
Individual outreach of the kind Marinella has been doing, along with setting up chapters from scratch using chapter seeding, seems quite clearly to be attributable to particular actions by particular of our staff. But itâs important to focus on only those kinds of really direct things. Advice from marketing seems to indicate that if you want to get people on board itâs important to have a strong brand, look professional etc. It would be very surprising if that wasnât important for us too. That kind of thing isnât really possible from organic growth. But itâs difficult to know precisely what effect it has â all we can do is learn as much as possible about what does and doesnât work for other organisations, try things out, and see how it seems to affect our growth.
Thanks so much for providing such detailed answers to my questions :). I think that I am getting a much better idea now. Thanks Sam for linking me to the comments of GWWCâs fundraising prospectus- the discussions there cover most of my questions. If itâs not too much to ask I would really like it if you answered a few more questions that I have.
Just to follow up on Samâs answer to my first and second questions.
To help me understand how I should feel about the ratio GWWC uses for actual donations to pledged donations could you please outline what percentage of members record enough information in My Giving for their information to be used in calculating the ratio of actual donations to pledged donations that was used in GWWCâs most recent realistic impact estimate?
To clarify, are you talking about the impact of changes to membersâ income over time, or asking whether weâre accounting for potential changes to donation patterns over time which affect the counterfactual ratio?â
I was talking about accounting for potential changes to donation patterns over time which affect the counterfactual ratio- sorry that wasnât clear :). I certainly agree that itâs really hard to know how accurate peopleâs counterfactual estimates are. I wonder what the best way of doing estimates for counterfactuals is. Do you guys think that it might be more accurate to ask people each year about their counterfactual estimates rather than asking people once and then using that number for the next 40 years to calculate the counterfactual impact of GWWC?
One last question. GWWC seems to be the only EA fundraising organization I know of which calculates its potential impact for 40 years into the future. Do you guys think that other EA fundraising organizations like Charity Science and Raising for Effective Giving should attempt to do this type of realistic impact calculation for 40 years into the future so that donors who are trying to decide between these organizations may be able to more accurately compare the different organizations?
On a sidenote, I think that up and down votes of comments provide a really valuable source of feedback. As a result I am interested if people have ideas about the possible reasoning behind the down vote or down votes of my previous comment so that I can hopefully improve my future comments :).
Iâm glad you feel youâre getting a proper sense! Itâs surprisingly hard to describe all the information in an informative and intelligible way.
Iâm afraid I have no idea why the comment was downvoted. Given the number of upvotes, it could have just been a mistake?
The calculation we used was for people who joined before 2013, and it was 137 out of 268 people, or 51%. Updating it using the 2013 and 2014 cohorts didnât make much difference (a very slight increase in the percentage), so we continued using the same figure. The cohort data linked to above gives an indication of how many members for 2013 and 2014 gave us enough information to calculate a pledge percentage.
With regard to the counterfactuals, we did at one point in the past ask that. People felt their answers werenât more accurate when given later on though. The thing that seemed to be going on was that while when youâre considering joining but havenât yet you have a fairly good sense of how you might act if you join, and if you donât, but the longer youâre a member the less you identify with the counterfactual of not having joined and so the weirder it feels to think about what would have happened if it had been true. Asking people about these counterfactuals makes it harder to get other information we need (like income and giving data, and updated professions), either because we put it on the same email/âsurvey making people more likely to put off responding to that, or because we send out an additional email/âsurvey, increasing peopleâs fatigue of being asked stuff by us. So overall asking this seemed to make our calculations worse rather than better.
Weâre actually not the only EA org who tries to calculate our overall impact rather than just value which has already come to fruition â for example, 80,000 Hours also does. This calculation is actually very different from one which would try to capture our âpotential impact for 40 years into the futureâ. If we ceased all our activities today, I would expect that quite a few people would join and donate due to our past activities. I also expect that weâve gained quite a bit of experience which will make us more effective in future than we have been in the past. So this calculation doesnât even capture the potential impact of all the activities weâve undertaken so far, let alone our potential impact into the future.
Yes, I think itâs very important for EA organisations to try to get a comprehensive picture of their overall impact, both for internal use and for informing others.
GWWC seems to be the only EA fundraising organization I know of which calculates its potential impact for 40 years into the future.
Even if you only focus on donations that have already been made, and ignore pledges, GWWC has a high positive multiplier. Moreover, completely ignoring the future value of pledges would be really pessimistic.
I think REG should try to value future pledged donations.
With CS you should value the future value of legacy commitments, even though they will take 20-60 years (though youâd need to apply discounting). (Presumably you do give them positive value even though theyâre a long way in the future? :)) The case for valuing future birthday fundraisers is less strong because thereâs no analogous long-term commitment.
Moreover, in all cases, we shouldnât get too hung up on historical impact. Most of our impact lies in the future, so what really matters is the potential upside and probability of that upside. Historical money moved is only a weak indication of that.
Even if you only focus on donations that have already been made, and ignore pledges, GWWC has a high positive multiplier. Moreover, completely ignoring the future value of pledges would be really pessimistic.
I totally agree and think that a really interesting question is what the future value of pledges should be. I think may also be worth mentioning that if we focus on donations that have already been made, my understanding is GWWCâs impact is an order of magnitude less than their current realistic impact estimate. I am not sure how exactly we should weigh that information.
With CS you should value the future value of legacy commitments, even though they will take 20-60 years (though youâd need to apply discounting). (Presumably you do give them positive value even though theyâre a long way in the future? :))
At CS we sure do value the future value of legacy commitments :). We havenât yet determined exactly how we will calculate their expected value.
Zooming out for a bit, if youâre trying to evaluate GWWCâs historical impact, thereâs five categories of impact:
Money pledgers have already donated
Money pledgers will donate in the future
People who will take the pledge in the future due to past investments (e.g. GWWCâs website will keep creating pledgers even if they closed tomorrow).
Money influenced by people who didnât take the pledge
Impact that isnât in the form of donations to top charities (creating the EA movement, founding CEA, their research contributions, spinning off 80k, substantial policy influence)
GWWC are very modest and only focus on (1) and (2) :)
To evaluate the total impact, you need to calculate the expected value for each component. You should make your own estimates given the best information you have.
Only counting (1) would be a very poor estimate. (2) to (5) are very unlikely to be zero. My personal guess is that:
(2) is about 10x (1) [because I think GWWCâs internal estimates are reasonable]
(3) and (4) are similar to (1)
(5) is much larger than (2)
However, even if you only cared about (1), GWWCâs multiplier would still be about 10x, which would make them more cost-effective than GiveWell recommended charities, and equal to or more cost-effective than the other EA fundraising orgs (in the past). So Iâm not sure itâs even that decision relevant.
Personally if GWWC central closed tomorrow, I expect most of the growth would stop. There would be a remaining constant trickle of members from the website (providing at most linear growth, whereas GWWC has been growing faster than linear, and most members donât join just from reading the website alone). Initially there would be continued growth from the chapters too, but the numbers would start to shrink. You may get continued growth from the EA community promoting GWWC, but I expect it would be much smaller.
More concretely, GWWC staff right now pursue several channels to growth that wouldnât happen without them:
Speaking to interested people one-on-one (few join just from written content alone; being made âthe askâ is really important)
Fostering chapters (e.g. giving them materials, arranging pledge drives, encouraging them, doing events) - most of this wouldnât happen otherwise.
Seeking press attention to get new people involvedâditto, mostly wouldnât happen otherwise.
And thereâs lots of other things they could do at the margin to get even more growth e.g. boosting the conversion rate of the website.
Finally, like Michelle says, much of the continued growth there would be if GWWC closed is due to staff efforts in past years (e.g. developing chapters in the first place), so should be included at some point in your evaluation. It would be wrong to just ignore new members from this year who didnât result from activities by staff this year. Future staff efforts can also be expected to generate members many years into the future.
In this sense, GWWC leverage ratio is a big underestimate of their effectiveness. Past efforts will actually generate a continued stream of members who arenât being counted.
More concretely, GWWC staff right now pursue several channels to growth that wouldnât happen without them:
Speaking to interested people one-on-one (few join just from written content alone; being made âthe askâ is really important)
Fostering chapters (e.g. giving them materials, arranging pledge drives, encouraging them, doing events) - most of this wouldnât happen otherwise.
Seeking press attention to get new people involvedâditto, mostly wouldnât happen otherwise.
Does GWWC have rough estimates of how many members each of these channels lead to? Or could you or other CEA staff hazard a guess, even rough?
Hi Michelle,
Thank you so much for answering questions like this. I think itâs really worthwhile :). When you have a free moment it would be great if you could answer these questions. Unlike Peter I have only 6 questions:
1.) Should GWWCâs realistic impact estimate include the % of people each year who fulfill their pledge? For instance, the summary of the 2014 Giving Review states that 20-55% of people who pledged from 2009-2013 didnât fulfill their pledge in 2014.
2.) On what information is the ratio of actual donations to pledged donations used in GWWCâs most recent realistic impact estimate based upon?
3.) Is the current technique that GWWC uses to calculate counterfactuals more likely to overestimate or underestimate GWWCâs impact given that the counterfactual percentage of all future donations is estimated when people initially take the pledge rather than when they make their donations in subsequent years?
4.) In Nick Becksteadâs 2013 quantitative performance review of GWWC he noted:
What is your response to this?
5.) Around 6 months ago GWWCâs realistic impact estimate was a ratio of 60:1 and itâs now 104:1. Why has this number changed and do you expect it to change that much in another 6 months?
6.) It seems that GWWCâs comparative advantage is in generating and maintaining pledges for effective charities. Given that there is a large overlap between GWWCâs and GiveWellâs charity recommendations and GiveWell being in a better position to continue charity evaluations, why is it worth GWWC continuing charity evaluations?
Also on Nick Becksteadâs review, the reason he made that comment is that in 2013, it looked like growth was linear (2011 and 2012 both added 100 members), which made it more unclear whether marginal activities were helping.
The situation is different todayâgrowth is definitely faster than linear.
Thereâs now been two new impact evaluations after Nick wrote his, with much more in-depth and up-to-date figures.
Hi Kieran,
Michelle is in a better position to answer some of these, but Iâll answer the ones I can. Iâd also suggest having a look at the comments section of our last fundraising prospectus, which covered some similar ground and which may provide more detail to some of your questions.
1) This is largely covered in the step Accounting for members donating a different amount than they pledged, which uses data from members who have reported their donations in My Giving, and comparing their actual donations with their pledges. The upper bound estimate in the Giving Review (80% of people keeping their pledge) uses the same dataset, but only takes into account a binary âpledge metâ vs. âpledge not metâ. The ratio of pledges to donations (117%) has more bearing our calculations because it captures both people who donate less than they pledge, and people who donate substantially more. Overall, due to people on average donating more than their pledge, the ratio is actually larger than 1:1 (so, even if only 80% of members hit their pledge amount, the number of people donating more than their pledge means the cohort as a whole donates more than it pledges).
The only quibble that you might have here is whether this cohort (people who report donations in My Giving) donates at a substantially different rate than people who donât report their donations (after weâve factored out people who have both gone silent, and stopped donating, as per the earlier Accounting for membership attrition step). We have good reason to think this isnât the case (we know a lot of people personally who choose not to use My Giving, but who keep their pledges), but if you were more pessimistic about this, you could downweight the Ratio of Actual Donations to Pledged Donations in the spreadsheet (currently 1.17).
2) As above, this was calculated using data from members who have reported their donations in My Giving, and taking the ratio between their pledged amount and their reported donations, averaging over all members. See this section of the impact page for more info.
3) To clarify, are you talking about the impact of changes to membersâ income over time, or asking whether weâre accounting for potential changes to donation patterns over time which affect the counterfactual ratio?
Weâre currently calculating our counterfactuals based on the amounts members say they would have donated without us â we havenât modelled behaviour changes into the future, and Iâm not sure what weâd base such a model on. Whether itâs conservative or optimistic is unclear, but Iâd say that this is probably a wash â itâs hard to know whether peopleâs predictions of what they would have donated are overall optimistic or pessimistic. In our conversations with members, many people who say that they would have donated 10% without us also tell us that weâre a useful commitment device (indicating that perhaps they wouldnât stick to their 10% without us, and that our counterfactual impact is actually greater than what weâve accounted for in the model).
If youâre just talking about the effect of membersâ income on the counterfactuals (because the calculations assume they will be static, when in reality income is likely to rise) then we think the calculation is fairly conservative. See the Donations Pledged By Members section of the impact page:
Footnotes 7 and 8 expand on this:
See also this comment made on the last prospectus for some discussion of the effect of modelling changing income over time. Using the same model with the updated figures yields a ratio of between 69:1 (using a fairly arbitrary starting pledged amount of $4,200,000 which produces donations over membersâ careers equivalent to the $344 million pledged amount, but accounts for income growth) and 157:1 (assuming that the current pledges correspond to current income levels, and that all members are at the beginning of their careers). You can play with this assumption by editing the figure in cell C2 on the âCalculationsâ sheet of this spreadsheet, and the income growth rates at the bottom of the column.
5) Weâve taken into account an additional year (2014), where we had strong growth, but where our costs were not significantly higher. Our membership more than doubled (386 in 2009-13 vs 792 in 2009-14) but our costs only went up by around 40% (ÂŁ238k in 2009-13 vs ÂŁ332k in 2009-14). The assumptions have remained essentially the same, so the difference in those ratios accounts for most of the difference (with a less significant amount being due to small changes in membership attrition and counterfactual pledge ratios).
As we note in the caveats, we do expect this amount to go down in future as our staff costs increase, and we donât want people to fixate on it as a predictor of our impact. We see it more as a sanity check of whether weâre a good bet, vs giving money to other effective causes.
The degree to which this will change significantly in future really depends on how strong our member growth vs. costs growth is. If the cost of creating a new member hits diminishing marginal returns soon (not at all unlikely), then itâs likely to drop back fairly quickly. We donât see this as particularly troubling â so long as our absolute number of members keeps increasing and the ratio is positive, then weâre still a good bet.
We doubled staff numbers over 2015 (3 > 6) and weâre hiring again now (6 > 8 or 9, depending on fundraising), so itâs likely that this will push it back down. We think that maximising future membership growth will be contingent on broadening the skillset of our team (and just having more hands on deck to do outreach work would be a huge help!). Expanding the team, strengthening our organisation, and increasing our growth rate seems very important right now (and much more important than maintaining this ratio at the current level). My guess is that it will settle somewhere between 20:1 and 60:1 â not quite as impressive as 104:1, but still suggestive that weâre making a big difference!
(This difference is similar to the reason that we donât think that using âoverheadâ is a good measure of a charityâs effectiveness. In effect, this is our overheads increasing, but so long as this leads to greater (counterfactual) overall member growth and donations to top charities, then we should be happy for the ratio to drop.)
6) Hauke, our Director of Research answers this question here. The short version:
Supplement GiveWell research and find new charities within our comparative area of advantage (global poverty reduction)
Independently check GiveWell recommendations and provide resilience to the effective charity evaluation system
Provide supervision for students/âearly career researchers who want to focus on effectiveness
Ensure in-house credibility when talking about top charities and fact-checking all of our public-facing material
Hope that helps!
Hi Kieran,
Glad itâs useful!
Samâs covered most of what I would say, but on 4): I think heâs right that itâs difficult to know to what extent growth is down to particular actions of our staff versus other growth factors (although we do have a bunch more information than we did back then). There are a lot of different factors at play, and often there are a lot of necessary conditions for someone becoming a member. Often itâs difficult even to know the difference between âorganic growthâ and âstaff activitiesâ â does someone joining due in part to an action of one of our staff 3 years ago count as organic or staff? A couple of concrete examples: 1) the Cambridge pledge event last year seemed to get around 80 people to join who wouldnât otherwise have joined. The pledge event wouldnât have happened without the Cambridge chapter, which wouldnât have started up without GWWC central. The central team also did various things to support the chapter, and built a website where itâs easy to join etc. But initiative was taken and work was done by a group of students acting mostly on their own. 2) Marinella has been writing to people who attended EA Global who werenât yet members of GWWC â learning about how they found out about us, answering questions they have etc. 9 of the people she contact joined GWWC directly after /â during their conversations with her. On the one hand, that seems concretely down to her reaching out. On the other hand, EA Global was also a necessary part. And on the other hand again, the GWWC staff were at EA Global and actively making people aware of GWWC and effective giving there.
Individual outreach of the kind Marinella has been doing, along with setting up chapters from scratch using chapter seeding, seems quite clearly to be attributable to particular actions by particular of our staff. But itâs important to focus on only those kinds of really direct things. Advice from marketing seems to indicate that if you want to get people on board itâs important to have a strong brand, look professional etc. It would be very surprising if that wasnât important for us too. That kind of thing isnât really possible from organic growth. But itâs difficult to know precisely what effect it has â all we can do is learn as much as possible about what does and doesnât work for other organisations, try things out, and see how it seems to affect our growth.
Hi Michelle and Sam,
Thanks so much for providing such detailed answers to my questions :). I think that I am getting a much better idea now. Thanks Sam for linking me to the comments of GWWCâs fundraising prospectus- the discussions there cover most of my questions. If itâs not too much to ask I would really like it if you answered a few more questions that I have.
Just to follow up on Samâs answer to my first and second questions.
To help me understand how I should feel about the ratio GWWC uses for actual donations to pledged donations could you please outline what percentage of members record enough information in My Giving for their information to be used in calculating the ratio of actual donations to pledged donations that was used in GWWCâs most recent realistic impact estimate?
I was talking about accounting for potential changes to donation patterns over time which affect the counterfactual ratio- sorry that wasnât clear :). I certainly agree that itâs really hard to know how accurate peopleâs counterfactual estimates are. I wonder what the best way of doing estimates for counterfactuals is. Do you guys think that it might be more accurate to ask people each year about their counterfactual estimates rather than asking people once and then using that number for the next 40 years to calculate the counterfactual impact of GWWC?
One last question. GWWC seems to be the only EA fundraising organization I know of which calculates its potential impact for 40 years into the future. Do you guys think that other EA fundraising organizations like Charity Science and Raising for Effective Giving should attempt to do this type of realistic impact calculation for 40 years into the future so that donors who are trying to decide between these organizations may be able to more accurately compare the different organizations?
On a sidenote, I think that up and down votes of comments provide a really valuable source of feedback. As a result I am interested if people have ideas about the possible reasoning behind the down vote or down votes of my previous comment so that I can hopefully improve my future comments :).
Iâm glad you feel youâre getting a proper sense! Itâs surprisingly hard to describe all the information in an informative and intelligible way.
Iâm afraid I have no idea why the comment was downvoted. Given the number of upvotes, it could have just been a mistake?
The calculation we used was for people who joined before 2013, and it was 137 out of 268 people, or 51%. Updating it using the 2013 and 2014 cohorts didnât make much difference (a very slight increase in the percentage), so we continued using the same figure. The cohort data linked to above gives an indication of how many members for 2013 and 2014 gave us enough information to calculate a pledge percentage.
With regard to the counterfactuals, we did at one point in the past ask that. People felt their answers werenât more accurate when given later on though. The thing that seemed to be going on was that while when youâre considering joining but havenât yet you have a fairly good sense of how you might act if you join, and if you donât, but the longer youâre a member the less you identify with the counterfactual of not having joined and so the weirder it feels to think about what would have happened if it had been true. Asking people about these counterfactuals makes it harder to get other information we need (like income and giving data, and updated professions), either because we put it on the same email/âsurvey making people more likely to put off responding to that, or because we send out an additional email/âsurvey, increasing peopleâs fatigue of being asked stuff by us. So overall asking this seemed to make our calculations worse rather than better.
Weâre actually not the only EA org who tries to calculate our overall impact rather than just value which has already come to fruition â for example, 80,000 Hours also does. This calculation is actually very different from one which would try to capture our âpotential impact for 40 years into the futureâ. If we ceased all our activities today, I would expect that quite a few people would join and donate due to our past activities. I also expect that weâve gained quite a bit of experience which will make us more effective in future than we have been in the past. So this calculation doesnât even capture the potential impact of all the activities weâve undertaken so far, let alone our potential impact into the future. Yes, I think itâs very important for EA organisations to try to get a comprehensive picture of their overall impact, both for internal use and for informing others.
Thanks again for answering these questions Michelle and Sam :)
Even if you only focus on donations that have already been made, and ignore pledges, GWWC has a high positive multiplier. Moreover, completely ignoring the future value of pledges would be really pessimistic.
I think REG should try to value future pledged donations.
With CS you should value the future value of legacy commitments, even though they will take 20-60 years (though youâd need to apply discounting). (Presumably you do give them positive value even though theyâre a long way in the future? :)) The case for valuing future birthday fundraisers is less strong because thereâs no analogous long-term commitment.
Moreover, in all cases, we shouldnât get too hung up on historical impact. Most of our impact lies in the future, so what really matters is the potential upside and probability of that upside. Historical money moved is only a weak indication of that.
https://ââ80000hours.org/ââ2015/ââ11/ââtake-the-growth-approach-to-evaluating-startup-non-profits-not-the-marginal-approach/ââ
Hi Ben,
Thanks for responding :)
I totally agree and think that a really interesting question is what the future value of pledges should be. I think may also be worth mentioning that if we focus on donations that have already been made, my understanding is GWWCâs impact is an order of magnitude less than their current realistic impact estimate. I am not sure how exactly we should weigh that information.
At CS we sure do value the future value of legacy commitments :). We havenât yet determined exactly how we will calculate their expected value.
How do should you weigh the information?
Zooming out for a bit, if youâre trying to evaluate GWWCâs historical impact, thereâs five categories of impact:
Money pledgers have already donated
Money pledgers will donate in the future
People who will take the pledge in the future due to past investments (e.g. GWWCâs website will keep creating pledgers even if they closed tomorrow).
Money influenced by people who didnât take the pledge
Impact that isnât in the form of donations to top charities (creating the EA movement, founding CEA, their research contributions, spinning off 80k, substantial policy influence)
GWWC are very modest and only focus on (1) and (2) :)
To evaluate the total impact, you need to calculate the expected value for each component. You should make your own estimates given the best information you have.
Only counting (1) would be a very poor estimate. (2) to (5) are very unlikely to be zero. My personal guess is that:
(2) is about 10x (1) [because I think GWWCâs internal estimates are reasonable] (3) and (4) are similar to (1) (5) is much larger than (2)
However, even if you only cared about (1), GWWCâs multiplier would still be about 10x, which would make them more cost-effective than GiveWell recommended charities, and equal to or more cost-effective than the other EA fundraising orgs (in the past). So Iâm not sure itâs even that decision relevant.
Personally if GWWC central closed tomorrow, I expect most of the growth would stop. There would be a remaining constant trickle of members from the website (providing at most linear growth, whereas GWWC has been growing faster than linear, and most members donât join just from reading the website alone). Initially there would be continued growth from the chapters too, but the numbers would start to shrink. You may get continued growth from the EA community promoting GWWC, but I expect it would be much smaller.
More concretely, GWWC staff right now pursue several channels to growth that wouldnât happen without them:
Speaking to interested people one-on-one (few join just from written content alone; being made âthe askâ is really important)
Fostering chapters (e.g. giving them materials, arranging pledge drives, encouraging them, doing events) - most of this wouldnât happen otherwise.
Seeking press attention to get new people involvedâditto, mostly wouldnât happen otherwise.
And thereâs lots of other things they could do at the margin to get even more growth e.g. boosting the conversion rate of the website.
Finally, like Michelle says, much of the continued growth there would be if GWWC closed is due to staff efforts in past years (e.g. developing chapters in the first place), so should be included at some point in your evaluation. It would be wrong to just ignore new members from this year who didnât result from activities by staff this year. Future staff efforts can also be expected to generate members many years into the future.
In this sense, GWWC leverage ratio is a big underestimate of their effectiveness. Past efforts will actually generate a continued stream of members who arenât being counted.
Does GWWC have rough estimates of how many members each of these channels lead to? Or could you or other CEA staff hazard a guess, even rough?