Thanks so much for providing such detailed answers to my questions :). I think that I am getting a much better idea now. Thanks Sam for linking me to the comments of GWWCās fundraising prospectus- the discussions there cover most of my questions. If itās not too much to ask I would really like it if you answered a few more questions that I have.
Just to follow up on Samās answer to my first and second questions.
To help me understand how I should feel about the ratio GWWC uses for actual donations to pledged donations could you please outline what percentage of members record enough information in My Giving for their information to be used in calculating the ratio of actual donations to pledged donations that was used in GWWCās most recent realistic impact estimate?
To clarify, are you talking about the impact of changes to membersā income over time, or asking whether weāre accounting for potential changes to donation patterns over time which affect the counterfactual ratio?ā
I was talking about accounting for potential changes to donation patterns over time which affect the counterfactual ratio- sorry that wasnāt clear :). I certainly agree that itās really hard to know how accurate peopleās counterfactual estimates are. I wonder what the best way of doing estimates for counterfactuals is. Do you guys think that it might be more accurate to ask people each year about their counterfactual estimates rather than asking people once and then using that number for the next 40 years to calculate the counterfactual impact of GWWC?
One last question. GWWC seems to be the only EA fundraising organization I know of which calculates its potential impact for 40 years into the future. Do you guys think that other EA fundraising organizations like Charity Science and Raising for Effective Giving should attempt to do this type of realistic impact calculation for 40 years into the future so that donors who are trying to decide between these organizations may be able to more accurately compare the different organizations?
On a sidenote, I think that up and down votes of comments provide a really valuable source of feedback. As a result I am interested if people have ideas about the possible reasoning behind the down vote or down votes of my previous comment so that I can hopefully improve my future comments :).
Iām glad you feel youāre getting a proper sense! Itās surprisingly hard to describe all the information in an informative and intelligible way.
Iām afraid I have no idea why the comment was downvoted. Given the number of upvotes, it could have just been a mistake?
The calculation we used was for people who joined before 2013, and it was 137 out of 268 people, or 51%. Updating it using the 2013 and 2014 cohorts didnāt make much difference (a very slight increase in the percentage), so we continued using the same figure. The cohort data linked to above gives an indication of how many members for 2013 and 2014 gave us enough information to calculate a pledge percentage.
With regard to the counterfactuals, we did at one point in the past ask that. People felt their answers werenāt more accurate when given later on though. The thing that seemed to be going on was that while when youāre considering joining but havenāt yet you have a fairly good sense of how you might act if you join, and if you donāt, but the longer youāre a member the less you identify with the counterfactual of not having joined and so the weirder it feels to think about what would have happened if it had been true. Asking people about these counterfactuals makes it harder to get other information we need (like income and giving data, and updated professions), either because we put it on the same email/āsurvey making people more likely to put off responding to that, or because we send out an additional email/āsurvey, increasing peopleās fatigue of being asked stuff by us. So overall asking this seemed to make our calculations worse rather than better.
Weāre actually not the only EA org who tries to calculate our overall impact rather than just value which has already come to fruition ā for example, 80,000 Hours also does. This calculation is actually very different from one which would try to capture our āpotential impact for 40 years into the futureā. If we ceased all our activities today, I would expect that quite a few people would join and donate due to our past activities. I also expect that weāve gained quite a bit of experience which will make us more effective in future than we have been in the past. So this calculation doesnāt even capture the potential impact of all the activities weāve undertaken so far, let alone our potential impact into the future.
Yes, I think itās very important for EA organisations to try to get a comprehensive picture of their overall impact, both for internal use and for informing others.
GWWC seems to be the only EA fundraising organization I know of which calculates its potential impact for 40 years into the future.
Even if you only focus on donations that have already been made, and ignore pledges, GWWC has a high positive multiplier. Moreover, completely ignoring the future value of pledges would be really pessimistic.
I think REG should try to value future pledged donations.
With CS you should value the future value of legacy commitments, even though they will take 20-60 years (though youād need to apply discounting). (Presumably you do give them positive value even though theyāre a long way in the future? :)) The case for valuing future birthday fundraisers is less strong because thereās no analogous long-term commitment.
Moreover, in all cases, we shouldnāt get too hung up on historical impact. Most of our impact lies in the future, so what really matters is the potential upside and probability of that upside. Historical money moved is only a weak indication of that.
Even if you only focus on donations that have already been made, and ignore pledges, GWWC has a high positive multiplier. Moreover, completely ignoring the future value of pledges would be really pessimistic.
I totally agree and think that a really interesting question is what the future value of pledges should be. I think may also be worth mentioning that if we focus on donations that have already been made, my understanding is GWWCās impact is an order of magnitude less than their current realistic impact estimate. I am not sure how exactly we should weigh that information.
With CS you should value the future value of legacy commitments, even though they will take 20-60 years (though youād need to apply discounting). (Presumably you do give them positive value even though theyāre a long way in the future? :))
At CS we sure do value the future value of legacy commitments :). We havenāt yet determined exactly how we will calculate their expected value.
Zooming out for a bit, if youāre trying to evaluate GWWCās historical impact, thereās five categories of impact:
Money pledgers have already donated
Money pledgers will donate in the future
People who will take the pledge in the future due to past investments (e.g. GWWCās website will keep creating pledgers even if they closed tomorrow).
Money influenced by people who didnāt take the pledge
Impact that isnāt in the form of donations to top charities (creating the EA movement, founding CEA, their research contributions, spinning off 80k, substantial policy influence)
GWWC are very modest and only focus on (1) and (2) :)
To evaluate the total impact, you need to calculate the expected value for each component. You should make your own estimates given the best information you have.
Only counting (1) would be a very poor estimate. (2) to (5) are very unlikely to be zero. My personal guess is that:
(2) is about 10x (1) [because I think GWWCās internal estimates are reasonable]
(3) and (4) are similar to (1)
(5) is much larger than (2)
However, even if you only cared about (1), GWWCās multiplier would still be about 10x, which would make them more cost-effective than GiveWell recommended charities, and equal to or more cost-effective than the other EA fundraising orgs (in the past). So Iām not sure itās even that decision relevant.
Hi Michelle and Sam,
Thanks so much for providing such detailed answers to my questions :). I think that I am getting a much better idea now. Thanks Sam for linking me to the comments of GWWCās fundraising prospectus- the discussions there cover most of my questions. If itās not too much to ask I would really like it if you answered a few more questions that I have.
Just to follow up on Samās answer to my first and second questions.
To help me understand how I should feel about the ratio GWWC uses for actual donations to pledged donations could you please outline what percentage of members record enough information in My Giving for their information to be used in calculating the ratio of actual donations to pledged donations that was used in GWWCās most recent realistic impact estimate?
I was talking about accounting for potential changes to donation patterns over time which affect the counterfactual ratio- sorry that wasnāt clear :). I certainly agree that itās really hard to know how accurate peopleās counterfactual estimates are. I wonder what the best way of doing estimates for counterfactuals is. Do you guys think that it might be more accurate to ask people each year about their counterfactual estimates rather than asking people once and then using that number for the next 40 years to calculate the counterfactual impact of GWWC?
One last question. GWWC seems to be the only EA fundraising organization I know of which calculates its potential impact for 40 years into the future. Do you guys think that other EA fundraising organizations like Charity Science and Raising for Effective Giving should attempt to do this type of realistic impact calculation for 40 years into the future so that donors who are trying to decide between these organizations may be able to more accurately compare the different organizations?
On a sidenote, I think that up and down votes of comments provide a really valuable source of feedback. As a result I am interested if people have ideas about the possible reasoning behind the down vote or down votes of my previous comment so that I can hopefully improve my future comments :).
Iām glad you feel youāre getting a proper sense! Itās surprisingly hard to describe all the information in an informative and intelligible way.
Iām afraid I have no idea why the comment was downvoted. Given the number of upvotes, it could have just been a mistake?
The calculation we used was for people who joined before 2013, and it was 137 out of 268 people, or 51%. Updating it using the 2013 and 2014 cohorts didnāt make much difference (a very slight increase in the percentage), so we continued using the same figure. The cohort data linked to above gives an indication of how many members for 2013 and 2014 gave us enough information to calculate a pledge percentage.
With regard to the counterfactuals, we did at one point in the past ask that. People felt their answers werenāt more accurate when given later on though. The thing that seemed to be going on was that while when youāre considering joining but havenāt yet you have a fairly good sense of how you might act if you join, and if you donāt, but the longer youāre a member the less you identify with the counterfactual of not having joined and so the weirder it feels to think about what would have happened if it had been true. Asking people about these counterfactuals makes it harder to get other information we need (like income and giving data, and updated professions), either because we put it on the same email/āsurvey making people more likely to put off responding to that, or because we send out an additional email/āsurvey, increasing peopleās fatigue of being asked stuff by us. So overall asking this seemed to make our calculations worse rather than better.
Weāre actually not the only EA org who tries to calculate our overall impact rather than just value which has already come to fruition ā for example, 80,000 Hours also does. This calculation is actually very different from one which would try to capture our āpotential impact for 40 years into the futureā. If we ceased all our activities today, I would expect that quite a few people would join and donate due to our past activities. I also expect that weāve gained quite a bit of experience which will make us more effective in future than we have been in the past. So this calculation doesnāt even capture the potential impact of all the activities weāve undertaken so far, let alone our potential impact into the future. Yes, I think itās very important for EA organisations to try to get a comprehensive picture of their overall impact, both for internal use and for informing others.
Thanks again for answering these questions Michelle and Sam :)
Even if you only focus on donations that have already been made, and ignore pledges, GWWC has a high positive multiplier. Moreover, completely ignoring the future value of pledges would be really pessimistic.
I think REG should try to value future pledged donations.
With CS you should value the future value of legacy commitments, even though they will take 20-60 years (though youād need to apply discounting). (Presumably you do give them positive value even though theyāre a long way in the future? :)) The case for valuing future birthday fundraisers is less strong because thereās no analogous long-term commitment.
Moreover, in all cases, we shouldnāt get too hung up on historical impact. Most of our impact lies in the future, so what really matters is the potential upside and probability of that upside. Historical money moved is only a weak indication of that.
https://āā80000hours.org/āā2015/āā11/āātake-the-growth-approach-to-evaluating-startup-non-profits-not-the-marginal-approach/āā
Hi Ben,
Thanks for responding :)
I totally agree and think that a really interesting question is what the future value of pledges should be. I think may also be worth mentioning that if we focus on donations that have already been made, my understanding is GWWCās impact is an order of magnitude less than their current realistic impact estimate. I am not sure how exactly we should weigh that information.
At CS we sure do value the future value of legacy commitments :). We havenāt yet determined exactly how we will calculate their expected value.
How do should you weigh the information?
Zooming out for a bit, if youāre trying to evaluate GWWCās historical impact, thereās five categories of impact:
Money pledgers have already donated
Money pledgers will donate in the future
People who will take the pledge in the future due to past investments (e.g. GWWCās website will keep creating pledgers even if they closed tomorrow).
Money influenced by people who didnāt take the pledge
Impact that isnāt in the form of donations to top charities (creating the EA movement, founding CEA, their research contributions, spinning off 80k, substantial policy influence)
GWWC are very modest and only focus on (1) and (2) :)
To evaluate the total impact, you need to calculate the expected value for each component. You should make your own estimates given the best information you have.
Only counting (1) would be a very poor estimate. (2) to (5) are very unlikely to be zero. My personal guess is that:
(2) is about 10x (1) [because I think GWWCās internal estimates are reasonable] (3) and (4) are similar to (1) (5) is much larger than (2)
However, even if you only cared about (1), GWWCās multiplier would still be about 10x, which would make them more cost-effective than GiveWell recommended charities, and equal to or more cost-effective than the other EA fundraising orgs (in the past). So Iām not sure itās even that decision relevant.