Thanks, Michael & Sjir, for writing this exhaustive report! The donation distribution is fascinating, and I loved reading your reasoning about & estimation of quantities I didn’t know were important. I’m glad GWWC has such a great giving multiplier!
After reading the summary & skimming the full report & working sheet, I’ve got three questions:
1. Does the dropping of the top 10 donors influence the decision on which groups to target? I think the reasoning to drop their 27% of donations from the giving multiplier is clear, but it’s not obvious to me that they should also be dropped when prioritizing, and I could not find this in the report. (You mention the multiplier is not the only thing that influences which groups to target, of course. To me the overall distribution seems to imply running a top-100 or top-200 survey could be useful in increasing the multiplier somehow, and this argument would be about twice as strong with the top 10 included vs without.)
2. Why did you choose the counterfactual as “what would have happened had GWWC never existed”? I think it’s somewhat plausible that the pre-2020 pledgees would still have donated 10% of their income from 2020 onward, had GWWC existed from 2020 onward in a very minimal form. This is about ~35% of the total donations, so this also seems a relevant counterfactual.
3. Similar to 2, I also didn’t quite get how you dealt with which GWWC year gets to count the impact from past pledges. It would seem off to me if both GWWC 2020-2022 and GWWC 2017-2019 get to fully count the donations a 2018 pledge-taker made in 2022 for their multipliers, although both of them definitely contributed to the donations, of course. How did you deal with this? Maybe this is addressed by the survey responses already?
The donation distribution statistics do not exclude our top 10 donors (neither on the pledge nor on the non-pledge side), so our takeaways from those aren’t influenced.
I should also clarify that we do not exclude all top 10 donors (on either the pledge or non-pledge side) from all of our donation estimates that influence the giving multiplier: we only exclude all top 10 pledge donors from our estimates of the value of the pledge (for more detail on how we treated large donors differently and why, see this appendix).
Please also note that we haven’t (yet) made any decisions on which groups to target more or less on the basis of these results: our takeaways will inform our organisation-wide strategic discussions going forwards but we haven’t had any of those yet—as we have only just finished this evaluation—and—as we also emphasize in the report—the takeaways provide updates on our views but not our all-things-considered views.
This is a great question and I think points at the importance of considering the difference between marginal and average cost-effectiveness when interpreting our findings: as you say, many people might have still donated “had GWWC existed from 2020 in a very minimal form”, i.e. the first few dollars spent on GWWC may be worth a lot more than the last few dollars spent on it. As we note in the plans for future evaluations section, we are interested in making further estimates like the one you suggest (i.e. considering only a part of our activities) and will consider doing so in the future; we just didn’t get to doing this in this evaluation and chose to make an estimate of our total impact (i.e. considering us not existing as the counterfactual) first.
We explain how we deal with this in this appendix. In short, we use two approaches, one of which counts all the impact of a pledge in the year the pledge is taken and the other in the years donations are made against that pledge. We take a weighted average of these two approaches to avoid double-counting our impact across different years.
Thanks, Michael & Sjir, for writing this exhaustive report! The donation distribution is fascinating, and I loved reading your reasoning about & estimation of quantities I didn’t know were important. I’m glad GWWC has such a great giving multiplier!
After reading the summary & skimming the full report & working sheet, I’ve got three questions:
1. Does the dropping of the top 10 donors influence the decision on which groups to target? I think the reasoning to drop their 27% of donations from the giving multiplier is clear, but it’s not obvious to me that they should also be dropped when prioritizing, and I could not find this in the report. (You mention the multiplier is not the only thing that influences which groups to target, of course. To me the overall distribution seems to imply running a top-100 or top-200 survey could be useful in increasing the multiplier somehow, and this argument would be about twice as strong with the top 10 included vs without.)
2. Why did you choose the counterfactual as “what would have happened had GWWC never existed”? I think it’s somewhat plausible that the pre-2020 pledgees would still have donated 10% of their income from 2020 onward, had GWWC existed from 2020 onward in a very minimal form. This is about ~35% of the total donations, so this also seems a relevant counterfactual.
3. Similar to 2, I also didn’t quite get how you dealt with which GWWC year gets to count the impact from past pledges. It would seem off to me if both GWWC 2020-2022 and GWWC 2017-2019 get to fully count the donations a 2018 pledge-taker made in 2022 for their multipliers, although both of them definitely contributed to the donations, of course. How did you deal with this? Maybe this is addressed by the survey responses already?
Thanks for asking these in-depth questions, Stan!
The donation distribution statistics do not exclude our top 10 donors (neither on the pledge nor on the non-pledge side), so our takeaways from those aren’t influenced. I should also clarify that we do not exclude all top 10 donors (on either the pledge or non-pledge side) from all of our donation estimates that influence the giving multiplier: we only exclude all top 10 pledge donors from our estimates of the value of the pledge (for more detail on how we treated large donors differently and why, see this appendix). Please also note that we haven’t (yet) made any decisions on which groups to target more or less on the basis of these results: our takeaways will inform our organisation-wide strategic discussions going forwards but we haven’t had any of those yet—as we have only just finished this evaluation—and—as we also emphasize in the report—the takeaways provide updates on our views but not our all-things-considered views.
This is a great question and I think points at the importance of considering the difference between marginal and average cost-effectiveness when interpreting our findings: as you say, many people might have still donated “had GWWC existed from 2020 in a very minimal form”, i.e. the first few dollars spent on GWWC may be worth a lot more than the last few dollars spent on it. As we note in the plans for future evaluations section, we are interested in making further estimates like the one you suggest (i.e. considering only a part of our activities) and will consider doing so in the future; we just didn’t get to doing this in this evaluation and chose to make an estimate of our total impact (i.e. considering us not existing as the counterfactual) first.
We explain how we deal with this in this appendix. In short, we use two approaches, one of which counts all the impact of a pledge in the year the pledge is taken and the other in the years donations are made against that pledge. We take a weighted average of these two approaches to avoid double-counting our impact across different years.
Thank you for the explanation & references, all three points make sense to me!