Hi Sjir, after quickly reading through the relevant parts in the detailed report, and following your conversation with Jeff, can I clarify that:
(1) Non-reporters are classified as donating $0, so that when you calculate the average donation per cohort it’s basically (sum of what reporters donate)/(cohort size including both reporters and non-reporters)?
(2) And the upshot of (1) would be that the average cohort donation each year is in fact an underestimate insofar as some nonreporters are in fact donation? I can speak for myself, at least—I’m part of the 2014 cohort, and am obviously still donating, but I have never reported anything to GWWC since that’s a hassle.
(3) Nonresponse bias is really hard to get around (as political science and polling is showing us these last few years), but you could probably try to get around this by either relying on the empirical literature on pledge attrition (e.g. https://www.sciencedirect.com/science/article/pii/S0167268122002992) or else just making a concerted push to reach out to non-reporters, and find the proportion who are still giving (though that in turn will also be subject to nonresponse bias, insofar as non-reporters twice over are different from non-reporters who respond to the concerted push (call them semi-reporters), and you would want to apply a further discount to your findings, perhaps based on the headline reporter/semi-reporter difference if you assume that reporter/semi-reporter donation difference = difference in semi-reporter/total non-reporter difference.
(4) My other big main concern beyond nonresponse bias is just counterfactuals, but looking at the report it’s clearly been very well-thought out, and I’m really impressed at the thoroughness and robustness. All I would add is that I would probably put more weight on the population base rate (even if you have to source US numbers), and even revise that downwards given the pool that EA draws from is distinctively non-religious and we have lower charitable donation rates.
(1) Are non-reporters counted as giving $0? Yes — at least for recorded donations (i.e., the donations that are within our database). For example, in cell C41 of our working sheet, we provide the average recorded donations of a GWWC Pledger in 2022-USD ($4,132), and this average assumes non-reporters are giving $0. Similarly, in our “pledge statistics” sheet, which provides the average amount we record being given per Pledger per cohort, and by year, we also assumed non-reporters are giving $0.
(2) Does this mean we are underestimating the amount given by Pledgers? Only for recorded donations — we also tried to account for donations made but that are not in our records. We discuss this more here —but in sum, for our best guess estimates, we estimated that our records only account for 79% of all pledge donations, and therefore we need to make an upwards adjustment of 1.27 to go from recorded donations to all donations made. We discuss how we arrived at this estimate pretty extensively in our appendix (with our methodology here being similar to how we analysed our counterfactual influence). For our conservative estimates, we did not make any recording adjustments, and we think this does underestimate the amount given by Pledgers.
(3) How did we handle nonresponse bias and could we handle it better? When estimating our counterfactual influence, we explicitly accounted for nonresponse bias. To do so, we treated respondents and nonrespondents separately, assuming a fraction of influence on nonrespondents compared to respondents for all surveys:
50% for our best-guess estimates.
25% for our conservative estimates.
We actually did consider adjusting this fraction depending on the survey we were looking at, and in our appendix we explain why we chose not to in each case. Could we handle this better? Definitely! I really appreciate your suggestions here — we explicitly outline handling nonresponse bias as one of the ways we would like to improve future evaluations.
(4) Could we incorporate population base rates of giving when considering our counterfactual influence? I’d love to hear more about this suggestion, it’s not obvious to me how we could do this. For example, one interpretation here would be to look at how much Pledgers are giving compared to the population base rate. Presumably, we’d find they are giving more. But I’m not sure how we could use that to inform our counterfactual influence, because there are at least two competing explanations for why they are giving more:
One explanation is that we are simply causing them to give more (so we should increase our estimated counterfactual influence).
Another is that we are just selecting for people who are already giving a lot more than the average population (in which case, we shouldn’t increase our estimated counterfactual influence).
But perhaps I’m missing the mark here, and this kind of reasoning/analysis is not really what you were thinking of. As I said, would love to hear more on this idea.
(Also, appreciate your kind words on the thoroughness/robustness)
Thanks for the clarifications, Michael, especially on non-reporters and non-response bias!
On base rates, my prior is that people who self select into GWWC pledges are naturally altruistic and so it’s right (as GWWC does) to use the more conservative estimate—but against this is a concern that self-reported counterfactual donation isn’t that accurate.
It’s really great that GWWC noted the issue of social desirability bias, but I suspect it works to overestimate counterfactual giving tendencies (rather than overestimating GWWC’s impact), since the desire to look generous almost certainly outweighs the desire to please GWWC (see research on donor overreporting: https://researchportal.bath.ac.uk/en/publications/dealing-with-social-desirability-bias-an-application-to-charitabl). I don’t have a good solution to this, insofar as standard list experiments aren’t great for dealing with quantification as opposed to yes/no answers—would be interested in hearing how your team plans to deal with this!
Hi Sjir, after quickly reading through the relevant parts in the detailed report, and following your conversation with Jeff, can I clarify that:
(1) Non-reporters are classified as donating $0, so that when you calculate the average donation per cohort it’s basically (sum of what reporters donate)/(cohort size including both reporters and non-reporters)?
(2) And the upshot of (1) would be that the average cohort donation each year is in fact an underestimate insofar as some nonreporters are in fact donation? I can speak for myself, at least—I’m part of the 2014 cohort, and am obviously still donating, but I have never reported anything to GWWC since that’s a hassle.
(3) Nonresponse bias is really hard to get around (as political science and polling is showing us these last few years), but you could probably try to get around this by either relying on the empirical literature on pledge attrition (e.g. https://www.sciencedirect.com/science/article/pii/S0167268122002992) or else just making a concerted push to reach out to non-reporters, and find the proportion who are still giving (though that in turn will also be subject to nonresponse bias, insofar as non-reporters twice over are different from non-reporters who respond to the concerted push (call them semi-reporters), and you would want to apply a further discount to your findings, perhaps based on the headline reporter/semi-reporter difference if you assume that reporter/semi-reporter donation difference = difference in semi-reporter/total non-reporter difference.
(4) My other big main concern beyond nonresponse bias is just counterfactuals, but looking at the report it’s clearly been very well-thought out, and I’m really impressed at the thoroughness and robustness. All I would add is that I would probably put more weight on the population base rate (even if you have to source US numbers), and even revise that downwards given the pool that EA draws from is distinctively non-religious and we have lower charitable donation rates.
Hi Joel — great questions!
(1) Are non-reporters counted as giving $0?
Yes — at least for recorded donations (i.e., the donations that are within our database). For example, in cell C41 of our working sheet, we provide the average recorded donations of a GWWC Pledger in 2022-USD ($4,132), and this average assumes non-reporters are giving $0. Similarly, in our “pledge statistics” sheet, which provides the average amount we record being given per Pledger per cohort, and by year, we also assumed non-reporters are giving $0.
(2) Does this mean we are underestimating the amount given by Pledgers?
Only for recorded donations — we also tried to account for donations made but that are not in our records. We discuss this more here —but in sum, for our best guess estimates, we estimated that our records only account for 79% of all pledge donations, and therefore we need to make an upwards adjustment of 1.27 to go from recorded donations to all donations made. We discuss how we arrived at this estimate pretty extensively in our appendix (with our methodology here being similar to how we analysed our counterfactual influence). For our conservative estimates, we did not make any recording adjustments, and we think this does underestimate the amount given by Pledgers.
(3) How did we handle nonresponse bias and could we handle it better?
When estimating our counterfactual influence, we explicitly accounted for nonresponse bias. To do so, we treated respondents and nonrespondents separately, assuming a fraction of influence on nonrespondents compared to respondents for all surveys:
50% for our best-guess estimates.
25% for our conservative estimates.
We actually did consider adjusting this fraction depending on the survey we were looking at, and in our appendix we explain why we chose not to in each case. Could we handle this better? Definitely! I really appreciate your suggestions here — we explicitly outline handling nonresponse bias as one of the ways we would like to improve future evaluations.
(4) Could we incorporate population base rates of giving when considering our counterfactual influence?
I’d love to hear more about this suggestion, it’s not obvious to me how we could do this. For example, one interpretation here would be to look at how much Pledgers are giving compared to the population base rate. Presumably, we’d find they are giving more. But I’m not sure how we could use that to inform our counterfactual influence, because there are at least two competing explanations for why they are giving more:
One explanation is that we are simply causing them to give more (so we should increase our estimated counterfactual influence).
Another is that we are just selecting for people who are already giving a lot more than the average population (in which case, we shouldn’t increase our estimated counterfactual influence).
But perhaps I’m missing the mark here, and this kind of reasoning/analysis is not really what you were thinking of. As I said, would love to hear more on this idea.
(Also, appreciate your kind words on the thoroughness/robustness)
Thanks for the clarifications, Michael, especially on non-reporters and non-response bias!
On base rates, my prior is that people who self select into GWWC pledges are naturally altruistic and so it’s right (as GWWC does) to use the more conservative estimate—but against this is a concern that self-reported counterfactual donation isn’t that accurate.
It’s really great that GWWC noted the issue of social desirability bias, but I suspect it works to overestimate counterfactual giving tendencies (rather than overestimating GWWC’s impact), since the desire to look generous almost certainly outweighs the desire to please GWWC (see research on donor overreporting: https://researchportal.bath.ac.uk/en/publications/dealing-with-social-desirability-bias-an-application-to-charitabl). I don’t have a good solution to this, insofar as standard list experiments aren’t great for dealing with quantification as opposed to yes/no answers—would be interested in hearing how your team plans to deal with this!