Thank you very much for sharing the cohort analysis. I find it very valuable to understanding GWWC’s membership flow. I also know it’s stressful to release the data and have it publicly scrutinized, especially during a high-stakes fundraising round. But I think it’s very important to have such public scrutiny when requesting public funds. :)
I have seven questions:
1.) I see from the Giving Review that GWWC had 386 members who joined prior to 2014, with 186 “going silent” (not responding with income and donation numbers) and 43 not completing the pledge, for an overall 40.7% “knowably kept pledge” rate.
But in the Fundraising doc, the “Realistic calculation” has a rate of leavers of 0.017 and a rate of people going silent of 0.031, for an overall estimate of “knowably kept pledge” rate of 95.2%. Do you have more information on how the attrition rate was calculated in the “Realistic calculation” and why these two numbers are so different?
-
2.) You state a counterfactual donation rate of 51%. Can you explain where that number comes from?
-
3.) You say that 66% of people say you’ve affected their choice of charity. Is this 66% of the entire population, or 66% of the 49% of people who would have donated anyway?
-
4.) What does the “effectiveness ratio” of 0.5 mean and how was it calculated?
-
5.) What credence, if any, do you give to the thought that the outside view for pledging working so successfully is pretty weak? My impression is that fundraising doesn’t make much use of pledges, hough they do frequently seek out “monthly donations” over one-time giving). If it were more successful, would other orgs do it? (Of course, the inside view could still be strong and it could be that GWWC has caught on to something innovative that other people just haven’t seen yet. And I suppose other orgs like REG have also had a lot of success with pledges.)
-
6.) This is a positive question where I think GWWC underestimates it’s impact—what credence do you give to Ben Todd’s position that EA is underemphasizing potential for growth and overestimating historical money moved metrics? Do you think GWWC is in a position to try new things it hasn’t before? What kind of learning value do you think is in that? Do you think GWWC will have more economies of scale in getting new pledgers as it grows?
-
7.) I hate to move the goalposts after releasing the cohort analysis, but I think what we ideally would like is “survival analysis”. I find it interesting to compare how likely a GWWC pledger is to keep a pledge to how likely a person taking out a loan is to pay back their loan (my job is to calculate that second one). One tool we use is survival analysis, which is kind of like an improved cohort analysis, where we make predictions like given that someone paid back in year 1, how likely are they to pay back in year 2, etc.? My guess is that we could do something very similar for GWWC members and I’d be happy to help do that when my time allows (maybe sometime early next year).
I’d also be really interested in using any data we have on members (e.g., demographics, income, how they heard about GWWC, level of contact with GWWC, etc.) to help predict which members are more likely to keep their pledge.
Thanks for your questions. As you say, it’s very important to make sure people have a really thorough understanding of what’s going on, both with their money, and in general in the sphere of helping others, so that we can build on each other’s work.
You might like to check out our impact page, which has more detail on the methodology of our calculations.
1) Silent members in the fundraising document are defined as members who we have not had any contact with/information from in at least 2 years i.e. people who fulfil the following: joined before 2013, have not filled in My Giving in at least 2 years, not known to staff at CEA personally, has not joined the online GWWC community facebook group (set up at the start of 2014), has not responded to any emails in at least 2 years. There are therefore relatively few of them compared with the number of members who we couldn’t persuade to fill out My Giving in a given year. It’s worth noting that while you might think that filling in My Giving is a small thing compared to giving 10% of your income, we know many people personally who do donate at least 10% but can’t be persuaded to fill in My Giving—many people just find recording donations in any way a big hassle.
The claim in the fundraising document is not that 95.2% of members will donate their pledge each year—it’s more realistically estimating the % of members who will stop donating per year, who will not resume donating in future years.
2) From looking at the % members pledged and subtracting the percentage members say they would have given anyway, and dividing by the percentage pledged for each member, then averaging over these. More info on the Impact page of the website: https://www.givingwhatwecan.org/impact/#realistic-impact-calculation-1031
3) 66% of the entire population
4) This represents how much more effective the charities people give to are than the ones they would have given to otherwise, for the case where people say we affected their donation. It’s simply an estimate, based on the reasoning that we’re probably if we affected someone’s donation it will be by nudging them to give in the developing world rather than developed world, or by nudging them from a typical developing world charity to one of our recommended charities. I therefore estimated that on average it might be something like a doubling in impact. That seems like a pretty conservative assumption to me.
5) I wouldn’t say there are no other examples of it—tithing works quite like this, and there are various examples of people encouraging each other to ‘promise’ to uphold some value that’s deemed important (I remember learning about the ‘silver ring thing’ in the US when I was at school).
I think the general principles of banding together to support each other upholding your values are fairly well established, and the outside view seems to be positive about the general psychological principles underlying pledging and forming a community. Most fundraisers are fundraising for one particular charity, and it doesn’t seem terribly surprising that pledges would seem less appealing if someone’s asking you just to pledge to their organisation. I also think that we shouldn’t be thinking of taking the pledge as the only thing—that’s the measurable thing, and the time when we can ask people the most easily about how we affect their donations. But in practice, the reason people pledge is to do with the overall community, with the information we can provide etc. So we shouldn’t think of it as ‘you put a pledge page online and then lots of behaviour changes’ which would be really surprising, but in a more holistic way.
Overall, I think it is somewhat surprising how the concept seems to be working. But my surprise is more with how many people are willing to actually give 10% of their income to effective charities, than with the fact that there being other people around inspiring them and holding them to account would have a big effect on that willingness.
6) Yes, I basically agree with Ben Todd on that, that’s why we tried to emphasise in our prospectus the importance of looking at our plan for growth, rather than the leverage numbers.
We’re always trying new ways of reaching out to people, changing the website to improve flow through and test different messages etc. I think it’s important we have credibility as a stable organisation in terms of the pledge remaining the same, and continuing to have regular contact with members. But when it comes to trying out different ways of outreach, we can be continuously improving.
Yes, I think there will be economies of scale. Eg. getting really good chapter resources made and online is high intensity, but can then be used by however many chapters we have. Last year we made a marginal assessment for members, but we were trying to focus more on our plans this year than on straightforward leverage ratios, so didn’t include that calculation.
7) Sounds good, I look forward to hearing from you when you have time for it.
P.S. One of my friends brings up a good point to me that ideally you want dollar-weighted attrition, where you look at the total amount of money donated in year N divided by the total amount of money pledged in year N-1 for year N. It’s possible this number could be above 100% if people underestimate how much they’ll donate.
P.P.S. Though for understanding a model of how GWWC members come and go, regular attrition is the preferred metric, though you may want to have a slight bias towards looking at employed members.
Hey Michelle,
Thank you very much for sharing the cohort analysis. I find it very valuable to understanding GWWC’s membership flow. I also know it’s stressful to release the data and have it publicly scrutinized, especially during a high-stakes fundraising round. But I think it’s very important to have such public scrutiny when requesting public funds. :)
I have seven questions:
1.) I see from the Giving Review that GWWC had 386 members who joined prior to 2014, with 186 “going silent” (not responding with income and donation numbers) and 43 not completing the pledge, for an overall 40.7% “knowably kept pledge” rate.
But in the Fundraising doc, the “Realistic calculation” has a rate of leavers of 0.017 and a rate of people going silent of 0.031, for an overall estimate of “knowably kept pledge” rate of 95.2%. Do you have more information on how the attrition rate was calculated in the “Realistic calculation” and why these two numbers are so different?
-
2.) You state a counterfactual donation rate of 51%. Can you explain where that number comes from?
-
3.) You say that 66% of people say you’ve affected their choice of charity. Is this 66% of the entire population, or 66% of the 49% of people who would have donated anyway?
-
4.) What does the “effectiveness ratio” of 0.5 mean and how was it calculated?
-
5.) What credence, if any, do you give to the thought that the outside view for pledging working so successfully is pretty weak? My impression is that fundraising doesn’t make much use of pledges, hough they do frequently seek out “monthly donations” over one-time giving). If it were more successful, would other orgs do it? (Of course, the inside view could still be strong and it could be that GWWC has caught on to something innovative that other people just haven’t seen yet. And I suppose other orgs like REG have also had a lot of success with pledges.)
-
6.) This is a positive question where I think GWWC underestimates it’s impact—what credence do you give to Ben Todd’s position that EA is underemphasizing potential for growth and overestimating historical money moved metrics? Do you think GWWC is in a position to try new things it hasn’t before? What kind of learning value do you think is in that? Do you think GWWC will have more economies of scale in getting new pledgers as it grows?
-
7.) I hate to move the goalposts after releasing the cohort analysis, but I think what we ideally would like is “survival analysis”. I find it interesting to compare how likely a GWWC pledger is to keep a pledge to how likely a person taking out a loan is to pay back their loan (my job is to calculate that second one). One tool we use is survival analysis, which is kind of like an improved cohort analysis, where we make predictions like given that someone paid back in year 1, how likely are they to pay back in year 2, etc.? My guess is that we could do something very similar for GWWC members and I’d be happy to help do that when my time allows (maybe sometime early next year).
I’d also be really interested in using any data we have on members (e.g., demographics, income, how they heard about GWWC, level of contact with GWWC, etc.) to help predict which members are more likely to keep their pledge.
Hi Peter,
Thanks for your questions. As you say, it’s very important to make sure people have a really thorough understanding of what’s going on, both with their money, and in general in the sphere of helping others, so that we can build on each other’s work. You might like to check out our impact page, which has more detail on the methodology of our calculations.
1) Silent members in the fundraising document are defined as members who we have not had any contact with/information from in at least 2 years i.e. people who fulfil the following: joined before 2013, have not filled in My Giving in at least 2 years, not known to staff at CEA personally, has not joined the online GWWC community facebook group (set up at the start of 2014), has not responded to any emails in at least 2 years. There are therefore relatively few of them compared with the number of members who we couldn’t persuade to fill out My Giving in a given year. It’s worth noting that while you might think that filling in My Giving is a small thing compared to giving 10% of your income, we know many people personally who do donate at least 10% but can’t be persuaded to fill in My Giving—many people just find recording donations in any way a big hassle. The claim in the fundraising document is not that 95.2% of members will donate their pledge each year—it’s more realistically estimating the % of members who will stop donating per year, who will not resume donating in future years.
2) From looking at the % members pledged and subtracting the percentage members say they would have given anyway, and dividing by the percentage pledged for each member, then averaging over these. More info on the Impact page of the website: https://www.givingwhatwecan.org/impact/#realistic-impact-calculation-1031
3) 66% of the entire population
4) This represents how much more effective the charities people give to are than the ones they would have given to otherwise, for the case where people say we affected their donation. It’s simply an estimate, based on the reasoning that we’re probably if we affected someone’s donation it will be by nudging them to give in the developing world rather than developed world, or by nudging them from a typical developing world charity to one of our recommended charities. I therefore estimated that on average it might be something like a doubling in impact. That seems like a pretty conservative assumption to me.
5) I wouldn’t say there are no other examples of it—tithing works quite like this, and there are various examples of people encouraging each other to ‘promise’ to uphold some value that’s deemed important (I remember learning about the ‘silver ring thing’ in the US when I was at school). I think the general principles of banding together to support each other upholding your values are fairly well established, and the outside view seems to be positive about the general psychological principles underlying pledging and forming a community. Most fundraisers are fundraising for one particular charity, and it doesn’t seem terribly surprising that pledges would seem less appealing if someone’s asking you just to pledge to their organisation. I also think that we shouldn’t be thinking of taking the pledge as the only thing—that’s the measurable thing, and the time when we can ask people the most easily about how we affect their donations. But in practice, the reason people pledge is to do with the overall community, with the information we can provide etc. So we shouldn’t think of it as ‘you put a pledge page online and then lots of behaviour changes’ which would be really surprising, but in a more holistic way. Overall, I think it is somewhat surprising how the concept seems to be working. But my surprise is more with how many people are willing to actually give 10% of their income to effective charities, than with the fact that there being other people around inspiring them and holding them to account would have a big effect on that willingness.
6) Yes, I basically agree with Ben Todd on that, that’s why we tried to emphasise in our prospectus the importance of looking at our plan for growth, rather than the leverage numbers. We’re always trying new ways of reaching out to people, changing the website to improve flow through and test different messages etc. I think it’s important we have credibility as a stable organisation in terms of the pledge remaining the same, and continuing to have regular contact with members. But when it comes to trying out different ways of outreach, we can be continuously improving. Yes, I think there will be economies of scale. Eg. getting really good chapter resources made and online is high intensity, but can then be used by however many chapters we have. Last year we made a marginal assessment for members, but we were trying to focus more on our plans this year than on straightforward leverage ratios, so didn’t include that calculation.
7) Sounds good, I look forward to hearing from you when you have time for it.
Happy to help with the analysis project alluded to above.
Sounds great—maybe we could collaborate. :)
P.S. One of my friends brings up a good point to me that ideally you want dollar-weighted attrition, where you look at the total amount of money donated in year N divided by the total amount of money pledged in year N-1 for year N. It’s possible this number could be above 100% if people underestimate how much they’ll donate.
P.P.S. Though for understanding a model of how GWWC members come and go, regular attrition is the preferred metric, though you may want to have a slight bias towards looking at employed members.