I’ve always been very pleasantly surprised that these things are so much of a success. My prior that you can just ask people to donate 10% for life and they will is very, very, very low—it just doesn’t seem like something that can happen and no other charity I know of does it. Everyone emphasizes making very small asks (though monthly donation asks are common) and ramping up steadily over time. So whenever I see the pledge drive work (twice now), I always feel like I’m looking at a miracle and wondering what the catch is.
My current working hypothesis is that the pledge works, but only on people who were already weakly involved in the EA movement and know other people who have pledged. So this would mean that the pledge event does create genuine impact by spurring people to action, but might be weak in (a) trying to build massive success outside EA and (b) in considering the counterfactual of whether pledgers who pledge during the drive would have just pledged later. (Overall, still probably worth doing.)
I think this makes sense of the evidence that you tried posting in various non-EA online groups that seem relevant but had low engagement and that the Tumblr posts went so well (since they were writing to somewhat EA audiences). This hypothesis also makes sense of the data that the 2014 pledge went better than the 2015 one despite the higher effort put into 2015, since in 2014 there were more EAs on the fence that you could convert over.
I’d use this hypothesis to predict that virtually no one will have joined from a Facebook ad and very few people will join from reading the HuffPo post. Instead, most of your people will report being somewhat familiar with EA before pledging and will come from Tumblr or FB posts. I’m super curious whether your interviews will validate this or not.
-
But clearly this is not counterfactual or time-adjusted. GWWC’s estimations is that each new pledger is worth ~60K in donations (counter-factual and time-adjusted).
If most of the new pledgers would have ended up taking the pledge anyway, the real counterfactual impact is the delta in how much sooner this pledge drive has resulted in people taking the pledge. For example, if you guesstimate that the average is one year.
I do think it’s worth seriously considering that you’re only bringing forward in time the pledge by a year or so.
Also, I think it’s worth seriously considering that the people you bring into the pledge through drive events will not be as committed as people who came to the pledge through their own volition (which constitute much of the people from which the $60K estimate was derived).
That being said, even if the 90 people are worth $600 each instead of $60K each, you’re still netting ~$22K/mo in counterfactually-adjusted impact, which seems clearly worth doing.
-
However, I think that this is too pessimistically low for three core reasons: 1)Haste consideration means that hearing about this pledge now is worth more than hearing about the pledge later
This doesn’t seem that valuable to me, except in so far as you get another year of donations. I would expect some social effect of getting more members sooner, but I expect this effect would not be large (say larger than the value of the additional year of donations) because my prior is that not that many GWWC members are successful at getting their friends to pledge… and even if they do get a friend to pledge, it will likely just bring them in one year sooner than they otherwise would.
-
However, since GWWC’s own “realistic” estimations is that their impact is ~100x, then the correct internal value of a new pledge relative to other things that GWWC could do, is about 600 dollars each.
This seems incorrect and needlessly unfair to the pledge event, since the pledge event could just be one of GWWC’s activities with the 100x ROI. Obviously GWWC has to do something and you should compare the pledge event to the calculated impact of other GWWC activities, but there’s no reason to just drop the impact by 100x.
-
2)The “average” of one year is too low: the median might be around there, but the right end of the counterfactual tail could end up being really high
I’ll admit that the “one year” number is quite made up, but I don’t see any reason to think that it is distributed with a wide right tail except that it tells a better story for counterfactuals. At minimum, I think chances are high that they’ll have just heard about it in the next pledge drive. ;)
Hi Peter, I really appreciate the detailed comments and nice words!!
“I’ve always been very pleasantly surprised that these things are so much of a success”
Agreed.
“My prior that you can just ask people to donate 10% for life and they will is very, very, very low—it just doesn’t seem like something that can happen and no other charity I know of does it.” Well, there are many things other charities don’t know and EAs do. :P
“This hypothesis also makes sense of the data that the 2014 pledge went better than the 2015 one despite the higher effort put into 2015, since in 2014 there were more EAs on the fence that you could convert over.”
Hmm...I would be surprised if there are more EAs on the fence in 2014 than 2015. I think there are specific features of the 2014 pledge drive that I will talk about in a follow-up post (not least the sheer degree that the Cambridge group was able to mobilize) that are difficult to reproduce. That said, we have more people pledging in Jan 2016 (104 so far, and still growing as of Jan 25th) than in Jan 2015 (94). (But December numbers are lower, 117 for Dec.2015 vs. 129 for Dec.2014, and since the base rate for pledges is considerably higher in 2015 vs. 2014, on balance I will definitely agree that 2014 was the more successful drive).
“I’d use this hypothesis to predict that virtually no one will have joined from a Facebook ad and very few people will join from reading the HuffPo post”
Agreed. That said, when the base rate is 50-60/month in a normal month and ~150 during the pledge drive, any nonzero number is significant! Another point to make is that the counterfactual value of people who pledge from HuffPo articles or FB ads is likely greater than the counterfactual value of pledges who hear from their friends.
“even if they do get a friend to pledge, it will likely just bring them in one year sooner than they otherwise would.”
Hmm...how will you realistically define the Haste consideration? I generally think of it as “A persuades B to join x months earlier who persuades C to join y months earlier, etc.” Put another way, the entirety of the Haste consideration is shifting the curve earlier, I don’t think persuading people earlier will significantly change the final number of the people who wound up doing EA things. That said, when the growth is exponential, I think persuading people to join earlier is quite significant.
“This seems incorrect and needlessly unfair to the pledge event, since the pledge event could just be one of GWWC’s activities with the 100x ROI.”
Not really. Consider: Assuming that the average ROI of GWWC’s activities are 100x. Then if GWWC doesn’t get unusually high funding, a simplified model would be that the organization should prioritize activities that have 100x or higher ROI before bothering with activities that have say 50x, or 5x. And from the perspective of an individual deciding between working on meta activities themselves or taking on a high-paying job and donating to GWWC or other meta orgs, this calculation might be helpful.
″ I don’t see any reason to think that it is distributed with a wide right tail except that it tells a better story for counterfactuals” Thanks for catching that: I actually got my directions wrong! I meant to say it is likely distributed with a LONG right tail, not high. I think this is defensible. The (like you said, probably very few) people who came across the pledge drive through serendipity or random happenstance rather than a close EA social network might counterfactually have taken years to hear about the pledge, and you don’t need that many of them to change the average considerably.
My prior that you can just ask people to donate 10% for life and they will is very, very, very low—it just doesn’t seem like something that can happen and no other charity I know of does it.
Well, there are many things other charities don’t know and EAs do. :P
That doesn’t seem fair to me. I don’t know any evidence that suggests EAs outperform professional non-EA fundraisers at fundraising.
-
This hypothesis also makes sense of the data that the 2014 pledge went better than the 2015 one despite the higher effort put into 2015, since in 2014 there were more EAs on the fence that you could convert over.
I think there are specific features of the 2014 pledge drive that I will talk about in a follow-up post (not least the sheer degree that the Cambridge group was able to mobilize) that are difficult to reproduce.
Yeah, you’d have to elaborate on that. ;)
-
even if they do get a friend to pledge, it will likely just bring them in one year sooner than they otherwise would.
Hmm...how will you realistically define the Haste consideration? I generally think of it as “A persuades B to join x months earlier who persuades C to join y months earlier, etc.” Put another way, the entirety of the Haste consideration is shifting the curve earlier, I don’t think persuading people earlier will significantly change the final number of the people who wound up doing EA things. That said, when the growth is exponential, I think persuading people to join earlier is quite significant.
I’m unimpressed by the haste consideration. You’d have to make it more concrete—what additional benefits do you expect to get beyond having one more year of donations? How do you expect those benefits to compound? Do they compound that way in practice?
-
This seems incorrect and needlessly unfair to the pledge event, since the pledge event could just be one of GWWC’s activities with the 100x ROI.
Not really. Consider: Assuming that the average ROI of GWWC’s activities are 100x. Then if GWWC doesn’t get unusually high funding, a simplified model would be that the organization should prioritize activities that have 100x or higher ROI before bothering with activities that have say 50x, or 5x. And from the perspective of an individual deciding between working on meta activities themselves or taking on a high-paying job and donating to GWWC or other meta orgs, this calculation might be helpful.
Well somebody needs to actually do the direct work—you can’t have 100% earn-to-give and 0% direct work or nothing actually gets done. So not everyone can use this argument. I got at this using a crank analogy earlier.
So the consideration for you is not what GWWC’s absolute ROI on activities is, but rather...
(a) is working on this GWWC project mean that a higher ROI project does not get done?
(b) could you earn to give and pay for someone to do your GWWC work for you and then some? If so, how much better are you at that GWWC work?
Even if the project has enormous ROI, I imagine that it is the case you could have paid someone else to do it and come out on a profit. But it’s possible that GWWC’s budget is tight and/or there isn’t anyone around who would do the work nearly as well.
-
I don’t see any reason to think that it is distributed with a wide right tail except that it tells a better story for counterfactuals
Thanks for catching that: I actually got my directions wrong! I meant to say it is likely distributed with a LONG right tail, not high. I think this is defensible. The (like you said, probably very few) people who came across the pledge drive through serendipity or random happenstance rather than a close EA social network might counterfactually have taken years to hear about the pledge, and you don’t need that many of them to change the average considerably.
I agree that’s defensible and possibly 2x-10x’s the total ROI of the project. But I’d also be concerned that these pledgers are lower quality on average and unlikely to return the full $60K estimated value.
“That doesn’t seem fair to me. I don’t know any evidence that suggests EAs outperform professional non-EA fundraisers at fundraising.”
It’s easier to get a long-term pledge for ‘whatever charity looks best at the time’ than for a particular charity you are working for at the time.
So it’s no surprise normal fundraisers, who are paid to raise money for their single present employer, will have left GWWC’s approach alone, even if it is much more efficient—which it seems to be.
So it’s no surprise normal fundraisers, who are paid to raise money for their single present employer, will have left GWWC’s approach alone, even if it is much more efficient
That strikes me as overconfident. Normal fundraisers ask for regular, life-long giving all the time—just in the form of a small, monthly donation. They could be asking for a monthly donation of 10% of your income, but they don’t.
Though, of course, there are lots of reasons why a good idea might be ignored by the mainstream.
The thing they don’t get you to do is pledge to give for the rest of your life. Because if you’re committed to just one organisation (their employer) almost nobody would say yes. But GWWC has shown people do say yes at a reasonable rate if the recipient is flexible.
“I don’t know any evidence that suggests EAs outperform professional non-EA fundraisers at fundraising.”
GWWC’s experience make me think it’s above a 10:1 ratio, which is apparently typical for commercial fundraisers.
I will think about the other points later. But I definitely agree that there’s a distinct possibility that people join because of the pledge drive are less likely to be committed than people who join normally, and that this worry, if true, will be a significant concern to the validity of any impact measurement for the drive.
That said, as a separate point I actually think (~60% confidence) that people who join from hearing about the drive through the media, rather than from a social network initially high in EAs, are more likely to be committed rather than less. This is because they had less initial contact with EAs before being convinced of the idea, suggesting that they had strong innate urges in that direction that did not require a lot of “push” from a social environment conducive to EAness. Does that make sense?
Another data point is how a sizeable minority of EAs (over-represented here and on the FB groups) barely need any convincing for EA...it just made sense as soon as they heard about it.
However, even if my theory about media pledgers being more committed than pledgers from a peer group high in EAs, this is not necessarily evidence against the worry that the pledge drive’s pledgers are on average less committed than normal, since like you said, most people who joined from the pledge drive are not due to media outreach.
This is because they had less initial contact with EAs before being convinced of the idea, suggesting that they had strong innate urges in that direction that did not require a lot of “push” from a social environment conducive to EAness. Does that make sense?
Yeah, I could buy that. If there’s a certain amount of people that will just get EA immediately (which seems plausible based on what we know so far), then it makes sense that the pledge drive—like any other media campaign—would uncover some of those people.
Hey, awesome!
I’ve always been very pleasantly surprised that these things are so much of a success. My prior that you can just ask people to donate 10% for life and they will is very, very, very low—it just doesn’t seem like something that can happen and no other charity I know of does it. Everyone emphasizes making very small asks (though monthly donation asks are common) and ramping up steadily over time. So whenever I see the pledge drive work (twice now), I always feel like I’m looking at a miracle and wondering what the catch is.
My current working hypothesis is that the pledge works, but only on people who were already weakly involved in the EA movement and know other people who have pledged. So this would mean that the pledge event does create genuine impact by spurring people to action, but might be weak in (a) trying to build massive success outside EA and (b) in considering the counterfactual of whether pledgers who pledge during the drive would have just pledged later. (Overall, still probably worth doing.)
I think this makes sense of the evidence that you tried posting in various non-EA online groups that seem relevant but had low engagement and that the Tumblr posts went so well (since they were writing to somewhat EA audiences). This hypothesis also makes sense of the data that the 2014 pledge went better than the 2015 one despite the higher effort put into 2015, since in 2014 there were more EAs on the fence that you could convert over.
I’d use this hypothesis to predict that virtually no one will have joined from a Facebook ad and very few people will join from reading the HuffPo post. Instead, most of your people will report being somewhat familiar with EA before pledging and will come from Tumblr or FB posts. I’m super curious whether your interviews will validate this or not.
-
I do think it’s worth seriously considering that you’re only bringing forward in time the pledge by a year or so.
Also, I think it’s worth seriously considering that the people you bring into the pledge through drive events will not be as committed as people who came to the pledge through their own volition (which constitute much of the people from which the $60K estimate was derived).
That being said, even if the 90 people are worth $600 each instead of $60K each, you’re still netting ~$22K/mo in counterfactually-adjusted impact, which seems clearly worth doing.
-
This doesn’t seem that valuable to me, except in so far as you get another year of donations. I would expect some social effect of getting more members sooner, but I expect this effect would not be large (say larger than the value of the additional year of donations) because my prior is that not that many GWWC members are successful at getting their friends to pledge… and even if they do get a friend to pledge, it will likely just bring them in one year sooner than they otherwise would.
-
This seems incorrect and needlessly unfair to the pledge event, since the pledge event could just be one of GWWC’s activities with the 100x ROI. Obviously GWWC has to do something and you should compare the pledge event to the calculated impact of other GWWC activities, but there’s no reason to just drop the impact by 100x.
-
I’ll admit that the “one year” number is quite made up, but I don’t see any reason to think that it is distributed with a wide right tail except that it tells a better story for counterfactuals. At minimum, I think chances are high that they’ll have just heard about it in the next pledge drive. ;)
Hi Peter, I really appreciate the detailed comments and nice words!!
“I’ve always been very pleasantly surprised that these things are so much of a success” Agreed.
“My prior that you can just ask people to donate 10% for life and they will is very, very, very low—it just doesn’t seem like something that can happen and no other charity I know of does it.” Well, there are many things other charities don’t know and EAs do. :P
“My current working hypothesis is that the pledge works, but only on people who were already weakly involved in the EA movement and know other people who have pledged.” Yes, this is my working hypothesis as well. It’s actually explicitly addressed in GWWC’s “Membership Pathway Model”: https://www.givingwhatwecan.org/sites/givingwhatwecan.org/files/attachments/givingwhatwecanpathway.pdf
“This hypothesis also makes sense of the data that the 2014 pledge went better than the 2015 one despite the higher effort put into 2015, since in 2014 there were more EAs on the fence that you could convert over.” Hmm...I would be surprised if there are more EAs on the fence in 2014 than 2015. I think there are specific features of the 2014 pledge drive that I will talk about in a follow-up post (not least the sheer degree that the Cambridge group was able to mobilize) that are difficult to reproduce. That said, we have more people pledging in Jan 2016 (104 so far, and still growing as of Jan 25th) than in Jan 2015 (94). (But December numbers are lower, 117 for Dec.2015 vs. 129 for Dec.2014, and since the base rate for pledges is considerably higher in 2015 vs. 2014, on balance I will definitely agree that 2014 was the more successful drive).
“I’d use this hypothesis to predict that virtually no one will have joined from a Facebook ad and very few people will join from reading the HuffPo post” Agreed. That said, when the base rate is 50-60/month in a normal month and ~150 during the pledge drive, any nonzero number is significant! Another point to make is that the counterfactual value of people who pledge from HuffPo articles or FB ads is likely greater than the counterfactual value of pledges who hear from their friends.
“even if they do get a friend to pledge, it will likely just bring them in one year sooner than they otherwise would.” Hmm...how will you realistically define the Haste consideration? I generally think of it as “A persuades B to join x months earlier who persuades C to join y months earlier, etc.” Put another way, the entirety of the Haste consideration is shifting the curve earlier, I don’t think persuading people earlier will significantly change the final number of the people who wound up doing EA things. That said, when the growth is exponential, I think persuading people to join earlier is quite significant.
“This seems incorrect and needlessly unfair to the pledge event, since the pledge event could just be one of GWWC’s activities with the 100x ROI.” Not really. Consider: Assuming that the average ROI of GWWC’s activities are 100x. Then if GWWC doesn’t get unusually high funding, a simplified model would be that the organization should prioritize activities that have 100x or higher ROI before bothering with activities that have say 50x, or 5x. And from the perspective of an individual deciding between working on meta activities themselves or taking on a high-paying job and donating to GWWC or other meta orgs, this calculation might be helpful.
″ I don’t see any reason to think that it is distributed with a wide right tail except that it tells a better story for counterfactuals” Thanks for catching that: I actually got my directions wrong! I meant to say it is likely distributed with a LONG right tail, not high. I think this is defensible. The (like you said, probably very few) people who came across the pledge drive through serendipity or random happenstance rather than a close EA social network might counterfactually have taken years to hear about the pledge, and you don’t need that many of them to change the average considerably.
Thanks a lot for your detailed comments!!
That doesn’t seem fair to me. I don’t know any evidence that suggests EAs outperform professional non-EA fundraisers at fundraising.
-
Yeah, you’d have to elaborate on that. ;)
-
I’m unimpressed by the haste consideration. You’d have to make it more concrete—what additional benefits do you expect to get beyond having one more year of donations? How do you expect those benefits to compound? Do they compound that way in practice?
-
Well somebody needs to actually do the direct work—you can’t have 100% earn-to-give and 0% direct work or nothing actually gets done. So not everyone can use this argument. I got at this using a crank analogy earlier.
So the consideration for you is not what GWWC’s absolute ROI on activities is, but rather...
(a) is working on this GWWC project mean that a higher ROI project does not get done?
(b) could you earn to give and pay for someone to do your GWWC work for you and then some? If so, how much better are you at that GWWC work?
Even if the project has enormous ROI, I imagine that it is the case you could have paid someone else to do it and come out on a profit. But it’s possible that GWWC’s budget is tight and/or there isn’t anyone around who would do the work nearly as well.
-
I agree that’s defensible and possibly 2x-10x’s the total ROI of the project. But I’d also be concerned that these pledgers are lower quality on average and unlikely to return the full $60K estimated value.
“That doesn’t seem fair to me. I don’t know any evidence that suggests EAs outperform professional non-EA fundraisers at fundraising.”
It’s easier to get a long-term pledge for ‘whatever charity looks best at the time’ than for a particular charity you are working for at the time.
So it’s no surprise normal fundraisers, who are paid to raise money for their single present employer, will have left GWWC’s approach alone, even if it is much more efficient—which it seems to be.
That strikes me as overconfident. Normal fundraisers ask for regular, life-long giving all the time—just in the form of a small, monthly donation. They could be asking for a monthly donation of 10% of your income, but they don’t.
Though, of course, there are lots of reasons why a good idea might be ignored by the mainstream.
The thing they don’t get you to do is pledge to give for the rest of your life. Because if you’re committed to just one organisation (their employer) almost nobody would say yes. But GWWC has shown people do say yes at a reasonable rate if the recipient is flexible.
“I don’t know any evidence that suggests EAs outperform professional non-EA fundraisers at fundraising.”
GWWC’s experience make me think it’s above a 10:1 ratio, which is apparently typical for commercial fundraisers.
I will think about the other points later. But I definitely agree that there’s a distinct possibility that people join because of the pledge drive are less likely to be committed than people who join normally, and that this worry, if true, will be a significant concern to the validity of any impact measurement for the drive.
That said, as a separate point I actually think (~60% confidence) that people who join from hearing about the drive through the media, rather than from a social network initially high in EAs, are more likely to be committed rather than less. This is because they had less initial contact with EAs before being convinced of the idea, suggesting that they had strong innate urges in that direction that did not require a lot of “push” from a social environment conducive to EAness. Does that make sense?
Another data point is how a sizeable minority of EAs (over-represented here and on the FB groups) barely need any convincing for EA...it just made sense as soon as they heard about it.
However, even if my theory about media pledgers being more committed than pledgers from a peer group high in EAs, this is not necessarily evidence against the worry that the pledge drive’s pledgers are on average less committed than normal, since like you said, most people who joined from the pledge drive are not due to media outreach.
Yeah, I could buy that. If there’s a certain amount of people that will just get EA immediately (which seems plausible based on what we know so far), then it makes sense that the pledge drive—like any other media campaign—would uncover some of those people.