Proposal for an Pre-registered Experiment in EA Outreach
Can talking about GWWC for 90 minutes actually get somebody to take the Pledge?
Note: I timed myself so I will not take more than 30 mins on the first draft (and hopefully less than an hour on this post overall)
Surface analysis of Giving What We Can members talking about the pledge with friends suggest that this results in unusually high marginal value in terms of getting new pledges. In particular, there appears to be a conversion rate of roughly 10% (1 in 10 people you talk to go on to take the pledge within a month).
This has naturally led to some skepticism of such extraordinary claims. Which is very understandable.
Thus, in a comment earlier, I suggest doing a pre-registered outreach trial to rectify this.
Here is the procedure:
-
We gather 5-101 people who:
-
Have never directly talked to non-GWWC members about GWWC in the last 6 months
-
Are willing to experiment with talking about Giving What We Can, and have spare time
-
Are themselves Giving What We Can members or Try Givers
-
-
The experimentees2 are known to each other.
-
Each experimentee is instructed to contact3 a minimum of 5 friends and a maximum of 204 who have not taken the pledge (but the experimentee thinks should take the pledge) within five days after the experiment starts.
-
Experimentees are free to use any of the GWWC/EA resources on convincing people, including talking to people very familiar with GWWC like myself.5 The one caveat is that they cannot ask CEA staff or core volunteers on who to contact. Also CEA should endeavor not to contact the same people within the experimental time period, though of course accidents might happen (and should be recorded).
-
Experimentees take notes6 on how many people they’ve messaged, and how many people responded to the initial conversation, reactions, etc., and (arguably most importantly) how much time it has taken them.
-
20 days after the experiment began, we look at how many of the people contacted actually went on to take the pledge.
-
The results of the study will be written up in the EAForum.
-
If anybody’s interested, we can do a follow-up on the same people 1 year later to see how much they have donated.
Prediction: My 90% credible interval7 is that this experiment should have a 0-20% success rate8 (median 7%).
This experiment does not have a control group. I believe this is okay because the base rate of random individuals (including EAs) taking the pledge within any 15-day span should be fairly low.
If you’re interested in participating in the experiment, and fit condition 1), PLEASE contact me ASAP at email.linch[at]gmail[dot]com. Thanks so much!
____
1 The experiment will not start until we have at least five people. We’ll try to wait a bit to get 10 people.
2 Peter Hurford has volunteered to be one such experimentee.
3 Experimentees can use any format: FB PM, text, phone, email or face-to-face. The object to generalize is whether “people who have not previously talked about GWWC can, using the best of their intellect and understanding of personal circumstances, get others to take the pledge in the time allotted,” not “Does FB messenger work better or worse than irl conversations?”
4 Again, the exact number is up to individual discretion. Some people may only have 6-7 friends who are likely GWWC material, others may have over 15 who just need a slight push. Allowing experimentees this freedom is reasonable given the information we’re trying to generalize.
5 This is reasonable, again, because the same resources will be available to people outside of the experiment.
6 Experimentees should at least try to coordinate on not messaging the same people. A few doubles is okay however as long as the final analysis doesn’t double-count impact.
7 This is somewhat contingent on how many people end up being contacted.I reserve the right to make another,hopefully more accurate, prediction right before the actual experiments starts.
8 Success rate is the % of people who wind up taking the pledge within 20 days of the experiment starting.
This sounds really great to me. I love the idea of having more RCTs in the EA sphere. I would definitely record how much they are giving 1 year later.
I also think it’s worth having a hold out set. People can pre-register the list of friends, than a random number generator can be used to randomly selects some friends not to make an explicit GWWC pitch to. It’s possible many of the friends/contacts who join GWWC and start donating are those who have already been exposed to EA ideas before over a long period of time, and the effect size of the direct GWWC pitch isn’t as large as it would appear. Having a hold out set would account for this. With a hold out set, CEA wouldn’t have to worry about who they contact. The holdout set would take care of this and make the estimate of the treatment effect unbiased.
I don’t think there will be enough datapoints to do this. But if there are enough people willing to be in this experiment and they think they have a lot of friends they’ll be willing to contact, I will include a holdout set.
Sounds great Linch. My only thought is… if this is supposed to inform the usual rank-and-file GWWC members about whether and how to approach talking to others, you should try to get a fairly normal distribution of GWWC members to be the experimentees. My guess is that Peter Hurford may well be more convincing to his friends than the average GWWC member.
I don’t know about that—my friends are not very EA inclined and I’m not really any less awkward about talking about GWWC than anyone else I know.
I guess we’ll find out :)
Not if Linch can’t get more volunteers!
I have three so far, including yourself!
I’m not sure we can find out even after the experiment. Since the base rate is at best 10-20%, I would expect the differences in random variance to swamp out person-to-person differences in skill, credibility etc.
UPDATE: I now have my needed number of volunteers, and intend to launch the experiment tomorrow evening. Please email, PM, or otherwise contact me in the next 12 hours if you’re interested in participating.