TLYCS Pamphlet Pilot Results
TLYCS has wrapped up the grass-roots, university focused pamphleting pilot that we posted about earlier this year. The entire writeup can be viewed here, but the major results (taken from the introduction) are given below. I’m happy to answer any questions in the comments, and if anyone wants to view the full data set we collected, I can talk with the folks at TLYCS about getting it to you. Thanks again for the helpful comments and critique you all provided at the outset of this pilot, and hopefully these results will be useful to someone down the line.
The Life You Can Save (TLYCS) ran a test-pilot for a grass roots pamphleting program during Spring of 2015. On five separate outings during May and June, roughly 3500 pamphlets were handed out to students on Los Angeles area university campuses. The metrics of interest for the pilot were the number of visitors to the TLYCS website generated by the pamphlets, and the associated cost per visitor.
In post-pilot analysis, the estimated acquisition rate of website visitors per-pamphlet for these outings was between 0.6 % and 1.1 %. This translates to a $125 to $70 cost per-website visitor at the $0.75 price-per-unit TLYCS paid for the pamphlets. However, this cost could be straightforwardly reduced to between $12 to $6 per-website visitor by more economically sourcing the pamphlets.
There is reason to believe that a more focused, strategic pamphlet design, along with offering an incentive in the pamphlet, could have driven up the acquisition rate. However, even a doubling of the rate would not have produced a per-visitor cost on par with TLYCS’s online advertising efforts. Therefore a decision was made in the wake of the pilot to discontinue the program.
FYI, the previous posts on this topic are:
http://effective-altruism.com/ea/dg/tlycs_pamphleting_pilot_program/
http://effective-altruism.com/ea/eh/tlycs_pamphleting_pilot_plan/
Thanks for writing this up, very interesting!
The cost presumably doesn’t include the volunteer time to hand out the pamphlets? I would guess the time cost associated with pamphleting is much more than the time cost associated with online advertising, and that the difference there is even more significant than the pure monetary difference. (Particularly given that it sounds like there was quite a lot of coordination effort, as well as the actual handing out leaflets.)
On the other hand, if someone takes the trouble to go on the website from having been handed a leaflet, they seem plausibly much more likely to take further action than if they just clicked on a link online. Looking into the donations to top charities, as Zach suggests, could be indicative here (though I guess it would be very hard!).
Yeah, one of the main “unmeasurables” of the pilot was how much more of an impression a pamphlet, handed out by a real volunteer, can have on someone over an online add. This goes into the “touchpoint” theory, where it may take multiple exposures to an idea before someone bites; and maybe getting a pamphlet is a much stronger “touchpoint” than seeing an online add. I personally think that getting handed a pamphlet by a non-paid volunteer is a really powerful thing; and that’s the reason that (again personally) I think that it would be great to see someone in the EA movement give a pamphleting program another shot, along the lines I sketched in the lessons learned (branded on EA, including multiple organizations, with a stronger, repeated ask).
We ran cost numbers including volunteer time and pamphlet development expenses, but I decided to ignore those for the final writeup, mainly because; the way we envisioned the program, the leaflets would be handed out on a volunteer basis by college students not paid TLYCS staff; and the cost of developing the pamphlets would amortize to a minimal effect over the lifetime of an ongoing pamphleting program. It’s definitely a debatable decision though.
Any data on the effect of visiting the website on donations to highly-effective charities? I’m wondering what the return was. Also curious about the return to website ads to get a sense of the overall effectiveness of such outreach measures.
The difficulty of actually getting people to hand out pamphlets is interesting and, I think, an underrated reason why it’s a difficult way to spread a cause.
Unfortunately, we could only track website visitors as far as clicking a “donate” button on TLYCS “Where To Donate” page (http://www.thelifeyoucansave.org/Where-to-Donate). After clicking the button they are directed to the individual charity website, so we don’t have visibility into if they actually followed through with making a donation, or how much that donation was.
That being said, we only had one visitor that plausibly came from the pamphlets that clicked a donate button; if we had paid closer to Vegan Outreach prices for the pamphlets ($0.07), the cost of the 3500 pamphlets would have been about $245, and so the donation would have had to be on that level to get parity. Would definitely have been an interesting data point to see if / how much our clicker actually donated.
Does LYCS have any estimates which speak to that, even tangentially, for instance for it’s general impact evaluations?
They must have some visibility into how many people are donating via their website, because they release yearly estimates for money moved. I’m not sure exactly how they go about doing this; it can’t be via TLYCS website analytics though, so maybe they work with the charities themselves to track donations originating from TYLCS. I only partnered up with them for this study, so unfortunately I don’t have any details.
TLYCS’s (very) rough estimate is that on average each “donate click” equates to ~$100 in donations. However, that’s on overall average for our site, so my guess for a new visitor, currently enrolled in college, would be significantly lower.
As a reminder, TLYCS also expects to generate a multiplier on its expenditures, so the true breakeven cost would have to be <33% or so of the expected donations.
Sorry to take so long to respond to this!