Proposed methodology for leafleting study

After some of the re­cent con­tro­versy about the state of the ev­i­dence sup­port­ing ACE’s recom­men­da­tions [1] I started think­ing about how best we could study leaflet­ing, and I think there is a strong op­por­tu­nity for some­one to do a much higher power study than has been man­aged be­fore, with rel­a­tively low effort.

Dis­claimer: All of the views/​plans pre­sented therein are my own, and not offi­cially af­fili­ated with or en­dorsed by Mercy For An­i­mals.

1. Background

1.1 A brief overview of pre­vi­ous research

A good re­view of the pre­vi­ous stud­ies on leaflet­ting effec­tive­ness can be found here: http://​​veg­​​les-fall-2016/​​

The pri­mary take­away is that most of the stud­ies con­ducted so far have not been con­trol­led mak­ing it im­pos­si­ble to in­fer the effect of leaflet­ing ver­sus a gen­eral trend to­wards veg­e­tar­i­anism. The stud­ies which have used con­trols have also always had ex­tremely small con­trol groups, the largest be­ing 57, mean­ing none of them had the power to mea­sure any statis­ti­cally sig­nifi­cant re­sults.

This has left leaflet­ing effec­tive­ness es­ti­mates al­most en­tirely dom­i­nated by per­sonal judge­ments, de­spite the fact it is a sig­nifi­cant tool used by effec­tive an­i­mal char­i­ties. [2]

1.2 A sum­mary of my plan

At a large uni­ver­sity (pseudo)-ran­domly split the stu­dents evenly into two groups, and put leaflets into the pi­geon­holes of all mem­bers of one of the groups, and none of the other. Then send a fol­low up sur­vey (pos­si­bly in­cen­tivised) to all stu­dents, a week later, ask­ing:

  1. A ques­tion to de­ter­mine if they were leafleted or not, with­out di­rectly ask­ing.

  2. If they have changed their diet in the last two weeks, and if so, how.

This would provide a con­trol­led trial with a sam­ple size of many thou­sands if the sur­vey re­sponse rate was high enough (see 3.4), enough to find an effect size of ap­prox­i­mately 1/​75-1/​250 leaflets cre­at­ing one veg­e­tar­ian (de­pend­ing on sam­ple size, as it ranges from 2,500 to 10,000, see 3.5).

1.3 My aim for this post

After con­tact­ing my uni­ver­sity it turns out run­ning such a study in Cam­bridge would not be pos­si­ble (as the Stu­dent union/​fac­ulty will not email out a sur­vey, and due to other con­sid­er­a­tions, see 3.1). How­ever I think such a study could still be ex­tremely valuable if con­ducted el­se­where.

By lay­ing out the plan in de­tail I hope to both get feed­back on ar­eas which could be im­proved, and hope­fully find a uni­ver­sity at which it could be im­ple­mented.

2. De­tailed plan

2.1 Preparations

Deter­mine which stu­dents you will leaflet, ei­ther us­ing some kind of ran­domi­sa­tion or any nat­u­ral di­vi­sions that ex­ist (see 3.1).

Con­tact the stu­dent union/​uni­ver­sity it­self and ask if they would email the sur­vey out to the en­tire stu­dent pop­u­la­tion. Get­ting the sur­vey to reach ev­ery­one is very im­por­tant, so it would be worth work­ing hard on this, meet­ing them in per­son if nec­es­sary etc. Also write some ar­ti­cles about the sur­vey to fea­ture in on­line stu­dent news­pa­pers (the TAB etc.) and dis­cuss get­ting them pub­lished in ad­vance. For both of these play­ing up the fact it is a po­ten­tially large and im­por­tant study be­ing con­ducted by uni­ver­sity stu­dents would prob­a­bly be helpful. In­cen­tivis­ing the sur­vey re­sponses with a fairly large amount (e.g. £5 each) might also be most use­ful at this stage, as even if it does not boost re­sponse rate very much (see 3.5) get­ting the sur­vey emailed out to ev­ery­one is very im­por­tant.

Work with Statis­ti­ci­ans Without Borders (or similar ex­perts) to check the statis­tics be­hind the study all work out, and pre-reg­ister the study, in­clud­ing which effects we are will be look­ing for (such as a higher than base-rate num­ber of peo­ple who were leafleted turn­ing veg­e­tar­ian).

Ob­tain as many leaflets are you are ex­pect­ing to hand out (in the UK An­i­mal Equal­ity dis­tribute them to stu­dents to hand out for free).

Con­tact/​hire an on­line sur­vey com­pany to set up a sur­vey that is linked with the uni­ver­sity’s emails or similar, so that each stu­dent can only fill it out once. The sur­vey should also have the fol­low­ing ques­tions:

  1. What is your last name? Or similar ques­tion de­signed to de­ter­mine if they were leafleted or not (see 3.1).

  2. Have you changed your diet in the last two weeks?

    If they se­lect yes them be­ing pre­sented with two fur­ther ques­tions:

  3. Which la­bel best de­scribed your diet be­fore the change?

  • Meat Re­duc­tion Diet (A diet re­duc­ing meat con­sump­tion, for ex­am­ple Meatless Mon­days)

  • Pesc­etar­ian Diet (eat fish, egg, and milk prod­ucts, but no other meat (in­clud­ing chicken))

  • Vege­tar­ian Diet (eat egg and milk prod­ucts, but no meat (in­clud­ing fish or chicken))

  • Ve­gan Diet (eat no meat (in­clud­ing fish or chicken), milk prod­ucts, egg, or other an­i­mal prod­ucts)

  • No spe­cific diet (A diet with no spe­cific prefer­ences or ex­clu­sions)

  • Other __________________________

  1. Which la­bel best de­scribes your diet af­ter the change?
    With the same op­tions.

If the sur­vey was in­cen­tivised there would then be a tick­box for “I would like a £5 ama­zon voucher” and a sub­mit but­ton, which would take them to a ‘thank you’ page with a refer­ral link to send to their friends, and the in­for­ma­tion that they would get an­other £2 ama­zon voucher for each of their friends that used it and filled out the sur­vey.

2.2 Implementation

Once all the setup is com­pleted, find one-two days where 4 vol­un­teers/​work­ers are free, split into two groups and work through putting leaflets in the pi­geon­holes of all the stu­dents of all the col­leges you ar­ranged to. (For the prac­ti­cal­ities of this we have found filling two suit­cases with leaflets al­lowed two peo­ple to trans­port about 2,500 at once). We have been able to give out an av­er­age of 1000 leaflets an hour as part of a two per­son team when mass-leaflet­ing like this, so it might be pos­si­ble to do it in only one day, or two days with only two peo­ple.

One week later send out the sur­vey (open for one week) via all your available chan­nels, with a fol­low up re­minder 3 days later if pos­si­ble. If the re­sponse rate is too low pos­si­bly con­sider try­ing to boost it via ad­di­tional meth­ods such as face­book ad­ver­tis­ing. At the end of the week close the sur­vey.

2.3 Analysis

Although the un­der­ly­ing statis­tics in case case seem quite sim­ple (see 3.5), I would sug­gest let­ting statis­ti­ci­ans with­out bor­ders or other ex­perts do the anal­y­sis, ac­cord­ing to the pre-reg­istered method­ol­ogy. This would just in­volve look­ing for differ­ence be­tween the con­trol and leafleted group in the rate veg*n dietary change, such as in­creased num­ber of peo­ple re­duc­ing meat con­sump­tion. (Again see 3.5)

3.Ex­pla­na­tion of the Details

3.1 How to split the pop­u­la­tion into two groups

I was in­spired to come up with this af­ter re­al­is­ing that ap­prox­i­mately half of the Cam­bridge col­leges al­low mass-pi­geon­hol­ing all of their stu­dents (and all cam­bridge stu­dents have a pi­geon­hole), and the other half none, cre­at­ing a nat­u­ral di­vi­sion. How­ever there are ac­tu­ally sev­eral fac­tors that make Cam­bridge col­leges a non-ideal par­ti­tion for this sur­vey and whilst it turns out that the Cam­bridge Stu­dent union would not send out a sur­vey in any case, these con­sid­er­a­tions also ap­ply to any other col­le­giate uni­ver­sity that such a sur­vey might be run at:

i. Differ­ent col­leges have differ­ent cul­tures and in­sti­tu­tions, e.g. differ­ent cafe­te­rias, which may serve differ­ing qual­ities of veg­e­tar­ian/​ve­gan food, in­fluenc­ing the effec­tive­ness of leaflet­ing.

ii. De­mo­graphic con­founders, as differ­ent col­leges have differ­ent sub­ject and gen­der ra­tios, which may cor­re­late with leaflet re­sponse rates. I con­sider this a less im­por­tant con­sid­er­a­tion as it could be con­trol­led for with care­ful study de­sign. (Such as by ask­ing about gen­der and sub­ject in the sur­vey)

iii. An ad­di­tional point against Cam­bridge is that quite a lot of pro-veg­gie leaflet­ing has already been car­ried out, in­clud­ing mass-pi­geon­hol­ing of many col­leges in pre­vi­ous years.

iv. Any form of clus­tered ran­domi­sa­tion re­duces statis­ti­cal power, al­though I am not sure about the size of this effect (see 3.5)

How­ever more gen­er­ally I think the im­por­tant crite­ria for good ways of split­ting the stu­dent pop­u­la­tion into two groups are:

i. It be­ing pos­si­ble to se­lec­tively leaflet all of the stu­dents in one group and none in the other.

ii. Which group any stu­dent falls into be­ing eas­ily dis­cernible via a sim­ple sur­vey ques­tion which does not rely on the stu­dent re­mem­ber­ing be­ing leafleted/​could in­fluence the stu­dents’ later an­swers.

iii. Split­ting the stu­dent pop­u­la­tion into ap­prox­i­mately equal sized groups to max­imise statis­ti­cal power.

iv. The split­ting be­ing ran­dom, or at least not cor­re­lated with any­thing that should af­fect the re­sponse to leaflet­ing.

With these con­di­tions in mind I think the ideal method might be to find a uni­ver­sity where all stu­dents could in the­ory be leafleted, and then se­lec­tively only leaflet­ing those whose sur­name starts with a let­ter in the first half of the alpha­bet (e.g. a-m) or some similar sys­tem. A po­ten­tial is­sue with this could be that last name start­ing let­ter may in­fluence the base chance to go veg­e­tar­ian (as it could re­flect class or similar dis­tinc­tions), how­ever when split­ting the pop­u­la­tion into only 2 groups this seems un­likely to be an is­sue.

As names are writ­ten on all stu­dent pi­geon­holes I would think most ran­domi­sa­tion sys­tems would need to de­pend on them, how­ever there is a trade off as the more ran­dom a sys­tem the harder it would be to im­ple­ment when ac­tu­ally do­ing the leaflet­ing.

3.2 Sur­vey Questions

I think the im­por­tant con­sid­er­a­tions in choos­ing what ques­tions to put on the sur­vey would be:

i. Find­ing out some­thing which we can ac­tu­ally put a value on, such as peo­ple go­ing veg­e­tar­ian (as op­posed to say, chang­ing their views about meat, where the im­pact is much less clear).

ii. Min­imis­ing the chance of in­fluenc­ing the an­swer, which would be an is­sue if you were e.g. ask­ing them about if they had been leafleted, mak­ing them think back to it.

iii. Within a given effect, choos­ing the ques­tion that max­imises statis­ti­cal power.
iv. Keep­ing the sur­vey short, to max­imise re­sponse rate.

In 2.1 I reused one of the diet ques­tions from MFA 2013 study on leaflet­ing [3], cut­ting out the ir­relevent op­tions and and adapt­ing them to a much shorter (two weeks vs 3 months) timescale, which should greatly in­crease the statis­ti­cal power. This is how­ever at the cost of cap­tur­ing longer term effects, such as the leaflets not hav­ing a di­rect effect but mak­ing peo­ple more sus­cep­ti­ble to other veg*n out­reach, which is dis­cussed fur­ther in the next section

3.3 Prac­ti­cal sur­vey considerations

I think ac­tual leaflet­ing should be car­ried out as rapidly as pos­si­ble, ideally over a one or two day pe­riod, so that the amount of time peo­ple have to pick up and read the leaflet is as uniform as pos­si­ble be­fore the sur­vey goes out.

One week af­ter dis­tribut­ing the leaflets a sur­vey (open for one week) should be sent out to the en­tire stu­dent pop­u­la­tion, con­sist­ing of one ques­tion about any changes they may have had in diet over the last 2 weeks, and then an­other to de­ter­mine if they fall in the leaflet­ing group or the con­trol. I chose this as I think one week is enough for ev­ery­one to have checked their pi­geon­hole, and I think most peo­ple will then ei­ther read the leaflet im­me­di­ately or at least in the same day, or throw it away. Any diet changes then have a few days to kick in be­fore the sur­vey goes out.

I chose this short turn around to max­imise the power of the sur­vey, as the shorter the timescale the lower the base rate of peo­ple con­vert­ing to veg­e­tar­i­anism, and so the greater the statis­ti­cal power. I sus­pect this is a trade off be­tween power and look­ing at the long-term effects how­ever, and prob­a­bly cuts off some im­por­tant effects. It might be worth con­duct­ing a fol­low up sur­vey a few months later, to see both if the ini­tial effects last, which would be very use­ful in es­ti­mat­ing the value of leaflet­ing in its own right, and also to see if any ad­di­tional ones man­i­fest.

3.4 Incentivisation

It is pos­si­ble that in­cen­tivis­ing the sur­vey to boast re­sponse rate would be worth it, how­ever it would sig­nifi­cantly add to the cost of the sur­vey, and I am un­sure of the benefits. I have not been able to find any con­clu­sive re­search as to how it would in­fluence the ex­pected re­sponse rate, with some pa­pers even find­ing in­cen­tivized sur­veys got lower re­sponse rates.

Even if it does not boost re­sponse rate, it seems that in­cen­tivis­ing the sur­vey would in­crease the will­ing­ness by uni­ver­sity-wide groups such as the stu­dent union to share the sur­vey. This is very im­por­tant as if we can­not get the sur­vey emailed out to the en­tire stu­dent pop­u­la­tion evenly then this would not only re­duce the sam­ple size, but also mean the we may end up with a non-rep­re­sen­ta­tive sam­ple.

If the sur­vey was in­cen­tivised I would sug­gest offer­ing a re­ward of say a £5 ama­zon voucher to each stu­dent who fills out the sur­vey, plus a bonus £2 for ev­ery other stu­dents you re­fer who fills it out. The refer­ral bonus should hope­fully en­gen­der a strong shar­ing amongst friends af­fect, which should lead to a high re­sponse rate, al­though I have found no stud­ies on this, so it is only in­tu­ition.

A sur­vey com­pany may need to be brought in to cre­ate and set up the refer­ral and re­ward sys­tems, which could add to the costs. As part of this some sys­tem would need to be in place to stop peo­ple filling out the sur­vey mul­ti­ple times (such as re­quiring a unique uni­ver­sity email ad­dress for ev­ery­one filling out the sur­vey, or us­ing a uni­ver­sity au­then­ti­ca­tion sys­tem as dis­cussed in my plan for Cam­bridge).

3.5 Statis­ti­cal power estimations

The power es­ti­mates are quite strongly de­pen­dant on what the base rate of stu­dents go­ing veg­e­tar­ian/​re­duc­ing meat con­sump­tion are. I have looked at looked at ACEs 2013 study to get the pro­por­tion of stu­dents go­ing veg­e­tar­ian in a given two week pe­riod as 0.234% [4]

For all the fol­low­ing power calcu­la­tions I used: https://​​​​~rol­lin/​​stats/​​ssize/​​b2.html

Note that if us­ing uni­ver­sity col­leges or similar large groups as the way to split the stu­dents then the power would be re­duced due to clus­ter­ing [5], but I do not know how to es­ti­mate this effect. The fol­low­ing calcu­la­tions are thus for a sys­tem like the name-based ran­domi­sa­tion de­scribed at the end of 3.1

For sim­plic­ity this is look­ing only at peo­ple go­ing veg­e­tar­ian as a re­sult of read­ing the leaflet, al­though as a study like this would likely be in­ves­ti­gat­ing mul­ti­ple hy­pothe­ses such as the effects of the leaflets on veg­e­tar­i­anism and ve­g­anism sep­a­rately, I have made a bon­fer­roni cor­rec­tion [6] of ⅕ mean­ing the fol­low­ing are calcu­lated for alpha = 0.01

To get the fol­low­ing figures I plugged the base rate of 0.234% and sam­ples sizes into the above calcu­la­tor, found the small­est effect size in the treat­ment pop­u­la­tion that gave 90% power, and sub­tracted the base rate to get the de­tectable effects of the leaflet­ing. [7]

A sam­ple of 10,000 would give a 90% chance of find­ing an 1202 effect if it ex­isted (i.e would provide a 90% chance to find­ing out if one in ev­ery 202 leaflets turned a stu­dent veg­e­tar­ian)

A sam­ple of 5,000 would give a 90% chance of find­ing an 1124 effect if it existed

A sam­ple of 2,500 would give a 90% chance of find­ing an 173 effect if it existed

(All as­sum­ing the sam­ple was split evenly be­tween con­trol and treat­ment groups)

The av­er­age un­ver­sity size in the UK seems to be about 20,000 stu­dents mak­ing sur­vey sizes of 10,000, 5,000 and 2,500 rep­re­sent ap­prox­i­mately 50%, 25% and 12.5% re­sponse rates re­spec­tively.

All these figures are ex­ceed­ingly rough, and more could be calcu­lated to find the power to de­tect e.g. con­ver­sions to ve­g­anism or gen­eral meat re­duc­tion, but they serve to show that with a rel­a­tively low re­sponse rate (12.5%) we could at­tain sig­nifi­cant statis­ti­cal power, par­tic­u­larly more than enough to test some ex­ist­ing claims, such as the 150 or 175 figure leaflets to a veg­e­tar­ian figure [8]

4. Fi­nal Thoughts
4.1 Limitations

A few things to con­sider are that the re­sults of a study like this would not be ob­vi­ously an im­me­di­ately gen­er­al­iz­able to other cases. Leaflets in pi­geon­holes could have a very differ­ent effect than when handed out nor­mally. Stu­dent pop­u­la­tions are also prob­a­bly far more likely to re­spond pos­i­tively to pro-veg­e­tar­ian mes­sages, al­though as most of the out­reach cur­rently seems to tar­get them any­way this is not much of an is­sue.

There is also a sig­nifi­cant range of val­ues in which it would be effec­tively im­pos­si­ble to tell if leaflets had an effect, but they could still be con­sid­ered very effec­tive if they did. E.g. if 110,000 leaflets made some­one turn veg­e­tar­ian this would work out to ap­prox­i­mately £600 per veg­e­tar­ian cre­ated [9] which some might con­sider worth it (if they wanted to e.g. offset their own diet), but would also re­quire such a large enough sam­ple (~7 mil­lion peo­ple) [10] as to be effec­tively untestable.

As such I think a likely out­come of a study like this would be find­ing no statis­ti­cally sig­nifi­cant effect, but an­i­mal char­i­ties con­tin­u­ing with leaflet­ing re­gard­less, which might well be jus­tified.

4.2 Pre­dicted cost

If no in­cen­tive was used with the sur­vey, then the cost of the en­tire could be ex­tremely low, ba­si­cally just that of the leaflets, ap­prox­i­mately 4p each in the UK, and 20-30 per­son hours of do­ing set up and hand­ing out the leaflets. At 10,000 leaflets and £10 an hour this would put the en­tire cost of the study at £700 ($900) or less.

If the sug­gested £5 per re­sponse and £2 refer­rals in­cen­tives were sup­plied, de­spite be­ing a fairly large re­ward per per­son this would al­low for a 10,000 per­son (rep­re­sent­ing a 50% re­sponse rate for a typ­i­cal uni­ver­sity, if all the stu­dents could be emailed the sur­vey) study to be con­ducted for around £75,000 ($90,000), which would have or­ders of mag­ni­tude more power than any other study con­ducted to date.

4.3 Conclusion

De­spite some sub­tleties about ex­actly how to split up the groups and max­imise re­sponse the re­sponse rate to the sur­vey I think there is a strong op­por­tu­nity here, and that a study run us­ing this method­ol­ogy would be sig­nifi­cantly valuable.

The main is­sue seems to be the availa­bil­ity of uni­ver­si­ties with pi­geon­holes and co­op­er­a­tive fac­ul­ties/​stu­dent unions to email out the sur­vey. As such I would be in­ter­ested in hear­ing from stu­dents at any uni­ver­si­ties they think fit the crite­ria.

In the mean­time please give me feed­back on any ar­eas you think are flawed, or ways things could be im­proved.

[1] https://​​​​@har­ri­son­nathan/​​re-eval­u­at­ing-an­i­mal-char­ity-eval­u­a­tors-c164231406f7

[2] e.g Mercy For An­i­mals, one of ACE’s top recom­mended char­i­ties pays its fel­lows to do this.

[3] https://​​an­i­malchar­i­tye­val­u­a­​​re­search/​​in­ter­ven­tions/​​leaflet­ing/​​2013-leaflet­ing-study/​​#main

[4] As ACE pub­lished all the data from their study [3], I was able to look at the ‘con­trol’ re­sponses from peo­ple who did not re­ceive a leaflet, or who re­ceived one un­re­lated to re­duc­ing an­i­mal product con­sump­tion. Re­mov­ing all those who did not an­swer the rele­vant ques­tions, I was left with 356 peo­ple who filled in their pre­sent diet, as well as what it had been 3 month pre­vi­ously.

Count­ing the num­ber of veg­e­tar­i­ans who had re­ported be­ing some­thing other than veg­e­tar­ian 3 months pre­vi­ously (and ex­clud­ing one ve­gan who be­cause veg­e­tar­ian) I got a base rate of 5356 stu­dents be­com­ing veg­e­tar­ian in a 3 month pe­riod, or ⅙*5/​356 = 0.234% be­com­ing veg­e­tar­ian in a two week pe­riod.

[5] For an ex­pla­na­tion of clus­ter­ing see e.g. http://​​­knowl­​​pub­lic-health-text­book/​​re­search-meth­ods/​​1a-epi­demiol­ogy/​​clus­tered-data

[6] https://​​en.wikipe­​​wiki/​​Bon­fer­roni_cor­rec­tion There might well be bet­ter ways of do­ing this, I just went with some­thing I could eas­ily use.

[7] In de­tail for the n = 10,000 case:
I en­tered the figures into the calcu­la­tor as so:
statistics blank.png

I then en­tered val­ues for p2 un­til find­ing the low­est that gave a power of 90%, which turned out to be 0.0073:


This cor­re­sponds to the leaflets hav­ing an effect of p2 - p1 = 0.00496 = 1202.


[9] When mass-leaflet­ing col­leges as in this study as a team of two we were able to hand out 1000 leaflets an hour. I have been told the leaflets cost 4p each, and as­sum­ing pay­ing the leafleters £10 an hour this gives a to­tal cost of £600 (2*10*£10 = £200 to pay the leafleters, and $0.04*10,000 = £400 for the cost of the leaflets).

[10] ob­tained by set­ting p2 in the calcu­la­tor to 0.0021, 0.0001 (or 110,000) more than the base rate of p1 = 0.002, with a power of 90%.