Thanks for reaching out and for including a nice note about our experiment.
A small typo you may want to fix: In our field study, no meals were sold because they were free at an event. We used this field study to establish ecological validity and then scaled the study by repeating the experiment design in an online experiment
A few more notes:
You have a subsection about how online studies have much smaller effect sizes than field studies. Is this the case in other behavioral sciences? I suggest you make a comparison.
Especially for when you write this for a journal, I suggest including motivation for the works you cover in the introduction. Why is transitioning away from MAP important?
Overall, I don’t find directly comparing the effect sizes of the various studies compelling because the studies differ across so many dimensions—region, intervention type, student vs non-student population, online vs field studies—and as you mention, publication bias is a concern.
I think what is needed are studies that compare different types of interventions within the same study population. For example, imagine a study where a random subset of the participants receive an intervention highlighting how a vegan meal has a lower price, and another random subset of participants receive an intervention highlighting how a vegan meal has a lower carbon impact.
Best of luck with the project! This work is important!
(The following is a lightly edited version of an email I sent to Alex earlier this weekend)
I just fixed that typo, TY [I actually fixed it in draft on Saturday and forgot to implement]
1. Comparing online to IRL studies—I will think about how to integrate, e.g. a study that finds similar results wr.t. the effects of intergroup contact on prejudice, but I’m not sure how much this generalizes across the behavioral sciences.
2. You’re right about motivations; for the EA forum and a preprint I think we can take for granted that people agree that we should collectively eat fewer animal products, and truth be told I’m not sure what kind of journal we’re going to aim for yet, so we left that kind of underspecified.
3. There are some studies that compare multiple strategies within one sample! See Feltz et al. (2022), Norris (2014) and Piester et al. (2022), though admittedly these are generally trying to test multiple implementations of one theoretical perspective, as opposed to your idea which puts the theoretical approaches head to head. I also think that’s promising. I am soon to put a research agenda on this subject together and I will think about how to incorporate that.
Thanks for engaging as always!
P.S. I went to a lovely vegan donut shop in Beacon this weekend and the person working there mentioned that they don’t always emphasize the vegan labels for certain customers because of the mixed connotations. Then again, a lot of the vegan places near me have vegan in their name—seasoned vegan, slutty vegan, and next stop vegan come to mind. This is probably a regional/NYC thing but still something I’ve been more on the look out for since reading your paper.
Thanks for reaching out and for including a nice note about our experiment.
A small typo you may want to fix: In our field study, no meals were sold because they were free at an event. We used this field study to establish ecological validity and then scaled the study by repeating the experiment design in an online experiment
A few more notes:
You have a subsection about how online studies have much smaller effect sizes than field studies. Is this the case in other behavioral sciences? I suggest you make a comparison.
Especially for when you write this for a journal, I suggest including motivation for the works you cover in the introduction. Why is transitioning away from MAP important?
Overall, I don’t find directly comparing the effect sizes of the various studies compelling because the studies differ across so many dimensions—region, intervention type, student vs non-student population, online vs field studies—and as you mention, publication bias is a concern.
I think what is needed are studies that compare different types of interventions within the same study population. For example, imagine a study where a random subset of the participants receive an intervention highlighting how a vegan meal has a lower price, and another random subset of participants receive an intervention highlighting how a vegan meal has a lower carbon impact.
Best of luck with the project! This work is important!
Alex
Thank you for engaging Alex!
(The following is a lightly edited version of an email I sent to Alex earlier this weekend)
I just fixed that typo, TY [I actually fixed it in draft on Saturday and forgot to implement]
1. Comparing online to IRL studies—I will think about how to integrate, e.g. a study that finds similar results wr.t. the effects of intergroup contact on prejudice, but I’m not sure how much this generalizes across the behavioral sciences.
2. You’re right about motivations; for the EA forum and a preprint I think we can take for granted that people agree that we should collectively eat fewer animal products, and truth be told I’m not sure what kind of journal we’re going to aim for yet, so we left that kind of underspecified.
3. There are some studies that compare multiple strategies within one sample! See Feltz et al. (2022), Norris (2014) and Piester et al. (2022), though admittedly these are generally trying to test multiple implementations of one theoretical perspective, as opposed to your idea which puts the theoretical approaches head to head. I also think that’s promising. I am soon to put a research agenda on this subject together and I will think about how to incorporate that.
Thanks for engaging as always!
P.S. I went to a lovely vegan donut shop in Beacon this weekend and the person working there mentioned that they don’t always emphasize the vegan labels for certain customers because of the mixed connotations. Then again, a lot of the vegan places near me have vegan in their name—seasoned vegan, slutty vegan, and next stop vegan come to mind. This is probably a regional/NYC thing but still something I’ve been more on the look out for since reading your paper.