I am a Research Scientist at the Humane and Sustainable Food Lab at Stanford and a nonresident fellow at the Kahneman-Treisman Center at Princeton. By trade, I am a meta-analyst.
Here is my date-me doc.
Seth Ariel Green đ¸
Karma: 1,148
Hi Wayne,
Great questions, Iâll try to give them the thoughtful treatment they deserve.
We donât place much (any?) credence in the statistical significance of the overall result, and I recognize that a lot of work is being done by the word âmeaningfullyâ in âmeaningfully reducing.â For us, changes on the order of a few percentage pointsâespecially given relatively small samples & vast heterogeneity of designs and contexts (hence our point about âwell-validatedââalmost nothing is directly replicated out of sample in our database) -- are not the kinds of transformational change that others in this literature have touted. Another way to slice this, if you were looking to evaluate results based on significance, is to look at how many results are, according to their own papers, statistical nulls: 95 out of 112, or about 85%. (On the other hand, many of these studies might be finding small but real effects but not be sufficiently powered to identify them: If you plan for d > 0.4, an effect of d = 0.04 is going to look like a null, even if real changes are happening). So my basic conclusion is that marginal changes probably are possible, so in that sense, yes, many of these interventions probably âwork,â but I wouldnât call the changes transformative. I think the proliferation of GLP-1 drugs is much more likely to be transformative.
Itâs true that cost-effectiveness estimates might still be very good even if the results are small. If there was a way to scale up the Jalil et al. intervention, Iâd probably recommend it right away. But I donât know of any such opportunity. (It requires getting professors to substitute out a normal economics lecture for one focused on meat consumption, and weâd probably want at least a few other schools to do measurement to validate the effect, and my impression from talking to the authors is that measurement was a huge lift). I also think that choice architecture approaches are promising and awaiting a new era of evaluation. My lab is working on some of these; for someone interested in supporting the evaluation side of things, donating to the lab might be a good fit.
This is in the supplement rather than the paper, but one of our depressing results is that rigorous evaluations published by nonprofits, such as The Humane League, Mercy For Animals, and Faunalytics, produce a small backlash on average (see table below). But itâs also my impression that a lot of these groups have changed gears a lot, and are now focusing less on (e.g.) leafletting and direct persuasion efforts and more on corporate campaigns, undercover investigations, and policy work. I donât know if they have moved this direction specifically because a lot of their prior work was showing null/âbacklash results, but in general I think this shift is a good idea given the current research landscape.
4. Pursuant to that, economists working on this sometimes talk about the consumer-citizen gap, where people will support policies that ban practices whose products theyâll happily consume. (People are weird!) For my money, if I were a significant EA donor on this space, I might focus here: message testing ballot initiatives, preparing for lengthy legal battles, etc. But as always with these things, the details matter. If you ban factory farms in California and lead Californians to source more of their meat from (e.g.) Brazil, and therefore cause more of the rainforest to be clearcutâwell thatâs not obviously good either.
5. Almost all interventions in our database targeted meat rather than other animal products (one looked at fish sauce and a couple also measured consumption of eggs and dairy). Also a lot of studies just say the choice was between a meat dish and a vegetarian dish, and whether that vegetarian dish contained eggs or milk is sometimes omitted. But in general, Iâd think of these as âless meatâ interventions.
Sorry I canât offer anything more definitive here about what works and where people should donate. An economist I like says his dadâs first rule of social science research was: âSometimes itâs this way, and sometimes itâs that way,â and I suppose I hew to that đ
đ Great questions!
Most studies in our dataset donât report these kinds of fine-grained results, but in general my impression from the texts is that the typical study gets a lot of people to change their behavior a little. (In part because if they got people to go vegan I expect they would say that.)
Some studies deliberately exclude vegetarians as part of their recruitment process, but most just draw from whatever population at large. Somewhere between 2 and 5% of people identify as vegetarians (and many of them eat meat sometimes), so I donât personally worry too much about this curtailing results. A few studies specifically recruit people who are motivated to change their diets and/âor help animals, e.g. Cooney (2016) recruited people who wanted to help Mercy for Animals evaluate its materials.
I think this is a fair mental model, but I think one of the main open questions of our paper is about how do we recruit people to cut back on meat in general vs. just cutting back on a few categories, e.g. red and processed meat. So I guess my mental model is that most people have heard that raising cows is bad for the environment and those who are cutting back are substituting partly to plant-based substitutes (reps from Impossible Foods noted at a recent meeting that most of their customers also eat meat) and partly to chicken and fish, e.g. the Mayo Clinicâs page on heart-healthy diets suggests âLean meat, poultry and fish; low-fat or fat-free dairy products; and eggs are some of the best sources of protein...Fish is healthier than high-fat meatsâ, although it also says that âEating plant protein instead of animal protein lowers the amounts of fat and cholesterol you take in.â
So Iâd say we still have a lot of open questions...
đ Our pleasure!
To the best of my recollection, the only paper in our dataset that provides a cost-benefit estimation is Jalil et al. (2023)
Calculations indicate a high return on investment even under conservative assumptions (~US$14 per metric ton CO2eq). Our findings show that informational interventions can be cost effective and generate long-lasting shifts towards more sustainable food options.
Thereâs also a red/âprocessed meat studyâEmmons et al. (2005) --- that does some cost-effectiveness analyses, but itâs almost 20 years old and its reporting is really sparse: changes to the eating environment âwere not reported in detail, precluding more detailed analyses of this intervention.â So Iâd stick with Jalil et al. to get a sense of ballpark estimates.
Agreed that itâs hard to implement: much easier to say âvegetarian food is popular at this cafe!Ⲡthan to convince people that they are expected to eat vegetarian.
See here for a review of the âdynamic normsâ part of this literature (studies that tell people that vegetarianism is growing in popularity over time): https://ââosf.io/ââpreprints/ââpsyarxiv/ââqfn6y
Thank you for your kind words!
putting SMDs into sensible terms is a continual struggle. I donât think itâll be easy to put vegetarians and meat eaters on a common scale because if vegetarians are all clustered around zero meat consumption, then the distance between vegs and meat eaters is just entirely telling you how much meat the meat eater group eats, and that changes a lot between populations.
Also, different disciplines have different ideas about what a âbigâ effect size is. Andrew Gelman writes something I like about this:
the first problem I noticed with that meta-analysis was an estimated average effect size of 0.45 standard deviations. Thatâs an absolutely huge effect, and, yes, there could be some nudges that have such a large effect, but thereâs no way the average of hundreds would be that large. Itâs easy, though, to get such a large estimate by just averaging hundreds of estimates that are subject to massive selection bias. So itâs no surprise that they got an estimate of 0.45, but we shouldnât take this as an estimate of treatment effects.
But by convention, an SMD of 0.5 is typically just considered a âmediumâ effect. I tend to agree with Gelman that changing peopleâs behavior by half a standard deviation on average is huge.
A different approach: here are a few studies, their main findings in normal terms, the SMD that translates to, and whether subjectively thatâs considered big or small
Jalil et al. (2023) | 5.6% reduction in meat eaten |SMD = 0.11 | small (but well-measured and has lasting effects)
Camp * Lawrence (2019) |self-reported decrease on meat items FFQ of .28 points | SMD = 0.4 | moderate/âlarge
Andersson & Nelander (2021 | 1% decrease in vegetarian meals sold | SMD = 0.16 | small (but, again, well measured)
So, for instance, the absolute change in the third study is a lot smaller than the absolute change in the first but has a bigger SMD because thereâs less variation in the dependent variable in that setting.
So anyway this is another hard problem. But in general, nothing here is equating to the kind of radical transformation that animal advocates might hope for.
MeanÂingfully reÂducÂing meat conÂsumpÂtion is an unÂsolved probÂlem: meta-analysis
I emailed this to my lab group â I got a lot out of last summerâs conference đ
Thatâs interesting, Iâm not sure what accounts for the differences (this is not my research area). If anything I would expect demand for the booster to be more price sensitive than for the initial dose.
ďNew reÂsearch on inÂcenÂtives to vaccinate
Hi Victoria, thanks for asking!
The stats classes I took in grad school typically had problem sets in R, so I learned that. I got better at it in summer 2016 when I used it for the paper I worked on. The first real job I got in tech was doing technical support for academic researchers who were using a computational reproducibility platform, so knowing a bit of R and being able to pick up enough of the other languages to get byâmostly some shell scripting and package installation commands in Python/âJulia/âetc. -- was helpful. Mostly I just learned the bits and pieces I needed to know and didnât really approach the question systematically.
The data analyst job I got was in an R shop. If I had been more motivated by the problem and a better fit at the company, I might still be doing that.
My cirÂcuitous, undiÂrected path to an EA job
Thanks, I think this was featured on Marginal Revolution last year â definitely good background.
Sure, there is an intuitive plausibility to this. But how extraordinary must the political dysfunction be for no one within Bangladesh to be capable of solving this themselves through political agitation without NGO support? If the DALY calculations are to be believed, the potential gains are enormous and comparatively cheap. As an outsider to the situation, I am looking for context on why this hasnât happened yet. A good theory in the social sciences will abstract from specifics of the situation, or map a theory onto specific things that have happened (or that havenât), rather than making the general observation that collective action problems cause some public goods to be underprovisioned.
I amended the text to be less inflammatory
Sure, there are more reasonable ways to express the argument, all of which boil down to âexperts have a comparative advantage at producing welfare gains.â And the people donât need to be poor for this theory to be in vogue, e.g. American public health officials giving very nuanced guidance about masks in March-April 2020 because they were looking to achieve a second-order effect (proper distribution). I think my broader point, however, about the necessity for such a theory, regardless of how it is expressed, holds. I went with a cynical version in part because I was trying to make a point that a theory can be a âfolk theoryâ and therefore impolite, but elucidating that was out of scope for the text.
Three use cases come to mind for the forum:
establishing a reputation in writing as a person who can follow good argumentative norms (perhaps as a kind of extended courtship of EA jobs/âorgs)
disseminating findings that are mainly meant for other forums, e.g. research reports
keeping track of what the community at large is thinking about/âworking on, which is mostly facilitated by organizations like RP & GiveWell using the forum to share their work.
I donât think I would use the forum for hashing out anything I was really thinking hard about; Iâd probably have in-person conversations or email particular persons.
Thanks for clarifying. That inevitably rests on a strong assumption about the relative importance of chicken welfare to human welfare, and it looks like your work builds on Bob Fischerâs estimates for conversion. Thatâs a fine starting point but for my tastes, this is a truly hard problem where the right answer is probably not knowable even in theory. When Iâm discussing this, Iâll probably stick to purely empirical claims, e.g., âwe can make X chickensâ lives better in Y waysâ or âwe can reduce meat consumption by Z poundsâ and be hand-wavy about the comparison between species. YMMV.
Thank you for the additional context!
re: Pure Earth: GiveWell notes that its Pure Earth estimates are âsubstantially less rigorous than both our top charity cost-effectiveness estimates,â so I donât want to read too much into it. However, a claim that an intervention is merely 18X better at helping poor people than they are at helping themselves still strikes me as extraordinary, albeit in a way that we become acclimated to over time.
As to what good social theory would look like here, there is some nice work in sociology on the causes and consequences of lead exposure in America (see Muller, Sampson, and Winter 2018 for a review). I donât expect EA orgs to produce this level of granularity when justifying their work, but some theory about why an opportunity exist would be very much appreciated, at least by me.
Iâve followed your work a bit w.r.t. animal welfare. Thatâs 15 chicken DALYs right? That seems plausible to me. The theory I would construct for this would start with the fact that there are probably more chickens living on factory farms at this moment than there are humans alive. Costco alone facilitates the slaughter of ~100M chickens/âyear. If you improve the welfare of just the Costco chickens by just 1% of a DALY per chicken, thatâs 1M DALYs. I could very much believe that a corporate campaign of that magnitude might cost about $66K (approximately 1M/â15). So I find this claim much less extraordinary.
As a potential title, maybe âDisability among farmed animals is neglected relative to human disabilityâ? or something like that
Hi there,
Delays run the gamut. Jalil et al (2023) measure three years worth of dining choices, Weingarten et al. a few weeks; other studies are measuring whatâs eaten at a dining hall during treatment and control but with no individual outcomes; and other studies are structured recall tasks like 3/â7/â30 days after treatment that ask people to say what they ate in a 24 hour period or over a given week. We did a bit of exploratory work on the relationship between length of delay and outcome size and didnât find anything interesting.
Iâm afraid we donât know that overall. A few studies did moderator analysis where they found that people who scored high on some scale or personality factor tended to reduce their MAP consumption more, but no moderator stood out to us as a solid predictor here. Some studies found that women seem more amenable to messaging interventions, based on the results of Piester et al. 2020 and a few others, but some studies that exclusively targeted women found very little. I think gendered differences are interesting here but we didnât find anything conclusive.