Iâm a bit confused about your framing of the takeaway. You state that âreducing meat consumption is an unsolved problemâ and that âwe conclude that no theoretical approach, delivery mechanism, or persuasive message should be considered a well-validated means of reducing meat and animal product consumption.â However, the overall pooled effects across the 41 studies show statistical significance w/â a p-value of <1%. Yes, the effect size is small (0.07 SMD) but shouldnât we conclude from the significance that these interventions do indeed work?
Having a small effect or even a statistically insignificant one isnât something EAs necessarily care about (e.g. most longtermism interventions donât have much of an evidence base). Itâs whether we can have an expected positive effect thatâs sufficiently cheap to achieve. In Arielâs comment, you point to a study that concludes its interventions are highly cost-effective at ~$14/âton of CO2eq averted. Thatâs incredible given many offsets cost ~$100/âton or more. So it doesnât matter if the effect is âsmallâ, only that itâs cost-effective.
Can you help EA donors take the necessary next step? It wonât be straightforward and will require additional cost and impact assumptions, but itâll be super useful if you can estimate the expected cost-effectiveness of different diet-change interventions (in terms of suffering alleviated).
Finally, in addition to separating out red meat from all animal product interventions, I suspect itâll be just as useful to separate out vegetarian from vegan interventions. It should be much more difficult to achieve persistent effects when youâre asking for a lot more sacrifice. Perhaps we can get additional insights by making this distinction?
Great questions, Iâll try to give them the thoughtful treatment they deserve.
We donât place much (any?) credence in the statistical significance of the overall result, and I recognize that a lot of work is being done by the word âmeaningfullyâ in âmeaningfully reducing.â For us, changes on the order of a few percentage pointsâespecially given relatively small samples & vast heterogeneity of designs and contexts (hence our point about âwell-validatedââalmost nothing is directly replicated out of sample in our database) -- are not the kinds of transformational change that others in this literature have touted. Another way to slice this, if you were looking to evaluate results based on significance, is to look at how many results are, according to their own papers statistical nulls: 95 out of 112, or about 85%. (On the other hand, may of these studies might be finding small but real effects but not just be sufficiently powered to identify them: If you expect d > 0.4 because you read past optimistic reviews, an effect of d = 0.04 is going to look like a null, even if real changes are happening). So my basic conclusion is that marginal changes probably are possible, so in that sense, yes, many of these interventions probably âwork,â but I wouldnât call the changes transformative. I think the proliferation of GLP-1 drugs is much more likely to be transformative.
Itâs true that cost-effectiveness estimates might still be very good even if the results are small. If there was a way to scale up the Jalil et al. intervention, Iâd probably recommend it right away. But I donât know of any such opportunity. (It requires getting professors to substitute out a normal economics lecture for one focused on meat consumption, and weâd probably want at least a few other schools to do measurement to validate the effect, and my impression from talking to the authors is that measurement was a huge lift). I also think that choice architecture approaches are promising and awaiting a new era of evaluation. My lab is working on some of these; for someone interested in supporting the evaluation side of things, donating to the lab might be a good fit.
This is in the supplement rather than the paper, but one of our depressing results is that rigorous evaluations published by nonprofits, such as The Humane League, Mercy For Animals, and Faunalytics, produce a small backlash on average (see table below). But itâs also my impression that a lot of these groups have changed gears a lot, and are now focusing less on (e.g.) leafletting and direct persuasion efforts and more on corporate campaigns, undercover investigations, and policy work. I donât know if they have moved this direction specifically because a lot of their prior work was showing null/âbacklash results, but in general I think this shift is a good idea given the current research landscape.
4. Pursuant to that, economists working on this sometimes talk about the consumer-citizen gap, where people will support policies that ban practices whose products theyâll happily consume. (People are weird!) For my money, if I were a significant EA donor on this space, I might focus here: message testing ballot initiatives, preparing for lengthy legal battles, etc. But as always with these things, the details matter. If you ban factory farms in California and lead Californians to source more of their meat from (e.g.) Brazil, and therefore cause more of the rainforest to be clearcutâwell thatâs not obviously good either.
5. Almost all interventions in our database targeted meat rather than other animal products (one looked at fish sauce and a couple also measured consumption of eggs and dairy). Also a lot of studies just say the choice was between a meat dish and a vegetarian dish, and whether that vegetarian dish contained eggs or milk is sometimes omitted. But in general, Iâd think of these as âless meatâ interventions.
Sorry I canât offer anything more definitive here about what works and where people should donate. An economist I like says his dadâs first rule of social science research was: âSometimes itâs this way, and sometimes itâs that way,â and I suppose I hew to that đ
Thanks so much for this very helpful post!
Iâm a bit confused about your framing of the takeaway. You state that âreducing meat consumption is an unsolved problemâ and that âwe conclude that no theoretical approach, delivery mechanism, or persuasive message should be considered a well-validated means of reducing meat and animal product consumption.â However, the overall pooled effects across the 41 studies show statistical significance w/â a p-value of <1%. Yes, the effect size is small (0.07 SMD) but shouldnât we conclude from the significance that these interventions do indeed work?
Having a small effect or even a statistically insignificant one isnât something EAs necessarily care about (e.g. most longtermism interventions donât have much of an evidence base). Itâs whether we can have an expected positive effect thatâs sufficiently cheap to achieve. In Arielâs comment, you point to a study that concludes its interventions are highly cost-effective at ~$14/âton of CO2eq averted. Thatâs incredible given many offsets cost ~$100/âton or more. So it doesnât matter if the effect is âsmallâ, only that itâs cost-effective.
Can you help EA donors take the necessary next step? It wonât be straightforward and will require additional cost and impact assumptions, but itâll be super useful if you can estimate the expected cost-effectiveness of different diet-change interventions (in terms of suffering alleviated).
Finally, in addition to separating out red meat from all animal product interventions, I suspect itâll be just as useful to separate out vegetarian from vegan interventions. It should be much more difficult to achieve persistent effects when youâre asking for a lot more sacrifice. Perhaps we can get additional insights by making this distinction?
Hi Wayne,
Great questions, Iâll try to give them the thoughtful treatment they deserve.
We donât place much (any?) credence in the statistical significance of the overall result, and I recognize that a lot of work is being done by the word âmeaningfullyâ in âmeaningfully reducing.â For us, changes on the order of a few percentage pointsâespecially given relatively small samples & vast heterogeneity of designs and contexts (hence our point about âwell-validatedââalmost nothing is directly replicated out of sample in our database) -- are not the kinds of transformational change that others in this literature have touted. Another way to slice this, if you were looking to evaluate results based on significance, is to look at how many results are, according to their own papers statistical nulls: 95 out of 112, or about 85%. (On the other hand, may of these studies might be finding small but real effects but not just be sufficiently powered to identify them: If you expect d > 0.4 because you read past optimistic reviews, an effect of d = 0.04 is going to look like a null, even if real changes are happening). So my basic conclusion is that marginal changes probably are possible, so in that sense, yes, many of these interventions probably âwork,â but I wouldnât call the changes transformative. I think the proliferation of GLP-1 drugs is much more likely to be transformative.
Itâs true that cost-effectiveness estimates might still be very good even if the results are small. If there was a way to scale up the Jalil et al. intervention, Iâd probably recommend it right away. But I donât know of any such opportunity. (It requires getting professors to substitute out a normal economics lecture for one focused on meat consumption, and weâd probably want at least a few other schools to do measurement to validate the effect, and my impression from talking to the authors is that measurement was a huge lift). I also think that choice architecture approaches are promising and awaiting a new era of evaluation. My lab is working on some of these; for someone interested in supporting the evaluation side of things, donating to the lab might be a good fit.
This is in the supplement rather than the paper, but one of our depressing results is that rigorous evaluations published by nonprofits, such as The Humane League, Mercy For Animals, and Faunalytics, produce a small backlash on average (see table below). But itâs also my impression that a lot of these groups have changed gears a lot, and are now focusing less on (e.g.) leafletting and direct persuasion efforts and more on corporate campaigns, undercover investigations, and policy work. I donât know if they have moved this direction specifically because a lot of their prior work was showing null/âbacklash results, but in general I think this shift is a good idea given the current research landscape.
4. Pursuant to that, economists working on this sometimes talk about the consumer-citizen gap, where people will support policies that ban practices whose products theyâll happily consume. (People are weird!) For my money, if I were a significant EA donor on this space, I might focus here: message testing ballot initiatives, preparing for lengthy legal battles, etc. But as always with these things, the details matter. If you ban factory farms in California and lead Californians to source more of their meat from (e.g.) Brazil, and therefore cause more of the rainforest to be clearcutâwell thatâs not obviously good either.
5. Almost all interventions in our database targeted meat rather than other animal products (one looked at fish sauce and a couple also measured consumption of eggs and dairy). Also a lot of studies just say the choice was between a meat dish and a vegetarian dish, and whether that vegetarian dish contained eggs or milk is sometimes omitted. But in general, Iâd think of these as âless meatâ interventions.
Sorry I canât offer anything more definitive here about what works and where people should donate. An economist I like says his dadâs first rule of social science research was: âSometimes itâs this way, and sometimes itâs that way,â and I suppose I hew to that đ