Great questions, I’ll try to give them the thoughtful treatment they deserve.
We don’t place much (any?) credence in the statistical significance of the overall result, and I recognize that a lot of work is being done by the word “meaningfully” in “meaningfully reducing.” For us, changes on the order of a few percentage points—especially given relatively small samples & vast heterogeneity of designs and contexts (hence our point about “well-validated”—almost nothing is directly replicated out of sample in our database) -- are not the kinds of transformational change that others in this literature have touted. Another way to slice this, if you were looking to evaluate results based on significance, is to look at how many results are, according to their own papers statistical nulls: 95 out of 112, or about 85%. (On the other hand, may of these studies might be finding small but real effects but not just be sufficiently powered to identify them: If you expect d > 0.4 because you read past optimistic reviews, an effect of d = 0.04 is going to look like a null, even if real changes are happening). So my basic conclusion is that marginal changes probably are possible, so in that sense, yes, many of these interventions probably “work,” but I wouldn’t call the changes transformative. I think the proliferation of GLP-1 drugs is much more likely to be transformative.
It’s true that cost-effectiveness estimates might still be very good even if the results are small. If there was a way to scale up the Jalil et al. intervention, I’d probably recommend it right away. But I don’t know of any such opportunity. (It requires getting professors to substitute out a normal economics lecture for one focused on meat consumption, and we’d probably want at least a few other schools to do measurement to validate the effect, and my impression from talking to the authors is that measurement was a huge lift). I also think that choice architecture approaches are promising and awaiting a new era of evaluation. My lab is working on some of these; for someone interested in supporting the evaluation side of things, donating to the lab might be a good fit.
This is in the supplement rather than the paper, but one of our depressing results is that rigorous evaluations published by nonprofits, such as The Humane League, Mercy For Animals, and Faunalytics, produce a small backlash on average (see table below). But it’s also my impression that a lot of these groups have changed gears a lot, and are now focusing less on (e.g.) leafletting and direct persuasion efforts and more on corporate campaigns, undercover investigations, and policy work. I don’t know if they have moved this direction specifically because a lot of their prior work was showing null/backlash results, but in general I think this shift is a good idea given the current research landscape.
4. Pursuant to that, economists working on this sometimes talk about the consumer-citizen gap, where people will support policies that ban practices whose products they’ll happily consume. (People are weird!) For my money, if I were a significant EA donor on this space, I might focus here: message testing ballot initiatives, preparing for lengthy legal battles, etc. But as always with these things, the details matter. If you ban factory farms in California and lead Californians to source more of their meat from (e.g.) Brazil, and therefore cause more of the rainforest to be clearcut—well that’s not obviously good either.
5. Almost all interventions in our database targeted meat rather than other animal products (one looked at fish sauce and a couple also measured consumption of eggs and dairy). Also a lot of studies just say the choice was between a meat dish and a vegetarian dish, and whether that vegetarian dish contained eggs or milk is sometimes omitted. But in general, I’d think of these as “less meat” interventions.
Sorry I can’t offer anything more definitive here about what works and where people should donate. An economist I like says his dad’s first rule of social science research was: “Sometimes it’s this way, and sometimes it’s that way,” and I suppose I hew to that 😃
Hi Wayne,
Great questions, I’ll try to give them the thoughtful treatment they deserve.
We don’t place much (any?) credence in the statistical significance of the overall result, and I recognize that a lot of work is being done by the word “meaningfully” in “meaningfully reducing.” For us, changes on the order of a few percentage points—especially given relatively small samples & vast heterogeneity of designs and contexts (hence our point about “well-validated”—almost nothing is directly replicated out of sample in our database) -- are not the kinds of transformational change that others in this literature have touted. Another way to slice this, if you were looking to evaluate results based on significance, is to look at how many results are, according to their own papers statistical nulls: 95 out of 112, or about 85%. (On the other hand, may of these studies might be finding small but real effects but not just be sufficiently powered to identify them: If you expect d > 0.4 because you read past optimistic reviews, an effect of d = 0.04 is going to look like a null, even if real changes are happening). So my basic conclusion is that marginal changes probably are possible, so in that sense, yes, many of these interventions probably “work,” but I wouldn’t call the changes transformative. I think the proliferation of GLP-1 drugs is much more likely to be transformative.
It’s true that cost-effectiveness estimates might still be very good even if the results are small. If there was a way to scale up the Jalil et al. intervention, I’d probably recommend it right away. But I don’t know of any such opportunity. (It requires getting professors to substitute out a normal economics lecture for one focused on meat consumption, and we’d probably want at least a few other schools to do measurement to validate the effect, and my impression from talking to the authors is that measurement was a huge lift). I also think that choice architecture approaches are promising and awaiting a new era of evaluation. My lab is working on some of these; for someone interested in supporting the evaluation side of things, donating to the lab might be a good fit.
This is in the supplement rather than the paper, but one of our depressing results is that rigorous evaluations published by nonprofits, such as The Humane League, Mercy For Animals, and Faunalytics, produce a small backlash on average (see table below). But it’s also my impression that a lot of these groups have changed gears a lot, and are now focusing less on (e.g.) leafletting and direct persuasion efforts and more on corporate campaigns, undercover investigations, and policy work. I don’t know if they have moved this direction specifically because a lot of their prior work was showing null/backlash results, but in general I think this shift is a good idea given the current research landscape.
4. Pursuant to that, economists working on this sometimes talk about the consumer-citizen gap, where people will support policies that ban practices whose products they’ll happily consume. (People are weird!) For my money, if I were a significant EA donor on this space, I might focus here: message testing ballot initiatives, preparing for lengthy legal battles, etc. But as always with these things, the details matter. If you ban factory farms in California and lead Californians to source more of their meat from (e.g.) Brazil, and therefore cause more of the rainforest to be clearcut—well that’s not obviously good either.
5. Almost all interventions in our database targeted meat rather than other animal products (one looked at fish sauce and a couple also measured consumption of eggs and dairy). Also a lot of studies just say the choice was between a meat dish and a vegetarian dish, and whether that vegetarian dish contained eggs or milk is sometimes omitted. But in general, I’d think of these as “less meat” interventions.
Sorry I can’t offer anything more definitive here about what works and where people should donate. An economist I like says his dad’s first rule of social science research was: “Sometimes it’s this way, and sometimes it’s that way,” and I suppose I hew to that 😃