To be honest Iâm having trouble pinning down what the central claim of the meta-analysis is.
To paraphrase Diddyâs character in Get Him to the Greek, âWhat are you talking about, the name of the [paper] is called â[Meaningfully reducing consumption of meat and animal products is an unsolved problem]!â (đ) That is our central claim. Weâre not saying nothing works; weâre saying that meaningful reductions either have not been discovered yet or do not have substantial evidence in support.
However the authors hedge this in places
Thatâs author, singular. I said at the top of my initial response that I speak only for myself.
I think âan unsolved problemâ could indicate several things. it could be
We have evidence that all of the commonly tried approaches are ineffective, i.e., we have measured all of their effects and they are tightly bounded as being very small
We have a lack of evidence, thus very wide credible intervals over the impact of each of the common approaches.
To me, the distinction is important. Do you agree?
You say above
meaningful reductions either have not been discovered yet or do not have substantial evidence in support
But even âdo not have substantial evidence in supportâ could mean either of the above ⌠a lack of evidence, or strong evidence that the effects are close to zero. At least to my ears.
As for âhedge thisâ, I was referring to the paper not to the response, but I can check this again.
For what itâs worth, I read that abstract as saying something like, âwithin the class of interventions studied so far, the literature has yet to settle onto any intervention that can reliably reduce animal product consumption by a meaningful amount, where meaningful amount might be a 1% reduction at Costco scale or long-term 10% reduction at a single cafeteria. The class of interventions being studied tends to be informational and nudge-style interventions like advertising, menu design, and media pamphlets. When effect sizes differ for a given type of intervention, the literature has not offered a convincing reason why a menu-design choice works in one setting versus another.â
Okay, now that Iâve typed that up, I can see why âunsolved problemâ is unclear.
And Iâm probably taking a lot of leaps of faith in interpretation here
From the POV of our core contention -- that we donât currently have a validated, reliable intervention to deploy at scaleâwhether this is because of absence of evidence (AoE) or evidence of absence (EoA) is hard to say. I donât have an overall answer, and ultimately both roads lead to âunsolved problem.â
We can cite good arguments for EoA (these studies are stronger than the norm in the field but show weaker effects, and that relationship should be troubling for advocates) or AoE (weâre not talking about very many studies at all), and ultimately I think the line between the two is in the eye of the beholder.
Going approach by approach, my personal answers are
choice architecture is probably AoE, it might work better than expected but we just donât learn very much from 2 studies (I am working on something about this separately)
the animal welfare appeals are more EoA, esp. those from animal advocacy orgs
social psych approaches, Iâm skeptical of but there werenât a lot of high-quality papers so Iâm not so sure (see here for a subsequent meta-analysis of dynamic norms approaches).
I would recommend health for older folks, environmental appeals for Gen Z. So there Iâd say we have evidence of efficacy, but to expect effects to be on the order of a few percentage points.
Were I discussing this specifically with a funder, I would say, if youâre going to do one of the meta-analyzed approachesâpsych, nudge, environment, health, or animal welfare, or some hybrid thereofâyou should expect small effect sizes unless you have some strong reason to believe that your intervention is meaningfully better than the category average. For instance, animal welfare appeals might not work in general, but maybe watching Dominion is unusually effective. However, as we say in our paper, there are a lot of cool ideas that havenât been tested rigorously yet, and from the point of view of knowledge, Iâd like to see those get funded first.
Hi David,
To paraphrase Diddyâs character in Get Him to the Greek, âWhat are you talking about, the name of the [paper] is called â[Meaningfully reducing consumption of meat and animal products is an unsolved problem]!â (đ) That is our central claim. Weâre not saying nothing works; weâre saying that meaningful reductions either have not been discovered yet or do not have substantial evidence in support.
Thatâs author, singular. I said at the top of my initial response that I speak only for myself.
I think âan unsolved problemâ could indicate several things. it could be
We have evidence that all of the commonly tried approaches are ineffective, i.e., we have measured all of their effects and they are tightly bounded as being very small
We have a lack of evidence, thus very wide credible intervals over the impact of each of the common approaches.
To me, the distinction is important. Do you agree?
You say above
But even âdo not have substantial evidence in supportâ could mean either of the above ⌠a lack of evidence, or strong evidence that the effects are close to zero. At least to my ears.
As for âhedge thisâ, I was referring to the paper not to the response, but I can check this again.
For what itâs worth, I read that abstract as saying something like, âwithin the class of interventions studied so far, the literature has yet to settle onto any intervention that can reliably reduce animal product consumption by a meaningful amount, where meaningful amount might be a 1% reduction at Costco scale or long-term 10% reduction at a single cafeteria. The class of interventions being studied tends to be informational and nudge-style interventions like advertising, menu design, and media pamphlets. When effect sizes differ for a given type of intervention, the literature has not offered a convincing reason why a menu-design choice works in one setting versus another.â
Okay, now that Iâve typed that up, I can see why âunsolved problemâ is unclear.
And Iâm probably taking a lot of leaps of faith in interpretation here
Itâs an interesting question.
From the POV of our core contention -- that we donât currently have a validated, reliable intervention to deploy at scaleâwhether this is because of absence of evidence (AoE) or evidence of absence (EoA) is hard to say. I donât have an overall answer, and ultimately both roads lead to âunsolved problem.â
We can cite good arguments for EoA (these studies are stronger than the norm in the field but show weaker effects, and that relationship should be troubling for advocates) or AoE (weâre not talking about very many studies at all), and ultimately I think the line between the two is in the eye of the beholder.
Going approach by approach, my personal answers are
choice architecture is probably AoE, it might work better than expected but we just donât learn very much from 2 studies (I am working on something about this separately)
the animal welfare appeals are more EoA, esp. those from animal advocacy orgs
social psych approaches, Iâm skeptical of but there werenât a lot of high-quality papers so Iâm not so sure (see here for a subsequent meta-analysis of dynamic norms approaches).
I would recommend health for older folks, environmental appeals for Gen Z. So there Iâd say we have evidence of efficacy, but to expect effects to be on the order of a few percentage points.
Were I discussing this specifically with a funder, I would say, if youâre going to do one of the meta-analyzed approachesâpsych, nudge, environment, health, or animal welfare, or some hybrid thereofâyou should expect small effect sizes unless you have some strong reason to believe that your intervention is meaningfully better than the category average. For instance, animal welfare appeals might not work in general, but maybe watching Dominion is unusually effective. However, as we say in our paper, there are a lot of cool ideas that havenât been tested rigorously yet, and from the point of view of knowledge, Iâd like to see those get funded first.