From the POV of our core contention -- that we donât currently have a validated, reliable intervention to deploy at scaleâwhether this is because of absence of evidence (AoE) or evidence of absence (EoA) is hard to say. I donât have an overall answer, and ultimately both roads lead to âunsolved problem.â
We can cite good arguments for EoA (these studies are stronger than the norm in the field but show weaker effects, and that relationship should be troubling for advocates) or AoE (weâre not talking about very many studies at all), and ultimately I think the line between the two is in the eye of the beholder.
Going approach by approach, my personal answers are
choice architecture is probably AoE, it might work better than expected but we just donât learn very much from 2 studies (I am working on something about this separately)
the animal welfare appeals are more EoA, esp. those from animal advocacy orgs
social psych approaches, Iâm skeptical of but there werenât a lot of high-quality papers so Iâm not so sure (see here for a subsequent meta-analysis of dynamic norms approaches).
I would recommend health for older folks, environmental appeals for Gen Z. So there Iâd say we have evidence of efficacy, but to expect effects to be on the order of a few percentage points.
Were I discussing this specifically with a funder, I would say, if youâre going to do one of the meta-analyzed approachesâpsych, nudge, environment, health, or animal welfare, or some hybrid thereofâyou should expect small effect sizes unless you have some strong reason to believe that your intervention is meaningfully better than the category average. For instance, animal welfare appeals might not work in general, but maybe watching Dominion is unusually effective. However, as we say in our paper, there are a lot of cool ideas that havenât been tested rigorously yet, and from the point of view of knowledge, Iâd like to see those get funded first.
Itâs an interesting question.
From the POV of our core contention -- that we donât currently have a validated, reliable intervention to deploy at scaleâwhether this is because of absence of evidence (AoE) or evidence of absence (EoA) is hard to say. I donât have an overall answer, and ultimately both roads lead to âunsolved problem.â
We can cite good arguments for EoA (these studies are stronger than the norm in the field but show weaker effects, and that relationship should be troubling for advocates) or AoE (weâre not talking about very many studies at all), and ultimately I think the line between the two is in the eye of the beholder.
Going approach by approach, my personal answers are
choice architecture is probably AoE, it might work better than expected but we just donât learn very much from 2 studies (I am working on something about this separately)
the animal welfare appeals are more EoA, esp. those from animal advocacy orgs
social psych approaches, Iâm skeptical of but there werenât a lot of high-quality papers so Iâm not so sure (see here for a subsequent meta-analysis of dynamic norms approaches).
I would recommend health for older folks, environmental appeals for Gen Z. So there Iâd say we have evidence of efficacy, but to expect effects to be on the order of a few percentage points.
Were I discussing this specifically with a funder, I would say, if youâre going to do one of the meta-analyzed approachesâpsych, nudge, environment, health, or animal welfare, or some hybrid thereofâyou should expect small effect sizes unless you have some strong reason to believe that your intervention is meaningfully better than the category average. For instance, animal welfare appeals might not work in general, but maybe watching Dominion is unusually effective. However, as we say in our paper, there are a lot of cool ideas that havenât been tested rigorously yet, and from the point of view of knowledge, Iâd like to see those get funded first.