Thanks Jacob! It’s great to see what was interesting / useful / confusing etc for people, and generally quite hard to get detailed feedback, so I appreciate you taking the time to read and reply.
I’m sure we could debate these topics at length; that’s a tempting prospect, but I’ll just reply to some specific parts here.
I don’t see reason to give up on trying to conduct high-quality studies
To summarise my view update from this review in other terms:
Lots of money has gone into health behaviour research. I expected the health behaviour literature to come to some fairly strong conclusions about the value of some intervention types over others. This didn’t seem to be the case, given various limitations and inconsistencies in the research. Hence, I’m less optimistic about the usefulness of conducting comparable research now, relative to other types of research that we could conduct.
It seems like social movement case studies are then limited generally, like most observational research, to understanding correlations and motivating causal theories about those correlations, rather than measuring causation itself.
have you considered the merits of regression discontinuity designs, instrumental variables estimation, propensity score matching and prospective cohort studies, for example
I have never heard of these things, let alone considered their merits! I don’t think that invalidates the view update I describe above, though if I look into these things more, it might restore my confidence?
To clarify, I suspect we have some agreement on (social movement) case studies: I do think they can provide evidence towards causation—literally that one should update their subjective Bayesian beliefs about causation based on social movement case studies. However, at least to my understanding of the current methods, they cannot provide causal identification, thus vastly limiting the magnitude of that update. (In my mind, to probably <10%.)
What I’m struggling to understand fundamentally is your conception of the quality of evidence. If you find the quality of evidence of the health behavior literature low, how does that compare to the quality of evidence of SI’s social movement case studies? One intuition pump might be that the health behavior literature undoubtedly contains scores of cross-sectional studies, which themselves could be construed as each containing hundreds of case studies, and these cross-sectional studies are still regarded as much weaker evidence than the scores of RCTs in the health behavior literature. So where then must a single case study lie?
For what it’s worth, in reflecting on an update which is fundamentally about how to make causal inferences, it seems like being unfamiliar with common tools for causal inference (eg, instrumental variables) warrants updating towards an uninformed prior. I’m not sure if they’ll restore your confidence, but I’d be interested to hear.
However, at least to my understanding of the current methods, they cannot provide causal identification, thus vastly limiting the magnitude of that update. (In my mind, to probably <10%.)
Interesting. Let’s imagine a specific question that we might be interested in, e.g. do incremental improvements (e.g. on welfare of animals or prisoners) encourage momentum for further change or complacency? ~10% sounds about right to me as an upper limit on an update from a single case study. But a case study will provide information on far more questions of interest than this single question. And as we look at several case studies and start to compare between them, then I can imagine an update of more like ~40% from historical social movement evidence in general on any single question of interest.
One intuition pump might be that the health behavior literature undoubtedly contains scores of cross-sectional studies, which themselves could be construed as each containing hundreds of case studies
That may be so, but they would be providing evidence on very different types of cause and effect relationships. E.g. the effects of motivational interviews on dietary behaviour, vs the effects of incremental improvements (e.g. on welfare of animals or prisoners) on a movement’s momentum for further change. When I’m thinking about the value of social movement case studies compared to RCTs, I’m also thinking about their ability to provide evidence on the questions that I think are most important.
Thank you for your replies, Jamie, I appreciate the discussion. As a last point of clarification when you say ~40%, does this, for example, mean that if a priori I was uninformed on momentum v complacency and so put 50/50% credence on either possibility, that a series of case studies might potentially update you to 90/10%?
When I’m thinking about the value of social movement case studies compared to RCTs, I’m also thinking about their ability to provide evidence on the questions that I think are most important
I don’t disagree—but my point with this intuition pump is the strength of inference a case study, or even series of case studies, might provide on any one of those questions.
Yes to the first part! (I was also thinking something like: If you had read some of the other available evidence but not the historical case studies and had 70/30% credence, then reading the historical case studies might update your views to 30/70%. But that’s a bit messier.)
And got it with the second; I think we mostly agree there.
Ah, I see—in that case, it makes a lot of sense for you to pursue these case studies. I appreciate the time you invested to get to a double crux here, thanks!
Thanks Jacob! It’s great to see what was interesting / useful / confusing etc for people, and generally quite hard to get detailed feedback, so I appreciate you taking the time to read and reply.
I’m sure we could debate these topics at length; that’s a tempting prospect, but I’ll just reply to some specific parts here.
I still think RCTs have their uses. It’s just that they can be limited in various ways and that other research methods have some advantages over them, as discussed in the “EAA RCTs v intuition/speculation/anecdotes v case studies v external findings” section you refer to.
To summarise my view update from this review in other terms:
Lots of money has gone into health behaviour research. I expected the health behaviour literature to come to some fairly strong conclusions about the value of some intervention types over others. This didn’t seem to be the case, given various limitations and inconsistencies in the research. Hence, I’m less optimistic about the usefulness of conducting comparable research now, relative to other types of research that we could conduct.
I don’t agree with this. I think that you can look for evidence that X caused Y in a particular case, rather than just that X preceded Y. (Of course, often the evidence is very weak or nonexistent that X caused Y.) I discuss that in more depth here. You then have the separate problems of How much weight should we place on strategic knowledge from individual historical cases? and/or How likely is it that correlations will replicate across movements? It’s hard to describe answers to those questions in precise and unambiguous terms, but I’d answer them with something like “not a lot” and “quite likely,” respectively.
I have never heard of these things, let alone considered their merits! I don’t think that invalidates the view update I describe above, though if I look into these things more, it might restore my confidence?
To clarify, I suspect we have some agreement on (social movement) case studies: I do think they can provide evidence towards causation—literally that one should update their subjective Bayesian beliefs about causation based on social movement case studies. However, at least to my understanding of the current methods, they cannot provide causal identification, thus vastly limiting the magnitude of that update. (In my mind, to probably <10%.)
What I’m struggling to understand fundamentally is your conception of the quality of evidence. If you find the quality of evidence of the health behavior literature low, how does that compare to the quality of evidence of SI’s social movement case studies? One intuition pump might be that the health behavior literature undoubtedly contains scores of cross-sectional studies, which themselves could be construed as each containing hundreds of case studies, and these cross-sectional studies are still regarded as much weaker evidence than the scores of RCTs in the health behavior literature. So where then must a single case study lie?
For what it’s worth, in reflecting on an update which is fundamentally about how to make causal inferences, it seems like being unfamiliar with common tools for causal inference (eg, instrumental variables) warrants updating towards an uninformed prior. I’m not sure if they’ll restore your confidence, but I’d be interested to hear.
Interesting. Let’s imagine a specific question that we might be interested in, e.g. do incremental improvements (e.g. on welfare of animals or prisoners) encourage momentum for further change or complacency? ~10% sounds about right to me as an upper limit on an update from a single case study. But a case study will provide information on far more questions of interest than this single question. And as we look at several case studies and start to compare between them, then I can imagine an update of more like ~40% from historical social movement evidence in general on any single question of interest.
That may be so, but they would be providing evidence on very different types of cause and effect relationships. E.g. the effects of motivational interviews on dietary behaviour, vs the effects of incremental improvements (e.g. on welfare of animals or prisoners) on a movement’s momentum for further change. When I’m thinking about the value of social movement case studies compared to RCTs, I’m also thinking about their ability to provide evidence on the questions that I think are most important.
Thank you for your replies, Jamie, I appreciate the discussion. As a last point of clarification when you say ~40%, does this, for example, mean that if a priori I was uninformed on momentum v complacency and so put 50/50% credence on either possibility, that a series of case studies might potentially update you to 90/10%?
I don’t disagree—but my point with this intuition pump is the strength of inference a case study, or even series of case studies, might provide on any one of those questions.
Yes to the first part! (I was also thinking something like: If you had read some of the other available evidence but not the historical case studies and had 70/30% credence, then reading the historical case studies might update your views to 30/70%. But that’s a bit messier.)
And got it with the second; I think we mostly agree there.
Ah, I see—in that case, it makes a lot of sense for you to pursue these case studies. I appreciate the time you invested to get to a double crux here, thanks!