Thanks for the post John! Very informative. I know some people thinking of doing another RCT on this and will definitely point them to it.
Also agree that heterogeneities in the actual intervention as well as population under study are major challenges here in generalizing the effects (and they are common in studies on social science interventions which probably lead to lower generalizability than medical trials).
One minor and meta comment on section 2: “How over-optimistic should we expect the evidence to be?” I’m not sure how I feel about having a section on this in a post like yours. It’s totally reasonable as a way to form your prior before examining the literature, but after you do that (motivated by your skepticism based on these reasons) your learning from examining the literature “screens off” the factors that made you skeptical in the first place. (E.g. it may well be that the studies turn out to have super rigorous methodology, even though they are psychological studies conducted by “true believes” etc., and the former should be the main factor influencing your posterior on the impact of meditation—unless the reasons that gave you a skeptical prior makes you think they may have fabricated data etc.)
So while what you said in that section is true in terms of forming a prior (before looking at the papers), I would have put it in a less prominent place in this post (perhaps at the end on “what made me particularly skeptical and hence more interested in examining the literature”). (It’s totally fine if readers feel what’s in section 3 mostly “screens off” what’s in section 2, but if not it may unfairly bias their perception against the studies.)
(Digression: in a completely different situation, if one didn’t examine the literature at all but just put out a skeptical prior based on these reasons—I would say that is the correct way of forming a prior, but it feels slightly unfair or irresponsible. But I probably would feel it’s okay if people highly qualify their statement, e.g. “I have a skeptical prior due to X, Y, and Z, but I really haven’t looked at the actual studies” and perhaps even “if I did look, things like A, B, and C would convince me the studies are actually reliable / unreliable”. I’m not sure about this point and curious for others’ thoughts, since this is probably how a lot of people talk about studies that they haven’t fully read on social media.)
Also a minor and concrete point on section 2: the 2nd bullet point “Most outcome metrics are subjective”. Here are some reasons we may or may not think (ex ante) the results may be overestimated.
If there’s a lot of noise in self-reported outcomes alone it actually doesn’t lead to bias (though in a case where the outcome variable is censored, as many psychological outcomes are, and outcomes are bunched near one end, that could happen).
Some relevant sources of bias are
Social desirability bias (respondents saying what they consider is socially desirable, should affect treatment and control respondents equally and apply to other psychological studies looking at the same outcome)
Courtesy bias (applies to treatment respondents, who may feel obligated to report positive impact)
And since these are self-reported outcomes that can’t be verified, 1) people may be less deterred from lying, 2) we will never find out the truth—so the two biases are potentially more severe (compared to a case where outcomes can be verified).
Thanks for the post John! Very informative. I know some people thinking of doing another RCT on this and will definitely point them to it.
Also agree that heterogeneities in the actual intervention as well as population under study are major challenges here in generalizing the effects (and they are common in studies on social science interventions which probably lead to lower generalizability than medical trials).
One minor and meta comment on section 2: “How over-optimistic should we expect the evidence to be?” I’m not sure how I feel about having a section on this in a post like yours. It’s totally reasonable as a way to form your prior before examining the literature, but after you do that (motivated by your skepticism based on these reasons) your learning from examining the literature “screens off” the factors that made you skeptical in the first place. (E.g. it may well be that the studies turn out to have super rigorous methodology, even though they are psychological studies conducted by “true believes” etc., and the former should be the main factor influencing your posterior on the impact of meditation—unless the reasons that gave you a skeptical prior makes you think they may have fabricated data etc.)
So while what you said in that section is true in terms of forming a prior (before looking at the papers), I would have put it in a less prominent place in this post (perhaps at the end on “what made me particularly skeptical and hence more interested in examining the literature”). (It’s totally fine if readers feel what’s in section 3 mostly “screens off” what’s in section 2, but if not it may unfairly bias their perception against the studies.)
(Digression: in a completely different situation, if one didn’t examine the literature at all but just put out a skeptical prior based on these reasons—I would say that is the correct way of forming a prior, but it feels slightly unfair or irresponsible. But I probably would feel it’s okay if people highly qualify their statement, e.g. “I have a skeptical prior due to X, Y, and Z, but I really haven’t looked at the actual studies” and perhaps even “if I did look, things like A, B, and C would convince me the studies are actually reliable / unreliable”. I’m not sure about this point and curious for others’ thoughts, since this is probably how a lot of people talk about studies that they haven’t fully read on social media.)
Also a minor and concrete point on section 2: the 2nd bullet point “Most outcome metrics are subjective”. Here are some reasons we may or may not think (ex ante) the results may be overestimated.
If there’s a lot of noise in self-reported outcomes alone it actually doesn’t lead to bias (though in a case where the outcome variable is censored, as many psychological outcomes are, and outcomes are bunched near one end, that could happen).
Some relevant sources of bias are
Social desirability bias (respondents saying what they consider is socially desirable, should affect treatment and control respondents equally and apply to other psychological studies looking at the same outcome)
Courtesy bias (applies to treatment respondents, who may feel obligated to report positive impact)
And since these are self-reported outcomes that can’t be verified, 1) people may be less deterred from lying, 2) we will never find out the truth—so the two biases are potentially more severe (compared to a case where outcomes can be verified).
(Please correct me if I’m wrong here!)