I try to avoid avoid the problem by discounting the average effect of psychotherapy. The point isn’t to try and find the “true effect”. The goal is to adjust for the risk of bias present in psychotherapy’s evidence base relative to the evidence base of cash transfers. We judge the CTs evidence to be higher quality. Psychotherapy has lower sample sizes on average and fewer unpublished studies, both of which are related to larger effect sizes in meta-analyses (MetaPsy, 2020; Vivalt, 2020, Dechartres et al., 2018 ;Slavin et al., 2016). FWIW I discuss this more in appendix C of the psychotherapy report.
I should note that I think the tool I use needs development. This issue of detecting and adjusting for the bias present in a study is a more general issue in social science.
I do worry about the effect sizes decreasing, but the hope is that the cost will drop to a greater degree as StrongMinds scales up.
We say “post-treatment effect” because it makes it clear the time point we are discussing. “Treatment effect” could refer either to the post-treatment effect or to the total effect of psychotherapy, where the total effect is the decision-relevant effect.
Hi Michael,
I try to avoid avoid the problem by discounting the average effect of psychotherapy. The point isn’t to try and find the “true effect”. The goal is to adjust for the risk of bias present in psychotherapy’s evidence base relative to the evidence base of cash transfers. We judge the CTs evidence to be higher quality. Psychotherapy has lower sample sizes on average and fewer unpublished studies, both of which are related to larger effect sizes in meta-analyses (MetaPsy, 2020; Vivalt, 2020, Dechartres et al., 2018 ;Slavin et al., 2016). FWIW I discuss this more in appendix C of the psychotherapy report.
I should note that I think the tool I use needs development. This issue of detecting and adjusting for the bias present in a study is a more general issue in social science.
I do worry about the effect sizes decreasing, but the hope is that the cost will drop to a greater degree as StrongMinds scales up.
We say “post-treatment effect” because it makes it clear the time point we are discussing. “Treatment effect” could refer either to the post-treatment effect or to the total effect of psychotherapy, where the total effect is the decision-relevant effect.