A few things that stand out to me that seem dodgy and make me doubt this analysis:
One of the studies you included with the strongest effect (Araya et al. 2003 in Chile with an effect of 0.9 Cohens d) uses antidepressants as part of the intervention. Why did you include this? How many other studies included non-psychotherapy interventions?
Some of the studies deal with quite specific groups of people eg. survivors of violence, pregnant women, HIV-affected women with young children. Generalising from psychotherapyâs effects in these groups to psychotherapy in the general population seems unreasonable.
Similarly, the therapies applied between studies seem highly variable including âAntenatal Emotional Self-Management Trainingâ, group therapy, one-on-one peer mentors. Lumping these together and drawing conclusions about âpsychotherapyâ generally seems unreasonable.
With the difficulty of blinding patients to psychotherapy, there seems to be room for the Hawthorne effect to be skewing the results of each of the 39 studies: with patients who are aware that theyâve received therapy feeling obliged to say that it helped.
Other minor things: - Multiple references to Appendix D. Where is Appendix D? - Maybe Iâve missed it but do you properly list the studies you used somewhere. âHusain, 2017â is not enough info to go by.
I addressed the variance in the primacy of psychotherapy in the studies in response to Nickâs comment, so Iâll respond to your other issues.
Some of the studies deal with quite specific groups of people eg. survivors of violence, pregnant women, HIV-affected women with young children. Generalising from psychotherapyâs effects in these groups to psychotherapy in the general population seems unreasonable.
I agree this would be a problem if we only had evidence from one quite specific group. But when we have evidence from multiple groups, and we donât have strong reasons for thinking that psychotherapy will affect these groups differently than the general populationâI think itâs better to include rather than exclude them.
I didnât show enough robustness checks like this, which is a mistake Iâll remedy in the next version. I categorised the population of every study as involving âconflict or violenceâ, âgeneralâ or âHIVâ. Running these trial characteristics as moderating factors suggests that, if anything, adding these additional populations underestimates the efficacy. But this is a point worth returning to.
Similarly, the therapies applied between studies seem highly variable including âAntenatal Emotional Self-Management Trainingâ, group therapy, one-on-one peer mentors. Lumping these together and drawing conclusions about âpsychotherapyâ generally seems unreasonable.
Iâm less concerned with variation in the type of therapy not generalising because as I say in the report (page 5) â...different forms of psychotherapy share many of the same strategies. We do not focus on a particular form of psychotherapy. Previous meta-analyses find mixed evidence supporting the superiority of any one form of psychotherapy for treating depression (Cuijpers et al., 2019).â
Due to the fact most types of psychotherapy seem about as effective, and expertise doesnât seem to be of first order importance, I formed the view that if you regularly get someone talk to about their problems in a semi-structured way itâll probably be pretty good for them. This isnât a view Iâd defend to the death, but I held it strongly enough to justify (at least to myself and the team) doing the simpler version of the analysis I performed.
With the difficulty of blinding patients to psychotherapy, there seems to be room for the Hawthorne effect to be skewing the results of each of the 39 studies: with patients who are aware that theyâve received therapy feeling obliged to say that it helped.
Right, but this is the case with most interventions (e.g., cash transfers). So long as the Hawthorne effect is balanced across interventions (which Iâm not implying is assured), then we should still be able to compare their cost-effectiveness using self-reports.
Furthermore, only 8 of the trials had waitlist or do nothing controls. The rest of the trials received some form of âcare as usualâ or a placebo like âHIV educationâ. Presumably these more active controls could also elicit a Hawthorne effect or response bias?
Hi Henry. Thanks for your feedback! Iâll let Joel respond to the substantive comments but just wanted to note that Iâve changed the âAppendix Dâ references to âAppendix Câ. Thanks very much for letting us know about that.
Iâm not sure why Appendix B has hyperlinks for some studies but not for others. Iâll check with Joel about that and add links to all the papers as soon as I can. In future, I plan to convert some of our data tables into embedded AirTables so that readers can reorder by different columns if they wish.
A few things that stand out to me that seem dodgy and make me doubt this analysis:
One of the studies you included with the strongest effect (Araya et al. 2003 in Chile with an effect of 0.9 Cohens d) uses antidepressants as part of the intervention. Why did you include this? How many other studies included non-psychotherapy interventions?
Some of the studies deal with quite specific groups of people eg. survivors of violence, pregnant women, HIV-affected women with young children. Generalising from psychotherapyâs effects in these groups to psychotherapy in the general population seems unreasonable.
Similarly, the therapies applied between studies seem highly variable including âAntenatal Emotional Self-Management Trainingâ, group therapy, one-on-one peer mentors. Lumping these together and drawing conclusions about âpsychotherapyâ generally seems unreasonable.
With the difficulty of blinding patients to psychotherapy, there seems to be room for the Hawthorne effect to be skewing the results of each of the 39 studies: with patients who are aware that theyâve received therapy feeling obliged to say that it helped.
Other minor things:
- Multiple references to Appendix D. Where is Appendix D?
- Maybe Iâve missed it but do you properly list the studies you used somewhere. âHusain, 2017â is not enough info to go by.
Hi Henry,
I addressed the variance in the primacy of psychotherapy in the studies in response to Nickâs comment, so Iâll respond to your other issues.
I agree this would be a problem if we only had evidence from one quite specific group. But when we have evidence from multiple groups, and we donât have strong reasons for thinking that psychotherapy will affect these groups differently than the general populationâI think itâs better to include rather than exclude them.
I didnât show enough robustness checks like this, which is a mistake Iâll remedy in the next version. I categorised the population of every study as involving âconflict or violenceâ, âgeneralâ or âHIVâ. Running these trial characteristics as moderating factors suggests that, if anything, adding these additional populations underestimates the efficacy. But this is a point worth returning to.
Iâm less concerned with variation in the type of therapy not generalising because as I say in the report (page 5) â...different forms of psychotherapy share many of the same strategies. We do not focus on a particular form of psychotherapy. Previous meta-analyses find mixed evidence supporting the superiority of any one form of psychotherapy for treating depression (Cuijpers et al., 2019).â
Due to the fact most types of psychotherapy seem about as effective, and expertise doesnât seem to be of first order importance, I formed the view that if you regularly get someone talk to about their problems in a semi-structured way itâll probably be pretty good for them. This isnât a view Iâd defend to the death, but I held it strongly enough to justify (at least to myself and the team) doing the simpler version of the analysis I performed.
Right, but this is the case with most interventions (e.g., cash transfers). So long as the Hawthorne effect is balanced across interventions (which Iâm not implying is assured), then we should still be able to compare their cost-effectiveness using self-reports.
Furthermore, only 8 of the trials had waitlist or do nothing controls. The rest of the trials received some form of âcare as usualâ or a placebo like âHIV educationâ. Presumably these more active controls could also elicit a Hawthorne effect or response bias?
Hi Henry. Thanks for your feedback! Iâll let Joel respond to the substantive comments but just wanted to note that Iâve changed the âAppendix Dâ references to âAppendix Câ. Thanks very much for letting us know about that.
Iâm not sure why Appendix B has hyperlinks for some studies but not for others. Iâll check with Joel about that and add links to all the papers as soon as I can. In future, I plan to convert some of our data tables into embedded AirTables so that readers can reorder by different columns if they wish.