I agree with Huw’s assessment re: books vs digital vs digital + guide. Here are a few less-discussed reasons why, hastily scribbled:
Recruitment and retention costs: The cost of delivering a very cost-effective therapy is often lower than the cost of convincing someone to seriously give it a go. People don’t really want to just read a book or just use an app; they overwhelmingly want to talk to a real person. It can therefore be cheaper to recruit and retain people when a person is involved.
Misinterpretation of non-significant: Psychologists often present their findings as though statistically non-significant differences should be ignored. Sometimes this results in treating an effect size of 0.3 and 0.6 as if they’re identical, leading to conclusions like “we found no significant differences between guided and unguided…”. Nobody has time to read the whole literature, so people skim—and can come away thinking there’s no real difference, when in practice it may be more like a 30–100% difference in effectiveness.
Greater publication bias in unguided RCTs: It’s insanely cheap to do RCTs on unguided interventions because the cost of delivery is near zero and logistics are simple. Since it’s usually the researcher or funder who developed the treatment, they’re unlikely to publish the mediocre results. What gets published instead are lots and lots of positive findings, creating a skewed picture where unguided looks consistently effective.
Retention IRL: Despite most mental health apps showing >50% completion in RCTs, they retain only ~1–3% of real users that long. Guided self-help interventions retain an order of magnitude more. You thus need recruit an order of magnitude more users to treat the same number of people. This not only undermines their cost-effectiveness, but also drives up recruitment costs for everyone else. Plus, a lot of people try something that doesn’t work for them, have their time and effort wasted, become more jaded, and are harder to convince to try again later.
All that being said, I think we focus far too much on differences between treatments and far too little on differences between clients. The latter explains roughly 4× more variance than the former, yet accounts for <1% of the research published.
Thanks for this reply! In general I agree on the effectiveness of guidance, as in my response to Huw above. The publication bias issue (3) is one that I hadn’t thought about enough and may well distort our evidence on some of these questions.
I agree with Huw’s assessment re: books vs digital vs digital + guide. Here are a few less-discussed reasons why, hastily scribbled:
Recruitment and retention costs: The cost of delivering a very cost-effective therapy is often lower than the cost of convincing someone to seriously give it a go. People don’t really want to just read a book or just use an app; they overwhelmingly want to talk to a real person. It can therefore be cheaper to recruit and retain people when a person is involved.
Misinterpretation of non-significant: Psychologists often present their findings as though statistically non-significant differences should be ignored. Sometimes this results in treating an effect size of 0.3 and 0.6 as if they’re identical, leading to conclusions like “we found no significant differences between guided and unguided…”. Nobody has time to read the whole literature, so people skim—and can come away thinking there’s no real difference, when in practice it may be more like a 30–100% difference in effectiveness.
Greater publication bias in unguided RCTs: It’s insanely cheap to do RCTs on unguided interventions because the cost of delivery is near zero and logistics are simple. Since it’s usually the researcher or funder who developed the treatment, they’re unlikely to publish the mediocre results. What gets published instead are lots and lots of positive findings, creating a skewed picture where unguided looks consistently effective.
Retention IRL: Despite most mental health apps showing >50% completion in RCTs, they retain only ~1–3% of real users that long. Guided self-help interventions retain an order of magnitude more. You thus need recruit an order of magnitude more users to treat the same number of people. This not only undermines their cost-effectiveness, but also drives up recruitment costs for everyone else. Plus, a lot of people try something that doesn’t work for them, have their time and effort wasted, become more jaded, and are harder to convince to try again later.
All that being said, I think we focus far too much on differences between treatments and far too little on differences between clients. The latter explains roughly 4× more variance than the former, yet accounts for <1% of the research published.
Thanks for this reply! In general I agree on the effectiveness of guidance, as in my response to Huw above. The publication bias issue (3) is one that I hadn’t thought about enough and may well distort our evidence on some of these questions.