Yes, I elided the distinction between guided and unguided. The intervention I described was maximally unguided, while most studies involve more guidance. It seems to me the 2019 meta-analysis you cite is unusually pessimistic about the unguided case (compare e.g. this 2024 meta-analysis) but agree that most of the evidence support some form of guidance. Really this is a spectrum, and the question is what is the maximally cost-effective point on the unguided to guided spectrum. Your earlier post on this makes a lot of sense to me.
I am also interested in your preference for apps over books. They are cheaper. You say “the only reason not to do this” is internet connectivity issues. But some research indicates user preference for print media over digital media, e.g. this 2022 study in a US population. Do you think switching from print to apps lowers effectiveness (but is justified by cost savings) or that it is of the same (or greater) effectiveness?
The devil’s in the details here. The meta-analysis you cite includes an overall estimate for unguided self-help which aggregates over different control condition types (waitlist, care-as-usual, others). When breaking down by control condition, and adding Karyotaki et al. (2021), which looks at guided vs unguided internet-based studies:
Waitlist control
Cuijpers et al. (2019): 0.52 (Guided: 0.87)
Karyotaki et al. (2021): 0.6 (Guided: 0.8)
Tong et al. (2024): 0.71
Care as usual control
Cuijpers et al. (2019): 0.13 (Guided: 0.47)
Karyotaki et al. (2021): 0.2 (Guided: 0.4)
Tong et al. (2024): 0.35
Now, Tong et al. (2024) does seem to find higher effects in general, but 32% of the interventions studied included regular human encouragement. These conditions found effect sizes of 0.62, compared to 0.47 for no support (I wish these would disaggregate against control conditions, but alas).
Tong has significantly more self-guided studies than Cuijpers. When both limit to just low risk-of-bias studies, Tong reports an effect sizes of 0.43 (not disaggregated across controls, unfortunately). Cuijpers reports 0.44 for waitlist controls, and 0.13 for care as usual, for the same restriction. So Tong has included more high risk-of-bias studies, which is quite dramatically lifting their effect size.
Now, as Cuijpers and Karyotaki are both co-authors on the Tong analysis, I’m sure that there’s value in including those extra studies, and Tong probably makes me slightly update on the effectiveness of unguided self-help. But I would be wary about concluding that Tong is ‘normal’ and Cuijpers is ‘unusually pessimistic’; probably the inverse is true.
(All up, though, I think that it’s quite likely that unguided interventions could end up being more optimally cost-effective than guided ones, and I’m excited to see how this space develops. I’d definitely encourage people to try and pursue this more! I don’t think the case for unguided interventions rests on their relative effectiveness, but much more on their relative costs.)
Apps vs books
I don’t have a strong sense here. The Jesus-Romero study is good and persuasive, but to be convinced, I’d want to see a study of revealed preferences rather than stated ones. To illustrate, one reason we think apps might be better is because our digital ads reach users while they’re on media apps like Instagram, which is a behaviour people are probably likely to engage in when depressed. I think there’s probably a much lower activity threshold to click on an ad and book a call with us, than there is to remember you ordered/were sent a book and to do an exercises from it.
Regardless, it’s likely that digital interventions are much cheaper (again, probably about $1 per participant in engineering vs. ~$5 (??) for book printing and delivery, assuming both interventions spend $1 on social media), and can scale much faster (printing and delivery requires a lot of extra coordination and volume prediction). There’s a good reason many for-profit industries have digitised; it’s just much cheaper and more flexible.
On the meta-analyses: that seems fair. I think my initial thought was just that the Cuijpers seemed very low relative to my priors, and the Tong seemed more in line with them. But maybe my priors are wrong! I take your point that the Tong may be too high because of how widely it casts the “unguided” net, though it still does find some meaningful difference. But on the main point I think we’re in agreement: guided > unguided, and the case for unguidedness, if there is one, will depend on its relative cost-effectiveness.
On apps v. books: I think there are so many potentially countervailing effects here it’s hard to trust my intuitive judgments. I see the consideration you cite, but on the other hand (I would guess) someone on a phone is more likely to defect away from self-help and to use their phones for all the other things that phones can be used for. It would be great to have more studies here. There are a few RCT’s comparing print with courses delivered via the internet on a desktop/laptop, which seem to find little difference either way, but these studies are very sparse, and they’re at some remove from the question of comparing self-help delivery via a printed book via self-help delivery via Whatapp.
I take the point about cost-effectiveness. Certainly the tendency in the for-profit space has been digitization. But here too there’s a countervailing consideration. Digitized self-help is a natural fit for the for-profit space, since a product that can be monetized in various ways (subscriptions, advertising) and produced at 0 marginal cost offers an attractive business model. But books do not fit that model. So perhaps one role for NGO’s in this space may be supporting interventions which are known to be effective but whose financials are less promising, and perhaps self-help books are a case of this.
I suspect the synthesis here is that unguided is very effective when adhered to, but the main challenge is adherence. The reason to believe this is that there is usually a strong dosage effect in psychotherapy studies, and that the Furukawa study I posted in the first comment found that the only value humans provided was for adherence, not effect size.
Unfortunately, this would then cause big problems, because there is likely a trial bias affecting adherence, potentially inflating estimates by 4× against real-world data. I’m surprised that this isn’t covered in the literature, and my surprise is probably good evidence that I have something wrong here. This is one of the reasons I’m keen to study our intervention’s real-world data in a comparative RCT.
You make a strong point about the for-profit space and relative incentives, which is partly why, when I had to make a decision between founding a for-profit unguided app and joining Kaya Guides, I chose the guided option. As you note, the way the incentives seem to work is that large for-profits can serve LMICs only when profit margins are competitive with expanding further in HICs. This is the case for unguided apps, because translation and adaptation is a cheap fixed cost. But as soon as you have marginal costs, like hiring humans (or buying books, or possibly, paying for AI compute), it stops making sense. This is why BetterHelp have only now begun to expand beyond the U.S. to other rich countries.
But I think you implicitly note—if one intervention has zero marginal cost, then surely it’s going to be more cost-effective and therefore more attractive to funders? One model I’ve wondered about for an unguided for-profit is essentially licensing its core technology and brand to a non-profit at cost, which would then receive donations, do translations, and distribute in other markets.
Hi Huw,
This is very helpful! A couple of quick notes:
Yes, I elided the distinction between guided and unguided. The intervention I described was maximally unguided, while most studies involve more guidance. It seems to me the 2019 meta-analysis you cite is unusually pessimistic about the unguided case (compare e.g. this 2024 meta-analysis) but agree that most of the evidence support some form of guidance. Really this is a spectrum, and the question is what is the maximally cost-effective point on the unguided to guided spectrum. Your earlier post on this makes a lot of sense to me.
I am also interested in your preference for apps over books. They are cheaper. You say “the only reason not to do this” is internet connectivity issues. But some research indicates user preference for print media over digital media, e.g. this 2022 study in a US population. Do you think switching from print to apps lowers effectiveness (but is justified by cost savings) or that it is of the same (or greater) effectiveness?
Tong et al. (2024)
The devil’s in the details here. The meta-analysis you cite includes an overall estimate for unguided self-help which aggregates over different control condition types (waitlist, care-as-usual, others). When breaking down by control condition, and adding Karyotaki et al. (2021), which looks at guided vs unguided internet-based studies:
Waitlist control
Cuijpers et al. (2019): 0.52 (Guided: 0.87)
Karyotaki et al. (2021): 0.6 (Guided: 0.8)
Tong et al. (2024): 0.71
Care as usual control
Cuijpers et al. (2019): 0.13 (Guided: 0.47)
Karyotaki et al. (2021): 0.2 (Guided: 0.4)
Tong et al. (2024): 0.35
Now, Tong et al. (2024) does seem to find higher effects in general, but 32% of the interventions studied included regular human encouragement. These conditions found effect sizes of 0.62, compared to 0.47 for no support (I wish these would disaggregate against control conditions, but alas).
Tong has significantly more self-guided studies than Cuijpers. When both limit to just low risk-of-bias studies, Tong reports an effect sizes of 0.43 (not disaggregated across controls, unfortunately). Cuijpers reports 0.44 for waitlist controls, and 0.13 for care as usual, for the same restriction. So Tong has included more high risk-of-bias studies, which is quite dramatically lifting their effect size.
Now, as Cuijpers and Karyotaki are both co-authors on the Tong analysis, I’m sure that there’s value in including those extra studies, and Tong probably makes me slightly update on the effectiveness of unguided self-help. But I would be wary about concluding that Tong is ‘normal’ and Cuijpers is ‘unusually pessimistic’; probably the inverse is true.
(All up, though, I think that it’s quite likely that unguided interventions could end up being more optimally cost-effective than guided ones, and I’m excited to see how this space develops. I’d definitely encourage people to try and pursue this more! I don’t think the case for unguided interventions rests on their relative effectiveness, but much more on their relative costs.)
Apps vs books
I don’t have a strong sense here. The Jesus-Romero study is good and persuasive, but to be convinced, I’d want to see a study of revealed preferences rather than stated ones. To illustrate, one reason we think apps might be better is because our digital ads reach users while they’re on media apps like Instagram, which is a behaviour people are probably likely to engage in when depressed. I think there’s probably a much lower activity threshold to click on an ad and book a call with us, than there is to remember you ordered/were sent a book and to do an exercises from it.
Regardless, it’s likely that digital interventions are much cheaper (again, probably about $1 per participant in engineering vs. ~$5 (??) for book printing and delivery, assuming both interventions spend $1 on social media), and can scale much faster (printing and delivery requires a lot of extra coordination and volume prediction). There’s a good reason many for-profit industries have digitised; it’s just much cheaper and more flexible.
On the meta-analyses: that seems fair. I think my initial thought was just that the Cuijpers seemed very low relative to my priors, and the Tong seemed more in line with them. But maybe my priors are wrong! I take your point that the Tong may be too high because of how widely it casts the “unguided” net, though it still does find some meaningful difference. But on the main point I think we’re in agreement: guided > unguided, and the case for unguidedness, if there is one, will depend on its relative cost-effectiveness.
On apps v. books: I think there are so many potentially countervailing effects here it’s hard to trust my intuitive judgments. I see the consideration you cite, but on the other hand (I would guess) someone on a phone is more likely to defect away from self-help and to use their phones for all the other things that phones can be used for. It would be great to have more studies here. There are a few RCT’s comparing print with courses delivered via the internet on a desktop/laptop, which seem to find little difference either way, but these studies are very sparse, and they’re at some remove from the question of comparing self-help delivery via a printed book via self-help delivery via Whatapp.
I take the point about cost-effectiveness. Certainly the tendency in the for-profit space has been digitization. But here too there’s a countervailing consideration. Digitized self-help is a natural fit for the for-profit space, since a product that can be monetized in various ways (subscriptions, advertising) and produced at 0 marginal cost offers an attractive business model. But books do not fit that model. So perhaps one role for NGO’s in this space may be supporting interventions which are known to be effective but whose financials are less promising, and perhaps self-help books are a case of this.
I suspect the synthesis here is that unguided is very effective when adhered to, but the main challenge is adherence. The reason to believe this is that there is usually a strong dosage effect in psychotherapy studies, and that the Furukawa study I posted in the first comment found that the only value humans provided was for adherence, not effect size.
Unfortunately, this would then cause big problems, because there is likely a trial bias affecting adherence, potentially inflating estimates by 4× against real-world data. I’m surprised that this isn’t covered in the literature, and my surprise is probably good evidence that I have something wrong here. This is one of the reasons I’m keen to study our intervention’s real-world data in a comparative RCT.
You make a strong point about the for-profit space and relative incentives, which is partly why, when I had to make a decision between founding a for-profit unguided app and joining Kaya Guides, I chose the guided option. As you note, the way the incentives seem to work is that large for-profits can serve LMICs only when profit margins are competitive with expanding further in HICs. This is the case for unguided apps, because translation and adaptation is a cheap fixed cost. But as soon as you have marginal costs, like hiring humans (or buying books, or possibly, paying for AI compute), it stops making sense. This is why BetterHelp have only now begun to expand beyond the U.S. to other rich countries.
But I think you implicitly note—if one intervention has zero marginal cost, then surely it’s going to be more cost-effective and therefore more attractive to funders? One model I’ve wondered about for an unguided for-profit is essentially licensing its core technology and brand to a non-profit at cost, which would then receive donations, do translations, and distribute in other markets.