Awesome great post, fantastic to see variations of this intervention being considered. The main concerns we focused on with unguided vs guided delivery methods for self-help were recruitment costs and retention.
As you show in the relative engagement in recent trials retention seems to play out in favour of unguided given the lower costs. I think this is a fair read, from what I’ve seen from comparing the net effect size of these interventions the no less than half figure is about right too.
For recruitment, given the average rate of download for apps we discounted organic growth as you have. Then the figure cited for cost per install is correct (can see another source here) but for general installs. For the unguided self-help intervention, we want a subset of people with the/a condition we are targeting (~5-15%). We then want to turn those installs into active engaged users of the program. I don’t know how much these would increase the costs compared to the $0.02 - $0.10 figure. Plausibly 6-20 times for targeting the subgroup with the condition for installs (although even the general population may benefit a bit from the intervention). Then the cost of turning those installs into active users. Some stats suggest that on day one there is a 23.01% retention rate and then as low as 2.59% by day 30 (source, similar to your 3.3% real-world app data). So again looking at maybe a 4 to 40 times increase in recruitment costs depending. Overall that could increase costs for recruiting an active user by 24 to 800 times (Or $0.48 to $8).
Although these would be identical across unguided vs guided the main consideration then becomes does it costs more to recruit more users or to guide existing users and increase the follow-through/ effect size. Adding the above factors quickly to your CEA for recruitment cost gives a cost per active user of 9.6 (1.6 to 32) and cost-effectiveness for a guided of mean 24 (12 to 47) and unguided of mean 21 (4.8 to 66). Taking all of this at face value which version looks more cost-effective will depend a lot on exactly where and how the intervention is being employed. Even then there is uncertainty with some of these figures in a real-world setting.
For the unguided app model you outline I agree if successful this would be incredibly cost-effective. Although at present I’d still be uncertain which version would look best. Ultimately that’s up to groups implementing this family of interventions, like Kaya Guides, to explore through implementing and experimenting.
Thank you so much! Your criticism has helped me identify a few mistakes, and I think can get us closer to clarity. The main difference between our models is around who counts as a ‘beneficiary’, or what it means to ‘recruit’ someone.
The main thing I want to focus on is that you’re predicting a cost per beneficiary that would be nearly 50% recruitment. I don’t think that passes the smell test. The main difference is you’re only counting the staff time for active participants, but even with modest dropout, we’d expect the vast majority of staff time to go to users who only complete one or two calls. But you’re right to point out that we should factor in dropout between installation and the first guidance call, and when I factor this in, unguided has 7% of the cost of guided at scale.
The rest of this comment is just my working out.
Mistakes I made
One of the mistakes I made was having different definitions for recruitment for each condition in my cost model. If the number in that model says there are 100,000 beneficiaries, in the unguided model this means we got 100,000 installs, but in the guided model it means we got 100,000 participants who had 50–180 minutes of staff time allocated to them. Obviously there are different costs to recruit these kinds of participants.
(Two other mistakes I found: I shouldn’t have multiplied the unguided recruitment cost by 2 to account for engagement differences, and I forgot to discount the office space costs for the nonexistent guides in the unguided model)
I should’ve been clearer about a ‘beneficiary’
To rectify this, let’s count a ‘beneficiary’ as someone who is in the targeted subgroup and completes pre-treatment. This is in line with most of the literature, which counts ‘dropout’ regardless of whether users complete any of the material, so long as they’ve done their induction. We don’t want to filter this down to ‘active’ users, since users who drop out will still incur costs.
They spent US$105 and got 875 initial user interactions for a cost per interaction of $0.12 (this isn’t quite cost per install but it’s close enough)
Let’s consider this a lower bound, Meta’s targeting should get cheaper at scale and after optimisation on Kaya Guides’ end. I think it could get down to the $0.02–0.10 figure pretty easily (and it may have already been there if a number of interactions didn’t lead to installs)
82.65% of users who completed their depression questionnaire scored above threshold
This should also be a lower bound. Kaya Guides used basic ads, but Meta is able to optimise for ‘app events’ (such as scoring highly on this questionnaire) so should be better than this at targeting for depression (scary!)
12.34% (108) of these initially interacting users scheduled and picked up an initial call
Their programmes involve an initial call, a second call at 3 days (I confirmed this directly), and then 5–8 weeks of subsequent calls. All calls are 15 minutes.
1–10 calls per participant (lognormal), this is roughly what Kaya Guides and Step-By-Step do. I distributed it as a lognormal to quickly account for dropout rates, but the model is somewhat sensitive to this parameter. More knowledge on its distribution could change the overall calculus.
15–40 minutes of time spent per call (again, roughly consistent with Kaya Guides and Step-By-Step, accounts for other time spent on that participant like notes & chat reviews)
Broke down cost per treatment starter
Kept cost per install at $0.02–0.10, since it’s consistent with real-world data
Set a discount rate at 12.34% (~5–20%) of installers starting the programme, consistent with Kaya Guides’ observations
I decided to extrapolate this to unguided, in absence of better data
I think this fairly accounts for everything you raised. I think you’re right to point out that my model should’ve accounted for the cost of a treatment starter (~8x higher). But I don’t think it’s right to only account for active users, since Kaya Guides spend 15 minutes of staff time on 12% of all installers, even if they drop out later. And as their ad targeting gets better, we’d only expect this number to increase, paradoxically widening the cost gap!
Plugging it all in, unguided has 7% (15–22%) of the cost of guided at scale.
The fact that it took over a month to find some pretty obvious flaws in my model is a concern, and my model is clearly somewhat sensitive to the parameters. However, even if I’m really pessimistic about the parameters, I can’t get it above 20% of the cost of guided, which would still make it more cost-effective.
The bigger doubt I’ve had since writing this report is learning from Kaya Guides that they actually do have an unguided condition—anyone who scores 0–9 on the PHQ-9 (no/mild depression), or anyone who scores above but explicitly doesn’t want a guide gets the ordinary programme, just without the calls. This has an astonishing 0% completion rate. I think the different subgroup, programme design, and lack of focus are mostly contributing to this, but it indicates that it’s gonna be hard to keep users engaged. I’ll chat with them some more and see if I can learn anything else.
Awesome great post, fantastic to see variations of this intervention being considered. The main concerns we focused on with unguided vs guided delivery methods for self-help were recruitment costs and retention.
As you show in the relative engagement in recent trials retention seems to play out in favour of unguided given the lower costs. I think this is a fair read, from what I’ve seen from comparing the net effect size of these interventions the no less than half figure is about right too.
For recruitment, given the average rate of download for apps we discounted organic growth as you have. Then the figure cited for cost per install is correct (can see another source here) but for general installs. For the unguided self-help intervention, we want a subset of people with the/a condition we are targeting (~5-15%). We then want to turn those installs into active engaged users of the program. I don’t know how much these would increase the costs compared to the $0.02 - $0.10 figure. Plausibly 6-20 times for targeting the subgroup with the condition for installs (although even the general population may benefit a bit from the intervention). Then the cost of turning those installs into active users. Some stats suggest that on day one there is a 23.01% retention rate and then as low as 2.59% by day 30 (source, similar to your 3.3% real-world app data). So again looking at maybe a 4 to 40 times increase in recruitment costs depending. Overall that could increase costs for recruiting an active user by 24 to 800 times (Or $0.48 to $8).
Although these would be identical across unguided vs guided the main consideration then becomes does it costs more to recruit more users or to guide existing users and increase the follow-through/ effect size. Adding the above factors quickly to your CEA for recruitment cost gives a cost per active user of 9.6 (1.6 to 32) and cost-effectiveness for a guided of mean 24 (12 to 47) and unguided of mean 21 (4.8 to 66). Taking all of this at face value which version looks more cost-effective will depend a lot on exactly where and how the intervention is being employed. Even then there is uncertainty with some of these figures in a real-world setting.
For the unguided app model you outline I agree if successful this would be incredibly cost-effective. Although at present I’d still be uncertain which version would look best. Ultimately that’s up to groups implementing this family of interventions, like Kaya Guides, to explore through implementing and experimenting.
Thank you so much! Your criticism has helped me identify a few mistakes, and I think can get us closer to clarity. The main difference between our models is around who counts as a ‘beneficiary’, or what it means to ‘recruit’ someone.
The main thing I want to focus on is that you’re predicting a cost per beneficiary that would be nearly 50% recruitment. I don’t think that passes the smell test. The main difference is you’re only counting the staff time for active participants, but even with modest dropout, we’d expect the vast majority of staff time to go to users who only complete one or two calls. But you’re right to point out that we should factor in dropout between installation and the first guidance call, and when I factor this in, unguided has 7% of the cost of guided at scale.
The rest of this comment is just my working out.
Mistakes I made
One of the mistakes I made was having different definitions for recruitment for each condition in my cost model. If the number in that model says there are 100,000 beneficiaries, in the unguided model this means we got 100,000 installs, but in the guided model it means we got 100,000 participants who had 50–180 minutes of staff time allocated to them. Obviously there are different costs to recruit these kinds of participants.
(Two other mistakes I found: I shouldn’t have multiplied the unguided recruitment cost by 2 to account for engagement differences, and I forgot to discount the office space costs for the nonexistent guides in the unguided model)
I should’ve been clearer about a ‘beneficiary’
To rectify this, let’s count a ‘beneficiary’ as someone who is in the targeted subgroup and completes pre-treatment. This is in line with most of the literature, which counts ‘dropout’ regardless of whether users complete any of the material, so long as they’ve done their induction. We don’t want to filter this down to ‘active’ users, since users who drop out will still incur costs.
We have some facts from Kaya Guides:
They spent US$105 and got 875 initial user interactions for a cost per interaction of $0.12 (this isn’t quite cost per install but it’s close enough)
Let’s consider this a lower bound, Meta’s targeting should get cheaper at scale and after optimisation on Kaya Guides’ end. I think it could get down to the $0.02–0.10 figure pretty easily (and it may have already been there if a number of interactions didn’t lead to installs)
82.65% of users who completed their depression questionnaire scored above threshold
This should also be a lower bound. Kaya Guides used basic ads, but Meta is able to optimise for ‘app events’ (such as scoring highly on this questionnaire) so should be better than this at targeting for depression (scary!)
12.34% (108) of these initially interacting users scheduled and picked up an initial call
Their programmes involve an initial call, a second call at 3 days (I confirmed this directly), and then 5–8 weeks of subsequent calls. All calls are 15 minutes.
Updating my model
I’ve updated my cost model:
Split office space costs between conditions
Removed the recruitment cost doubling in unguided
Broke down staff time per participant
1–10 calls per participant (lognormal), this is roughly what Kaya Guides and Step-By-Step do. I distributed it as a lognormal to quickly account for dropout rates, but the model is somewhat sensitive to this parameter. More knowledge on its distribution could change the overall calculus.
15–40 minutes of time spent per call (again, roughly consistent with Kaya Guides and Step-By-Step, accounts for other time spent on that participant like notes & chat reviews)
Broke down cost per treatment starter
Kept cost per install at $0.02–0.10, since it’s consistent with real-world data
Set a discount rate at 12.34% (~5–20%) of installers starting the programme, consistent with Kaya Guides’ observations
I decided to extrapolate this to unguided, in absence of better data
I think this fairly accounts for everything you raised. I think you’re right to point out that my model should’ve accounted for the cost of a treatment starter (~8x higher). But I don’t think it’s right to only account for active users, since Kaya Guides spend 15 minutes of staff time on 12% of all installers, even if they drop out later. And as their ad targeting gets better, we’d only expect this number to increase, paradoxically widening the cost gap!
Plugging it all in, unguided has 7% (15–22%) of the cost of guided at scale.
Earlier, I also sense-checked with Kaya Guides’ direct cost-per-beneficiary, which they estimate to be $3.93. If the unguided cost per beneficiary is $0.41 (as in the updated model), then the limiting proportion increases a bit to 11%.
My doubts
The fact that it took over a month to find some pretty obvious flaws in my model is a concern, and my model is clearly somewhat sensitive to the parameters. However, even if I’m really pessimistic about the parameters, I can’t get it above 20% of the cost of guided, which would still make it more cost-effective.
The bigger doubt I’ve had since writing this report is learning from Kaya Guides that they actually do have an unguided condition—anyone who scores 0–9 on the PHQ-9 (no/mild depression), or anyone who scores above but explicitly doesn’t want a guide gets the ordinary programme, just without the calls. This has an astonishing 0% completion rate. I think the different subgroup, programme design, and lack of focus are mostly contributing to this, but it indicates that it’s gonna be hard to keep users engaged. I’ll chat with them some more and see if I can learn anything else.