I run another EA mental health charity. Here are my hastily scribbled thoughts:
Firstly, why did you opt to not have a control group?
When psychotherapy interventions fail, it’s usually not because they don’t reduce symptoms. They fail by failing to generate supply / demand cost-effectively enough, finding pilot and middle stage funding, finding a scalable marketing channel, or some other logistical issue.
Given that failing to reduce symptoms is not that bigger risk, we and every other EA mental health startup I can name did not use a control group for our pilots. Doing so would increase the cost of recruitment by ~10x and the cost of the pilot by ~30% or so.
The #1 reason is that so long as you’re using an evidence-based intervention, cost explains most of the variance in cost-effectiveness.
Secondly, isn’t it a massive problem that you only look at the 27% that completed the program when presenting results? You write that you got some feedback on why people were not completing the program unrelated to depression, but I think it’s more than plausible that many of the dropouts dropped out because they were depressed and saw no improvement
It’s also possible that they started feeling better and they didn’t need it any more. IMO, this is a little tangential because most dropout isn’t much to do with symptom reduction, it’s more to do with:
1 - (A lack of) Trust in and rapport with the therapist
2 - Not enjoying the process
3 - Not having faith it will work for them
4 - Missing a few sessions out of inconvenience and losing the desire to continue
It’s somewhat analogous to an online educational course. You probably aren’t dropping out because you aren’t learning fast enough; it’s probably that you don’t enjoy it or life got in the way so you put it on the back-burner
...[likely] many of the dropouts dropped out because they were depressed and saw no improvement. This choice makes stating things like “96% of program completers said they were likely or very likely to recommend the program” at best uninformative.
This is good point. These statistics are indeed uninformative, but it’s also not clear what better one would be. We use “mean session rating” and get >9/10, which I perceive as unrealistically high. Presumably, this would have gotten around the completer problem (as we’re sampling after every session and we include dropouts in our analysis), but it doesn’t seem it to have. I think it might be because both our services are free, and people don’t like to disparage free services unless they REALLY suck.
Thanks for this thorough and thoughtful response John!
I think most of this makes sense. I agree that if you are using an evidence based-intervention, it might not make sense to increase the cost by adding a control group. I would for instance not think of this as a big issue for bednet distribution in an area broadly similar to other areas bednet distribution works. Given that in this case they are simply implementing a programme from WHO with two positive RCTs (which I have not read), it seems reasonable to do an uncontrolled pilot.
I pushed back a little in a comment from you further down, but I think this point largely addresses my concerns there.
With regards to your explanations for why people drop out, I would argue that at least 1,2 and 3 are in fact because of the ineffectiveness of the intervention, but it’s mostly a semantic discussion.
The two RCTs cited seem to be about displaced Syrians, which makes me uncomfortable straightforwardly assuming it will transfer to the context in India. I would also add that there is a big difference between the evidence base for ITN distribution compared to this intervention. I look forward to seeing what the results are in the future!
Specifically on the cited RCTs, the Step-By-Step intervention has been specifically designed to be adaptable across multiple countries & cultures[1][2][3][4][5]. Although they initially focused on displaced Syrians, they have also expanded to locals in Lebanon across multiple studies[6][7][8] and found no statistically significant differences in effect sizes[8:1] (the latter is one of the studies cited in the OP). Given this, I would be default surprised if the intervention, when adapted, failed to produce similar results in new contexts.
I share your concerns and in our org at least we haven’t improved cost-effectiveness with scale. I think tech orgs though can sometimes be different as Stan said. Even with tech scaling though, ncreases in management staff especially can be a big source of extra costs.
I run another EA mental health charity. Here are my hastily scribbled thoughts:
When psychotherapy interventions fail, it’s usually not because they don’t reduce symptoms. They fail by failing to generate supply / demand cost-effectively enough, finding pilot and middle stage funding, finding a scalable marketing channel, or some other logistical issue.
Given that failing to reduce symptoms is not that bigger risk, we and every other EA mental health startup I can name did not use a control group for our pilots. Doing so would increase the cost of recruitment by ~10x and the cost of the pilot by ~30% or so.
The #1 reason is that so long as you’re using an evidence-based intervention, cost explains most of the variance in cost-effectiveness.
It’s also possible that they started feeling better and they didn’t need it any more. IMO, this is a little tangential because most dropout isn’t much to do with symptom reduction, it’s more to do with:
1 - (A lack of) Trust in and rapport with the therapist
2 - Not enjoying the process
3 - Not having faith it will work for them
4 - Missing a few sessions out of inconvenience and losing the desire to continue
It’s somewhat analogous to an online educational course. You probably aren’t dropping out because you aren’t learning fast enough; it’s probably that you don’t enjoy it or life got in the way so you put it on the back-burner
This is good point. These statistics are indeed uninformative, but it’s also not clear what better one would be. We use “mean session rating” and get >9/10, which I perceive as unrealistically high. Presumably, this would have gotten around the completer problem (as we’re sampling after every session and we include dropouts in our analysis), but it doesn’t seem it to have. I think it might be because both our services are free, and people don’t like to disparage free services unless they REALLY suck.
Thanks for this thorough and thoughtful response John!
I think most of this makes sense. I agree that if you are using an evidence based-intervention, it might not make sense to increase the cost by adding a control group. I would for instance not think of this as a big issue for bednet distribution in an area broadly similar to other areas bednet distribution works. Given that in this case they are simply implementing a programme from WHO with two positive RCTs (which I have not read), it seems reasonable to do an uncontrolled pilot.
I pushed back a little in a comment from you further down, but I think this point largely addresses my concerns there.
With regards to your explanations for why people drop out, I would argue that at least 1,2 and 3 are in fact because of the ineffectiveness of the intervention, but it’s mostly a semantic discussion.
The two RCTs cited seem to be about displaced Syrians, which makes me uncomfortable straightforwardly assuming it will transfer to the context in India. I would also add that there is a big difference between the evidence base for ITN distribution compared to this intervention. I look forward to seeing what the results are in the future!
Specifically on the cited RCTs, the Step-By-Step intervention has been specifically designed to be adaptable across multiple countries & cultures[1][2][3][4][5]. Although they initially focused on displaced Syrians, they have also expanded to locals in Lebanon across multiple studies[6][7][8] and found no statistically significant differences in effect sizes[8:1] (the latter is one of the studies cited in the OP). Given this, I would be default surprised if the intervention, when adapted, failed to produce similar results in new contexts.
Carswell, Kenneth et al. (2018) Step-by-Step: a new WHO digital mental health intervention for depression, mHealth, vol. 4, pp. 34–34.
Sijbrandij, Marit et al. (2017) Strengthening mental health care systems for Syrian refugees in Europe and the Middle East: integrating scalable psychological interventions in eight countries, European Journal of Psychotraumatology, vol. 8, p. 1388102.
Burchert, Sebastian et al. (2019) User-Centered App Adaptation of a Low-Intensity E-Mental Health Intervention for Syrian Refugees, Frontiers in Psychiatry, vol. 9, p. 663.
Abi Ramia, J. et al. (2018) Community cognitive interviewing to inform local adaptations of an e-mental health intervention in Lebanon, Global Mental Health, vol. 5, p. e39.
Woodward, Aniek et al. (2023) Scalability of digital psychological innovations for refugees: A comparative analysis in Egypt, Germany, and Sweden, SSM—Mental Health, vol. 4, p. 100231.
Cuijpers, Pim et al. (2022) Guided digital health intervention for depression in Lebanon: randomised trial, Evidence Based Mental Health, vol. 25, pp. e34–e40.
Abi Ramia, Jinane et al. (2024) Feasibility and uptake of a digital mental health intervention for depression among Lebanese and Syrian displaced people in Lebanon: a qualitative study, Frontiers in Public Health, vol. 11, p. 1293187.
Heim, Eva et al. (2021) Step-by-step: Feasibility randomised controlled trial of a mobile-based intervention for depression among populations affected by adversity in Lebanon, Internet Interventions, vol. 24, p. 100380.
Very interesting, thanks for highlighting this!
I share your concerns and in our org at least we haven’t improved cost-effectiveness with scale. I think tech orgs though can sometimes be different as Stan said. Even with tech scaling though, ncreases in management staff especially can be a big source of extra costs.