Pilots typically are meant to indicate whether the intervention may have potential, mainly in terms of feasibility; ours certainly isn’t the definitive assessment of its causal effect. For this we will need to run an RCT. I intended it to be clear from the post that there was no control group but rereading the executive summary, I can see that indeed this was not clear in this first section given that I mention estimated effect size. I have revised accordingly, thanks for pointing this out. We decided not to have a control group for the initial pilot given the added logistics and timeline as well as it being so early on with a lot of things not figured out yet. I’ve removed the how to donate section from the summary section to avoid the impression that is the purpose of this post, as it is not. The spontaneous remission seen in the WHO RCT is noted in “reasons to be skeptical,” but I’ve added this to the executive summary as well for clarity. There’s a lot to consider regarding the finding of a 50% spontaneous remission rate in high-income countries (this post does a good deep-dive into the complexities https://forum.effectivealtruism.org/posts/qFQ2b4zKiPRcKo5nn/strongminds-4-of-9-psychotherapy-s-impact-may-be-shorter), but it’s important to note that the landscape for mental healthcare is quite different in high-income contexts compared to LMIC contexts; people in high-income countries have alternative options for care, whereas our participants are unlikely to get any other form of help.
On the second point, it’s certainly possible that many people stopped engaging because they were not seeing improvements. I have shared the feedback we have so far. We are continuing to collect feedback from partial completers to learn more about their experiences and their reasons for deciding not to continue. It’s important to also understand the experiences of program completers and if/how they’ve benefited from the program, so we’ve shared the feedback.
On your third point, the justification is in the section “2025 Projected Cost-Effectiveness.” The figure is based on the cost-effectiveness estimate on the WHO RCT’s effect size and our projected budget for next year.
Regarding Joel’s assessment, Joel has said his availability doesn’t allow for a formalized public-facing assessment at this time, but the Happier Lives Institute is doing a much more in-depth analysis that they’ve said they aim to publish in 2024.
Thanks for making these changes and responding to my concerns! Also great to hear that HLI is doing a more in-depth analysis, that will be exciting to read.
With regards to the projections, it seems to me you just made up the number 10 000 participants? As in, there is no justification for why you chose this value. Perhaps I am missing something here, but it feels like without further context this projection is pretty meaningless.
My guess is that a WhatsApp-based MH intervention would be almost arbitrarily scalable. 10 000 participants ($300,000) may reflect the scale of the grants they are looking for.
I’ll rewrite completely because I didn’t explain myself very clearly
10,000 participants is possible since they are using Whatsapp, in a large country, and recruiting users does not seem to be a bottleneck
10,000 participants is relevant as it represents the scale they might hope to expand to at the next stage
Presumably they used the number 10,000 to estimate the cost-per-treatment by finding the marginal cost per treatment and adding 1⁄10,000th of their expected fixed costs.
So if they were to expand to 100,000 or 1,000,000 participants, the cost-per-treatment would be even lower.
I hope this is not what is happening. It’s at best naive. This assumes no issues will crop up during scaling, that “fixed” costs are indeed fixed (they rarely are) and that the marginal cost per treatment will fall (this is a reasonable first approximation, but it’s by no means guaranteed). A maximally optimistic estimate IMO. I don’t think one should claim future improvements in cost effectiveness when there are so many incredibly uncertain parameters in play.
My concrete suggestion would be to rather write something like: “We hope to reach 10 000 participants next year with our current infrastructure, which might further improve our cost-effectiveness.”
Hi Håkon, thank you for these questions!
Pilots typically are meant to indicate whether the intervention may have potential, mainly in terms of feasibility; ours certainly isn’t the definitive assessment of its causal effect. For this we will need to run an RCT. I intended it to be clear from the post that there was no control group but rereading the executive summary, I can see that indeed this was not clear in this first section given that I mention estimated effect size. I have revised accordingly, thanks for pointing this out. We decided not to have a control group for the initial pilot given the added logistics and timeline as well as it being so early on with a lot of things not figured out yet. I’ve removed the how to donate section from the summary section to avoid the impression that is the purpose of this post, as it is not. The spontaneous remission seen in the WHO RCT is noted in “reasons to be skeptical,” but I’ve added this to the executive summary as well for clarity. There’s a lot to consider regarding the finding of a 50% spontaneous remission rate in high-income countries (this post does a good deep-dive into the complexities https://forum.effectivealtruism.org/posts/qFQ2b4zKiPRcKo5nn/strongminds-4-of-9-psychotherapy-s-impact-may-be-shorter), but it’s important to note that the landscape for mental healthcare is quite different in high-income contexts compared to LMIC contexts; people in high-income countries have alternative options for care, whereas our participants are unlikely to get any other form of help.
On the second point, it’s certainly possible that many people stopped engaging because they were not seeing improvements. I have shared the feedback we have so far. We are continuing to collect feedback from partial completers to learn more about their experiences and their reasons for deciding not to continue. It’s important to also understand the experiences of program completers and if/how they’ve benefited from the program, so we’ve shared the feedback.
On your third point, the justification is in the section “2025 Projected Cost-Effectiveness.” The figure is based on the cost-effectiveness estimate on the WHO RCT’s effect size and our projected budget for next year.
Regarding Joel’s assessment, Joel has said his availability doesn’t allow for a formalized public-facing assessment at this time, but the Happier Lives Institute is doing a much more in-depth analysis that they’ve said they aim to publish in 2024.
Thanks again for the critical read and input!
Thanks for making these changes and responding to my concerns!
Also great to hear that HLI is doing a more in-depth analysis, that will be exciting to read.
With regards to the projections, it seems to me you just made up the number 10 000 participants? As in, there is no justification for why you chose this value. Perhaps I am missing something here, but it feels like without further context this projection is pretty meaningless.
My guess is that a WhatsApp-based MH intervention would be almost arbitrarily scalable. 10 000 participants ($300,000) may reflect the scale of the grants they are looking for.
I don’t understand what you are saying here, could you elaborate?
I’ll rewrite completely because I didn’t explain myself very clearly
10,000 participants is possible since they are using Whatsapp, in a large country, and recruiting users does not seem to be a bottleneck
10,000 participants is relevant as it represents the scale they might hope to expand to at the next stage
Presumably they used the number 10,000 to estimate the cost-per-treatment by finding the marginal cost per treatment and adding 1⁄10,000th of their expected fixed costs.
So if they were to expand to 100,000 or 1,000,000 participants, the cost-per-treatment would be even lower.
I hope this is not what is happening. It’s at best naive. This assumes no issues will crop up during scaling, that “fixed” costs are indeed fixed (they rarely are) and that the marginal cost per treatment will fall (this is a reasonable first approximation, but it’s by no means guaranteed). A maximally optimistic estimate IMO. I don’t think one should claim future improvements in cost effectiveness when there are so many incredibly uncertain parameters in play.
My concrete suggestion would be to rather write something like: “We hope to reach 10 000 participants next year with our current infrastructure, which might further improve our cost-effectiveness.”