I’m personally excited about more work in the EA space on topics around mental health and subjective well-being, and was initially excited to see StrongMinds (SM) come so strongly recommended. I do have a few Qs about the incredible success the pilots have shown so far:[1]
I couldn’t find number needed to treat (NNT)[2] figures anywhere (please let me know if I’ve missed this!), so I’ve had a rough go based on the published results, and came to an NNT of around 1.35.[3] Limitations of the research aside, this suggests StrongMinds is among the most effective interventions in all of medicine in terms of achieving its stated goals.
If later RCTs and replications showed much higher NNT figures, what do you think would be the most likely reason for this? For comparison:
This meta-analysis suggests a response rate of 41% and an NNT of 4 when comparing therapy to ‘waitlist’ conditions (and lower when only considering IPT in subgroup analyses); or
this meta-analysis which suggests an NNT of 7 when comparing psychotherapies to placebo pill.
Admittedly, there are many caveats here—the various linked studies aren’t a perfect comparison to SM’s work, NNT clearly shouldn’t be used as sole basis for comparison between interventions, and I haven’t done enough work here to feel super confident about the quality of SM’s research. But my initial reaction upon skimming and seeing response to treatment in the range of 94-99%, or 100+ people with PHQ-9 scores of over 15 basically all dropping down to 1-4[5] (edit: an average improvement of 12 points after conclusion of therapy) after 12-16 weeks of group IGT by lay counsellors was that this seemed far “too good to be true”, and fairly incongruent with ~everything I’ve learnt or anecdotally seen in clinical practice about the effectiveness of mental health treatments (though clearly I could be wrong!). This is especially surprising given SM dropped the group of participants with minimal or mild depression from the analysis.[6]
Were these concerns ever raised by the researchers when writing up the reports? Do you have any reason to believe that the Ugandan context or something about the SM methodology makes your intervention many times more effective than basically any other intervention for depression? [Edit: I note that the 99% figure in the phase 2 trial was disregarded, but the 94% figure in phase 1 trial wasn’t, despite presumably the same methodology? Also curious about the separate analysis that came to 92%, which states: “Since this impact figure was collected at a regular IPT group meeting, as had been done bi-weekly throughout the 12- week intervention, it is unlikely that any bias influenced the figure.” I don’t quite understand how collection at a regular IPT group meeting makes bias unlikely—could you clarify this? Presumably participants knew in advance how many weeks the intervention would be?]
How did you come to the 10% figure when adjusting for social desirability bias?
Was there a reason an RCT couldn’t have been done as a pilot? Just noting that “informal control populations were established for both Phase 1 and Phase 2 patients, consisting of women who screened for depression but did not participate”, and the control group in both the pilots were only 36 people, compared to the 244 and 270 in the treatment arm for phase 1 and phase 2 respectively. As a result, 11 / 24 of the villages where the interventions took place did not have a control arm at all. (pg 9)
Are you happy to go into a bit more detail about the background of the lay counsellors? E.g. what they know prior to the SM pilots, how much training (in number of hours) they receive, and who runs it (what relevant qualifications / background? How did the trainers get their IPT-G certification—e.g. is this a postgrad psychology qualification, or a one-off training course?) I briefly skimmed the text (appendix A + E) but also got a bit confused over the difference between “lay counsellor”, “mental health facilitator”, “mental health supervisor” and “senior technical advisor” and how they’re relevant for the intervention.
Can you give us a cost breakdown of $170 / person figure for delivering the programme (Or $134 for 2021)? See Joel’s response and subsequent discussion for more details. Specifically, whether the methodology for working out the cost / client by dividing total clients reached over SM’s total expenses means that this includes the clients reached by the partners, but not their operating costs / expenses. For example, ~48% of clients were treated through partners in 2021, and Q2 results (pg 2)suggest StrongMinds is on track for ~79% of clients treated through partners in 2022.[7] Or are all expenses of SM partners covered by SM and included in the tax returns?
In the most recent publication (pg 5), published 2017, the report says: “Looking forward, StrongMinds will continue to strengthen our evaluation efforts and will continue to follow up with patients at 6 or 12 month intervals. We also remain committed to implementing a much more rigorous study, in the form of an externally-led, longitudinal randomized control trial, in the coming years.”
Have either the follow-up or the externally-led longitudinal RCT happened yet? If so, are the results shareable with the public? (I note that there has been a qualitative study done on a teletherapy version published in 2021, but no RCT.)
The pivot to teletherapy in light of COVID makes sense, though the evidence-base for its effectiveness is ?presumably weaker.
What’s the breakdown of % clients reached via teletherapy versus clients reached via group IGT as per the original pilots (i.e. in person)
In the 2021 report on a qualitative assessment of teletherapy (pg 2), it says: “Data from StrongMinds shows that phone-based IPT-G is as effective as in-person group therapy in reducing depression symptoms among participants”. Is this research + methodology available to the public? (I searched for phone and telehealth in the other 4 reports which returned no hits)
Does StrongMinds have any other unpublished research?
What’s the plan with telehealth going forward? Was this a temporary thing for COVID, or is this a pivot into a more / similarly effective approach?
I also saw in the HLI report that SM defines treated patients treated for purpose of cost analysis as “attending more than six sessions (out of 12) for face-to-face modes and more than four (out of 8) for teletherapy.”—is this also the definition for the treatment outcomes? i.e. how did SM assess the effectiveness of SM for people who attended 7 sessions and then dropped out? Do we have more details around about how many people didn’t do all sessions, how they responded, and how this was incorporated into SM’s analyses?
Thanks again!
(Commenting in personal capacity etc)
[Edited after Joel’s response to include Q7, Q8, and an update to Q1c and Q5, mainly to put all the unresolved Qs in one place for Sean and other readers’ convenience.]
Apologies in advance if I’ve missed anything—I’ve only briefly skimmed your website’s publications, and I haven’t engaged with this literature for quite a while now!
SM’s results of 95% depression-free (85% after the 10% adjustment for social desirability bias) give an EER of 0.15 after adjustment. By a more conservative estimate, based on this quote (pg 3): “A separate control group, which consisted of depressed women who received no treatment, experienced a reduction of depressive symptoms in only 11% of members over the same 12-week intervention period” and assuming all of those are clinically significant reductions in depressive symptoms, the CER is 0.89, which gives an NNT of 1 / (0.89 − 0.15) = 1.35. The EER can be adjusted upwards because not all who started in the treatment group were depressed, but this is only 2% and 6% for phase 1 and 2 respectively—so in any case the NNT is unlikely to go much higher than 1.5 even by the most conservative estimate.
They also concluded: “We did not find convincing evidence supporting or refuting the effect of interpersonal psychotherapy or psychodynamic therapy compared with ‘treatment as usual’ for patients with major depressive disorder. The potential beneficial effect seems small and effects on major outcomes are unknown. Randomized trials with low risk of systematic errors and low risk of random errors are needed.”
A total of 56 participants with Minimal or Mild Depression (anyone with total raw scores between 1-9) at baseline in both the treatment intervention (46 participants) and control (10 participants) groups were dropped from the GEE analysis of determining the depression reduction impact. In typical practice around the world, individuals with Minimal/Mild Depression are not considered for inclusion in group therapy because their depressive symptoms are relatively insignificant. StrongMinds consciously included these Minimal/Mild cases in Phase Two because these patients indicated suicidal thoughts in their PHQ-9 evaluation. However, their removal from the GEE analysis serves to ensure that the Impact Evaluation is not artificially inflated, since reducing the depressive symptoms of Minimal/Mild Depressive cases is generally easier to do.
Thanks for these questions! I appreciate the time you took to really look at our data, and I think some of the questions will help us ponder what we need to be looking at next within StrongMinds. Please note, as the Founder, I’m not a researcher or clinician by trade, so my answers may not be as granular as you would hope, but I’ll do my best to respond. I’m going to tackle the first four tonight.
a. The NNTs could be different due to a variety of theoretical factors. NNTs are really only as good and accurate as the data provided. We are planning an RCT for 2023, so hopefully, we will have more to consider and dig into once those numbers are available. b/c. Yes, the numbers are surprising. As I’ve mentioned in some of the other responses, I largely think the success has to do with the interpersonal nature of IPT and how it works within the community-centered culture of Uganda and Zambia.
I believe the 10% figure is part of the HLI analysis, and I am not an expert on that, so I’ll let their team speak to that or let the numbers speak for themselves.
An RCT certainly could have been done as a pilot, but they are pretty costly to complete, and at the time, StrongMinds was just finding its footing and searching for funders. I was working as a volunteer at StrongMinds for the first 18 months of operation.
Our facilitators are all lay community members. In Uganda, 100% of our facilitators are Ugandan. In Zambia, they are all Zambian. They don’t need to have a college degree in Psychology; in fact, they don’t even need to have a high school degree. The most critical factor when we are interviewing potential facilitators is their empathy level. We have found that the higher their empathy level, the better they are as facilitators.
No worries! I should say that I’ve spent less than 3 hours looking through SM/HLI documents around this, so I’m not highly confident about most of these points. I have a lot of respect for anyone who is trying their best to make as much impact in the world as they can—thank you for all your work thus far, and thank you for engaging with all the questions!
I should also clarify that the digging was largely prompted by HLI’s strong endorsements:
We’re now in a position to confidently recommend StrongMinds as the most effective way we know of to help other people with your money.
And while this was a result of over 3 years and 10,000 hours of work, I generally aim to be more hesitant to take such strong claims at face value.
But I mention this because I want to emphasise that even if it’s the case that after this conversation, I decide that I’m not personally quite convinced that StrongMinds is the single most cost effective way to help other people, it doesn’t mean this is a reflection of the effort you have and continue to put into SM! It doesn’t necessarily mean SM isn’t a great charity. It doesn’t mean it’s not possible for StrongMinds to be the best charity in the future, or be the best under different philosophical assumptions. It’s just really hard to be the most cost effective charity.
And I’m mindful that this conversation has been possible precisely because of your shared commitment to transparency and willingness to engage, which I have a lot of respect for. We are both on the same team of wanting to do as much good as we can, and I hope you interpret this barrage (apologies!) of questions in that light.
Lastly, I’m also happy to continue via email and update folks later with a summary, if you think that would be helpful for getting answers that you may not be able to answer immediately etc.
With that in mind, some followups:
1) Just re-flagging the question RE: bias, though as you pointed out, this may be better suited for a researcher on the team / someone who was more in-the-weeds with the research:
a) What’s the justification behind the 94% figure not being found to be invalid when the 99% was? Was it based on different methodology between the two pilots, or something else? As far as I can tell, the difference in methodology RE: recording PHQ-9 scores was that in phase 1 these were scored weekly from week 5-16, with a post-assessment scoring at week 17, and for phase 2, these were done biweekly from week 2-12, with a post-assessment at week 14. It’s not clear that this difference leads to bias in one but not the other.
b) Also curious about the separate analysis that came to 92%, which states: “Since this impact figure was collected at a regular IPT group meeting, as had been done bi-weekly throughout the 12- week intervention, it is unlikely that any bias influenced the figure.” I don’t quite understand how collection at a regular IPT group meeting makes bias unlikely—could you clarify this? Presumably participants knew in advance how many weeks the intervention would be?
2) I took the 10% from StrongMinds’ 2017 report (pg 2), not an HLI analysis (though if HLI independently came to that conclusion or have reviewed it and agreed I’d be interested too):
While both the Phase 1 and 2 patients had 95% depression-free rates at the completion of formal sessions, our Impact Evaluation reports and subsequent experience has helped us to understand that those rates were somewhat inflated by social desirability bias, roughly by a factor of approximately ten percentage points. This was due to the fact that their Mental Health Facilitator administered the PHQ-9 at the conclusion of therapy. StrongMinds now uses external data collectors to conduct the post-treatment evaluations. Thus, for effective purposes, StrongMinds believes the actual depression-free rates for Phase 1 and 2 to be more in the range of 85%.
[emphasis added]
I couldn’t find a justification of this figure in that report or any of the preceding reports. (admittedly I just very quickly searched for various combinations of 10/85/95% and didn’t read the entire report)
3) Makes sense—looking forward to the results of the RCT! I assume it will be independent and pre-registered?
4) Thanks! Just looked in a bit more detail—in Appendix A (pg 30) it says:
“Use of lay community workers as the IPT-G facilitators, or Mental Health Facilitators(MHFs)
MHFs require at least a high-school diploma, and are employed and salaried by StrongMinds. They receive two weeks of training by a certified IPT-G expert and receive on-going supervision and guidance by a mental health professional. Since they are community members themselves, they are well-received by the depressed patients. The IPT-G training curriculum includes modules on mental illness in general, depression, interpersonal psychotherapy, management of suicidality, and the goals and objectives for each weekly session of the 16 total sessions that are held. The training extensively uses role-playing to recreate group meeting settings.”
“StrongMinds completed training its initial cadre of four MHFs in March, 2014. The training lasted 10 days and was conducted by two Ugandans certified in IPT-G by Columbia University. In addition, the training was monitored long-distance via Skype by our senior technical advisor who is an international expert on IPT-G from Columbia University.”
“In Phase One of the 2014-2015 pilot in Uganda, our 4 MHFs were supervised by the two Ugandan IPT-G experts noted above. In Phase Two, StrongMinds hired a full-time Mental Health Supervisor (MHS) who both conducted IPT groups and supervised the 4 MHFs. This MHS was actually a member of the 2002 RCT in Uganda and has over ten years of IPT-G experience”
a) Just confirming that “lay counsellor” is the same as “IPT-G facilitator” and “mental health facilitator”? If not, what are the differences? How much training do they get and what’s their role in the intervention etc.
b) How does StrongMinds select for empathy? E.g. questionnaire, interview, etc.
c) What does it mean to be a “certified IPT-G expert”? For example, it sounds like there are a lot of various levels of certification. From a quick google the best match I found for the description of the training was this, which suggests a “certified IPT-G expert” is someone who has completed this specific 6-day course, i.e., with a level A certification? (Happy to be corrected—just couldn’t find any details of this). If true, am I understanding correctly that the StrongMinds lay counsellors / mental health facilitators take a 10 day training course which is delivered by someone who has taken a 6-day course? Do the certified IPT-G experts play a role in SM other than the initial training of lay counsellors?
d) What does it mean to be a “mental health supervisor”? What’s their role in SM?
e) [Minor] I just realised the appendix said MHFs require at least a high school diploma, contra what you said earlier: “in fact, they don’t even need to have a high school degree”. I assume this was just a more recent change in policy. Not a big deal, just clarifying.
5) I had another question, which came up as I was going through the tax returns Joel linked to work out the cost per client reached—in the tax return it says
STRONGMINDS IS A SOCIAL ENTERPRISE THAT PROVIDES LIFECHANGING DEPRESSION TREATMENT TO LOW-INCOME INDIVIDUALS IN SUB-SAHARAN AFRICA.
This didn’t exist in the tax returns before 2019, but came up every year from 2019 onwards.
a) Was there a change in model in terms of revenue streams or business model for StrongMinds—if so, what changed?
b) You’ll probably cover this in some of the other questions, but how do the partnerships work? Do partners pay you for the year of training? What does this training look like?
c) Are there other revenue streams that StrongMinds have outside of donors / grants? (To be clear—I don’t have an issue with StrongMinds being a social enterprise, just wanting to clarify so I have the facts right!)
Thank you! I appreciate your curiosity, and I’m not put off by the questions or anything; it’s just many of them are not in my area of expertise, and this happens to be a pretty busy time of year at StrongMinds. It may take some time to fully gather what you’re asking for. We aren’t a large research institute by any means, so our clinical team is relatively small. Additionally, some of the work you are referencing is nearly a decade old, so we have shifted some of the ways we operate to be more effective or better based on our learnings. That said, I will dig back in when I can to help answer your additional questions via email or direct message.
To answer the remaining four from your original note to close the loop:
5) Since HLI generated the $170 figure, they have the best information on that particular breakdown, but I am collecting the most recent info on our CPP for another question, and I will share that with you later this week when I have the updated numbers.
6) As mentioned above, we are currently in the process of assessing the right questions and framework for an RCT looking at the results and impact of our therapy model. We are hoping to be able to launch the RCT late in 2023.
7) We switched our model to teletherapy to continue to treat clients during the pandemic lockdowns. It was not ideal, but we wanted to continue reaching as many women as possible despite the challenges. Though it proved tricky in some cases to reach our target demographic, we did find that some women preferred the flexibility teletherapy offered them. For the most part, we have switched back to our original model, but we still see some groups via teletherapy in Uganda. All research shared publicly from our initial year using teletherapy can be found here.
8) We track individuals that attend most of their therapy sessions, as we saw that the effects of therapy were still statistically significant and that attending additional sessions did not produce incremental impact. Due to the individual roles and responsibilities of the women that attend, it’s sometimes challenging for them to make it to all 12 sessions.
I can maybe help with question 5, since the $170 figure originates from my analysis.
I finalized the cost figures during COVID when their cost figures were very high ($400 per person). I tried to project what they’d be over the next 3 years (starting in 2020) and assumed it’d come down, but the costs have come down faster than I imagined. They now say they expect 2022 to cost 105 USD per person treated.
They regularly update their cost and expense figures in their quarterly reports.
And here’s the general breakdown of their expenses according to their 2021 tax returns (page 10).
RE: Q5 - sorry, just to clarify, I was interested in a breakdown of the $170 figure (or the 109 / 134/ 79 figure in the cost-per-patient graph). What does it consist of?
StrongMinds records the average cost of providing treatment to an additional person (i.e. total annual expenses / no. treated) and has shared the most recent figures for each programme with us.
But I’m interested in something more fine-grained than “total annual expenses, or even “program service expenses” (per tax returns). e.g.: $A to train lay counsellors $B / hour for facilitators * number of hours $C operating costs for StrongMinds (SM) $D for outreach to SM partners $E for SM partner operating costs etc
I’m mindful this is asking a lot of info, sorry! I just assumed it’d be readily available, but it looks like you’ve just deferred to SM here.
I had a very brief look through the tax returns—per the tax returns you linked, the total expenses for 2021 come to 5,186,778. Per the quarterly report you linked, the total clients reached in 2021 was 42482. This means the $ per client figure should be $122? But that’s not the $134 figure reported, so I’m probably doing something wrong here.
Also, ~48% of clients were treated through partners in 2021, but does the methodology of working out cost effectiveness by dividing clients reached by SM expenses include expenses and operating costs of the partners? Q2 results (pg 2)suggest StrongMinds is on track for ~79% of clients treated through partners in 2022. If the expenses of the partners aren’t covered by SM but the clients reached are then this will make SM look more cost-effective than it is.
I also saw in the HLI report that SM defines treated patients treated here as “attending more than six sessions (out of 12) for face-to-face modes”—is this also the definition for the treatment? i.e. how did the pilot assess the effectiveness of SM for people who attended 7 sessions and then dropped out?
Do you know the answers to the other Qs too? If so, I’d be interested in your take as well! But also no worries if you prefer to leave it to Sean (I’ve edited the comment above to incorporate these Qs).
Sorry if I missed it, I just ctrl+F’ed 170 in the forum post you linked which didn’t give me a result, so I skimmed section 5 in the full HLI report. I also looked at the Q report and the tax returns but it doesn’t quite answer the question.
But I’m interested in something more fine-grained than “total annual expenses, or even “program service expenses” (per tax returns). e.g.: $A to train lay counsellors $B / hour for facilitators * number of hours $C operating costs for StrongMinds (SM) $D for outreach to SM partners $E for SM partner operating costs etc
Unfortunately, I don’t know if I can share any information beyond the pie chart I shared above. So I’ll leave that for StrongMinds.
Also, ~48% of clients were treated through partners in 2021, but does the methodology of working out cost effectiveness by dividing clients reached by SM expenses include expenses and operating costs of the partners?
We did our analysis before they shifted models, so we hadn’t incorporated this. I don’t think StrongMinds includes partner costs. This will be something we revisit when we update our StrongMinds CEA (expected in 2023).
I see this as more of a concern for counterfactual impact. Where I see it as “StrongMinds got these organizations to do IPT-g, how much better is this than what they’d otherwise be doing?” But maybe I’m thinking about this wrong.
Many thanks for doing this AMA!
I’m personally excited about more work in the EA space on topics around mental health and subjective well-being, and was initially excited to see StrongMinds (SM) come so strongly recommended. I do have a few Qs about the incredible success the pilots have shown so far:[1]
I couldn’t find number needed to treat (NNT)[2] figures anywhere (please let me know if I’ve missed this!), so I’ve had a rough go based on the published results, and came to an NNT of around 1.35.[3] Limitations of the research aside, this suggests StrongMinds is among the most effective interventions in all of medicine in terms of achieving its stated goals.
If later RCTs and replications showed much higher NNT figures, what do you think would be the most likely reason for this? For comparison:
This meta-analysis suggests an NNT of 3 when comparing IPT to a control condition;
This systematic review suggests an NNT of 4 for interpersonal therapy (IPT) compared to treatment as usual[4];
This meta-analysis suggests a response rate of 41% and an NNT of 4 when comparing therapy to ‘waitlist’ conditions (and lower when only considering IPT in subgroup analyses); or
this meta-analysis which suggests an NNT of 7 when comparing psychotherapies to placebo pill.
Admittedly, there are many caveats here—the various linked studies aren’t a perfect comparison to SM’s work, NNT clearly shouldn’t be used as sole basis for comparison between interventions, and I haven’t done enough work here to feel super confident about the quality of SM’s research. But my initial reaction upon skimming and seeing response to treatment in the range of 94-99%, or 100+ people with PHQ-9 scores of over 15 basically all dropping down to 1-4[5] (edit: an average improvement of 12 points after conclusion of therapy) after 12-16 weeks of group IGT by lay counsellors was that this seemed far “too good to be true”, and fairly incongruent with ~everything I’ve learnt or anecdotally seen in clinical practice about the effectiveness of mental health treatments (though clearly I could be wrong!). This is especially surprising given SM dropped the group of participants with minimal or mild depression from the analysis.[6]
Were these concerns ever raised by the researchers when writing up the reports? Do you have any reason to believe that the Ugandan context or something about the SM methodology makes your intervention many times more effective than basically any other intervention for depression?
[Edit: I note that the 99% figure in the phase 2 trial was disregarded, but the 94% figure in phase 1 trial wasn’t, despite presumably the same methodology? Also curious about the separate analysis that came to 92%, which states: “Since this impact figure was collected at a regular IPT group meeting, as had been done bi-weekly throughout the 12- week intervention, it is unlikely that any bias influenced the figure.” I don’t quite understand how collection at a regular IPT group meeting makes bias unlikely—could you clarify this? Presumably participants knew in advance how many weeks the intervention would be?]
How did you come to the 10% figure when adjusting for social desirability bias?
Was there a reason an RCT couldn’t have been done as a pilot? Just noting that “informal control populations were established for both Phase 1 and Phase 2 patients, consisting of women who screened for depression but did not participate”, and the control group in both the pilots were only 36 people, compared to the 244 and 270 in the treatment arm for phase 1 and phase 2 respectively. As a result, 11 / 24 of the villages where the interventions took place did not have a control arm at all. (pg 9)
Are you happy to go into a bit more detail about the background of the lay counsellors? E.g. what they know prior to the SM pilots, how much training (in number of hours) they receive, and who runs it (what relevant qualifications / background? How did the trainers get their IPT-G certification—e.g. is this a postgrad psychology qualification, or a one-off training course?) I briefly skimmed the text (appendix A + E) but also got a bit confused over the difference between “lay counsellor”, “mental health facilitator”, “mental health supervisor” and “senior technical advisor” and how they’re relevant for the intervention.
Can you give us a cost breakdown of $170 / person figure for delivering the programme (Or $134 for 2021)? See Joel’s response and subsequent discussion for more details. Specifically, whether the methodology for working out the cost / client by dividing total clients reached over SM’s total expenses means that this includes the clients reached by the partners, but not their operating costs / expenses. For example, ~48% of clients were treated through partners in 2021, and Q2 results (pg 2) suggest StrongMinds is on track for ~79% of clients treated through partners in 2022.[7] Or are all expenses of SM partners covered by SM and included in the tax returns?
In the most recent publication (pg 5), published 2017, the report says: “Looking forward, StrongMinds will continue to strengthen our evaluation efforts and will continue to follow up with patients at 6 or 12 month intervals. We also remain committed to implementing a much more rigorous study, in the form of an externally-led, longitudinal randomized control trial, in the coming years.”
Have either the follow-up or the externally-led longitudinal RCT happened yet? If so, are the results shareable with the public? (I note that there has been a qualitative study done on a teletherapy version published in 2021, but no RCT.)
The pivot to teletherapy in light of COVID makes sense, though the evidence-base for its effectiveness is ?presumably weaker.
What’s the breakdown of % clients reached via teletherapy versus clients reached via group IGT as per the original pilots (i.e. in person)
In the 2021 report on a qualitative assessment of teletherapy (pg 2), it says: “Data from StrongMinds shows that phone-based IPT-G is as effective as in-person group therapy in reducing depression symptoms among participants”. Is this research + methodology available to the public? (I searched for phone and telehealth in the other 4 reports which returned no hits)
Does StrongMinds have any other unpublished research?
What’s the plan with telehealth going forward? Was this a temporary thing for COVID, or is this a pivot into a more / similarly effective approach?
I also saw in the HLI report that SM defines treated patients treated for purpose of cost analysis as “attending more than six sessions (out of 12) for face-to-face modes and more than four (out of 8) for teletherapy.”—is this also the definition for the treatment outcomes? i.e. how did SM assess the effectiveness of SM for people who attended 7 sessions and then dropped out? Do we have more details around about how many people didn’t do all sessions, how they responded, and how this was incorporated into SM’s analyses?
Thanks again!
(Commenting in personal capacity etc)
[Edited after Joel’s response to include Q7, Q8, and an update to Q1c and Q5, mainly to put all the unresolved Qs in one place for Sean and other readers’ convenience.]
[Edited to add this disclaimer.]
[Edited to include a link to a newer post StrongMinds should not be a top-rated charity (yet), which includes additional discussion.]
Apologies in advance if I’ve missed anything—I’ve only briefly skimmed your website’s publications, and I haven’t engaged with this literature for quite a while now!
Quick primer on NNT for other readers. Lower = better, where NNT = 1 means your treatment gets the desired effect 100% of the time.
SM’s results of 95% depression-free (85% after the 10% adjustment for social desirability bias) give an EER of 0.15 after adjustment. By a more conservative estimate, based on this quote (pg 3): “A separate control group, which consisted of depressed women who received no treatment, experienced a reduction of depressive symptoms in only 11% of members over the same 12-week intervention period” and assuming all of those are clinically significant reductions in depressive symptoms, the CER is 0.89, which gives an NNT of 1 / (0.89 − 0.15) = 1.35. The EER can be adjusted upwards because not all who started in the treatment group were depressed, but this is only 2% and 6% for phase 1 and 2 respectively—so in any case the NNT is unlikely to go much higher than 1.5 even by the most conservative estimate.
They also concluded: “We did not find convincing evidence supporting or refuting the effect of interpersonal psychotherapy or psychodynamic therapy compared with ‘treatment as usual’ for patients with major depressive disorder. The potential beneficial effect seems small and effects on major outcomes are unknown. Randomized trials with low risk of systematic errors and low risk of random errors are needed.”
See Appendix B, pg 30. for more context about what the PHQ-9 scoring is like.
As pointed out in the report (pg 9):
% clients reached by partners:
20392 / 42482 in 2021
33148 / (33148+8823) in 2022
Thanks for these questions! I appreciate the time you took to really look at our data, and I think some of the questions will help us ponder what we need to be looking at next within StrongMinds. Please note, as the Founder, I’m not a researcher or clinician by trade, so my answers may not be as granular as you would hope, but I’ll do my best to respond. I’m going to tackle the first four tonight.
a. The NNTs could be different due to a variety of theoretical factors. NNTs are really only as good and accurate as the data provided. We are planning an RCT for 2023, so hopefully, we will have more to consider and dig into once those numbers are available. b/c. Yes, the numbers are surprising. As I’ve mentioned in some of the other responses, I largely think the success has to do with the interpersonal nature of IPT and how it works within the community-centered culture of Uganda and Zambia.
I believe the 10% figure is part of the HLI analysis, and I am not an expert on that, so I’ll let their team speak to that or let the numbers speak for themselves.
An RCT certainly could have been done as a pilot, but they are pretty costly to complete, and at the time, StrongMinds was just finding its footing and searching for funders. I was working as a volunteer at StrongMinds for the first 18 months of operation.
Our facilitators are all lay community members. In Uganda, 100% of our facilitators are Ugandan. In Zambia, they are all Zambian. They don’t need to have a college degree in Psychology; in fact, they don’t even need to have a high school degree. The most critical factor when we are interviewing potential facilitators is their empathy level. We have found that the higher their empathy level, the better they are as facilitators.
No worries! I should say that I’ve spent less than 3 hours looking through SM/HLI documents around this, so I’m not highly confident about most of these points. I have a lot of respect for anyone who is trying their best to make as much impact in the world as they can—thank you for all your work thus far, and thank you for engaging with all the questions!
I should also clarify that the digging was largely prompted by HLI’s strong endorsements:
And while this was a result of over 3 years and 10,000 hours of work, I generally aim to be more hesitant to take such strong claims at face value.
But I mention this because I want to emphasise that even if it’s the case that after this conversation, I decide that I’m not personally quite convinced that StrongMinds is the single most cost effective way to help other people, it doesn’t mean this is a reflection of the effort you have and continue to put into SM! It doesn’t necessarily mean SM isn’t a great charity. It doesn’t mean it’s not possible for StrongMinds to be the best charity in the future, or be the best under different philosophical assumptions. It’s just really hard to be the most cost effective charity.
And I’m mindful that this conversation has been possible precisely because of your shared commitment to transparency and willingness to engage, which I have a lot of respect for. We are both on the same team of wanting to do as much good as we can, and I hope you interpret this barrage (apologies!) of questions in that light.
Lastly, I’m also happy to continue via email and update folks later with a summary, if you think that would be helpful for getting answers that you may not be able to answer immediately etc.
With that in mind, some followups:
1) Just re-flagging the question RE: bias, though as you pointed out, this may be better suited for a researcher on the team / someone who was more in-the-weeds with the research:
a) What’s the justification behind the 94% figure not being found to be invalid when the 99% was? Was it based on different methodology between the two pilots, or something else? As far as I can tell, the difference in methodology RE: recording PHQ-9 scores was that in phase 1 these were scored weekly from week 5-16, with a post-assessment scoring at week 17, and for phase 2, these were done biweekly from week 2-12, with a post-assessment at week 14. It’s not clear that this difference leads to bias in one but not the other.
b) Also curious about the separate analysis that came to 92%, which states: “Since this impact figure was collected at a regular IPT group meeting, as had been done bi-weekly throughout the 12- week intervention, it is unlikely that any bias influenced the figure.” I don’t quite understand how collection at a regular IPT group meeting makes bias unlikely—could you clarify this? Presumably participants knew in advance how many weeks the intervention would be?
2) I took the 10% from StrongMinds’ 2017 report (pg 2), not an HLI analysis (though if HLI independently came to that conclusion or have reviewed it and agreed I’d be interested too):
I couldn’t find a justification of this figure in that report or any of the preceding reports. (admittedly I just very quickly searched for various combinations of 10/85/95% and didn’t read the entire report)
3) Makes sense—looking forward to the results of the RCT! I assume it will be independent and pre-registered?
4) Thanks! Just looked in a bit more detail—in Appendix A (pg 30) it says:
“Use of lay community workers as the IPT-G facilitators, or Mental Health Facilitators(MHFs)
MHFs require at least a high-school diploma, and are employed and salaried by StrongMinds. They receive two weeks of training by a certified IPT-G expert and receive on-going supervision and guidance by a mental health professional. Since they are community members themselves, they are well-received by the depressed patients. The IPT-G training curriculum includes modules on mental illness in general, depression, interpersonal psychotherapy, management of suicidality, and the goals and objectives for each weekly session of the 16 total sessions that are held. The training extensively uses role-playing to recreate group meeting settings.”
In Appendix E (pg 33) it says:
“StrongMinds completed training its initial cadre of four MHFs in March, 2014. The training lasted 10 days and was conducted by two Ugandans certified in IPT-G by Columbia University. In addition, the training was monitored long-distance via Skype by our senior technical advisor who is an international expert on IPT-G from Columbia University.”
“In Phase One of the 2014-2015 pilot in Uganda, our 4 MHFs were supervised by the two Ugandan IPT-G experts noted above. In Phase Two, StrongMinds hired a full-time Mental Health Supervisor (MHS) who both conducted IPT groups and supervised the 4 MHFs. This MHS was actually a member of the 2002 RCT in Uganda and has over ten years of IPT-G experience”
a) Just confirming that “lay counsellor” is the same as “IPT-G facilitator” and “mental health facilitator”? If not, what are the differences? How much training do they get and what’s their role in the intervention etc.
b) How does StrongMinds select for empathy? E.g. questionnaire, interview, etc.
c) What does it mean to be a “certified IPT-G expert”? For example, it sounds like there are a lot of various levels of certification. From a quick google the best match I found for the description of the training was this, which suggests a “certified IPT-G expert” is someone who has completed this specific 6-day course, i.e., with a level A certification? (Happy to be corrected—just couldn’t find any details of this). If true, am I understanding correctly that the StrongMinds lay counsellors / mental health facilitators take a 10 day training course which is delivered by someone who has taken a 6-day course? Do the certified IPT-G experts play a role in SM other than the initial training of lay counsellors?
d) What does it mean to be a “mental health supervisor”? What’s their role in SM?
e) [Minor] I just realised the appendix said MHFs require at least a high school diploma, contra what you said earlier: “in fact, they don’t even need to have a high school degree”. I assume this was just a more recent change in policy. Not a big deal, just clarifying.
5) I had another question, which came up as I was going through the tax returns Joel linked to work out the cost per client reached—in the tax return it says
This didn’t exist in the tax returns before 2019, but came up every year from 2019 onwards.
a) Was there a change in model in terms of revenue streams or business model for StrongMinds—if so, what changed?
b) You’ll probably cover this in some of the other questions, but how do the partnerships work? Do partners pay you for the year of training? What does this training look like?
c) Are there other revenue streams that StrongMinds have outside of donors / grants? (To be clear—I don’t have an issue with StrongMinds being a social enterprise, just wanting to clarify so I have the facts right!)
(commenting in personal capacity etc)
Thank you! I appreciate your curiosity, and I’m not put off by the questions or anything; it’s just many of them are not in my area of expertise, and this happens to be a pretty busy time of year at StrongMinds. It may take some time to fully gather what you’re asking for. We aren’t a large research institute by any means, so our clinical team is relatively small. Additionally, some of the work you are referencing is nearly a decade old, so we have shifted some of the ways we operate to be more effective or better based on our learnings. That said, I will dig back in when I can to help answer your additional questions via email or direct message.
To answer the remaining four from your original note to close the loop:
5) Since HLI generated the $170 figure, they have the best information on that particular breakdown, but I am collecting the most recent info on our CPP for another question, and I will share that with you later this week when I have the updated numbers.
6) As mentioned above, we are currently in the process of assessing the right questions and framework for an RCT looking at the results and impact of our therapy model. We are hoping to be able to launch the RCT late in 2023.
7) We switched our model to teletherapy to continue to treat clients during the pandemic lockdowns. It was not ideal, but we wanted to continue reaching as many women as possible despite the challenges. Though it proved tricky in some cases to reach our target demographic, we did find that some women preferred the flexibility teletherapy offered them. For the most part, we have switched back to our original model, but we still see some groups via teletherapy in Uganda. All research shared publicly from our initial year using teletherapy can be found here.
8) We track individuals that attend most of their therapy sessions, as we saw that the effects of therapy were still statistically significant and that attending additional sessions did not produce incremental impact. Due to the individual roles and responsibilities of the women that attend, it’s sometimes challenging for them to make it to all 12 sessions.
Thanks again for the questions!
I can maybe help with question 5, since the $170 figure originates from my analysis.
I finalized the cost figures during COVID when their cost figures were very high ($400 per person). I tried to project what they’d be over the next 3 years (starting in 2020) and assumed it’d come down, but the costs have come down faster than I imagined. They now say they expect 2022 to cost 105 USD per person treated.
They regularly update their cost and expense figures in their quarterly reports.
And here’s the general breakdown of their expenses according to their 2021 tax returns (page 10).
Thanks for this Joel!
RE: Q5 - sorry, just to clarify, I was interested in a breakdown of the $170 figure (or the 109 / 134/ 79 figure in the cost-per-patient graph). What does it consist of?
On skimming the HLI report it says: [1]
But I’m interested in something more fine-grained than “total annual expenses, or even “program service expenses” (per tax returns). e.g.:
$A to train lay counsellors
$B / hour for facilitators * number of hours
$C operating costs for StrongMinds (SM)
$D for outreach to SM partners
$E for SM partner operating costs
etc
I’m mindful this is asking a lot of info, sorry! I just assumed it’d be readily available, but it looks like you’ve just deferred to SM here.
I had a very brief look through the tax returns—per the tax returns you linked, the total expenses for 2021 come to 5,186,778. Per the quarterly report you linked, the total clients reached in 2021 was 42482. This means the $ per client figure should be $122? But that’s not the $134 figure reported, so I’m probably doing something wrong here.
Also, ~48% of clients were treated through partners in 2021, but does the methodology of working out cost effectiveness by dividing clients reached by SM expenses include expenses and operating costs of the partners? Q2 results (pg 2) suggest StrongMinds is on track for ~79% of clients treated through partners in 2022. If the expenses of the partners aren’t covered by SM but the clients reached are then this will make SM look more cost-effective than it is.
I also saw in the HLI report that SM defines treated patients treated here as “attending more than six sessions (out of 12) for face-to-face modes”—is this also the definition for the treatment? i.e. how did the pilot assess the effectiveness of SM for people who attended 7 sessions and then dropped out?
Do you know the answers to the other Qs too? If so, I’d be interested in your take as well! But also no worries if you prefer to leave it to Sean (I’ve edited the comment above to incorporate these Qs).
Sorry if I missed it, I just ctrl+F’ed 170 in the forum post you linked which didn’t give me a result, so I skimmed section 5 in the full HLI report. I also looked at the Q report and the tax returns but it doesn’t quite answer the question.
Unfortunately, I don’t know if I can share any information beyond the pie chart I shared above. So I’ll leave that for StrongMinds.
We did our analysis before they shifted models, so we hadn’t incorporated this. I don’t think StrongMinds includes partner costs. This will be something we revisit when we update our StrongMinds CEA (expected in 2023).
I see this as more of a concern for counterfactual impact. Where I see it as “StrongMinds got these organizations to do IPT-g, how much better is this than what they’d otherwise be doing?” But maybe I’m thinking about this wrong.