An evaluation of Mind Ease, an anti-anxiety app
Cross posted from my blog https://hauke.substack.com/p/mindease
Luke Muehlhauser recently published “EA needs consultancies”. This is an example of an EA consulting project. If you’re interested in consulting services from me, get in touch at: H@EA.do
Acknowledgements
This work was funded by the Total Portfolio Project. Thanks to Derek Foster from Rethink Priorities and the Total Portfolio Project for their help in improving this manuscript. All opinions are mine.
Our process
This report was written in roughly 20 days of desk research in early 2020. The evaluation was partly based on a prototype beta version of Mind Ease and pilot data on the effectiveness of it dating back even further. Other parts of our analysis assume a more idealized and further optimized version of Mind Ease. Mind Ease’s team is constantly working to improve the app and has for instance added an additional intervention since this review was conducted and made lots of other improvements. Since the review, the Mind Ease team reports to have “additional evidence that was not evaluated in this review, in particular: (1) lots of app usage data from our actual users with information about how much they improve, and (2) a randomized controlled trial that we conducted early on in the life of the project that we used to narrow down our set of planned interventions to use in the app.”
Executive Summary
This report evaluates the social impact of Mind Ease, an app to reduce anxiety. The main goal of this report is to estimate the number of disability adjusted life years (DALYs) averted per user. This is to inform a separate analysis on whether to make an impact investment.
Our key question here was: how large of a counterfactual social impact does Mind Ease have?
To answer this, we tackled the following four related questions:
As a cause area, is anxiety a large and neglected problem?
Are Mind Ease exercises based on evidence-based psychotherapeutic interventions that are effective in an offline context?
Is there evidence from the scientific literature that anxiety can be effectively reduced if these offline psychotherapeutic interventions are ported over to mental health apps?
Is there evidence that Mind Ease in particular is effective in reducing anxiety?
In chapter 1, we briefly consider anxiety as a cause area. In brief, we find that the scale of untreated anxiety is relatively big and neglected.
We find that more than 1% of all ill-health and death—27 million disability adjusted life years—were caused by anxiety. This is similar to the global burden of violence. Globally, 284 million people—3.8% of all people—have anxiety disorders. Other estimates suggest that this might be even higher: according to the CDC, 11% of U.S. adults report regular feelings of worry, nervousness, or anxiety and ~19% had any anxiety disorder in the past year according to the NIH and Anxiety and Depression Association of America.
Mental health in general has been described as a “truly neglected area of global health policy”. The net present cost of scaling up anxiety treatment to an adequate level from 2016–30 has been estimated to be $56 billion. This could perhaps be seen as the overall value of Mind Ease’s market size that could be “disrupted”. The expected returns of scaled-up treatment would be 6 million extra years of healthy life valued at $52 billion. Given that this is the case, the average cost-effectiveness of anxiety treatment might be quite high already (roughly $10k per QALY gained=$56 billion / 6 million years of healthy life), it is plausible that there are even more highly cost-effective interventions to treat anxiety. Scaled-up treatment might also increase labour productivity by $169 billion, leading to benefit cost ratios 3 (or 4, when the value of health returns is also included).
In chapter 2, we review the scientific literature on the psychotherapeutic interventions Mind Ease tries to reduce anxiety with (e.g. Cognitive Therapy, Progressive Muscle Relaxation).
On the whole, Mind Ease’s current ten interventions seem carefully selected to be evidence-based. Though the scientific literature in clinical psychology often has poor methodology, many of Mind Ease’s interventions are backed up by relatively substantial evidence from systematic reviews and meta-analyses, and, where those are unavailable, plausible other evidence exists that suggest that the interventions are beneficial. The interventions usually have small to moderate effects in an offline therapeutic context, but these effect sizes might be reduced when the interventions are implemented in an app.
In chapter 3, we review this literature on mobile health (mHealth), a burgeoning field in health where apps are used to treat illness effectively at scale. In particular, we focus on mental mHealth to treat anxiety. We find that there is some emerging evidence that the effects of the interventions reviewed in the previous chapter transfer to an online context and that apps can reduce anxiety, but likely with a smaller effect size. We find that there is some evidence from systematic reviews and meta-analyses that apps, especially evidence-based apps like Mind Ease, can indeed reduce anxiety, but the effect sizes will likely be smaller than in an offline context.
In chapter 4, we evaluate direct evidence for Mind Ease’s efficacy such as user reports, internal self-evaluation, and their internal study. We find that results from internal studies are promising and perhaps hint at Mind Ease being effective to reduce anxiety. However, there are several methodological concerns and only one study outcome is useful for comparing it to other health interventions. We suggest an ideal trial to test Mind Ease’s effectiveness going forward. Taken together with user reports and the evidence reviewed in the previous chapters, it is plausible that the usage of an optimized future version of Mind Ease might significantly reduce anxiety for some people—also with non-trivial effect sizes. We also review the evidence of competitor apps and find they might also be quite effective at reducing anxiety, which could reduce Mind Ease’s counterfactual impact.
Chapter 1-4 served as priors for chapter 5, where we analyse the cost-effectiveness of Mind Ease in terms of quality adjusted life years gained (or disability adjusted life year averted) per additional user.
In a final literature review we find that, generally, studies often show that mental health apps have very high cost-effectiveness. This is due to their zero marginal cost per user, compared to the high cost of conventional psychotherapy, which is highly-skilled labor intensive. We also review what the size of the quality of life gains / reductions in disability weights of reducing anxiety in other studies are. We use these to feed them into our cost-effectiveness analysis.
Ultimately, we find that the long-term benefits of anxiety reduction of consistent long-term usage of an optimized future version of Mind Ease targeted at moderately anxious populations could conceivably have a benefit of 1 DALY averted per user. However, due to some inherent uncertainty in the data, estimated conservatively it might be as low as 0.002 DALY averted per user or as high as 6.11 DALY averted per user in an optimistic scenario. To counterfactually adjust for the fact that users might find other treatment, through apps or other forms of therapy, our best guess is that Mind Ease counterfactually averts 0.25 DALY per user. However, again due to the inherent uncertainty in the data, the counterfactual might be as low as 0.0001 DALY per user under conservative assumptions, and 4.07 DALY per user averted under optimistic assumptions.
For a crude comparison, the Against Malaria Foundation has a cost-effectiveness of roughly $50 per DALY averted (this figure might be somewhat out of date, but should be roughly correct). This is because a death of an under 5-year-old is equivalent to ~34 Years of Life lost (YLL) per AMF death. One of the most effective global health charities—the Deworm the World Initiative—roughly averts a DALY equivalent for $14, while GiveDirectly, the philanthropic benchmark averts a DALY equivalent for roughly $860. This means that if Mind Ease can reach ~4 users for less than $50 (or $12.5 per user), and avert 0.25 DALYs in each, then it could be as cost-effective as AMF. The promise of mobileHealth (mHealth) is that at scale apps often have ‘zero marginal cost’ per user (much less than $12.50) and so plausibly are very cost-effective. One can calculate the cost-effectiveness of an impact investment differently, and this conservative “societal perspective” includes the cost to the user and assumes a philanthropic subsidy. In contrast, if investing in Mind Ease returns profits above market rate in expectation, if we only look at the cost to the government and the philanthropist, the cost-effectiveness of Mind Ease might even be “negative”, i.e. they might save users, the government, and the philanthropist money.
We close with some qualitative arguments in the form of crucial considerations. For instance, funding Mind Ease might have benefits beyond its direct impact, through the value of improved global mental health and the value of information of researching what works in mental mHealth. We provide some toy models showing that Mind Ease’s research might plausibly have even higher cost-effectiveness than its direct impact if it were to improve the field of anxiety treatment as a whole.
Full Report
Please read the full report here.
Further reading
Founders Pledge Mental Health Cause Area Report
Anxiety apps: Can you lessen anxiety by playing a game on your phone?
I’m really pleased to see this: I have been wondering how one would do an EA-minded evaluation of the cost-effectiveness of a start-up that runs it head to head with things like AMF. I’m particularly pleased to see an analysis of a mental health product.*
I only have one comment. You say:
It doesn’t seem quite that tech products have zero marginal cost. Shouldn’t one include the cost of acquiring (and supporting?) a user, e.g. through advertising? This has a cost, and this cost would need to be lower than $12.50 per user, given your other assumptions. I have no idea what user acquisition costs are and if $12.5 is high or low.
*(Semi-obligatory disclaimer: Peter Brietbart, MindEase’s CEO, is the chair of the board of trustees for HLI, the organisation I run)
Cheers- thanks for the comment!
I’m using the term zero marginal cost colloquially as is common parlance in the tech sector.
Your app might spread through word of mouth, the server costs are trivial and then you can scale at ~zero marginal cost.
As you say in practise tech firms often spend a few dollars on acquiring new users/customers.
I could believe that mHealth/iCBT/online therapy is very cost-effective, but the claims here seem to be about Mind Ease in particular. As such the meat of the report is likely to be found in chapter 4, the actual evaluation of Mind Ease – but this section is inaccessible.
Is there a reason this was omitted from the public report?
Just re Anxiety prevalence: It seems to me that Anxiety would be a kind of continuum, and you may be able to say 50% of people are suffering from anxiety or 5%, depending on where you make the cutoff. Your description implicitly seems to support exactly this view (“Globally, 284 million people—3.8% of all people—have anxiety disorders. Other estimates suggest that this might be even higher: according to the CDC, 11% of U.S. adults report regular feelings of worry, nervousness, or anxiety and ~19% had any anxiety disorder in the past year according to the NIH and Anxiety and Depression Association of America.”), plus maybe that the Anxiety America guys like to quote impressive numbers for their domain. ⇒ Could be useful if you found more tangible ways to express what’s going on anxiety-wise in how many heads.
Yes, excellent point- I go into more detail about this in the full report:
“Anxiety is a highly prevalent condition, with lifetime rates for its derived mental disorders between 14.5% and 33.7% in Western countries (Alonso and Lepine, 2007; Kessler et al., 2012), and global estimates across countries between 3.8% to 25.0% (Remes et al., 2016).
Many more might have trait social anxiety which is not quite clinical yet still causes suffering. Indeed, trait social anxiety may have evolved to protect our ancestors from social threat. Similarly, generalized anxiety might have evolved to protect us from other threats. Thus, anxiety might be natural and very widespread.”
https://docs.google.com/document/d/1Y0Mc0pI-pDMQMPg8M4F0zA1KYiXuvW5q7MPXRH9sX7k/edit#bookmark=id.h2x1ikourvk6
I’ve heard a lot about this anxiety-fighting app, but haven’t had a chance to try it yet. For the last couple of years I have been suffering from constant bouts of anxiety and anxiety. Previously, this manifested itself in the fact that it was constantly difficult for me to breathe, there were signs of depression and apathy. I did not know what it could be and started looking on the Internet. I stumbled upon one site https://fherehab.com/anxiety/treatment/ and read there that I most likely have anxiety attacks. I decided to turn to a specialist and I was prescribed a treatment that soon helped me.