Is mindfulness good for you?

Mind­ful­ness med­i­ta­tion seems to be quite pop­u­lar in the EA com­mu­nity and in ad­ja­cent com­mu­ni­ties. Many EAs I meet seem to be in­ter­ested in med­i­ta­tion or ac­tively prac­tic­ing it, and there has been some writ­ten dis­cus­sion of its value by EAs.[1]

I per­son­ally find mind­ful­ness use­ful for re­duc­ing ru­mi­na­tion. Its main value is re­veal­ing that your mind is ba­si­cally go­ing com­pletely ba­nanas all the time, that pat­terns of thought emerge which are difficult to con­trol. I also find that lov­ing and kind­ness med­i­ta­tion im­proves my mood in the short-term. How­ever, I think the strength of the ev­i­dence on the benefits of med­i­ta­tion is of­ten over­stated by EAs and quasi-fel­low trav­el­lers like Sam Har­ris.

Here, I dis­cuss the over­all strength of the ev­i­dence on med­i­ta­tion as a treat­ment for anx­iety and de­pres­sion.

1. What is mind­ful­ness?

With its roots in Bud­dhism, at­ten­tion to­wards mind­ful­ness has grown enor­mously since the early 2000s.[2] Two com­monly stud­ies forms of mind­ful­ness are Mind­ful­ness-Based Stress Re­duc­tion and Mind­ful­ness-Based Cog­ni­tive Ther­apy. Mind­ful­ness-Based Stress Re­duc­tion is an 8-week group-based pro­gram con­sist­ing of:

● 20-26 hours of for­mal med­i­ta­tion prac­tice, in­clud­ing:

○ Weekly 2 to 2.5 hour sessions

○ A whole-day re­treat (6 hours)

○ Home prac­tice ~45 mins per day for 6 days a week.[3]

Mind­ful­ness-Based Cog­ni­tive Ther­apy in­cor­po­rates cog­ni­tive ther­apy into the ses­sions. Mind­ful­ness-Based Stress Re­duc­tion can be led by laypeo­ple, whereas Mind­ful­ness-Based Cog­ni­tive Ther­apy must be led by a li­censed health care provider.

In my ex­pe­rience, most peo­ple prac­tis­ing mind­ful­ness use an app such as Headspace or the Sam Har­ris Wak­ing Up app. I per­son­ally find the Wak­ing Up app far su­pe­rior to the Headspace app.

2. How over-op­ti­mistic should we ex­pect the ev­i­dence to be?

Mind­ful­ness has many fea­tures that should make us sus­pect that the strength of the ev­i­dence claimed in the liter­a­ture is over­stated:

● A form of psy­chol­ogy/​so­cial psy­chol­ogy research

● Most out­come met­rics are sub­jec­tive

● Many of those re­search­ing it seem to be true be­liev­ers[4]

● Hints of re­li­gion, al­ter­na­tive medicine, and woo

Re­search fields with these fea­tures are ripe for repli­ca­tion crisis trauma. We should ex­pect in­flated claims of im­pact which are then brought back to Earth by repli­ca­tions or fur­ther anal­y­sis of the ex­ist­ing re­search.

3. Main prob­lems with the evidence

3.1. Re­port­ing bias

Re­port­ing bias in­cludes:

  1. Study pub­li­ca­tion bias - the pub­li­ca­tion of sig­nifi­cant re­sults and the failure to pub­lish in­signifi­cant results

  2. Selec­tive out­come re­port­ing bias, in which out­comes pub­lished are cho­sen based on statis­ti­cal sig­nifi­cance with non-sig­nifi­cant out­comes not pub­lished.

  3. Selec­tive anal­y­sis re­port­ing bias, in which data are an­a­lyzed with mul­ti­ple meth­ods but are re­ported only for those that pro­duce pos­i­tive re­sults.

  4. Other bi­ases, such as rel­e­ga­tion of non-sig­nifi­cant pri­mary out­comes to sec­ondary sta­tus when re­sults are pub­lished.

There is good ev­i­dence of re­port­ing bias in mind­ful­ness re­search.

Coron­ado-Mon­toya et al (2016) test for re­port­ing bias by es­ti­mat­ing the ex­pected num­ber of pos­i­tive tri­als that mind­ful­ness-based ther­apy would have pro­duced if its effect size were the same as in­di­vi­d­ual psy­chother­apy for de­pres­sion, d = 0.55.[5] As we will see be­low, this is very likely an over­es­ti­mate of the true effect of mind­ful­ness-based ther­apy, and there­fore the method used un­der­states re­port­ing bias in mind­ful­ness stud­ies.

Of the 124 RCTs in­cluded in Coron­ado-Mon­toya et al’s (2016) study, 108 (87%) were clas­sified as pos­i­tive and 16 (13%) as nega­tive. If the true effect size of mind­ful­ness-based ther­apy was d = 0.55, then we would ex­pect 68 of 124 stud­ies to be pos­i­tive, rather than 108, mean­ing that the ra­tio of ob­served to ex­pected pos­i­tive stud­ies was 1.6.[6] This is clear ev­i­dence of re­port­ing bias.

More­over, Coron­ado-Mon­toya et al (2016) also looked at 21 tri­als that were pre-reg­istered. Of these, none speci­fied which vari­able would be used to de­ter­mine suc­cess, and 13 (62%) were still un­pub­lished 30 months af­ter the trial was com­pleted.[7]

A re­cent Na­ture pa­per found that in psy­chol­ogy, due to se­lec­tive re­port­ing, meta-analy­ses pro­duce sig­nifi­cantly differ­ent effect sizes to large-scale pre-reg­istered repli­ca­tions in 12 out of 15 cases. Where there was a differ­ence, on av­er­age, the effect size in the meta-anal­y­sis was 3 times larger than the repli­ca­tions.[8] This shows that re­port­ing bias is usu­ally not ad­e­quately cor­rected for in meta-analy­ses.

3.2. Effect size of med­i­ta­tion com­pared to other interventions

Goyal et al con­ducted a meta-anal­y­sis of the effect of mind­ful­ness-based ther­apy for well-be­ing.[9] They key facts are:

Effect size

○ Co­hen’s d rang­ing from 0.22 to 0.38 for anx­iety symp­toms.[10]

○ 0.23 to 0.30 for de­pres­sive symp­toms.[11]

○ Th­ese were each usu­ally com­pared to a non­spe­cific ac­tive con­trol.

○ How­ever, nei­ther of these es­ti­mates cor­rect for re­port­ing bias.[12] I think it is plau­si­ble that this bi­ases the es­ti­mate of the effect size up­wards by a fac­tor of 2 to 3.

Com­par­i­son to al­ter­na­tive treatments

■ In the 20 RCTs ex­am­in­ing com­par­a­tive effec­tive­ness, mind­ful­ness and mantra pro­grams did not show sig­nifi­cant effects when the com­para­tor was a known treat­ment or ther­apy.[13]

■ Sam­ple sizes in the com­par­a­tive effec­tive­ness tri­als were small (mean size of 37 per group), and none was ad­e­quately pow­ered to as­sess non­in­fe­ri­or­ity or equiv­alence.[14]

Antidepressants

● Ac­cord­ing to a re­cent meta-anal­y­sis, an­tide­pres­sants have an effect size of 0.3 for de­pres­sion vs placebo.

CBT

● Ac­cord­ing to one meta-anal­y­sis, com­pared to wait-list con­trols, CBT has a Co­hen’s d = 0.88 on de­pres­sion

● Com­pared to care as usual or non-spe­cific con­trols, it has a Co­hen’s d of 0.38.[15]

Goyal et al as­sess­ment of strength of evidence

○ Only 10 of the 47 in­cluded stud­ies had a study qual­ity rat­ing of ‘good’, with the re­main­der hav­ing a rat­ing of only ‘fair’ or ‘poor’.[16]

○ Goyal et al state that “none of our con­clu­sions yielded a high strength of ev­i­dence grade for a pos­i­tive or null effect.”[17]

This sug­gests that the strength of the ev­i­dence on med­i­ta­tion is weak, but that there is some ev­i­dence of small to mod­er­ate pos­i­tive effect on anx­iety and de­pres­sion. How­ever, the ev­i­dence seems to be much weaker than the ev­i­dence for CBT and an­tide­pres­sants, and CBT and an­tide­pres­sants seem to have a greater effect on de­pres­sion.

We should be­ware the man of one study, but also be­ware the man of a meta-anal­y­sis that doesn’t cor­rect for re­port­ing bias or other sources of bias. In­deed, as ar­gued in the sec­tion on re­port­ing bias, there is good rea­son to think that a pre-reg­istered high-pow­ered repli­ca­tion would cut the es­ti­mated meta-an­a­lytic effect size for med­i­ta­tion by a fac­tor of 2 to 3.

3.3. Large vari­a­tion among mind­ful­ness-based in­ter­ven­tions

Most mind­ful­ness-based ther­apy has been based on the idea of mind­ful­ness-based stress re­duc­tion. How­ever, there are large vari­a­tions among stud­ied mind­ful­ness-based in­ter­ven­tions.

● Time com­mit­ment[18]

○ The prac­tice hours of the in­ter­ven­tion in­cluded in the Goyal et al meta-anal­y­sis range from 7.5 hours to 78 hours.

○ The home­work hours are: of­ten not speci­fied, ex­ceed 30 hours in many stud­ies and even reach up to 1,310 hours in one study.

● Meth­ods for teach­ing and prac­tic­ing mind­ful states.[19]

Van Dam et al (2017) con­tend that there is far greater het­ero­gene­ity among mind­ful­ness in­ter­ven­tions than among other in­ter­ven­tion types such as CBT.[20] This het­ero­gene­ity across in­ter­ven­tion types means that we should be cau­tious about broad claims about the effi­cacy of mind­ful­ness for de­pres­sion and anx­iety.

It is es­pe­cially im­por­tant to con­sider this het­ero­gene­ity given that most peo­ple prac­tic­ing med­i­ta­tion prac­tice for 10-20 min­utes per day us­ing an app, mak­ing their ex­pe­rience very differ­ent to a full mind­ful­ness-based stress re­duc­tion course.

3.4. Shaky fMRI evidence

Many stud­ies as­sess the im­pact of med­i­ta­tion on brains states us­ing fMRI imag­ing. Th­ese meth­ods are highly sus­pect. There are nu­mer­ous po­ten­tial con­founds in fMRI stud­ies, such as head move­ment, pace of breath­ing, and heart rate.[21] Th­ese fac­tors can con­found a posited re­la­tion­ship be­tween med­i­ta­tion and change in ac­tivity in the amyg­dala. More­over, calcu­lat­ing valid es­ti­mates of effect sizes across groups in neu­roimag­ing data is very difficult. Con­se­quently, the prac­ti­cal im­port of such stud­ies re­mains un­clear.

Nonethe­less, ac­cord­ing to Van Dam et al (2017), meta-analy­ses of neu­roimag­ing data sug­gest mod­est changes in brain struc­ture as a re­sult of prac­tic­ing mind­ful­ness. Some con­comi­tant mod­est changes have also been ob­served in neu­ral func­tion. How­ever, similar changes have been ob­served fol­low­ing other forms of men­tal and phys­i­cal skill ac­qui­si­tion, such as learn­ing to play mu­si­cal in­stru­ments and learn­ing to rea­son, sug­gest­ing that they

may not be unique to mind­ful­ness.[22]

It would be in­ter­est­ing to com­pare the effects of med­i­ta­tion on the brain with the effects of other ac­tivi­ties such as read­ing, ex­er­cise, sport, or hav­ing a con­ver­sa­tion with friends. I sus­pect that the effects on fMRI scans would be quite similar for many mun­dane ac­tivi­ties, though I have not looked into this.

4. Use­ful re­sources and reading

● Coron­ado-Mon­toya et al, ‘Re­port­ing of Pos­i­tive Re­sults in Ran­dom­ized Con­trol­led Tri­als of Mind­ful­ness-Based Men­tal Health In­ter­ven­tions’, Plos One (2016)

○ A high-qual­ity anal­y­sis of re­port­ing bias in mind­ful­ness re­search.

● Goyal et al, ‘Med­i­ta­tion Pro­grams for Psy­cholog­i­cal Stress and Well-be­ing: A Sys­tem­atic Re­view and Meta-anal­y­sis’ JAMA In­ter­nal Medicine 2014.

○ A re­view com­mis­sioned by the U.S. Agency for Health­care Re­search and Qual­ity. Pro­vides a good overview of the qual­ity of the ev­i­dence and es­ti­mates of effect size but cru­cially does not cor­rect for re­port­ing bias.

● Van Dam et al ‘Mind the Hype: A Crit­i­cal Eval­u­a­tion and Pre­scrip­tive Agenda for Re­search on Mind­ful­ness and Med­i­ta­tion’, Per­spec­tives on Psy­cholog­i­cal Science (2017)

○ A very crit­i­cal re­view of the ev­i­dence on mind­ful­ness, which raises sev­eral prob­lems with the ev­i­dence. How­ever, I think the tone is over­all too crit­i­cal given the ev­i­dence pre­sented.

● Sam Har­ris—Wak­ing Up book

○ An in my opinion overly rosy re­view of the ev­i­dence on med­i­ta­tion, es­pe­cially in chap­ter 4.

● The Wak­ing Up med­i­ta­tion app.

○ The best med­i­ta­tion app I have tried.


References

[1] See for ex­am­ple this piece by Rob Mather, and this by Louis Dixon


[2] Ni­cholas T. Van Dam et al., ‘Mind the Hype: A Crit­i­cal Eval­u­a­tion and Pre­scrip­tive Agenda for Re­search on Mind­ful­ness and Med­i­ta­tion’, Per­spec­tives on Psy­cholog­i­cal Science 13, no. 1 (2018): 36–37.


[3] Stephanie Coron­ado-Mon­toya et al., ‘Re­port­ing of Pos­i­tive Re­sults in Ran­dom­ized Con­trol­led Tri­als of Mind­ful­ness-Based Men­tal Health In­ter­ven­tions’, PLOS ONE 11, no. 4 (8 April 2016): 1, https://​​doi.org/​​10.1371/​​jour­nal.pone.0153220.


[4] For ex­am­ple, a re­view of RCT ev­i­dence on mind­ful­ness by Creswell opens with a quote from John Ka­bat-Zinn, a lead­ing figure in mind­ful­ness stud­ies: “There are few peo­ple I know on the planet who couldn’t benefit more from a greater dose of aware­ness”. [creswell ref]


[5] Coron­ado-Mon­toya et al., ‘Re­port­ing of Pos­i­tive Re­sults in Ran­dom­ized Con­trol­led Tri­als of Mind­ful­ness-Based Men­tal Health In­ter­ven­tions’, 5.


[6] Coron­ado-Mon­toya et al., 9.


[7] Coron­ado-Mon­toya et al., 10.


[8] Amanda Kvar­ven, Eirik Strøm­land, and Mag­nus Jo­han­nes­son, ‘Com­par­ing Meta-Analy­ses and Pr­ereg­istered Mul­ti­ple-Lab­o­ra­tory Repli­ca­tion Pro­jects’, Na­ture Hu­man Be­havi­our, 23 De­cem­ber 2019, 1–12, https://​​doi.org/​​10.1038/​​s41562-019-0787-z.


[9] Mad­hav Goyal et al., ‘Med­i­ta­tion Pro­grams for Psy­cholog­i­cal Stress and Well-Be­ing: A Sys­tem­atic Re­view and Meta-Anal­y­sis’, JAMA In­ter­nal Medicine 174, no. 3 (1 March 2014): 357–68, https://​​doi.org/​​10.1001/​​ja­main­tern­med.2013.13018.


[10] Goyal et al., 364.


[11] Goyal et al., 364.


[12] Goyal et al., 361.


[13] Goyal et al., 364.


[14] Goyal et al., 365.


[15] Ellen Driessen and Steven D. Hol­lon, ‘Cog­ni­tive Be­hav­ioral Ther­apy for Mood Di­sor­ders: Effi­cacy, Moder­a­tors and Me­di­a­tors’, Psy­chi­a­tric Clinics 33, no. 3 (1 Septem­ber 2010): 2, https://​​doi.org/​​10.1016/​​j.psc.2010.04.005.


[16] Goyal et al., ‘Med­i­ta­tion Pro­grams for Psy­cholog­i­cal Stress and Well-Be­ing’, Table 2.


[17] Goyal et al., 365.


[18] Goyal et al., Table 2.


[19] Van Dam et al., ‘Mind the Hype’, 40.


[20] Van Dam et al., 46.


[21] Van Dam et al., 49–51.


[22] Van Dam et al., 50–51.