Made you look!
I’m less irritating in person than I am online—my DM’s are open if anyone wishes to entertain my over-opinioned thoughts on mental health any further. I’m always keen to learn. UK doctor / EA outsider / well-intentioned visitor to your community (I come in peace).
I was told to mention that Gregory Lewis is responsible for my philosophy training, as well as Kristen Bell—please direct all criticism appropriately.
This is an interesting read, thanks so much for putting it together. I can’t really talk about things on a larger scope e.g. companies and official consultancy offers, but I suppose on the more micro-level of cakes at conferences or research on a forum...
I think you set out some of the conflicts in a way that helped me structure my thoughts a lot more clearly. I offered an organisation some support in putting together a meta-analysis on this forum in a field I specialise in—this was out of a general sense of altruism and support of the over-arching cause, as well as soothing my conscience a little for otherwise providing critique. At the same time, I did wonder if doing so ‘de-valued’ the offer i.e. suggested my skills were worth what I was offering (nothing), I had an abundance of time to be doing this (i.e. I’m not in demand), and as an EA-outsider using a pseudonym, I don’t come pre-vetted [While my brother is ‘known’ in EA, I wouldn’t feel comfortable using my searchable legal name on a public forum due to my profession putting me at high risk of stalking, and (no doubt) a combination of my age/gender/physicality etc meaning I’ve had a lot of unwelcome attention online].
I think all of this skews naturally to EA projects selecting in-group (trusted) members for specific tasks, or having to pay for commercial services, which will likely come with a higher price due to any overhead of running a company (unless offering an EA-discount, making it more likely to be EA-aligned and receive promotion through EA—it’s perhaps a quid-pro-quo). E.g. I have no idea about accountancy and if I was hiring an accountant, I wouldn’t be able to look at their work and know if they were good or not—I’d likely rely on reputation, recommendations, the appearance they are ‘legitimate’ etc, and if they fit my budget. It makes sense to me for EA projects, this would favour EAs and EA-orgs.
Perhaps there is an issue with a more ‘grassroots’ concept in EA in the broader sphere of cost-effectiveness where some people likely struggle to properly evaluate their worth, don’t charge for their time, and therefore won’t get the ‘signals’ to build on their contributions to realise they should. It’s quite a leap from offering ad-hoc services to running a company. On the other hand, with receiving grants/funding or reputation being the markers of ‘valuable’ work in the community, this is open to corruption. Let’s say my family is wealthy, so my Dad is happy to give me some money to start an organisation, there’s little/no personal risk if it fails, and I don’t need to work for six months so I can focus entirely on getting it off the ground. I could set it up as something official and make it look like a ‘proper’ company, get a flashy website, devote all of my time to developing content and applying for grants, offer discounts to solicit clients and positive endorsements, advertise, network in EA, etc. Even if my work was terrible, if it is niche enough, it’s likely I would seem extremely legitimate within the community and I could enter a positive-feedback loop of increasing revenue, over-confidence and over-recommendation. It seems that there is risk of over- and under-funding effort along the same lines.
This is perhaps a silly question, but do people ever use a sort of Patreon model or have links to individual cash apps like PayPal etc.? [FWIW my offer wasn’t taken up so I’m relatively unbiased here] To be basic, I suppose how people might endorse/reward creators of content on social media platforms (a bit like Asimov in your example having a ‘Give whatever you like’ tin vs pricing the cakes). I’ve not seen this and it might be against the rules (sorry) or be a gauche idea, but this forum feels relatively unique in the amount of work people put into posts and other community-facing offers from regular contributors which provide a lot of value. It might allow for people to make better cost evaluations of their time/effort and perhaps encourage people who are doing good work to continue and recognise they should be charging for their services, which would allow the organic development to perhaps something official. This might also provide a slight counter-lever to the power dynamics and provide scope for reduced insularity in EA, as mentioned in your post, in encouraging smaller service-providers to not work for free, and allowing some questioning of bigger organisations utilising ‘free’ services if there was an option to fairly compensate people for their work (even if this was voluntary).
Oh gosh! Not at all, thanks so much for letting me know in that case—I’m sure others are reading it the same way if that was your interpretation.
I’m just being quite cautious in trying to not come across as some strangely specific and verbose troll with an HLI problem. My post wasn’t about HLI and I found it difficult striking a balance to respond to that comment without seeming to make it about HLI i.e. perhaps there could be an interpretation I’m acting in bad faith (should that interpretation exist at all) not due to previous critique of HLI but my short history on this forum. I personally think that is a fair thing to find suspicious (!) and it’s a reason I’ve felt there has been a higher burden on me vs established members in demonstrating I’m trying very hard to understand EA’s philosophy and methods to allow a good faith interpretation vs providing lazy, half-baked criticism—I was trying to allude to this in the end of that comment and in my response to Ray below.
The culture is clearly a bit more challenging for me, and I’m often doing a few things at once when I’m online, so please continue to call me out if I risk being misunderstood.
I don’t think there have been any edits, and I’m sure Michael would identify them if he made any. I was trying to make my comment less personally-directed by broadening the scope (i.e not just talking about HLI and turning to a more general sentiment about input from ‘anonymous’ outsiders on this forum and how they might be perceived—I remain a bit confused about the ‘real name’ comment).
FWIW, I’ve had lots of positive interactions with EA members on this forum (in public and in private), even from such short-lived involvement, so it wasn’t a chip-on-the-shoulder type statement. I mean no smear against EA as a whole. Thanks for prompting me to clarify.
Nope, I’ve not heard from any current HLI members regarding this in public or private.
Hello,
I just wanted to follow-up, as I’ve read your links as promised. I share a genetic trait of speed-reading, so this wasn’t too onerous – please forgive me for not watching a video of your EAG talk (this is inefficient for how I work) but you linked your write-up on the forum which I read instead.
I feel you may be approaching this assuming I’m a bad actor, and I’m going to further endeavour to demonstrate this isn’t the case (I thought offering my meta-analysis skills for free, my arguments above about not over-interpreting an RCT, and this post in general would demonstrate that I’m approaching this in good faith). I care about this topic and I want EA to continue having interest in mental health/wellbeing – I think this could do a lot of good. I’m not sure why being non-EA would suggest I disagree with effective ways of doing altruistic work, or that I’m incapable of contributing anything to a discussion of mental health and wellbeing given this is literally my job, and so shouldn’t offer my perspective on a public-facing forum. I want to understand more about EA approaches to my field but I’m not so selfish to ask someone to do all the work for me (a stranger), hence putting a lot of effort into my post to show I am genuinely interested in trying to understand (side note: thanks again to people who have offered to chat off-forum—it’s really appreciated, and I’m sure my opinions will change in the coming weeks with your help).
I’m wondering if this is coming across instead that I care about making another anti-HLI post – I don’t. I tried approaching this problem (how does mental illness relate to wellbeing) from first principles to see what happened; I thought this was a sort of EA-oriented approach. I vaguely referenced the HLI post I commented on in my introductory comments to explain how this work came to be and interrogate my motivations (allowing others to do the same) - not linking the post/naming HLI, calling my contribution an ‘overly technical diatribe’ and not mentioning HLI at all was meant to dissaude from the idea my involvement in that thread should be considered on it’s merits. It was a reflection of my internal responses which I thought was important to interrogate in trying to adopt a less biased approach in my work (appraisal of SWB as a whole). We reflect all the time on unconscious vs conscious in psychiatry (blame Jung) but maybe this was weird to mention. Either way, It’s not like I’m competing for any EA space/funding which would make this something sinister or detract from HLI, and I’m not a big deal at all to think I could do this even if I wanted to.
Anyway, I’m confused why you’ve said “’we’ (i.e. happiness researchers) know quite a bit about the nature and measurement of wellbeing and it’s causes and correlates, but relatively little about the ways to increase it…” when you’ve linked several resources that suggest my heuristic approach isn’t too bad and my main question was about putting this in practice. I’m not too sure I understand how your comment seems to contrast the idea of my overly/”deliberately” shallow non-academic review with several sources supporting my approach. Isn’t that a good argument for interdisciplinary work? I’m not really understanding the tone of your comment when my post wasn’t ‘here’s my systematic review of wellbeing, it’s a new field with nothing useful to say’ but rather, ‘this is what makes sense to me, this doesn’t seem too out-there?, could this be used helpfully?’
In terms of the Diener, et al. paper– I had come across this author before in my work for this post (but not this specific paper). They use the comment about 170,000 articles (paper was accepted in 2017) in their introductory paragraph as a set-up for their narrative review of the literature – so I don’t need to talk about Google Scholar or try and ‘verify’ the numbers. They only mention it to talk about the growth of interest in SWB, which we all agree with.
This was a lovely piece to read, and I’m glad you directed me to it. It’s an extremely thorough review of wellbeing literature (32 pages excluding refs in PDF form, 439 different references). I’m not too sure why you’ve linked it, as it seems to be really in support of a lot of my work? Perhaps this was your intent as you don’t comment on my more specific criticisms, or my overall framework or findings. It’s not clear in how you framed it, which seemed to be as a counterpoint to me post, but maybe it was an agreement with the broader points you didn’t want to address in detail.
While I’m sure you’ve read this paper as you referenced it, for anyone else following – I can summarise the areas where this review supports my own:
It discusses concerns with single-measure SWB that I identified e.g. issues of subjectivity as a whole, reluctance providing honest answers to researchers, differences of meaning in various contexts, how people retrieve the relevant information to respond (likely too quickly to provide an evaluative judgement) and influenced by current mood/prior discussion (my joke about coffee). They point out that studies ‘on their own, provide little evidence regarding the validity of the measures themselves’, and conclude that, “although different camps have emerged that advocate for one set of measures over others, we believe that such advocacy is premature… we recommend that, when possible, researchers should include a broad array of measures….”
It lists several predictors, correlates and causes of SWB: social relationships/social capital; income and wealth; religiosity; demographic factors (e.g. age); health; genetic and personality factors; internal factors (e.g. self-esteem); basic needs (e.g. food, shelter) and safety; autonomy; ‘freedom’; ‘balanced life’; ‘adaptation’ to life events such as disability; cognitive processes; ‘fair and efficient’ law/government… These were all factors I either included in my framework or identified in review and explained my reasons for excluding.
The paper was arguably even more in favour of cultural contextualisation than I was e.g. “Even when a relevant term such as happiness is clearly defined, it is worth exploring whether the term [is] used in similar ways across cultures and time. Most philosophers and historians agree that the concept of happiness has changed over the years. In antiquity [the authors reference ancient Greece], this concept centred around good luck and fortune, whereas contemporary Americans view happiness as a pleasant experience over which they have control and something that they can actively pursue.” The authors go on to mention the differences of contemporary understandings of happiness between American (personal achievement and intense positive emotion) and Chinese people (spiritual enrichment, harmony and dialectic relation between happiness and unhappiness).
It spends a long time talking about the difficulties of interpreting causal vs correlative relationships, or third-variables, in describing how these factors relate to SWB.
They are consistent in pointing out all the ‘open questions’ that still need to be answered in understanding and measuring SWB, and how to apply this to policy decision-making. They also suggest research into other methodologies to answer questions about predictors and outcomes of wellbeing to define these relationships better, and describe a variety of ways SWB can be measured (outside of various self-report tools).
The conclusion of this paper:
As this review makes clear, growth in the science of SWB over the past decades has been quite rapid, and there are no signs that this growth is slowing. Yet, there are still large numbers of questions in need of research. At this point the most important research goes beyond cross-sectional correlations of self-reported measures of SWB, and includes experimental and longitudinal designs, as well as varying methods for assessing SWB. Furthermore, what is needed now are studies focused on the underlying psychological processes that influence SWB, as well as research on how SWB affects future outcomes. SWB research has become a truly interdisciplinary enterprise that includes all areas of the social, behavioral, and brain sciences.
It seems they think there is a place for me within wellbeing research and I’d discourage anyone adopting the idea that only evaluating a problem from one perspective strengthens your understanding – it’s much more likely the ‘most right’ answer lies in-between various disciplines to maximise experience and expertise.
Your next link – the World Happiness Report, seems to just be cited here to further illustrate growing interest in SWB (which I haven’t disputed). It does highlight as a pull-quote on the first page: “The recent pandemic has likely had a strong impact on popular conceptions of what is most important for a good life, and indeed on how society can foster collective improvements in well-being”
…Again, I’m not sure how this discounts my (poorly-worded) comments about the effect of COVID on the field. It might be to suggest it was an oversight to omit ‘happiness’ from my search which would produce a different graph than the one I used, but I’m struggling to see the relevance of this above what I’ve already addressed (i.e. your graph includes any paper mentioning ‘health’ or a number of other vague terms due to the search) – it seems we all agree there is growing interest in SWB, and there is an increasing interest over the last few years, for one reason or another.
The report goes on to mention the same issue I did in that there are a few early-adopter countries in the field of wellbeing research, but this is becoming more global over time from the ‘earliest mentions’ in the economics literature from the 1970s (they made this comment, not me, hence their graphs starting in 1970):
This was what I was representing in my pie chart (I’ll admit this is slightly prettier) to talk about interpreting the existing research (i.e. the issues the Diener paper raised about cultural understandings of wellbeing/‘happiness’). The report also mentions the diverse number of measures being used for SWB (10 domains and 38 individual measures) to discuss how best to measure world happiness. These are elaborated upon further in the Appendix to the chapter you linked, along with other graphs representing country-based origins of SWB.
In the ‘challenges’ section, it explicitly talks about the difficulties in composing ‘any new indicator framework aimed at capturing a meaningful concept such as well-being or progress’ and how that can combine different measures, or aggregate individual experience into summary numbers for groups or populations, to ‘express values through numerical weights’ – this is exactly what I had a go at doing with my framework/matrix. Interestingly, it also encourages not focusing on over-analytical approaches to “unanswerable questions” (they talk about climate change as an example) and delaying actions which provide benefit for the sake of attempting ‘futile’ over-analytic approaches of far-future impact calculations. This seems to be similar to my conclusion.
The ”To WELLBY or not to WELLBY” I read prior to this work and I’m afraid I still disagree on re-read for the same points I made in my original post (single-measures in isolation, life satisfaction, conversion to different settings). I’ve highlighted that this isn’t a ‘fringe’ opinion derived from my ignorance when looking at the other materials you linked. I also clearly will not agree with ‘Affective mental health, usually measured with depression scales, involves questions about how people feel, which will directly relate to SWB’ (an unreferenced footnote) – I feel I have explained this point very clearly in my post and cannot take HLI’s report as ‘stronger’ evidence that would lead me to reconsider; I have addressed this from my relevant expertise and from listing the PHQ-9 in a previous comment.
Also, I’m really sorry, but that article mentions repeatedly HLI’s psychotherapy meta-analysis I’ve already talked about on this forum (as have others) in terms like “(3) Using SWB will reveal previously under-captured benefits, such as it has already done for psychotherapy,” and alludes to future plans to publish research on issues such as the ‘neutral point’ on SWB scales, and agrees comparing SWB scales isn’t supported in literature (i.e. to convert to the LS evaluation used to support the utility of the WELLBY) but suggests things that ‘could’ be done to manage this. It’s putting me in a difficult position to discuss this link when I’ve explicitly stated I don’t want to reopen old arguments and I feel any further critique of HLI isn’t appropriate at present on this forum. I feel statements like those in the article are otherwise inciting me to dig around HLI’s website to see what was built off these prospective ideas, and it’s just going to devolve from there. Perhaps you could highlight the parts that disagree with my original post in terms of your opinion vs that of HLI if you really think this offers strong counterpoints?
In terms of your linked EAG debate, I’m running into similar problems. I didn’t see much disagreement in the early parts discussing the importance of SWB and ways of measuring it, but otherwise it focusses on HLI’s work and references the meta-analysis, framing it as: “We looked at cash transfers and group therapy for depression. We did a meta-analysis of each. We looked at the available studies. We had about 140,000 people total across both. So we really tried to look at all the available evidence that was there. We did some kind of technical jiggery-pokery to convert it into wellbeing scores. And what we found was that the $1000 cash transfer has about the same per-treatment effect as group therapy for well-being. You can see that the cash transfer does have an effect, has a much longer lasting effect. But the effect of treating depression is larger, but fades sooner.”
From my previous comment on the HLI post on this forum, you must see how this is problematic to link here. It’s not arguing against anything I’ve said in my post, and I can’t go into the reasons why I disagree with later points without restarting settled arguments. I’m assuming you didn’t reply to my comment on your post about HLI as you didn’t want to engage with me about my issues with the meta-analysis etc, and I think it would be implicitly crossing a boundary for me to reproduce those arguments here. I think you are and were completely within your rights to disengage with that discussion and I respect that choice—I don’t want to put you in a position where you feel you have to engage again in something you found negative to avoid appearing to ‘concede’ any points about HLI and so I cannot comment on HLI-related links.
I’m not really ‘online’ very much and it was part of my reflection about my comment on the HLI post—I felt it was only fair to similarly ‘put myself out there’ and invite criticism of my work given how I imagined you and the HLI team might have felt in that thread. I tend to post random artwork etc online—it would feel really harsh if in the middle of some drama, a critic’s sister happened to be an art professor and came in to critique my art. I thought it was only fair to allow the same to happen to me.
Awkwardly, your opponent in this debate seems to support my approach in this post on several fronts. Their conclusion:
OK, so what’s my upshot to conclude? I totally agree with Michael that effective altruism should think harder about well-being. What does it mean for a life to go well? How can we promote good lives, not just more lives? However, I think that we should think about good life in a multidimensional way, in a rich way that takes into account structure and sociological conditions. I think we shouldn’t fixate on RCTs and CBA and shouldn’t use them as a hammer in search of nails. Effective treatments for complex problems are likely to be messy, and we need to develop epistemic frameworks for appreciating that messiness and tolerating it, not assuming it away.
I felt I was doing that but trying to put it in EA terms for CEA-purposes, and it’s not clear to me why this rebuttal was considered ‘wrong’ (?). FWIW, I have also come across the @ryancbriggs post on the capability approach (which is something referenced in the counter-argument) and I felt much more closely aligned with these ideas than HLI’s. [I’m sorry, I don’t know the etiquette on ‘tagging’ here—please ignore me if this isn’t expected when linking to someone’s work].
I’m not going to go through each of the items on the HLI reading list you also suggested (as it’ll be too long), but similarly it wasn’t presented in reference to anything specific, and I’m not sure if it’s just to demonstrate the idea I might benefit from more reading into SWB. I’m struggling to agree that I’m vastly under-competency from the resources you did link to understand how these would be revolutionary to my understanding.
Honestly, I’m pretty confused, perhaps I’m not reading the tone of your post correctly—very possible. I do appreciate you linking these papers/reports but I’m struggling to understand your overall point (my interpretation was you didn’t think I knew enough for my post to be very valuable, especially on the basis of a search you felt I had conducted poorly/with an aim in mind to downplay SWB—FWIW I didn’t think 699 papers was weirdly small to anticipate this being the perception). From the HLI-related links you’ve chosen, perhaps we just disagree on certain things, which is fine, but your comment also didn’t highlight all the ways my approach fit in with other work in this area, and in complete reading of those papers. I think that omission from someone considered well-known in this area in EA does invite negative conclusions of the post as a whole and suggests those sources devalue my work much more broadly.
I’ve been clear I’m not approaching this with any expertise out of my field (mental health), and I have a lot to learn—I’m open to the idea further discussion will lead me to change my mind [I don’t necessarily expect Michael to respond, he certainly shouldn’t feel any obligation, I’m referencing other people kindly offering their time!]. But common ground is helpful as it suggests interdisciplinary work (similar to what went into the DALY, etc) is possible for wellbeing measures too. I’m not sure if taking a stance like “happiness researchers” just know more/everything about wellbeing/SWB is anything more than severely limiting – if so, I suppose I could be saying no philosopher has any right to comment on mental health as “we” (psychologists/psychiatrists) have got it, so shh.
I disagree with this—not only is this a wild take in psychiatry of all disciplines, but experience has taught me multidisciplinary work is key in providing the best care for patients and solve problems much more effectively. It prevents people working in silos and wasting time. Is it really the case in EA that any particular speciality field (or EA as a whole) ‘stakes claim’ on something like wellbeing which is inherently cross-discipline, to say there’s no benefit from other ideas? I think there’s benefit in keeping concepts accessible vs telling people they won’t understand and should consider a ‘reading list’ to contribute—the point of interdisciplinary working is to avoid the need for everyone to be competent in philosophy, economics, psychiatry, psychology and social science etc to have an opinion—you invite people with detailed understanding of their specific area to discuss and find solutions. I’m not suggesting EA people need to do medicine/psychiatry, read textbooks and papers to understand mental health, just maybe there is value outside of the ‘bubble’ of “happiness researchers”.
It 100% may just be a communication thing, but this was more of an olive branch than perhaps it appeared. I think perhaps it is too late for me to strike this tone with HLI and its members (again, this is fine, it doesn’t offend me—it’s completely fair as a take) but felt it was worth responding anyway in case others might have felt I’ve been mean-spirited. Just culturally, this is a small forum with a lot of power, and it’s difficult to understand as an outsider. I was surprised to learn a lot of these linked resources seemed to suggest a closer alignment than I assumed and perhaps there is a role for non-adversarial non-EAs approaching some otherwise intimidating concepts in good faith for good outcomes.
Hi MichaelPlant, [Edit: Jk—I don’t get the comment about my username/real name, I saw a mix being used on the forum, but I might have missed some etiquette—would you like my real name? Just ‘hello’ is fine if you’d prefer—no offence taken.]
Thanks so much for taking the time to read and respond! I was hoping to get more insight from people within EA who might be able to fill me in on some of the more philosophical/economic aspects as I’m aware these aren’t my areas of expertise (it was very much a ‘paper’ EA-hat I was trying on!) - I felt furthering my online searches wasn’t as helpful as getting more insight into the underlying concepts from experts and hoped my post would at least show I was interested in hearing more. Thanks for the links as well—I did come across a few of them in my approach to this work, but will take your advice these are worth looking at again if you think I’ve not appraised them properly—you definitely know best in this regard!
Also, apologies—you might be right in saying I didn’t structure a paragraph very well if it has left anyone with the impression I was suggesting subjective wellbeing research has only been in existance since COVID. My own graph disproves this, for starters! I think it’s this paragraph from the first section I’ve not phrased well (italics added).
From my quick literature review, the interest in wellbeing as a research area is a relatively recent phenomenon. There has been rapid growth in papers being published about subjective wellbeing from around 2020 onwards. I’d guess this is due to (1) the COVID-19 pandemic and lockdown restrictions making this a hugely important topic for public health officials and politicians, and (2) the recent interest in using wellbeing as an outcome when evaluating the effect of a broad range of policy decisions, which has subsequently driven interest in quantifying ‘wellbeing’ for use in cost-impact analyses.
I was trying to emphasise the relatively steep growth in interest over the last few years due to questions about cost-effectiveness (e.g. WELLBY), which as you mention is ‘barely older than COVID’. I don’t actually think we disagree here so I’ll need to think how to rephrase it to avoid conflating this with SWB research as a whole—to be clear, I don’t think your reading of this was unfair and I can phrase it better.
I’m not too sure I was ever arguing I was doing an exhaustive literature review (?) - I felt I stated a few times this was non-scientific, should have no weight, etc. My goal was just trying to get a quick overview as more of a sense-check, but didn’t want to limit my reading to the first x number of pages in case that was biased—I chose very limited terms and stated them so it would be clear how I did the search, which allowed you to double-check I wasn’t pulling anything dishonest (impressive that 7 papers have been added since I did the search last week—clearly there is a lot of interest!). If I set out to do a completely exhaustive review, you’re right to suggest the terms you did (I would add “SWB” as well), but I’m not sure that would be a reasonable expectation of something like an “abstract-only” non-academic review from a visitor with a full-time job when that returns thousands of results...
I’m sorry if it appeared that selecting limited search terms was an attempt to ‘downplay’ SWB research as a field—I mentioned it was my time constraints that were the problem but it’s easy to miss in a long post. I felt I was trying to explain throughout how interesting/useful I found this piece of work, spoke about how it changed my mind in the later sections and identified blind-spots I wasn’t aware I had. I don’t think I was critical of ‘subjective wellbeing’ research as whole—I tried to lay out my very specific concerns very clearly (e.g. in “life satisfaction” being used as an isolated measure for subjective wellbeing) but my overall conclusion was in support of finding a way to incorporate more of the diverse value these researchers were adding to the understanding of how we think about wellbeing estimates e.g. other measures, combining measures, qualitiative research, etc. I was approaching the problem with an open mind and left the exercise positively so I’m sorry to see it might have appeared mal-intentioned.
I hope this doesn’t seem like nit-picking, and it’s not intended as criticism of you personally as you similarly were upfront about not being familiar with PubMed; it can be tricky to get to grips with, but it might be helpful to share a quick point.
For the search that you linked that returned 150k results—if you go to “Advanced” under the search bar and click through you can see this searched “All Fields” and expanded the terms you used as they weren’t in quotation marks (it’s a quirk of the database that perhaps isn’t true for others!). This was the actual search run from your link—for all I know this was your intent, but in case it wasn’t (just judging this as a possibility based on the hyperlink):
With quotations to limit results to those specific terms (I’m not sure if that was your intent) and using Title/Abstract (PubMed also doesn’t really like hyphens as you see above but I used “subjective well” to try and work around this) you get more like 20-30k results. You can use MeSH terms (key words instead), other variations, etc to push it up or down (I tried a few variations to work with ‘well-being’ hence this being #8). And as you rightly say, you can always add more terms to include more papers.
Again, my aim isn’t to down-play SWB research at all with this point (I like the field!), it’s just in case it’s helpful as I use PubMed all the time and it’s one of many databases which contains mental health literature. Whether 2k or 150k+ results (even a few well-written papers wouldn’t necessarily be an argument against the field—it just suggests to me it’s new), I still stand by my OP in being broadly positive about the area as a whole, hence coming up with a framework of my own in inspiration, I just haven’t shifted much in my specific critiques of certain applications of specific aspects of SWB but I’ll have a read of your links and see if it changes my mind! Similarly, if you had any further thoughts, I’d appreciate hearing them for your feedback.
Ah, okay, thanks for this—I think I see why I’ve been confused. I skipped through my arguments for psychosis/schizophrenia research in my OP because I felt my post was already ridiculously long—I see now that it could be read as suggesting a high HCP investment is needed for schizophrenia treatment in an LMIC. You make the fair point (that I entirely agree with) that anything requiring significant additional use of HCP resources is a major problem. Sorry, I didn’t understand where you were coming from and so I’m sure my response to your comment makes absolutely no sense—I thought you were making a completely separate point about HCPs in LMICs and so I was a bit perplexed and needlessly theoretical in my response.
The things I skipped over in the schizophrenia/psychosis research section were ways to reduce the needs of health services through treatment of psychosis/schizophrenia that would not require increasing numbers of HCPs. Again, none of this is an EA area suggestion—I do not have any authority to talk about LMICs. I’m explaining why I mentioned these things in my OP as I covered this poorly. It was intended as highly speculatively-reasoned areas for a potential deeper look if my groundwork seemed okay
Given prevalence is relatively similar across the world and static over time, LMICs are managing schizophrenia at present—it’s not an illness which is influenced by societal problems or ‘increased awareness’ etc. It’s why I assume that community-approaches and spiritual understandings might be very similar to things like CBT-p and family intervention—I wouldn’t assume these should be ‘rolled out’ to LMICs (rather the opposite that we could have learnt from other cultures a lot sooner to help our patients—we tended to use institutions/asylums instead) and there are non-HCP/non-medical approaches that can and do help people with schizophrenia. We might give the same ideas medical terms and run RCTs, but we likely didn’t ‘invent’ these approaches, nor would it be considered groundbreaking that this helps elsewhere in the world with different cultural understandings of psychosis. It’s an argument against an assumption about the need for therapy/therapists in all LMICs for psychosis.
It helps schizophrenia to treat it early (reducing duration of untreated psychosis to have best chance of complete recovery, faster recovery, reduced length of subsequent hospital admission) and my personal assumption would be in LMICs the issue is access to antipsychotics, particularly newer drugs which have fewer side-effects and are likely under patent—not HCPs/psychiatrists. I could be wrong—it just doesn’t necessarily require advanced training to recognise psychosis when someone is that unwell they need treatment (if it did, expertise-sharing/teleconsults with specialists isn’t necessarily a bad approach if you want to support one doctor working in a remote setting expected to manage a whole range of problems independently—it’s not an exclusively psych issue to consider access to specialities without training more HCPs). Using older drugs is a problem as they have disabling (e.g. Parkinson’s-type) side effects amongst many others, which means patients stop taking them and therefore are prone to relapse (needing more acute/hospital-based care) - I assume these are more commonly used in LMICs as they are much cheaper.
Some of the other (slightly newer) treatments cause obesity, diabetes, high blood pressure, heart disease etc, so will increase the need for other types of healthcare in the long-term. I guess, it could be one option to look into in-patent drug access agreements for LMICs to access the newest drugs for lowest side effect burdens, but as I wasn’t talking about LMICs, research for new treatments which work better/faster or are better-tolerated is another good option. If people can stay well on a medication, they will not need the kind of frequent HCP/social support they otherwise absolutely do need with chronic forms of schizophrenia.
I also assume clozapine is not easily accessed in resource-poor settings. This is the only medication with any evidence in treatment-resistant schizophrenia (which similarly would be highly-consumptive of HCP resources if not treated). Part of this reason is that it requires intensive blood monitoring—weekly blood tests for first 18 weeks, then every 2 weeks for the first year, then monthly thereafter. Heart health could be checked with (at least) an ECG machine which isn’t so rare, but the blood monitoring is clearly an issue if you don’t have ready access to a lab to process blood samples (and the infrastructure/cold chain to support transport/reliable results). It’s risky using clozapine without this monitoring. This is not just about checking levels of clozapine in the blood, but due to the known issue that taking clozapine can cause levels of white blood cells to crash and leave people unable to fight off infection—another issue in any setting with high prevalence of infectious disease (but it can be fatal anywhere). Finger-prick monitoring which I referenced briefly could have huge impact in remote/LMIC settings as it wouldn’t require a lab or anyone able to draw bloods to provide monitoring and might therefore support access to this effective treatment.
And it’s true that our long history of relative neglect of people with psychosis has set us really far back in terms of research. As an example, NMDA-r encephalitis was only discovered in 2007 (“Brain on Fire” is based on this illness) - a treatable form of psychosis when it was discovered patients were actually suffering with an autoimmune encephalitis and so needed immunotherapy, not antipsychotics. It’s generated a flurry of interest in understanding mechanisms for schizophrenia and was a significant breakthrough for identification of auotimmune encephalitis and directing treatment which undoubtably has saved lives. It seems plausible to me that there is a lot of potential gain in psychosis research when we are starting so far behind other areas of medicine.
Schizophrenia (and bipolar) are a little unique as they are considered more ‘organic’ than others i.e. they cannot arise from environmental factors alone, there has to be an underling ‘switch’ waiting to be flipped which is likely genetic in origin. While in illnesses like depression/anxiety/substance misuse, people can be more genetically prone to these problems, they can also arise entirely as a result of someone’s environment and—to me—a high prevalence of these illnesses in a country is a ‘canary in the mine’ for greater issues in public health/society. It’s why I’m a bit reluctant to assume medicalised treatments are helpful and think there is room to think about addressing poverty, inequality, hunger, preventable disease etc as ‘societal-level’ highly effective mental health treatments.
I didn’t mention it as a key area for mental health-style interventions for this reason (and thought I’d come too close to talking about SM). In my example about offering CBT vs cash as a trade-off CEA; that isn’t to suggest people in LMICs devalue depression treatments etc, it’s that they might know exactly why they are depressed and it’s therefore more effective to direct treatment to the cause than assume using therapy/medication can help people in the same way while they are living in very objectively bad circumstances. This wouldn’t favour HCPs either.
I hope that’s more helpful to explain where I’m coming from in thinking schizophrenia/psychosis is worth more thought from a wellbeing perspective as it potentially offers scope for larger gains through some of these simpler things outside of HCP training/retention. I’d be relatively certain psychosis research is EA-aligned, but would not suggest any other intervention without more detailed work in psychosis or other mental health conditions in LMICs.
In case my repetition about cultural relativism seems a bit hokey to anyone reading, this is my favourite paper I often suggest as a ‘gateway’ to anyone who doesn’t find psychosis interesting or wants to know more. It’s not overly technical/medical. It’s based on trying to understand how people who were born deaf (i.e. have never perceived sound) or lost hearing later in life/have partial hearing loss “hear voices” as a hallucination experience. It was research conducted by someone who uses sign-language and explains lots of the issues in how certain words are understood to a ‘hearing’ person vs how they are meant in BSL and how they worked around this. It might help explain some of my reluctance to make assumptions and why I constantly talk about within-community research—it’s so easy to have blind-spots (we don’t know how much we don’t know).
Thanks so much for commenting, you make an interesting point!
It’s stretching my competency a wee bit to discuss mental healthcare in LMICs (and I won’t touch poor StrongMinds again!). With the prevalence of schizophrenia being relatively static over time, I suppose the whole concept of this requiring highly-trained medical professionals is with a “Western” bias—I’d be open to the idea that LMICs may manage mental illnesses differently with good results, hence my interest in supporting research from LMICs and having some flexibility in the idea RCTs are ‘best’ to allow us to learn from other settings. It’s not like psychiatry has always been ‘right’ in its approach and has occasionally been very wrong, historically speaking, as has medicine as a whole. We can only do the best we can with the knowledge we have now, share and learn as much as possible, but still, who’s to say in 100 years we’re not going to be scoffed at in textbooks for being so misinformed.
CBT for psychosis (CBT-p) and some of the family interventions are based around helping people find meaning in their experiences, set their own goals for recovery (a strictly ‘medical’ model might suggest eliminating ‘symptoms’ is the goal; while many people are not concerned by hallucinations etc, if they understand them and they are not intrusive/distressing), and promoting open discussion to destigmatise experiences which can be frightening to both experience oneself and witness in a loved one. I can see community-based or spiritual concepts organically mirroring those ideas outside of any medical framing/labelling, whereas it’s perhaps a bit of a course-correct for how psychosis can be understood in countries where it is otherwise highly stigmatised. I have seen lots of UK/US people use ‘psychotic’ or ‘mentally ill’, or even ‘schizophrenic’, as a lay term to mean ‘unhinged’, ‘violent’, etc—it makes sense to me why a diagnosis of schizophrenia can be incredibly challenging and isolating for patients when it must feel misunderstood amongst friends/family/colleagues etc.
So, I wouldn’t necessarily assume that a mass scale-up of healthcare workers in LMICs would be uniformly desirable or even the first priority for managing psychosis in many settings—access to medications might be more pressing, for example. I kept my recommendation to ‘research’ to avoid making specific LMIC suggestions which would be ill-informed without more work on my part. Advances in treatment are helpful regardless as medications are considered first-line in many settings when someone is very unwell, and this does have a positive effect on long-term recovery and wellbeing. I’d be disappointed if current treatment is the best we can do forever and I have hope there are promising developments on the horizon from what I’ve seen. I generally think research in psychosis/schizophrenia is needed to understand these illnesses more completely towards that aim.
That being said, if LMICs identify lack of health workers as their main need, I’m not sure mandated service is the answer and I would be deeply uncomfortable with funding from wealthy countries for education having those strings attached. Simply put, training is expensive, and having a number of years post-qualification for doctors/nurses in LMICs would likely be brilliant for wealthy countries struggling to retain HCWs as this will create a pool of relatively senior HCWs the receiving country won’t even have to train after recruitment. It creates a perverse incentive for this funding, and might degrade working conditions in LMICs (why worry if you know people have to work for you), strip health systems of mostly senior clinicians, and there’s a bigger issue of people in caring roles working with vulnerable people who might feel burned out or hate their jobs but also that they have no choice but to stay—at best, this will just increase mistakes and worsen care, at worst it can lead to abuse and cause sickness and harm in your workforce.
I mean my hands are tied given the industrial action in my profession in the UK, where I’ve heard the same suggestions about keeping doctors in the NHS—I can’t agree with something I vehemently oppose for myself and my future colleagues so I might not be the best person to ask! It’s one of the reasons I’ve found the relative complacency about understanding the retention problems in the UK so frustrating—the issue of knock-on effects to health worker migration is relatively overlooked as a predictable consequence of worsening retention and the mandatory service suggestion seems to dismiss the very real issue with morale and working conditions for doctors and nurses which is causing the problem in the first place. I personally feel countries with the means to do so have an ethical responsibility to address problems in their health workforce to avoid contributing to worsening global health inequalities, and think you can’t really take push or pull factors in isolation to fix the problem.
While I’ve ended up apparently arguing for a pay raise for me and my colleagues (I’m joking!), hopefully I’ve balanced this by mainly talking about my redundancy to say I don’t think our industrial action should be an EA priority either. This might be a roundabout way of saying you’re right to point out my work is lacking practical-level steps and my suggestions are vague/speculative. I suppose I wanted to check my groundwork before risking making any concrete suggestions off of a shaky foundation. I’m expecting this to be trashed imminently, but if it survives in some form and there’s interest I can try taking it a bit further and looking at a couple of my broad areas of potential gain in more detail. [I think anything even approaching a CEA might be embarrassing without some help though!]
That’s kind of you to say—it’s definitely a sobering perspective working in mental health and you end up feeling very strongly (clearly...) about wishing people struggling with mental illnesses had more support.
Of course that makes me biased, and it’s worth saying I’m still learning—if I presented this to my psychiatry trainee colleagues, I’m sure they would all have different takes, let alone more senior clinicians or doctors in other specialities. Clinical work means, of course, we don’t think about ‘evaluating’ illnesses outside of the patients in front of us and it’s highly individualised as a field.
I think that makes it easy to go with my initial reaction to all of this (“Mental health is too complicated, shouldn’t be simplified to numbers, you all don’t understand,” etc). It makes me uneasy to think about ‘comparing’ suffering—it’s much more comfortable to stay railing against the machine in my position, and it’s historically why I’ve not felt aligned with EA or utilitarianism more broadly.
But obviously CEAs happen all the time in medicine, it’s just at a level way over my head so I don’t have to think about it. Reading some of the work that went into the DALY was pretty fascinating to see how people approached this problem on a global health scale (I also favoured the DALY most out of the frameworks I encountered). I think my overall takeaway is a greater sympathy for what EA is trying to do, and I definitely learned a lot in the process—it’s been humbling trying to think from this perspective (even if massively long forum posts are not the usual behaviour of the humble).
[Speaking from a UK perspective with much less knowledge of non-medical psychotherapy training]
I think the importance is having a strong mental health research background, particularly in systematic review and meta-analysis. If you have an expert in this field then the need for clinical experience becomes less important (perhaps, depends on HLI’s intended scope).
It’s fair to say psychology and psychiatry do commonly blur boundaries with psychotherapy as there are different routes of qualification—it can be with a PhD through a psychology/therapy pathway, or there is a specialism in psychotherapy that can be obtained as part of psychiatry training (a bit like how neurologists are qualified through specialism in internal medicine training). Psychotherapists tend to be qualified in specific modalities in order to practice them independently e.g. you might achieve accreditation in psychoanalytic psychotherapy, etc. There are a vast number of different professionals (me included, during my core training in psychiatry) who deliver psychotherapy under supervision of accredited practitioners so the definition of therapist is blurry.
Psychotherapy is similarly researched through the perspective of delivering psychotherapy which perhaps has more of a psychology focus, and as a treatment of various psychiatric illnesses (+/- in combination or comparison with medication, or novel therapies like psychadelics) which perhaps is closer to psychiatric research. Diagnosis of psychiatric illnesses like depression and directing treatment tends to remain the responsibility of doctors (psychiatrists or primary care physicians), and so psychiatry training requires the development of competencies in psychotherapy, even if delivery of psychotherapy does not always form the bulk of day-to-day practice, as it relates to formulating treatment plans for patients with psychiatric illness.
The issues I raise relate to the clinical presentation of depression as it pertains to impairment/wellbeing, diagnosis of depression, symptom rating scales, psychotherapy as a defined treatment, etc.; as well as the wide range of psychopathology captured in the dataset. My feeling is the breadth of this would benefit from a background in psychiatry for the assumptions I made about HLI’s focus of the meta-analysis. However, if the importance is the depth of understanding IPT as an intervention, or perhaps the hollistic outcomes of psychotherapy particularly related to young women/girls in LMICs, then you might want a psychotherapist (PhD or psychiatrist) working with accreditation in the modality or with the population of interest. If you found someone who regularly publishes systematic reviews and meta-analyses of psychotherapy efficacy then that would probably trump both regardless of clinical background. Or perhaps all three is best.
You’re both right to clarify this, though—I was giving my opinion from my background in clinical/academic psychiatry and so I talk about it a lot! When I mention the field of study etc, I meant mental health research more broadly given it depends on HLI’s aims/scope to know what specific area this would be.
[Edit—Sorry, I’ve realised my lack of digging into the background of HLI members/contributors to this research could render the above highly offensive if there are individuals from this field on staff, and also makes me appear extremely arrogant. For clarity, it’s possible all of my concerns were actually fully-rationalised, deliberate choices by HLI that I’ve not understood from my quick sense-check, or I might disagree with but are still valid.
[However, my impression from the work, in particular the design and methodology, is that there is a lack of psychiatric and/or psychotherapy knowledge (given the questions I had from a clinical perspective); and a lack of confidence in systematic review and meta-analysis from how far this deviates from Cochrane/PRISMA that I was trying to explain in more accessible terms in my comment without being exhaustive. It’s possible contributors to this work did have experience in these areas but were not represented in the write-up, or not involved at the appropriate times in the work, etc. I’m not going to seek out whether or not that is the case as I think it would make this personal given the size of the organisation, and I’m worried that if I check I might find a psychotherapy professor on staff I’ve now crossed (jk ;-)).
[It’s interesting to me either way, as both seem like problems—HLI not identifying they lacked appropriate skills to conduct this research, or seemingly not employing those with the relevant skills appropriately to conduct or communicate it—and it has relevance outside of this particular meta-analysis in the consideration of further outputs from HLI, or evaluation of orgs by EA. In any case, peer-review offers reassurance to the wider EA community that external subject-matter expertise has been consulted in whatever field of interest (with the additional benefit of shutting people like me down very quickly), and provides an opportunity for better research if deficits identified from peer-review suggest skills need to be reallocated or additional skills sought in order to meet a good standard.]
Hi everyone,
To fully disclose my biases: I’m not part of EA, I’m Greg’s younger sister, and I’m a junior doctor training in psychiatry in the UK. I’ve read the comments, the relevant areas of HLI’s website, Ozler study registration and spent more time than needed looking at the dataset in the Google doc and clicking random papers.
I’m not here to pile on, and my brother doesn’t need me to fight his corner. I would inevitably undermine any statistics I tried to back up due to my lack of talent in this area. However, this is personal to me not only wondering about the fate of my Christmas present (Greg donated to Strongminds on my behalf), but also as someone who is deeply sympathetic to HLI’s stance that mental health research and interventions are chronically neglected, misunderstood and under-funded. I have a feeling I’m not going to match the tone here as I’m not part of this community (and apologise in advance for any offence caused), but perhaps I can offer a different perspective as a doctor with clinical practice in psychiatry and on an academic fellowship (i.e. I have dedicated research time in the field of mental health).
The conflict seems to be that, on one hand, HLI has important goals related to a neglected area of work (mental health, particularly in LMICs). I also understand the precarious situation they are in financially, and the fears that undermining this research could have a disproportionate effect on HLI vs critiquing an organisation which is not so concerned with their longevity. There might be additional fears that further work in this area will be scrutinised to a uniquely high degree if there is a precedent set that HLI’s underlying research is found to be flawed. And perhaps this concern is compounded by the stats from people in this thread, which perhaps is not commonly directed to other projects in the EA-sphere, and might suggest there is an underlying bias against this type of work.
I think it’s fair to hold these views, but I’d argue this is likely the mechanism by which HLI has escaped scrutiny before now – people agree more work and funding should be directed to mental health and wanted to support an organisation addressing this. It possibly elevated the status of HLI in people’s minds, appearing more revolutionary in redirecting discussions in EA as a whole. Again, Greg donated to Strongminds on my behalf and, while he might now feel a sense of embarrassment for not delving into this research prior, in my mind I think it reflects a sense of affirmation in this cause and trust in this community which prides itself on being evidence-based. I’m mentioning it, because I think everyone here is united on these points and it’s always easier to have productive discussions from the mutual understanding of shared values and goals.
However, there are serious issues in the meta-analysis which appears to underlie the CEA, and therefore the strength of claims made by HLI. I think it is possible to uncouple this statement from arguments against HLI or any of the above points (where I don’t see disagreement). It seems critical to acknowledge the flaws in this work given the values of EA as an objective, data-driven approach to charitable giving. Failing to do this will risk the reputation of EA, and suggest there is a lack of critical appraisal and scrutiny which perhaps is driven by personal biases i.e. the number of reassurances in this thread that HLI is a good organisation where members are known personally to others in the community. Good people with good intentions can produce flawed research. Similarly, from the perspective of a clinical academic in psychiatry, there is a long history in my field of poorly-conducted, misinterpreted and rushed research which has meant establishing evidence-based care and attracting funding for research/interventions particularly difficult. Poor research in this area risks worsening this problem and mis-allocating very limited resources – it’s fairly shocking seeing the figures quoted here in terms of funding if it is based wholly or in part on outputs such as this meta-analysis which were accepted by EA. Again, as an outsider, it’s difficult for me to judge how critical this research was in attracting this allocation of funds.
While I think the issues with the analysis and all the statistics discussions are valid critiques of this work, it’s important to establish that this is only part of the reason this study would fall down under peer review. It’s concerning to me that peer-review is not the standard for organisations supported by EA; this is not just about scrutinising how the research was conducted and arguing about statistics, but establishes the involvement of expertise within the field of study. As someone who works in this field, the assumptions this meta-analysis makes about psychotherapy, outcome measures in mental health, etc, are problematic but perhaps not readily identified to those without a clinical background, and this is a much greater problem if there is an increasing interest in addressing mental health within EA. I’m not familiar with the backgrounds of people involved in HLI, but I’d be curious about who was consulted in formulating this work given the tone seems to reflect more philosophical vs psychiatric/psychotherapeutic language.
The way the statistical analysis has been heavily debated in this thread likely reflect the skills-mix in the EA community (clearly stats are well-covered!), but the statistics are somewhat irrelevant if your study design and inputs into the analysis are flawed to start with. Even if the findings of this research were not so unusual (perhaps something else which could have been flagged sooner) or were based on concrete stats, the research would still be considered flawed in my field. I imagine this will prompt some reflection in EA on this topic, but peer-review as a requirement could have avoided the bad timing of these discussions and would reduce the reliance on community members to critique research. I think this thread has demonstrated that critical appraisal is time-intensive and relies on specialist skills – it’s not likely that every area of interest will have representation within the EA community so the problem of ‘not knowing what you don’t know’ or how you weight the importance of voices in the community vs their amplification would be greatly helped by peer-review and reduce these blind spots. If the central goal of EA is using money to do the most good, and there is no robust system to evaluate research prior to attracting funding, this is an organisational problem rather than a specific issue with HLI/Strongminds.
My unofficial peer review.
Given inclusion/exclusion criteria aren’t stated clearly in the meta-analysis and the aim is pretty woolly, It seems the focus of the upcoming RCT and Strongminds research is evaluating:
Training non-HCPs in delivering psychotherapy in LMICs
Providing treatment (particularly to young women and girls) with symptoms suggestive of moderate to severe depression (PHQ-9 score of 10 and above)
Measuring the efficacy of this treatment on subjective symptom rating scales, such as PHQ-9, and other secondary outcome measures which might reflect broader benefits not captured in the symptom rating scales.
Finding some way to compare the cost-effectiveness of this treatment to other interventions such as cash transfers in broader discussions of life satisfaction and wellbeing which it obviously complicated compared to using QALYs, but important to do as the impact of mental illness is under-valued using measures geared towards physical morbidity. Or maybe it’s trying to understand effectiveness of treating symptoms vs assumed precipitating/perpetuating factors like poverty.
Grand.
However, the meta-analysis design seems to miss the mark on developing anything which would support a CEA along these lines. Even from the perspective of favouring broad inclusion criteria, you would logically set these limits:
Population
LMIC setting, people with depressive symptoms. It’s not clear if this is about effectively treating depression with psychotherapy and extrapolating that to a comment on wellbeing; or using psychotherapy as a tool to improve wellbeing, which for some reason is being measured in a reduction in various symptom scales for different mental health conditions and symptoms – this needs to be clearly stated. If it’s the former, what you accept as a diagnosis of depression (ICD diagnostic codes, clinical assessment by trained professional, symptom scale cut-offs, antidepressant treatment, etc) should be defined.
If not defining the inclusion criteria of depression as a diagnosis, it’s worth considering if certain psychiatric/medical conditions or settings should be excluded e.g. inpatients. As a hypothetical, extracting data on depression symptom scales for a non-HCP delivered psychotherapy in bipolar patients will obviously be misleading in isolation (i.e. the study likely accounted for measuring mania symptoms in their findings, but would be lost in this meta-analysis). One study included in this analysis (Richter et al) looked at an intervention which encouraged adherence to anti-retroviral medications via peer support for women newly diagnosed with HIV. Fortunately, this study shouldn’t have been included as it didn’t involve delivering psychotherapy, but for the sake of argument, is that fair given the neuropsychiatric complications of HIV/AIDS? Again, it’s not about preparing for every eventuality, but it’s having clear inclusion/exclusion criteria so there’s no argument about cherry-picking studies because this has been discussed prior to search and analysis.
Intervention
Delivery of a specific psychotherapeutic modality (IPT, etc) by a non-HCP. While I can agree there are shared core concepts between different modalities of psychotherapy, you absolutely have to define what you mean by psychotherapy because your dataset containing a column labelled ‘therapyness’ (high/medium/low) undermines a lot of confidence, as do some of the interventions you’ve included as meeting the bar for psychotherapy treatment. If you want to include studies which perhaps are not focussed on treating depression and might therefore involve other forms of therapy but still have benefit in alleviating depressive symptoms e.g. where the presenting complaint is trauma, the intervention might be EMDR (a specific therapy for PTSD) but the authors collected a number of outcome measures including symptom rating scales for anxiety and depression as secondary outcomes, it would be logical to stratify studies in this manner as a plan for analysis. I.e. psychotherapeutic intervention with evidence-base in relieving depressive symptoms (CBT, IPT, etc), psychotherapeutic intervention not specifically targeted at depressive symptoms (EMDR, MBT etc), with non-(psychotherapy) intervention as the control.
Several studies instead use non-psychotherapy as the intervention under study and this confusion seems to be down to papers describing them as having a ‘psychotherapeutic approach’ or being based on principles in any area of psychotherapy. This would cover almost anything as ‘psychotherapeutic’ as an adjective just means understanding people’s problems through their internal environment e.g. thoughts, feelings, behaviours and experiences. In my day-to-day work, I maintain a psychotherapeutic approach in patient interactions, but I do not sit down and deliver 14-week structured IPT. You can argue that generally having a supportive environment to discuss your problems with someone who is keen to hear them is equally beneficial to formal psychotherapy, but this leads to the obvious question of how you can use the idea of any intervention which sounds a bit ‘psychotherapy-y’ to justify the cost of training people to specifically deliver psychotherapy in a CEA from this data.
The fundamental lack of definition or understanding of these clinical terms leads to odd issues in some of the papers I clicked on i.e. Rojas et al (2007) compares a multicomponent group intervention involving lots of things but notably not delivery of any specific psychotherapy, to normal clinical care in a postnatal clinic. The next sentence describes part of normal clinical care to be providing ‘brief psychotherapeutic interventions’ – perhaps this is understood by non-clinicians as not highly ‘therapyish’ but this term is often used to describe short-term focussed CBT, or CBT-informed interventions. Not defining the intervention clearly means the control group contains patients receiving evidence-based psychotherapy of a specific modality and a treatment arm of no specific psychotherapy which is muddled by the MA.
Comparison
As alluded to above, you need to be clear about what is an acceptable control and it’s simply not enough to state you are not sure what the ‘usual care’ is in research by Strongminds you have weighted so heavily. It can’t be then justified by an assumption mental health is neglected in LMICs so probably wouldn’t involve psychotherapy (with no citation). Especially as the definition of psychotherapy in this meta-analysis would deem someone visiting a pastor in church once a week as receiving psychotherapy. Without clearly defining the intervention, it’s really difficult to understand what you are comparing against what.
Outcome
This meta-analysis uses a range of symptom rating scales as acceptable outcome measures, favouring depression and anxiety rating scales, and scales measuring distress. This seems to be based on idea that these clusters of symptoms are highly adverse to wellbeing. This makes the analysis and discussion really confused, in my opinion, and seems to be a sign the analysis, expected findings, extrapolation to wellbeing and CEA was mixed into the methodology.
To me, the issue arises from not clearly defining the aim and inclusion/exclusion criteria. This meta-analysis could be looking at psychotherapy as a treatment for depression/depressive symptoms. This would acknowledge that depression is a psychiatric illness with congitive, psychological and biological symptoms (as captured by depression rating scales). As a clinical term, it is not just about ‘negative affect’ - low mood is not even required for a diagnosis as per ICD criteria. It absolutely does negatively affect wellbeing, as would any illness with unpleasant/distressing symptoms, but this therefore means generating some idea for how much patients’ wellbeing improves from treatment has to be specific to depression. The subsequent CEA would then need to account for this and evaluate only psychotherapies with an evidence-base in depression. In the RCT design, I’d guess this is the rationale for a high PHQ cut-off—it’s a proxy for relative certainty in a clinical diagnosis of depression (or at least a high burden of symptoms which may respond to depression treatments and therefore demonstrate a treatment effect); it’s not supporting the idea that some general negative symptoms impacting a concept of wellbeing, short of depression, will likely benefit from specific psychotherapy to any degree of significance, and it would be an error to take this assumption and then further assume a linear relationship between PHQ and wellbeing/impairment.
If you are looking at depressive symptom reduction, you need to only include evaluation tools for depressive symptoms (PHQ, etc). You need to define which tools you would accept prior to the search and that these are validated for the population under study as you are using them in isolation—how mental illness is understood and presents is highly culturally-bound and these tools almost entirely developed outside of LMICs.
If, instead, you’re looking at a range of measures you feel reflect poor mental health (including depression, anxiety and distress) in order to correlate this to a concept of wellbeing, these tools similarly have to be defined and validated. You also need to explain why some tools should be excluded, because this is unclear e.g. in Weiss et al, a study looking at survivors of torture and militant attacks in Iraq, the primary outcome measure was a trauma symptom scale (the HTQ), yet you’ve selected the secondary outcome measures of depression and anxiety symptom scores for inclusion. I would have assumed that reducing the highly distressing symptoms of PTSD in this group would be most relevant to a concept of wellbeing, yet that is not included in favour of the secondary measures. Including multiple outcome measures with no plan to stratify/subgroup per symptom cluster or disorder seems to accept double/triple counting participants who completed multiple outcome measures from the same intervention. Importantly, you can’t then use this wide mix of various scales to make any comment on the benefits of psychotherapy for depression in improving wellbeing (as lots of the included scores are not measuring depression).
In both approaches, you do need to show it is accepted to pool these different rating scales to still answer your research question. It’s interesting to state you favour subjective symptom scores over functional scores (which are excluded), when both are well-established in evaluating psychotherapy. Other statements made by HLI suggest symptom rating scores include assesment of functioning—I’ve reproduced the PHQ-9 below for people to draw their own conclusions, but it’s safe to say I disagree with this. It’s not clear to me if it’s understood functional scores are also commonly subjective measures, like the WSAS—patients are asked how rate how well they feel they are managing work activities, social activities etc. Ignoring functioning as a blanket rule seems to miss the concept of ‘insight’ in mental health, where people can struggle identifying symptoms as symptoms but are severely disabled due to an illness (this perhaps should also be considered in excluding scales completed by an informant or close relative, particularly thinking about studies involving children or more severe psychopathology). Incorporating functional scoring captures the holistic nature of psychotherapy, where perhaps people may still struggle with symptoms of anxiety/depression after treatment, but have made huge strides in being able to return to work. Again, you need to be clear why functional scores are excluded and be clear this was done when extrapolating findings to discussions of life satisfaction or wellbeing. This research has made a lot of assumptions in this regard that I don’t follow.
x. Grouping measures and relating this to wellbeing:
On that note – using a mean change in symptoms scores is a reasonable evaluation of psychotherapy as a concept if you are so inclined but I would strongly argue that this cannot be used in isolation to make any inference about how this correlates to wellbeing. As others have alluded to in this thread, symptom scores are not linear. To isolate depression as an example, this is deemed mild/moderate/severe based on the number of symptoms experienced, the presence of certain concerning symptoms (e.g. psychosis) and the degree of functional impact.
Measures like the PHQ-9 score the number of depressive symptoms present and how often they occur from 0 (not at all) to 3 (nearly every day) over the past two weeks:
Little interest or pleasure in doing things?
Feeling down, depressed or hopeless?
Trouble falling or staying asleep, or sleeping too much?
Feeling tired or having little energy?
Poor appetite or overeating?
Feeling bad about yourself—or that you are a failure or have let yourself or your family down?
Trouble concentrating on things, such as reading the newspaper or watching television?
Moving or speaking so slowly that other people have noticed? Or the opposite—being so fidgety or restless that you have been moving around a lot more than usual?
Thoughts that you would be better off dead, or of hurting yourself in some way?
If you take the view that a symptom rating score has a linear relationship to ‘negative affect’ or suffering in depression, you would then imagine that the outcomes of PHQ-9 (no depression, mild, moderate, severe) would be regularly distributed in the 27-item score i.e. a score of 0-6 should be no depression, 7-13 mild depression, 14-20 moderate depression and 21-27 severe. This is not the case as the actual PHQ-9 scores are 0-4 no depression, 5-9 mild depression, 10-14 moderate, 15-19 moderately severe, 20-27 severe. This is because the symptoms asked about in the PHQ are diagnostic for depression – it’s not an attempt at trying to gather how happy or sad someone is on a scale from 0-27 (in fact 0 just indicates ‘no depression symptoms’, not happiness or fulfilment, and it’s likely people with very serious depression will not be able to complete a PHQ-9). Hopefully it’s clear from the PHQ-9 why the cut-offs are low and why the severity increases so sharply; the symptoms in question are highly indicative of pathology if occuring frequently. It’s also in the understanding that a PHQ-9 would be administered when there is clinical suspicion of depression to elicit severity or in evaluation of treatment (i.e. in some contexts, like bereavement, experiencing these symptoms would be considered normal, or if symptoms are better explained by another illness the PHQ is unhelpful) and it’s not used for screening (vs the Edinburgh score for postnatal depression which is a screening tool and features heavily in included studies). Critically, it’s why you can’t assume it’s valid to lump all symptom scales together, especially cross-disorders/symptom clusters as in this meta-analysis.
x. Search strategy
I feel this should go without saying, but once you’ve ironed out these issues to have a research question you could feasibly inform with meta-analysis, you then need to develop a search strategy and conduct a systematic review. It’s guaranteed that papers have been missed with the approach used here, and I’ve never read a peer-reviewed meta-analysis where a (10-hour) time constraint was used as part of this strategy. While I agree the funnel plot is funky, it’s likely reflecting the errors in not conducting this search systematically rather than assuming publication bias – it’s likely the highly cited papers etc were more easily found using this approach and therefore account for the p-clustering. If the search was a systematic review and there were objective inclusion/exclusion criteria and the funnel plot looked like that, you can make an argument for publication bias. As it stands, the outputs are only as good as the inputs i.e. you can’t out-analyse a poor study design/methodology to produce reliable results.
Simply put, the most critical problem here is that without even getting into the problems with the data extraction I found, or the analysis as discussed in this thread, from this study design which doesn’t seek to justify why any of these decisions were made, any analytic outputs are destined to be unreliable. How much of this was deliberate on the part of HLI can’t be determined as there is no possible way of replicating the search strategy they used (this is the reason to have a robust strategy as part of your study design). I think if you want to call this a back-of-napkin scoping review to generate some speculative numbers, you could describe what you found as there being early signals that psychotherapy could be more cost-effective than assumed and therefore there’s need to conduct a vigorous SR/MA. It perhaps may have been more useful in a shallow review to actually exclude the Strongminds study and evaluate existing research through the framework of (1) do the SM results make sense in the context of previous studies and (2) can we explain any differences in a narrative review. It seems instead this work generated figures which were treated as useful or reliable and fed into a CEA, which was further skewed by how this was discussed by HLI.
TL;DR
This is obviously very long and not going to be read in any detail on an online forum, but from the perspective of someone within this field, there seem to be a raft of problems with how this research was conducted and evaluated by HLI and EA. I’m not considered the Queen Overload of Psychiatry, I don’t have a PhD, but I suppose I’m trying to demonstrate that having a different background raises different questions, which seems particularly relevant if there is a recognition of the importance of peer-review (hopefully, I’m assuming, outside of EA literature). I’m also going to caveat this by saying I’ve not poured over HLI’s work, it’s just what immediately stood out to me, and haven’t made any attempt to cite my own knowledge derived from my practice – to me this is a post on a forum I’m not involved with rather than an ‘official’ attempt at peer review so I’m not holding myself to the same standard, just commenting in good faith.
I get the difficult position HLI are in with reputational salvage, but there is a similar risk to EA’s reputation if there are no checks in place given this has been accessible information for some time and did not raise questions earlier. While this might feel like Greg’s younger sister joining in to dunk on HLI, and I see from comments in this thread that perhaps criticism said passionately can be construed as hostile online, I don’t think this is anyone’s intent. Incredibly ironically given our genetic aversion to team sports, perhaps critique is intended as a fellow teammate begging a striker to get off the field when injured as they are hurting themselves and the team. Letting that player limp on is not being a supportive teammate. Personally, I hope these discussions drive discussions in HLI and EA which provide scope for growth.
In my unsolicited and unqualified opinion, I would advise withdrawing the CEA and drastically modifying the weight HLI puts on this work so it does not appear to be foundational to HLI as an organisation. Journals are encouraging the submission of meta-analysis study protocols for peer-review and publication (BMJPsych Open is one – I have acted as a peer reviewer for this journal to be transparent) in order to improve the quality of research. While conducting a whole SR/MA and publication takes time which could allow further loss of reputation, this is a quick way of acknowledging the issues here and taking concrete steps to rectify them. It’s not acceptable, to me, for the same people to offer a re-analysis or review this work because I’m sceptical this would not produce another flawed output and it seems there is a real need to involve expertise from the field of study (i.e. in formal peer review) at an earlier stage to right the ship.
Again, I do think the aims of HLI are important and I do wish them the best; and I’m interested to see how these discussions evolve in EA as it seems straying into a subject I’m passionate about. I come in peace and this feedback is genuinely meant constructively, so in the spirit of EA and younger-sibling disloyalty, I’m happy to offer HLI help above what’s already provided if they would like it.
[Edit for clarity mostly under ‘outcomes’ and ‘grouping measures’, corrected my horrid formatting/typos, and included the PHQ-9 for context. Kept my waffle and bad jokes for accountability, and was using the royal ‘you’ vs directing any statements at OP(!)]
For UK universities (I see a few have EA clubs) - it is really weird that student volunteers receive individual funding. I think this applies to US as well but can’t be 100% sure:
UK student clubs fall under the banner of their respective student union, which is a charitable organisation to support the needs, interests and development of the students at the university. They have oversight of clubs, and a pot of money that clubs can access (i.e. they submit a budget for their running costs/events for the year and the union decides what is/isn’t reasonable and what it can/can’t fund). They also have a platform to promote all clubs through the union website, Freshers’ week, university brochures, etc.
Some external organisations sponsor clubs. This is usually to make up ‘gaps’ in funding from the union e.g. If a bank wanted to fund a finance club so they can provide free caviar and wine at all events to encourage students to attend, in return for their logo appearing in club newsletters, this makes sense; the union would not be funding the ‘caviar and wine’ line item in the budget as this is not considered essential to supporting the running of the finance club as per the union’s charitable aims (and they have 100s of clubs to fund).
Student clubs do really impressive things and often need support in achieving this e.g. if a club wanted to run a pan-London summer school to support widening access to STEM it’s likely this would be supported by the union and corporate sponsorship to cover costs. They can also access free support/time from student union staff on the operational/finance side. The students wouldn’t be paid to run the club though, and are often recognised for their extraordinary voluntary service in other ways e.g. student awards, etc.
The PETA program you linked to is paying individual students arrange ad-hoc protests etc on campuses, and will pay a reasonably small sum and provide materials to support that. It’s not under a banner of a student club (from what I can tell i.e. not Oxford PETA). It’s different than PETA paying Oxford PETA’s President $600 a semester just for being President. It also isn’t paying someone to set up the club or keep it going year-on-year by ensuring committee members (which should be based on enough students having an interest).
It seems OP was paid in his role as the latter i.e. to run a club at a university (a voluntary position). If so, I share OP’s assessment this is weird—there’s something about it which feels uncomfortable: wondering what people are being paid for, how they are selected, if this is a salary (with a contract and protections) or just a golden handshake, how that changes dynamics within the club, or potential conflicts of interest e.g. being asked to run an EA conference in finals week, how safe a student would feel in declining if they had accepted a large amount of money throughout the year.
I’m curious how much money these roles are attracting and if this changes between universities (even a ballpark figure would help)? I’d be curious to know if this is generally known at universities and what evidence there is this helps—personally, if I was back in Freshers’ Week and I came across an EA stall, it would markedly downgrade my estimation of EA to learn the otherwise ‘volunteer’ positions in other fantastic clubs were paid here i.e. I’m not seeing passionate students devoting their time to a cause they care about, it’s also/mainly a money-making venture (especially with OP’s description of ‘outrageous’). I’m perplexed why this is being done at all—EA could be funding conferences, retreats, etc without this type of ‘weird’, and surely there is sufficient interest within students to not have to pay volunteers.
[You’re not the only person to make this point, so please don’t think I’m challenging you personally as I pretty much agree with your stance on everything except for the ‘weirdness’ - I’ve just seen a few misconceptions in this thread about uni clubs and thought I could clear some up here vs a new comment.]