In an attempt to avoid this confusion, we denoted all the effects as changes in ‘SDs’ or ‘SD-years’ of subjective wellbeing / affective mental health in all the reports (1,2,3,4,5) that were direct results in the intervention comparison.
This is exactly what confused me. In all the analytical pieces (and places linked to in the reports defining WELLBY on the 0-10 scale) you use SD but then there’s a chart which uses WELLBY and I couldn’t find where you convert from one to another.
That standard deviation differences (I.e., Cohen’s d or Hedges g effect sizes) are reasonable ways to do meta-analyses?
I think this is a very reasonable way to do meta-analyses
Or is your concern more that even if SDs are reasonable for meta-analyses, they aren’t appropriate for comparing the effectiveness of interventions? We flag some possible concerns in Section 7 of the psychotherapy report. But we haven’t found sufficient evidence after several shallow dives to change our minds.
Yes. This is exactly my confusion, specifically:
A potential issue with using SD changes is that the mental health (MH) scores for recipients of different programmes might have different size standard deviations – e.g. SD could be 15 for cash transfers and 20 for psychotherapy, on a given mental health scale. We currently do not have much evidence on this. If we had more time we would test and adjust for any bias stemming from differences in variances of psychological distress between intervention samples by comparing the average SD for equivalent measures across intervention samples
In the absence of evidence my prior is very strong that a group of people selected to have a certain level of depression is going to have a lower SD than a group of randomly sampled people. This is exactly my confusion. Furthermore, I would expect the SD of “generally healthy people” to be quite low and interventions to have low impact. For example, giving a health person an PS5 for Christmas might massively boost their subjective well-being, but probably doen’t do much for mental health. (This is related to your third point, but is more about the magnitude of changes I’d expect to see rather than anything else)
Or, you may be concerned that similar changes in subjective wellbeing and affective mental health don’t represent similar changes in wellbeing? (We discuss this in Appendix A of the psychotherapy report).
So I also have issues with this, although it’s not the specific issue I’m raising here.
Or is it something else I haven’t articulated?
Nope—it’s pretty much exactly point 2.
Most of these issues are technical, and we recognise that our views could change with further work. However, we aren’t convinced there’s a ready-to-use method that is a better alternative for use with subjective wellbeing analyses.
Well, my contention is subjective wellbeing analyses shouldn’t be the sole basis for evaluation (but again, that’s probably a separate point).
I also welcome further explanation of your issues with our analysis, public or private. If you’d like to have low stakes chat about our work, you can schedule a time here. If that doesn’t work, email or message me, and we can make something work.
Thanks! I’ve (hopefully) signed up to speak to you tomorrow
This is exactly what confused me. In all the analytical pieces (and places linked to in the reports defining WELLBY on the 0-10 scale) you use SD but then there’s a chart which uses WELLBY and I couldn’t find where you convert from one to another.
I think this is a very reasonable way to do meta-analyses
Yes. This is exactly my confusion, specifically:
In the absence of evidence my prior is very strong that a group of people selected to have a certain level of depression is going to have a lower SD than a group of randomly sampled people. This is exactly my confusion. Furthermore, I would expect the SD of “generally healthy people” to be quite low and interventions to have low impact. For example, giving a health person an PS5 for Christmas might massively boost their subjective well-being, but probably doen’t do much for mental health. (This is related to your third point, but is more about the magnitude of changes I’d expect to see rather than anything else)
So I also have issues with this, although it’s not the specific issue I’m raising here.
Nope—it’s pretty much exactly point 2.
Well, my contention is subjective wellbeing analyses shouldn’t be the sole basis for evaluation (but again, that’s probably a separate point).
Thanks! I’ve (hopefully) signed up to speak to you tomorrow