Fwiw, I think that it might be better if you were to simply drop the “Strength of effect: %” column from your sheet and not rank interventions according to this.
As an earlier commenter pointed out this is comparing “% reductions” in very different things (e.g. percentage reduction in cortisol vs percentage change in stress scores). But it also seems like this is going to be misleading in a variety of other ways. As far as I can tell, it’s not only comparing different metrics for different interventions, it’s sometimes combining percentage changes in different metrics to generate a single percentage change score for the same intervention. This systems also means that some of the most relevant pieces of evidence also get left out (e.g. the effect sizes for the meta-analyses for CBT), because they’re reported as standardised effect sizes, rather than percentage change. I would probably also drop ‘percentage of studies [we reviewed ] where results were statistically significant’ as a metric since this seems likely to be misleading (I also wasn’t sure how this was working in some cases e.g. CBT has 86 studies reviewed, but it seems to have 2⁄2 studies with significant results. Is this counting the two meta-analyses as single studies?).
I think it might be better to do this and instead take the approach of presenting this as a collection of possible interventions with some evidence people could evaluate and potentially try (collating the evidence so people can evaluate it themselves) rather than trying to rank the interventions according to simple, but likely to be misleading metrics. If you were to lean into this approach you could also include a lot more potential interventions which have at least as much of an evidence base as things listed here. For example, Examine.com reviews over 20 interventions for stress (and more for anxiety, mood, depression) and there are plenty of things which could be included (for example, here’s a meta-analysis and systematic review of b-vitamins for stress).
I think there can be reasonable disagreement about whether it’s good to include more things, with less review and/or a lower bar for evidence, or fewer things with more review and/or a stricter bar for evidence. But in this case it seems like it may not be feasible within your project to provide meaningful review of the evidence for the different interventions and it seems like a lot of things are included which aren’t better evidenced than things which are excluded.
I appreciate the detailed notes/ feedback on the research process. I think the points you make are very reasonable and definitely helpful.
I expect that I will come back to this stress article, and the sleep one before that, to improve the quality of recommendations through a better research process. I can see a number of ways to do that including points that you bring up. - I’d like the evidence tables to cover 90+% of interventions that appear valuable from the outside/ have been suggested elsewhere, which I agree is not a bar I think this article reaches. - Leaving out the ‘% of studies that were significant’ makes good sense, especially given it’s quite inaccurate as I’ve put it here in its treatment of meta-analyses. - I’d like to keep some form of prioritisation by effect size. I think there’s probably a better way of doing this than what I’ve done here but prioritising interventions seems particularly valuable to me in terms of increasing the practicality of the recommendations and the likelihood a reader starts doing something useful. Perhaps it would be better to include the effect sizes as a range and highlight the different measures used (e.g. subjective stress vs. cortisol) to make this issue clearer to the reader. More work to do though on figuring this out.
Thanks for collating these different ideas!
Fwiw, I think that it might be better if you were to simply drop the “Strength of effect: %” column from your sheet and not rank interventions according to this.
As an earlier commenter pointed out this is comparing “% reductions” in very different things (e.g. percentage reduction in cortisol vs percentage change in stress scores). But it also seems like this is going to be misleading in a variety of other ways. As far as I can tell, it’s not only comparing different metrics for different interventions, it’s sometimes combining percentage changes in different metrics to generate a single percentage change score for the same intervention. This systems also means that some of the most relevant pieces of evidence also get left out (e.g. the effect sizes for the meta-analyses for CBT), because they’re reported as standardised effect sizes, rather than percentage change. I would probably also drop ‘percentage of studies [we reviewed ] where results were statistically significant’ as a metric since this seems likely to be misleading (I also wasn’t sure how this was working in some cases e.g. CBT has 86 studies reviewed, but it seems to have 2⁄2 studies with significant results. Is this counting the two meta-analyses as single studies?).
I think it might be better to do this and instead take the approach of presenting this as a collection of possible interventions with some evidence people could evaluate and potentially try (collating the evidence so people can evaluate it themselves) rather than trying to rank the interventions according to simple, but likely to be misleading metrics. If you were to lean into this approach you could also include a lot more potential interventions which have at least as much of an evidence base as things listed here. For example, Examine.com reviews over 20 interventions for stress (and more for anxiety, mood, depression) and there are plenty of things which could be included (for example, here’s a meta-analysis and systematic review of b-vitamins for stress).
I think there can be reasonable disagreement about whether it’s good to include more things, with less review and/or a lower bar for evidence, or fewer things with more review and/or a stricter bar for evidence. But in this case it seems like it may not be feasible within your project to provide meaningful review of the evidence for the different interventions and it seems like a lot of things are included which aren’t better evidenced than things which are excluded.
I appreciate the detailed notes/ feedback on the research process. I think the points you make are very reasonable and definitely helpful.
I expect that I will come back to this stress article, and the sleep one before that, to improve the quality of recommendations through a better research process. I can see a number of ways to do that including points that you bring up.
- I’d like the evidence tables to cover 90+% of interventions that appear valuable from the outside/ have been suggested elsewhere, which I agree is not a bar I think this article reaches.
- Leaving out the ‘% of studies that were significant’ makes good sense, especially given it’s quite inaccurate as I’ve put it here in its treatment of meta-analyses.
- I’d like to keep some form of prioritisation by effect size. I think there’s probably a better way of doing this than what I’ve done here but prioritising interventions seems particularly valuable to me in terms of increasing the practicality of the recommendations and the likelihood a reader starts doing something useful. Perhaps it would be better to include the effect sizes as a range and highlight the different measures used (e.g. subjective stress vs. cortisol) to make this issue clearer to the reader. More work to do though on figuring this out.