While I am as much a fan of wellbeing research as the next fellow—indeed, probably a much bigger fan—I have to say I found the methodology and conclusions of this research rather confusing.
If I were approaching this topic, I would have (1) done a review of the existing literature to find out what people thought was effective and what the possible interventions were, then (2) tried to assess the options in terms of (a) a comparable metric of effectiveness and (b) cost, so readers could think about what would do the most for them at the least effort.
As it is, this research seems to have missed out many of the standard pieces of advice like avoiding alcohol, napping after 3pm, having a large meal before bed, or having a sleep routine. The author doesn’t mention having looked at the existing literature, but does note that other EAs have mentioned sleep. I don’t mean to single out the OP, but I do want to deride the myopic and self-referential tendency among effective altruists in general to overlook work done outside effective altruism. Lots of good has happened ‘out there’ and we ignore it at our peril.
What I found least satisfying about this research was how this (partial list of) interventions were assessed. As far as I can see, the ‘weighted-factor model’ involved assigning unexplained subjective numbers to various seemingly-arbitrarily chosen properties, then assigning a seemingly-arbitrary weight to each factor to aggregate them. * I am reminded of “Garbage-In-Garbage-Out” concept in computer science where nonsense inputs products nonsense outputs. As a reader, I have no idea how to interpret the rankings or numbers—what does it mean that melatonin gets “5.95/10″ or that CBT-I gets “5.78”? - or how much to update off them. The results are basically uninterpretable.
I would strongly recommend that the OP heavily revise their methods and the presentation of their research for any further work. The main thing would be to present the results of the interventions in a standardised metric, e.g. total sleep time, or standard deviations of something, so readers can make a comparison themselves, then make comments on cost and, if necessary research quality. I am happy to provide advice if that’s helpeful.
*I recognise the weighted-factor model is something Charity Entrepreneurship use. I have raised it with them several times that, for the reasons given, I find this approach hard to follow or justify and thus questionably useful.
There’s a few points you make that I feel are important to clarify but I want to first acknowledge that the format and presentation of the research is a work in progress and could definitely be improved. In particular, I can agree that the “5.95/10” numbers aren’t particularly useful given the lack of any context/ scale.
To respond quickly to a few specific points: 1) This research was overwhelmingly based on the existing literature. I chose not to include a reference list or in-text citations to maximise the readability of the text but perhaps this was an error. 2) The interventions’ strength of effect, which accounts for about 40% of the score, is an average of improvements in sleep efficiency and total sleep time found in the literature I reviewed. In hindsight, I think it could have been better to highlight these findings on their own in the text and may add them in. 3) The focus of this research was to explore interventions that the reader is less likely to have heard of previously. I assumed that ideas like avoiding alcohol and late naps are things that nearly all readers would already know. On that basis, I felt that highlighting them in an article like this is unlikely to produce any change in behaviour, though I can see now the potential usefulness of presenting the size of effect. 4) I think an article like this recommending low-cost, personal interventions can reasonably have a different approach and level of rigour to one recommending charitable interventions and/or shifting large sums of money. A weighted factor model may not be the best way to frame this research but I think additional considerations like the potential risks and additional benefits of a recommendation are important and necessary to highlight.
These points make me realise that a more explicit description of the methods used and the literature reviewed would be valuable for future posts, rather than linking to them in out-of-text docs or leaving them out for a marginal improvement to the conciseness of the text.
This is the first post intended in a series and I expect to revise and improve the methods involved with each post and certainly the feedback I get on posts helps to direct that process. As a first attempt, some of the process was not as rigorous as it could or perhaps should be. In part, this is a relatively time-limited project for now (~3 months) so I am sacrificing some potential added depth in each post for the ability to cover more topics.
On a final note, I have immense respect and appreciation for the work of HLI and so really appreciate the feedback from someone who does wellbeing research at a much higher level!
Good to know this was based on existing literature. In most cases, it helps to show the reader you know that literature, to outline what it is, and then go on say what your new contribution is. Like I say, you missed a few of the obvious things, which is unfortunate. A piece of “what works for X” should, I say, include the things that work for X, then perhaps go on to flag which of these are likely to be a surprise, rather than assuming on the reader’s behalf what they will already know. If you are going to have a piece on “what works for X that might but that might surprise you” you should at least clearly flag that, and then point to something such as “standard guidance on X”.
Re strength of interventions being “40%” that still seems a confused way of presenting the information. 40% of what? Of a maximum score? A maximum score of what? Of cost-effectiveness? Well, why not just present the effectiveness numbers and divide them by the costs then?
I agree that this sort of thing can have a lower level of rigour but I stand by my concern that the method you use is so puzzling it’s questionably useful at all. You gathered quite a bit of relevant info, but I think you presented it in a less-than-ideal way. Here, simpler would have been better: I’d have preferred a post that just said “here’s a list of evidence-based ways to improve sleep” and then listed them and provided a brief discussion on each. That seems the way to go unless you have the data and time to do a quantitative (cost-)effectiveness analysis.
Glad you think we (at HLI) do good work. Like I see, feel free to reach out if you want to chat about research methods etc.! You can get me at michael@happierlivesinstitute.org
While I am as much a fan of wellbeing research as the next fellow—indeed, probably a much bigger fan—I have to say I found the methodology and conclusions of this research rather confusing.
If I were approaching this topic, I would have (1) done a review of the existing literature to find out what people thought was effective and what the possible interventions were, then (2) tried to assess the options in terms of (a) a comparable metric of effectiveness and (b) cost, so readers could think about what would do the most for them at the least effort.
As it is, this research seems to have missed out many of the standard pieces of advice like avoiding alcohol, napping after 3pm, having a large meal before bed, or having a sleep routine. The author doesn’t mention having looked at the existing literature, but does note that other EAs have mentioned sleep. I don’t mean to single out the OP, but I do want to deride the myopic and self-referential tendency among effective altruists in general to overlook work done outside effective altruism. Lots of good has happened ‘out there’ and we ignore it at our peril.
What I found least satisfying about this research was how this (partial list of) interventions were assessed. As far as I can see, the ‘weighted-factor model’ involved assigning unexplained subjective numbers to various seemingly-arbitrarily chosen properties, then assigning a seemingly-arbitrary weight to each factor to aggregate them. * I am reminded of “Garbage-In-Garbage-Out” concept in computer science where nonsense inputs products nonsense outputs. As a reader, I have no idea how to interpret the rankings or numbers—what does it mean that melatonin gets “5.95/10″ or that CBT-I gets “5.78”? - or how much to update off them. The results are basically uninterpretable.
I would strongly recommend that the OP heavily revise their methods and the presentation of their research for any further work. The main thing would be to present the results of the interventions in a standardised metric, e.g. total sleep time, or standard deviations of something, so readers can make a comparison themselves, then make comments on cost and, if necessary research quality. I am happy to provide advice if that’s helpeful.
*I recognise the weighted-factor model is something Charity Entrepreneurship use. I have raised it with them several times that, for the reasons given, I find this approach hard to follow or justify and thus questionably useful.
Thank you for the feedback!
There’s a few points you make that I feel are important to clarify but I want to first acknowledge that the format and presentation of the research is a work in progress and could definitely be improved. In particular, I can agree that the “5.95/10” numbers aren’t particularly useful given the lack of any context/ scale.
To respond quickly to a few specific points:
1) This research was overwhelmingly based on the existing literature. I chose not to include a reference list or in-text citations to maximise the readability of the text but perhaps this was an error.
2) The interventions’ strength of effect, which accounts for about 40% of the score, is an average of improvements in sleep efficiency and total sleep time found in the literature I reviewed. In hindsight, I think it could have been better to highlight these findings on their own in the text and may add them in.
3) The focus of this research was to explore interventions that the reader is less likely to have heard of previously. I assumed that ideas like avoiding alcohol and late naps are things that nearly all readers would already know. On that basis, I felt that highlighting them in an article like this is unlikely to produce any change in behaviour, though I can see now the potential usefulness of presenting the size of effect.
4) I think an article like this recommending low-cost, personal interventions can reasonably have a different approach and level of rigour to one recommending charitable interventions and/or shifting large sums of money. A weighted factor model may not be the best way to frame this research but I think additional considerations like the potential risks and additional benefits of a recommendation are important and necessary to highlight.
These points make me realise that a more explicit description of the methods used and the literature reviewed would be valuable for future posts, rather than linking to them in out-of-text docs or leaving them out for a marginal improvement to the conciseness of the text.
This is the first post intended in a series and I expect to revise and improve the methods involved with each post and certainly the feedback I get on posts helps to direct that process. As a first attempt, some of the process was not as rigorous as it could or perhaps should be. In part, this is a relatively time-limited project for now (~3 months) so I am sacrificing some potential added depth in each post for the ability to cover more topics.
On a final note, I have immense respect and appreciation for the work of HLI and so really appreciate the feedback from someone who does wellbeing research at a much higher level!
Hello Ben!
Good to know this was based on existing literature. In most cases, it helps to show the reader you know that literature, to outline what it is, and then go on say what your new contribution is. Like I say, you missed a few of the obvious things, which is unfortunate. A piece of “what works for X” should, I say, include the things that work for X, then perhaps go on to flag which of these are likely to be a surprise, rather than assuming on the reader’s behalf what they will already know. If you are going to have a piece on “what works for X that might but that might surprise you” you should at least clearly flag that, and then point to something such as “standard guidance on X”.
Re strength of interventions being “40%” that still seems a confused way of presenting the information. 40% of what? Of a maximum score? A maximum score of what? Of cost-effectiveness? Well, why not just present the effectiveness numbers and divide them by the costs then?
I agree that this sort of thing can have a lower level of rigour but I stand by my concern that the method you use is so puzzling it’s questionably useful at all. You gathered quite a bit of relevant info, but I think you presented it in a less-than-ideal way. Here, simpler would have been better: I’d have preferred a post that just said “here’s a list of evidence-based ways to improve sleep” and then listed them and provided a brief discussion on each. That seems the way to go unless you have the data and time to do a quantitative (cost-)effectiveness analysis.
Glad you think we (at HLI) do good work. Like I see, feel free to reach out if you want to chat about research methods etc.! You can get me at michael@happierlivesinstitute.org