I’m not sure I agree with some parts of your argument in that write-up. My main points of contention are:
1. The accuracy of self-reports. You have given low/negligible weight to all studies relying on self-report, but I don’t think that is warranted. Firstly, lots of widely cited research on sleep medicine uses sleep diaries only or sleep diaries + objective measures to measure sleep. This suggests that even though it is widely recognised that sleep diaries are less accurate than objective measures, they are nevertheless considered to be important evidence by experts in the field. Most of the statements you cite in support of your stance (except Bauer and Blunden which I can’t access) just say that sleep diaries are less accurate than objective measures, they don’t say that sleep diary measures should be nearly completely ignored. You cite two studies showing that sleep diary measures and objective measures come apart, but one of these uses parents’ reporting the sleep of their autistic children, which seems so different to self-reported sleep as to be completely irrelevant to the accuracy of the latter.
Secondly, your proposed test outlined in footnote 51 suggests you think there is a fairly strong correlation between self-reported measures and objective measures. You think there is a 70% chance that at least 35% of studies will have the sign of the effect on objective and subjective measures coming apart. So, am I right to infer that you think that in expectation up to 75.5% of studies will have the same sign on both measures, in which case the measures are fairly strongly correlated?
Thirdly, my a priori intuition is that self-reported sleep measures will be pretty accurate in that they will note the direction in which your sleep is moving quite well. I can tell when I have slept badly (what feels like 6 hours) and slept well (what feels like 8 hours). The actual numbers might be wrong, but I think the direction of travel would almost always be correct.
2. Long-term effects. I have higher confidence that the effects of CBT-I will persist into the longer-term, so I think the endline studies are better evidence of long-term effect that you. Indeed, this seems to be a view shared in various papers—the effects are more likely to persist than drugs. E.g. the NIH says “Moreover, there are indications that the beneficial effects of CBT, in contrast to those produced by medications, may last well beyond the termination of active treatment’’. I’ve read similar claims in various other studies. This chimes with my personal experience and with my prior. I can see why we would build up a tolerance to sleeping pills, which would limit long-term efficacy. But on the posited mechanism of CBT-I, the mechanism is breaking the psychological association between bed and lack of sleep, and making permanent changes to your sleep routine and environment. I don’t see why this kind of effect would decay.
3. Deference to experts/epistemic modesty. NICE and the NIH both say that CBT-I is effective and recommend it as the first line of defence against chronic insomnia. A pan-European body of sleep researchers at major European universities also comes to the same conclusion. Until I see some leading sleep researchers who publicly disagree with these major scientific bodies, or agree with the arguments you make in your piece, I think it makes more sense to go with the scientific bodies.
I would also add—the American Academy of Sleep Medicine agrees that CBT-I should be the first line of defence against insomnia.
Pushing on the epistemic modesty point, if you didn’t interview any experts when writing your brief, it seems overconfident to disagree with the scientific establishment. If you have a tentatively held conclusion and you’ve not engaged with other experts, I think the sensible thing to do is defer.
More generally, I think extensive interviews with experts should be part of all reviews of evidence on interventions
Interesting—I didn’t look into the evidence that much and was happy to defer to NICE on it, which if what you say is correct (I haven’t checked), may not have been right. As you say, sleep restriction+hygiene still seems worth trying even if, as you argue, the evidence isn’t that strong, given that it has a plausible mechanism and is unlikely to do harm.
I will look deeper into the evidence and add an edit to the main text in the meantime
I disagree about the strength of evidence for CBT-I effectiveness.
Hi,
I’m not sure I agree with some parts of your argument in that write-up. My main points of contention are:
1. The accuracy of self-reports. You have given low/negligible weight to all studies relying on self-report, but I don’t think that is warranted. Firstly, lots of widely cited research on sleep medicine uses sleep diaries only or sleep diaries + objective measures to measure sleep. This suggests that even though it is widely recognised that sleep diaries are less accurate than objective measures, they are nevertheless considered to be important evidence by experts in the field. Most of the statements you cite in support of your stance (except Bauer and Blunden which I can’t access) just say that sleep diaries are less accurate than objective measures, they don’t say that sleep diary measures should be nearly completely ignored. You cite two studies showing that sleep diary measures and objective measures come apart, but one of these uses parents’ reporting the sleep of their autistic children, which seems so different to self-reported sleep as to be completely irrelevant to the accuracy of the latter.
Secondly, your proposed test outlined in footnote 51 suggests you think there is a fairly strong correlation between self-reported measures and objective measures. You think there is a 70% chance that at least 35% of studies will have the sign of the effect on objective and subjective measures coming apart. So, am I right to infer that you think that in expectation up to 75.5% of studies will have the same sign on both measures, in which case the measures are fairly strongly correlated?
Thirdly, my a priori intuition is that self-reported sleep measures will be pretty accurate in that they will note the direction in which your sleep is moving quite well. I can tell when I have slept badly (what feels like 6 hours) and slept well (what feels like 8 hours). The actual numbers might be wrong, but I think the direction of travel would almost always be correct.
2. Long-term effects. I have higher confidence that the effects of CBT-I will persist into the longer-term, so I think the endline studies are better evidence of long-term effect that you. Indeed, this seems to be a view shared in various papers—the effects are more likely to persist than drugs. E.g. the NIH says “Moreover, there are indications that the beneficial effects of CBT, in contrast to those produced by medications, may last well beyond the termination of active treatment’’. I’ve read similar claims in various other studies. This chimes with my personal experience and with my prior. I can see why we would build up a tolerance to sleeping pills, which would limit long-term efficacy. But on the posited mechanism of CBT-I, the mechanism is breaking the psychological association between bed and lack of sleep, and making permanent changes to your sleep routine and environment. I don’t see why this kind of effect would decay.
3. Deference to experts/epistemic modesty. NICE and the NIH both say that CBT-I is effective and recommend it as the first line of defence against chronic insomnia. A pan-European body of sleep researchers at major European universities also comes to the same conclusion. Until I see some leading sleep researchers who publicly disagree with these major scientific bodies, or agree with the arguments you make in your piece, I think it makes more sense to go with the scientific bodies.
I would also add—the American Academy of Sleep Medicine agrees that CBT-I should be the first line of defence against insomnia.
Pushing on the epistemic modesty point, if you didn’t interview any experts when writing your brief, it seems overconfident to disagree with the scientific establishment. If you have a tentatively held conclusion and you’ve not engaged with other experts, I think the sensible thing to do is defer.
More generally, I think extensive interviews with experts should be part of all reviews of evidence on interventions
Interesting—I didn’t look into the evidence that much and was happy to defer to NICE on it, which if what you say is correct (I haven’t checked), may not have been right. As you say, sleep restriction+hygiene still seems worth trying even if, as you argue, the evidence isn’t that strong, given that it has a plausible mechanism and is unlikely to do harm.
I will look deeper into the evidence and add an edit to the main text in the meantime