How would the possibility of scale norming with life satisfaction scores (using the scales differently across people or over time, in possibly predictable ways) affect these results? There’s a recent paper on this, and also an attempt to correct for this here (video here). (I haven’t read any of these myself; just the abstracts.)
(Disclosure: I’m the author of the second linked paper, board member of HLI, and a collaborator on some of its research.)
Hi Michael!
In my paper on scale use, I generally find that people who become more satisfied tend to also become more stringent in the way they report their satisfaction (i.e., for a given satisfaction level, they report a lower number). As a consequence, effects tend to be underestimated.
If effects are underestimated by the same amount across different variables/​treatments, scale norming is not an issue (apart from costing us statistical power). However, in the context of this post, if (say) the change in reporting behaviour is stronger for cash-transfers than for psychotherapy, then cash-transfers will seem relatively less cost-effective than psychotherapy .
To assess whether this is indeed a problem, we’d either need data on so-called vignettes (link), or people’s assessment of their past wellbeing. Unfortunately, as far as I know, this data does not currently exist.
That being said, in my paper (which is based on a sample from the UK), I find that accounting for changes in scale use does not, compared to the other included variables, result in statistically significantly larger associations between income and satisfaction.
How would the possibility of scale norming with life satisfaction scores (using the scales differently across people or over time, in possibly predictable ways) affect these results? There’s a recent paper on this, and also an attempt to correct for this here (video here). (I haven’t read any of these myself; just the abstracts.)
(Disclosure: I’m the author of the second linked paper, board member of HLI, and a collaborator on some of its research.)
Hi Michael!
In my paper on scale use, I generally find that people who become more satisfied tend to also become more stringent in the way they report their satisfaction (i.e., for a given satisfaction level, they report a lower number). As a consequence, effects tend to be underestimated.
If effects are underestimated by the same amount across different variables/​treatments, scale norming is not an issue (apart from costing us statistical power). However, in the context of this post, if (say) the change in reporting behaviour is stronger for cash-transfers than for psychotherapy, then cash-transfers will seem relatively less cost-effective than psychotherapy .
To assess whether this is indeed a problem, we’d either need data on so-called vignettes (link), or people’s assessment of their past wellbeing. Unfortunately, as far as I know, this data does not currently exist.
That being said, in my paper (which is based on a sample from the UK), I find that accounting for changes in scale use does not, compared to the other included variables, result in statistically significantly larger associations between income and satisfaction.