I don’t however think these issues with point estimates are biggest problem with wellbeing research, these issues are important yes for calibration, but a bigger problem is whether reported increases in wellbeing after an intervention are real or biased. I have said this before, apologies for being a stuck record.
These two biases which don’t necessarily affect point estimates (like you discuss above) but affect before and after measurements...
Demand/ courtesy bias. Giving higher wellbeing score after the intervention because you think that is what the researcher wants.
“Future hope” bias. Giving higher scores after any intervention, thinking (often rationally and correctly) that the positive report will make you more likely to get other, even different types of help in future. This could be a huge problem in surveys among the poor but there’s close to no research on it.
These might be hard to research and are undrafted, but I think it is important to try.
We should keep in mind though these two bias don’t only affect wellbeing surveys, but to some degree any self reported survey, for example the majority of give directly’s data.
Thanks for pointing out both kinds of biases. These biases can cause a failure of comparability. Concretely, if an intervention causes you to give counterfactually higher scores as a matter of ‘courtesy’ to the researcher, then the intervention changed the meaning of each given response category.
I therefore take it that you don’t think that our particular tests of comparability will cover the two biases you mention. If so, I agree. However, my colleague has given reasons for why we might not be as worried about these sorts of biases.
I don’t think this can be tested in our current survey format, but it might be testable in a different design. We are open to suggestions!
Not only courtesy, but also future hope (which I think may be more important here).
Yeah it’s really hard to test. I think validity of point estimates are pretty reasonable for wellbeing surveys and I agree with most of the reasoning on this post.
It’s very had to test those biases ethically, but probably possible. Not in this kind of survey anyway.
The reasons he gave for not being worried about those biases were not unreasonable, but based on flimsy evidence. Especially future hope bias which may not have been researched at all.
Thanks for this, it is interesting and important.
I don’t however think these issues with point estimates are biggest problem with wellbeing research, these issues are important yes for calibration, but a bigger problem is whether reported increases in wellbeing after an intervention are real or biased. I have said this before, apologies for being a stuck record.
These two biases which don’t necessarily affect point estimates (like you discuss above) but affect before and after measurements...
Demand/ courtesy bias. Giving higher wellbeing score after the intervention because you think that is what the researcher wants.
“Future hope” bias. Giving higher scores after any intervention, thinking (often rationally and correctly) that the positive report will make you more likely to get other, even different types of help in future. This could be a huge problem in surveys among the poor but there’s close to no research on it.
These might be hard to research and are undrafted, but I think it is important to try.
We should keep in mind though these two bias don’t only affect wellbeing surveys, but to some degree any self reported survey, for example the majority of give directly’s data.
Hi Nick,
Thanks for pointing out both kinds of biases. These biases can cause a failure of comparability. Concretely, if an intervention causes you to give counterfactually higher scores as a matter of ‘courtesy’ to the researcher, then the intervention changed the meaning of each given response category.
I therefore take it that you don’t think that our particular tests of comparability will cover the two biases you mention. If so, I agree. However, my colleague has given reasons for why we might not be as worried about these sorts of biases.
I don’t think this can be tested in our current survey format, but it might be testable in a different design. We are open to suggestions!
Not only courtesy, but also future hope (which I think may be more important here).
Yeah it’s really hard to test. I think validity of point estimates are pretty reasonable for wellbeing surveys and I agree with most of the reasoning on this post.
It’s very had to test those biases ethically, but probably possible. Not in this kind of survey anyway.
The reasons he gave for not being worried about those biases were not unreasonable, but based on flimsy evidence. Especially future hope bias which may not have been researched at all.