Thanksâdespite already having received great feedback from you on a survey Iâm developing, I still found this comment also really useful.
open comments questions are often dramatically more time-consuming and demanding than fixed response questions and so their inclusion can greatly reduce the quality of the responses to the fixed questions.
Meaning people start getting tired or bored and so give less thoughtful responses to the fixes questions? (Rather than just some people stopping taking the survey, which would be a quantity issue.)
And do you think that holds even if you make it clear that those open comment boxes are very optional? In particular, I plan to say at the start of the survey something like âFeel free to skip questions about which youâd rather not express an opinion, e.g. because you donât feel you know enough about it. Feel especially free to skip the questions about the robustness of your views and the comment boxes.â (I donât then remind people at each comment box that that comment box is optional.)
I think all of the following (and more) are possible risks:
- People are tired/âbored and so answer less effortfully/âmore quickly
- People are annoyed and so answer in a qualitatively different way
- People are tired/âbored/âannoyed and so skip more questions
- People are tired/âbored/âannoyed and dropout entirely
Note that people skipping questions/âdropping out is not merely a matter of quantity (reduced numbers of responses), because the dropout/âskipping is likely to be differential. The effect of the questions will be to lead to precisely those respondents who are more likely to be bored/âtired/âannoyed by those questions and to skip questions/âdropout if bored/âtired/âannoyed to be less likely to give responses.
Regrettably, I think that specifying extremely clearly that the questions are completely optional influences some respondents (it also likely makes many simply less likely to answer these questions), but doesnât ameliorate the harm for others. You may be surprised how many people will provide multiple exceptionally long open comments and then complain that the survey took them longer than the projected average. That aside, depending on the context, I think itâs sometimes legitimate for people to be annoyed by the presence of lots of open comment questions even if they are explicitly stated to be optional because, in context, it may seem like they need to answer them anyway.
Thanksâdespite already having received great feedback from you on a survey Iâm developing, I still found this comment also really useful.
Meaning people start getting tired or bored and so give less thoughtful responses to the fixes questions? (Rather than just some people stopping taking the survey, which would be a quantity issue.)
And do you think that holds even if you make it clear that those open comment boxes are very optional? In particular, I plan to say at the start of the survey something like âFeel free to skip questions about which youâd rather not express an opinion, e.g. because you donât feel you know enough about it. Feel especially free to skip the questions about the robustness of your views and the comment boxes.â (I donât then remind people at each comment box that that comment box is optional.)
I think all of the following (and more) are possible risks:
- People are tired/âbored and so answer less effortfully/âmore quickly
- People are annoyed and so answer in a qualitatively different way
- People are tired/âbored/âannoyed and so skip more questions
- People are tired/âbored/âannoyed and dropout entirely
Note that people skipping questions/âdropping out is not merely a matter of quantity (reduced numbers of responses), because the dropout/âskipping is likely to be differential. The effect of the questions will be to lead to precisely those respondents who are more likely to be bored/âtired/âannoyed by those questions and to skip questions/âdropout if bored/âtired/âannoyed to be less likely to give responses.
Regrettably, I think that specifying extremely clearly that the questions are completely optional influences some respondents (it also likely makes many simply less likely to answer these questions), but doesnât ameliorate the harm for others. You may be surprised how many people will provide multiple exceptionally long open comments and then complain that the survey took them longer than the projected average. That aside, depending on the context, I think itâs sometimes legitimate for people to be annoyed by the presence of lots of open comment questions even if they are explicitly stated to be optional because, in context, it may seem like they need to answer them anyway.