WAY too many of the questions only allow checking a single box, or a limited number of boxes. I’m not sure why you’ve done this? From my perspective it almost never seems like the right thing, and it’s going to significantly reduce the accuracy of the measurements you get, at least from me.
Thanks for your comment. A lot of the questions are verbatim requests from other orgs, so I can’t speak to exactly what the reasons for different designs are. Another commenter is also correct to mention the rationale of keeping the questions the same across years (some of these date back to 2014), even if the phrasing isn’t what we would use now. There are also some other practical considerations, like wanting to compare results to surveys that other orgs have already used themselves.
That said, I’m happy to defend the claim that allowing respondents to select only a single option is often better than allowing people to select any number of boxes. People (i.e. research users) are often primarily interested in the _most_ important or _primary_ factors for respondents, for a given question, rather than in all factors. With a ‘select all’ format, one loses the information about which are the most important. Of course, ideally, one could use a format which captures information about the relative importance of each selected factor, as well as which factors are selected. For example, in previous surveys we’ve asked respondents to rate the degree of importance of each factor , as well as which factors they did not have a significant interaction with. But the costs here are very high, as answering one of these questions takes longer and is more cognitively demanding than answering multiple simpler questions. So, given significant practical constraints (to keep the survey short, while including as many requests as possible), we often have to use simpler, quicker question formats.
Regarding politics specifically, I would note that asking about politics on a single scale is exceptionally common (I’d also add that single-select format for religion is very standard e.g. in the CES). I don’t see this as implying a belief that individuals believe in a single “simple political identity or political theory.” The one wrinkle in our measure is that ‘libertarian’ is also included a distinct category (which dates back to requests in 2014-2015 and, as mentioned above, the considerations in favour of keeping questions consistent across years are quite strong). Ideally we could definitely split this out so we have (at least) one scale, plus a distinct question which captures libertarian alignment or more fine-grained positions. But there are innumerable other questions which we’d prioritise over getting more nuanced political alignment data.
With a ‘select all’ format, one loses the information about which are the most important
Have you found that people answer that way? I’ll only tend to answer with more than one option if they’re all about equally important.
You might expect that it’s uncommon for multiple factors to be equally important, I think one of the reasons it is common, in the messy reality that we have (which is not the reality that most statisticians want): multiple factors are often crucial dependencies. Example: A person who spends a lot of their political energy advocating for Quadratic Funding (a democratic way of deciding how public funding is allocated) cannot be said to be more statist than they are libertarian, or vice versa, because the concept of QF just wouldn’t exist and cannot be advocated without both schools of thought, there may be ways of quantifying the role of each school in its invention, but they’re arbitrary (you probably don’t want to end up just measuring which arbitrary quantifications of qualitative dependencies respondants might have in mind today) The concept rests on principles from both schools, to ask which is more important to them is like asking whether having skin is more important to an animal than having blood.
Thanks for your comment. A lot of the questions are verbatim requests from other orgs, so I can’t speak to exactly what the reasons for different designs are. Another commenter is also correct to mention the rationale of keeping the questions the same across years (some of these date back to 2014), even if the phrasing isn’t what we would use now. There are also some other practical considerations, like wanting to compare results to surveys that other orgs have already used themselves.
That said, I’m happy to defend the claim that allowing respondents to select only a single option is often better than allowing people to select any number of boxes. People (i.e. research users) are often primarily interested in the _most_ important or _primary_ factors for respondents, for a given question, rather than in all factors. With a ‘select all’ format, one loses the information about which are the most important. Of course, ideally, one could use a format which captures information about the relative importance of each selected factor, as well as which factors are selected. For example, in previous surveys we’ve asked respondents to rate the degree of importance of each factor , as well as which factors they did not have a significant interaction with. But the costs here are very high, as answering one of these questions takes longer and is more cognitively demanding than answering multiple simpler questions. So, given significant practical constraints (to keep the survey short, while including as many requests as possible), we often have to use simpler, quicker question formats.
Regarding politics specifically, I would note that asking about politics on a single scale is exceptionally common (I’d also add that single-select format for religion is very standard e.g. in the CES). I don’t see this as implying a belief that individuals believe in a single “simple political identity or political theory.” The one wrinkle in our measure is that ‘libertarian’ is also included a distinct category (which dates back to requests in 2014-2015 and, as mentioned above, the considerations in favour of keeping questions consistent across years are quite strong). Ideally we could definitely split this out so we have (at least) one scale, plus a distinct question which captures libertarian alignment or more fine-grained positions. But there are innumerable other questions which we’d prioritise over getting more nuanced political alignment data.
Have you found that people answer that way? I’ll only tend to answer with more than one option if they’re all about equally important.
You might expect that it’s uncommon for multiple factors to be equally important, I think one of the reasons it is common, in the messy reality that we have (which is not the reality that most statisticians want): multiple factors are often crucial dependencies.
Example: A person who spends a lot of their political energy advocating for Quadratic Funding (a democratic way of deciding how public funding is allocated) cannot be said to be more statist than they are libertarian, or vice versa, because the concept of QF just wouldn’t exist and cannot be advocated without both schools of thought, there may be ways of quantifying the role of each school in its invention, but they’re arbitrary (you probably don’t want to end up just measuring which arbitrary quantifications of qualitative dependencies respondants might have in mind today) The concept rests on principles from both schools, to ask which is more important to them is like asking whether having skin is more important to an animal than having blood.