I was curious about the formatting of some of your demographic questions. For example this question;
28. Your gender:
provides only a free text box, with no standard options. This is often considered poor survey technique, because it can lead to a very broad range of responses, which require a lot of manual work on the backend. You will need to manually determine whether ‘woman’, ‘Female’, ‘Lady’, ‘f’ etc. are the same thing, and what you want to do with someone who says ‘Dude’. Not only is this time consuming but it adds subjectivity to your analysis. It also increases the amount of work required from your respondents—if they are on their iPhone they will have to manipulate the keypad, rather than just pressing once.
Since you are using SurveyMonkey, you have access to their SurveyMonkey Certified Questions:
This certified question was added from our Question Bank. It was written by our methodologists to minimize bias and get the most accurate responses.
If you edit the wording of this question, it’ll no longer be certified, which means it might be subject to bias and accuracy issues.
Most of their accredited gender questions avoid these problems by giving you simple options to click. This will likely be optimal for the vast majority of your respondents, and if you wanted to be politically correct you could always include an ‘Other’ box!
Strangely, it seems like for the race/ethnicity option you go in the opposite direction, by providing the full list of standard US options for people to select from. This includes ‘Native Hawaiian or Other Pacific Islander’, even though I think less than 0.1% of the global population fall into this composite category. If you are concerned about space limitations I would have considered removing this category, as well as the Alaskan Native one, implicitly folding them into the ‘other’ box.
The gender question and many of the other demographic questions were selected largely to ensure comparability with other surveys run by CEA.
That aside, I think your claim that open comment gender questions are “considered poor survey technique” is over-stated. The literature discusses pros and cons to both formats. From this recent article in the International Journal of Social Research Methodology:
One of the simplest ways to collect data on gender identity is to use an open text box (see
Figure 2) which allows participants the freedom to describe their gender in whatever way they
see fit while accommodating changing norms around acceptable terminology. Terms commonly used around gender evolve over time… It would therefore be misguided of researchers to attempt to find the most contemporary terminology and use it to the exclusion of all other terms. Research teams are also likely to find such a process difficult and frustrating (Herman et al., 2012). Thus, an open text box is certainly the most accommodating approach to a range of evolving terms to describe gender identity.
If open text boxes are used for research that intends to analyze by category, however,
researchers will still ultimately be categorizing the gender identities in order to define groups
for statistical analysis and groups to which the findings might be generalized… These decisions will also need to be made if researchers using a multiple-choice approach choose to provide a long list of as many gender identity terms as possible. This approach is a fine option, but researchers need to be cognisant that terminology that was in common use when a tool was published may no longer be current when research is conducted using that tool… Good arguments can be made for the value of participants being able to see the specific term for their gender identity among a list of possibilities, but even Herman’s and Kuper’s lists, published within the past decade, contain terms that are increasingly considered problematic and do not contain some terms that are more common today.
An approach which provides a smaller number of options for gender identity has benefits
and drawbacks. Providing fewer categories inevitably forces gender minority participants to
place themselves into categories that the researcher provides, but gives the advantage that the
participant, not researcher, chooses the categories in which they will be included.
I was curious about the formatting of some of your demographic questions. For example this question;
provides only a free text box, with no standard options. This is often considered poor survey technique, because it can lead to a very broad range of responses, which require a lot of manual work on the backend. You will need to manually determine whether ‘woman’, ‘Female’, ‘Lady’, ‘f’ etc. are the same thing, and what you want to do with someone who says ‘Dude’. Not only is this time consuming but it adds subjectivity to your analysis. It also increases the amount of work required from your respondents—if they are on their iPhone they will have to manipulate the keypad, rather than just pressing once.
Since you are using SurveyMonkey, you have access to their SurveyMonkey Certified Questions:
Most of their accredited gender questions avoid these problems by giving you simple options to click. This will likely be optimal for the vast majority of your respondents, and if you wanted to be politically correct you could always include an ‘Other’ box!
Strangely, it seems like for the race/ethnicity option you go in the opposite direction, by providing the full list of standard US options for people to select from. This includes ‘Native Hawaiian or Other Pacific Islander’, even though I think less than 0.1% of the global population fall into this composite category. If you are concerned about space limitations I would have considered removing this category, as well as the Alaskan Native one, implicitly folding them into the ‘other’ box.
Hi Dale. Thanks for your comment.
The gender question and many of the other demographic questions were selected largely to ensure comparability with other surveys run by CEA.
That aside, I think your claim that open comment gender questions are “considered poor survey technique” is over-stated. The literature discusses pros and cons to both formats. From this recent article in the International Journal of Social Research Methodology: