WAY too many of the questions only allow checking a single box, or a limited number of boxes. I’m not sure why you’ve done this? From my perspective it almost never seems like the right thing, and it’s going to significantly reduce the accuracy of the measurements you get, at least from me.
An example would be, there’s a question, “what is the main type of impact you expect to have” or something, and I expect to do things that are entrepreneurial, which involve or largely consist of communitybuilding, communication and research. I don’t know which of those for impact types are going to be the largest (it’s not even possible to assess that, and I’m not sure it’s a meaningful question considering that the impacts are often dependent on more than one of those factors at the same time: We can’t blame any one factor), but even if I did know how to assess that, the second place category might have similar amounts of impact as first, meaning that by only asking for the peak, you’re losing most of the distribution.
Other examples, which are especially galling, are the question about religious or political identity. The notion that people can only adhere to one religion is actually an invention of monotheist abrahamic traditions, arguably a highly spiritually corrosive assumption, I’m not positioned to argue that, but the survey shouldn’t be imposing monotheistic assumptions. The idea that most EAs would have a simple political identity or political theory is outright strange to me. Have you never actually seen EAs discussing politics? Do you think people should have discrete political identities? I think having a discrete, easily classifiable political identity is pretty socially corrosive as well and shouldn’t be imposed by the survey! (although maybe an ‘other’ or ‘misc’ category is enough here. People with mixed political identities tend not to be big fans of political identity in general.)
WAY too many of the questions only allow checking a single box, or a limited number of boxes. I’m not sure why you’ve done this? From my perspective it almost never seems like the right thing, and it’s going to significantly reduce the accuracy of the measurements you get, at least from me.
Thanks for your comment. A lot of the questions are verbatim requests from other orgs, so I can’t speak to exactly what the reasons for different designs are. Another commenter is also correct to mention the rationale of keeping the questions the same across years (some of these date back to 2014), even if the phrasing isn’t what we would use now. There are also some other practical considerations, like wanting to compare results to surveys that other orgs have already used themselves.
That said, I’m happy to defend the claim that allowing respondents to select only a single option is often better than allowing people to select any number of boxes. People (i.e. research users) are often primarily interested in the _most_ important or _primary_ factors for respondents, for a given question, rather than in all factors. With a ‘select all’ format, one loses the information about which are the most important. Of course, ideally, one could use a format which captures information about the relative importance of each selected factor, as well as which factors are selected. For example, in previous surveys we’ve asked respondents to rate the degree of importance of each factor , as well as which factors they did not have a significant interaction with. But the costs here are very high, as answering one of these questions takes longer and is more cognitively demanding than answering multiple simpler questions. So, given significant practical constraints (to keep the survey short, while including as many requests as possible), we often have to use simpler, quicker question formats.
Regarding politics specifically, I would note that asking about politics on a single scale is exceptionally common (I’d also add that single-select format for religion is very standard e.g. in the CES). I don’t see this as implying a belief that individuals believe in a single “simple political identity or political theory.” The one wrinkle in our measure is that ‘libertarian’ is also included a distinct category (which dates back to requests in 2014-2015 and, as mentioned above, the considerations in favour of keeping questions consistent across years are quite strong). Ideally we could definitely split this out so we have (at least) one scale, plus a distinct question which captures libertarian alignment or more fine-grained positions. But there are innumerable other questions which we’d prioritise over getting more nuanced political alignment data.
With a ‘select all’ format, one loses the information about which are the most important
Have you found that people answer that way? I’ll only tend to answer with more than one option if they’re all about equally important.
You might expect that it’s uncommon for multiple factors to be equally important, I think one of the reasons it is common, in the messy reality that we have (which is not the reality that most statisticians want): multiple factors are often crucial dependencies. Example: A person who spends a lot of their political energy advocating for Quadratic Funding (a democratic way of deciding how public funding is allocated) cannot be said to be more statist than they are libertarian, or vice versa, because the concept of QF just wouldn’t exist and cannot be advocated without both schools of thought, there may be ways of quantifying the role of each school in its invention, but they’re arbitrary (you probably don’t want to end up just measuring which arbitrary quantifications of qualitative dependencies respondants might have in mind today) The concept rests on principles from both schools, to ask which is more important to them is like asking whether having skin is more important to an animal than having blood.
I think one consideration is that they want to make the surveys comparable year to year, and if people can select many categories, that would be make it difficult.
For adding multiple options, I think there’s another sense of challenge, where if someone could select different political identities or religions, that would make the result difficult to interpret. It seems sort of “mainstream” for better or worse, that there is one category for some of the things you mentioned.
Zooming out, it seems that instead of seeing things like single/multiple as sort of a didactic/right or wrong choice or trying to impose a viewpoint, these seem to be design decisions, that is sort of inherently imperfect in some sense, and part of some bigger vision or something.
I think one consideration is that they want to make the surveys comparable year to year
Makes sense. But I guess if it’s only been one year, there wouldn’t have been much of a cost to changing it this year, or, the cost would have been smaller than the cost of not having it right in future years.
if someone could select different political identities or religions, that would make the result difficult to interpret
Could you explain why? I don’t see why it should, really.
Could you explain why? I don’t see why it should, really.
Well, in one sense that is shallow, what would an agnostic person + (some other religion mean)?
Maybe more deeper (?): it seems like some religions like Buddhism, which accepts other practices, would be understood to accept other practices. So it’s not clear if a Buddhist who selected multiple options had different beliefs, or was just very being very comprehensive and communicative like a good EA.
Well, in one sense that is shallow, what would an agnostic person + (some other religion mean)?
Uh that specifically? Engaging in practices and being open to the existence of the divine but ultimately not being convinced. This is not actually a strange or uncommon position. (What if there are a lot of statisticians who are trying to make their work easier by asking questions that make the world seem simpler than it is.)
it seems like some religions like Buddhism, which accepts other practices, would be understood to accept other practices [but not believe in them or practice them?]
That just sounds like a totally bizarre way to answer the question as I understood it (and possibly as it was stated, I don’t remember the details). I wouldn’t expect a buddhist with no other affiliations to answer that way. I don’t believe the ambiguity is there.
WAY too many of the questions only allow checking a single box, or a limited number of boxes. I’m not sure why you’ve done this? From my perspective it almost never seems like the right thing, and it’s going to significantly reduce the accuracy of the measurements you get, at least from me.
An example would be, there’s a question, “what is the main type of impact you expect to have” or something, and I expect to do things that are entrepreneurial, which involve or largely consist of communitybuilding, communication and research. I don’t know which of those for impact types are going to be the largest (it’s not even possible to assess that, and I’m not sure it’s a meaningful question considering that the impacts are often dependent on more than one of those factors at the same time: We can’t blame any one factor), but even if I did know how to assess that, the second place category might have similar amounts of impact as first, meaning that by only asking for the peak, you’re losing most of the distribution.
Other examples, which are especially galling, are the question about religious or political identity. The notion that people can only adhere to one religion is actually an invention of monotheist abrahamic traditions, arguably a highly spiritually corrosive assumption, I’m not positioned to argue that, but the survey shouldn’t be imposing monotheistic assumptions.
The idea that most EAs would have a simple political identity or political theory is outright strange to me. Have you never actually seen EAs discussing politics? Do you think people should have discrete political identities? I think having a discrete, easily classifiable political identity is pretty socially corrosive as well and shouldn’t be imposed by the survey! (although maybe an ‘other’ or ‘misc’ category is enough here. People with mixed political identities tend not to be big fans of political identity in general.)
Thanks for your comment. A lot of the questions are verbatim requests from other orgs, so I can’t speak to exactly what the reasons for different designs are. Another commenter is also correct to mention the rationale of keeping the questions the same across years (some of these date back to 2014), even if the phrasing isn’t what we would use now. There are also some other practical considerations, like wanting to compare results to surveys that other orgs have already used themselves.
That said, I’m happy to defend the claim that allowing respondents to select only a single option is often better than allowing people to select any number of boxes. People (i.e. research users) are often primarily interested in the _most_ important or _primary_ factors for respondents, for a given question, rather than in all factors. With a ‘select all’ format, one loses the information about which are the most important. Of course, ideally, one could use a format which captures information about the relative importance of each selected factor, as well as which factors are selected. For example, in previous surveys we’ve asked respondents to rate the degree of importance of each factor , as well as which factors they did not have a significant interaction with. But the costs here are very high, as answering one of these questions takes longer and is more cognitively demanding than answering multiple simpler questions. So, given significant practical constraints (to keep the survey short, while including as many requests as possible), we often have to use simpler, quicker question formats.
Regarding politics specifically, I would note that asking about politics on a single scale is exceptionally common (I’d also add that single-select format for religion is very standard e.g. in the CES). I don’t see this as implying a belief that individuals believe in a single “simple political identity or political theory.” The one wrinkle in our measure is that ‘libertarian’ is also included a distinct category (which dates back to requests in 2014-2015 and, as mentioned above, the considerations in favour of keeping questions consistent across years are quite strong). Ideally we could definitely split this out so we have (at least) one scale, plus a distinct question which captures libertarian alignment or more fine-grained positions. But there are innumerable other questions which we’d prioritise over getting more nuanced political alignment data.
Have you found that people answer that way? I’ll only tend to answer with more than one option if they’re all about equally important.
You might expect that it’s uncommon for multiple factors to be equally important, I think one of the reasons it is common, in the messy reality that we have (which is not the reality that most statisticians want): multiple factors are often crucial dependencies.
Example: A person who spends a lot of their political energy advocating for Quadratic Funding (a democratic way of deciding how public funding is allocated) cannot be said to be more statist than they are libertarian, or vice versa, because the concept of QF just wouldn’t exist and cannot be advocated without both schools of thought, there may be ways of quantifying the role of each school in its invention, but they’re arbitrary (you probably don’t want to end up just measuring which arbitrary quantifications of qualitative dependencies respondants might have in mind today) The concept rests on principles from both schools, to ask which is more important to them is like asking whether having skin is more important to an animal than having blood.
I think one consideration is that they want to make the surveys comparable year to year, and if people can select many categories, that would be make it difficult.
For adding multiple options, I think there’s another sense of challenge, where if someone could select different political identities or religions, that would make the result difficult to interpret. It seems sort of “mainstream” for better or worse, that there is one category for some of the things you mentioned.
Zooming out, it seems that instead of seeing things like single/multiple as sort of a didactic/right or wrong choice or trying to impose a viewpoint, these seem to be design decisions, that is sort of inherently imperfect in some sense, and part of some bigger vision or something.
Makes sense. But I guess if it’s only been one year, there wouldn’t have been much of a cost to changing it this year, or, the cost would have been smaller than the cost of not having it right in future years.
Could you explain why? I don’t see why it should, really.
Well, in one sense that is shallow, what would an agnostic person + (some other religion mean)?
Maybe more deeper (?): it seems like some religions like Buddhism, which accepts other practices, would be understood to accept other practices. So it’s not clear if a Buddhist who selected multiple options had different beliefs, or was just very being very comprehensive and communicative like a good EA.
Uh that specifically? Engaging in practices and being open to the existence of the divine but ultimately not being convinced. This is not actually a strange or uncommon position. (What if there are a lot of statisticians who are trying to make their work easier by asking questions that make the world seem simpler than it is.)
That just sounds like a totally bizarre way to answer the question as I understood it (and possibly as it was stated, I don’t remember the details). I wouldn’t expect a buddhist with no other affiliations to answer that way. I don’t believe the ambiguity is there.