Thanks for doing this! It’s very difficult to figure out what the right way of sampling EAs is since it’s unclear how you define an EA. Is it someone who acts as an EA or identifies as one? How much altruism (donations, volunteering) etc. is required and how much effectiveness (e.g. being cause-neutral versus just seeking to be effective within a cause) is required?
Just by skimming this, it strikes me that SSC is overrepresented, I’m guessing because of the share of the survey being more salient there and having an overall large readership base. Facebook seems overrepresented too, although I think that’s less of an issue since the SSC audience has its own culture in the way that EAs on Facebook probably do not. I do wonder if there would be a way to get better participation from EA organizations’ audiences, as I think that’s probably the best way to get a picture of things, and I imagine the mailing lists are not very salient. (I don’t remember the survey link and am on several of those mailing lists.)
For reference, the question we used for people to self-identify as an EA (and be included in the analysis) was:
Could you, however loosely, be described as “an Effective Altruist”? Please answer “Yes” if you are broadly in agreement with the core ideas associated with Effective Altruism, even if you don’t like applying the term to yourself (e.g. because it sounds presumptuous or you are an “aspiring Effective Altruist”).
You make some really good points about how tricky it is to sample EA, not the least because defining “EA” is so hard. We don’t have any good information to reweigh the population of the survey, so it’s hard to say if our populations are over- or underrepresented, though I would agree with your intuitions.
I don’t remember the survey link and am on several of those mailing lists.
For one example, we did have an entire EA Newsletter dedicated solely to the EA Survey—I’m not sure how to make it more apparent without being obnoxious. I am a bit surprised by how few sign-ups we got through the EA Newsletter. I do agree that the inclusion in some of the other newsletters was more subtle.
Yeah, upon reflection I don’t think the problem is how the survey is presented in a newsletter but whether people read the newsletter at all. Most newsletters end up in the promotions folder on Gmail and I imagine a similar folder elsewhere. What I’ve found in the past is you have to find a way to make it appear as a personal email (not have images or formatting, for example, and use people’s names), and that can get around it sometimes.
What I’ve found in the past is you have to find a way to make it appear as a personal email (not have images or formatting, for example, and use people’s names), and that can get around it sometimes.
We did also send 1-1 personal emails to everyone who gave an email address for past surveys, though we only got three survey completions from this. Maybe we did something wrong here?
I believe the 1-1 personal emails were still sent through a mail service. In my experience (from being the sender for similar emails) those still get caught in promotions or spam a lot of the time.
A solution for this would be to send “normal” emails (not bulk, no images) from a “normal” email address (like Gmail). I’ll definitely consider this for the 2018 survey.
Thanks for doing this! It’s very difficult to figure out what the right way of sampling EAs is since it’s unclear how you define an EA. Is it someone who acts as an EA or identifies as one? How much altruism (donations, volunteering) etc. is required and how much effectiveness (e.g. being cause-neutral versus just seeking to be effective within a cause) is required?
Just by skimming this, it strikes me that SSC is overrepresented, I’m guessing because of the share of the survey being more salient there and having an overall large readership base. Facebook seems overrepresented too, although I think that’s less of an issue since the SSC audience has its own culture in the way that EAs on Facebook probably do not. I do wonder if there would be a way to get better participation from EA organizations’ audiences, as I think that’s probably the best way to get a picture of things, and I imagine the mailing lists are not very salient. (I don’t remember the survey link and am on several of those mailing lists.)
For reference, the question we used for people to self-identify as an EA (and be included in the analysis) was:
Could you, however loosely, be described as “an Effective Altruist”? Please answer “Yes” if you are broadly in agreement with the core ideas associated with Effective Altruism, even if you don’t like applying the term to yourself (e.g. because it sounds presumptuous or you are an “aspiring Effective Altruist”).
You make some really good points about how tricky it is to sample EA, not the least because defining “EA” is so hard. We don’t have any good information to reweigh the population of the survey, so it’s hard to say if our populations are over- or underrepresented, though I would agree with your intuitions.
For one example, we did have an entire EA Newsletter dedicated solely to the EA Survey—I’m not sure how to make it more apparent without being obnoxious. I am a bit surprised by how few sign-ups we got through the EA Newsletter. I do agree that the inclusion in some of the other newsletters was more subtle.
Yeah, upon reflection I don’t think the problem is how the survey is presented in a newsletter but whether people read the newsletter at all. Most newsletters end up in the promotions folder on Gmail and I imagine a similar folder elsewhere. What I’ve found in the past is you have to find a way to make it appear as a personal email (not have images or formatting, for example, and use people’s names), and that can get around it sometimes.
We did also send 1-1 personal emails to everyone who gave an email address for past surveys, though we only got three survey completions from this. Maybe we did something wrong here?
I believe the 1-1 personal emails were still sent through a mail service. In my experience (from being the sender for similar emails) those still get caught in promotions or spam a lot of the time.
A solution for this would be to send “normal” emails (not bulk, no images) from a “normal” email address (like Gmail). I’ll definitely consider this for the 2018 survey.