This is not a criticism (think that it’s sick you do this survey, and grateful for your work), but I’m curious whether Rethink (or anyone else) has had a go at adjusting for a non-random sample of the total EA community (however that is defined) being made aware of the survey and choosing to participate.
Probably out of scope for this project—and maybe not even that useful—but I wonder whether it could be useful to survey a sample from an actually random frame to just get some approximate idea of the size and demographics of the EA community. That might help inform EA orgs in working out how much to update based on the survey responses.
credence: not a stats guy, maybe this has been done already, maybe it doesn’t matter very much, open to being told why this doesn’t matter actually.
Thanks! We think about this a lot. We have previously discussed this and conducted some sensitivity testing in this dynamic document.
I wonder whether it could be useful to survey a sample from an actually random frame to just get some approximate idea of the size and demographics of the EA community.
The difficulty here is that it doesn’t seem to be possible to actually randomly sample from the EA population. At best, we could randomly sample from some narrower frame (e.g. people on main newsletter mailing list, EA Forum users), but these groups are not likely to be representative of the broader community. In the earliest surveys, we actually did also report results from a random sample drawn from the main EA Facebook group. However, these days the population of the EA Facebook group seems quite clearly not representative of the broader community, so the value of replicating this seems lower.
The more general challenge is that no-one knows what a representative sample of the EA community should look like (i.e. what is the true composition of the EA population). This is in contrast to general population samples where we can weight results relative to the composition found in the US census. I think the EA Survey itself represents the closest we have to such a source of information.
That said, I don’t think we are simply completely in the dark when it comes to assessing representativeness. We can test some particular concerns about potential sources of unrepresentativeness in the sample (and have done this since the first EA Survey). For example, if one is concerned that the survey samples a disproportionate number of respondents from particular sources (e.g. LessWrong), then we can assess how the samples drawn from those sources differ and how the results for the respondents drawn from those sources differ. Last year, for example, we examined how the reported importance of 80,000 Hours differed if we excluded all respondents referred to the survey from 80,000 Hours, and still found it to be very high. We can do more complex/sophisticated robustness/sensitivity checks on request. I’d encourage people to reach out if they have an interest in particular results, to see what we can do on a case by case basis.
sorry, I see now that you’ve discussed the point in my comment below (which I’ve now put in italics) in the linked document. I’m grateful for, but not surprised at, the care and thought that’s gone into this.
If it’s not too much of your time, I just am curious about one more thing. Is the paragraph below saying that surveying the general population would not provide useful information, or is it saying something like ‘this would help, but would not totally address the issue’. Like, is there any information value in doing this—or would it basically be pointless/pseudoscientific?
Surveying the general (non-EA) population as part of larger representative surveys to get a sense of the overall composition of EAs (e.g., the gender ratio). However, differential non-response, to these larger surveys would again throw this in doubt. Standard corrections may not be easy: and relative non-response among EAs (e.g., male versus female EAs) may differ from the relative non-response to such surveys in other populations.
***
Original comment:
Thanks for the in depth response, David.
The difficulty here is that it doesn’t seem to be possible to actually randomly sample from the EA population
Sorry, I explained poorly what I meant. What I meant to ask was whether you could randomly sample from a non-EA frame, identify EAs based on their responses (presumably a self identification question), and then use that to get some sense of the attributes of EAs.
One problem might be that the prevalence of EAs in that non-EA population might be so minuscule that you’d need to survey an impractical number of people to know much about EAs.
Another response is that it just wouldn’t be that useful to know, although the cost involved in hiring polling companies in a few places to do this maybe is not that much when weighed against the time cost of lots of EAs doing the survey at 10min/response.
I was a pretty motivated EA (donated, sometimes read EA literature) who did consider myself an EA but was entirely disengaged from the community from 2013-2017, and then barely engaged from 2017-2020. Additionally, when I speak with other lawyers it’s not uncommon to hear that someone is either interested in EA or has begun donating to an EA charity, but that they haven’t gotten involved with the community because they don’t see how that would help them or anyone else do more good.
I don’t know how useful you think it would be to know more about the makeup and size of that population of unengaged EAs (or EA-adjacent folk, or whatever the label). Maybe it just wouldn’t be very decision-relevant for the orgs who have expressed interest in using the data. My initial sense is that it would be useful, but I don’t really know.
Is the paragraph below saying that surveying the general population would not provide useful information, or is it saying something like ‘this would help, but would not totally address the issue’.
It’s just describing limitations. In principle, you could definitely update based on representative samples of the general population, but there would still be challenges.
Notably, we have already run a large representative survey (within the US), looking at how many people have heard of EA (for unrelated reasons). It illustrates one of the simple practical limitations of using this approach to estimate the composition of the EA community, rather than just to estimate how many people in the public have heard of EA.
Even with a sample of n=6000, we still only found around 150 people who plausibly even knew what effective altruism was (and we think this might still have been an over-estimate). Of those, I’d say no more than 1-3 seemed like they might have any real engagement with EA at all. (Incidentally, this is roughly a ratio that seems plausible to me for how many people who hear of EA actually then engaged with EA at all, i.e. 150-50:1 or less.) Note that we weren’t trying to see whether people were members of the EA community in this survey, so the above estimate is just based on those who happened to mention enough specifics- like knowing about 80,000 Hours- that it seemed like they might have been at all engaged with EA). So, given that, we’d need truly enormous survey samples to sample a decent number of ‘EAs’ via this method, and the results would still be limited by the difficulties mentioned above.
This is not a criticism (think that it’s sick you do this survey, and grateful for your work), but I’m curious whether Rethink (or anyone else) has had a go at adjusting for a non-random sample of the total EA community (however that is defined) being made aware of the survey and choosing to participate.
Probably out of scope for this project—and maybe not even that useful—but I wonder whether it could be useful to survey a sample from an actually random frame to just get some approximate idea of the size and demographics of the EA community. That might help inform EA orgs in working out how much to update based on the survey responses.
credence: not a stats guy, maybe this has been done already, maybe it doesn’t matter very much, open to being told why this doesn’t matter actually.
Thanks! We think about this a lot. We have previously discussed this and conducted some sensitivity testing in this dynamic document.
The difficulty here is that it doesn’t seem to be possible to actually randomly sample from the EA population. At best, we could randomly sample from some narrower frame (e.g. people on main newsletter mailing list, EA Forum users), but these groups are not likely to be representative of the broader community. In the earliest surveys, we actually did also report results from a random sample drawn from the main EA Facebook group. However, these days the population of the EA Facebook group seems quite clearly not representative of the broader community, so the value of replicating this seems lower.
The more general challenge is that no-one knows what a representative sample of the EA community should look like (i.e. what is the true composition of the EA population). This is in contrast to general population samples where we can weight results relative to the composition found in the US census. I think the EA Survey itself represents the closest we have to such a source of information.
That said, I don’t think we are simply completely in the dark when it comes to assessing representativeness. We can test some particular concerns about potential sources of unrepresentativeness in the sample (and have done this since the first EA Survey). For example, if one is concerned that the survey samples a disproportionate number of respondents from particular sources (e.g. LessWrong), then we can assess how the samples drawn from those sources differ and how the results for the respondents drawn from those sources differ. Last year, for example, we examined how the reported importance of 80,000 Hours differed if we excluded all respondents referred to the survey from 80,000 Hours, and still found it to be very high. We can do more complex/sophisticated robustness/sensitivity checks on request. I’d encourage people to reach out if they have an interest in particular results, to see what we can do on a case by case basis.
Edit:
sorry, I see now that you’ve discussed the point in my comment below (which I’ve now put in italics) in the linked document. I’m grateful for, but not surprised at, the care and thought that’s gone into this.
If it’s not too much of your time, I just am curious about one more thing. Is the paragraph below saying that surveying the general population would not provide useful information, or is it saying something like ‘this would help, but would not totally address the issue’. Like, is there any information value in doing this—or would it basically be pointless/pseudoscientific?
***
Original comment:
Thanks for the in depth response, David.
Sorry, I explained poorly what I meant. What I meant to ask was whether you could randomly sample from a non-EA frame, identify EAs based on their responses (presumably a self identification question), and then use that to get some sense of the attributes of EAs.
One problem might be that the prevalence of EAs in that non-EA population might be so minuscule that you’d need to survey an impractical number of people to know much about EAs.
Another response is that it just wouldn’t be that useful to know, although the cost involved in hiring polling companies in a few places to do this maybe is not that much when weighed against the time cost of lots of EAs doing the survey at 10min/response.
I was a pretty motivated EA (donated, sometimes read EA literature) who did consider myself an EA but was entirely disengaged from the community from 2013-2017, and then barely engaged from 2017-2020. Additionally, when I speak with other lawyers it’s not uncommon to hear that someone is either interested in EA or has begun donating to an EA charity, but that they haven’t gotten involved with the community because they don’t see how that would help them or anyone else do more good.
I don’t know how useful you think it would be to know more about the makeup and size of that population of unengaged EAs (or EA-adjacent folk, or whatever the label). Maybe it just wouldn’t be very decision-relevant for the orgs who have expressed interest in using the data. My initial sense is that it would be useful, but I don’t really know.
Thanks for the comments!
It’s just describing limitations. In principle, you could definitely update based on representative samples of the general population, but there would still be challenges.
Notably, we have already run a large representative survey (within the US), looking at how many people have heard of EA (for unrelated reasons). It illustrates one of the simple practical limitations of using this approach to estimate the composition of the EA community, rather than just to estimate how many people in the public have heard of EA.
Even with a sample of n=6000, we still only found around 150 people who plausibly even knew what effective altruism was (and we think this might still have been an over-estimate). Of those, I’d say no more than 1-3 seemed like they might have any real engagement with EA at all. (Incidentally, this is roughly a ratio that seems plausible to me for how many people who hear of EA actually then engaged with EA at all, i.e. 150-50:1 or less.) Note that we weren’t trying to see whether people were members of the EA community in this survey, so the above estimate is just based on those who happened to mention enough specifics- like knowing about 80,000 Hours- that it seemed like they might have been at all engaged with EA). So, given that, we’d need truly enormous survey samples to sample a decent number of ‘EAs’ via this method, and the results would still be limited by the difficulties mentioned above.
Thanks for taking the time to explain, David!