sorry, I see now that you’ve discussed the point in my comment below (which I’ve now put in italics) in the linked document. I’m grateful for, but not surprised at, the care and thought that’s gone into this.
If it’s not too much of your time, I just am curious about one more thing. Is the paragraph below saying that surveying the general population would not provide useful information, or is it saying something like ‘this would help, but would not totally address the issue’. Like, is there any information value in doing this—or would it basically be pointless/pseudoscientific?
Surveying the general (non-EA) population as part of larger representative surveys to get a sense of the overall composition of EAs (e.g., the gender ratio). However, differential non-response, to these larger surveys would again throw this in doubt. Standard corrections may not be easy: and relative non-response among EAs (e.g., male versus female EAs) may differ from the relative non-response to such surveys in other populations.
***
Original comment:
Thanks for the in depth response, David.
The difficulty here is that it doesn’t seem to be possible to actually randomly sample from the EA population
Sorry, I explained poorly what I meant. What I meant to ask was whether you could randomly sample from a non-EA frame, identify EAs based on their responses (presumably a self identification question), and then use that to get some sense of the attributes of EAs.
One problem might be that the prevalence of EAs in that non-EA population might be so minuscule that you’d need to survey an impractical number of people to know much about EAs.
Another response is that it just wouldn’t be that useful to know, although the cost involved in hiring polling companies in a few places to do this maybe is not that much when weighed against the time cost of lots of EAs doing the survey at 10min/response.
I was a pretty motivated EA (donated, sometimes read EA literature) who did consider myself an EA but was entirely disengaged from the community from 2013-2017, and then barely engaged from 2017-2020. Additionally, when I speak with other lawyers it’s not uncommon to hear that someone is either interested in EA or has begun donating to an EA charity, but that they haven’t gotten involved with the community because they don’t see how that would help them or anyone else do more good.
I don’t know how useful you think it would be to know more about the makeup and size of that population of unengaged EAs (or EA-adjacent folk, or whatever the label). Maybe it just wouldn’t be very decision-relevant for the orgs who have expressed interest in using the data. My initial sense is that it would be useful, but I don’t really know.
Is the paragraph below saying that surveying the general population would not provide useful information, or is it saying something like ‘this would help, but would not totally address the issue’.
It’s just describing limitations. In principle, you could definitely update based on representative samples of the general population, but there would still be challenges.
Notably, we have already run a large representative survey (within the US), looking at how many people have heard of EA (for unrelated reasons). It illustrates one of the simple practical limitations of using this approach to estimate the composition of the EA community, rather than just to estimate how many people in the public have heard of EA.
Even with a sample of n=6000, we still only found around 150 people who plausibly even knew what effective altruism was (and we think this might still have been an over-estimate). Of those, I’d say no more than 1-3 seemed like they might have any real engagement with EA at all. (Incidentally, this is roughly a ratio that seems plausible to me for how many people who hear of EA actually then engaged with EA at all, i.e. 150-50:1 or less.) Note that we weren’t trying to see whether people were members of the EA community in this survey, so the above estimate is just based on those who happened to mention enough specifics- like knowing about 80,000 Hours- that it seemed like they might have been at all engaged with EA). So, given that, we’d need truly enormous survey samples to sample a decent number of ‘EAs’ via this method, and the results would still be limited by the difficulties mentioned above.
Edit:
sorry, I see now that you’ve discussed the point in my comment below (which I’ve now put in italics) in the linked document. I’m grateful for, but not surprised at, the care and thought that’s gone into this.
If it’s not too much of your time, I just am curious about one more thing. Is the paragraph below saying that surveying the general population would not provide useful information, or is it saying something like ‘this would help, but would not totally address the issue’. Like, is there any information value in doing this—or would it basically be pointless/pseudoscientific?
***
Original comment:
Thanks for the in depth response, David.
Sorry, I explained poorly what I meant. What I meant to ask was whether you could randomly sample from a non-EA frame, identify EAs based on their responses (presumably a self identification question), and then use that to get some sense of the attributes of EAs.
One problem might be that the prevalence of EAs in that non-EA population might be so minuscule that you’d need to survey an impractical number of people to know much about EAs.
Another response is that it just wouldn’t be that useful to know, although the cost involved in hiring polling companies in a few places to do this maybe is not that much when weighed against the time cost of lots of EAs doing the survey at 10min/response.
I was a pretty motivated EA (donated, sometimes read EA literature) who did consider myself an EA but was entirely disengaged from the community from 2013-2017, and then barely engaged from 2017-2020. Additionally, when I speak with other lawyers it’s not uncommon to hear that someone is either interested in EA or has begun donating to an EA charity, but that they haven’t gotten involved with the community because they don’t see how that would help them or anyone else do more good.
I don’t know how useful you think it would be to know more about the makeup and size of that population of unengaged EAs (or EA-adjacent folk, or whatever the label). Maybe it just wouldn’t be very decision-relevant for the orgs who have expressed interest in using the data. My initial sense is that it would be useful, but I don’t really know.
Thanks for the comments!
It’s just describing limitations. In principle, you could definitely update based on representative samples of the general population, but there would still be challenges.
Notably, we have already run a large representative survey (within the US), looking at how many people have heard of EA (for unrelated reasons). It illustrates one of the simple practical limitations of using this approach to estimate the composition of the EA community, rather than just to estimate how many people in the public have heard of EA.
Even with a sample of n=6000, we still only found around 150 people who plausibly even knew what effective altruism was (and we think this might still have been an over-estimate). Of those, I’d say no more than 1-3 seemed like they might have any real engagement with EA at all. (Incidentally, this is roughly a ratio that seems plausible to me for how many people who hear of EA actually then engaged with EA at all, i.e. 150-50:1 or less.) Note that we weren’t trying to see whether people were members of the EA community in this survey, so the above estimate is just based on those who happened to mention enough specifics- like knowing about 80,000 Hours- that it seemed like they might have been at all engaged with EA). So, given that, we’d need truly enormous survey samples to sample a decent number of ‘EAs’ via this method, and the results would still be limited by the difficulties mentioned above.
Thanks for taking the time to explain, David!