The 2014 Survey of Effective Altruists: Results and Analysis
It’s my great pleasure to announce that, after seven months of hard work and planning fallacy, the EA Survey is finally out.
It’s a long document, however, so we’ve put it together in an external PDF.
Introduction
In May 2014, a team from .impact and Charity Science released a survey of the effective altruist community. The survey offers data to supplement and clarify those anecdotes, with the aim of better understanding the community and how to promote EA.
In addition it enabled a number of other valuable projects—initial seeding of EA Profiles, the new EA Donation Registry and the Map of EAs. It also let us put many people in touch with local groups they didn’t know about, and establish presences in over 40 new cities and countries so far.
Summary of Important Findings
-
The survey was taken by 2,408 people, 1,146 (47.6%) of whom provided enough data to be considered, and 813 of whom considered themselves members of the EA movement (70.9%) and were included for the entire analysis.
-
The top three sources people in our sample first heard about EA from were LessWrong, friends, or Giving What We Can. LessWrong, GiveWell, and personal contact were cited as the top three reasons people continued to get more involved in EA. (Keep in mind that EAs in our sample might not mean all EAs overall… more on this later.)
-
66.9% of the EAs in our sample are from the United States, the United Kingdom, and Australia, but we have EAs in many countries. You can see the public location responses visualized on a map!
-
The Bay Area had the most EAs in our sample, followed by London and then Oxford. New York and Washington DC have surprisingly many EAs and may have flown under the radar.
-
The EAs in our sample in total donated over $5.23 million in 2013. The median donation size was $450 in 2013 donations.
-
238 EAs in our sample donated 1% of their income or more, and 84 EAs in our sample give 10% of their income. You can see the past and planned donations that people have chosen to made public on the EA Donation Registry.
-
The top three charities donated to by EAs in our sample were GiveWell’s three picks for 2013 -- AMF, SCI, and GiveDirectly. MIRI was the fourth largest donation target, followed by unrestricted donations to GiveWell.
-
Poverty was the most popular cause among EAs in our sample, followed by metacharity and then rationality.
-
33.1% of EAs in our sample are either vegan or vegetarian.
-
34.1% of EAs in our sample who indicated a career indicated that they were aiming to earn to give.
The Full Document
You can read the rest at the linked PDF! -->
A Note on Methodology
One concern worth putting in the forefront is that we used a convenience sample, trying to sample as many EAs as we can in places we knew where to find them. But we didn’t get everyone.
It’s easy to survey, say, all Americans in a reliable way, because we know where Americans live and we know how to send surveys to a random sample of them. Sure, there may be difficulties with subpopulations who are too busy or subpopulations who don’t have landlines (though surveys now call cell phones).
Contrast this with trying to survey effective altruists. It’s hard to know who is an EA without asking them first, but we can’t exactly send surveys to random people all across the world and hope for the best. Instead, we have to do our best to figure out where EAs can be found, and try to get the survey to them.
We did our best, but some groups may have been oversampled (more survey respondents, by percentage, from that group than are actually in the true population of all EAs) or undersampled (not enough people in our sample from that subpopulation to be truly representative). This is a limitation that we can’t fully resolve, though we’ll strive to improve next year. At the bottom of this analysis, we include a methodological appendix that has a detailed discussion of this limitation and why we think our survey results are still useful.
You can find much more than you’d ever want in the methodological appendix at the bottom of the PDF.
-
In sum, this is probably the most exhaustive study of the effective altruism movement in existence. It certainly exhausted us!
I’m really excited about the results and look forward to how they will be able to inform our movement.
- EA Survey 2018 Series: Do EA Survey Takers Keep Their GWWC Pledge? by (16 Jun 2019 23:04 UTC; 87 points)
- EA Survey 2018 Series: Donation Data by (9 Dec 2018 3:58 UTC; 83 points)
- EA Survey 2018 Series: Cause Selection by (18 Jan 2019 16:55 UTC; 69 points)
- EA Survey 2018 Series: Geographic Differences in EA by (18 Feb 2019 23:34 UTC; 68 points)
- EA Survey 2018 Series: How Long Do EAs Stay in EA? by (31 May 2019 0:32 UTC; 53 points)
- EA Survey 2018 Series: How welcoming is EA? by (28 Feb 2019 2:42 UTC; 50 points)
- EA Survey Series 2018 : How do people get involved in EA? by (18 Nov 2018 0:06 UTC; 50 points)
- EA Survey 2018 Series: Community Demographics & Characteristics by (21 Sep 2018 4:13 UTC; 39 points)
- EA Survey 2018 Series: Where People First Hear About EA and Influences on Involvement by (11 Feb 2019 6:05 UTC; 34 points)
- EA Survey 2018 Series: Group Membership by (11 Feb 2019 6:04 UTC; 34 points)
- EA Survey Series 2018: Subscribers and Identifiers by (26 Nov 2018 6:57 UTC; 30 points)
- 's comment on The marketing gap and a plea for moral inclusivity by (9 Jul 2017 0:41 UTC; 21 points)
- EA Survey 2018 Series: Distribution and Analysis Methodology by (18 Oct 2018 2:06 UTC; 15 points)
- EA Survey 2017 Series: Community Demographics & Beliefs by (29 Aug 2017 18:36 UTC; 15 points)
- .impact’s pivot to focus projects by (29 Apr 2016 19:59 UTC; 14 points)
- Please Take the 2018 Effective Altruism Survey! by (LessWrong; 2 May 2018 7:58 UTC; 14 points)
- EA Survey 2017 Series: Have EA Priorities Changed Over Time? by (6 Oct 2017 15:51 UTC; 12 points)
- Giving What We Can Pledge thoughts by (10 Jan 2017 21:09 UTC; 12 points)
- EA Facebook New Member Report by (26 Jul 2015 16:35 UTC; 11 points)
- EA Survey 2017 Series: Donation Data by (12 Sep 2017 1:29 UTC; 11 points)
- Common Misconceptions about Effective Altruism by (23 Mar 2015 9:25 UTC; 10 points)
- The 2017 Effective Altruism Survey—Please Take! by (24 Apr 2017 21:01 UTC; 10 points)
- EA Survey 2017 Series: Cause Area Preferences by (1 Sep 2017 14:55 UTC; 10 points)
- EA Survey 2017 Series: Demographics II by (18 Sep 2017 15:51 UTC; 9 points)
- The 2017 Effective Altruism Survey—Please Take! by (LessWrong; 24 Apr 2017 21:08 UTC; 9 points)
- EA Survey 2017 Series: How do People Get Into EA? by (17 Nov 2017 4:44 UTC; 7 points)
- 's comment on The Value of Those in Effective Altruism by (17 Feb 2016 13:35 UTC; 7 points)
- 's comment on EA Survey 2020: Demographics by (4 Sep 2021 6:56 UTC; 6 points)
- EA Survey 2017 Series: Distribution and Analysis Methodology by (29 Aug 2017 18:31 UTC; 6 points)
- EA Survey 2017 Series: Qualitative Comments Summary by (21 Sep 2017 1:36 UTC; 3 points)
- Peter’s 2015 Q1 Personal Review by (25 Apr 2015 1:33 UTC; 2 points)
- What Cause(s) Do You Support? And Why? by (22 Mar 2015 0:13 UTC; 2 points)
- 's comment on The Effective Altruism Newsletter & Open Thread – March 2016 Edition by (7 Mar 2016 17:00 UTC; 1 point)
- 's comment on Open Thread #38 by (24 Aug 2017 3:25 UTC; 1 point)
- Meetup : 2015 EA Survey Deep Dive and General Discussion by (15 May 2015 5:34 UTC; 0 points)
Thank you for doing this survey and analysis. I regret that the feedback from me was primarily critical, and that this reply will follow in a similar vein. But I don’t believe the data from this survey is interpretable in most cases, and I think that the main value of this work is as a cautionary example.
A biased analogy
Suppose you wanted to survey the population of Christians at Oxford: maybe you wanted to know their demographics, the mix of denominations, their beliefs on ‘hot button’ bioethical topics, and things like that.
Suppose you did it by going around the local churches and asking the priests to spread the word to their congregants. The local catholic church is very excited, and the priest promises to mention at the end of his sermon; you can’t get through to the Anglican vicar, but the secretary promises she’ll mention it in the next newsletter; the evangelical pastor politely declines.
You get the results, and you find that Christians in Oxford are overwhelmingly catholic, that they are primarily White and Hispanic, and tend conservative on most bioethical issues, and are particularly opposed to abortion and many forms of contraception.
Surveys and Sampling
Of course, you shouldn’t think that, because this sort of survey is shot through with sampling bias. You’d expect Catholics are far more likely to respond to the survey than evangelicals, so instead of getting a balanced picture of the ‘Christians in Oxford’ population, you get a picture of a ‘primarily Catholics in Oxford with some others’ – and predictably the ethnicity data and the bioethical beliefs are skewed.
I hope EA is non-denominational (or failing that, ecumenical), but there is a substructure to the EA population – folks who hang around LessWrong tend to be different from those who hang around Giving What We Can, for example. Further they likely differ in ways the survey is interested in: their gender, their giving, what causes they support, and so on. To survey of ‘The Effective Altruism Movement’, the EAs who cluster in both need to be represented proportionately (ditto all the other subgroups).
The original plan (as I understand) was to obviate the sampling concerns by just sampling the entire population. This was highly over-confident (when has a voluntary survey captured 90%+ of a target population?) and the consequences of its failure to become a de facto ‘EA census’ significant. The blanket advertising of the survey was taken up by some sources more than others: LessWrong put in on their main page, whilst Giving What We Can didn’t email it around – for example. Analogous to the Catholics and the Pentecostals, you would anticipate LWers to be significantly over-sampled versus folks in GWWC (or, indeed, versus many other groups, as I’d guess LW’s ‘reach’ to its membership via its main page is much better than many other groups). Consequently results like the proportion of EAs who care about AI/x-risk, where most EAs live, or what got them involved in EA you would predict to be slanted towards what LWers care about, where LWers live (bay area), or how LWers got involved in EA (LW!).
If the subgroups didn’t differ, we could breathe a sigh of relief. Alas, not so: the subgroups identified by URL significantly differ across a variety of demographic information, and their absolute size (often 10-20%) makes the difference practically as well as statistically significant – I’d guess if you compared ‘where you heard about EA’ against URL, you’d see an even bigger difference. It may understate the case – if one moved from 3 groups (LW, EA FB, contacts) to 2 (LW, non-LW), one may see more differences, and the missing variable issues and smaller subgroup size mean the point estimates for (e.g.) what proportion of LWers care about X-risk is not that reliable.
Convenience sampling is always dicey, as unlike probabilistic sampling any error in parameter estimate due to bias will not expectedly diminish as you increase the sample size. However, the sampling strategy in this case is particularly undesirable as the likely bias runs pretty much parallel to the things you are interested in: you might hope that (for example) the population of the EA facebook might not be too slanted in terms of cause selection compared to the ‘real’ EA population – not a group like GWWC, LW, CFAR, etc.
What makes it particularly problematic is that it is very hard estimate the ‘size’ of this bias: I wouldn’t be surprised if this survey only oversampled LWers by 5-10%, but I wouldn’t be that surprised if it oversampled LWers by a factor of 3 either. The problem is that any ‘surprise’ I get from the survey mostly goes to adjusting my expectation of how biased it is. Suppose I think ‘EA’ is 50% male and I expect the survey to overestimate the %age male by 15%. Suppose the survey said EA was 90% male. I am going to be much more uncertain about the degree of over-representation than I am about what I think the ‘true EA male fraction’ is. So the update will be to something like 52% male and the survey overestimating by 28%. To the extent I am not an ideal epistemic agent, feeding me difficult to interpret data might make my estimates worse, not better.
To find fault is easy; to plan well, difficult
Science rewards caution and planning; many problems found in analysis could only have been fixed in design, and post-hoc cleaning of data is seldom feasible and still seldomer easy. Further planning could have made the results more interpretable. Survey design has a variety of jargon like “population definition”, “sampling frame”. More careful discussion of what the target population was and how they were going to be reached could have flagged the sampling bias worry sooner, likewise how likely a ‘saturation’ strategy was to succeed. As it was most of the discussion seemed to be focused on grabbing as many people as possible.
Similarly, ‘baking in’ the intended analysis plan with the survey itself would have helped to make sure the data could be analysed in the manner intended (my understanding – correct me if I’m wrong! – is that the planning of exactly what analysis would be done happened after the survey was in the wild). In view of the sampling worries, the analysis was planned to avoid giving aggregate measures sensitive to sampling bias, but instead explore relationships between groups via regression (e.g. what factors predict amount given to charity). However, my understanding is this pre-registered plan had to be abandoned as the data was not amenable. Losing the pre-registered plan for a new one which shares no common elements is regrettable (especially as the new results are very vulnerable to sampling bias), and a bit of a red flag.
On getting better data, and on using data better
Given the above, I think the survey offers extremely unreliable data. I’m not sure I agree with the authors it is ‘better than nothing’, or better than our intuitions—given most of us are imperfect cognizers, it might lead us more astray to the ‘true nature’ of the EA community. I am pretty confident it is not worth the collective time and energy it has taken: it probably took a couple of hundred hours of the EA community’s time to fill in the surveys, leave alone the significant work from the team in terms of design, analysis, etc.
Although some things could not have been helped, I think many things could have, and there were better approaches ex ante:
1) It is always hard to calibrate one’s lack of knowledge about something. But googling things like ‘survey design’, ‘sampling’, and similar are fruitful – if nothing else, they suggest that ‘doing a survey’ is not always straightforward and easy, and put one on guard for hidden pitfalls. This sort of screening should be particularly encouraged if one isn’t a domain expert: many things in medicine concord with common sense, but some things do not, likewise statistics and analysis, and no doubt likewise many other matters I know even less about.
2) Clever and sensible the EA community generally is, it may not always be sufficient to ask for feedback on a survey idea and then interpreting the lack of response as a tacit green light. Sometimes ‘We need expertise and will not start until we have engaged some’, although more cautious, is also more better. I’d anticipate this concern will grow in significance as EAs tackle things ‘further afield’ from their background and training.
3) You did get a relative domain expert raise the sampling concerns to you within a few hours of going live. Laudable though it was that you were responsive to this criticism and (for example) tracked URL data to get a better handle on sampling concerns, invited your critics to review prior drafts and analysis, and mention the methodological concerns prominently, it took a little too long to get there. There also seemed a fair about of over-confidence and defensiveness – not only from some members of the survey team, but from others who thought that, although they hadn’t considered X before and didn’t know a huge amount about X, that on the basis of summary reflection X wasn’t such a big deal. Calling a pause very early may have been feasible, and may have salvaged the survey from the problems above.
This all comes across as disheartening. I was disheartened too: effective altruism intends to put a strong emphasis on being quantitative, getting robust data, and so forth. Yet when we try to practice what we preach, our efforts leave much to be desired (this survey is not the only – or the worst – example). In the same way good outcomes are not guaranteed by good intentions, good information is not guaranteed by good will and hard work. In some ways we are trailblazers in looking hard at the first problem, but for the second we have the benefit of the bitter experience of the scientists and statisticians who have gone before us. Let us avoid recapitulating their mistakes.
Thanks for sharing such detailed thoughts on this Greg. It is so useful to have people with significant domain expertise in the community who take the time to carefully explain their concerns.
It’s worth noting there was also significant domain expertise on the survey team.
Why isn’t the survey at least useful count data? It allows me to considerably sharpen my lower bounds on things like total donations and the number of Less Wrong EAs.
I think count data is the much more useful kind to take away even ignoring sampling bias issues, because the data in the survey is over a year old, i.e. Even if it were a representative snapshot of EA in early 2014, that snapshot would be of limited use. Whereas most counts can safely be assumed to be going up.
I agree the survey can provide useful count data along lines of providing lower bounds. With a couple of exceptions though, I didn’t find the sort of lower bounds the survey gives hugely surprising or informative—if others found them much moreso, great!
Very thoughtful post.
Are there any types of analysis you think could be usefully performed on the data?
Once could compare between clusters (or, indeed, see where there are clusters), and these sorts of analyses would be more robust to sampling problems: even if LWers are oversampled compared to animal rights people, one can still see how they differ. Similar things like factor analysis, PCA etc. etc. could be useful to see whether certain things trend together, especially for when folks could pick multiple options.
Given that a regression-style analysis was abandoned, I assume actually performing this sort of work on the data is much easier said than done. If I ever get some spare time I might look at it myself, but I have quite a lot of other things to do...
One approach would be to identify a representative sample of the EA population and circulate among folks in that sample a short survey with a few questions randomly sampled from the original survey. By measuring response discrepancies between surveys (beyond what one would expect if both surveys were representative), one could estimate the size of the sampling bias in the original survey.
ETA: I now see that a proposal along these lines is discussed in the subsection ‘Comparison of the EA Facebook Group to a Random Sample’ of the Appendix. In a follow-up study, the authors of the survey randomly sampled members of the EA Facebook group and compared their responses to those of members of that group in the original survey. However, if one regards the EA Facebook group as a representative sample of the EA population (which seems reasonable to me), one could also compare the responses in the follow-up survey to all responses in the original survey. Although the authors of the survey don’t make this comparison, it could be made easily using the data already collected (though given the small sample size, practically significant differences may not turn out to be statistically significant).
I think it’s right to say that the survey was premised on the idea that there is no way to know the true nature of the EA population and no known-to-be-representative sampling frame. If there were such a sampling frame or a known-to-be-representative population, we’d definitely have used that. Beforehand, and a little less so now, I would have strongly expected the EA Facebook group to not be representative. For that reason I think randomly sampling the EA FB group is largely uninformative- and I think that this is now Greg’s view too, though I could be wrong.
I agree that could work, although doing it is not straightforward—for technical reasons, there aren’t many instances where you get added precision by doing a convenience survey ‘on top’ of a random sample, although they do exist.
(Unfortunately, random FB sample was small, with something like 80% non-response, thus making it not very helpful to sample sampling deviation from the ‘true’ population. In some sense the subgroup comparisons do provide some of this information by pointing to different sub-populations—what they cannot provide is a measure as to whether these subgroups are being represented proportionally or not. A priori though, that would seem pretty unlikely.)
As David notes, the ‘EA FB group’ is highly unlikely to be a representative sample. But I think it is more plausibly representative along axes we’d be likely to be interested in the survey. I’d guess EAs who are into animal rights are not hugely more likely to be in facebook in contrast to those who are into global poverty, for example (could there be some effects? absolutely—I’d guess FB audience skews young and computer savvy, so maybe folks interested in AI etc. might be more likely to be found there, etc. etc.)
The problem with going to each ‘cluster’ of EAs is that you are effectively sampling parallel rather than orthogonal to your substructure: if you over-sample the young and computer literate, that may not throw off the relative proportions of who lives where or who cares more about poverty than the far future; you’d be much more fearful of this if you oversample a particular EA subculture like LW.
I’d be more inclined to ‘trust’ the proportion data (%age male, %xrisk, %etc) if the survey was ‘just’ of the EA facebook group, either probabilistically or convenience sampled. Naturally, still very far from perfect, and not for all areas (age, for example). (Unfortunately, you cannot just filter the survey and just look at those who clicked through via the FB link to construct this data—there’s plausibly lots of people who clicked through via LW but would have clicked through via FB if there was no LW link, so ignoring all these responses likely inverts anticipated bias).
Thanks for running the survey, writing it up, and posting the data. I think this is chiefly valuable for giving people an approximate overview of what we know about the movement, so it’s great to have the summary document which does that.
I would have preferred fewer attempts to look for statistical significance, as I’m not sure they ever helped much and think they have led you to at least one misleading conclusion. In particular:
On the contrary, I think the main message from the data is that in the sample collected, they are roughly evenly split. The biggest of the four beats the smallest by less than a factor of two—this is a relatively small difference when there are no mechanisms I can see which should equalise their size (I would not have been shocked if you’d found an order of magnitude difference between some two of them).
Doing a test here for statistical significance is basically checking the hypothesis that survey participants were drawn from a distribution with exactly equal proportions supporting the four causes. But that’s obviously nonsense—we don’t need a big survey to tell us that. It does tell us that poverty is the biggest (where we might not have been confident about which was), but statistical significance is misleading in terms of what that means—the raw ratios are more informative.
Thanks for the feedback. I agree that particular test/conclusion was unnecessary/misleading. I think we’ll be more careful to avoid tests like that in future survey analyses :)
It’s hard to say. Others have told me that they greatly preferred backing up these kinds of statements with statistical testing. I guess I can’t make everyone happy. :)
OK, I guess I inferred the causality as being you did the test, then wrote the statement. If you were going to use the same language anyway, I agree that the test doesn’t hurt—but I think that this statement might have been better left out or weakened.
I agree with the spirit of this criticism, though it seems that the problem is not significance testing as such, but a failure to define the null hypothesis adequately.
Thank you to the survey team for completing what is an easy-to-underestimate volume of work. Thank you also to the many who completed this survey, helping us to both understand different EA communities better and to improve this process of learning about ourselves as a wider group in future years.
I have designed and analysed several consumer surveys professionally as part of my job as a strategy consultant.
There is already a discussion of sample bias so I will leave those issues alone in this post and focus on three simple suggestions to make the process easier and more reliable for when this valuable exercise is repeated next year.
Firstly, we should use commercial software to operate the survey rather than trying to build something ourselves. These are both less effort and more reliable. For example, SurveyMonkey could have done everything this survey does for about £300. I’m happy to pay that myself next year to avoid some of the data quality issues.
Secondly, we should use live data validation to improve data collection, data integrity and ease of analysis. SurveyMonkey or other tools can help John to fill in his age in the right box. It could refuse to believe the 7 year old, and suggest that they have another go at entering their age. It could also be valuable to do some respondent validation by asking people to answer a question with a given answer, removing any random clickers or poor quality respondents who are speeding through (eg “Please enter the number ‘2’ in letters into the textbox to prove you are not a robot. For example, the number ‘1’ in letters is ‘one’”)
Thirdly, we should do more testing by trying out draft versions with respondents who have not written the survey. It is very, very hard to estimate how people are going to read a particular question, or which options should be included in multiple choice questions. Within my firm, it is typical for an entire project team to run through a survey several times before sending it out to the public. Part of the value here is that most team members were not closely involved in writing the survey, and so won’t necessarily be reading it in the way the author expected them to read it. I would suggest you want to try any version of the survey out with a large group (at least twenty) of different people who might answer it, to catch the interpretations of questions which different groups might have. Does the EA affiliation filter work as hoped for? Are there important charities which we should include in the prompt list? It does not seem unreasonable to pilot and redraft a few times with a diverse group of willing volunteers before releasing generally.
The analysis throws up several interesting conclusions, and I have learned a lot by reading through it. The main shocks are: the relatively low levels of donations in $ terms by many self-identified EAs, the relatively low proportion of EAs identifying chapters/local groups as a reason for joining or identifying with the community and, (for me) the encouragingly high proportion of respondents who are vegetarian or vegan.
I’m going to set aside some time in May to go through the data in a ‘consulting’ sort of way to see if that approach throws up anything interesting or different to others and will circulate with the survey team before publishing here.
Thanks Chris, all very useful info.
(On the 0 donors question: I’ve written about this elsewhere in the comments and a sizeable majority of these respondents were full time students or low income or had made significant past donations or had pledged at least (and often much more) of future income). Once all these people are taken account of, the number of 0 donors was pretty low. There was a similar (if not even stronger) trend for people donating <$500).
Thanks Chris, this is useful feedback and we’ll go through it. For example, I think trying out draft versions would be valuable. I may ask you some more questions, e.g. about SurveyMonkey’s features.
Happy to answer these any time, and happy to help out next year (ideally in low time commitment ways, given other constraints).
Thanks for this, and thanks for putting the full data on github. I’ll have a sift through it tonight and see how far I get towards processing it all (perhaps I’ll decide it’s too messy and I’ll just be grateful for the results in the report!).
I have one specific comment so far: on page 12 of the PDF you have rationality as the third-highest-ranking cause. This was surprisingly high to me. The table in imdata.csv has it as “Improving rationality or science”, which is grouping together two very different things. (I am strongly in favour of improving science, such as with open data, a culture of sharing lab secrets and code, etc.; I’m pretty indifferent to CFAR-style rationality.)
Good point. Yes, this is my bad, I forgot that part. Definitely a mistake.
Definitely will break that apart next time.
“238 EAs in our sample donated 1% of their income or more, and 84 EAs in our sample give 10% of their income.”
I was surprised by this. In particular, 22% (127/588) of people identifying as EAs do not donate. (Of course they may have good reasons for not donating, e.g. if they are employed by an EA charity or if they are currently investing in order to give more in the future). Do we know why so many people identify as EAs but do not presently donate?
Probably because the average age is so low (~25) - lots of students and people just starting out their careers.
Because self-identifying as EA is a lot easier than being self-sacrificing and donating. I saw the numbers with students removed and they did not improve as much as you would think.
The raw data seems to show that a lot of people who have donated zero have nevertheless pledged to donate a significant amount (e.g. everything above living expenses etc.).
The survey question only asked if people though they could be described ‘however loosely’ as an effective altruist. I suspect this question did not perform as intended—we know it included people who said they not heard of the term.
People will say anything on surveys. Many respondents go through clicking randomly. You can write a question that says, “Please answer C,” and >10% of respondents will still click something other than C.
This year it might be worth including a mandatory questions saying something like “Check C to promise not to go through clicking randomly”, both as a test and a reminder.
I regularly do this when designing consumer surveys as part of m professional work—the concern in those instances is that respondents are mainly completing the survey for a small monetary reward and so are incentivised to click through as fast as possible. To help my own survey development skills, I participate in several online panels and can confirm that whilst not exactly standard practice, a non-negligible proportion of online consumer surveys will include questions like this used to screen out respondents who are not paying attention.
This is less of a concern for the EA survey, but is almost costless to include such a screening question so seems like an easy way to help validate any resulting analysis or conclusions.
What evidence could we get now or in the future that’d speak to the different hypotheses being offered in response to this?
I think as it stands the evidence is already pretty good (on this particular question). I only skimmed through the zero donors briefly, and I may go through in more detail in the future, or someone else can do it, but I found that a pretty substantial majority of the zero donors were either students or on a “low income” based on my quick informal and pretty conservative coding of people as “low income” if they were on roughly the UK minimum wage or less. I would guesstimate at least around 70% were full time or “low income” by this measure. But I then went through counting those who had donated a significant sum in the past (roughly $1000 or higher) or had pledged more than 10% in the future, and again, this was a majority, including some people (roughly 10) who weren’t in the student/low income group, so yet more were students or low income or pledged 10% or previously donated substantial amount.
I also extended this analysis to the people pledging <$500 but more than $0, and the figures were even more overwhelming: by my count roughly 90% were students or low income.
I’m being conservative given the rough nature of my coding and counting, but I think these figures are still too low, because I didn’t count people who were on around $20,000 per year (which was quite a few, and still a lowish income), people who pledged 5% (which is still respectable) or just said something vague like “Yes” about future donations. Just to give a flavour of the pledges: a significant number were even higher than the GWWC pledge (i.e. for roughly every 2 people pledging 10%, within the zero donors, there was 1 pledging much more than that: 50% income, 90% income, everything above basic living expenses and so on). So far from being not-EAs at all, a lot of these people seem like exemplary EAs!
Is this is a typo? Do you mean “not full time” or not working or “full time students”?
Yeh, thanks for the spot, that is just a typo: it should be “full time students”
I haven’t looked at correlations between various data-sets yet, so I’m not confident. I’ll update on this later. Anyway, many who took the survey are likely university students, including perhaps a disproportionate number of graduate students as effective altruism is skewed towards the academic world. This may delay who’s in a good position to donate by a few years.
I’ve made a bar chart plotter thing with the survey data: link.
Also, do you have the GitHub code for your plotter? Would love to see.
Woah, this is an impressive data viz accomplishment! You should make it a top-level post—it’s cooler than a comment. :)
-
Also, …
How did you do that so quickly? We had to pay $60 to get it done manually via virtual assistants.
Thanks Peter! I’ll make the top-level post later today.
(I might have given the impression that I did this all during a weekend. This isn’t quite right—I spent 2-3 evenings, about 8 hours in total, going from the raw csv files to nice and compact .js function. Then I wrote the plotter on the weekend.)
I did this bit in Excel. If the money amounts were in column A, I insert three columns to the right: B for the currency (assumed USD unless otherwise specified), C for the min of the range given, D for the max. In column C, I started with =IF(ISNUMBER(A2), A2, “”) and dragged that formula down the column. Then I went through line by line, reading off any text entries, and turning them into currency/min/max (if a single value was reported, I entered it as the min, and left the max blank). currency, tab, number, enter, currency, tab, number, tab, number, enter, currency, tab...
It’s not a fun way to spend an evening (hence why I didn’t do the lifetime donations as well), but it doesn’t actually take that long.
Then: new column E for the AVERAGE(C2:D2) dragged down the column. Then I typed in the average currency conversions for 2013 into a new sheet and did a lookup (most users would use VLOOKUP I think, I used MATCH and OFFSET) to get my final USD numbers in column F.
As a fierce partisan of the “_final_really_2” school of source control, I’m yet to learn how to GitHub. You can view the Javascript source easily enough though, and save it locally. (I suggest deleting the Google Analytics “i,s,o,g,r,a,m” script if you do this, or your browser might go looking for Google in your file system for a few seconds before plotting the graphs). The two scripts not in the HTML file itself are d3.min.js and ea_survey.data.js. (EDIT: can’t be bothered fixing the markdown underscores here.)
A zip file with my ready-to-run CSV file and the R script to turn it into a Javascript function is here.
Well I admire your dedication to do it yourself and not use my conversions. :)
-
Aww man, you should learn. It’s tremendously useful. Not to mention a requirement for any programming job.
PDF link doesn’t exist anymore. @Peter_Hurford
Thanks. We’re working on relocating it and will fix the link when we do.
UPDATE: fixed
The link is fixed now.
The first 17 entries in imdata.csv have some mixed-up columns, starting (at latest) from
Have you volunteered or worked for any of the following organisations? [Machine Intelligence Research Institute]
until (at least)
Over 2013, which charities did you donate to? [Against Malaria Foundation].
Some of this I can work out (volunteering at “6-10 friends” should obviously be in the friends column), but the blank cells under the AMF donations have me puzzled.
Yes, it looks like the first 17 entries are corrupted for some reason. I’ll look into it.
The previous link to the survey results died, so I edited to update the link.
Do you have any sense of the extent to which people who put down ‘friend’ as how they got into effective altruism might have learned about EA via their friend taking them to a meet-up group, or might have ended up getting properly committed because they made friends with people through a meet-up group? I was just thinking about how I would classify myself as having got into EA through friends, but that you might think it was more accurate to describe it as a meet-up group in Oxford getting me involved.
So, there were 2 questions, one speaking to how people “learned about EA” and one to how they “ended up getting properly committed”:
“How did you first hear about ‘Effective Altruism’?” (single response)
“Which factors were important in ‘getting you into’ Effective Altruism, or altering your actions in its direction?” (multiple response)
The second question covers the intersection you describe; I think you can see the overlap by selecting it as both the primary and secondary categories at http://pappubahry.com/misc/ea_survey/ . Beyond that, we don’t have any data (except comments at the end of the survey). I don’t know of course, but I’d guess that in your case talking to friends was important in getting you into it and attending local group events regularly maybe wasn’t—even if those friends attended local group events.
Thanks, that’s interesting. The reason I was thinking that it might be more accurately attributed to a local group was that it seems unlikely I would have really formed friendships with any of the people around if they hadn’t been setting up Giving What We Can.
Ah, OK—I was thinking of you being friends with Will MacAskill before from being in his masters cohort. I guess the central GWWC organisation in Oxford counts as a local group, just of a different sort than I was thinking of! (And then of course you’d have gone to events GWWC put on in Oxford University too.)
I’m trying to make sense of all the missing data. It seems very strange to have such a high non response rate (nearly 20%) to simple demographic questions such as gender and student status, and this suggests a problem with the data.
You say here that a ‘survey response’ was generated each time somebody opened the survey, even if they answered no questions. Does that mean there wasn’t a ‘complete and submit’ step? Was every partially completed survey considered a separate ‘person’? If so, was there any way to determine if individuals were opening multiple times?
If each (however complete) opening of the survey was considered an entry (ie whatever data was entered has been counted as a person), that would suggest that individuals making several attempts to complete the survey are being multiply counted. That would be supported if the non response rates are generally higher later in the survey, which I can’t tell from this report.
If multiple attempts can’t be excluded, these numbers are unlikely to be valid. Missing data is a difficult problem, but my first thought is that a safer approach would be to only include complete responses
When ACE and HRC talked to statisticians and survey researchers as part of developing our Survey Guidelines for animal charities beginning to evaluate their own work, they consistently said that demographic questions should go at the end of the survey because they have high non-response rates and some people don’t proceed past questions they aren’t answering. So while it’s intuitively surprising that people don’t answer these simple questions, it’s not obviously different from what (at least some) experts would expect. I don’t know, however, whether 20% is an especially high non-response rate even taking that into account.
That’s interesting to know, thank you for sharing it! Looking at this study (comparing mail and web surveys) they cited non response rates to demographic items at 2-5%. However I don’t know how similar the target population here is to the ‘general population’ in these behaviours. http://surveypractice.org/index.php/SurveyPractice/article/view/47/html
Yes, these questions were right at the end. You can see the order of the questions in the spreadsheet that Peter linked to—they correspond to the order of the columns.
Thanks Tom. I’m limited in my spreadsheet wrangling at the moment I’m afraid, but looking at non response rates that are cited in the document above, and comparing to the order or questions, non responses seem to be low (30-50) until the question on income and donation specifics, after which they are much higher (150-220). A question that requires financial specifics seems likely to require someone to stop and seek documents, so could well cause someone to abandon the survey at least temporarily. If somebody abandoned the survey at that point, would the information they had entered so far be submitted? Or would they have to get to the end and cluck submit for any of their data to be included?
That’s a good point, could well have happened, and is something we should consider changing.
The questions were split into a few pages, and people’s answers got saved when they clicked the ‘Continue’ button at the bottom of each page—so if they only submitted 2 pages, only those pages would be saved. We searched for retakes and saw a small number which we deleted.
Oh cool. How were you able to identify duplicates?
I looked for identical names or email addresses, and then manually checked them. The other thing we could do would be to record people’s IP addresses and look for identical ones. However I chose not to record them due to privacy concerns. I would promise not to use them to try guessing who people are, and this identifying data never gets shared with anyone but me—I’d appreciate feedback on whether people would be comfortable with IP addresses getting recorded given this.
Tom was the one who created the survey taking architecture, so I’ve asked him to get back to you, just to make sure I don’t give you incorrect information.
Thanks for the survey! An interesting read. One question, two comments:
1 How do I read the graph on p10?
2
It cannot be correct to say that poverty is clearly leading the four because in the next sentence you say that that metacharity was arguably more popular than it.
3
Thanks for all the work put into this, and it’ll be good to see what thinking unfolds from it.
x-axis is the percent of income donated. y-axis is the number of people who have donated that percent or more.
-
It is “clearly leading the four” under one interpretation, which I go on to suggest is probably a narrow one.
1 thanks, that makes sense. Maybe it’s just me, but when I see this kind of graph, I’m expecting the y-axis to be the response variable e.g. percent of income donated, and the x-axis to represent the rank of the people like this.
2 I know the report acknowledges that its interpretation is questionable there but I think that understates the problem. Rather, I think it proposes then misapplies a categorisation...
Imagine that I make a survey regarding the world’s religions, and I made survey with options “Anglican”, “Catholic”, “Islamic”, “Buddhist”, “Hindu”, et cetera, and then I say that reading theological scholar so-and-so’s categorisation of the four great religions—Christian, Islamic, Buddhist and Hindu—Hindu is clearly the most popular. And then I go on to clarify that if you add Anglican and Catholic together, you end up with more than in Hindu. Then I go on to say that these are two different “interpretations”, one of which is “probably narrow”...
I’m not au fait with metacharity or rationality—can you explain why rationality should be bundled under metacharity? What is the meta plan behind promotion of rationality (particular in it’s specific forms, like the organisation CFAR)?
Is this really true as strongly as that Anglican and Catholic should be bundled under Christian? I guess the point of your analogy may have been that Four Focus Areas of Effective Altruism classed rationality/CFAR under meta EA, which is fair.
Yeah, I don’t think that they’re as similar as Anglicanism or Catholicism, I’m just saying that you have to consistently apply your chosen scheme.
But I do in fact think that they’re decently related. EA Outreach wants to increase the number of people trying to do good. 80,000 Hours wants to move altruists to impactful careers (increasing quality of altruistic activities). GPP wants to get altruists working on higher-value projects. CFAR wants altruists to be equipped with better thinking skills (and non-altruists). The common thread is that they all want to increase the number, build the capacities and improve the target projects of altruists.
Makes sense, but points to meta being an unusually broad and unspecific description.
That’s fair. It’s a cumulative frequency graph.
-
I mean it’s a matter of opinion, so I thought I’d provide the different interpretations. Sorry if you feel that I editorialized too much.
It’s interesting that only around 10% of self-identified EA’s report donating 10% or more of their income. That makes me feel less guilty about “only” giving 10%. :)
Is there a way to easily pull out the % of their income that people gave in 2013 in a more granular way—e.g. people who gave 10.0% vs. 10.1% vs 12%? I don’t see the data with the currencies converted in github.
This is action relevant for us at Charity Science because assuming GWWC pledgers stick carefully to 10.0% getting them to donate to a particular fundraiser has no counterfactual value.
23 people donated precisely 10%.
1 person donated precisely 15%.
46 people donated between 10 and 15%, such as 11.8%, 12.4% 12.6%, 14.9%, 10.6%...
So it does seem more common than not.
Someone at GWWC might be able to confirm with data from MyGiving.
Tom mentioned that the raw data would be shared (in an anonymized form)?
Also, I recall there being questions about politics—maybe about which other movements we were involved in?
Yes. All data and scripts are publicly available in the GitHub repository for the project.
-
Yes, these questions were in the survey (and available in the anonymized data), but were not included in our analysis. Other people can feel free to do further analysis with the politics data.
Right: 21 Centre-right: 17 Centre: 60 Centre-left: 222 Left: 299
Libertarian: 128
Particularly high non-response rate though (was one of the last questions).
I’m sure you’d get high non-response on political questions regardless, for obvious reasons.