New data suggests the ‘leaders’’ priorities represent the core of the community

Benjamin_Todd11 May 2020 13:07 UTC
106 points
26 commentsEA link

There have been several surveys of attendees of what was previously called the ‘EA Leaders Forum’ about what they believe the community’s cause priorities should be (2017, 2018, 2019).

The relevance of these results have been criticised each year for not representing the views of the wider community (e.g. in 2019). You can see some hypotheses about why there might be differences in views between the Leaders Forum group and others here. In particular, the former group is selected to be more into longtermism than the average, potentially skewing the results.

Similarly, CEA and 80,000 Hours have been criticised for not having their priorities line up with the community (e.g. this post summarises criticisms about representativeness, and 80k was recently criticised here).

Critics of the Leaders Forum survey often appeal to the EA Survey as a reflection of the community’s priorities. This is a much broader survey with about 2,500 respondents, and when we compare the ideal portfolio chosen in the Leaders Forum Survey to the top cause preferences of the EA Survey respondents (as explained in more depth later), we see significant differences:

The problem with this is the EA Survey is open to anyone to take, and without knowing more about who’s taking it, it’s unclear why this should be used to set community priorities.

Fortunately, the 2019 EA Survey lets us make progress on this question. It asked many more questions about people’s level of engagement, which means we can further subdivide the responses.

In this post, I do a quick analysis of these new results to show that based on this data, the Leaders Forum survey priorities seem to reflect the most engaged half of the community – around 1,000-2,000 people – pretty well. This suggests the issue we actually face is not a difference between the leaders and the core of the community, but rather between the core and those who are new or moderately engaged.

The data

In the 2019 EA Survey, one question asked about how engaged people feel they are on a scale of 1 to 5, where 5 was defined as “I am heavily involved in the effective altruism community, perhaps helping to lead an EA group or working at an EA-aligned organization. I make heavy use of the principles of effective altruism when I make decisions about my career or charitable donations.”[1]

About 450 out of 2,100 respondents to this question reported a “5”. My estimate is that only about 40% of engaged EAs filled out the survey in 2019,[2] which would suggest there are around ~1,000 people with this level of engagement, so it’s a much wider group than the ~30 or so people at the leaders forum.

If we look at what different members of this group said they think is the ‘top cause’, and compare that to the ‘ideal portfolio’ chosen by the mean attendee of the Leaders Forum,[3] (making some assumptions about how the categories line up) we find:

As you can see, these line up pretty well, except that the EA Survey group actually has a greater collective preference for ‘meta’ cause prioritisation work. (Note that ‘other near term’ wasn’t offered as a category on the leaders forum survey so may be undercounted – I hope this will be fixed this year.)

Using broader buckets, among the leaders survey we get 54% longtermist, 20% meta, and 24% near term and 2% other. Among the survey respondents we find 48% longtermist, 28% meta, and 24% near term.

(The 2020 Leaders Forum results are also not published, but at a glance they line up even better, with more agreement on AI and other near term causes.)

I should be very clear: these questions are asking about different things. The Leaders Forum one is about the ideal portfolio of resources, while the question in the EA survey is about what people believe is the top cause. However, it seems notable that leaders think resources should be allocated roughly lines up the distribution of most-engaged EAs’ top cause areas.[4]

To double check whether this is a problem, the EA Survey also asked people about which cause they currently work on. When normalized to 100% and compared again to the mean Leaders Forum respondents’ ideal distribution of resources, this showed:

Again, the picture is fairly similar, suggesting that the allocation of labour is not far from what the leaders guess is the ideal portfolio.

The main difference is that AI seems underinvested in (perhaps suggesting practical career support should focus here), and ‘other near term’ may be overinvested in. Note that the Leaders Forum question was about how both money and time should be allocated e.g. global health receives more funding via GiveWell than any other issue, so if we were looking at funding as well, global health would be over-allocated towards.

One potential problem with this analysis is that you might think it’s circular: it asks about things like working at an EA org or attending EAG, and maybe it’s hard to do these if you’re not already into longtermism. If that’s true, then it would be no wonder that engaged EAs are, like Leaders Forum attendees, pretty into longtermism. What other samples might be better?

In Joey Savoie’s piece arguing in favour of greater representativeness, he suggests surveying EA Forum respondents as a good proxy for what’s representative. We now have this data too:

Again, the picture is similar, suggesting that the views of those taking the Leaders Forum survey views line up with Forum users fairly well, who also line up with ‘5/​5 engaged’ community members. There are about 1,500 active users of the Forum in total.

I also quickly looked at cause preferences among those who reported that they work at EA orgs, and again the picture was similar.

Likewise, if you look at ‘length of time involved’ among all EA survey respondents as a proxy for engagement, you see a clear tilt towards longtermist issues compared to the whole sample.

If we broaden to consider those who reported 4 or 5 out of 5 on engagement, then the differences get a bit larger. “4” on engagement was defined as: “I’ve engaged extensively with effective altruism content (e.g. attending an EA Global conference, applying for career coaching, or organizing an EA meetup). I often consider the principles of effective altruism when I make decisions about my career or charitable donations.”

For instance, the longtermist percentage is now 43% compared to 54% among the leaders, and 33% is near-termist compared to 24%. Still, these strike me as fairly modest differences.

People who clicked “4” or “5” represent about the top half of EAs who responded to the question in terms of engagement (approx 900 people). My guess is that there are over 2000 people at about this level of engagement in the community in total.

If we look at those who reported 3 out of 5 on engagement, then the most popular two causes are Global Poverty (29.8%) and Climate Change (19.5%), which is now a significant difference with the leaders forum survey and the most engaged members.

In brief, newer and less engaged members of the community are more likely to stick to more common sense priorities, which is not surprising.

What might this mean?

Nowadays at least,[5] the Leaders Forum respondents represent the priorities of the most engaged 1,000-2,000 people pretty well.[6]

At 80k, we recently said we’d put around 50% of effort into our priority paths, 30% into other ways of helping solve our priority problems & other issues we list here, and 20% in other problems, such as global health and factory farming. This also lines up with the priorities for the core of the community.

My view is that it would be reasonable to design some other infrastructure, such as EA Global, around the priorities of the core ~2000 people also.

Instead of there being a difference between the Leaders Forum respondents and the core, the divergence is mainly between the core and those who are new or moderately engaged (‘the middle’).

This suggests that the main issue we face is how to best form a bridge between new members and the core. As long as the core focuses on unusual priorities – which using neglectedness as a heuristic for prioritization will mean is likely – there’s a risk that new members get surprised when they find out about these unusual priorities, putting off people who (i) could have a significant impact in other problem areas, or (ii) would later become convinced of the core’s priorities.

I think what to do about this is a really difficult question, which I’m not going to go into here besides to say the following: It’s tempting to try to solve this by ‘meeting people where they are’, and talking a lot more about climate change and global health. But I don’t expect this indirect approach is better at finding people who might end up in the core, it means we spend less time on higher-impact areas (where we still have a lot of gaps), and it can lead to the perception of a ‘bait and switch’ when people discover the ‘true’ priorities. These are some of the reasons we moved away from that approach at 80k. Today our approach is to state our priorities directly, and to focus on explaining them as clearly and reasonably as we can, while trying to remain welcoming to those interested in other areas.

Thank you to David Moss for helping to prepare the data and charts, and to Aaron for posting.

  1. ↩︎

    1: No engagement: I’ve heard of effective altruism, but do not engage with effective altruism content or ideas at all 2: Mild engagement:_ I’ve engaged with a few articles, videos, podcasts, discussions, events on effective altruism (e.g. reading Doing Good Better or spending ~5 hours on the website of 80,000 Hours)_ 3: Moderate engagement:_ I’ve engaged with multiple articles, videos, podcasts, discussions, or events on effective altruism (e.g. subscribing to the 80,000 Hours podcast or attending regular events at a local group). I sometimes consider the principles of effective altruism when I make decisions about my career or charitable donations._ 4: Considerable engagement: I’ve engaged extensively with effective altruism content (e.g. attending an EA Global conference, applying for career coaching, or organizing an EA meetup). I often consider the principles of effective altruism when I make decisions about my career or charitable donations.

  2. ↩︎

    This is based on looking at how many people took the survey from specific organisations, and comparing to their total number of employees. It’s also based on an informal poll of CEA staff.

  3. ↩︎

    Medians seem to match slightly better, if anything.

  4. ↩︎

    One difference is that the leaders might think a cause is less effective, but should still receive an allocation in the portfolio. Whereas the EA survey respondents were asked about which cause they think is most effective. Relatedly, the leaders might think the portfolio should be representative rather than reflect their own views.

  5. ↩︎

    It’s true that this shift might have happened due to the priorities of the leaders in the past, and perhaps at that point there was a disconnect between the core and the Leaders Forum respondents. However, for decisions going forward, we should focus on the community as it exists today.

  6. ↩︎

    Though there could still be differences here that aren’t picked up in this method. For instance, maybe the leaders think we should ideally allocate even more to ‘meta’ but that we don’t have enough people able to do it, so they set the portfolio lower. In addition, the EA survey asked about peoples’ top cause priority, while the Leaders Forum Survey asked about the ideal portfolio – someone might think a cause should be in the portfolio even if it’s a lower priority. I think there are also difficult communication and cultural challenges in how we talk about the issues that receive a smaller allocation in the portfolio.

Benjamin_Todd11 May 2020 13:07 UTC
106 points
26 commentsEA link