2017 LEAN Impact Assessment: Quantitative Findings

Quantitative Findings
Qualitative Findings
Evaluation & Strategic Conclusions
Methodology

The Local Effective Altruism Network (LEAN) is a Rethink Charity project initiated in 2015, which focuses on providing material and informational assistance to university and local EA groups around the world.

This document is the first in the LEAN Impact Assessment Series, summarising relevant data from the 2017 Local Group Survey, which is used to assess the EA local group network and the effectiveness of LEAN’s services.

The assessment utilises a mixed method social research strategy, including both quantitative and qualitative components. Our quantitative research relies upon relevant data from the 2017 Local Group Survey [1], which was conducted in collaboration with the Effective Altruism Foundation (EAF) and the Centre for Effective Altruism (CEA) mid-2017. Our qualitative research is made up of over 30 semi-structured interviews that LEAN held by video call with group organisers around the world, ranging from 20 to 40 minutes in duration.

Findings are divided into two overarching categories: “EA Groups” and “Support and Resources”. The first includes data related to the scale, nature and impact of groups, (e.g. membership numbers or funds raised). The second includes data on particular group support strategies, and their popularity and impact.

In this report, and in the qualitative finding summary to follow, we offer descriptive commentary only, leaving full strategic analysis and interpretation for a later article.

Table of Contents

Survey Sample

EA Groups

Group Demographics

Commitment and Lifestyle Changes

Funds Moved

Support and Resources

General EA organisation feedback

Specific services and resources

Survey Sample

The 2017 Local Group Survey was sent by LEAN staff to every EA group organiser on record [2] via email and Facebook message. The survey was also posted in EA Facebook groups, and advertised in the EA Newsletter and the Local Groups Newsletter.

Although 374 entries were submitted to the survey (138 of which identified as group leaders and 236 as group members), 292 entries remained once the data was cleaned [3]. Among these, 98 identified as organisers, and 194 as members.

The survey was split into sections containing questions reserved for organisers, and questions open to organisers and members. Where groups had more than one organiser, they were asked to nominate one organiser to complete the questions designated for organisers. Surplus organisers completed the survey as members. Organisers answered on behalf of their groups for the organiser questions, and on behalf of themselves for the member questions.

Some questions attracted many more responses than others, and several participants chose to skip certain questions. Throughout this report, response levels are indicated as fractions of the total respondent category, in order to signal this difference. For example, if there are 75 responses to a question restricted to organisers, this is displayed as ⁷⁵⁄₉₈ where 98 is the total number of organisers that completed the survey.

Our methodology is explained in more detail in a forthcoming post in this series.

EA Groups

Group Demographics

Number of Groups and Group Size

Organisers reported an average of 50 group members, and a median of 10. The data are partially determined by the different criteria respondents used for for defining membership. For instance, some used the size of their Facebook or Meetup groups, while others counted only individuals who had attended group activities on a regular basis. Based on these numbers though, slightly more than 78% of members are within the top 10% of groups by size.

Group Type

56 participating organisers were from local groups, and 37 organisers were from University groups. Of course, many members of local city based groups may still be university students. Some groups may be better understood as hybrids between the two categories. It is also possible that as the EA community is aging, EAs who joined the movement during university are progressing into local non-university groups.

Group Leader Succession

As a rough indicator of the stability of groups, we asked organisers to estimate the likeliness that their groups would continue were the current organisers to step down.

The results suggest a degree of vulnerability, and dependence on particular organisers.

EA Activities of Organisers and Members

We asked members and organisers whether or not they had ever engaged in the following activities:

We supplied no participation count for this question because respondents were only given the option to add a mark if they engaged in the relevant activity. Therefore, a lack of marks from a respondent could mean either a lack of engaging in the question, or it could mean that the respondent doesn’t engage in any of the activities. Any participation count would therefore not be informative.

An additional limitation of these results is the fact that categories such as “volunteered at an EA organisation” were not sufficiently defined. For example, some respondents interpreted time spent organising their groups as voluntary work for an EA organisation, while others did not.

In an open ended addendum to the question, respondents reported additional ways of investing time in EA:

Thinking about EA
Direct EA work
Producing EA content
Researching EA
EA informed career transition
Applying for EA related grants
Pitching EA to individuals
EA aligned policy work
Earning to give

This suggests that members (and organisers) of EA groups are engaged in Effective Altruist activities and the movement more broadly. Almost by definition, group membership involves social interaction with other EAs. It is clear, however, that this is just one of many activities that members are involved in.

Commitment & Lifestyle Changes

Increasing Engagement with EA

Perhaps the most important success criteria for EA groups is their ability to attract people to Effective Altruism, and to retain their interest and commitment. We included the following questions in the Local Group Survey in order to gauge this:

[All respondents] “Did you consider yourself an EA before you attended a group meeting?”
[Organisers] “In the last year, roughly how many people have attended at least one of your events who weren’t familiar (or were barely familiar) with Effective Altruism?”
[All respondents] “How much of a factor has being involved with your group been for engagement with EA?”
[Organisers] “How many members have become actively committed to EA as a result of being in your group? (Examples of commitment include lifestyle changes, direct action, or donation to effective causes)”

The following table and graph summarise data on how many respondents considered themselves Effective Altruists prior to attending their first group meeting:

It may appear striking that some organisers did not consider themselves EAs until after their first group meeting. Note that it may be that organisers were converted to EA after attending their first group meeting as a member, but became an organiser after becoming an EA. It may also be the case that some respondents are reluctant to apply the label ‘EA’ to themselves e.g. until they’ve started doing something they see as concrete for EA (such as organising an EA group).

While it is difficult to discern causation from these figures, they do at least confirm that a sizeable number individuals become EAs only after beginning to attend a group. This confirms that groups are not merely reaching people who already identity as EAs and nor are they merely reaching non-EAs who never subsequently come to identify with the movement.

This graph displays the number of people who group organisers estimated attended each group’s events with little or no familiarity with Effective Altruism:

Responses are widely distributed, with a small number of groups reporting very large numbers, but most responses clustered around 30 new event attendees with little familiarity with EA, and with most responses (44/78) falling between 5 and 50. It is important to see this in the context of group size, with all but 10 of the groups who responded to this question had <50 group members. The ratio between group members and total event attendees who were unfamiliar with EA varied widely, between 1:0.375 (a group with 40 members and 15 new event attendees) and 1:43 (a group with 14 members and 600 event attendees who were unfamiliar with EA).

How much of a factor was group involvement for engagement with EA?

When asked how much of a factor group involvement was for their engagement with EA, respondents gave the following answers:

Most members (102/178) and organisers (45/72) report involvement with their EA group to be ‘large’ or ‘very large’ factor for their engagement with EA, with the majority of the rest being ‘moderate’ responses.

Number of members becoming ‘actively committed’ to EA due to groups

We also asked for counterfactual estimates of the number of members who became ‘actively committed’ to EA (for example, lifestyle changes, direct action or donating money to effective causes) as a result of involvement in groups.

The median number of estimated counterfactual active commitments is 5 and the majority (43/63) fall between 1 and 10. A small number of groups report substantially higher numbers (e.g. 50-100 counterfactual commitments). The top ¹⁰⁄₆₃ groups account for slightly more than half the reported commitments (352/617.5).

Looking at this in relation to ‘group size’ finds a positive correlation between the number of members a group has and its number of reported active commitments.

Thought and Behavior Change Since Being Involved with a Local Group

We asked all respondents whether their world views or behaviours had changed since becoming involved in an EA group.

We also asked them to indicate whether or not any changes reported were likely to be impactful.

A substantial majority of members and organisers alike report that the way they think about the world and behave has changed since being involved with a local group and that they expect to have more social impact as a result of this change. This does not necessarily suggest causation between the local group and their increased engagement and efficacy.

How many of your current members do you expect to choose careers based on EA recommendations or thinking?

Most groups report between 1 and 5 members choosing careers based on EA principles

(median 4). The mean (7.29) is dragged upwards by a small number of groups with much higher (up to 50) numbers of career choices based on EA.

A natural question to ask is how this relates to group size. Are the largest groups simply accounting for many more of these outcomes (due to their much greater size)? The first graph shown here would seem to suggest this, with the group with the largest number of EA career choices by some way, also being the largest, and all but one of the groups with the highest number of EA career choices having >100 members.

It is a further question, however, whether the largest groups are better at making conversions (e.g. getting members to make EA career choices). The graph below would not support this conclusion. We see here only a weak correlation, but it might appear that the largest groups (responsible for the most EA career choices) in absolute terms, have a relatively lower % of members making EA career choices. This does not seem sufficient to suggest that larger EA groups are worse at making conversions however. A plausible explanation might be that smaller groups contain a disproportionate number of dedicated EAs (for example, a small group with 5 members might contain two EAs sufficiently dedicated to found and run a group), compared to the largest groups which may have many hundreds of new members.

If applicable, how much of a factor are or were EA principles in planning your career?

A majority of members and organisers alike report that EA principles were a large or very large factor in planning their careers. Notably, though perhaps unsurprisingly, organisers disproportionately indicated that they were a “very large” factor in planning their careers, whereas among members there were relatively more moderate and large responses.

Examples of Notable Group Members Becoming Active EAs

Organisers responded to the question: “Please name any current or past group members who have gone on to become active in the wider EA community. This would include going on to work at an EA organization, starting an EA project, becoming a thought leader in the movement, earning to give, and/or representing EA in other public ways.” We refer to these individuals as ‘hits’ who initiated or increased their level of involvement in EA after group involvement.

In total 121 ‘hits’ were reported in this open comment question. Note that low response rates may obscure the number of group organisers who would report 0 ‘hits.’

How valuable do you find your group’s activities?

As this graph shows, majorities of both organisers and members rate their group’s activities as valuable or very valuable. Notably, members appear strikingly more positive than organisers, “very valuable” being their most frequent response by some way (80), followed by valuable, with 135 out of 144 selecting these two options, whereas organisers’ responses are centred around valuable (30) and moderately valuable (22).

Funds moved

Group organisers provided estimates of the money raised through group fundraising activities, the money raised through the private donations of members, and the counterfactual GWWC pledges raised by groups.

While substantial funds have been collectively raised by groups, the majority of the funds come from a small number of groups.

Nevertheless an appreciable number of groups have fundraised significant (i.e. $100 to $1000) amounts, as seen below (note the log scale on the y axis).

Note that we would expect that non-response would be higher for groups who have not run fundraisers or who raised very little, so there may be a longer ‘tail’ of groups raising $0.

This table summarises group organisers’ estimates of the private donations made by group members who counterfactually would not have donated (but for group involvement):

Total estimated counterfactual private donations are dominated by a small number of groups reporting very high figures (note the log scale on the x axis). However, a substantial number of groups are reporting significant sums being donated. An important caveat is that respondents may have over-estimated some of the figures for various reasons (for example, including future donations).

Note, as above, that non-response rates may conceal a higher number of groups who would estimate very low donations.

Finally, organisers provided estimates of the counterfactual Giving What We Can pledges secured through their groups.

Most organisers report few counterfactual pledges (pledges which would not have been taken without the influence of the group), with most reporting between 1 and 5. Indeed, the vast majority of responses fall within 1 and 11, while 2 groups report 40 and 75 counterfactual pledges respectively, and 16 report 0 counterfactual pledges.

Overall, these data on funds are speculative, and should be treated as such. However, it appears that groups have a non-trivial role in the movement of funds to effective causes.

LEAN Support and Resources

In this part of the report we summarise evidence regarding the usefulness of services which LEAN provides in assisting the operation of groups [4]. More data on groups’ experiences of outside support will be shared in the qualitative report.

General feedback on LEAN and other EA organisations

We asked organisers: “What outside help has been the most useful to the operation of your group?” (Respondents could select multiple options.)

“Other” was made up of specific University student unions, larger EA groups in similar regions (EA NTNU, EA London and EA ANU), value aligned local organisations, and individual EAs.

CEA (39) was the organisation most often selected as ‘most useful’ to the operation of groups by organisers. Note that respondents could select multiple organisations if they wished. LEAN (22) and EAF (22) were joint second most commonly selected organisations. TLYCS was selected 11 times, and remaining options were each selected 2 or fewer times. CEA-affiliated 80,000 Hours (21) and Giving What We Can (13) were selected separately by some respondents. A favourable bias towards LEAN is possible given the fact that LEAN distributed the survey.

Feedback on specific services and resources

Personal Feedback

LEAN offers organisers personal support, on demand, via video call, social media and email. We asked organisers: “In your opinion, how useful is personal feedback and support via social media, email and video call?”

A clear majority report that personal feedback and support of this kind is useful or very useful.

Practical support and new ideas

We asked organisers: “In your opinion, how useful is it to receive practical support and new ideas for group activities?”

Practical support and new ideas for group activities are generally rated as useful or very useful (75/80), with only (3/8) finding them either not useful or not at all useful.

Video Calls

LEAN hosts video calls to help share best practices between groups. Organisers responded to the question: “In your opinion, how useful is it to host video calls about group management topics?”

A majority of organisers reported video calls about group management to be either useful or very useful.

Written Guides

LEAN is among many EA individuals and organisations to have produced written content for EA organisers. We asked organisers: “In your opinion, how useful are written guides (with a focus on practical and strategic aspects of organising groups)?”

A clear majority of respondents (65/78) considered written guides to be useful or very useful with only 1 respondent out of 78 offering a negative rating.

Websites and Technical Support

LEAN provides hosting, domains and basic content management for over fifty EA group websites. Organisers were asked: “If your group uses a website, do you believe that it makes a non-trivial difference in the effectiveness of your group’s outreach efforts?”

While a majority of the groups who used (group) websites find them significantly useful, a notable minority find them no more than trivially useful.

In addition, we asked: “In your opinion, how useful is technical support (for instance, subscriptions for online services, free websites, group email addresses)?”

A majority (52/73) of respondents report that technical support of this nature is either useful or very useful, compared to 13 and 8 groups being neutral or not finding it useful, respectively.

Premium Meetup.com subscription

LEAN provides free Meetup.com accounts for interested organizers. Organisers were asked: “If your group uses Meetup.com, please give an estimation of the % more attendees you have attracted as a result of using the platform in addition to—or instead of—alternatives?”

*It should be noted that most EA Groups don’t use Meetup.com and would not have been able to answer this question.

While many groups gained modest increases in members from using meetup.com (median 15%, mean 21.42%), a small number gained very significant increases.

Local Group Newsletter

LEAN leads a regular newsletter for EA groups with support from EAF and CEA. We asked organisers: “In your opinion, how useful is the Local Group Newsletter?”

Many more respondents rated the local groups newsletter (N.B. not the EA Newsletter) as useful or very useful, (32) than not useful or not at all useful (34), though many were neutral (24).

These results should be contextualised, however, by responses at the end of the survey which asked whether respondents wished to be added to the local organiser newsletter:

This shows that 49 organisers or more have not received the newsletter, which limits the usefulness of the earlier responses.

Group Organisers’ Mentoring Programme

With support from CEA, LEAN launched a mentoring trial programme, connecting experienced organisers with new ones from August 2017. We asked organisers: “In your opinion, how useful is the EA Organiser’s Mentoring Programme?”

The majority (32) of respondents found this program neither useful nor useless, with 16 finding it useful or very useful and 4 finding it not useful. These results may indicate that the majority of organisers are simply unfamiliar with the program due to it’s recent release.

EA Organisers’ Facebook Community

LEAN supports a Facebook group for EA Organisers in collaboration with EAF and CEA.

We asked organisers: “In your opinion, how useful is the Facebook community of group organisers?”

A decisive majority of organisers found the Facebook group to be useful or very useful.

EA Groups Slack Team

LEAN supports a Slack channel for EA Groups in collaboration with EAF and CEA. Organisers were asked: “In your opinion, how useful is the EA Groups Slack Team?”

Slightly more organisers found the Slack team to be useless (13) rather than useful (10), with the majority (34) being neutral.

Conclusion

Evaluating the impact of LEAN and the strategic implications of these results will be deferred until the LEAN Assessment Strategy report, which will follow in this series of articles. We will also draw on the qualitative data we have gathered in a separate report to help interpret these findings.

Endnotes

[1] Due to the number of personal identifiers in the data set, it is not possible at this point in time to make the raw survey results publicly available. At a later date it may be possible to release partial anonymised findings.

[2] LEAN collaborates with CEA and EAF to maintain up to date, comprehensive records of EA groups and their organisers.

[3] Entries were deleted if they were blank, or sufficiently incomplete as to render the submitted data useless. Other deletions included garbled or illegible responses and duplicates.

[4] Of the support categories included in this section, some have historically been provided only by LEAN, whereas others have been provided by various individuals and organisations in EA.

[5] The LEAN Impact Assessment is distinct from the 2017 Local Group Survey. While the survey results supply a substantive base for the assessment, the survey was a collaborative project between the Centre for Effective Altruism, the Effective Altruism Foundation, and The Local Effective Altruism Network (LEAN). Findings from the survey that were not relevant to this assessment may be shared at a later date.

Acknowledgements

This report was written by Richenda Herzig. David Moss, Peter Hurford and Richenda Herzig analysed the 2017 Local Group Survey data. Editorial input was provided by Peter Hurford and Tee Barnett. Thanks to Ellen McGeoch for assisting in survey design and formatting for the 2017 Local Group Survey. Thanks to David Vatousious for distributing the survey across the network and for recruiting participants. Thanks also to Kaitlin Alcantara for data entry and filtering.

We are highly grateful to Greg Lewis for his input as an external advisor.

We would also like to express our thanks to Harri Besceli from the Centre for Effective Altruism (CEA) and Jonas Vollmer from the Effective Altruism Foundation (EAF), who collaborated in writing the 2017 Local Group Survey. We are grateful to CEA for generously supplying free EA t-shirts to respondents.

Last but not least, a big thank you to all organisers and members who took and shared the survey!