More empirical data on ‘value drift’

Benjamin_Todd29 Aug 2020 11:44 UTC
114 points
23 comments12 min readEA link

Here I carry out a quick review of estimates of value drift in the EA community—including several unpublished estimates and a new estimate for attendees of the 2014 CEA weekend away—and based on these, try to come to an overall view.

Overall this review of the evidence updated me towards lower drop out rates among those who are highly engaged, in part because the new samples I found suggested lower drop out rates than previous estimates.

In this post, I don’t want to address the issue of what exactly ‘value drift’ means. Instead, I will try to estimate the more neutral ‘drop out rate’, which is the rate at which people both (i) stop engaging with the effective altruism community and (ii) stop working on paths commonly considered high-impact within the community. I won’t try to assess whether people dropped out for good or bad reasons. Where possible, I look at both the rate at which people fully dropped out, and the rate at which people decreased involvement.

I will also make all the estimates conditional on ‘EA continuing in something like its current form’. If the effective altruism community falls apart, then lots of people will drop out. However, I prefer to let people make a separate estimate of EA collapse risk and then overlay that on drop out estimates.

I focus most on estimating the rate among the most highly engaged ~2,000 community members, though will also cover some data relevant to the broader group.

What’s coming up:

  • What priors should we have when estimating drop out?

  • A review of six drop out rate estimates

  • Summary tables of all data

  • A list of other considerations to take into account

  • My overall estimates

  • Suggestions for further work

Priors

I think we should expect that drop out rates will be very different among different groups.

My guess is that the most significant factor is someone’s degree of social integration—i.e. I expect that people with friends or colleagues who are into EA are less likely to drop out of the community.

Relatedly, I think the degree to which someone identifies with EA will be important. For instance, someone who has been featured in the media as being into EA seems much less likely to drop out. We could think of both of these as aspects of ‘engagement’.

Some other relevant factors seem likely to be: someone’s level of dedication (i.e. how much they prioritise impact compared to other factors), the length of time they’ve been involved, and whether they support ‘weirder’ or more mainstream issues. Of course, many of these factors overlap.

Another important consideration is that I expect drop out rates will decline with time—i.e. someone who has been involved for a year is much more likely to drop out next year than someone who has been involved for ten years (even holding their level of engagement constant). This means we need to be careful not to simply extrapolate forward a short-term drop out rate.

Empirical data

1. Joey Savoie’s informal survey

This is perhaps the most well known and earliest estimate (original post), so I start with it, though I think the later estimates are more useful (since engagement is more clearly defined). The later estimates also show lower drop out rates among more engaged people.

Joey considered a group of people he knew in the community, and categorised them in terms of dedication, as either (i) ‘50% donors’, defined as someone doing something equivalent to donating 50% of their income or (ii) 10% donors.

The results over 5 years are as follows:

16 people were ~50% donors → 9⁄16 stayed around 50%

22 people were ~10% donors → 8⁄22 stayed around 10%

This is a 44% and 64% drop out rate over 5 years.

Joey was also asked if any of the ‘50% donors’ dropped to being ‘10% donors’, and answered:

I did not break down the data that way when I made it, but a quick look would suggest ~75% moved from 50% to 10% and drifting was mildly concentrated at the beginning.

This would imply that among the ‘50% donors’, only ~2 out of 16 (12.5%) fully dropped out, and 31% decreased their level of engagement.

One issue with this data is that it categorises via dedication rather than social integration. My guess is that if these people had been working at EA organisations in a key EA hub, the rate would be lower—an idea supported by the later samples.

2. CEA early employees and volunteers

In the same thread as Joey’s estimate, an anonymous poster posted their estimates for the drop out rates among CEA’s team in 2011.

Overall I don’t think this is an especially useful estimate due to the small sample and the issues I note below, but I also post it here for completeness. One advantage of this sample is that the measure of engagement is fairly high, objective, and was recorded at the time.

The commenter estimated that perhaps 4 out of 17 (24%) had dropped out after 6.5 years (evaluated two years ago when the post was released).

My personal estimate as of 2020 is that 3 dropped out (18%) and another 2 decreased involvement (12%) after 9 years.

Note that this estimate is about half of Joey’s—the time period is almost twice as long, and the drop out rates are slightly lower.

I think probably one person dropped out in the last 2-3 years, which would mean the marginal rate is about 3% per year.

My impression is also that drop out is concentrated into earlier years.

The advantage of this sample is that most of these people were highly integrated. On the other hand, the sample was not further filtered by level of dedication – and ranges from people who had volunteered for under a year to people who helped launch the community in the first place.

I also have the intuition that drop out from the very early days (i.e. 2013 and earlier) will be higher, since at that point it was undecided what EA even was, and everyone was new and somewhat flung together. EA has also become more professional and respectable since then, with higher salaries. I’d expect that if you looked at the team in 2013-2015, the rate would be lower. This motivated me to find the data for the CEA weekends away, discussed below.

2a. Early employees at other organisations

We also did a private analysis of the early staff at another key EA organisation to check whether the CEA estimate was unusual, and found that the drop out rates were similar to CEA from 2011 (~14% over ~6 years). (This is actually lower, but given the tiny sample sizes I think the best we can conclude is ‘roughly similar’.)

3. CEA 2014 weekend away

CEA ran two weekend retreats with about 40-80 people in the summer of 2013 and 2014. I have an attendees list for the 2014 weekend away, which I used to make a new estimate.

This sample is useful because it covers a people pursuing a wide range of career paths, but also those who have some degree of social integration. Unlike the CEA founding team list, it’s also from 2014, at which point the movement seemed more solidified, and the sample is much larger.

One downside is that attendees spanned a spectrum of engagement, from the founders of the movement to CEA volunteers to people who had only recently heard about EA and were interested in getting more involved. We looked at the drop out rate across the whole sample and then again among a more engaged subset.

Rate across the entire sample

The list contains 69 names. Several of the team went over the names checking who is still involved. We started with our own knowledge, and then checked our own engagement data for ambiguous cases.

We counted someone as ‘still involved’ if they were doing something as engaged as fulfilling a GWWC pledge.

On this basis, we counted about 10 people we think have ‘dropped out’, which is 14% of the total in 6 years.

The number of people who decreased involvement (e.g from planning a career in EA towards only fulfilling the pledge) is somewhat higher, though I haven’t done a formal estimate of this.

More engaged subset

Eyeballing the results, drop out seems very concentrated among the people who were newer and less engaged at the time.

To correct for this, we can use some additional data from 2014. In the attendees list, people were classified as either ‘speakers’ or ‘staff’.

Of the 14 classified as staff, I don’t count any clear cases of drop out. 12 are working at EA organisations, and I think the remaining 2 would still be interested in high-impact roles in the future whether at EA organisations or otherwise.

So, we could summarise this as 0% drop out over 6 years, and 14% becoming less involved, though not clearly in a permanent way.

Of the 24 categorised as ‘speakers’ (which mostly don’t overlap), I could only count 1 case of drop out (4%).

It’s also interesting to note that 20 out of the 24 (83%) are currently working in EA organisations.

I think this supports my prior that people who were highly engaged and socially integrated after 2013 were not likely to drop out.

Less engaged subset

If we subtract the 24 speakers from the sample, then drop out among the remainder was 20% over 6 years, with more decreasing their level of involvement.

4. EA Survey data

The EA Survey in 2018 offered an interesting way to estimate drop out rates across a much wider group, which was carried out by Peter Hurford.

In 2014, about 1,500 people took the EA Survey, suggesting that at least that many people were involved at that date.

In the 2018 EA Survey, 885 people said they joined the EA community in 2014 or earlier.

This suggests that roughly 60% of the people who filled out the survey in 2014 are still filling it out in 2018—four years later—suggesting a drop out rate of 40% over 4 years.

We know from the 2019 survey data that this sample spans the whole spectrum of engagement. In 2019, only about half of the respondents reported a 5⁄5 or a 4⁄5 level of engagement with EA (someone working at an EA organisation would be at ‘5’). So, we should also expect it to be an overestimate of the drop out rate among the more engaged.

In 2020 we will be able to apply the same method among a subset of more engaged respondents.

5. 80k’s ‘top plan change’ data

We looked at people whose plan changes we have at some point classified as ‘top plan changes’ or ‘rated-100 plan changes’ (in our previous system), and considered whether they seemed to us to have dropped out of EA.

There are 16 top plan changes, and another ~11 who were counted as rated-100 plan changes in our old system, covering a wide range of career paths.

We think the level of engagement of this group is very high: we’ve previously estimated that there are only around 300 longtermist EAs who are at the level of capability/​dedication/​engagement typically needed to make these plan changes – so this is probably a stricter measure of engagement than self-rating as 5⁄5 on the EA survey, working at an EA organisation, or being a 50% donor equivalent.

Among this group, we didn’t find any cases of drop out from the community.

The average top plan change has been counted for about 2 years, so we could roughly say we measured a 0% drop out rate over 2 years.

One counter might be that people are unlikely to drop out in the couple of years after changing jobs.

6. Giving What We Can’s data

In their 2014 impact evaluation, GWWC very roughly estimated a 5% per year drop out rate. This would be 23% over 5 years.

I would expect this rate to be on the high end, since many GWWC members are less integrated into the community than those in some of the other samples, though it’s also less demanding—which could make it easier to stick with.

There was also a more recent estimate using the 2018 EA Survey data, which aimed to estimate which fraction of those in the survey are fulfilling their pledge. My view of how to interpret this data is in line with AGB, who said:

My best guess thinking over all this would be that 73% of the GWWC members in this EA survey sample are compliant with the pledge, with extremely wide error bars (90% confidence interval 45%-88%).

Note that many of those who don’t fulfil the pledge, still donate a significant amount, so this is more an estimate for decreased involvement than drop out.

On the other hand, the GWWC members who fill out the EA Survey are likely to be more engaged than average, so this could suggest a higher drop out rate across the whole sample.

It’s unclear from the data how this rate is changing with time.

Summary tables

Note that the samples involve people with very different levels of engagement, and it can be misleading to compare them side-by-side. Please see the relevant sections for more explanation of each figure.

Drop out rates

Decreased engagement rates

Other considerations

In favour of greater drop out rates:

  • EA hasn’t been through a major public scandal or disaster and has had decent growth. We might expect drop out to be concentrated among these events, and so the past record underestimates it.

  • The community is still young. If you think most drop out happens in people’s 30s and 40s—especially as people transition to wanting to have a family—then this won’t be in the sample yet. This doesn’t seem to be happening among those I know who are having children, so my best guess is there won’t be a big uptick in drop out here (I think the transition from university to work is also a very big one), but I’m pretty uncertain.

In favour of lower drop out rates:

  • My impression is that the community is better managed than in the early days. It’s also more respected (e.g. AI safety is a lot less weird than in the past) and more professionalised, which should reduce drop out rates.

  • Reactivation – some people who appear to have dropped out might get involved again in the future, and I’ve seen examples in the past.

Outside views:

  • What is the drop out rate in similar movements? I don’t have any data here, and that seems like an interesting direction for future research.

  • One point is that there seems to be a strong trend of people being more involved in social activism when young and decreasing their involvement as they age (though I would guess still supporting the issues).

My overall estimates

Note that:

  • I focus on the rate of decreasing involvement.

  • My estimates are conditional on EA continuing. I think it’s clearer to separately estimate the chance of EA collapsing.

  • These are not especially robust.

Over 5 years

The rate seems very sensitive to how engaged and socially integrated the people in the sample are.

If you roughly add up all the estimates for more engaged people, you end up with about 15% over 5 years. However, I think there remains significant differences in engagement between this group.

I put the most weight on the CEA 2014 weekend away sample for speakers, which found only ~7% becoming less involved or dropping out after 6 years.

Overall:

  • My estimate for someone who’s highly engaged, enthusiastic, and socially integrated would be about ~10% over 5 years. I estimate there are ~500 people in the community who are at this level of risk for value drift. This is a little higher than the CEA weekend away sample (despite a shorter time horizon) due to (i) the outside view (ii) putting some weight on other samples (iii) concern that this wider group is less integrated than that sample.

  • For someone who’s in the top ~2000 most engaged people (which would roughly correspond to the typical level of someone working at an EA org, though would mostly not be people actually in that path), it would be a little higher, perhaps 20%.

  • Among those who are fans (e.g. 4⁄5 or 5⁄5 ‘engaged’ in the EA Survey) but not necessarily very socially integrated, I estimate 30%.

  • Among those who are GWWC members but not very involved otherwise, I estimate 40%.

Year 5-10 drop out rates

My prior is that drop out decreases over time, and in the data we have, drop out seems to be slightly more concentrated in the earlier years, so I would expect that the chances of someone who has already been highly engaged for 5 years to be much lower than when they are new.

My guess is that the rate for years 5-10 should be perhaps 50-75% of the rate for years 0-5.

If we go beyond 10 years, we don’t have any existing data. However, my prior is that the drop out rate will be declining (if the community continues in its current form), and that is also what extrapolating our current data would suggest, so my best guess is that the drop out rates over years 10-30 will be even lower, and that perhaps half of drop out will be concentrated in the first 10 years.

One big uncertainty is whether something along the lines of having children/​buying a house/​saving for retirement/​middle age ends up being a big cause of drop out. I haven’t seen major signs of this so far, though most community members are under the age of 35, so there is little data, and the outside view seems to suggest this is a big transition.

Expected lifetime engagement

Previously, I put more weight on higher initial drop out rates for the most engaged, and would often assume that drop out rates wouldn’t decline as much. This made me think that the typical community member would stay involved for less than half of their career, which would mean that most of their expected impact would come from the earlier and middle parts of their career.

This analysis updated me towards the idea that for the most engaged current community members, much of their impact will come from work they do 15+ years in the future (depending on your assumptions about discount rates and how much productivity increases over a career).

For instance, if the drop out rate for the most engaged core is:

  • Year 0-5: 10%

  • Year 5-10: 7%

  • Year 10-30: 15%

Then, the chance of staying involved the rest of their career is about 70%, which would mean for someone with a 40 career ahead of them, the expected length of engagement is roughly 30 years (see Larks’ calculation in the comments).

Further work

There is a lot of further work that could be done, including:

  • Collect more overall estimates of drop out rates based on the data we already have. An average of estimates should be more accurate than just mine alone.

  • A second category of work would be trying to do more surveys.

    • The ideal survey would pre-identify a sample of people, collect enough information on them that we can control for and subdivide by the predictors of drop out, and then do heavy follow up in 5 years to make sure nobody is missed.

    • We will be able to do something a bit like this with 80k’s top plan change data.

    • It’s possible the EA Survey team could also do something like this if they collect more contact information from a subset of respondents.

  • There are probably more older samples that people could dig out and follow up on themselves, such as the analysis of CEA’s early team.

  • It could be useful to look at more outside view estimates, such as the rates of drop out among other social movements, or how much people’s attitudes and priorities tend to change over their life on average.

  • My estimates could be carried out with real statistical analysis. For instance, we could use the samples to estimate confidence intervals on the drop out rates, or carry out a mini meta-analysis with all of the samples. I also wonder if statistical models (like Laplace’s rule of succession) could be used to improve our estimates.

Benjamin_Todd29 Aug 2020 11:44 UTC
114 points
23 comments12 min readEA link