Retention in EA—Part III: Retention Comparisons
Summary
I spent one day trying to compare retention in EA to other movements. I was unable to find anything that I was very happy with, but I’m sharing these comparisons in case they are helpful to others. Hopefully Cunningham’s law will kick in and someone will be spurred to do a better analysis.
I found semi-comparable estimates of retention for Southern Baptist megachurch youth ministry participants, vegetarians, and civil rights activists who participated in Freedom Summer. While none of these data sources seemed precise or perfectly comparable, my best guess is that EA is slightly better at retention than they are, and unlikely to be substantially worse.
How does EA compare to other movements?
Most interviewees remarked that, once someone gets to the point where their career is driven by EA considerations, it’s unlikely that they will leave. This might suggest that CEA should not focus on retention, as it is already going well.
For some insight on this, I tried to compare Ben Toddd’s estimates with estimates from other movements. My best guess is that EA has better retention than Southern Baptist megachurches and Freedom Summer participants, and similar or slightly worse retention to vegetarianism. Most of my uncertainty comes from how reliable these estimates are and how comparable they are to each other.
Because of how wide my error bars are, it’s hard to say too much with confidence. I don’t think EA has a severe retention problem (compared to e.g. Baptist churches). We also don’t seem substantially better at retention than others (e.g. vegetarianism). So my guess is that EA is slightly above average at retaining highly engaged members (compared to other movements), which makes me think it’s worth investing a moderate amount of resources into retention.
| Five-year dropout rate | |
| Ben Todd’s estimate for top 2000 most engaged EA’s | 20% |
| “High involvement” Southern Baptist megachurch youth ministry participants | 54% |
| Vegetarians | 14-26% |
| Civil rights activists who signed up for Freedom Summer | 15-51% |
Ben Todd’s estimates
Taken from More empirical data on ‘value drift’.
| Type of person | Five-year dropout rate | Implied expected lifetime[1] |
| Highly engaged, enthusiastic, and socially integrated | 10% | 45 years |
| Top 2000 most engaged people (I think this is roughly CEA’s definition of “engaged EA”) | 20% | 22 years |
| Answered >= ⅘ engaged on EA survey but not socially integrated | 30% | 14 years |
| GWWC member but not otherwise involved | 40% | Nine years |
Youth Ministry Adherence
An Assessment of Dropout Rates of Former Youth Ministry Participants in Conservative Southern Baptist Megachurches classified youth ministry participants (teenagers who attended a certain type of religious education) into one of four engagement levels based on their answer to a survey (asking e.g. how frequently they attend church) and measured their retention.
| Dropout rate (don’t self-report as moderate or high involvement) | Dropout rate (don’t self-report as high involvement) | |
| Disengaged prospect | 100% | 100% |
| Normative attender | 26% | 58% |
| Enthusiastic follower | 21% | 65% |
| Engaged disciple | 8% | 42% |
I think we are mostly interested in those listed as “engaged disciples” who continue to self-report as “high involvement”, so the 42% number is what I’m going to focus on.
This study did not control for the fact that people who were older at the time of surveying were more likely to have dropped out.[2] Also note that the survey was retrospective and presumably suffers from response bias.
To try to handle the fact they didn’t control for age, I tried to find the parameters for a geometric distribution which fit their overall results of 42% dropout. A constant 14.5% annual dropout rate results in 42% of respondents having dropped out:
| Age | Elapsed years | Percent of respondents | Expected dropout rate | Percent of total sample who dropped out from this age bracket |
18 | 0 | 0.4 | 0.00 | 0.00 |
19 | 1 | 10 | 0.15 | 1.45 |
20 | 2 | 24.4 | 0.27 | 6.56 |
21 | 3 | 20.1 | 0.37 | 7.54 |
22 | 4 | 14 | 0.47 | 6.52 |
23 | 5 | 7.2 | 0.54 | 3.91 |
24 | 6 | 8.6 | 0.61 | 5.24 |
25 | 7 | 4.7 | 0.67 | 3.13 |
26 | 8 | 6.1 | 0.71 | 4.36 |
27 | 9 | 4.7 | 0.76 | 3.55 |
| Total | 45 | 100.2 | 42.26 |
This assumes that “engaged disciples” are evenly distributed across the ages, which is probably not completely true but I would guess mostly true. A 14.5% annual dropout rate is a 54% five year dropout rate.
I’m not very confident in the 54% number, but I feel relatively confident that the number is higher than Ben Todd’s estimate of 20%. The paper cites another survey which found that 70-90% of teenagers stop attending church within two years of high school graduation. (One wrinkle is that there is some evidence people reengage with the church in their late 20s/30s, meaning that it’s a little hard to define what “dropout” actually means.)
Vegetarian adherence
ACE estimates that the average vegetarian stays vegetarian for 3.9-7.2 years,[3] implying a five-year dropout rate of 53%-77%.[4] My guess is that the average vegetarian is equivalent to Ben’s lowest level of engagement (GWWC member but not otherwise involved) or even less engaged, and being vegetarian seems about as big of a commitment as keeping the GWWC pledge. This implies a dropout rate higher than what Ben estimated for EA.
US Civil Rights Activists—Freedom Summer
Freedom Summer was a 1964 project with extremely high asks of its participants: they were mostly college students who gave up their summer to be unpaid volunteers facing severe violence (several were murdered). Participants needed to apply beforehand, and those who applied but withdrew from the project before it started give some estimate of adherence amongst civil rights activists, split by engagement level.
I think that attending Freedom Summer is a sign of involvement in the civil rights movement very roughly comparable to how having attended early EA retreats is a sign of being at one of Ben’s two highest levels of EA involvement. So that’s what I’m going to compare it to, even though the comparison is quite inexact.[5]
The Biographical Consequences of Activism followed up with participants approximately 20 years after Freedom Summer and found the following levels of engagement:
| % Active in civil rights immediately after Freedom Summer[6] | Implied 5 year dropout rate | |
| Volunteers | 90% | 41% |
| No-shows | 74% | 88% |
| % Currently active in any social movement 20 years later | Implied 5 year dropout rate | |
| Volunteers | 48% | 15% |
| No-shows | 33% | 24% |
| % “Very involved” in antinuclear movement 20 years later | Implied 5 year dropout rate | |
| Volunteers | 11% | 42% |
| No-shows | 6% | 51% |
Again, the methodology makes comparisons hard, but it seems like the dropout rate from Freedom Summer is higher than Ben Todd’s estimates for EA.
Religious adherence
Social Predictors of Retention in and Switching from the Religious Faith of Family of Origin: Another Look Using Religious Tradition Self-Identification found that age was positively correlated with remaining in a religion, implying a negative dropout rate. I’m not sure whether this is because people drop out when they are young and then rejoin when they are older or if people who were born longer ago are just more committed to their religion, but either way it seems different enough from EA that the comparison isn’t useful.
Footnotes
In this and the rest of this section I’m going to assume a constant annual dropout rate, i.e. a geometric distribution, unless otherwise specified.
In the section where the author analyzes demographic variables, they say that only educational ones had a significant impact. Taken at face value, this implies that age does not correlate with dropout rate, which seems unlikely to me, but possible.
Taken from their “final estimate” but excluding the effect of cultured meat alternatives, death, and GCRs, since those do not seem relevant to the question I’m trying to answer here.
Assuming a geometric distribution whose expectation is 1/p
There are a lot of ways in which Freedom Summer is different from EA, the most notable being the violence participants experienced. Another major concern is that the Civil Rights Act of 1968 is seen by some as the culmination of the civil rights movement, meaning that many participants perhaps should have been expected to move to other causes after 1968 (e.g. anti-Vietnam War). But I think we would have expected many of them to be active in at least some social movement, if they were truly “retained”.
This is the percentage who were active in civil rights at some point within the five years following Freedom Summer, but I am simplifying this to assume it’s an annual dropout rate. My justification for this is a) there probably aren’t many people who didn’t engage for 4 years and then became active again in the fifth, and b) to the extent that these people do exist, it’s arguably more accurate to classify them as having disengaged and then reengaged instead of having been retained.
Don’t forget the 2018 EA Survey analysis that suggests a ~40% EA drop out rate after 4-5 years.
So my first reaction to the Youth Ministry Adherence data was the basically the opposite of yours, in that I looked at it and thought ‘seems like they are doing a (slightly) better job of retention’. Reviewing where we disagree, I think there’s a tricky thing here about distinguishing between ‘dropout’ rates and ‘decreased engagement’ rates. Ben Todd’s estimates which you quote are explicitly trying to estimate the former, but when you compare to:
...I think you might end up estimating the latter. ‘High involvement’ was the highest of four possible levels, and ‘Engaged disciple’ was also the highest of four possible levels. By default I’d look at the number who are neither moderately nor highly involved, i.e. 8% rather than 42%.
More generally, my understanding is that Ben was counting someone as not having dropped out if they were still doing something as engaged as fulfulling as GWWC pledge, based on quote below. So if you start at a much higher level than that (like...attending the Weekend Away), there’s a lot of room to decrease engagement substantially while still being above the bar and not having ‘dropped out’. Which in turn means I’d generally be aiming to have similar leeway for ‘regression to the mean’ in the comparisons. Or you can compare everything to Ben’s GWWC dropout number of 40%, which has no such leeway, as you do with the similarly-no-leeway case of vegetarianism.
I appreciate this is a highly subjective call though, this is very much just my two cents. I could easily imagine changing my mind if I looked at the Youth Ministry information more closely and decided that ‘Engaged Disciple’ actually constituted some kind of ‘super-high-involvement’ category.
Interesting, thanks! Something which probably isn’t obvious without reading the methods (pages 125-127) is that study participants were recruited through church mailing lists and Facebook groups. So the interpretation of that statistic is “of the people who answer surveys from their church, 92% report at least moderate engagement”.
“Moderate engagement” is defined as an average of a bunch of questions, but roughly it means someone who attends church at least once per month.
I think that definition of “moderate engagement” is a bit higher than “willing to answer surveys from my church” (as evidenced by the people who answered the survey but did not report moderate engagement), but it’s not a ton higher, so I’m hesitant to read too much into the percentage who report moderate engagement.
I felt like “high engagement” was enough above “willing to answer a survey” that some value could be gotten from the statistic, but even there I’m hesitant to conclude too much, and wouldn’t blame someone who discounted the entire result because of the research method (or interpreted the result in a pretty different way from me).
If we want to compare it to Ben’s EA estimates: I guess one analog would be to look at people who attended that weekend away but also answered the EA survey five years later. I’m not sure if such a data set exists.
Thanks for writing this! I found it a really interesting series and kudos to you for sharing earlier-stage thinking publicly. I definitely know that I can find sharing such thinking pretty daunting!
Small point:
>ACE estimates that the average vegetarian stays vegetarian for 3.9-7.2 years, implying a five-year dropout rate of 53%-77%.
I think that 3.9-7.2 is their estimate for (i) the average vegetarian adherence length, but you might be interpreting that here as more like (ii) the median length of vegetarian adherence?
From that ACE report:
> We can see in the Guesstimate model that after multiplying, the length of adherence for current vegetarians is 32–49 years.
And then they calculate the length for former vegetarians using the following as a basis:
I’d say the underlying distribution here is pretty skewed, so the difference between that average vegetarian length and median vegetarian length might be pretty significant.
So I guess my pretty quick sense is that the median vegetarian adherence length may be a fair bit shorter than 3.9 − 7.2 years. And if you are interpreting that 3.9-7.2 as being more about the median length, and it was in fact a fair bit shorter, then that could meaningfully change some of your conclusions here.
I think that is all somewhat nitpicky though, and I could certainly be wrong about it! Regardless, thanks again for sharing all of this. :)
Thanks Kieren! I was interpreting it to be the expectation of a geometric distribution (i.e. mean length assuming a constant annual probability of leaving), which I think is the correct way to interpret that number? Let me know if that’s wrong though!
The assumption that length is geometrically distributed might not be warranted, I’m not sure.
Thanks, Ben! That all seems fair enough for these purposes.
Fwiw, I think that number might be more the arithmetic mean of some observations. Interpreting it as a geometric mean seems like it doesn’t strongly violate much, but I think the geometric mean is going to be a little bit lower.
But yeah, I doubt it makes much of a difference in the scheme of things!
Thanks again for sharing all this :)
Cool series, thanks for sharing on the forum. One nitpick:
I’m not sure how your rate is being calculated from ACE’s figures here, but at first pass it seems wrong? Since 5 years is within but slightly towards the lower end of the range given for how long the average vegetarian stays vegetarian, I’d assume we’d end up with something more like a ~45% five-year dropout rate. By contrast, a 14-26% five-year dropout rate would suggest that >50% of vegetarians are still vegetarian after two such periods, i.e. 10 years.
If I’m misunderstanding either stat, just let me know and will happily retract.
Thanks Alex! You are correct. I accidentally put the annual dropout rates there instead of five-year dropout rates.
The implied five-year rate is 53%-77%, approximately in line with Ben’s estimates for GWWC members. I’ve updated the text accordingly.