That makes sense. That said, while it might not be possible to quantify the extent of selectin bias at play, I do think the combination of a) favoring simpler explanations and b) the pattern I observed in the data makes a pretty compelling case that dissatisfied people being less likely to take the survey is probably much more of an issue than dissatisfied people being more likely to take the survey to voice their dissatisfaction.
AnonymousEAForumAccount
Karma: 3,692
As I mentioned in point 3 of this comment:
Looking at the distribution of satisfaction scores in late 2022 vs late 2023, we see the reduction in satisfaction over that period coming from fewer people giving high ratings of 9 or 10, and more people giving low ratings of 3-5. But for the lowest ratings, we see basically no change in the number of people giving a rating of 2, and fewer people (almost nobody) now giving a rating of 1.
This suggests we could crudely estimate the selection effects of people dropping out of the community and therefore not answering the survey by assuming that there was a similar increase in scores of 1 and 2 as there was for scores of 3-5. My guess is that this would still understate the selection bias (because I’d guess we’re also missing people who would have given ratings in the 3-5 range), but it would at least be a start. I think it would be fair to assume that people who would have given satisfaction ratings of 1 or 2 but didn’t bother to complete the survey are probably also undercounted in the various measures of behavioral change.
Great post! Another excellent example of the invaluable work Rethink Priorities does.
My observations and takeaways from this latest survey are:
Overall satisfaction is low. On a 10 point scale, a 6.77 rating isn’t very good. Only ~12% give a rating of 9 or 10, and <40% give a rating of 8 or higher. (I’m aware that a 6.8 rating on a ten point scale still suggests more satisfaction than dissatisfaction, but given that we’re talking about people rating their satisfaction with a community they’ve chosen to participate in I think we should interpret this result as damning with faint praise).
The change in satisfaction from pre-FTX to now is quite large. The original survey in the immediate aftermath of FTX’s collapse found a substantial drop in satisfaction vs. pre-FTX. One commenter at the time observed that “[decreased satisfaction by 0.5-1 points on a 10-point scale is] a big drop! In practice I’ve only ever seen this type of satisfaction scale give results between about 7⁄10 through 9.5/10… so that decline is a real big chunk of the scale’s de facto range.” This latest survey suggests that the satisfaction has dropped another ~.2-.4 points. That means from pre-FTX to today, satisfaction has probably dropped something like .7-1.4 points, which is huge for a 10 point scale with a much narrower effective range (and the drop could easily be understated due to selection bias). The mean reversion of satisfaction to pre-FTX levels that some expected has not happened.
Selection bias is probably causing us to overstate satisfaction, and understate the degree to which satisfaction has dropped. As the report notes, selection bias could plausibly cut both ways. That said, I think most people would guess that selection bias is operating in the direction of excluding the least satisfied members from the data set. And the data seems to give some indication to support that view. Looking at the distribution of satisfaction scores in late 2022 vs late 2023, we see the reduction in satisfaction over that period coming from fewer people giving high ratings of 9 or 10, and more people giving low ratings of 3-5. But for the lowest ratings, we see basically no change in the number of people giving a rating of 2, and fewer people (almost nobody) now giving a rating of 1. Given falling satisfaction, we should expect more people to be giving these lowest ratings; the fact that we don’t suggests that the least satisfied people are excluded from the sample.
The behavioral changes described in this survey are significant and warrant attention. The number of people making concrete behavioral changes surprised me (even though I already believed FTX had a significant negative impact on community growth and activity). I was particularly surprised at how many people reported permanently ceasing to promote EA or engage in key EA activities, especially since selection bias likely suppresses the reported numbers. The December 2022 survey indicated a community desire for additional transparency, reflections, and post-mortems. This more recent survey suggests the community has been unimpressed by the limited public efforts on these fronts, with concrete negative implications.
This analysis shows large drops in satisfaction and significant behavior changes, but only gives limited insight into what caused these. Issues like JDEI concerns and concerns about cause prioritization are clearly reasons why people might be dissatisfied, but it’s not obvious why these longstanding issues should cause people to be more dissatisfied now than they were in late 2022. We can hypothesize that the changes were driven by FTX (and to a lesser extent subsequent scandals), but it would be preferable to look directly at the data. I think it would be very valuable to closely examine the cohort of people who report having changed their behavior. (It would also be interesting to look at a cohort of people who report significant decreases in satisfaction, but it seems more important to understand drivers of behavior change.) For this cohort, I would like to know:
What are the largest causes of dissatisfaction? How does this cohort compare to the full sample?
Does dissatisfaction appear to be driven by one major factor, or a confluence of concerns (as Luke Freeman has suggested might be the case)?
How much did reported satisfaction decrease since previous surveys?
What levels of satisfaction [or sizes of decreases in satisfaction] are associated with behavior change (i.e. can we say something like “if satisfaction drops to 6 or lower [or decreases by 1 or more points], that’s when we start seeing people shift behaviors”?)
How does perception of issues for this cohort compare to the full sample?
Do perceptions of issues look different for modest behavior changes (e.g. stopping referring to “EA” but still promoting) vs. larger behavior changes (e.g. permanently stopping promoting)?
Do we have any insights into the temporarily stopped promoting cohort (a sizeable 15% of the sample)? Do we have any idea whether these are people who stopped promoting for a few months but are now promoting again vs. people who are still not promoting but expect to do so again at some point? I realize the most recent survey might not give enough information to tell, but this seems like a very important issue to understand and something we might want to dig into in the full EA Survey.
How much overlap is there across the different types of behavior change?
Updating analyses of community growth seems like it should be a high priority. The last public effort was based on data collected in August 2023. I’ve been a longstanding proponent of conducting regular analyses of community growth, but refreshing that analysis (perhaps with some methodological tweaks) seems particularly important given the results of this survey. Various metrics that are publicly available don’t look great, continuing trends I first noted in October (though note that some of the metrics that performed strongest in the immediate aftermath of FTX, most notably 80k’s metrics, aren’t publicly available):
Donations and donors to EA Funds continue to shrink year over year. The decline in donations has been a multi-year trend, but the decline in donors represents a reversal of a multi-year growth trend , and the timing of that reversal aligns perfectly with FTX’s collapse.
GWWC donations and donors were down in 2023 vs. 2022, and pledges in Dec 23/Jan 24 were also down vs Dec 22/Jan 23 even though that latter period immediately followed FTX’s collapse and GWWC was “instructed to pause all our giving season campaigns around the time of the crisis.”
EA Forum metrics, which had experienced years of consistent and rapid growth leading up to FTX’s collapse, have either flatlined at the levels they were at 2 years ago or have been steadily falling since the collapse (even as resources invested in the forum have increased significantly since then).
Per CEA’s dashboard, the last few months have seen the lowest traffic ever to the effectivealtruism.org intro page (in a data series going back to 2017) and the EA newsletter subscription base has shown no sign of regaining the growth trajectory it was on for 2 years prior to FTX’s collapse; since the collapse subscribers have actually shrank slightly.
As Jason astutely observed, the massive increase in public awareness around AI over the last year should have provided a significant tailwind to many metrics. I would argue the same is true for the hugely successful promotion around WWOTF. And demographic factors should generally support an expectation of growth in donation related metrics (almost everyone in the EA community is at an age where they are entering the workforce or increasing their earnings; virtually nobody is at retirement age). The weak performance we’ve observed in publicly available metrics looks even worse in this context.
Me too
Hi David! Any update on what (if anything) is going on with this survey and sharing its results? Was this part of the survey that was conducted in late December?
Thanks for these updates Zach!
FWIW I’m pretty surprised and disappointed that my question about whether EV will continue to exist after the spinoffs never got a response. It’s a key piece of information about one of the most important EA orgs, and it’s clear that others are curious about it from the karma of my question, the comments replying to it, and the same question being raised in another thread. Even if the answer is “we don’t know yet”, that would be valuable to communicate. And if the answer is known, giving clarity to the rest of the community feels like extraordinarily low hanging fruit.
Thanks for this update! Two questions…
When all the sponsored projects have been spun out, will EV continue to exist? If so, what will it do?
“I plan to share other non-privileged information on lessons learned in the aftermath of FTX and encourage others to share their reflections as well.” Do you have an estimated timeline for this?
Thanks Ben, I appreciate this detailed response!
Re: 2, very helpful to know CEA’s top of funnel target. To the best of my knowledge, this hasn’t been shared before. Are there also targets for middle and bottom of funnel growth, and if so, would you mind sharing those?
Re: 3, I agree that both of your points suggest raising the target might make sense. But in the other direction, all else equal we should expect growth rates to slow over time (30% annual growth obviously isn’t sustainable in perpetuity).
Re: 4, I would VERY much like to see EA develop growth channels that aren’t dependent on a public figure (particularly a philosopher) releasing a book, going on a tour, publicizing his multibillion dollar crypto exchange, etc. More organic channels (e.g. campus outreach) seem more sustainable, more scalable, and less prone to the hero worship that often seems to be found in EA.
Thanks for clarifying this! I agree with Jason’s point that the percentage of CEA’s funding coming from OP’s GCRCB team is high enough that the exact figure doesn’t really change the incentives much. But I’m very glad that CEA has made the effort to try and diversify OP’s funding.
I do want to flag that if your understanding is correct, it seems quite bad/weird that the incentives/priorities/strategies of the most important EA community building organization are significantly (mainly?) influenced by differences in “willingness to pay for talent” between OP’s GCRCB and GHWCR teams.
Has CEA ever solicited funding from Open Phil’s Global Health and Wellbeing team and/or does CEA have plans to do so? If not, why not?
IMO a world where CEA got e.g. 50% of its funding from OpenPhil’s GCR team and 35% of its funding from OpenPhil’s GHW team would be vastly preferable (due to incentives being better aligned with the broader community) to CEA getting 80% of its funding from OpenPhil GCR.
I’ve looked over Angelina’s (excellent) report on EA growth rates, and it raised some questions about how you see top of funnel growth rates for the community.
In July’s mid-year update on CEA, you wrote: “our priorities haven’t changed since we wrote about our work in [December] 2022: helping people who have heard about EA to deeply understand the ideas, and to find opportunities for making an impact in important fields. We continue to think that top-of-funnel growth is likely already at or above healthy levels, so rather than aiming to increase the rate any further, we want to make that growth go well.” (emphasis added).
Angelina’s analysis suggests that in 2022 top-of-funnel growth was quite rapid (73% growth in 2022 vs. 2021) but has slowed significantly so far this year (30% in 2023 vs. 2022). Do you agree with her finding that top-of-funnel growth has slowed significantly (though IMO the extent of the drop might be a bit overstated by her analysis due to e.g. huge 2022 spikes in search activity for EA in the wake of FTX)?
Do you still think top-of-funnel growth is already at or above healthy levels; and if so, is that because your assessment of a healthy growth rate has dropped significantly similar to how actual growth has fallen? If not, is CEA planning any strategic shifts as a result? What do you consider a healthy level for top-of-funnel growth over the next year or two?
That’s great, thanks!
Thank you Angelina for this excellent post! I really hope that this analysis, or something quite similar, gets conducted on a regular basis and shared with the entire community.
A few thoughts on this:
There are a lot of complications, judgement calls, assumptions that need to go into something like this. I think you did a very good job of articulating these and the associated limitations, and also made reasonable judgement calls along the way.
You mention that some metrics measure “levels” while others measure “changes”. I haven’t fully thought through this, but my hunch is that this is a pretty important issue and any future iterations of this report would benefit a lot from really thinking through the implications of the different types of measurement.
I probably lean toward “change” measures giving a better representation of recent growth patterns. The levels of a fair number of EA metrics grew a lot in 2022 (due to long-term growth trends and WWOTF) then flattened out or declined in 2023 (due to aftermath of FTX and subsequent scandals). Comparing annual levels, you’d see positive growth in 2023 vs. 2022 (for metrics that flattened out or declined modestly) or zero growth (for metrics that had a 2023 decline that perfectly mirrored 2022 growth). Comparing annual changes would show 2023 as worse than 2022, which IMO is how we should be interpreting this general pattern.
In addition to the AI and Effective Giving aggregations, I’d be curious to see an aggregation for effective careers (80k + Animal Advocacy Careers + any relevant redacted orgs)
I think the summary chart would be a lot easier to read/interpret if it were presented as a table.
Where there’s no data available for 2021, I think it would be helpful to include totals that are on an apples-to-apples basis (i.e. that include only projects for which there are both 2022 vs. 2021 and 2023 v. 2022 growth rates.)
Sure! Here’s a comment I wrote in response to a similar question in March. I don’t think it is exhaustive, but should be fairly illustrative.
Some specific examples of EA leaders putting SBF on a pedestal that I found with a bit of brief digging:
At the time FTX blew up, SBF was featured on 80k’s homepage. Also, if you clicked “start here” on that homepage (the first link aside from a subscription form) you were brought to an article that featured SBF as one of three individual profiles.
Both of these mentions linked to a more in depth profile of SBF that had been created in 2014 and regularly updated, and clearly “puts him on a pedestal” (“This approach — where he donates a significant proportion of his income to organisations aiming to make the world a better place as effectively as possible — is allowing Sam to have a pretty staggering impact.”)
When Will wrote about how the EA Funding situation had changed, he praised SBF not just for his donations but also his personal virtue “I think the fact that Sam Bankman-Fried is a vegan and drives a Corolla is awesome, and totally the right call.”
In Will’s appearance on the 80k podcast, he uses SBF as the exemplar of earning to give, and says convincing SBF to pursue ETG was “really the important impact.” He also uses SBF to illustrate “fat tails” of impact. On another 80k podcast, 80k staff talks about SBF to illustrate the impact of their 1:1 team.
In a talk at EAG London 2021, Ben Todd argues “that the recent success of Sam Bankman-Fried is an additional reason to aim high.” (Ben also mentions SBF in a variety of posts discussing EA’s funding levels, but that seems to me like an unavoidable part of discussing that issue and less like putting him on a pedestal.)
All those examples come from the period after concerns about SBF had been raised to EA leaders. Prior to that, there are plenty more examples especially for 80k (e.g. SBF was on the 80k homepage from late 2014 to late 2017.)
As far as I can tell, EA leaders started promoting SBF in 2014 or so, seeing him as a great example of altruistic career choice in general, and of ETG (a counterintuitive model of altruistic career choice that originated in EA) specifically. Then leaders kept promoting SBF despite the warnings they got from Alameda co-founders, and continued to do so until FTX blew up.
Another data point that speaks to SBF being on a pedestal was the individual who at EAG London 2022 (without consent of the conference organizers)distributed hundreds of stickers depicting Sam on a bean bag with the text “what would SBF do?” Whether this individual was themselves putting SBF on a pedestal or was making a joke about the extent SBF was already being put on a pedestal, I think this anecdote illustrates how Sam was often portrayed.
IMO the big mistakes before the collapse were 1) making SBF an EA poster child and 2) not giving enough weight to the possibility of a massive drawdown or complete collapse in FTX’s valuation.*
In each case, I think the magnitude of the mistake was dramatically increased by the fact that some parts of EA leadership apparently reasons to think SBF was/could be shady well prior to the FTX collapse. (Though to be clear, it does not seem reasonable to think that those leaders should have foreseen the extent of the fraud that appears to have taken place). I also believe EA’s previous experience with Ben Delo (who also became a crypto billionaire through running an offshore exchange before having legal troubles) should have given the community/leadership pause before putting SBF on a pedestal and assuming FTX funding would be in place.
After the collapse, I’d guess the lack of communication around FTX from EA leaders/orgs will probably turn out to be a mistake (though I’m less sure about this as I’m not privy to info about e.g. legal liability).
*The drawdown/collapse scenarios weren’t completely ignored, but my sense is that very little weight was put on them. See for example Will MacAskill’s 6300 word piece on “EA and the current funding situation” from May 2022. Will only mentions downside volatility twice, very much in passing: “bear in mind the difficulty of valuing crypto assets, and their volatility” and “total potential funding could still decrease by a lot, especially if FTX crashes.” Even at the time, there were comments from the community about paying more attention to the inherent volatility of FTX.
Thanks Angelina! Your post is terrific, and I hope that you or someone else will update it in the future. It seems like a great starting point for the sort of regular analysis of community metrics I’ve called for. And I appreciate your clarification on some of the data points in my analysis.
Luke Freeman from GWWC has shared some excellentobservations around reduced willingness for community members to advocate for GWWC post-FTX (partially but not solely due to FTX). I think it would be quite valuable if the follow-up survey examined the dynamics he describes, as I doubt they are unique to GWWC.
Thanks for clarifying. FWIW, given community desires for more communication from leadership and greater transparency, I think it would have been good for the board to share this decision publicly.
Thanks so much for this additional data and analysis! Really interesting stuff here. To me, the most interesting things are:
The midpoint of the satisfaction scale being a pretty good threshold for the point at which we see behavior changes
The relatively high frequency of people with high satisfaction temporarily stopping promoting EA (and the general flatness of this curve)
I was surprised that for the cohort that changed their behavior, “scandal” was just one of many reasons for dissatisfaction and didn’t really stand out. The data you provide looks quite consistent with Luke Freeman’s observation: “My impression is that that there was a confluence of things that peaking around the FTX-collapse. There was building hostility towards some of the more avant garde EA ideas an actions of those associated with EA[1] (towards both accurate and also misrepresentations of those ideas and actions) that seemed to get traction around just prior to the launch of WWOTF which meant there were a lot of people/opinions that got a lot more light when WWOTF was getting lots of attention and FTX failed so spectacularly. Then there was so much energy and angst in the system (both within the community and its critics) that I think the other issues compounded more than any individual one would have. The confluence of all this has sadly left a bad taste in a lot of people’s mouths that costs fairly uncontroversially good things a lot in terms of action and advocacy.”