“2022 was a year of continued growth for CEA and our programs.”—A bit of a misleading way to summarise CEA’s year?
“maintaining high retention and morale”—to me there did seem to be a dip in morale at the office recently
“[EA Forum] grew by around 2.9x this year.”—yes, although a bit of this was due to the FTX catastrophe
“Overall, we think that the quality of posts and discussion is roughly flat over the year, but it’s hard to judge.”—this year, a handful of people told me they felt the quality had decreased, which didn’t happen in previous years, and I noticed this too.
“Recently the community took a significant hit from the collapse of FTX and the suspected illegal and/or immoral behaviour of FTX executives.”—this is a very understated way to note that a former board member of CEA committed one of the largest financial frauds of all time.
I realise there are legal and other constraints, so maybe I am being harsh, but overall, several components of this post seemed not very “real” or straightforward relative to what I would usually expect from this sort of EA org update.
Hey Ryan, thanks for writing this up—keen to explain this and reflect more.
First, I do think that I could have written this post better: e.g. given the disclaimer about not focusing on FTX up front.
I think that a lot of this is a substantive disagreement about how CEA’s year went, which I think might be driven by a substantive disagreement about what CEA is.[1]
I’m aware that some people (possibly including you) have a model that’s a bit like “CEA is responsible for what happens in the EA community. A big bad thing happened to the EA community, so CEA is responsible, and so they had a pretty bad year.” (I expect that this is a bit of an oversimplification of your view, but hopefully it helps make the direction clear.)
An oversimplification of my view is “CEA is responsible for running EAG, EAGx, UGAP, groups support, virtual programs, the EA Forum, EA.org. A couple of these things had middling years, but many of them seemed to grow or improve significantly (like 5x, 2.9x growth). So CEA had a pretty good year, and “a year of continued growth” is a good summary.”
The key way in which that is an oversimplification is that the Community Health & Special Projects team does have a broader mandate: scanning for risks and trajectory changes to the community and trying to address them. As mentioned in that section, we clearly didn’t prioritise or eliminate this risk—we’re reflecting on how much that was a mistake, versus a good decision (to focus on other significant risks). We are reflecting on what we should do differently to reduce this and related risks in the future, and we’ll communicate more about that in due course. As I say in response to Larks, there may well be things that we end up thinking were big mistakes here, that should cause us to course-correct (and should go on our mistakes page). But we’re not yet confident in if/what the mistakes were, so we didn’t want to add them in this post yet. I should have noted that in that section though—I’ll go and edit.
So one way I think of this is, should CEA’s annual review be kinda the same thing as EA’s annual review? If I were writing EA’s annual review, there would be a bunch of negative stuff in there, and the overall tone would be pretty different to the above. If this is what CEA’s annual review should be, I did a pretty bad job and you’re right.
Or should it be more like Rethink Priorities[2] annual review? I think that Rethink Priorities could reasonably be like “Our core work went well! (Footnote: though it’s impact may be attenuated by one of EA’s major funders blowing up in a very visible way. We’ll reflect on what that changes for us.)” I think there’s a legit version of CEA’s annual review that is like this, if you view CEA as more being responsible for running EAG, groups support, etc.
Overall, I think that CEA is clearly somewhere in between these extremes—we’re more focused on and accountable for how EA overall goes than RP are. But my current take, based on how I view CEA’s role/mandate, is that it’s a bit more on the “Rethink Priorities” end, whereas I guess you think it’s more on the “EA annual review” end. I expect that this partly explains why I took a different approach to the annual review.
Going through your nitpicks:
Agree that morale dipped around FTX stuff. My impression is that it’s more back-to-normal, but we should probably have noted this. (And I think it’s a caveat to the overall thing of “doubled the team size and maintained morale/retention” thing though.)
Yep, some of the growth was due to FTX attention (but again, this feels like a bit of a footnote—a significant majority of the growth was not to do with FTX, and I think that the FTX growth was an example of the Forum providing real value, by helping people to process the news).
This is a fair point about Forum quality—I’m aware that some people think this. Our current take, having looked into this a bit and looked over top post lists from previous years, is that the quality is pretty constant but that people are struggling to find the most interesting posts as post volume has increased. This is still a problem, and something that we’re looking to fix. I think that this is an important amendment to the statement in the post, and something I’ll correct.
Understatement of the FTX thing—I agree the tone is maybe a bit bland here, but I’m not sure it really understates it? “Community took a significant hit” seems about the right level.
To sum up, despite the big-picture disagreement above, I think that you’re right that we should have added a bit more on the setbacks and caveats to the big picture. But my perspective is that the big picture on CEA’s year really is positive.
As a meta point, I’m aware that I’m clearly biased to think that CEA went well, and to want to say that CEA went well. This post is an attempt to accurately convey the big picture as I see it, but I’m aware that I may be biased here! I think that I can speak somewhat definitively to how I view CEA’s scope though.
Thanks for the response. Out of the four responses to nitpicks, I agree with the first two. I broadly agree about the third, forum quality. I just think that peak post quality is at best a lagging indicator—if you have higher volume and even your best posts are not as good anymore, that would bode very poorly. Ideally, the forum team would consciously trade off between growth and average post quality, and in some cases favouring the latter, e.g. performing interventions that would improve the latter even if they slowed growth. And the fourth, understatement, I don’t think we disagree that much.
As for summarising the year, it’s not quite that I want you say that CEA’s year was bad. In one sense, CEA’s year was fine, because these events don’t necessarily reflect negatively on CEA’s current operations. But in another important sense, it was a terrible year for CEA, because these events have a large bearing on whether CEA’s overarching goals are being reached. And this could bear on on what operating activities should be performed in future. I think an adequate summary would capture both angles. In an ideal world (where you were unconstrained by legal consequences etc.), I think an annual review post would note that when such seismic events happen, the standard metrics become relatively less important, while strategy becomes more important, and the focus of discussion then rests on the actually important stuff. I can accept that in the real world, that discussion will happen (much) later, but it’s important that it happens.
Ideally, the forum team would consciously trade off between growth and average post quality [emphasis added]
I disagree. I think the metric I care about is “quality of the average post that a person reads.”
The Forum will have a long tail of posts that are written by newbies just exploring some area for the first time, or are kinda confused, or are bad takes, etc. Many of these posts are net positive! I’m rather in favor of people having their learning experiences in public. Many times the comments on those posts are good places to recapitulate the best of EA. The good news is that most of those posts don’t get much karma or readership. I’m sure you can think of posts that you don’t like that got lots of karma. There’s a complex conversation to be had there, I hope Lizka will post her draft on that soon. But I’m talking more about the much more common post that sits at 0-20 karma. There are lots of them. But they’re by and large pretty harmless. I don’t want my metric to be reducing them.
I don’t know how many people use RSS feeds, but they interact somewhat poorly with this, because the algorithmic post prioritization does not affect RSS readers.
I don’t mean for me personally—I just mean it will bias the web analytics, because low-karma posts will look like they make up a smaller fraction of impressions than in reality (assuming most RSS feed users do not apply such a cutoff).
As mentioned in that section, we clearly didn’t prioritise or eliminate this risk—we’re reflecting on how much that was a mistake, versus a good decision (to focus on other significant risks).
Hm, if you don’t mind the question — what are other significant risks you are thinking about? IMO, the FTX fraud scheme is a big deal if anything deserves to be called a big deal, from what I’ve read in the devastating legal documents about the situation, and given how SBF was framed as a poster child of EA and was a significant funder.
The only more significant risk I can think of is EA funding dangerous AI capabilities research via Anthropic, but that isn’t even unrelated to FTX (since FTX slash Alameda slash their leadership was Anthropic’s main funder). Also, my guess is that mitigating AI risk is not within the Community Health team’s scope.
Several nitpicks:
“2022 was a year of continued growth for CEA and our programs.”—A bit of a misleading way to summarise CEA’s year?
“maintaining high retention and morale”—to me there did seem to be a dip in morale at the office recently
“[EA Forum] grew by around 2.9x this year.”—yes, although a bit of this was due to the FTX catastrophe
“Overall, we think that the quality of posts and discussion is roughly flat over the year, but it’s hard to judge.”—this year, a handful of people told me they felt the quality had decreased, which didn’t happen in previous years, and I noticed this too.
“Recently the community took a significant hit from the collapse of FTX and the suspected illegal and/or immoral behaviour of FTX executives.”—this is a very understated way to note that a former board member of CEA committed one of the largest financial frauds of all time.
I realise there are legal and other constraints, so maybe I am being harsh, but overall, several components of this post seemed not very “real” or straightforward relative to what I would usually expect from this sort of EA org update.
Hey Ryan, thanks for writing this up—keen to explain this and reflect more.
First, I do think that I could have written this post better: e.g. given the disclaimer about not focusing on FTX up front.
I think that a lot of this is a substantive disagreement about how CEA’s year went, which I think might be driven by a substantive disagreement about what CEA is.[1]
I’m aware that some people (possibly including you) have a model that’s a bit like “CEA is responsible for what happens in the EA community. A big bad thing happened to the EA community, so CEA is responsible, and so they had a pretty bad year.” (I expect that this is a bit of an oversimplification of your view, but hopefully it helps make the direction clear.)
An oversimplification of my view is “CEA is responsible for running EAG, EAGx, UGAP, groups support, virtual programs, the EA Forum, EA.org. A couple of these things had middling years, but many of them seemed to grow or improve significantly (like 5x, 2.9x growth). So CEA had a pretty good year, and “a year of continued growth” is a good summary.”
The key way in which that is an oversimplification is that the Community Health & Special Projects team does have a broader mandate: scanning for risks and trajectory changes to the community and trying to address them. As mentioned in that section, we clearly didn’t prioritise or eliminate this risk—we’re reflecting on how much that was a mistake, versus a good decision (to focus on other significant risks). We are reflecting on what we should do differently to reduce this and related risks in the future, and we’ll communicate more about that in due course. As I say in response to Larks, there may well be things that we end up thinking were big mistakes here, that should cause us to course-correct (and should go on our mistakes page). But we’re not yet confident in if/what the mistakes were, so we didn’t want to add them in this post yet. I should have noted that in that section though—I’ll go and edit.
So one way I think of this is, should CEA’s annual review be kinda the same thing as EA’s annual review? If I were writing EA’s annual review, there would be a bunch of negative stuff in there, and the overall tone would be pretty different to the above. If this is what CEA’s annual review should be, I did a pretty bad job and you’re right.
Or should it be more like Rethink Priorities[2] annual review? I think that Rethink Priorities could reasonably be like “Our core work went well! (Footnote: though it’s impact may be attenuated by one of EA’s major funders blowing up in a very visible way. We’ll reflect on what that changes for us.)” I think there’s a legit version of CEA’s annual review that is like this, if you view CEA as more being responsible for running EAG, groups support, etc.
Overall, I think that CEA is clearly somewhere in between these extremes—we’re more focused on and accountable for how EA overall goes than RP are. But my current take, based on how I view CEA’s role/mandate, is that it’s a bit more on the “Rethink Priorities” end, whereas I guess you think it’s more on the “EA annual review” end. I expect that this partly explains why I took a different approach to the annual review.
Going through your nitpicks:
Agree that morale dipped around FTX stuff. My impression is that it’s more back-to-normal, but we should probably have noted this. (And I think it’s a caveat to the overall thing of “doubled the team size and maintained morale/retention” thing though.)
Yep, some of the growth was due to FTX attention (but again, this feels like a bit of a footnote—a significant majority of the growth was not to do with FTX, and I think that the FTX growth was an example of the Forum providing real value, by helping people to process the news).
This is a fair point about Forum quality—I’m aware that some people think this. Our current take, having looked into this a bit and looked over top post lists from previous years, is that the quality is pretty constant but that people are struggling to find the most interesting posts as post volume has increased. This is still a problem, and something that we’re looking to fix. I think that this is an important amendment to the statement in the post, and something I’ll correct.
Understatement of the FTX thing—I agree the tone is maybe a bit bland here, but I’m not sure it really understates it? “Community took a significant hit” seems about the right level.
To sum up, despite the big-picture disagreement above, I think that you’re right that we should have added a bit more on the setbacks and caveats to the big picture. But my perspective is that the big picture on CEA’s year really is positive.
As a meta point, I’m aware that I’m clearly biased to think that CEA went well, and to want to say that CEA went well. This post is an attempt to accurately convey the big picture as I see it, but I’m aware that I may be biased here! I think that I can speak somewhat definitively to how I view CEA’s scope though.
I chose Rethink Priorities kind of at random as an example—this could be many EA orgs, and I don’t mean it to reflect anything on RP in particular.
Thanks for the response. Out of the four responses to nitpicks, I agree with the first two. I broadly agree about the third, forum quality. I just think that peak post quality is at best a lagging indicator—if you have higher volume and even your best posts are not as good anymore, that would bode very poorly. Ideally, the forum team would consciously trade off between growth and average post quality, and in some cases favouring the latter, e.g. performing interventions that would improve the latter even if they slowed growth. And the fourth, understatement, I don’t think we disagree that much.
As for summarising the year, it’s not quite that I want you say that CEA’s year was bad. In one sense, CEA’s year was fine, because these events don’t necessarily reflect negatively on CEA’s current operations. But in another important sense, it was a terrible year for CEA, because these events have a large bearing on whether CEA’s overarching goals are being reached. And this could bear on on what operating activities should be performed in future. I think an adequate summary would capture both angles. In an ideal world (where you were unconstrained by legal consequences etc.), I think an annual review post would note that when such seismic events happen, the standard metrics become relatively less important, while strategy becomes more important, and the focus of discussion then rests on the actually important stuff. I can accept that in the real world, that discussion will happen (much) later, but it’s important that it happens.
I disagree. I think the metric I care about is “quality of the average post that a person reads.”
The Forum will have a long tail of posts that are written by newbies just exploring some area for the first time, or are kinda confused, or are bad takes, etc. Many of these posts are net positive! I’m rather in favor of people having their learning experiences in public. Many times the comments on those posts are good places to recapitulate the best of EA. The good news is that most of those posts don’t get much karma or readership. I’m sure you can think of posts that you don’t like that got lots of karma. There’s a complex conversation to be had there, I hope Lizka will post her draft on that soon. But I’m talking more about the much more common post that sits at 0-20 karma. There are lots of them. But they’re by and large pretty harmless. I don’t want my metric to be reducing them.
Totally, this is what I had in mind—something like the average over posts based on how often they are served on the frontpage.
I don’t know how many people use RSS feeds, but they interact somewhat poorly with this, because the algorithmic post prioritization does not affect RSS readers.
Not sure if this fully helps but you can use a karma cutoff for the RSS feed
I don’t mean for me personally—I just mean it will bias the web analytics, because low-karma posts will look like they make up a smaller fraction of impressions than in reality (assuming most RSS feed users do not apply such a cutoff).
Hm, if you don’t mind the question — what are other significant risks you are thinking about? IMO, the FTX fraud scheme is a big deal if anything deserves to be called a big deal, from what I’ve read in the devastating legal documents about the situation, and given how SBF was framed as a poster child of EA and was a significant funder.
The only more significant risk I can think of is EA funding dangerous AI capabilities research via Anthropic, but that isn’t even unrelated to FTX (since FTX slash Alameda slash their leadership was Anthropic’s main funder). Also, my guess is that mitigating AI risk is not within the Community Health team’s scope.