I work on the CEA Online Team, which runs the Forum, but I am only speaking for myself. Others on my team may disagree with me. I wrote this relatively quickly so I wouldnât be surprised if I changed my mind on things upon reflection.
Overall, I really appreciated Nuñoâs article, and did not find it to be harsh, overly confrontational, or unpleasant to read. I appreciated the nice messages that he included to me, a person working on the Forum, at the start and end of the piece.
On the design change and addition of features:
People have different aesthetic preferences, and I personally think the current Forum design looks nicer than the 2018 version, plus I think it has better usability in various ways. I like minimalism in some contexts, but I care more about doing good than about making the Forum visually pleasing to me. To that end, I think it is correct for the Forum to have more than just a simple frontpage list of posts plus a ârecent discussionâ feed (which seems to be the entirety of the 2018 version).
For example, I think adding the âquick takesâ and âpopular commentsâ sections to the home page have been really successful. By making quick takes more salient, weâve encouraged additional discussions on the site (since quick takes are intended for less polished posts). âPopular commentsâ helps to highlight ongoing discussions in posts that may not be visible elsewhere. I take the fact that LessWrong borrowed these sections for their site as further evidence of their value, and in fact LessWrong has had some impactful discussions happen in their quick takes section. As another example, features like the âGroups directoryâ and the âPeople directoryâ are not available anywhere else online, and I view them as more like âessential infrastructure for the EA communityâ. I think itâs reasonable for those to live on the EA Forum, where people already gather to talk about EA things and look for some EA resources.
On the costs of the Forum:
Here, I want to clarify what the actual costs are. I will focus on budgeted costs over true costs for privacy reasons.
I believe that the $2M/âyear was the highest ever (overall, budgeted) cost of the CEA Online Team, when we were a team of 8 people, and it was only the case for approximately 6 months (between Jan 2023 and July 2023) (see update below). We were probably 7 people for another ~6 months total, and less than 7 people for the rest of the time.
Update 2024-12-09: I was just thinking about this again, and I believe actually the largest size of the team was 10 people, between Jan 2023 and Mar 2023. I donât know what the budget implications are â probably not much since we budget per year.
We are currently at 6 people, and our teamâs budget for the year is close to $1.6M/âyear. I joined in late 2021, when the team was 5 people (including me). 6 is a more representative number for how many people have been on the Online Team in the past few years.
My understanding is that about $960K/âyear of the budget is staff costs, broadly construed.[1] The rest are things such as SaaS costs (which we have made significant strides in reducing in the past year).
Our true staff costs are lower, because multiple people on the Online Team choose to reduce their take-home pay. We budget for their full salaries because CEA needs to be able to pay them that salary if they choose to take it. I think itâs reasonable to allow people to keep this information private, so we do not report the true staff costs.
The Online Team does not spend all of its time working on the Forum (although it is a majority of our time). The actual time split depends on when other projects arise, either via external partnerships or internal initiatives (such as the big EA.org redesign in 2022). I would broadly estimate that we spent 10% of Online Team staff time on other projects over the past 3 years.
Therefore, the actual average yearly cost of Forum staff time has been roughly $860K minus voluntary salary reductions.
In the future, I expect more Online Team staff time to be used for non-Forum projects, so I expect that cost to naturally decrease over time. (We spent roughly 20% of staff time on non-Forum projects in the past 3 months, for example.)
EDIT: I will also note that there are additional overheads to staff that are not captured even in the âbroadly construedâ figures above. For example, we have a People Ops team whose job is focused on CEA staff, so one could argue that part of the People Ops budget goes towards the Forum team. Part of the rest of the $1.6M/âyear is also ops overhead related to having staff. You could even argue that some of the SaaS costs are overhead on staff. I didnât want to go line by line and delay posting this quick take, but Iâm adding this here to highlight that âstaff costsâ can vary a lot depending on what you consider part of that cost. I just picked a âstaff costâ value that I thought would generally match the definition that readers would have in mind (i.e. the most direct costs to the company, which is mostly salary plus benefits). I would guess that from the CEA perspective (like if we were considering hiring another person on our team), we would take into account other overhead costs to some extent.
[I will skip responding to the moderation section because I have not been involved with that.]
On Forum culture evolution:
I guess I generally think things like, making the Forum more accessible and decreasing barriers to entry, are good. There are many people in the world who can do good via the broad project of effective altruism, and I want the Forum to work for them as much as is reasonable. Of course there are tradeoffs, and we do need to keep the interests of more experienced users in mind and be mindful not to lose parts of the Forum culture that are valuable, but I donât see why that should mean we stop making the Forum more accessible. In my opinion, new Forum users are some of the most valuable contributors, and it would be to our detriment to ignore them.
On cost-effectiveness:
I think we agree that what matters is marginal cost-effectiveness. I believe that the marginal expense of the Online Team is a software engineer. Our team has done an internal marginal impact analysis to evaluate how much value the marginal engineer produces via the Forum (broadly, by looking at the software engineering work we have done in a year, BOTEC-ing how much counterfactual value it has produced, then attributing an appropriate fraction to the marginal engineer), and the resulting 90% confidence interval was above break even.[2]
Multiple people within CEA red-teammed it (and there is not full agreement on the current version). The last time I looked at the Squiggle model, the analysis had over 100 factors. That doesnât prove its accuracy but hopefully it demonstrates that we take this seriously. I hope to publish more about this publicly, but it will take a good chunk of time to extract the private information and write a version that is clear for a public audience. And, as I mentioned in another comment, I actually donât know how to incorporate the time I spend on writing it up into the marginal impact analysis, so by default I feel like itâs a poor use of my work time (like, by spending time writing a public version, I would actually be decreasing the value of the marginal engineer, though this is slightly complicated by the fact that Iâm currently technically in a non-engineering interim role).
Responding to the bottom part of your second footnote:
To me, it seems pretty important for Forum Team members (especially the interim lead!) to be communicating with Forum users. I therefore think itâs a mistake for you to assign zero value to your posts and comments, relative to your other work.
How much value to assign to one of your posts or comments? I would crudely model this as:
(Size of your postâs or commentâs contribution)/â(Size of all Forum contributions in a year) x (Forumâs total value per year)
Youâll have better-informed figures/âestimates than I do, but Iâd guess that the size of all Forum contributions, measured in karma,[1] in a year, is around 100,000, and that the value of the Forum per year is around $10M.[2] A thoughtful comment might get 10 karma, on average, and a thoughtful post might get 50.
Iâd therefore roughly value a comment from you at (10 karma)/â(100,000 karma) x $10M = $1000, and a post from you at $5000.
(My model may well be off in some way; I invite readers to improve on it.)
I donât take karma to be a perfect measure of value by any meansâsee, e.g., âKarma overrates some topicsââbut I think itâs a reasonable-enough measure for carrying out this BOTEC.
Why $10M? Mostly Iâm working off the value of an âEA projectâ as estimated in cell C4 of this spreadsheet by Nuño. (This was the accompanying post.)
Thanks Will! I really appreciate this comment, and I think this is a great suggestion. I buy something in this general direction (the Forum produces a certain amount of value per year, and so contributing on the Forum is worth some value relative to the total). I think my Forum-related posts/âcomments tend not to be very actionable â like, I would guess a post from me is actually worth less (in value to the world) than a post about a new job opening, and probably worth less than an object-level post with the same amount of karma. But I also think that coordination/âinformation-sharing is a key source of value for the Forum, so Iâm convinced enough to dig into this further and value my comms at more than $0. (The value per hour is likely going to look less promising though, since I am pretty slow at writing...)
As a bonus, I think this is a good general pitch for everyone to write more on the Forum! ^^
Another (complementary) framing here is that I assume your âday jobâ is mostly improving the Forum. So you can ~directly compare how much an additional hour of commenting or post on the Forum is worth in terms of total value add to the Forum, relative to the expected marginal impact of improving the codebase by an hour of coding or other work you do[1].
This a) sidesteps the absolute value of Forum work, which is potentially more contentious, and b) gives you a direct way to assess how valuable things are relative to the opportunity cost.
Interesting â I feel like I donât have good intuitions for how to compare an hour of coding with an hour of writing. But I believe youâre suggesting something like, convert both of these into âhours of Forum engagementâ per hour of my work (like, comparing how much engagement my post gets, divided by the hours I spent writing it, to the hours of engagement I add to the Forum for adding a feature, divided by the hours I spent building it).
If I were to do that comparison, Iâm guessing that coding looks way better. My best post got ~23 hours of engagement total, and I donât expect to get significantly more than that. One of our best features wrt engagement is adding AI-narrated audio for posts. This gets us very approximately 8 hours of engagement per day, was relatively quick to do, and we expect it will last for years. If it lasts 5 years that would be 14,600 hours of engagement. Even if it took 10x as long to build as the post did to write, building the feature seems clearly better.
Perhaps this is some evidence that I should not spend more time writing on the Forum, though I think thereâs some value from posting that is not captured by this, so maybe converting everything into dollars is still better if it allows me to account for more factors.
No easily summarizable comment on the rest of it, but as a LessWrong dev I do think the addition of Quick Takes to the front page of LW was very goodâmy sense is that itâs counterfactually responsible for a pretty substantial amount of high quality discussion. (I havenât done any checking of ground-truth metrics, this is just my gestalt impression as a user of the site.)
Agree! I am a bit worried the discussion is a bit social-drama heavy on LW, since itâs kind of the only outlet for that kind of stuff on the frontpage, but that in itself is also somewhat of a success (having a place for more community-oriented discussion without taking over the whole site).
Some quick responses to Nuñoâs article about EA Forum stewardship
I work on the CEA Online Team, which runs the Forum, but I am only speaking for myself. Others on my team may disagree with me. I wrote this relatively quickly so I wouldnât be surprised if I changed my mind on things upon reflection.
Overall, I really appreciated Nuñoâs article, and did not find it to be harsh, overly confrontational, or unpleasant to read. I appreciated the nice messages that he included to me, a person working on the Forum, at the start and end of the piece.
On the design change and addition of features:
People have different aesthetic preferences, and I personally think the current Forum design looks nicer than the 2018 version, plus I think it has better usability in various ways. I like minimalism in some contexts, but I care more about doing good than about making the Forum visually pleasing to me. To that end, I think it is correct for the Forum to have more than just a simple frontpage list of posts plus a ârecent discussionâ feed (which seems to be the entirety of the 2018 version).
For example, I think adding the âquick takesâ and âpopular commentsâ sections to the home page have been really successful. By making quick takes more salient, weâve encouraged additional discussions on the site (since quick takes are intended for less polished posts). âPopular commentsâ helps to highlight ongoing discussions in posts that may not be visible elsewhere. I take the fact that LessWrong borrowed these sections for their site as further evidence of their value, and in fact LessWrong has had some impactful discussions happen in their quick takes section. As another example, features like the âGroups directoryâ and the âPeople directoryâ are not available anywhere else online, and I view them as more like âessential infrastructure for the EA communityâ. I think itâs reasonable for those to live on the EA Forum, where people already gather to talk about EA things and look for some EA resources.
On the costs of the Forum:
Here, I want to clarify what the actual costs are. I will focus on budgeted costs over true costs for privacy reasons.
I believe that the $2M/âyear was the highest ever (overall, budgeted) cost of the CEA Online Team,
when we were a team of 8 people, and it was only the case for approximately 6 months (between Jan 2023 and July 2023)(see update below). We were probably 7 people for another ~6 months total, and less than 7 people for the rest of the time.Update 2024-12-09: I was just thinking about this again, and I believe actually the largest size of the team was 10 people, between Jan 2023 and Mar 2023. I donât know what the budget implications are â probably not much since we budget per year.
We are currently at 6 people, and our teamâs budget for the year is close to $1.6M/âyear. I joined in late 2021, when the team was 5 people (including me). 6 is a more representative number for how many people have been on the Online Team in the past few years.
My understanding is that about $960K/âyear of the budget is staff costs, broadly construed.[1] The rest are things such as SaaS costs (which we have made significant strides in reducing in the past year).
Our true staff costs are lower, because multiple people on the Online Team choose to reduce their take-home pay. We budget for their full salaries because CEA needs to be able to pay them that salary if they choose to take it. I think itâs reasonable to allow people to keep this information private, so we do not report the true staff costs.
The Online Team does not spend all of its time working on the Forum (although it is a majority of our time). The actual time split depends on when other projects arise, either via external partnerships or internal initiatives (such as the big EA.org redesign in 2022). I would broadly estimate that we spent 10% of Online Team staff time on other projects over the past 3 years.
Therefore, the actual average yearly cost of Forum staff time has been roughly $860K minus voluntary salary reductions.
In the future, I expect more Online Team staff time to be used for non-Forum projects, so I expect that cost to naturally decrease over time. (We spent roughly 20% of staff time on non-Forum projects in the past 3 months, for example.)
EDIT: I will also note that there are additional overheads to staff that are not captured even in the âbroadly construedâ figures above. For example, we have a People Ops team whose job is focused on CEA staff, so one could argue that part of the People Ops budget goes towards the Forum team. Part of the rest of the $1.6M/âyear is also ops overhead related to having staff. You could even argue that some of the SaaS costs are overhead on staff. I didnât want to go line by line and delay posting this quick take, but Iâm adding this here to highlight that âstaff costsâ can vary a lot depending on what you consider part of that cost. I just picked a âstaff costâ value that I thought would generally match the definition that readers would have in mind (i.e. the most direct costs to the company, which is mostly salary plus benefits). I would guess that from the CEA perspective (like if we were considering hiring another person on our team), we would take into account other overhead costs to some extent.
[I will skip responding to the moderation section because I have not been involved with that.]
On Forum culture evolution:
I guess I generally think things like, making the Forum more accessible and decreasing barriers to entry, are good. There are many people in the world who can do good via the broad project of effective altruism, and I want the Forum to work for them as much as is reasonable. Of course there are tradeoffs, and we do need to keep the interests of more experienced users in mind and be mindful not to lose parts of the Forum culture that are valuable, but I donât see why that should mean we stop making the Forum more accessible. In my opinion, new Forum users are some of the most valuable contributors, and it would be to our detriment to ignore them.
On cost-effectiveness:
I think we agree that what matters is marginal cost-effectiveness. I believe that the marginal expense of the Online Team is a software engineer. Our team has done an internal marginal impact analysis to evaluate how much value the marginal engineer produces via the Forum (broadly, by looking at the software engineering work we have done in a year, BOTEC-ing how much counterfactual value it has produced, then attributing an appropriate fraction to the marginal engineer), and the resulting 90% confidence interval was above break even.[2]
Inclusive of benefits, travel, allowances for equipment, etc. (Not all of which is spent.)
Multiple people within CEA red-teammed it (and there is not full agreement on the current version). The last time I looked at the Squiggle model, the analysis had over 100 factors. That doesnât prove its accuracy but hopefully it demonstrates that we take this seriously. I hope to publish more about this publicly, but it will take a good chunk of time to extract the private information and write a version that is clear for a public audience. And, as I mentioned in another comment, I actually donât know how to incorporate the time I spend on writing it up into the marginal impact analysis, so by default I feel like itâs a poor use of my work time (like, by spending time writing a public version, I would actually be decreasing the value of the marginal engineer, though this is slightly complicated by the fact that Iâm currently technically in a non-engineering interim role).
Responding to the bottom part of your second footnote:
To me, it seems pretty important for Forum Team members (especially the interim lead!) to be communicating with Forum users. I therefore think itâs a mistake for you to assign zero value to your posts and comments, relative to your other work.
How much value to assign to one of your posts or comments? I would crudely model this as:
(Size of your postâs or commentâs contribution)/â(Size of all Forum contributions in a year) x (Forumâs total value per year)
Youâll have better-informed figures/âestimates than I do, but Iâd guess that the size of all Forum contributions, measured in karma,[1] in a year, is around 100,000, and that the value of the Forum per year is around $10M.[2] A thoughtful comment might get 10 karma, on average, and a thoughtful post might get 50.
Iâd therefore roughly value a comment from you at (10 karma)/â(100,000 karma) x $10M = $1000, and a post from you at $5000.
(My model may well be off in some way; I invite readers to improve on it.)
I donât take karma to be a perfect measure of value by any meansâsee, e.g., âKarma overrates some topicsââbut I think itâs a reasonable-enough measure for carrying out this BOTEC.
Why $10M? Mostly Iâm working off the value of an âEA projectâ as estimated in cell C4 of this spreadsheet by Nuño. (This was the accompanying post.)
Thanks Will! I really appreciate this comment, and I think this is a great suggestion. I buy something in this general direction (the Forum produces a certain amount of value per year, and so contributing on the Forum is worth some value relative to the total). I think my Forum-related posts/âcomments tend not to be very actionable â like, I would guess a post from me is actually worth less (in value to the world) than a post about a new job opening, and probably worth less than an object-level post with the same amount of karma. But I also think that coordination/âinformation-sharing is a key source of value for the Forum, so Iâm convinced enough to dig into this further and value my comms at more than $0. (The value per hour is likely going to look less promising though, since I am pretty slow at writing...)
As a bonus, I think this is a good general pitch for everyone to write more on the Forum! ^^
Another (complementary) framing here is that I assume your âday jobâ is mostly improving the Forum. So you can ~directly compare how much an additional hour of commenting or post on the Forum is worth in terms of total value add to the Forum, relative to the expected marginal impact of improving the codebase by an hour of coding or other work you do[1].
This a) sidesteps the absolute value of Forum work, which is potentially more contentious, and b) gives you a direct way to assess how valuable things are relative to the opportunity cost.
(assuming your direct counterfactual with writing comments is writing code or otherwise improving the forum)
Interesting â I feel like I donât have good intuitions for how to compare an hour of coding with an hour of writing. But I believe youâre suggesting something like, convert both of these into âhours of Forum engagementâ per hour of my work (like, comparing how much engagement my post gets, divided by the hours I spent writing it, to the hours of engagement I add to the Forum for adding a feature, divided by the hours I spent building it).
If I were to do that comparison, Iâm guessing that coding looks way better. My best post got ~23 hours of engagement total, and I donât expect to get significantly more than that. One of our best features wrt engagement is adding AI-narrated audio for posts. This gets us very approximately 8 hours of engagement per day, was relatively quick to do, and we expect it will last for years. If it lasts 5 years that would be 14,600 hours of engagement. Even if it took 10x as long to build as the post did to write, building the feature seems clearly better.
Perhaps this is some evidence that I should not spend more time writing on the Forum, though I think thereâs some value from posting that is not captured by this, so maybe converting everything into dollars is still better if it allows me to account for more factors.
No easily summarizable comment on the rest of it, but as a LessWrong dev I do think the addition of Quick Takes to the front page of LW was very goodâmy sense is that itâs counterfactually responsible for a pretty substantial amount of high quality discussion. (I havenât done any checking of ground-truth metrics, this is just my gestalt impression as a user of the site.)
Agree! I am a bit worried the discussion is a bit social-drama heavy on LW, since itâs kind of the only outlet for that kind of stuff on the frontpage, but that in itself is also somewhat of a success (having a place for more community-oriented discussion without taking over the whole site).