How do you feel retrospectively about the change to lower community engagement on the EA forum? Do you have any numbers about activity/engagement since then?
Just a quick clarification: I don’t think this was a “change to lower community engagement.” Adding the community section was a change, and it did (probably[1]) lower engagement with community posts, but that wasn’t the actual point (which is a distinction I think is worth making, although maybe some would say it’s the same). In my view, the point was to avoid suffocating other discussions and to make the Forum feel less on-edge/anxiety-inducing (which we were hearing was at least some people’s experience). In case it helps, this outlines our (or at least my) thinking about it.
And before I get into my personal takes, here’s a graph of engagement hours on Community posts vs. other content:
and here’s a graph of the percent of overall engagement that’s spent on Community posts (the blue line is a running 30-day average):
And another graph that might be interesting is this one — percent of karma (orange) given that comes from downvotes (the black line is the total number of votes in a period):
On to how I feel about the change:
I’m pretty happy with it, with some caveats.
I was quite worried about it initially,[2] but eventually got convinced/decided that it was a good thing to try. I’m much more positive about it now, although I still think that we might be able to find a better and more elegant solution at some point.
Some reasons I feel positive about it:
(1) We get a lot of positive feedback on the Community Section, from a broad range of people.
For the initial test, we surveyed people on the Forum (more here), and got the following results:
(45 responses. On a scale of 1-7 where “1” was negative and “7″ was positive, 38% of responses put 6, 22% put 7, 13% put 5, 11% put 4, 7% put 3 and 2 (each), and 2% put 1.)
We’d also done some user interviews and gotten quick takes from people (again, a bit more here).
Since then, we’ve continued hearing from people who really appreciate it. At EAG and in other ~EA spaces, I’ve been, unprompted, told things like “I’m going on the Forum a lot more now because I see interesting and useful content” and “The Forum seems much better now.”
A number of people have also asked for ways to hide the section for themselves entirely.
(2) I feel that the Forum has gotten healthier.
I could try to point you to discussions that, before the change, I’d have expected would blow up in some unpleasant ways, I could pull out the number for how often moderators have to ban or warn someone per engagement hour in different periods, and I can point you to the “percent of karma that comes from downvotes” that I shared above, but I think this is mostly a subjective judgement, and factors in things that others might not agree with me on, like “how much does karma track what I think is more valuable?”
I also find it easier to read what I want to read on the Forum right now. I hadn’t expected a huge change in my own experience — I read a lot of Community posts! — but I actually quite like having the section.
(3) I was worried about various downsides, and they haven’t seemed to have gotten very bad — although that’s my main uncertainty.
Growth in total engagement hours has stalled. I don’t know if that will continue, and I don’t fully know why that is. Maybe people are tired, disenchanted with EA (or more precisely, maybe new folks aren’t finding EA as much), using Twitter/LW more, or too busy working on slowing down AI or the like. (I’ve seen data about other ~EA engagement metrics that suggest there’s at least some lull.) Or maybe we’ve discouraged authors of important and engaging Community content from posting on the Forum, and this has knock-on effects on non-Community engagement.
But we didn’t see a mass exodus, which I was somewhat worried about, and it’s not on a clear down trajectory. And evaluating all of this is pretty hard given the quick growth in 2022 due to FTX and WWOTF, and the crazy engagement levels in November.
I was really worried about the community/individual people missing out on important discussions. This is hard to notice — I shouldn’t expect to know what discussions I’m missing out on. But I was tracking some topics that I’d discussed with folks off the Forum that hadn’t made it on to the Forum, and I think most of them showed up during Strategy Fortnight, so I’m excited about more pushes/events like that (and other things listed here and elsewhere).
Here’s another bad and subjective datapoint: the average karma of some Community posts compared to some non-Community posts that are of a similar minimum quality/value (according to me) is quite similar (with Community posts winning a bit).[3]
There are other downsides, like:
Authors are sometimes sad that their posts are in the Community section
We’ve introduced some messiness/complexity to how the Forum is structured
Maybe we’ve implicitly made communicated something about what is and isn’t “valuable” that I don’t endorse
My personal approach here is to keep an eye on this, see if people are engaging to the extent that they endorse, and continue looking for better solutions.
I think this is almost certainly true — engagement with community posts went down —but there was also less big news to discuss, and that might have been the actual cause of the decrease.
Initially I was particularly worried that we would miss out on important conversations, and worried about our reasoning (I was worried that we were getting feedback from a heavily biased sample of people). I think I was also worried that there was something “special” about the Forum as it was at the time — I’ve now articulated this in terms of “we’re at relatively ~stable eliquibrium points on some important parameters and structural/external changes can mess with that stability” — that we would mess with by doing something big like this.
Here’s a bad baseline sanity check that I did at some point: (1) Take the posts shared in the Digest — which gives me a baseline of ~quality/value-to-readers (according to my own judgement). Note that I don’t filter strongly for karma in the Digest; if something gets a lot of karma and I’m not too sure about it, I’d maybe err on the side of including, but I don’t really use karma as a big factor. (I might also be more likely to miss low-karma good posts, and there are other reasons to share or not share something in the Digest, but let’s put that aside for now.) (2) I compared how much karma different types of posts were getting. The posts that I was putting as ~top 3 in the list order for the Digets had noticeably more karma, on average. Setting those aside, though, Community posts in the Digest got ~100 karma on average, and non-Community posts got ~80 karma on average. (Note that there’s a lot of variation, though.)
Regarding “I feel that the Forum has gotten healthier,” is that statement in comparison to pre-FTX levels of health?
It doesn’t surprise me that a series of scandals, as well as the various pro-reform/anti-reform posts, would trigger a rise in downvote percentage and in warnable events per hour in comparison to a calmer mix of topics. Without more, I would not be inclined to view the 4Q 22 / 1Q 23 trends as strong evidence of a generalized health decline rather than a reflection of the topic mix that needed to be discussed because of events that happened external to the Forum.
This is a good question. I was mostly not thinking of pre-FTX levels when I was writing that part of my response. I’m not sure if I think my claim holds on longer time scales! Things in the community have been calmer recently (outside of the Forum); I don’t know what would have happened had we not added the section (i.e on the Forum. whether things would have calmed down by themselves), nor what would happen if we removed the section now (and unfortunately we can’t e.g. A/B test this kind of thing, but maybe we should test in some other way at some point).
Aside (not related to what I was talking about in the “Forum has gotten healthier” section above): I agree that 4Q 22 / 1Q 23 aren’t representative in many ways, but a key underlying motivation for the Community section is giving posts that have a more niche audience a better chance by separating out posts that tend to get a lot of karma so they’re not “fighting for the same spots” (which are determined by ~karma). That phenomenon (and goal) is unaffected by recent events; ie. what we described in Karma overrates some topics; resulting issues and potential solutions should hold at all times. (On a skim, the top-by-karma posts of 2023, 2022, to some extent 2021 are Community posts. Curiously, this seems less true of 2020 but again more true of 2019.)
How do you feel retrospectively about the change to lower community engagement on the EA forum? Do you have any numbers about activity/engagement since then?
Just a quick clarification: I don’t think this was a “change to lower community engagement.” Adding the community section was a change, and it did (probably[1]) lower engagement with community posts, but that wasn’t the actual point (which is a distinction I think is worth making, although maybe some would say it’s the same). In my view, the point was to avoid suffocating other discussions and to make the Forum feel less on-edge/anxiety-inducing (which we were hearing was at least some people’s experience). In case it helps, this outlines our (or at least my) thinking about it.
And before I get into my personal takes, here’s a graph of engagement hours on Community posts vs. other content:
and here’s a graph of the percent of overall engagement that’s spent on Community posts (the blue line is a running 30-day average):
And another graph that might be interesting is this one — percent of karma (orange) given that comes from downvotes (the black line is the total number of votes in a period):
On to how I feel about the change:
I’m pretty happy with it, with some caveats.
I was quite worried about it initially,[2] but eventually got convinced/decided that it was a good thing to try. I’m much more positive about it now, although I still think that we might be able to find a better and more elegant solution at some point.
Some reasons I feel positive about it:
(1) We get a lot of positive feedback on the Community Section, from a broad range of people.
For the initial test, we surveyed people on the Forum (more here), and got the following results:
(45 responses. On a scale of 1-7 where “1” was negative and “7″ was positive, 38% of responses put 6, 22% put 7, 13% put 5, 11% put 4, 7% put 3 and 2 (each), and 2% put 1.)
We’d also done some user interviews and gotten quick takes from people (again, a bit more here).
Since then, we’ve continued hearing from people who really appreciate it. At EAG and in other ~EA spaces, I’ve been, unprompted, told things like “I’m going on the Forum a lot more now because I see interesting and useful content” and “The Forum seems much better now.”
A number of people have also asked for ways to hide the section for themselves entirely.
(2) I feel that the Forum has gotten healthier.
I could try to point you to discussions that, before the change, I’d have expected would blow up in some unpleasant ways, I could pull out the number for how often moderators have to ban or warn someone per engagement hour in different periods, and I can point you to the “percent of karma that comes from downvotes” that I shared above, but I think this is mostly a subjective judgement, and factors in things that others might not agree with me on, like “how much does karma track what I think is more valuable?”
I also find it easier to read what I want to read on the Forum right now. I hadn’t expected a huge change in my own experience — I read a lot of Community posts! — but I actually quite like having the section.
(3) I was worried about various downsides, and they haven’t seemed to have gotten very bad — although that’s my main uncertainty.
Growth in total engagement hours has stalled. I don’t know if that will continue, and I don’t fully know why that is. Maybe people are tired, disenchanted with EA (or more precisely, maybe new folks aren’t finding EA as much), using Twitter/LW more, or too busy working on slowing down AI or the like. (I’ve seen data about other ~EA engagement metrics that suggest there’s at least some lull.) Or maybe we’ve discouraged authors of important and engaging Community content from posting on the Forum, and this has knock-on effects on non-Community engagement.
But we didn’t see a mass exodus, which I was somewhat worried about, and it’s not on a clear down trajectory. And evaluating all of this is pretty hard given the quick growth in 2022 due to FTX and WWOTF, and the crazy engagement levels in November.
I was really worried about the community/individual people missing out on important discussions. This is hard to notice — I shouldn’t expect to know what discussions I’m missing out on. But I was tracking some topics that I’d discussed with folks off the Forum that hadn’t made it on to the Forum, and I think most of them showed up during Strategy Fortnight, so I’m excited about more pushes/events like that (and other things listed here and elsewhere).
Here’s another bad and subjective datapoint: the average karma of some Community posts compared to some non-Community posts that are of a similar minimum quality/value (according to me) is quite similar (with Community posts winning a bit).[3]
There are other downsides, like:
Authors are sometimes sad that their posts are in the Community section
We’ve introduced some messiness/complexity to how the Forum is structured
Maybe we’ve implicitly made communicated something about what is and isn’t “valuable” that I don’t endorse
My personal approach here is to keep an eye on this, see if people are engaging to the extent that they endorse, and continue looking for better solutions.
I think this is almost certainly true — engagement with community posts went down —but there was also less big news to discuss, and that might have been the actual cause of the decrease.
Initially I was particularly worried that we would miss out on important conversations, and worried about our reasoning (I was worried that we were getting feedback from a heavily biased sample of people). I think I was also worried that there was something “special” about the Forum as it was at the time — I’ve now articulated this in terms of “we’re at relatively ~stable eliquibrium points on some important parameters and structural/external changes can mess with that stability” — that we would mess with by doing something big like this.
Here’s a bad baseline sanity check that I did at some point: (1) Take the posts shared in the Digest — which gives me a baseline of ~quality/value-to-readers (according to my own judgement). Note that I don’t filter strongly for karma in the Digest; if something gets a lot of karma and I’m not too sure about it, I’d maybe err on the side of including, but I don’t really use karma as a big factor. (I might also be more likely to miss low-karma good posts, and there are other reasons to share or not share something in the Digest, but let’s put that aside for now.) (2) I compared how much karma different types of posts were getting. The posts that I was putting as ~top 3 in the list order for the Digets had noticeably more karma, on average. Setting those aside, though, Community posts in the Digest got ~100 karma on average, and non-Community posts got ~80 karma on average. (Note that there’s a lot of variation, though.)
Regarding “I feel that the Forum has gotten healthier,” is that statement in comparison to pre-FTX levels of health?
It doesn’t surprise me that a series of scandals, as well as the various pro-reform/anti-reform posts, would trigger a rise in downvote percentage and in warnable events per hour in comparison to a calmer mix of topics. Without more, I would not be inclined to view the 4Q 22 / 1Q 23 trends as strong evidence of a generalized health decline rather than a reflection of the topic mix that needed to be discussed because of events that happened external to the Forum.
This is a good question. I was mostly not thinking of pre-FTX levels when I was writing that part of my response. I’m not sure if I think my claim holds on longer time scales! Things in the community have been calmer recently (outside of the Forum); I don’t know what would have happened had we not added the section (i.e on the Forum. whether things would have calmed down by themselves), nor what would happen if we removed the section now (and unfortunately we can’t e.g. A/B test this kind of thing, but maybe we should test in some other way at some point).
Aside (not related to what I was talking about in the “Forum has gotten healthier” section above): I agree that 4Q 22 / 1Q 23 aren’t representative in many ways, but a key underlying motivation for the Community section is giving posts that have a more niche audience a better chance by separating out posts that tend to get a lot of karma so they’re not “fighting for the same spots” (which are determined by ~karma). That phenomenon (and goal) is unaffected by recent events; ie. what we described in Karma overrates some topics; resulting issues and potential solutions should hold at all times. (On a skim, the top-by-karma posts of 2023, 2022, to some extent 2021 are Community posts. Curiously, this seems less true of 2020 but again more true of 2019.)
Wow this is a spectacularly detailed answer thanks!