Getting a feel for changes of karma and controversy in the EA Forum over time

Every time I come across an old post in the EA forum I wonder if the karma score is low because people did not get any value from it or if people really liked it and it only got a lower score because fewer people were around to upvote it at that time. Fortunately, you can send queries to the EA Forum API to get the data that could answer this question. In the following post I will describe how karma and post controversy (measured by the amount of votes a post got and the relation of downvotes to upvotes, more detailed explanation below) developed over time in the EA forum and provide a list of the best rated posts relative to the amount of activity in the forum. Before you read the post, take a moment to think about the following questions:

  • What is your best guess for the most well received post, relative to the amount of activity in the forum at the time it was posted?

  • Do you think the controversy of posts in the EA Forum has decreased or increased over time or stayed the same?

Karma

I got the metadata of the posts by simply sending a query to GraphiQL. Unfortunately, this apparently maxes out at 5000 posts, but this also seems to be around the same number of posts in the forum overall. Therefore, likely some random posts were excluded from this analysis, but the overall trend should be right. Still, I would be interested in how to circumvent this for future analysis. After bringing the answer of the query into a reasonable format I could calculate the average monthly karma of posts (you can find the data and code used here). If we plot the average monthly karma, we can see a steady increase over time. Interestingly, it seems to have stagnated a bit in the last ~ two years. This might be partly explained by the increase in the amount of posts, as 2020 saw more than three times as much posts as 2018 did.

We can use this mean monthly karma to scale all our posts by simply dividing the karma every post has by the mean karma of the month it was published. This allows us to determine which posts received the most praise in relation to how active the forum was at the time it was published. Here are the top 15 posts of the EA Forum for 2014 - March 2021 according to this metric:

At first glance this list seems pretty similar to the one you get if you simply order the forum by top post. However, the post that was best received in relation to the activity in the forum at that time is from 2014 and the posts on 11, 12 and 13 are all from 2015. Still, most of the best received posts are from 2020 and 2021. This is especially interesting, as we also saw in the earlier figure that the overall karma of the post stagnated at the same time. Apparently, we are getting more really good and more really unremarkable posts as the forum keeps on growing.

Controversy

As I had already extracted all the data, I thought it might also be interesting to take a look at how controversial posts are in the EA Forum. I simply calculated controversy the same way reddit does:

if else

This means a post is more controversial when it has a) many votes overall and b) the more those votes are split equally between the number of upvotes and downvotes.

The controversy score in the EA Forum is quite low, as most posts are either well received or don’t get much votes at all. The results here show that there are several distinct spikes in how controversial the average post in the forum was in 2016 and 2018. After that the controversy continuously decreased. As I only really started to engage with EA in 2018 I do not have a good overview of what might have caused these spikes. Maybe some people who have been around for longer can shed a light on this.

This analysis also provided me with a list of the most controversial posts of the EA Forum. I do not link them here directly, as those posts already created a lot of discussion and it might be counterproductive to start them again. However, they might be valuable for some and you can PM me if you want the list (or look in the repository I linked earlier). Many of those posts seem to handle sensitive issues like criticising organizations, norms or individual people, which might be difficult to handle in a way that only few will find disagreeable. Before looking at the list I would have expected that common partisan topics would be the most controversial. I am not sure what to make of this. Maybe it is this just natural for any community that critique is always difficult to swallow or those critiques might just not have been written carefully enough (if have not read most of them yet). This might have been the case as several of the most well received posts in the forum criticize ideas and institutions as well.

Overall, I think we can gain some interesting insights by taking a deeper look at the metadata of the forum. Especially the trend that controversy seemed to have decreased over time seems relevant to me and I am curious if this trend will continue.

Thanks to Magdalena Wache, Max Räuker and Ekaterina Illin for providing feedback on this post.