I would consider something to reduce the karma users can get from commenting on controversial posts. Right now it seems easy to get very high scores by making not really that great comments in such places.
As an example, I think this comment I made is decent. It makes a true and relevant point that no-one else had mentioned . But it’s not great; the topic of that thread is not that important, and the all the comments in it, let alone mine alone, do not resolve the issue. Most importantly, that comment is definitely not over 50% as good as this article I wrote. I would say the article is at least a thousand times more important, and took at least a thousand times longer to write.
I’m not sure how exactly you would do this though, as all the most obvious methods have significant drawbacks.
Maybe turn off strong voting in comments or even comment karma from counting to users’ total karma in such posts? How do we decide which posts to consider controversial, though? Just the mods do it (they kept object-level election posts in the personal blog)?
An approach some forum use is the ratio of up and downvotes: −38+40 is not the same as +2 ! This allows you to have a smooth measure of the degree of controversy rather than a binary classification.
One underlying reason your comment got a lot of upvotes was because the post was viewed many times. Controversy leads to pageviews. Arguably “net upvotes” is an OK metric for post quality (where popularity is important) whereas “net upvotes”/”pageviews” might make more sense for comments.
Side-issue: isn’t Karma from posts weighted at 10x compared to Karma in comments? Or at least, I think it once was. And that would help a bit in this particular instance.
We no longer weigh frontpage posts 10x, though we might want to reinstitute some kind of weighing again. I think the 10x was historically too much, and made it so that by far the primary determinant of who had how much karma was how many frontpage posts you had, which felt like it undervalued comments, but it’s pretty plausible (and even likely to me) that the current system is now too skewed in the other direction.
My current relationship towards karma is something like: The point of karma for comments is to provide local information in a thread about a mixture of importance, quality and readership, and it’s pretty hard to disentangle those without making the system much more complex. Overall the karma of a post is a pretty good guess on how many people will want to read it, so it makes sense to use it for some recommendation systems, but the karma of comments feel a lot more noisy to me. As a long-term reward I think we shouldn’t really rely on karma at all and instead use systems like the LessWrong review to establish in a much more considered way which posts were actually good.
We’ve also deemphasized how much karma someone has on the site quite a bit because I don’t want to create the impression that it’s at all a robust measure of the quality of someone’s contributions. So, for example, we no longer have karma leaderboards.
A topic could be controversial in society but the votes could still go mostly one way on the EA Forum itself, though. For example, I wouldn’t be surprised if Democrat-favouring election posts were not scored as very controversial on the EA Forum, given the political leanings of EA. Do we also want to consider posts on controversial topics more broadly?
I would consider something to reduce the karma users can get from commenting on controversial posts. Right now it seems easy to get very high scores by making not really that great comments in such places.
As an example, I think this comment I made is decent. It makes a true and relevant point that no-one else had mentioned . But it’s not great; the topic of that thread is not that important, and the all the comments in it, let alone mine alone, do not resolve the issue. Most importantly, that comment is definitely not over 50% as good as this article I wrote. I would say the article is at least a thousand times more important, and took at least a thousand times longer to write.
I’m not sure how exactly you would do this though, as all the most obvious methods have significant drawbacks.
Maybe turn off strong voting in comments or even comment karma from counting to users’ total karma in such posts? How do we decide which posts to consider controversial, though? Just the mods do it (they kept object-level election posts in the personal blog)?
An approach some forum use is the ratio of up and downvotes: −38+40 is not the same as +2 ! This allows you to have a smooth measure of the degree of controversy rather than a binary classification.
One underlying reason your comment got a lot of upvotes was because the post was viewed many times. Controversy leads to pageviews. Arguably “net upvotes” is an OK metric for post quality (where popularity is important) whereas “net upvotes”/”pageviews” might make more sense for comments.
Side-issue: isn’t Karma from posts weighted at 10x compared to Karma in comments? Or at least, I think it once was. And that would help a bit in this particular instance.
We no longer weigh frontpage posts 10x, though we might want to reinstitute some kind of weighing again. I think the 10x was historically too much, and made it so that by far the primary determinant of who had how much karma was how many frontpage posts you had, which felt like it undervalued comments, but it’s pretty plausible (and even likely to me) that the current system is now too skewed in the other direction.
My current relationship towards karma is something like: The point of karma for comments is to provide local information in a thread about a mixture of importance, quality and readership, and it’s pretty hard to disentangle those without making the system much more complex. Overall the karma of a post is a pretty good guess on how many people will want to read it, so it makes sense to use it for some recommendation systems, but the karma of comments feel a lot more noisy to me. As a long-term reward I think we shouldn’t really rely on karma at all and instead use systems like the LessWrong review to establish in a much more considered way which posts were actually good.
We’ve also deemphasized how much karma someone has on the site quite a bit because I don’t want to create the impression that it’s at all a robust measure of the quality of someone’s contributions. So, for example, we no longer have karma leaderboards.
A topic could be controversial in society but the votes could still go mostly one way on the EA Forum itself, though. For example, I wouldn’t be surprised if Democrat-favouring election posts were not scored as very controversial on the EA Forum, given the political leanings of EA. Do we also want to consider posts on controversial topics more broadly?
Just saw this and wanted to add my strong agreement that (a) this is a problem, and (b) I don’t know how to fix it.
My lizard brain has definitely learned that commenting on high-controversy posts gets me lots of karma, and I don’t like it.