My take on this is: maybe this is fine actually, because, for precisely the reasons you said, high karma is a sign of high accessibility and high popularity...which is useful for users! If I see that a post has high karma, that’s a reliable signal to me that it’s both interesting and accessible to the general reader (i.e., to me). If all the highest-rated posts were highly-technical, long treatises on niche topics, even if they were very good quality, then high-karma wouldn’t be such a good signal that I would get something out of reading it, if that makes sense? So karma would then be a less good tool at nudging people to read stuff that they might actually enjoy/get something out of.
I do take your point that there can be a snowball effect where high-quality but high-effort-to-read posts can just get completely pushed off the frontpage before anyone has even seen them, while middling community posts just hang around forever, accumulating more karma. That is a problem.
I guess a question underlying all of this is ‘what is karma for?’ An implication of this post seems to be that karma should reflect quality, or how serious people think the issues are, all things considered. But I think that’s too big a responsibility to place on upvotes and downvotes. I don’t think the Forum norms say that you should use them this way (they say you can upvote if ‘you want others to see it’ and ‘generally like it’, not only if you think it’s objectively really important), and even if they did, I don’t think it’s reasonable to expect that people really would use them that way, because people don’t have that much brainspace to devote to “is this post really impactful/serious?” And the majority of EA Forum readers are never going to be qualified to say whether a niche post is high-quality or not, because they don’t have the expertise (they can say if they found it interesting, but things can be interesting, accessible, and also wrong).
Amber—I agree. EA Forum is for the EA community. Community-related posts that interest a high proportion of the community should get plenty of karma.
Treated ‘community-related posts’ as if they’re inferior to more specialist, technical posts seems to kinda miss the whole point of EA Forum—to be a nexus for discussion of EA topics of broad interest to the EA community.
My ideal self spends most of my EA Forum time reading technical posts about various cause areas, both to stay up to date on the ones I know a lot about and to learn more about the ones I’m less familiar with.
My actual self disproportionately reads Community posts because they take a lot less energy to read.
But I reserve almost all my upvotes for more technical posts to help nudge myself and others toward reading those ones more.
I didn’t vote, but your assertion that the “whole point” of the Forum is to discuss “topics of broad interest” seems overstated to me. That’s a purpose, but more technical/specialized discussion is also a purpose, and there are legitimate concerns that some of that is being drowned out.
My take on this is: maybe this is fine actually, because, for precisely the reasons you said, high karma is a sign of high accessibility and high popularity...which is useful for users! If I see that a post has high karma, that’s a reliable signal to me that it’s both interesting and accessible to the general reader (i.e., to me).
I don’t think there really is a regular reader, there are just a bunch of different readers with different interests. If I had to choose between these two scenarios --
Scenario A: Front page of 10 posts where all 10 are mediocre, but I can relate to and interact with all of them.
Scenario B: Front page of 10 posts where all are high quality, but I can only relate to and interact with 3 of them.
I would personally go with (B). Would you prefer (A), or do you think quality wouldn’t be compromised in that way, i.e. I’m presenting a false choice? (By “quality” I mean some combination of useful/interesting takeaways, good reasoning transparency, engages with previous work, treats an important subject matter, etc.)
I think this is a false choice, because I don’t think the top karma posts are usually mediocre. I think high karma is a good proxy for high quality, but low karma isn’t a good proxy for poor quality, because some low karma posts are (as OP said) good, but too technical or niche for general readership, or perhaps just not many people have seen it. In other words, I think there are lots of false negatives with karma but few false positives (is that metaphor at all clarifying, lol).
I do think it’s a shame if good non-community-drama posts never even get seen; on going onto the Forum, I’d love to see a front page featuring articles on a range of topics.
I think this is a false choice, because I don’t think the top karma posts are usually mediocre.
“Mediocre” was too strong—I should’ve written “high quality” in scenario A versus “excellent quality” in scenario B.
I think high karma is a good proxy for high quality, but low karma isn’t a good proxy for poor quality, because some low karma posts are (as OP said) good, but too technical or niche for general readership, or perhaps just not many people have seen it. In other words, I think there are lots of false negatives with karma but few false positives (is that metaphor at all clarifying, lol).
I mostly agree, but I think there’s some tendency for some of the best, most front-page-worthy posts to get stuck at “quite a lot of karma, but not the most” due to some combination of being (necessarily) long, specialized, technical, rigorous, difficult and/or dealing with a non-topical subject matter.
I do think it’s a shame if good non-community-drama posts never even get seen; on going onto the Forum, I’d love to see a front page featuring articles on a range of topics.
I’ll try to engage with the other things that you said, but I just want to clarify a specific claim first. You write:
I guess a question underlying all of this is ‘what is karma for?’ An implication of this post seems to be that karma should reflect quality, or how serious people think the issues are, all things considered.
I actually do not believe this. I think the primary/key point of karma is ordering the Frontpage & providing a signal of what to read (and ordering other pages, like when you’re exploring posts on a given topic). We don’t need to use only karma for ordering the Frontpage — and I really wish that more people used topic filters to customize their Frontpages, etc. — but I do think that’s a really important function of karma. This means that karma needs to reflect usefulness-of-reading-something to a certain extent. This post is about correcting one type of issue that arises given this use.
Note that we also correct in other ways. The Frontpage isn’t just a list of posts from all time sorted by (inflation-adjusted) karma, largely because people find it useful to read newer content (although not always), we have topic tags, etc.
So I don’t directly care about whether a post that’s 1000x more net useful than another post has 1000x (or even simply more) karma; I just want people to see the posts that will be most useful for them to engage with. (I think some people care quite a bit about karma correlating strongly with the impact of posts, and don’t think this is unreasonable as a desire, but I personally don’t think it’s that important. I do think there are other purposes to karma, like being a feedback mechanism to the authors, a sign of appreciation, etc.)
[Writing just for myself, not my employer or even my team. I am working on the Forum, and that’s probably hard to separate from my views on this topic— but this is a quickly-written comment, not something that I feedback on from the rest of the team, etc.]
Thanks for this!
My take on this is: maybe this is fine actually, because, for precisely the reasons you said, high karma is a sign of high accessibility and high popularity...which is useful for users! If I see that a post has high karma, that’s a reliable signal to me that it’s both interesting and accessible to the general reader (i.e., to me). If all the highest-rated posts were highly-technical, long treatises on niche topics, even if they were very good quality, then high-karma wouldn’t be such a good signal that I would get something out of reading it, if that makes sense? So karma would then be a less good tool at nudging people to read stuff that they might actually enjoy/get something out of.
I do take your point that there can be a snowball effect where high-quality but high-effort-to-read posts can just get completely pushed off the frontpage before anyone has even seen them, while middling community posts just hang around forever, accumulating more karma. That is a problem.
I guess a question underlying all of this is ‘what is karma for?’ An implication of this post seems to be that karma should reflect quality, or how serious people think the issues are, all things considered. But I think that’s too big a responsibility to place on upvotes and downvotes. I don’t think the Forum norms say that you should use them this way (they say you can upvote if ‘you want others to see it’ and ‘generally like it’, not only if you think it’s objectively really important), and even if they did, I don’t think it’s reasonable to expect that people really would use them that way, because people don’t have that much brainspace to devote to “is this post really impactful/serious?” And the majority of EA Forum readers are never going to be qualified to say whether a niche post is high-quality or not, because they don’t have the expertise (they can say if they found it interesting, but things can be interesting, accessible, and also wrong).
Amber—I agree. EA Forum is for the EA community. Community-related posts that interest a high proportion of the community should get plenty of karma.
Treated ‘community-related posts’ as if they’re inferior to more specialist, technical posts seems to kinda miss the whole point of EA Forum—to be a nexus for discussion of EA topics of broad interest to the EA community.
My ideal self spends most of my EA Forum time reading technical posts about various cause areas, both to stay up to date on the ones I know a lot about and to learn more about the ones I’m less familiar with.
My actual self disproportionately reads Community posts because they take a lot less energy to read.
But I reserve almost all my upvotes for more technical posts to help nudge myself and others toward reading those ones more.
PS—as always, for folks who disagree-voted on this, I’d appreciate seeing why specifically you disagree.
I didn’t vote, but your assertion that the “whole point” of the Forum is to discuss “topics of broad interest” seems overstated to me. That’s a purpose, but more technical/specialized discussion is also a purpose, and there are legitimate concerns that some of that is being drowned out.
Ok, fair enough. Imagine I said ‘a major point’ rather than ‘the whole point’.
I don’t think there really is a regular reader, there are just a bunch of different readers with different interests. If I had to choose between these two scenarios --
Scenario A: Front page of 10 posts where all 10 are mediocre, but I can relate to and interact with all of them.
Scenario B: Front page of 10 posts where all are high quality, but I can only relate to and interact with 3 of them.
I would personally go with (B). Would you prefer (A), or do you think quality wouldn’t be compromised in that way, i.e. I’m presenting a false choice? (By “quality” I mean some combination of useful/interesting takeaways, good reasoning transparency, engages with previous work, treats an important subject matter, etc.)
I think this is a false choice, because I don’t think the top karma posts are usually mediocre. I think high karma is a good proxy for high quality, but low karma isn’t a good proxy for poor quality, because some low karma posts are (as OP said) good, but too technical or niche for general readership, or perhaps just not many people have seen it. In other words, I think there are lots of false negatives with karma but few false positives (is that metaphor at all clarifying, lol).
I do think it’s a shame if good non-community-drama posts never even get seen; on going onto the Forum, I’d love to see a front page featuring articles on a range of topics.
“Mediocre” was too strong—I should’ve written “high quality” in scenario A versus “excellent quality” in scenario B.
I mostly agree, but I think there’s some tendency for some of the best, most front-page-worthy posts to get stuck at “quite a lot of karma, but not the most” due to some combination of being (necessarily) long, specialized, technical, rigorous, difficult and/or dealing with a non-topical subject matter.
I agree!
Thanks for this comment, Amber!
I’ll try to engage with the other things that you said, but I just want to clarify a specific claim first. You write:
I actually do not believe this. I think the primary/key point of karma is ordering the Frontpage & providing a signal of what to read (and ordering other pages, like when you’re exploring posts on a given topic). We don’t need to use only karma for ordering the Frontpage — and I really wish that more people used topic filters to customize their Frontpages, etc. — but I do think that’s a really important function of karma. This means that karma needs to reflect usefulness-of-reading-something to a certain extent. This post is about correcting one type of issue that arises given this use.
Note that we also correct in other ways. The Frontpage isn’t just a list of posts from all time sorted by (inflation-adjusted) karma, largely because people find it useful to read newer content (although not always), we have topic tags, etc.
So I don’t directly care about whether a post that’s 1000x more net useful than another post has 1000x (or even simply more) karma; I just want people to see the posts that will be most useful for them to engage with. (I think some people care quite a bit about karma correlating strongly with the impact of posts, and don’t think this is unreasonable as a desire, but I personally don’t think it’s that important. I do think there are other purposes to karma, like being a feedback mechanism to the authors, a sign of appreciation, etc.)
[Writing just for myself, not my employer or even my team. I am working on the Forum, and that’s probably hard to separate from my views on this topic— but this is a quickly-written comment, not something that I feedback on from the rest of the team, etc.]