I[1] sketch out a way to visualize the usefulness of posts (based on how many people read them and how useful the average reading is), and brief thoughts on how this interacts with Forum karma.
Image TL;DR:
Few posts are both extremely useful to the average reader and broadly relevant or accessible
This is a Draft Amnesty Day draft. That means it’s not polished, it’s probably not up to my standards, the ideas are not thought out, and I haven’t checked everything. I was explicitly encouraged[2] to post something unfinished!
I wrote this a while back and am sharing mostly unedited. It’s rough and probably wrong in important ways.
Commenting and feedback guidelines: I’m going with the default — please be nice. But constructive feedback is appreciated; please let me know what you think is wrong. Feedback on the structure of the argument is also appreciated.
Usefulness of a post is the product of the number of readers the post will have and the extent to which the average reader finds the post useful. (Sort of.[3])
Here’s a way to visualize it:
If you know exactly who you’re writing to, you’ll know exactly what will be interesting and useful to them, what information they already know, etc. Compare this to a case when you’re writing for a broader audience — you’ll need to make guesses about what is common knowledge and what isn’t, you won’t be able to tailor the information to the needs of the readers, etc. This weakness of broad-appeal posts means that they’ll probably be less useful to the average reader than more niche posts.
This — as well as experience interacting with lots of posts — leads me to believe that there’s an anti-correlation between breadth of appeal of a post, and usefulness of the post for a reader; nicher posts tend to be more useful to the average reader, and broader posts tend to be less useful to the average reader.
Visualized:
Few posts are both extremely useful to the average reader and broadly relevant or accessible
Some weaknesses of the model:
Averages are hard to track, interpret, and use when a distribution is long-tailed (because samples don’t get the long tails). [EXPLAIN?]
Some posts are only useful if multiple people see them?
[I was going to write more, never did]
Karma rewards broad-appeal more than niche-but-extremely-useful posts
If users could report exactly how useful a post was to them[4] in util-points or something, and they actually did that accurately and consistently, and the Forum took that, added the util-scores from all readers, and reported the overall usefulness of a post in the total-util-score, then we’d be tracking something like overall usefulness of the post.
But this system would be difficult to set up, and probably impossible to do well. For one thing, it is incredibly game-able (a user can report 100000 util-points on their friend’s post, for instance), and we can’t rely on the honesty of every user.
Instead, we have karma. We encourage people to upvote posts they found useful, so that others will see them. For posts that are accessible and of interest to everyone — posts that are on the right (and, generally, on the bottom) side of the chart above, karma works pretty well; lots of people will upvote the posts they found useful, and people won’t upvote (or will even downvote) the posts they thought were uninformative or misleading. For posts in the upper left-hand corner of the chart, however, we’ll see that a few people will find them extremely useful, but each of these people will only give a strong upvote at most. So the karma distribution of a bunch of posts that are approximately equally useful will look something like:
All posts shown below are approximately as useful. Their karma will be different, however; karma goes up as posts become more broad-appeal and less niche.
I basically think that’s fine, as long as everyone who needs to see a useful post on the left side does see it. Unfortunately, that does always not happen, as those posts will disappear from the Frontpage too fast (this is one of the reasons we’re working on subforums).
Niche vs. broad-appeal posts (& how this relates to usefulness/karma) (a sketch)
Meta-TL;DR:
I[1] sketch out a way to visualize the usefulness of posts (based on how many people read them and how useful the average reading is), and brief thoughts on how this interacts with Forum karma.
Image TL;DR:
This is a Draft Amnesty Day draft. That means it’s not polished, it’s probably not up to my standards, the ideas are not thought out, and I haven’t checked everything. I was explicitly encouraged[2] to post something unfinished!
I wrote this a while back and am sharing mostly unedited. It’s rough and probably wrong in important ways.
Usefulness of a post is the product of the number of readers the post will have and the extent to which the average reader finds the post useful. (Sort of.[3])
Here’s a way to visualize it:
If you know exactly who you’re writing to, you’ll know exactly what will be interesting and useful to them, what information they already know, etc. Compare this to a case when you’re writing for a broader audience — you’ll need to make guesses about what is common knowledge and what isn’t, you won’t be able to tailor the information to the needs of the readers, etc. This weakness of broad-appeal posts means that they’ll probably be less useful to the average reader than more niche posts.
This — as well as experience interacting with lots of posts — leads me to believe that there’s an anti-correlation between breadth of appeal of a post, and usefulness of the post for a reader; nicher posts tend to be more useful to the average reader, and broader posts tend to be less useful to the average reader.
Visualized:
Some weaknesses of the model:
Averages are hard to track, interpret, and use when a distribution is long-tailed (because samples don’t get the long tails). [EXPLAIN?]
Some posts are only useful if multiple people see them?
[I was going to write more, never did]
Karma rewards broad-appeal more than niche-but-extremely-useful posts
If users could report exactly how useful a post was to them[4] in util-points or something, and they actually did that accurately and consistently, and the Forum took that, added the util-scores from all readers, and reported the overall usefulness of a post in the total-util-score, then we’d be tracking something like overall usefulness of the post.
But this system would be difficult to set up, and probably impossible to do well. For one thing, it is incredibly game-able (a user can report 100000 util-points on their friend’s post, for instance), and we can’t rely on the honesty of every user.
Instead, we have karma. We encourage people to upvote posts they found useful, so that others will see them. For posts that are accessible and of interest to everyone — posts that are on the right (and, generally, on the bottom) side of the chart above, karma works pretty well; lots of people will upvote the posts they found useful, and people won’t upvote (or will even downvote) the posts they thought were uninformative or misleading. For posts in the upper left-hand corner of the chart, however, we’ll see that a few people will find them extremely useful, but each of these people will only give a strong upvote at most. So the karma distribution of a bunch of posts that are approximately equally useful will look something like:
I basically think that’s fine, as long as everyone who needs to see a useful post on the left side does see it. Unfortunately, that does always not happen, as those posts will disappear from the Frontpage too fast (this is one of the reasons we’re working on subforums).
I am speaking for myself, not my team or my employer.
By… myself. Let’s ignore that, I guess. I didn’t think through how this template would work if I used it. :)
Obviously, this model is imperfect. I note some failures [below].
(or how much value they think the world got from them reading the post, if the post was a request by the author that wasn’t useful to the reader)