See here for the explanation of the potential problem.
To test it, we could do an experiment where some bot (or server-side process) randomly upvotes or downvotes new posts. We measure final karma after some fixed time, and see if that single vote snowballed.
I love that as a mechanism for measuring the effect of info cascades. It’s cheap, non-obtrusive, and certain. It’s from this study.
But I no longer like the solution for it I suggested in the Occlumency post. I think there are better ways using karma to mitigate info cascades and diversify what people read/discuss.
Thanks for this suggestion. Do you suspect that this is a big deal and have any intuition as to why?
My intuition is that it’s quite an interesting experiment, but seems unlikely to be a major influence on the Forum based on the fact that most posts with high karma are actually pretty decent.
“this effect was present in all parts of the distribution”
I was curious about this when I skimmed the paper, but I couldn’t find a breakdown of the impact of the random upvotes on, say, the top 5% highest upvoted posts. Do you know where to find that breakdown or what you mean with this?
Positively manipulated comments did receive higher ratings at all parts of the distribution, which means that they were also more likely to collect extremely high scores.
But since I don’t know what effect sizes they’re talking about at the top of the distribution, I don’t think this sentence is very informative.
Check if information cascades / social influence bias is a problem on EA Forum.
If it is, maybe we could implement Emrik’s idea to counter it, or some similar mechanism.
See here for the explanation of the potential problem.
To test it, we could do an experiment where some bot (or server-side process) randomly upvotes or downvotes new posts. We measure final karma after some fixed time, and see if that single vote snowballed.
relevant discussion
I love that as a mechanism for measuring the effect of info cascades. It’s cheap, non-obtrusive, and certain. It’s from this study.
But I no longer like the solution for it I suggested in the Occlumency post. I think there are better ways using karma to mitigate info cascades and diversify what people read/discuss.
Thanks for this suggestion. Do you suspect that this is a big deal and have any intuition as to why?
My intuition is that it’s quite an interesting experiment, but seems unlikely to be a major influence on the Forum based on the fact that most posts with high karma are actually pretty decent.
I don’t suspect it to be that bad. More like some noise added to each post’s score, and some posts not getting enough attention because of that.
In the reddit experiment single upvotes caused posts to have 25% higher mean score later (this effect was present in all parts of the distribution).
But the effect size was very dependent on the topic, so I’m curious how would that turn out for EA Forum.
I was curious about this when I skimmed the paper, but I couldn’t find a breakdown of the impact of the random upvotes on, say, the top 5% highest upvoted posts. Do you know where to find that breakdown or what you mean with this?
Ah, no, I just read the report of results on Wikipedia (that’s how they worded it). Hm, it’s strange if that’s not in the paper.
Ah, yeah, I read this on Wikipedia:
But since I don’t know what effect sizes they’re talking about at the top of the distribution, I don’t think this sentence is very informative.