The most trivial one, and I guess the less glamorous one, is to defer to the community
Hmm one issue I have with deferring to the community is that even in situations where it’s individually epistemically valid, it seems to me to be bad for group epistemics to not form your own position. An analogy I use is a stock market where everybody only invests in index funds.
In the meantime, a few random thoughts. First, the index fund analogy suggests a self-correcting mechanism. Players defer to the community only to the degree that they expect it to track the truth more reliably than their individual judgment, given their time and ability constraints. As the reliability of the community prediction changes, in response to changes in the degree to which individual players defer to it, so will these players’s willingness to defer to the community.
Second, other things equal, I think it’s a desirable property of a prediction platform that it makes it rational for players to sometimes defer to the community. This could be seen as embodying the important and neglected truth that in many areas of life one can generally do better by deferring to society’s collective wisdom than by going with one’s individual opinion. Furthermore, it requires considerable ability to determine when and to what degree one should defer to others in any given case. In fact, this metacognitive skill of knowing how much more (or less) reliable other opinions are relative to one’s own seems like a core epistemic virtue, and one that can be assessed only if users are allowed to defer to others.
Finally, insofar as there are reasons for wanting players not to defer to the community, I think the appropriate response is to change the scoring function rather than to ask players to exercise self-restraint. As fellow forecaster Tom Adamczewski reminded me, the Metaculus Scoring System page describes one such possible change:
It’s easy to account for the average community prediction pc by adding a constant to each of these. For example, Slog(p,pc)=Slog(p)−Slog(pc). This way a player would get precisely zero points if they just go along with the community average.
Perhaps Metaculus could have two separate leaderboards: in addition to the current ranking, it could also display a ranking of players with the community component subtracted. These two rankings could be seen as measuring the quality of a player’s “credences” and “impressions”, respectively.
Hmm one issue I have with deferring to the community is that even in situations where it’s individually epistemically valid, it seems to me to be bad for group epistemics to not form your own position. An analogy I use is a stock market where everybody only invests in index funds.
Some Metaculus discussion here.
The link is broken; can you fix it?
In the meantime, a few random thoughts. First, the index fund analogy suggests a self-correcting mechanism. Players defer to the community only to the degree that they expect it to track the truth more reliably than their individual judgment, given their time and ability constraints. As the reliability of the community prediction changes, in response to changes in the degree to which individual players defer to it, so will these players’s willingness to defer to the community.
Second, other things equal, I think it’s a desirable property of a prediction platform that it makes it rational for players to sometimes defer to the community. This could be seen as embodying the important and neglected truth that in many areas of life one can generally do better by deferring to society’s collective wisdom than by going with one’s individual opinion. Furthermore, it requires considerable ability to determine when and to what degree one should defer to others in any given case. In fact, this metacognitive skill of knowing how much more (or less) reliable other opinions are relative to one’s own seems like a core epistemic virtue, and one that can be assessed only if users are allowed to defer to others.
Finally, insofar as there are reasons for wanting players not to defer to the community, I think the appropriate response is to change the scoring function rather than to ask players to exercise self-restraint. As fellow forecaster Tom Adamczewski reminded me, the Metaculus Scoring System page describes one such possible change:
Perhaps Metaculus could have two separate leaderboards: in addition to the current ranking, it could also display a ranking of players with the community component subtracted. These two rankings could be seen as measuring the quality of a player’s “credences” and “impressions”, respectively.