The evidence suggests Brier score improvement after reaching 10 predictors is slow, and in practice I think I will now have a very similar level of confidence in a prediction if it has 10 or 30 predictors, whereas before I might have more easily dismissed the former.
So do you think this means it’d be better if forecasters “spread themselves more evenly” across questions—like moving a bit in the direction of forecasting on questions that so far have fewer unique predictors (especially those that have <10), relative to those which have more unique predictors?
Do you think there’s a good way for this to be encouraged/incentivised? Some ideas that come to mind:
Just highlight that that is a way for forecasters to more efficiently provide the public good of more accurate aggregate forecasts (holding other factors constant)
Though in practice questions with more predictors might tend to be more important and so a smaller accuracy boost there could be better than a larger accuracy boost elsewhere
Metaculus could make an “achievement” or show stats for how often people have forecasted on questions that have few unique predictors so far, maybe excluding questions people create themselves
But this might mostly incentivise jumping on questions really fast, rather than jumping on questions that would otherwise remain neglected
And it again runs into the issue that questions with more predictors might tend to be more important
I think one issue with the current system is it has the opposite incentives—you get more points for predicting on popular questions. I don’t know that going all the way in the opposite direction makes sense but reducing this seems good to me.
Perhaps there’s another clever mechanism they could implement, but I’d guess that’s the lowest hanging fruit.
(Also just realised that I should flag that I was implicitly assuming the patterns you found are causal, not correlational, but we should remain uncertain about that. But it does seem fairly likely there’s a fairly large causal component, so I still basically endorse the above comment.)
Thanks for this post!
So do you think this means it’d be better if forecasters “spread themselves more evenly” across questions—like moving a bit in the direction of forecasting on questions that so far have fewer unique predictors (especially those that have <10), relative to those which have more unique predictors?
Do you think there’s a good way for this to be encouraged/incentivised? Some ideas that come to mind:
Just highlight that that is a way for forecasters to more efficiently provide the public good of more accurate aggregate forecasts (holding other factors constant)
Though in practice questions with more predictors might tend to be more important and so a smaller accuracy boost there could be better than a larger accuracy boost elsewhere
Metaculus could make an “achievement” or show stats for how often people have forecasted on questions that have few unique predictors so far, maybe excluding questions people create themselves
But this might mostly incentivise jumping on questions really fast, rather than jumping on questions that would otherwise remain neglected
And it again runs into the issue that questions with more predictors might tend to be more important
Tweak the scoring rule to reward this behaviour
But this again has the above issues
I think one issue with the current system is it has the opposite incentives—you get more points for predicting on popular questions. I don’t know that going all the way in the opposite direction makes sense but reducing this seems good to me.
Perhaps there’s another clever mechanism they could implement, but I’d guess that’s the lowest hanging fruit.
Interesting, thanks.
(Also just realised that I should flag that I was implicitly assuming the patterns you found are causal, not correlational, but we should remain uncertain about that. But it does seem fairly likely there’s a fairly large causal component, so I still basically endorse the above comment.)