Nuño might have additional thoughts, but I have a couple of concerns here.
It’s possible to run into the following issues even (/especially) when people are “playing perfectly”, at least in terms of trying to maximise points:
Correctly making the same forecast as the crowd doesn’t have 0 value, as it makes the crowd prediction more robust to future bad predictions, however it does not earn you any points.
You are very strongly disincentivised from posting evidence that the crowd is wrong when you are in fact correct to disagree with the crowd.
Somewhat seperately, I think this particular scoring system risks people making some bad decisions from both a points perspective and a good forecasting perspective:
There’s a fine line between people understanding “I get more points if I am correct and the crowd is wrong” and “I get more points if I disagree with the crowd”, with the second line of reasoning potentially leading to people updating their forecasts away from the median in order to maximise their points potential.
Given how good crowds tend to be, most of the time when you think the crowd is very wrong, you are the person who is very wrong.
Edit: I re-ordered the points above in order to try to be more clear, not all of them are concerned about exactly the same thing.
There’s a fine line between people understanding “I get more points if I am correct and the crowd is wrong” and “I get more points if I disagree with the crowd”, with the second line of reasoning potentially leading to people updating their forecasts away from the median in order to maximise their points potential.
This shouldn’t be a problem in the limit with a proper scoring rule.
Nuño might have additional thoughts, but I have a couple of concerns here.
It’s possible to run into the following issues even (/especially) when people are “playing perfectly”, at least in terms of trying to maximise points:
Correctly making the same forecast as the crowd doesn’t have 0 value, as it makes the crowd prediction more robust to future bad predictions, however it does not earn you any points.
You are very strongly disincentivised from posting evidence that the crowd is wrong when you are in fact correct to disagree with the crowd.
Somewhat seperately, I think this particular scoring system risks people making some bad decisions from both a points perspective and a good forecasting perspective:
There’s a fine line between people understanding “I get more points if I am correct and the crowd is wrong” and “I get more points if I disagree with the crowd”, with the second line of reasoning potentially leading to people updating their forecasts away from the median in order to maximise their points potential.
Given how good crowds tend to be, most of the time when you think the crowd is very wrong, you are the person who is very wrong.
Edit: I re-ordered the points above in order to try to be more clear, not all of them are concerned about exactly the same thing.
This shouldn’t be a problem in the limit with a proper scoring rule.