The organization-wide Brier score (measuring both calibration and resolution) is .217, which is somewhat better than chance (.250). This requires careful interpretation, but in short we think that our reasonably good Brier score is mostly driven by good calibration, while resolution has more room for improvement (but this may not be worth the effort). [more]
Another explanation for the low resolution, besides the limited time you spend on the forecasts, might be that you chose questions that you are most uncertain about (i.e. that you are around 50% certain about resolving positively), right?
This is something I noticed when making my own forecasts. To remove this bias I sometimes use a dice to chose the number for questions like
By Jan 1, 2018,the grantee will have staff working in at least [insert random number from a reasonable range] European countries
Our forecasting questions are indeed maximally uncertain in some absolute sense because our base rate is ~50%, but it may also be the case that they’re particularly uncertain to the person making the prediction as you suggest.
Thanks for sharing, super interesting!
Another explanation for the low resolution, besides the limited time you spend on the forecasts, might be that you chose questions that you are most uncertain about (i.e. that you are around 50% certain about resolving positively), right?
This is something I noticed when making my own forecasts. To remove this bias I sometimes use a dice to chose the number for questions like
Interesting, thanks for sharing that trick!
Our forecasting questions are indeed maximally uncertain in some absolute sense because our base rate is ~50%, but it may also be the case that they’re particularly uncertain to the person making the prediction as you suggest.