Executive summary: The post investigates methods for estimating the precision of a set of probability forecasts, in order to determine at what level of granularity the forecasts add information. It finds issues with rounding-based approaches and instead proposes perturbing forecasts by adding noise.
Key points:
Precision should be measured in bits, not probabilities, using the logit function.
Rounding probabilities to bins worsens scores in unintuitive ways on small datasets.
Adding uniform noise to log-odds probabilities produces smoothly declining scores.
Binary search finds the point where score worsens faster but may underestimate.
Forecasting datasets have similar precision: ~1.2 bits for Metaculus, ~1.3 bits for PredictionBook.
Knowing precision allows sensitivity analysis and avoids false precision claims.
Noising perturbations seem better than rounding but more analysis is needed.
This comment was auto-generated by the EA Forum Team. Feel free to point out issues with this summary by replying to the comment, andcontact us if you have feedback.
Executive summary: The post investigates methods for estimating the precision of a set of probability forecasts, in order to determine at what level of granularity the forecasts add information. It finds issues with rounding-based approaches and instead proposes perturbing forecasts by adding noise.
Key points:
Precision should be measured in bits, not probabilities, using the logit function.
Rounding probabilities to bins worsens scores in unintuitive ways on small datasets.
Adding uniform noise to log-odds probabilities produces smoothly declining scores.
Binary search finds the point where score worsens faster but may underestimate.
Forecasting datasets have similar precision: ~1.2 bits for Metaculus, ~1.3 bits for PredictionBook.
Knowing precision allows sensitivity analysis and avoids false precision claims.
Noising perturbations seem better than rounding but more analysis is needed.
This comment was auto-generated by the EA Forum Team. Feel free to point out issues with this summary by replying to the comment, and contact us if you have feedback.