It’s not clear to me that “fitting a Beta distribution and using one of it’s statistics” is different from just taking the mean of the probabilities.
I fitting a beta distribution to Metaculus forecasts and looked at:
Median forecast
Mean forecast
Mean log-odds / Geometric mean of odds
Fitted beta median
Fitted beta mean
Scattering these 5 values against each other I get:
We can see fitted values are closely aligned with the mean and mean-log-odds, but not with the median. (Unsurprising when you consider the ~parametric formula for the mean / median).
The performance is as follows:
brier
log_score
questions
geo_mean_odds_weighted
0.116
0.37
856
beta_median_weighted
0.118
0.378
856
median_weighted
0.121
0.38
856
mean_weighted
0.122
0.391
856
beta_mean_weighted
0.123
0.396
856
My intuition for what is going on here is that the beta-median is an extremized form of the beta-mean / mean, which is an improvement
Looking more recently (as the community became more calibrated), the beta-median’s performance edge seems to have reduced:
It’s not clear to me that “fitting a Beta distribution and using one of it’s statistics” is different from just taking the mean of the probabilities.
I fitting a beta distribution to Metaculus forecasts and looked at:
Median forecast
Mean forecast
Mean log-odds / Geometric mean of odds
Fitted beta median
Fitted beta mean
Scattering these 5 values against each other I get:
We can see fitted values are closely aligned with the mean and mean-log-odds, but not with the median. (Unsurprising when you consider the ~parametric formula for the mean / median).
The performance is as follows:
My intuition for what is going on here is that the beta-median is an extremized form of the beta-mean / mean, which is an improvement
Looking more recently (as the community became more calibrated), the beta-median’s performance edge seems to have reduced: