You can get a sense for these sorts of numbers just by looking at a binomial distribution.
e.g., Suppose that there are n events which each independently have a 45% chance of happening, and a noisy/biased/inaccurate forecaster assigns 55% to each of them.
Then the noisy forecaster will look more accurate than an accurate forecaster (who always says 45%) if >50% of the events happen, and you can use the binomial distribution to see how likely that is to happen for different values of n. For example, according to this binomial calculator, with n=51 there is a 24% chance that at least 26⁄51 of the p=.45 events will resolve as True, and with n=201 there is a 8% chance (I’m picking odd numbers for n so that there aren’t ties).
With slightly more complicated math you can look at statistical significance, and you can repeat for values other than trueprob=45% & forecast=55%.
You can get a sense for these sorts of numbers just by looking at a binomial distribution.
e.g., Suppose that there are n events which each independently have a 45% chance of happening, and a noisy/biased/inaccurate forecaster assigns 55% to each of them.
Then the noisy forecaster will look more accurate than an accurate forecaster (who always says 45%) if >50% of the events happen, and you can use the binomial distribution to see how likely that is to happen for different values of n. For example, according to this binomial calculator, with n=51 there is a 24% chance that at least 26⁄51 of the p=.45 events will resolve as True, and with n=201 there is a 8% chance (I’m picking odd numbers for n so that there aren’t ties).
With slightly more complicated math you can look at statistical significance, and you can repeat for values other than trueprob=45% & forecast=55%.
Good comment, thank you!