Agreed. However, the higher the uncertainty of each forecaster, the greater the variation across forecasters’ best guesses will tend to be, and therefore the wider the 95 % confidence interval of the median?
That’s true for means, where we can simply apply the CLT. However, this is a median. Stack Exchange suggests that only the density at the median matters. That means a very peaky distribution, even with wide tails will still lead to a small confidence interval. Due to forecasters rounding answers, the distribution is plausibly pretty peaky.
The confidence interval width still goes with sample size as 1/√n. There’s a decent sample size here of superforecasters.
Intuitively: you don’t care how spread out the tails are for the median, only how much of the mass is in the tails.
I was thinking as follows. The width of the confidence interval of quantile q for a confidence level alpha isF(q2 = q + z*(q*(1 - q)/n)^0.5) - F(q1 = q—z*(q*(1 - q)/n)^0.5), where P(z ⇐ X | X ~ N(0, 1)) = 1 - (1 - alpha)/2. A greater variation across estimates does not change q1 nor q2, but it increases the width F(q2) - F(q1).
That being said, I have to concede that what we care about is the mass in the tails, not the tails of the median. So one should care about the difference between e.g. the 97.5th and 2.5th percentile, not F(q2) - F(q1).
The confidence intervals are for what the median person in the class would forecast. Each forecaster’s uncertainty is not reflected.
Hi Joshua,
Agreed. However, the higher the uncertainty of each forecaster, the greater the variation across forecasters’ best guesses will tend to be, and therefore the wider the 95 % confidence interval of the median?
That’s true for means, where we can simply apply the CLT. However, this is a median. Stack Exchange suggests that only the density at the median matters. That means a very peaky distribution, even with wide tails will still lead to a small confidence interval. Due to forecasters rounding answers, the distribution is plausibly pretty peaky.
The confidence interval width still goes with sample size as 1/√n. There’s a decent sample size here of superforecasters.
Intuitively: you don’t care how spread out the tails are for the median, only how much of the mass is in the tails.
Thanks for following up!
I was thinking as follows. The width of the confidence interval of quantile q for a confidence level alpha is F(q2 = q + z*(q*(1 - q)/n)^0.5) - F(q1 = q—z*(q*(1 - q)/n)^0.5), where P(z ⇐ X | X ~ N(0, 1)) = 1 - (1 - alpha)/2. A greater variation across estimates does not change q1 nor q2, but it increases the width F(q2) - F(q1).
That being said, I have to concede that what we care about is the mass in the tails, not the tails of the median. So one should care about the difference between e.g. the 97.5th and 2.5th percentile, not F(q2) - F(q1).