Second, identify the people who are consistently more accurate than everyone else — say, those in the top 0.1% for accuracy, for multiple years in a row. These are your “superforecasters.”
My worry is this: if enough people are trying to make forecasts, just by random chance you will get some people that attain whatever arbitrary amount of accuracy you desire (e.g. getting several forecasts in a row right for several years). How do we tell if a “superforecaster” is really that or they just got lucky until now? If the latter, their past success is not an indicator of future success.
My worry is this: if enough people are trying to make forecasts, just by random chance you will get some people that attain whatever arbitrary amount of accuracy you desire (e.g. getting several forecasts in a row right for several years). How do we tell if a “superforecaster” is really that or they just got lucky until now? If the latter, their past success is not an indicator of future success.