Is the one-hour training module publicly available?
One might worry that training improves accuracy by motivating the trainees to take their jobs more seriously. Indeed it seems that the trained forecasters made more predictions per question than the control group, though they didn’t make more predictions overall. Nevertheless it seems that the training also had a direct effect on accuracy as well as this indirect effect.34
I could not find results like the ones in Table 4 in which the Brier scores are based only on the first answer that forecasters provide. Allowing forecasters to update their forecasts as frequently as they want (while reporting average daily Brier scores) plausibly gives an advantage to the forecasters who are willing to invest more time in their task.
The paper from which Table 4 is from stated that “Training was a significant predictor of average number of forecasts per question for year 1 and the number of forecasts per question was also significant predictor of accuracy (measured as mean standardized Brier score)”. Consider Table 10 in the paper that shows “Forecasts per question per user by year”. Notice that in year 3 the forecasters that got training made 4.27 forecasts per question, while forecasters that did not get training made only 1.90 forecasts per question. The paper includes additional statistical analyses related to this issue (unfortunately I don’t have the combination of time and background in statistics to understand them all).
In case it helps others decide whether or not to take the Superforecasting Fundamentals course, I’m reposting a brief message I sent to the CEA Slack workspace back in August 2017:
I took it a year or so ago. The course is very good, but also very basic: I clearly wasn’t the target audience, since I was already quite familiar with most of the content. I wouldn’t recommend it unless you don’t know anything about forecasting.
For sure, forecasters who devoted more effort to it tended to make more accurate predictions. It would be surprising if that wasn’t true!
I agree. But I am not referring to an extra effort that makes a person provide a better forecast (e.g. by spending more time looking for arguments), but rather an extra effort that allows one to improve their average daily Brier scores by simply using new public information that was not available when the question was first presented (e.g. new poll results).
I agree that this was probably a factor that contributed to the accuracy gains of people who made more frequent forecasts. It may even have been doing most of the work; I’m not sure.
Thank you for writing this.
Is the one-hour training module publicly available?
I could not find results like the ones in Table 4 in which the Brier scores are based only on the first answer that forecasters provide. Allowing forecasters to update their forecasts as frequently as they want (while reporting average daily Brier scores) plausibly gives an advantage to the forecasters who are willing to invest more time in their task.
The paper from which Table 4 is from stated that “Training was a significant predictor of average number of forecasts per question for year 1 and the number of forecasts per question was also significant predictor of accuracy (measured as mean standardized Brier score)”. Consider Table 10 in the paper that shows “Forecasts per question per user by year”. Notice that in year 3 the forecasters that got training made 4.27 forecasts per question, while forecasters that did not get training made only 1.90 forecasts per question. The paper includes additional statistical analyses related to this issue (unfortunately I don’t have the combination of time and background in statistics to understand them all).
The exact training module they used is probably not public, but they do have a training module on their website. It costs money though.
For sure, forecasters who devoted more effort to it tended to make more accurate predictions. It would be surprising if that wasn’t true!
In case it helps others decide whether or not to take the Superforecasting Fundamentals course, I’m reposting a brief message I sent to the CEA Slack workspace back in August 2017:
I agree. But I am not referring to an extra effort that makes a person provide a better forecast (e.g. by spending more time looking for arguments), but rather an extra effort that allows one to improve their average daily Brier scores by simply using new public information that was not available when the question was first presented (e.g. new poll results).
I agree that this was probably a factor that contributed to the accuracy gains of people who made more frequent forecasts. It may even have been doing most of the work; I’m not sure.