Thanks for this, it’s really helpful! I find it very plausible to me that “generalist forecasters are the most accurate source for predictions on ~any question” has become too much of a community shibboleth. This is a useful correction.
Given how widely the “forecasters are better than experts!” meme has spread, point 3a seems particularly important to me (emphasis mine):
A common misconception is that superforecasters outperformed intelligence analysts by 30% [...] The forecaster prediction market performed about as well as the intelligence analyst prediction market [...] [85% confidence]
I would have found a couple more discussion paragraphs helpful. As written, it’s difficult for me to tell which studies you think are most influential in shaping the conclusions you lay out in the summary paragraph at the beginning of the post. The “Summary” section of the post isn’t actually summarizing the rest of the post; instead, that’s just where your discussion and conclusions are being presented.
I’m excited to potentially see more critical analysis of the forecasting literature! Plus ideas for new studies that can help identify the conditions under which forecasters are most accurate/helpful.
Thanks for this, it’s really helpful! I find it very plausible to me that “generalist forecasters are the most accurate source for predictions on ~any question” has become too much of a community shibboleth. This is a useful correction.
Given how widely the “forecasters are better than experts!” meme has spread, point 3a seems particularly important to me (emphasis mine):
I would have found a couple more discussion paragraphs helpful. As written, it’s difficult for me to tell which studies you think are most influential in shaping the conclusions you lay out in the summary paragraph at the beginning of the post. The “Summary” section of the post isn’t actually summarizing the rest of the post; instead, that’s just where your discussion and conclusions are being presented.
I’m excited to potentially see more critical analysis of the forecasting literature! Plus ideas for new studies that can help identify the conditions under which forecasters are most accurate/helpful.
Renamed the summary section, thanks
Thank you! We might consider editing the summary. This particular point is mostly supported by our takes on Goldstein et al (2015) and by Appendix A.