Error

Interesting side-finding: prediction markets seem notably worse than cleverly aggregated prediction pools (at least when liquidity is as low as in the play markets). Not many studies, but see Appendix A for what we’ve found.

Thank you for writing this overview! I think it’s very useful. A few notes on the famous “30%” claim:

• Part of the problem with fully understanding the performance of IC analysts is that much of the information about the tournaments and the ICPM is classified.

• What originally happened is that someone leaked info about ACE to David Ignatius, who then published it in his column. (The IC never denied the claim.^[1]) The document you cite is part of a case study by MITRE that’s been approved for public release.

One under-appreciated takeaway that you hint at is that prediction markets (rather than non-market aggregation platforms) are poorly suited to classified environments. Here’s a quote from a white paper I co-wrote last year:^[2]

“Prediction markets are especially challenging to implement in classified environments because classified markets will necessarily have large limitations on participation, requiring the use of algorithmic correctives to solve liquidity problems. Good liquidity, like that of a well-functioning stock market, is difficult to achieve in prediction markets like the ICPM, requiring prediction markets to have corrective tools like setting liquidity parameters and using automated market makers, which attempt to simulate efficient market behavior in electronic prediction markets.”

More broadly, I would like to push back a little against the idea that your point 3(a) ( whether supers outperform IC analysts) is really much evidence for or against 3 (whether supers outperform domain experts).

First, the IARPA tournaments asked a wide range of questions, but intelligence analysts tend to be specialized. If you’re looking at the ICPM, are you really looking at the performance of domain experts? Or are you looking at e.g. an expert on politics in the Horn of Africa trying to forecast the price of the Ruble? On the one hand, since participants self-selected which questions they answered, we might expect domain experts to stick to their domain. On the other, analysts might have seen it as a “game,” a “break,” or “professional development”—in short, an opportunity to try their hand something outside their expertise. The point is that we simply don’t know whether the ICPM really reflects “expert” opinion.

Second, I am inclined to believe that comparisons between IC analysts and supers may tell us more about the secrecy heuristic than about forecaster performance. From the same white paper:

“Experimental research on using secrecy as a heuristic for informational quality demonstrates that people tend to weigh secret information more heavily than publicly available information, viewing secret information as higher quality than public information.^[3] Secrecy does matter, especially in situations where information asymmetry exists, but a pervasive secrecy bias may negatively affect the accuracy of a classified crowd in some cases.”

I personally see much of the promise of forecasting platforms not as a tool for beating experts, but as a tool for identifying them more reliably (more reliably than by the usual signals, like a PhD).

1. ^

   Tetlock discusses this a bit in Chapter 4 of Superforecasting.

2. ^

   Keeping Score: A New Approach to Geopolitical Forecasting, https://​​global.upenn.edu/​​sites/​​default/​​files/​​perry-world-house/​​Keeping%20Score%20Forecasting%20White%20Paper.pdf.

3. ^

   Travers et al., “The Secrecy Heuristic,” https://​​www.jstor.org/​​stable/​​43785861.

David Manheim’s 2020 viewpoint prefigures some of the above, but goes further in questioning the superforecaster phenomenon (by reducing it to intelligence + open-mindedness + giving a damn).

Thanks for this, it’s really helpful! I find it very plausible to me that “generalist forecasters are the most accurate source for predictions on ~any question” has become too much of a community shibboleth. This is a useful correction.

Given how widely the “forecasters are better than experts!” meme has spread, point 3a seems particularly important to me (emphasis mine):

A common misconception is that superforecasters outperformed intelligence analysts by 30% [...] The forecaster prediction market performed about as well as the intelligence analyst prediction market [...] [85% confidence]

I would have found a couple more discussion paragraphs helpful. As written, it’s difficult for me to tell which studies you think are most influential in shaping the conclusions you lay out in the summary paragraph at the beginning of the post. The “Summary” section of the post isn’t actually summarizing the rest of the post; instead, that’s just where your discussion and conclusions are being presented.

I’m excited to potentially see more critical analysis of the forecasting literature! Plus ideas for new studies that can help identify the conditions under which forecasters are most accurate/​helpful.

These claims about Superforecasting are eye-catching. However, it’s difficult to draw any conclusions when most of the research cited doesn’t in fact include Superforecasters. In our view, it isn’t a matter of Superforecasters vs experts: the Boolean is “and” as much as possible to get the best results.

For those who are interested in taking a deeper dive into the peer-reviewed literature, though, take a look here:
https://​​goodjudgment.com/​​about/​​the-science-of-superforecasting/​​

Some of our work on combining forecasters and experts is here:
https://​​www.foreignaffairs.com/​​articles/​​united-states/​​2020-10-13/​​better-crystal-ball
https://​​warontherocks.com/​​2021/​​07/​​did-sino-american-relations-have-to-deteriorate-a-better-way-of-doing-counterfactual-thought-experiments/​​

Where’s the “delta” upvote when I need it? :)

In principle, I like the research question and the comparison above is probably the most you can make out from what is published. That said, it is the year 2022, capabilites and methodology have advanced enormously at least with those PM firms operating successfully in the commercial world markets. So it’s the proverbial comparing apples and oranges on several dimensions to talk about how “prediction markets” (sic) perform for whatever. Different platform implementations have very different capabilities suited to very different tasks. Moreover, like any advanced tool, practical application of the more advanced PM platforms need a high degree of methodic knowhow on how to use their specific capabilities—based on real experience of what works and what does’t.

So here’s a potentially fatal flaw in this analysis:

You write, “Goldstein et al showed that superforecasters outperformed the intelligence community....”

But the Goldstein paper was not about the Superforecasters. Your analysis, footnote 4, says, “‘All Surveys Logit’ takes the most recent forecasts from a selection of individuals in GJP’s survey elicitation condition....”

Thousands of individuals were in GJP’s survey elicitation condition, of whom only fraction (a few dozen) were Superforecasters.

So Goldstein did not find that “superforecasters outperformed the intelligence community”; rather, he found that [thousands of regular forecasters + a few dozen Superforecasters] outperformed the intelligence community. That’s an even lower bar.

Please check for yourself. All GJP data is publicly-available here: https://​​dataverse.harvard.edu/​​dataverse/​​gjp.

Asking forecasters about their expertise, or about their thinking patterns is not useful in terms of predicting which individuals will prove consistently accurate. Examining their behaviors, such as belief updating patterns, as well as their psychometric scores related to fluid intelligence offer more promising avenues. Arguably the most impressive performance in our study was for registered intersubjective measures, which rely on comparisons between individual and consensus estimates. Such measures proved valid as predictors of relative accuracy.

From the conclusion of this new paper https://​​psyarxiv.com/​​rm49a/​​

We checked to see if Tetlock’s 2005 book had anything to tell us about our question.

Despite my own and others’ recollection that it shows that top generalists match experts, the main RFE experiment turns out to compare PhD area experts against PhD experts outside their precise area. The confusion arises because he uses the word “dilettante” for these latter experts, and doesn’t define this until the last appendix.

Be sure to check out the vast chasm between the experts and random undergrads.

One nice little study which was out of scope: ClearerThinking vs Good Judgment Inc vs MTurk on Trump policies. (This has been advertised as superforecasters vs experts, but it isn’t.)

this might be due to a change on EA forum since you initially posted this post, but the left and right columns of the table of studies are quite unreadable for me on desktop, on both Chrome and Edge. see screenshot for what it looks like from my end. is there any other format I can read this post in?

Our recent submission, “Training experts to be forecasters”, to the cause exploration prize may be of interest (I certainly found this post interesting as a justification for some of the ideas we experiment with).

https://​​forum.effectivealtruism.org/​​posts/​​WFbf2d4LHjgvWJCus/​​cause-exploration-prizes-training-experts-to-be-forecasters

Can someone clarify these statements from Summary (3a)? They seem to be at odds....

A: “A common misconception is that superforecasters outperformed intelligence analysts by 30%.”

B: “Instead: Goldstein et al showed that superforecasters outperformed the intelligence community...”[then a table listing the ICPM MMDB as 0.23 versus the GJP Best MMDB as 0.15].

--> Wouldn’t that be 34% better?

Curious: You say the 2015 Seth Goldstein “unpublished document” was “used to justify the famous ‘Supers are 30% better than the CIA’ claim.”

But that was reported two years earlier, in 2013: https://​​www.washingtonpost.com/​​opinions/​​david-ignatius-more-chatter-than-needed/​​2013/​​11/​​01/​​1194a984-425a-11e3-a624-41d661b0bb78_story.html.

So how was the 2015 paper the justification?