Superforecasting in a nutshell

Link post

Let’s say you want to know how likely it is that an innovative new product will succeed, or that China will invade Taiwan in the next decade, or that a global pandemic will sweep the world — basically any question for which you can’t just use “predictive analytics,” because you don’t have a giant dataset you can plug into some statistical models like (say) Amazon can when predicting when your package will arrive.

Is it possible to produce reliable, accurate forecasts for such questions?

Somewhat amazingly, the answer appears to be “yes, if you do it right.”

Prediction markets are one promising method for doing this, but they’re mostly illegal in the US, and various implementation problems hinder their accuracy for now. Fortunately, there is also the “superforecasting” method, which is completely legal and very effective.

How does it work? The basic idea is very simple. The steps are:

First, bother to measure forecasting accuracy at all. Some industries care a lot about their forecasting accuracy and therefore measure it, for example hedge funds. But most forecasting-heavy industries do not even bother to measure their forecasting accuracy,^[1] for example the US intelligence community or philanthropy.^[2]
Second, identify the people who are consistently more accurate than everyone else — say, those in the top 0.1% for accuracy, for multiple years in a row. These are your “superforecasters.”
Finally, pose your forecasting questions to the superforecasters, and use an aggregate of their predictions.

Technically, the usual method is a bit more complicated than that,^[3] but these three simple steps are the core of the superforecasting method.

So, how well does this work?

A few years ago, the US intelligence community tested this method in a massive, rigorous forecasting tournament that included multiple randomized controlled trials and produced over a million forecasts on >500 geopolitical forecasting questions such as “Will there be a violent incident in the South China Sea in 2013 that kills at least one person?” This study found that:

This method produced forecasts that were very well-calibrated, in the sense that forecasts made with 20% confidence came true 20% of the time, forecasts made with 80% confidence came true 80% of the time, and so on. The method is not a crystal ball; it can’t tell you for sure whether China will invade Taiwan in the next decade, but if it tells you there’s a 10% chance, then you can be pretty confident the odds really are pretty close to 10%, and decide what policy is appropriate given that level of risk.^[4]
This method produced forecasts that were far more accurate than those of a typical forecaster or other approaches that were tried, and ~30% more accurate than intelligence community analysts who (unlike the superforecasters^[5]) had access to expensively-collected classified information and years of training in the geopolitical issues they were making forecasts about.^[6] Those are pretty amazing results! And from an unusually careful and rigorous study, no less!^[7]

So you might think the US intelligence community has eagerly adopted the superforecasting method, especially since the study was funded by the intelligence community, specifically for the purpose of discovering ways to improve the accuracy of US intelligence estimates used by policymakers to make tough decisions. Unfortunately, in my experience, very few people in the US intelligence and national security communities have even heard of these results, or even the term “superforecasting.”^[8]

A large organization such as the CIA or the Department of Defense has enough people, and makes enough forecasts, that it could implement all steps of the superforecasting method itself, if it wanted to. Smaller organizations, fortunately, can just contract already-verified superforecasters to make well-calibrated forecasts about the questions of greatest importance to their decision-making. In particular:

The superforecasters who out-predicted intelligence community analysts in the forecasting tournament described above are available to be contracted through Good Judgment Inc.
Another company, Hypermind, offers aggregated forecasts from “champion forecasters,” i.e. the most accurate forecasters across thousands of forecasting questions for corporate clients going back (in some cases) almost two decades.^[9]
Several other projects, for example Metaculus, are also beginning to identify forecasters with unusually high accuracy across hundreds of questions.

These companies each have their own strengths and weaknesses, and Open Philanthropy has commissioned forecasts from all three in the past couple years. If you work for a small organization that regularly makes important decisions based on what you expect to happen in the future, including what you expect to happen if you make one decision vs. another, I suggest you try them out. (All three offer “conditional” questions, e.g. “What’s the probability of outcome X if I make decision A, and what’s the probability of that same outcome if I instead make decision B?”)

If you work for an organization that is very large and/or works with highly sensitive information, for example the CIA, you should consider implementing the entire superforecasting process internally. (Though contracting one or more of the above organizations might be a good way to test the model cheaply before going all-in.)

↩︎
Except to the extent they’re able to use predictive analytics for particular questions for which they have rich data sets, which isn’t the subject of this post. I’m focused here on “general-purpose” forecasting methods, i.e. methods that can generate forecasts for any reasonably well-specified forecasting questions, and not just for those conducive to predictive analytics.
↩︎
In both example industries, there are a few exceptions, for example the intelligence community prediction market in the US intelligence community, or Open Philanthropy in philanthropy.
↩︎
E.g. for higher accuracy you might want to “team” the superforecasters in a certain way. See Superforecasting for details.
↩︎
By saying the odds “really are” close to 10%, I just mean that the 10%-confident predictions from this process are well-calibrated; I don’t mean to imply an interpretation of probability other than standard subjective Bayesianism.
↩︎
A few superforecasters had a geopolitics background of some kind, but most did not.
↩︎
For various accuracy comparisons, see Superforecasting, Mellers et al. (2014), and Goldstein et al. (2015). For high-level summaries of some of these results, see this page from Good Judgment Inc. and also AI Impacts (2019).
↩︎
One limitation of the currently available evidence is that we don’t know how effective superforecasting (or really, any judgment-based forecasting technique) is on longer-range forecasting questions (see here). I have a hunch that superforecasting is capable of producing forecasts on well-specified long-range questions that are well-calibrated even if they’re not very strong on “resolution” (explained here), but that’s just a hunch.
↩︎
For example, economist Tyler Cowen recently asked John Brennan (CIA Director until 2017): “You’re familiar with Philip Tetlock’s superforecasters project?” Brennan was not familiar.
↩︎
Technically, Hypermind’s usual aggregation algorithm also includes forecasts from other forecasters too, but gives much greater weight to the forecasts of the “champion forecasters.”