Forecasting and estimation are important tools for improving the future, because good forecasts and estimates can help us appropriately plan interventions and assess risks. Over the past several decades there has been significant research and investment in forecasting and estimation techniques, tools, and organizations. This continues to be an area of investment for improving our ability to make good decisions.
The State of Forecasting within EA
There are some major branches of forecasting within the EA movement:
Personal forecasting—individuals forecasting to improve their decision-making or for status and personal enjoyment
Forecasting consultancies—EA organisations pay for forecasting by groups of top forecasters or Metaculus
Forecasting research—Academic research on the accuracy of forecasting and how to do it better (eg by FRI)
Institutional forecasting—Seeking for forecasting to be used inside government and large institutions
Forecasting technology—Building new tools to quantify with (eg Squiggle)
These areas in more depth
Institutional forecasting.
Forecasting in institutions can range from predicting broad metrics to specific outcomes based on specific decisions. There can often be problems with buy-in from key stakeholders, who either see this as an unnecessary step or are concerned for their own status.
Forecasting Techniques
Forecasting is hard but many top forecasters use common techniques. This suggests that forecasting is a skill that can be learnt and practised.
Base rates
Reference Class Forecasting on Wikipedia
Suppose we are trying to find the probability that an event will occur within the next 5 years. One good place to start is by asking “of all similar time periods, what fraction of the time does this event occur?”. This is the base rate.
If we want to know the probability that Joe Biden is President of the United States on Nov. 1st, 2024, we could ask
What fraction of presidential terms are fully completed (last all 4 years)? The answer to this is 49 out of the 58 total terms, or around 84%.
On the other hand, we know that Biden has already made it through 288 days of his term. If we remove the 5 presidents who left office before that, there are 49 out of 53 or around 92%.
But alternately, Joe Biden is pretty old (78 to be exact). If we look up death rate per year in actuarial tables, it’s around 5.1% per year, so this leaves him with a ~15% chance of death or a 85% chance of surviving his term.
These are all examples of using base rates. [These examples are taken from Base Rates and Reference Classes by jsteinhardt.]
Base rates represent the outside view for a given question. They are a good place to start but can often be improved on by updating the probability according to an inside view.
Note that there are often several reference classes we could use, each implying a different base rate. The problem of deciding which class to use is known as the reference class problem.
Calibration training
A forecaster is said to be calibrated if the events they say have a X% chance of happening, happen X% of the time.
Most people are overconfident. When they say an event has a 99% chance of happening, often the events happen much less frequently than that.
This natural overconfidence can be corrected with calibration training. In calibration training, you are asked to answer a set of factual questions, assigning a probability to each of your answers.
A list of calibration training exercises can be found here.
Question decomposition
Much like Fermi estimation, questions about future events can often be decomposed into many different questions, these questions can be answered, and the answers to these questions can be used to reconstruct an answer to the original question.
Suppose you are interested in whether AI will cause a catastrophe by 2100. For AI to cause such an event, several things need to be true: (1) it needs to be possible to build advanced AI with agentic planning and strategic awareness by 2100, (2) there need to be strong incentives to apply such a system, (3) it needs to be difficult to align such a system should it be deployed, (4) a deployed and unaligned AI would act in unintended and high-impact power seeking ways causing trillions of dollars in damage, (5) of these consequences will result in the permanent disempowerment of all humanity and (6) this disempowerment will constitute an existential catastrophe. Taking the probabilities that Eli Lifland assigned to each question gives a 80%, 85%, 75%, 90%, 80% and 95% chance of events 1 through 6 respectively. Since each event is conditional on the ones before it, we can find the probability of the original question by multiplying all the probabilities together. This gives Eli Lifland a probability of existential risk from misaligned AI before 2100 to be approximately 35%. For more detail see Eli’s original post here.
Decomposing questions into their constituent parts, assigning probabilities to these sub-questions, and combining these probabilities to answer the original questions is believed to improve forecasts. This is because, while each forecast is noisy, combining the estimates from many questions cancels the noise and leaves us with the signal.
Question decomposition is also good at increasing epistemic legibility. It helps forecasters to communicate to others why they’ve made the forecast that they did and it allows them to identify their specific points of disagreement.
Premortems
A premortem is a strategy used once you’ve assigned a probability to an event. You ask yourself to imagine that the forecast was wrong and you then work backwards to determine what could potentially have caused this.
It is simply a way to reframe the question “in what ways might I be wrong?” but in a way that reduces motivated reasoning caused by attachment to the bottom line.
Practice
Getting Started on the Forecasting Wiki
While the above techniques are useful, they are no substitute for actually making predictions. Get out there and make predictions! Use the above techniques. Keep track of your predictions. Periodically evaluate questions that have been resolved and review your performance. Assess the degree to which you are calibrated. Look out for systematic mistakes that you might be making. Make more predictions! Over time, like with any skill, your ability can and should improve.
Other Resources
Other resources include:
Superforcasting by Philip Tetlock and Dan Gardener
Intro to Forecasting by Alex Lawson
Forecasting Newsletter by Nuño Sempere
State of the Art
For many years there have been calls to apply forecasting techniques to non-academic domains including journalism, policy, investing and business strategy. Several organisations now exist within these niche.
Metaculus
Metaculus is a popular and established web platform for forecasting. Their questions mainly focus on geopolitics, the coronavirus pandemic and topics of interest to Effective Altruism.
They host prediction competitions with real money prizes and collect and track public predictions made by various figures.
Cultivate Labs
Cultivate Labs build tools that companies can use to crowdsource information from among their employees. This helps leadership to understand the consensus of people working on the ground and use this to improve the decisions they make.
Kalshi
Kalshi provide real money prediction markets on geopolitical events. The financial options they provide are intended to be used as hedges for political risk.
Manifold.Markets
Manifold.Markets is a prediction market platform that uses play money. It is noteworthy for its ease of use, great UI and the fact that the market creator decides how the market resolves.
QURI
QURI is a research organisation that builds tools that make it easier to make good forecasts. Their most notable tool is Squiggle—a programming language designed to be used to make legible forecasts in a wide range of contexts.
This is a broad topic group that captures several sub-topics:
Concepts
Tools and Platforms
Tutorials
Alex Lawsen’s video series
Metaculus’s tutorials
Organizations