Forecasting tools and prediction markets allow you to state your credence in something precisely and view others’ credences. This can be an important way of gathering information from a group or sharing information precisely between individuals.
This post will introduce a few tools for generating and sharing predictions. Many of these tools are excellent; they’ve been grouped together because of their similarity and number.
One way I think forecasting tools can be useful for collaborative truth-seeking is that spending some time using them can improve both your calibration and your ability to precisely define questions. I think this can then greatly improve your ability to communicate your implicit beliefs explicitly in conversation or text. I’d be interested to hear other people’s opinions on this view!
Forecasting tools and putting precise numbers on predictions in general is important because it lets a community approach truth over time better by letting people have a more transparent track record to learn who is in fact good at predicting.
This post will discuss the following tools:
Metaculus
Manifold Markets
Pastcasting
Confido
PredictionBook
Elicit
When can forecasting/prediction tools be useful?
Practice predicting so that you’re better able to communicate your beliefs verbally (especially PredictionBook, Pastcasting)
Get community predictions relevant to deciding where to donate, e.g.:
AI milestones
Cultured meat
Factory farming
Global poverty
Politics
Make a private question to ask people in your organisation to predict project outcomes or engagement (Confido)
Add a prediction question on your post to see if people’s views change after reading (Elicit)
Find out how people feel about a controversial topic (with people who feel more strongly able to influence the result more; e.g. here)
The official Metaculus tutorials are suitable for beginners as a way to introduce forecasting, probability distributions, and a basic understanding of how to behave under uncertainty.
Tutorial #1 is over-explaining for those with a basic understanding of the forecasting, but still a good introduction to using Metaculus specifically.
For those familiar with forecasting, tutorials #2 and #3 are more useful as a (small) amount of calibration practice and building familiarity with the Metaculus interface.
Examples
Note that you can embed metaculus questions in forum posts by posting the direct link into the editor:
Note that you can embed Manifold Markets questions into forum posts by pasting the direct link into the editor:
Other Tools
Pastcasting is a forecasting platform using resolved questions from other sites (e.g. Metaculus) to shorten the feedback loop and practice forecasting, particularly over long time horizons. The official FAQ is a great introduction. In partcular, I’d like to highlight Pastcasting’s feature Vantage Search:
Vantage Search is a custom search engine that only shows results from before the vantage point to prevent information leaks from the future.
Confidois tool that provides an instance of a closed forecasting platform. Forecasts are entered similarly to on metaculus, but questions are intended for a single group or organisation. This is useful for questions requiring insider knowledge like “How will the EA conference we are planning go” or “How many employees will we have by the end of the year”. Trying the demo version is a good way to understand the tool.
PredictionBookis simple tool for prediction by entering a % credence only. You write can write a question, put the date when you should know the true answer by, and enter a credence. On the date you entered, you’ll be notified to give the true answer.
Predictions can be public or private; I find it’s particularly useful for producing personal questions “in 2023, will I go 60 days without allergy symptoms?” as a way of improving your calibration (including an automatic calibration graph[2]).
Elicit Forecast is a tool that lets you easily set up questions people can put credences on, and shows an aggregate of them, but is currently unsupported[3]. At present, only binary questions (with a % chance input) work as far as I can tell; date or number estimate questions are currently non-functioning. There is no guarantee any functionality will continue.
I don’t honestly use prediction or forecasting tools much in my day-to-day life[4]. If this were to change I think I’d be most likely to use PredictionBook to improve my calibration on everyday questions that I need to be able to predict, or manifold markets simply for fun.
I think if you’re regularly making big decisions like grant making or running projects in AI, biosecurity, etc., forecasting tools like Metaculus or Manifold could be useful, particularly the comments (where other people explain their reasoning). Similarly, I think Confido would be useful for a sufficiently-sized organisation.
Elicit is useful because of its ability to be embedded & interacted with in forum posts. If somebody picks up supporting the tool & fixes the current issues, I imagine I’d add it to a lot more forum posts.
Try it Yourself!
You can access the below market to buy shares yourself here.
I’d also suggest making an account on PredictionBook and formalising 3 predictions that you’re currently making implicitly! Here are some suggestions to get the ball rolling:
The completion date of a current project
How much money you’ll donate to charity before 31/12/2023
How many people will read your next forum post
How successful you will be with your new year’s resolution
Your total spending this month
What time you’ll get home today
Tomorrow at 6pm GMT we’ll also be running a short event in the EA GatherTown to discuss forecasting tools & do a short exercise about them!
i.e. a graph of “% you assigned to an outcome” against “% chance that that actually happened” (which would be the line y=x if you were perfectly calibrated)
Forecasting tools and Prediction Markets: Why and How
Forecasting tools and prediction markets allow you to state your credence in something precisely and view others’ credences. This can be an important way of gathering information from a group or sharing information precisely between individuals.
This post will introduce a few tools for generating and sharing predictions. Many of these tools are excellent; they’ve been grouped together because of their similarity and number.
One way I think forecasting tools can be useful for collaborative truth-seeking is that spending some time using them can improve both your calibration and your ability to precisely define questions. I think this can then greatly improve your ability to communicate your implicit beliefs explicitly in conversation or text. I’d be interested to hear other people’s opinions on this view!
Forecasting tools and putting precise numbers on predictions in general is important because it lets a community approach truth over time better by letting people have a more transparent track record to learn who is in fact good at predicting.
This post will discuss the following tools:
Metaculus
Manifold Markets
Pastcasting
Confido
PredictionBook
Elicit
When can forecasting/prediction tools be useful?
Practice predicting so that you’re better able to communicate your beliefs verbally (especially PredictionBook, Pastcasting)
Get community predictions relevant to deciding where to donate, e.g.:
AI milestones
Cultured meat
Factory farming
Global poverty
Politics
Make a private question to ask people in your organisation to predict project outcomes or engagement (Confido)
Add a prediction question on your post to see if people’s views change after reading (Elicit)
Find out how people feel about a controversial topic (with people who feel more strongly able to influence the result more; e.g. here)
Learn to forecast (especially Pastcasting)
Perform curious self-referential research on prediction markets[1] (Manifold Markets)
Metaculus
Metaculus is a reputation-based site for soliciting and aggregating predictions.
The official Metaculus tutorials are suitable for beginners as a way to introduce forecasting, probability distributions, and a basic understanding of how to behave under uncertainty.
Tutorial #1 is over-explaining for those with a basic understanding of the forecasting, but still a good introduction to using Metaculus specifically.
For those familiar with forecasting, tutorials #2 and #3 are more useful as a (small) amount of calibration practice and building familiarity with the Metaculus interface.
Examples
Note that you can embed metaculus questions in forum posts by posting the direct link into the editor:
Manifold Markets
Manifold Markets is a play money prediction market with user-created questions. The play money can be redeemed as a charitable donation. Because it works as a market, it can be used in lots of interesting ways—you can ask people to predict which opportunities your organisation will fund (as one factor to influence your decision, maybe), you can play chess, and you can even implement futarchy.
A text tutorial is available on the wiki here.
Examples
Note that you can embed Manifold Markets questions into forum posts by pasting the direct link into the editor:
Other Tools
Pastcasting is a forecasting platform using resolved questions from other sites (e.g. Metaculus) to shorten the feedback loop and practice forecasting, particularly over long time horizons. The official FAQ is a great introduction. In partcular, I’d like to highlight Pastcasting’s feature Vantage Search:
Confido is tool that provides an instance of a closed forecasting platform. Forecasts are entered similarly to on metaculus, but questions are intended for a single group or organisation. This is useful for questions requiring insider knowledge like “How will the EA conference we are planning go” or “How many employees will we have by the end of the year”. Trying the demo version is a good way to understand the tool.
PredictionBook is simple tool for prediction by entering a % credence only. You write can write a question, put the date when you should know the true answer by, and enter a credence. On the date you entered, you’ll be notified to give the true answer.
Predictions can be public or private; I find it’s particularly useful for producing personal questions “in 2023, will I go 60 days without allergy symptoms?” as a way of improving your calibration (including an automatic calibration graph[2]).
Elicit Forecast is a tool that lets you easily set up questions people can put credences on, and shows an aggregate of them, but is currently unsupported[3]. At present, only binary questions (with a % chance input) work as far as I can tell; date or number estimate questions are currently non-functioning. There is no guarantee any functionality will continue.
Binary elicit forecasts currently work as embeds:
Personal Experience
I don’t honestly use prediction or forecasting tools much in my day-to-day life[4]. If this were to change I think I’d be most likely to use PredictionBook to improve my calibration on everyday questions that I need to be able to predict, or manifold markets simply for fun.
I think if you’re regularly making big decisions like grant making or running projects in AI, biosecurity, etc., forecasting tools like Metaculus or Manifold could be useful, particularly the comments (where other people explain their reasoning). Similarly, I think Confido would be useful for a sufficiently-sized organisation.
Elicit is useful because of its ability to be embedded & interacted with in forum posts. If somebody picks up supporting the tool & fixes the current issues, I imagine I’d add it to a lot more forum posts.
Try it Yourself!
You can access the below market to buy shares yourself here.
I’d also suggest making an account on PredictionBook and formalising 3 predictions that you’re currently making implicitly! Here are some suggestions to get the ball rolling:
The completion date of a current project
How much money you’ll donate to charity before 31/12/2023
How many people will read your next forum post
How successful you will be with your new year’s resolution
Your total spending this month
What time you’ll get home today
Tomorrow at 6pm GMT we’ll also be running a short event in the EA GatherTown to discuss forecasting tools & do a short exercise about them!
Also, join us at 6pm GMT today in the GatherTown to discuss the use of Squiggle!
“This market resolves to NO if there’s at least two people holding 157 YES shares, otherwise it resolves YES.”, Manifold Markets
i.e. a graph of “% you assigned to an outcome” against “% chance that that actually happened” (which would be the line y=x if you were perfectly calibrated)
Private communication, Ought
Though we did get feedback from people who do on various drafts of this post, don’t worry!