Based on reading through this, it seems promising—the question is whether it’s worth someone overcoming the startup costs / bottlenecks to run a pilot, and I’m not sure it is unless there is a specific promising use case that makes this seem much better than current approaches. Any ideas for specific markets where this is valuable?
1. CEA / Open Phil wants to contract [Metaculus/Manifold/Good Judgement/Epoch] to come up with forecasts for 100 continuous questions regarding clean meat. They just care that the forecasts are good, they don’t care much for which specific methods are used. They also would like to see new forecasting players join the fray and compete for cost-effective work.
The put out an open call for proposals, and say that much of the money will come in the form of a simple Accuracy Agreement (one that perhaps can’t be re-sold, for legal simplicity reasons, for now).
2. A government agency wants to outsource “Predictions of construction costs, for a megaproject”. This set of predictions will be very expensive to produce and update. Instead of paying consultants in the traditional way, they organize much of this as an Accuracy Agreement.
3. In 2030, in a hypothetical future world, there are many big forecasting players and purchasers. Complicated forecasting Type Definitions are routinely created for things. Like, “For every US regional area, and every point in time for the next 5 years, how many of each disease will be monitored/registered in that area?”. Forecasting purchasers can expect that a public bidding market will return a reasonable result, similar to some current large project bidding procedures now.
I see this approach as a more complex but powerful alternative to the model of buyers now just contracting with specific firms like Metaculus or Good Judgement, which has started to happen in the last few years (for judgemental questions).
One weakness that these Agreements have is that they require the client (or a third party) to ensure that questions are written and scored, instead of the consultant.
This is a similar issue that Prediction Markets have, but not one that existing forecasting contracts often have. These contracts often have the forecasting contractors do the work of question specification and resolution.
So, Accuracy Agreements are probably in-between Prediction Markets and current contractor agreements, in complexity.
For big government construction projects, I believe some firms/agencies will do a lot of preparation and outlining, before a bidding process might begin. Getting things specific enough for a large bidding process is itself a fair bit of work. This can be useful for large projects, or in cases where the public has little trust in the key decision makers, but is probably cost-prohibitive for other situations.
Based on reading through this, it seems promising—the question is whether it’s worth someone overcoming the startup costs / bottlenecks to run a pilot, and I’m not sure it is unless there is a specific promising use case that makes this seem much better than current approaches. Any ideas for specific markets where this is valuable?
Some ideas:
1. CEA / Open Phil wants to contract [Metaculus/Manifold/Good Judgement/Epoch] to come up with forecasts for 100 continuous questions regarding clean meat. They just care that the forecasts are good, they don’t care much for which specific methods are used. They also would like to see new forecasting players join the fray and compete for cost-effective work.
The put out an open call for proposals, and say that much of the money will come in the form of a simple Accuracy Agreement (one that perhaps can’t be re-sold, for legal simplicity reasons, for now).
2. A government agency wants to outsource “Predictions of construction costs, for a megaproject”. This set of predictions will be very expensive to produce and update. Instead of paying consultants in the traditional way, they organize much of this as an Accuracy Agreement.
3. In 2030, in a hypothetical future world, there are many big forecasting players and purchasers. Complicated forecasting Type Definitions are routinely created for things. Like, “For every US regional area, and every point in time for the next 5 years, how many of each disease will be monitored/registered in that area?”. Forecasting purchasers can expect that a public bidding market will return a reasonable result, similar to some current large project bidding procedures now.
I see this approach as a more complex but powerful alternative to the model of buyers now just contracting with specific firms like Metaculus or Good Judgement, which has started to happen in the last few years (for judgemental questions).
One weakness that these Agreements have is that they require the client (or a third party) to ensure that questions are written and scored, instead of the consultant.
This is a similar issue that Prediction Markets have, but not one that existing forecasting contracts often have. These contracts often have the forecasting contractors do the work of question specification and resolution.
So, Accuracy Agreements are probably in-between Prediction Markets and current contractor agreements, in complexity.
For big government construction projects, I believe some firms/agencies will do a lot of preparation and outlining, before a bidding process might begin. Getting things specific enough for a large bidding process is itself a fair bit of work. This can be useful for large projects, or in cases where the public has little trust in the key decision makers, but is probably cost-prohibitive for other situations.