Executive summary: This post outlines a research agenda for improving judgmental forecasting, identifying key challenges, assessing current forecasting ability, and proposing techniques like question decomposition and using language models to enhance forecasts.
Key points:
The five main challenges in forecasting are long time horizons, reward-correlated predictions, low probability events, out-of-distribution situations, and hard-to-specify events.
Open questions remain about current forecasting ability, including performance on long-term and low-probability questions, convergence behavior, and comparisons between prediction markets, teams, and models.
Potential improvements include developing better scoring rules, techniques for handling unclear resolution criteria and incentivizing predictions on challenging questions, and empirically testing question decomposition methods.
Question decomposition can be multiplicative, additive (MECE), or recursive, and may be enhanced by using large language models, though more research is needed.
Other research directions include analyzing existing forecast datasets, studying question quality, developing aggregation methods, and assessing the robustness of current prediction platforms.
This comment was auto-generated by the EA Forum Team. Feel free to point out issues with this summary by replying to the comment, andcontact us if you have feedback.
Executive summary: This post outlines a research agenda for improving judgmental forecasting, identifying key challenges, assessing current forecasting ability, and proposing techniques like question decomposition and using language models to enhance forecasts.
Key points:
The five main challenges in forecasting are long time horizons, reward-correlated predictions, low probability events, out-of-distribution situations, and hard-to-specify events.
Open questions remain about current forecasting ability, including performance on long-term and low-probability questions, convergence behavior, and comparisons between prediction markets, teams, and models.
Potential improvements include developing better scoring rules, techniques for handling unclear resolution criteria and incentivizing predictions on challenging questions, and empirically testing question decomposition methods.
Question decomposition can be multiplicative, additive (MECE), or recursive, and may be enhanced by using large language models, though more research is needed.
Other research directions include analyzing existing forecast datasets, studying question quality, developing aggregation methods, and assessing the robustness of current prediction platforms.
This comment was auto-generated by the EA Forum Team. Feel free to point out issues with this summary by replying to the comment, and contact us if you have feedback.