Executive summary: Metaculus is launching a series of AI forecasting benchmark contests with $120k in prizes to measure the state of the art in AI forecasting capabilities compared to human forecasters.
Key points:
The contests aim to benchmark AI forecasting accuracy, calibration, and logical consistency over time.
Bots will compete on 250-500 binary questions per contest, with performances compared against each other and human forecasters.
Bots must provide a rationale for each forecast to ensure reasoning transparency.
Metaculus provides a prompting interface and Google Colab notebook templates to help participants get started with building forecasting bots.
Participants are encouraged to experiment with prompt engineering and can seek support for model credits if needed.
Feedback and discussion are welcome via comments, a private form, and a dedicated Discord channel.
This comment was auto-generated by the EA Forum Team. Feel free to point out issues with this summary by replying to the comment, andcontact us if you have feedback.
Executive summary: Metaculus is launching a series of AI forecasting benchmark contests with $120k in prizes to measure the state of the art in AI forecasting capabilities compared to human forecasters.
Key points:
The contests aim to benchmark AI forecasting accuracy, calibration, and logical consistency over time.
Bots will compete on 250-500 binary questions per contest, with performances compared against each other and human forecasters.
Bots must provide a rationale for each forecast to ensure reasoning transparency.
Metaculus provides a prompting interface and Google Colab notebook templates to help participants get started with building forecasting bots.
Participants are encouraged to experiment with prompt engineering and can seek support for model credits if needed.
Feedback and discussion are welcome via comments, a private form, and a dedicated Discord channel.
This comment was auto-generated by the EA Forum Team. Feel free to point out issues with this summary by replying to the comment, and contact us if you have feedback.