christian

Launch­ing the AI Fore­cast­ing Bench­mark Series Q3 | $30k in Prizes

• christianJul 1, 2024, 8:22 PM

  2 points

  0 ∶ 0

  on: chris­tian’s Shortform

  Winners of the China and Global Cooperation Tournament

  The China and Global Cooperation Tournament has ended, thank you to everyone who forecasted! Here are the top three finishers:

  1. twsummer – $687

  2. Ab5A8bd20V – $194

  3. Inertia – $80

  The China and Global Cooperation Tournament featured questions related to the future of China and its role in the world stage. Participants forecasted on questions such as the global uptake of China’s currency, the number of Japanese Air Force responses to Chinese aircraft, and the combined market cap of large Chinese companies.

• christianJun 28, 2024, 8:03 PM

  1 point

  0 ∶ 0

  on: chris­tian’s Shortform

  Metaculus: Winners of the Global Pulse Tournament + Forecaster Impact

  The Global Pulse Tournament has concluded: Forecasters contributed to two academic studies conducted by Dr. Philipp Schoenegger, Associate Professor Barbara Fasolo, and Associate Professor Matteo Galizzi, all at the Behavioural Lab at the London School of Economics.
  

  Congratulations to the top three winners:

  1. skmmcj – $233

  2. datscilly – $231

  3. SpottedBear – $110

  Dr. Schoenegger shared the following about the research the tournament supported:

  I wanted to thank all of you for your hard work in this tournament! It was really amazing to follow the forecasts and discussions over the past few months (even though I, at times, really wanted to jump in and forecast myself)!
  
  We used the forecasts you generated in two academic studies, where we presented a report made up of of your forecasts to two samples of decision-makers who were forecasting on questions that were directly connected to but distinct from those asked in the tournament. While I am not yet able to share results as we are still in the analysis phase, I can tell you that we wouldn’t have been able to study this in such a thorough way without the actual tournament data you provided, so thank you!
  
  And last but not least, congratulations to the winners!

  You can find more details about the research shared by Dr. Schoenegger here. You can also see more details and the full leaderboard for the Global Pulse Tournament here.

• christianJun 26, 2024, 6:28 PM

  14 points

  0 ∶ 0

  on: chris­tian’s Shortform

  Metaculus FluSight Challenge ²⁰²³⁄₂₄ Winners + Forecasting Impact

  
  Metaculus forecaster impact in our 3rd annual FluSight Challenge:

  • Strengthened CDC flu forecasts
  • Helped train mechanistic models
  • 1 successful abstract submission by our partner, Dr. Thomas McAndrew, who leads Lehigh University’s Computational Uncertainty Lab
  • Sped progress on 2 manuscripts
  
  The flu burden has subsided for the season and with it the third annual FluSight Challenge has concluded. The top three prize-winners earning a share of the $5,000 prize pool are:

  1. SpottedBear – $3,432

  2. skmmcj – $914

  3. datscilly – $297

  Congratulations to the top three winners – who also earned the gold, silver, and bronze medals in the tournament – and congratulations to all the prizewinners!

  Our returning partner for this project, Dr. Thomas McAndrew, who leads Lehigh University’s Computational Uncertainty Lab, shared the following:

  We wish to thank everyone who submitted forecasts of the peak timing and intensity of influenza hospitalizations throughout this ²⁰²³⁄₂₄ season. Forecasts supported training a mechanistic model to generate forecasts for all 50 states in the country. Your contributions have led to a successful abstract submission (below), and this work has spurred two manuscripts: one focused on a computational technique to extend forecasts of peak timing/​intensity from 10 states to all 50 states; and a second focused on how to train a mechanistic model on human judgment ensemble forecasts of peak timing/​intensity. Your work here builds upon previous human judgment-support mechanistic modeling on a synthetic outbreak. Thank you again for all the hard work and dedication during the influenza season.
  
  Abstract: For the 2022/​2023 season, influenza accounted for 31m cases, 360k hospitalizations and 21k deaths, costing US healthcare ~$4.6 billion. Forecasts of influenza hospitalizations provide health officials with advanced warning. Prior research has revealed that forecasts generated by human judgment are similar in accuracy to mathematical models. However, little work has compared an equally weighted vs performance-weighted human judgment ensemble. We collected weekly human judgment forecasts of the peak number of incident hospitalizations (peak intensity) and the epidemic week where this peak occurs (peak time) for 10 US states from Oct 2023 to March 2024. The median, 25th, and 75th percentiles were extracted for each week. We found that, for the performance-weighted ensemble, forecast uncertainty decreased just before and after peak intensity and peak timing. However, uncertainty for the equally-weighted ensemble does not decrease. For both ensembles, the median prediction of peak intensity is smaller than the truth at the beginning of the season and approaches the truth monotonically. For peak time, the performance-weighted ensemble median prediction is later than the truth and approaches the truth only after the true peak time is observed. We have observed that the performance-weighted ensemble tends to produce more accurate forecasts of peak timing and intensity compared to an equally-weighted ensemble. One potential mechanism for this boost in accuracy was that the performance-weighted ensemble correctly assigned large weights to experienced forecasters and small weights to inexperienced forecasters. Experienced forecasters may be needed when asked to predict peak timing/​intensity of an ongoing outbreak.

  You can see more details and the full leaderboard for the FluSight Challenge ²⁰²³⁄₂₄ here.

  We will be in touch with prize winners to arrange payments. We are also happy to facilitate donations to the organizations listed here.

  Congratulations again to SpottedBear, skmmcj, and datscilly!

• christianJun 25, 2024, 8:40 PM

  1 point

  0 ∶ 0

  in reply to: elifland’s comment on: An­nounc­ing the AI Fore­cast­ing Bench­mark Series | July 8, $120k in Prizes

  Thanks, I’ve created a new link which shouldn’t expire and I’ve updated the post.

Open Sourc­ing Metaculus

An­nounc­ing the AI Fore­cast­ing Bench­mark Series | July 8, $120k in Prizes

Me­tac­u­lus: Q2 Cup Kick­off + Q1 Winners

• christianApr 1, 2024, 2:41 PM

  1 point

  0 ∶ 0

  in reply to: Austin’s comment on: Fore­cast in the Un­der­stand­ing AI Series With Ti­mothy B. Lee

  Great to see this! Absolutely, we’re looking forward to sharing more Metaculus collaborations with more interesting public thinkers in the near future.

Fore­cast in the Un­der­stand­ing AI Series With Ti­mothy B. Lee

AI Bench­marks Series — Me­tac­u­lus Ques­tions on Eval­u­a­tions of AI Models Against Tech­ni­cal Benchmarks

• christianMar 27, 2024, 4:28 PM

  1 point

  0 ∶ 0

  in reply to: david_reinstein’s comment on: Fore­cast Which Psy­chol­ogy Stud­ies Repli­cate With Me­tac­u­lus for the Trans­par­ent Repli­ca­tions Project

  Thanks for sharing that. About to dm you.

Me­tac­u­lus Launches Ques­tion Series With Bryan Caplan

• christianMar 19, 2024, 6:09 PM

  3 points

  0 ∶ 0

  on: chris­tian’s Shortform

  Metaculus Concludes Nuclear Risk Tournament Supporting Rethink Priorities

  Metaculus has concluded the Nuclear Risk Tournament, supporting Rethink Priorities’ efforts by helping to inform funding, policy, research, and career decisions aimed at reducing existential risk.

  Thank you to Rethink Priorities for sponsoring the tournament, thank you to the forecasters who contributed their talents, and congratulations to the tournament winners.

  Learn more

Me­tac­u­lus Launches Fu­ture of AI Series, Based on Re­search Ques­tions by Arb

Me­tac­u­lus In­tro­duces Bet­ter Ques­tion Discovery

• christianFeb 5, 2024, 11:35 PM

  5 points

  0 ∶ 0

  on: chris­tian’s Shortform

  Now You Can Create Multiple Choice Questions on Metaculus

  Create multiple choice questions and bring greater clarity to topics with multiple potential outcomes where one and only one will occur.

  To get started, simply Create a Question and set the Question Type to ‘multiple choice’.

  Give the Group Variable a clear label, e.g., ‘Option’, ‘Team’, ‘Country’.

  Fill in the Multiple Choice Options, adding more fields as needed.

  After you share additional details including background information on your topic, we’ll be excited to review and publish your multiple choice question!

• christianJan 11, 2024, 8:37 PM

  12 points

  1 ∶ 0

  on: chris­tian’s Shortform

  

  Metaculus launches round 2 of the Chinese AI Chips Tournament

  Help bring clarity to key questions in AI governance and support research by the Institute for AI Policy and Strategy (IAPS).

  Start forecasting on new questions tackling broader themes of Chinese AI capability like:

  Will we see a frontier Chinese AI model before 2027?

  Will a Chinese firm order a large number of domestic AI chips?

  Will a Chinese firm order a large number of US or US-allied AI chips?

Me­tac­u­lus Launches Q1 2024 Quar­terly Cup for Cur­rent Events-Fo­cused, Fast-Re­solv­ing Questions

Me­tac­u­lus Hosts ACX 2024 Pre­dic­tion Contest