A quick and crude comparison of epidemiological expert forecasts versus Metaculus forecasts for COVID-19

[Jotto]

Katherine Milkman on Twitter notes how far off the epidemiological expert forecasts were in the linked sample:

https://​​twitter.com/​​katy_milkman/​​status/​​1244668082062348291

They gave an average estimate of 20,000 cases. The actual outcome was 122,653 by the stated date in the U.S. That’s off by a factor of 6.13.

I was curious how this compares to the Metaculus community forecast (note: not the machine learning fed one, just the simple median prediction). Unfortunately the interface doesn’t tell me the full distribution at date x, it just says what the median was at the time. If the expert central tendency was off by a factor of 6.13, how far off was it for Metaculus?

I looked into it in this document:

https://​​docs.google.com/​​document/​​d/​​1TcG1gceeVgimr8lBTCAYUscOt_ADyuSpdmt1d6Scve8/​​edit?usp=sharing

Sadly a direct comparison is not really feasible, since we weren’t predicting the same questions. But suppose if all predictions of importance were inputted into platforms such as the Good Judgement Project Open or Metaculus. Then making comparisons between groups could be trivial and continuous. This isn’t even “experts versus non-experts”. The relevant comparison is at the platform-level. It is “untrackable and unworkable one-off PDFs of somebody’s projections” versus proper scoring and aggregation over time. Since Metaculus accounts can be entirely anonymous, why wouldn’t we want every expert to input their forecast into a track record? That would make it possible to find out if the person is a dart-throwing chimp. You should assume half of them are.

[Benjamin_Todd]

There have been some claims that the 538 article put the wrong date on the expert’s forecasts, and we haven’t been able to figure out whether that’s true or not by contacting them, so unfortunately I wouldn’t use the 538 article by itself.

[Jotto]

If I’m reading this Tweet thread correctly, anna wiederkehr from 538 seems to say the graphic was correct and the error really is that much.

https://​​twitter.com/​​wiederkehra/​​status/​​1245026159529807873?s=20

It is further implied by them in another tweet:

https://​​twitter.com/​​wiederkehra/​​status/​​1244956725150650368?s=20

Hey @katy_milkman
you seemed to have missed our follow up (and we’ll do another this week) that shows a different picture in case estimates from the surveyed experts. It takes time to understand what’s happening.

[Kirsten]

It’s worth noting that epidemiologists can do modelling of disease given particular assumptions or policy choices, but they do not make a career out of predicting policies, which is a lot of what this question is about.

[Linch]

I messaged Khorton on Twitter, but just paraphrasing what I said here:

I think this description is incomplete. By ~late April/​May, in direct comparisons, generalist predictors on Metaculus and GJ Open generally outpredicted epidemiologist’s surveys for forecasts 1 week out. This is relevant because there was no way for changes due to new gov’t interventions or behavioral changes in the future to show up in case statistics in less than 7 days.

It takes several days for infections to get symptoms, a few days for people to get tested, and another X days (decreases as pandemic progressed) for tests to resolve.

So a question like “how many cases/​deaths in 6 days” can almost be entirely decomposed to your distributions on

- how many infected *now*,

- how testing will change in a week

- probability of data updates.

The latter two does incorporate some elements of politics and behaviorial change. However, “amateur forecasters better at predicting political trends” can’t be the full story, because in a lot of weeks data updates/​testing didn’t change *that much*, so amateur predictors essentially had a better internal model of how many ppl currently infected.

There are a lot of caveats, of course.

[Linch]

Metaculus predictions are now featured in those surveys (yay!) so I was able to make a more direct comparison for the first survey where you can compare those predictions head-to-head.

tl;dr: Experts have broadly outperformed the Metaculus aggregative predictions, however the differences were not exceptionally large.

[Linch]

UPDATE: With more data, Metaculus users have instead done better again.

[Jotto]

Thank you very much! Apologies for not replying to your earlier comment. I was predicting that the Metaculus community prediction would outperform the surveys, and it is gratifying to see.