I recently had to dive into the Metaculus data for a report I’m writing and I produced the following plot along the way. I’m posting it here because it didn’t make it into the final report, but I felt it was worth sharing anyway.
Each dot corresponds to the Brier score for the community prediction on every non-ambiguously resolved question as a function of time horizon (i.e. time remaining until resolution when the prediction was made). There are up to 101 predictions per question for the reasons you describe in the post. The red line is a moving average and the shaded area is a (t-distributed) 95% confidence interval around the mean.
Thanks for doing these analyses!
I recently had to dive into the Metaculus data for a report I’m writing and I produced the following plot along the way. I’m posting it here because it didn’t make it into the final report, but I felt it was worth sharing anyway.
Each dot corresponds to the Brier score for the community prediction on every non-ambiguously resolved question as a function of time horizon (i.e. time remaining until resolution when the prediction was made). There are up to 101 predictions per question for the reasons you describe in the post. The red line is a moving average and the shaded area is a (t-distributed) 95% confidence interval around the mean.
Nice graph, thanks!