Why does Metaculus default to āresolve timeā when in your analysis you think it is better to present āall timesā? And given my goal of using Metaculus, which āevaluated atā setting should I pick?
The Brier score evaluated at āall timesā applies to the whole period during which the question was open. It is the mean Brier score, i.e. the one I would see if I selected a random time during which the question was open. I used it because it contains more information.
I think the setting one should pick depends on the context. If you are looking into:
A question which has already closed, but not yet resolved, I would pick āclose timeā.
A question which is still open, I would check āall timesā, and āother timeā matching your current conditions (for example, 1 year āprior to resolve timeā). The less data I had for the āother timeā option, the more weight I would give to āall timesā (everything else equal).
I want to demonstrate to people these are probably the best estimates available of what threats society and individuals are most likely to face in the coming decades and therefore a good way to think about how to build resilience against these threats.
I think it is hard to know how reliable Metaculusā predictions will be with respect to these questions, as Metaculusā track record does not yet contain data about long-range questions. There are only 8 questions whose Brier can be evaluated 5 years prior to resolve time. For communicating risk to your audience, one could try to make a case for the possibility of the next few decades being wild (if Metaculusā nearterm predictions about AI are to be trusted), and the possibility of this being the most important century.
Thanks again for your excellent work and for you patience with my questions.
Thanks for the follow-up questions!
The Brier score evaluated at āall timesā applies to the whole period during which the question was open. It is the mean Brier score, i.e. the one I would see if I selected a random time during which the question was open. I used it because it contains more information.
I think the setting one should pick depends on the context. If you are looking into:
A question which has already closed, but not yet resolved, I would pick āclose timeā.
A question which is still open, I would check āall timesā, and āother timeā matching your current conditions (for example, 1 year āprior to resolve timeā). The less data I had for the āother timeā option, the more weight I would give to āall timesā (everything else equal).
I think it is hard to know how reliable Metaculusā predictions will be with respect to these questions, as Metaculusā track record does not yet contain data about long-range questions. There are only 8 questions whose Brier can be evaluated 5 years prior to resolve time. For communicating risk to your audience, one could try to make a case for the possibility of the next few decades being wild (if Metaculusā nearterm predictions about AI are to be trusted), and the possibility of this being the most important century.
No worries; you are welcome!