I wonder what would happen if you were to do the same exercise with the fixed-year predictions under a âconstant riskâ model, i.e.P(t) = 1 - exp(-l*t) with l = - year / log(1 - P(year)), to get around the problem that weâre still 3 years away from 2026. Given that timelines are systematically longer with a fixed-year framing, I would expect the Brier score of those predictions would be worse. OTOH, the constant risk model doesnât seem very reasonable here, so the results wouldnât have a straightforward interpretation.
Iâd be interested in seeing this too! Although Iâm not planning to spend the time to do this soon, Iâd be up for having a quick chat if someone else was up for it. The constant risk model seems to me like a better-than-nothing model, so could be a slight update if it gives different results. Or could just use a linear increase of probability from 0 years to 10 years.
FWIWâfrom this survey alone, Iâm not convinced that timelines are systematically longer with a fixed-year framing. I sampled ten forecasts where probabilities were given in both framings on a 10 year timescale, and five of them (Subtitles, Transcribe, Top forty, Random game, Explain) gave later forecasts when asked with âprobability in N yearsâ rather than âyear that the probability is Mâ, three of them (Video scene, Read aloud, Atari) gave the same forecasts, and two of them (Rosetta, Taylor) gave an earlier forecast. So from this sample, it seems theyâre commonly longer.
Great analysis!
I wonder what would happen if you were to do the same exercise with the fixed-year predictions under a âconstant riskâ model, i.e.
P(t) = 1 - exp(-l*t)withl = - year / log(1 - P(year)), to get around the problem that weâre still 3 years away from 2026. Given that timelines are systematically longer with a fixed-year framing, I would expect the Brier score of those predictions would be worse. OTOH, the constant risk model doesnât seem very reasonable here, so the results wouldnât have a straightforward interpretation.Iâd be interested in seeing this too! Although Iâm not planning to spend the time to do this soon, Iâd be up for having a quick chat if someone else was up for it. The constant risk model seems to me like a better-than-nothing model, so could be a slight update if it gives different results. Or could just use a linear increase of probability from 0 years to 10 years.
FWIWâfrom this survey alone, Iâm not convinced that timelines are systematically longer with a fixed-year framing. I sampled ten forecasts where probabilities were given in both framings on a 10 year timescale, and five of them (Subtitles, Transcribe, Top forty, Random game, Explain) gave later forecasts when asked with âprobability in N yearsâ rather than âyear that the probability is Mâ, three of them (Video scene, Read aloud, Atari) gave the same forecasts, and two of them (Rosetta, Taylor) gave an earlier forecast. So from this sample, it seems theyâre commonly longer.