Hey Dan, thanks for sanity-checking! I think you and feruell are correct to be suspicious of these estimates, we laid out reasoning and probabilities for people to adjust to their taste/confidence.
I agree outliers are concerning (and find some of them implausible), but I likewise have an experience of being at 10..20% when a crowd was at ~0% (for a national election resulting in a tie) and at 20..30% when a crowd was at ~0% (for a SCOTUS case) [likewise for me being ~1% while the crowd was much higher; I also on occasion was wrong updating x20 as a result, not sure if peers foresaw Biden-Putin summit but I was particularly wrong there].
I think the risk is front-loaded, and low month-to-year ratios are suspicious, but I don’t find them that implausible (e.g., one might expect everyone to get on a negotiation table/emergency calls after nukes are used and for the battlefield to be “frozen/shocked” – so while there would be more uncertainty early on, there would be more effort and reasons not to escalate/use more nukes at least for a short while — these two might roughly offset each other).
Yeah, it was my prediction that conjunction vs. direct wouldn’t match for people (really hard to have a good “sense” of such low probabilities if you are not doing a decomposition). I think we should have checked these beforehand and discussed them with folks.
It would be interesting whether the forecasters with outlier numbers stand by those forecasts on reflection, and to hear their reasoning if so. In cases where outlier forecasts reflect insight, how do we capture that insight rather than brushing them aside with the noise? Checking in with those forecasters after their forecasts have been flagged as suspicious-to-others is a start.
The p(month|year) number is especially relevant, since that is not just an input into the bottom line estimate, but also has direct implications for individual planning. The plan ‘if Russia uses a nuclear weapon in Ukraine then I will leave my home to go someplace safer’ looks pretty different depending on whether the period of heightened risk when you will be away from home is more like 2 weeks or 6 months.
Hey Dan, thanks for sanity-checking! I think you and feruell are correct to be suspicious of these estimates, we laid out reasoning and probabilities for people to adjust to their taste/confidence.
I agree outliers are concerning (and find some of them implausible), but I likewise have an experience of being at 10..20% when a crowd was at ~0% (for a national election resulting in a tie) and at 20..30% when a crowd was at ~0% (for a SCOTUS case) [likewise for me being ~1% while the crowd was much higher; I also on occasion was wrong updating x20 as a result, not sure if peers foresaw Biden-Putin summit but I was particularly wrong there].
I think the risk is front-loaded, and low month-to-year ratios are suspicious, but I don’t find them that implausible (e.g., one might expect everyone to get on a negotiation table/emergency calls after nukes are used and for the battlefield to be “frozen/shocked” – so while there would be more uncertainty early on, there would be more effort and reasons not to escalate/use more nukes at least for a short while — these two might roughly offset each other).
Yeah, it was my prediction that conjunction vs. direct wouldn’t match for people (really hard to have a good “sense” of such low probabilities if you are not doing a decomposition). I think we should have checked these beforehand and discussed them with folks.
It would be interesting whether the forecasters with outlier numbers stand by those forecasts on reflection, and to hear their reasoning if so. In cases where outlier forecasts reflect insight, how do we capture that insight rather than brushing them aside with the noise? Checking in with those forecasters after their forecasts have been flagged as suspicious-to-others is a start.
The p(month|year) number is especially relevant, since that is not just an input into the bottom line estimate, but also has direct implications for individual planning. The plan ‘if Russia uses a nuclear weapon in Ukraine then I will leave my home to go someplace safer’ looks pretty different depending on whether the period of heightened risk when you will be away from home is more like 2 weeks or 6 months.