[Question] Are there superforecasts for existential risk?

[Alex HT]

The closest thing I could find was the Metaculus Ragnarök Question Series, but I’m not sure how to interpret it because:

• The answers seem inconsistent (eg. a 1% chance of >95% of humans being killed by 2100, but a 2% chance of humans going extinct by 2100). Maybe this isn’t all that problematic but I’m not sure

• The incentives for accuracy seem weird. These questions only resolve by 2100, and, if there is a catastrophe, nobody will care about their Brier score. Again, this might not be a problem but I’m not sure

• The ‘community prediction’ (the median) was much higher than the ‘Metaculus prediction’ (some weighted combination of each user’s prediction). Is that because more accurate forecasters were less worried about existential risk, or because there’s something that makes a good near-term forecaster that makes people underestimate existential risk?

Related: here’s a list of database of existential risk estimates, and here’s a list of AI-risk prediction market question suggestions.

I wonder if questions around existential risk would better be estimated by a smaller group of forecasters, rather than a prediction market or something like Metaculus (for the above reasons and other reasons).

[Davidmanheim]

Yes, but it is hard, and they don’t work well. They can, however, be done at least slightly better.

Good Judgement was asked to forecast the risk of a nuclear war in the next year—which helps somewhat with the time frame question. Unfortunately, the brier score incentives are still really weak.

Ozzie Gooen and others have talked a lot about how to make forecasting better. Some of the ideas that he has suggested relate to how to forecast longer term questions. I can’t find a link to a public document, but here’s one example (which may have been someone else’s suggestion):

You ask people to forecast what probability people will assign in 5 years to the question “will there be a nuclear war by 2100?” (You might also ask whether there will be a nuclear war in the next 5 years, of course.) By using this trick, you can have the question (s) resolve in 5 years, and have an approximate answer based on iterated expectation. But extending this, you can also have them predict what probability people will assign in 5 years to the probability they will assign in another 5 years to the question “will there be a nuclear war by 2100″ - and by chaining predictions like this, you can transform very long term questions into series of shorter term questions.

There is other work in this vein, but to simplify, all of it takes the form “can we do something clever to slightly reduce the issues that exist with the fundamentally hard question of getting short term answers to long term questions.” As far as I can see, there aren’t any simple answers.

[Alex HT]

Thanks for the answer.

Will MacAskill mentioned in this comment that he’d ‘expect that, say, a panel of superforecasters, after being exposed to all the arguments, would be closer to my view than to the median FHI view.’

You’re a good forecaster right? Does it seem right to you that a panel of good forecasters would come to something like Will’s view, rather than the median FHI view?

[Davidmanheim]

I’ll speak for the consensus when I say I think there’s not a clear way to decide if this is correct without actually doing it—and the outcome would depend a lot on what level of engagement the superforecasters had with these ideas already. (If I got to pick the 5 superforecasters, even excluding myself, I could guarantee it was either closer to FHI’s viewpoints, or to Will’s.) Even if we picked from a “fair” reference class, if I could have them spend 2 weeks at FHI talking to people there, I think a reasonable proportion would be convinced—though perhaps this is less a function of updating neutrally towards correct ideas as it is the emergence of consensus in groups.

Lastly, I have tremendous respect for Will, but I don’t know that he’s calibrated particularly well to make a prediction like this. (Not that I know he isn’t—I just don’t have any reason to think he’s spent much time working on this skillset.)

[zdgroff]

Thanks for writing this. I’ve had similar questions myself.

I think the incentives issue here is a big one. One way I’ve wondered about addressing it is to find a bunch of people who forecast really well and whose judgments are not substantially affected by forecasting incentives. Then have them forecast risks. Might that work, and has anyone tried it?

[HaydnBelfield]

You might be interested in these two papers:

Identifying and Assessing the Drivers of Global Catastrophic Risk by Simon Beard & Phil Torres.

An Analysis and Evaluation of Methods Currently Used to Quantify the Likelihood of Existential Hazards by Simon Beard, Thomas Rowe & James Fox.

[Alex HT]

Thanks, those look good and I wasn’t aware of them

[Linch]

As someone who is one of the most active users of Metaculus (particularly 2 months ago), I find a lot of the Metaculus estimates on existential risk pretty suspicious:

• There’s no incentive to do well on those questions.

• The feedback loops are horrible

  • Indeed, some people have actually joked betting low on the more existential questions since they won’t get a score if we’re all dead (at least, I hope they’re joking)

• At the object-level, I just think people are really poorly calibrated about x-risk questions

  • My comment here arguably changed the community’s estimates by ~10%

    • It wasn’t a very insightful comment by EA standards

• People are not well-trained on low probabilities

  • Both that they literally have no way of doing so on Metaculus (can’t go <1% or >99%) and that they haven’t predicted enough questions to be calibrated at those levels.

• My inside view is that people on Metaculus are more well-read than most people on existential risk matters, but probably on average less well-read than, eg, your typical organizer of a EA local or college group.

• My guess is that as of summer 2020, there’s no strong reason to think Metaculus will do better than a group of well-read EAs on existential risk questions.

[alex lawsen]

As another very active user of metaculus recently, who’s come from an EA background, I basically agree with all of the above, except that I don’t think all the users are joking about betting low on the existential questions.

[Linch]

I find it hard to parse where our disagreement is. Can you point me to what quote you disagree with?

[alex lawsen]

“Indeed, some people have actually joked betting low on the more existential questions since they won’t get a score if we’re all dead (at least, I hope they’re joking)”

I think several of them aren’t joking, they care more about feeling “smart” because they’ve gamed the system than the long-term potential consequences of inaccurate forecasts.

I do, like you, really hope that I’m wrong and they are in fact joking.

[Max_Daniel]

Thanks, this is quite useful. I hadn’t considered the issue of incentives sufficiently before, and the OP and your comment make me put less weight on the Metaculus x-risk forecasts than I did previously.

(Though I didn’t put a lot of absolute weight on them, and I can’t think of any decision or downstream discussion that would be significantly affected by the update on Metaculus.)

[Alex HT]

Thanks for the answer.

Will MacAskill mentioned in this comment that he’d ‘expect that, say, a panel of superforecasters, after being exposed to all the arguments, would be closer to my view than to the median FHI view.’

You’re a good forecaster right? Does it seem right to you that a panel of good forecasters would come to something like Will’s view, rather than the median FHI view?

[Linch]

I’m not sure. I mentioned as a reply to that comment that I was unimpressed with the ability of existing “good” forecasters to think about low-probability and otherwise out-of-distribution problems. My guess is that they’d change their minds if “exposed” to all the arguments, and specifically have views very close to the median FHI view, if “exposed” → reading the existing arguments very carefully and put lots of careful thought into them. However, I think this is a very tough judgement call, and does seem like the type of thing that’d be really bad if we get it wrong!

My beliefs here are also tightly linked to me thinking that the median FHI view is more likely to be correct than Will’s view, and it is a well-known bias that people think their views are more common/​correct than they actually are.