Error

One possibility is that there aren’t many risks that are truly unknown, in the sense that they fall outside of the categories Toby enumerates, for the simple reason that some of those categories are relatively broad, so cover much of the space of possible risks.

Even if that were true, there might still be (fine-grained) risks we haven’t thought about within those categories, however—e.g. new ways in which AI could cause an existential catastrophe.

Great post!

But based on Rowe & Beard’s survey (as well as Michael Aird’s database of existential risk estimates), no other sources appear to have addressed the likelihood of unknown x-risks, which implies that most others do not give unknown risks serious consideration.

I don’t think this is true. The Doomsday Argument literature (Carter, Leslie, Gott etc.) mostly considers the probability of extinction independently of any specific risks, so these authors’ estimates implicitly involve an assessment of unknown risks. Lots of this writing was before there were well-developed cases for specific risks. Indeed, the Doomsday literature seems to have inspired Leslie, and then Bostrom, to start seriously considering specific risks.

Leslie explicitly considers unknown risks (p.146, End of the World):

Finally, we may well run a severe risk from something-we-know-not-what: something of which we can say only that it would come as a nasty surprise like the Antarctic ozone hole and that, again like the ozone hole, it would be a consequence of technological advances.

As does Bostrom (2002):

We need a catch-all category. It would be foolish to be confident that we have already imagined and anticipated all significant risks. Future technological or scientific developments may very well reveal novel ways of destroying the world.

I agree with all the points you make in the “Implications” section. I also provide some relevant ideas in Some thoughts on Toby Ord’s existential risk estimates, in the section on how Ord’s estimates (especially the relatively high estimates for “other” and “unforeseen” risks) should update our career and donation decisions.

To deal with currently-possible unknown risks, we could spend more effort thinking about possible sources of risk, but this strategy probably wouldn’t help us predict x-risks that depend on future technology.

I was confused by this claim. Wouldn’t you put risks from AI, risks from nanotechnology, and much of biorisk in the category of “x-risks that depend on future technology”, rather than currently possibly yet unrecognised risks? And wouldn’t you say that effort thinking about possible sources of risk has helped us identify and mitigate (or strategise about how to mitigate) those three risks? If so, I’d guess that similar efforts could help us with similar types of risks in future, as well.

If we expect that deliberate efforts to reduce x-risk will likely come into play before new and currently-unknown x-risks emerge, that doesn’t mean we should deprioritize unknown risks. Our actions today to prioritize unknown risks could be the very reason that such risks will not seriously threaten future civilization.

I agree. I’d also add that I think a useful way to think about this is in terms of endogenous vs exogenous increases in deliberate efforts to reduce x-risk, where the former essentially means “caused by people like us, because they thought about arguments like this”, and the latter essentially means “caused by other people or other events” (e.g., mainstream governments and academics coming to prioritise x-risk more for reasons other than the influence of EA). The more we expect exogenous increases in deliberate efforts to reduce x-risk, the less we should prioritise unknown risks. The same is not true with regards to endogenous increases, because deprioritising this area makes the endogenous increases unlikely.

(This distinction came to my mind due to SjirH making a similar distinction in relation to learning about giving opportunities.)

This is similar to discussions about how “sane” society is, and how much we should expect problems to be solved “by default”.

We developed nuclear weapons in 1945, but it was not until almost 40 years later that we realized their use could lead to a nuclear winter.

You seem to be using this data point to support the argument “There are many risks we’ve discovered only recently, and we should be worried about these and about risks we have yet to discover.” That seems fair.

But I think that, if you’re using that data point in that way, it’s probably worth also more explicitly noting the following: People were worried about existential catastrophe from nuclear war via mechanisms that we now know don’t really warrant concern (as alluded to in the Ord quote you also provide). So there seems to be a companion data point which supports the argument “In the past, many risks that were recently discovered later turned out to not be big deals, so we should be less worried about recently discovered or to-be-discovered risks than we might otherwise think.”

(And I’d say that both data points provide relatively weak evidence for their respective arguments.)

Thanks for this post—I think this is a very important topic! I largely agree that this argument deserves substantial weight, and that we should probably think more about unknown existential risks and about how we should respond to the possibility of them.

In general, the more recently we discovered a particular existential risk, the more probable it appears. I proposed that this pattern occurs because technological growth introduces increasingly-significant risks. But we have an alternative explanation: Perhaps all existential risks are unlikely, but the recently-discovered risks appear more likely due to bad priors plus wide error bars in our probability estimates.

(I’m not sure if the following is disagreeing with you or just saying something separate yet consistent)

I agree that recently discovered risks seem the largest source of existential risk, and that two plausible explanations of that are “technological growth introduces increasingly significant risks” and “we have bad priors in general and wider error bars for probability estimates about recently discovered risks than about risks we discovered earlier”. But it also seems to me that we can believe all that and yet still think there’s a fairly high chance we’ll later realise that the risks we recently discovered are less risky than we thought, and that future unknown risks in general aren’t very risky, even if we didn’t have bad priors.

Essentially, this is because, if we have a few decades or centuries without an existential catastrophe (and especially if this happens despite the development of highly advanced AI, nanotechnology, and biotech), that’ll represent new evidence not just about those particular risks, but also about how risky advanced technologies are in general and how resilient to major changes civilization is in general.

Thousands of years where asteroids could’ve wiped humanity out but didn’t updates us towards thinking extinction risk from asteroids is low. I think that, in the same way, developing more things in the reference class of “very advanced tech with hard-to-predict consequences” without an existential catastrophe occurring should update us towards thinking existential risk from developing more things like that is relatively low.

I think we can make that update in 2070 (or whatever) without saying our beliefs in 2020 were foolish, given what we knew. And I don’t think we should make that update yet, as we haven’t yet seen that evidence of “general civilizational stability” (or something like that).

Does that make sense?

Admin: The “Pamlin & Armstrong (2015)” link was broken — I updated it.