Suppose that Alice and Bob have different beliefs about existential risk over the next 50 years. While Alice thinks it comes overwhelmingly from a single threat, Bob views it as a disjunction of four independent threats.
Alice:
Threat A: 20%
Bob:
Threat A: 6%
Threat B: 6%
Threat C: 5%
Threat D: 5%
Combined risk (given independence): ~20%
Setting aside your beliefs about actual threats such as AI misalignment or nuclear war, who looks more reasonable to you?
I think many people would say Bob. While Alice’s focus on a single big threat can come across as fanatical, Bob’s broader portfolio of lower credences has a more balanced and humble look.
But I take the opposite view. Serious existential threats seem to arise rarely: it’s debatable whether we even have one yet. This makes it decidedly unlikely that we have evidence of four near-term risks of 5 per cent or more. A single big risk is much more credible (if still unlikely), given the plausible hypothesis that the risk distribution is heavy-tailed.
Existential risk communication has often given comparable attention to several unrelated threats, creating the impression that they pose risks of roughly the same order. But if the near-term risk is indeed high, it’s more likely to be concentrated in one or two threats – AI and possibly synthetic biology. So it’s better to emphasise those threats than to use broader framings that make existential risk seem more disjunctive than it is.
I’ve written a short post on the problem with disjunctive existential risk.
Suppose that Alice and Bob have different beliefs about existential risk over the next 50 years. While Alice thinks it comes overwhelmingly from a single threat, Bob views it as a disjunction of four independent threats.
Alice:
Threat A: 20%
Bob:
Threat A: 6%
Threat B: 6%
Threat C: 5%
Threat D: 5%
Combined risk (given independence): ~20%
Setting aside your beliefs about actual threats such as AI misalignment or nuclear war, who looks more reasonable to you?
I think many people would say Bob. While Alice’s focus on a single big threat can come across as fanatical, Bob’s broader portfolio of lower credences has a more balanced and humble look.
But I take the opposite view. Serious existential threats seem to arise rarely: it’s debatable whether we even have one yet. This makes it decidedly unlikely that we have evidence of four near-term risks of 5 per cent or more. A single big risk is much more credible (if still unlikely), given the plausible hypothesis that the risk distribution is heavy-tailed.
Existential risk communication has often given comparable attention to several unrelated threats, creating the impression that they pose risks of roughly the same order. But if the near-term risk is indeed high, it’s more likely to be concentrated in one or two threats – AI and possibly synthetic biology. So it’s better to emphasise those threats than to use broader framings that make existential risk seem more disjunctive than it is.