Quantifying the probability of existential catastrophe: A reply to Beard et al.

Link post

Seth Baum of GCRI has pub­lished an ex­cel­lent new pa­per. Here’s the ab­stract:

A re­cent ar­ti­cle by Beard, Rowe, and Fox (BRF) eval­u­ates ten method­olo­gies for quan­tify­ing the prob­a­bil­ity of ex­is­ten­tial catas­tro­phe. This ar­ti­cle builds on BRF’s valuable con­tri­bu­tion. First, this ar­ti­cle de­scribes the con­cep­tual and math­e­mat­i­cal re­la­tion­ship be­tween the prob­a­bil­ity of ex­is­ten­tial catas­tro­phe and the sever­ity of events that could re­sult in ex­is­ten­tial catas­tro­phe. It dis­cusses com­pli­ca­tions in this re­la­tion­ship aris­ing from catas­tro­phes oc­cur­ring at differ­ent speeds and from mul­ti­ple con­cur­rent catas­tro­phes. Se­cond, this ar­ti­cle re­vis­its the ten BRF method­olo­gies, find­ing an in­verse re­la­tion­ship be­tween a method­ol­ogy’s ease of use and the qual­ity of re­sults it pro­duces—in other words, achiev­ing a higher qual­ity of anal­y­sis will in gen­eral re­quire a larger in­vest­ment in anal­y­sis. Third, the manuscript dis­cusses the role of prob­a­bil­ity quan­tifi­ca­tion in the man­age­ment of ex­is­ten­tial risks, de­scribing why the prob­a­bil­ity is only some­times needed for de­ci­sion-mak­ing and ar­gu­ing that analy­ses should sup­port real-world risk man­age­ment de­ci­sions and not just be aca­demic ex­er­cises. If the find­ings of this ar­ti­cle are taken into ac­count, to­gether with BRF’s eval­u­a­tions of spe­cific method­olo­gies, then risk analy­ses of ex­is­ten­tial catas­tro­phe may tend to be more suc­cess­ful at un­der­stand­ing and re­duc­ing the risks.

(He also wrote a blog post about the pa­per.)

I’d highly recom­mend read­ing the whole pa­per. I’d also recom­mend the Beard et al. pa­per; I found it very use­ful when con­struct­ing a database of ex­is­ten­tial risk es­ti­mates, as well as when think­ing about the pros and cons of do­ing so. (Un­for­tu­nately, Beard et al. is be­hind a pay­wall. But I think that this freely available work­ing pa­per is es­sen­tially a draft of that pa­per, though I haven’t read it to check.)

Ex­is­ten­tial risk is a func­tion of prob­a­bil­ity of oc­cur­rence and prob­a­bil­ity of suffi­cient severity

I want to high­light and briefly com­ment on one part of the pa­per in par­tic­u­lar:

Quan­tify­ing the prob­a­bil­ity of spe­cific ex­is­ten­tial catas­tro­phe events (such as a nu­clear war or Earth-as­ter­oid col­li­sion) re­quires ad­di­tional at­ten­tion to sever­ity [of those events]. The prob­a­bil­ity can be de­com­posed into two con­stituent parts as fol­lows:

In Eq. (1) [the above equa­tion], PEC is the prob­a­bil­ity of ex­is­ten­tial catas­tro­phe from some event; P1 is the prob­a­bil­ity of the ini­tial catas­tro­phe event; and P2 is the prob­a­bil­ity that the event will re­sult in a harm greater or equal to the col­lapse of civ­i­liza­tion [Baum defers in this pa­per to Beard et al.’s non­stan­dard us­age of the term “ex­is­ten­tial risk”; see also]. For ex­am­ple, P1 could rep­re­sent the prob­a­bil­ity of nu­clear war and P2 could rep­re­sent the prob­a­bil­ity that nu­clear war would re­sult in the col­lapse of civ­i­liza­tion or worse. The oc­cur­rence of the ini­tial catas­tro­phe event does not nec­es­sar­ily en­tail the col­lapse of civ­i­liza­tion—that de­pends on how effec­tively the sur­vivors can cope with the af­ter­math of the event.

Calcu­lat­ing PEC via Eq. (1) re­quires two dis­tinct analy­ses: one for each of P1 and P2. Anal­y­sis of P1 is the anal­y­sis of the prob­a­bil­ity of ini­tial events, and can fol­low many con­ven­tions of prob­a­bil­is­tic risk anal­y­sis. In con­trast, quan­tify­ing P2 re­quires anal­y­sis of the sever­ity, with at­ten­tion to the suc­cess of catas­tro­phe sur­vivors. This is a rather differ­ent sort of anal­y­sis than is needed to quan­tify the prob­a­bil­ity of ini­tial catas­tro­phe events rep­re­sented by P1. How­ever, P2 is not equiv­a­lent to sever­ity. P2 is a prob­a­bil­ity vari­able rep­re­sent­ing the prob­a­bil­ity that the sever­ity will ex­ceed a cer­tain thresh­old. P2 can be ob­tained by cre­at­ing a prob­a­bil­ity dis­tri­bu­tion for the sever­ity of an ini­tial event and then calcu­lat­ing the por­tion of that dis­tri­bu­tion that ex­ceeds the thresh­old for ex­is­ten­tial catas­tro­phe:

In Eq. (2), P2 is as in Eq. (1); S is sever­ity of some ini­tial event; and ST is the min­i­mum sever­ity thresh­old of ex­is­ten­tial catas­tro­phe (the col­lapse of civ­i­liza­tion in BRF [again, this is non­stan­dard us­age of the term “ex­is­ten­tial catas­tro­phe”]). Eq. (2) is illus­trated in Fig. 1.

I think this is a very use­ful way to break down and think about the like­li­hoods of var­i­ous ex­is­ten­tial risks. (Though there are also of course other use­ful ways to do so.)

But what sever­ity level is suffi­cient?

How­ever, I think the above para­graphs fail to make one im­por­tant point ex­plicit: We’re un­cer­tain about what the min­i­mum thresh­old is in the first place, not just how se­vere an event will be (if it oc­curs). Both our un­cer­tainty about the thresh­old and our un­cer­tainty about the ex­pected sever­ity of an event con­tribute to our un­cer­tainty about the like­li­hood that that event would cause an ex­is­ten­tial catas­tro­phe (if it oc­curred).

For ex­am­ple, with nu­clear war, we’re un­cer­tain about how se­vere the “short-term” con­se­quences would be (e.g., how many states will col­lapse and how many peo­ple will die?), and about what sever­ity of con­se­quences would be suffi­cient for un­re­cov­er­able civ­i­liza­tional col­lapse (e.g., is the col­lapse of all states and death of 99% of the pop­u­la­tion “enough”?).

We could adapt the above di­a­gram to rep­re­sent this by also show­ing a prob­a­bil­ity dis­tri­bu­tion over pos­si­ble thresh­olds, rather than a sin­gle ver­ti­cal line (which is effec­tively a point es­ti­mate about what the thresh­old is).

I think Baum would agree with these points, given his pa­per Uncer­tain Hu­man Con­se­quences in As­teroid Risk Anal­y­sis and the Global Catas­tro­phe Thresh­old. And these points don’t con­tra­dict his state­ments in this new pa­per. One rea­son why these points don’t con­tra­dict his state­ments is that we could ar­guably just in­cor­po­rate our un­cer­tainty about the thresh­old into our un­cer­tainty about how likely it is that the event (if it oc­curs) would ex­ceed the thresh­old.

But it seems to me con­cep­tu­ally use­ful to ex­plic­itly think of the prob­a­bil­ity of an ini­tial event ex­ceed­ing the rele­vant thresh­old as be­ing de­ter­mined by both:

  • a prob­a­bil­ity dis­tri­bu­tion for the sever­ity of the event (con­di­tional on its oc­cur­rence)

  • and an­other prob­a­bil­ity dis­tri­bu­tion for where the rele­vant thresh­old is

I’d also guess that that’d some­times be use­ful when ac­tu­ally try­ing to quan­tify risk lev­els, but I’m less sure about that.