Critical Review of ‘The Precipice’: A Reassessment of the Risks of AI and Pandemics


In this es­say I will pre­sent a crit­i­cal re­sponse to Toby Ord’s re­cent book The Precipice (page num­bers re­fer to the soft cover ver­sion of this book). Rather than at­tempt­ing to ad­dress all of the many is­sues dis­cussed by Ord, I will fo­cus on what I con­sider to be one of the most crit­i­cal claims of the book. Namely, Ord claims that the pre­sent cen­tury is a time of un­prece­dented ex­is­ten­tial risk, that “we stand at a cru­cial mo­ment in the his­tory of our species” (p. 3), a situ­a­tion which is “un­sus­tain­able” (p. 4). Such views are en­cap­su­lated in Ord’s es­ti­mate of the prob­a­bil­ity of an ex­is­ten­tial catas­tro­phe over the next cen­tury, which he places at one in six. Of this roughly sev­en­teen per­cent chance, he at­tributes roughly ten per­centage points to the risks posed by un­al­igned ar­tifi­cial in­tel­li­gence, and an­other three per­centage points to the risks posed by en­g­ineered pan­demics, with most of the rest of the risk is due to un­fore­seen and ‘other’ an­thro­pogenic risks (p. 167). In this es­say I will fo­cus on the two ma­jor sources of risk iden­ti­fied by Ord, ar­tifi­cial in­tel­li­gence and en­g­ineered pan­demics. I will con­sider the anal­y­sis pre­sented by Ord, and ar­gue that by ne­glect­ing sev­eral crit­i­cal con­sid­er­a­tions, Ord dra­mat­i­cally over­es­ti­mates the mag­ni­tude of the risks from these two sources. This short es­say is in­suffi­cient to provide a full jus­tifi­ca­tion for all of my views about these risks. In­stead, my aim is to high­light some of what I be­lieve to be the ma­jor flaws and omis­sions of Ord’s ac­count, and also to out­line some of the key con­sid­er­a­tions that I be­lieve sup­port a sig­nifi­cantly lower as­sess­ment of the risks.

Why prob­a­bil­ity es­ti­mates matter

Be­fore analysing the de­tails of Ord’s claims about the risks of en­g­ineered pan­demics and un­al­igned ar­tifi­cial in­tel­li­gence, I will first ex­plain why I think it is im­por­tant to es­tab­lish as ac­cu­rate as pos­si­ble es­ti­mates of the mag­ni­tude of these ex­is­ten­tial risks. After all, it could be ar­gued that even if the risks are sig­nifi­cantly less than those pre­sented by Ord, nev­er­the­less the risks are still far higher than we would like them to be, and causes such as un­al­igned AI and en­g­ineered pan­demics are clearly ne­glected and re­quire much more at­ten­tion than they cur­rently re­ceive. As such, does it re­ally mat­ter what pre­cise prob­a­bil­ities we as­sign to these risks? I be­lieve it does mat­ter, for a num­ber of rea­sons.

First, Ord’s core the­sis in his book is that hu­man­ity faces a ‘precipice’, a rel­a­tively short pe­riod of time with uniquely high and un­sus­tain­able lev­els of ex­is­ten­tial risk. To sub­stan­ti­ate this claim, Ord needs to show not just that ex­is­ten­tial risks are high enough to war­rant our at­ten­tion, but that ex­is­ten­tial risk is much higher now than in the past, and that the risks are high enough to rep­re­sent a ‘precipice’ at which hu­man­ity stands at the edge. Ord ar­tic­u­lates this in the fol­low­ing pas­sage:

“If I’m even roughly right about their (the risks’) scale, then we can­not sur­vive many cen­turies with risk like this. It is an un­sus­tain­able level of risk. Thus, one way or an­other, this pe­riod is un­likely to last more than a small num­ber of cen­turies. Either hu­man­ity takes con­trol of its des­tiny and re­duces the risk to a sus­tain­able level, or we de­stroy our­selves.” (p. 31)

Crit­i­cal here is Ord’s link­age of the scale of the risk with our in­abil­ity to sur­vive many cen­turies of this scale of risk. He goes on to ar­gue that this is what leads to the no­tion of a precipice:

This com­par­a­tively brief pe­riod is a unique challenge in the his­tory of our species… His­to­ri­ans of the fu­ture will name this time, and schoolchil­dren will study it. But I think we need a name now. I call it the Precipice. The Precipice gives our time im­mense mean­ing. (p. 31)

Given these pas­sages, it is clear that there is a di­rect con­nec­tion be­tween the mag­ni­tude of the ex­is­ten­tial risks over the next cen­tury or so, and the ex­is­tence of a ‘precipice’ that uniquely defines our time as his­tor­i­cally spe­cial. This is a dis­tinct ar­gu­ment from the weaker claim that ex­is­ten­tial risks are far higher than we should be com­fortable with, and that more should be done to re­duce them. My ar­gu­ment in this es­say is that the main sources of the ab­nor­mally high risk iden­ti­fied by Ord, namely en­g­ineered pan­demics and un­al­igned ar­tifi­cial in­tel­li­gence, do not pose nearly as high a risk as Ord con­tends, and there­fore his ar­gu­ment that the pre­sent pe­riod con­sti­tutes a ‘precipice’ is un­per­sua­sive.

Se­cond, I think pre­cise es­ti­mates of the prob­a­bil­ities mat­ter be­cause there is a very long his­tory of pre­dict­ing the end of the world (or the end of civil­i­sa­tion, or other ex­is­ten­tial catas­tro­phes), so the baseline for ac­cu­racy of such claims is poor. As such it seems rea­son­able to ex­er­cise some scep­ti­cism and cau­tion when eval­u­at­ing such claims, and en­sure that they are based on suffi­ciently plau­si­ble ev­i­dence and rea­son­ing to be taken se­ri­ously. This is also im­por­tant for con­vinc­ing oth­ers of such risks, as ex­ag­ger­a­tion of risks to hu­man­ity is very com­mon, and is likely to re­duce the cred­i­bil­ity of those at­tempt­ing to raise aware­ness of such risks. Ord makes a similar ar­gu­ment when he ad­vises:

Don’t ex­ag­ger­ate the risks. There is a nat­u­ral ten­dency to dis­miss claims of ex­is­ten­tial risk as hy­per­bole. Ex­ag­ger­at­ing the risks plays into that, mak­ing it much harder for peo­ple to see that there is sober, care­ful anal­y­sis amidst the noise. (p. 213)

Third, I think that ac­cu­rate es­ti­mates of prob­a­bil­ities of differ­ent forms of ex­is­ten­tial risk are im­por­tant be­cause it helps us to al­ign our efforts and re­sources in pro­por­tion to the amount of risk posed by differ­ent causes. For ex­am­ple, if one type of risk is es­ti­mated to pose one hun­dred times as much risk as an­other, this im­plies a differ­ent dis­tri­bu­tion of efforts com­pared to if both causes posed roughly com­pa­rable amounts of risk. Ord makes this ar­gu­ment as fol­lows:

This vari­a­tion (in risk) makes it ex­tremely im­por­tant to pri­ori­tise our efforts on the right risks. And it also makes our es­ti­mate of the to­tal risk very sen­si­tive to the es­ti­mates of the top few risks (which are among the least well un­der­stood). So get­ting bet­ter un­der­stand­ing and es­ti­mates for those be­comes a key pri­or­ity. (p. 168)

As such, I be­lieve it is im­por­tant to care­fully con­sider the prob­a­bil­ity of var­i­ous pro­posed ex­is­ten­tial risk sce­nar­ios. In the sub­se­quent two sec­tions I will con­sider risks of en­g­ineered pan­demics and un­al­igned ar­tifi­cial in­tel­li­gence.

Eng­ineered Pandemics

Ex­tinc­tion level agent exists

One ini­tial con­sid­er­a­tion that must be ad­dressed is how likely it is that any biolog­i­cal pathogen can even kill enough peo­ple to drive hu­man­ity to ex­tinc­tion. This places an up­per limit on what any biotech­nol­ogy could achieve, re­gard­less of how ad­vanced. Note that here I am refer­ring to an agent such as a virus or bac­terium that is clearly biolog­i­cal in na­ture, even if it is en­g­ineered to be more deadly than any nat­u­rally-oc­cur­ring pathogen. I am not in­clud­ing en­tities that are non-biolog­i­cal in na­ture, such as ar­tifi­cial nan­otech­nol­ogy or other chem­i­cal agents. Whilst it is im­pos­si­ble to de­ter­mine the ul­ti­mate limits of biol­ogy, one rele­vant point of com­par­i­son is the most deadly nat­u­rally-oc­cur­ring in­fec­tious dis­ease. To my knowl­edge, the high­est fatal­ity rate for any in­fec­tious biolog­i­cal agent that is read­ily trans­mis­si­ble be­tween liv­ing hu­mans is the Zaire ebo­lavirus, with a fatal­ity rate of around 90%. It is un­clear whether such a high fatal­ity rate would be sus­tained out­side of the so­cial and cli­mac­tic en­vi­ron­ment of West Africa whence the dis­ease origi­nated, but nev­er­the­less we can con­sider this to be a plau­si­ble baseline for the most deadly known hu­man in­fec­tious pathogen. Crit­i­cally, it ap­pears un­likely that the death of even 90% of the world pop­u­la­tion would re­sult in the ex­tinc­tion of hu­man­ity. Death rates of up to 50% dur­ing the Black Death in Europe do not ap­pear to have even come close to caus­ing civil­i­sa­tional col­lapse in that re­gion, while pop­u­la­tion losses of up to 90% in Me­soamer­ica over the course of the in­va­sion and plagues of the 16th cen­tury did not lead to the end of civ­i­liza­tion in those re­gions (though so­cial and poli­ti­cal dis­rup­tion dur­ing these events were mas­sive).

If we think the min­i­mal vi­able hu­man pop­u­la­tion is roughly 7,000 (which is near the up­per end of the figures cited by Ord (p. 41), though rounded for sim­plic­ity), then a pathogen would need to di­rectly or in­di­rectly lead to the deaths of more than 99.9999% of the cur­rent world pop­u­la­tion in or­der to lead to hu­man ex­tinc­tion. One could ar­gue that the pathogen would only need to di­rectly cause a much smaller num­ber of deaths, with the re­main­ing deaths caused by sec­ondary dis­rup­tions such as war or famine. How­ever to me this seems very un­likely, con­sid­er­ing that such a dev­as­tat­ing pathogen would sig­nifi­cantly im­pair the abil­ity of na­tions to wage war, and it is hard to see how war­fare would af­fect all ar­eas of the globe suffi­ciently to bring about such sig­nifi­cant pop­u­la­tion loss. Global famine also seems un­likely, given that the greater the num­ber of pan­demic deaths, the more food stores would be available to sur­vivors. Per­haps the most dev­as­tat­ing sce­nario would be a mas­sive global pan­demic fol­lowed by a full-scale nu­clear war, though it is un­clear why should a nu­clear ex­change would fol­low a pan­demic. One can of course de­vise var­i­ous hy­po­thet­i­cal sce­nar­ios, but over­all it ap­pears to me that a pathogen would have to have an ex­tremely high fatal­ity rate in or­der to have the po­ten­tial to cause hu­man ex­tinc­tion.

In ad­di­tion to a high fatal­ity rate, an ex­tinc­tion-level pathogen would also have to be suffi­ciently in­fec­tious such that it would be able to spread rapidly through hu­man pop­u­la­tions. It would need to have a long enough in­cu­ba­tion time such that in­fected per­sons can travel and in­fect more peo­ple be­fore they can be iden­ti­fied and quaran­tined. It would also need to be able to sur­vive and prop­a­gate in a wide range of tem­per­a­tures and cli­mac­tic con­di­tions. Fi­nally, it would also need to be suffi­ciently dan­ger­ous to a wide range of ages and ge­netic pop­u­la­tions, since any pock­ets of im­mu­nity would ren­der ex­tinc­tion con­sid­er­ably less likely. Over­all, it is highly un­clear whether any biolog­i­cal agent with all these prop­er­ties is even pos­si­ble. In par­tic­u­lar, pathogens which are suffi­ciently viru­lent to cause 99% or more fatal­ity rates are likely to place such a bur­den on hu­man phys­iol­ogy such that they would have a short in­cu­ba­tion time, po­ten­tially ren­der­ing it eas­ier to quaran­tine in­fected per­sons. Of course we do not know what is pos­si­ble at the limits of biol­ogy, but given the ex­treme prop­er­ties re­quired of such an ex­tinc­tion-level pathogen, in my view it is very un­likely that such a pathogen is even pos­si­ble.

Ex­tinc­tion level agent tech­nolog­i­cally feasible

Even if biolog­i­cal agents with the po­ten­tial of wiping out hu­man­ity are the­o­ret­i­cally pos­si­ble, the ques­tion re­mains as to how long it will be un­til it be­comes tech­nolog­i­cally fea­si­ble to en­g­ineer such an agent. While our cur­rent sci­en­tific un­der­stand­ing places sig­nifi­cant limi­ta­tions on what can be en­g­ineered, Ord ar­gues that “it is not twen­tieth-cen­tury bioweaponry that should alarm us, but the next hun­dred years of im­prove­ments” (p. 133), which in­di­cates that he be­lieves that biotech­nolog­i­cal ad­vances over the next cen­tury are likely to en­able the cre­ation of a much wider range of dan­ger­ous biolog­i­cal agents. Of course, it is im­pos­si­ble to know how rapidly such tech­nol­ogy will de­velop in the com­ing decades, how­ever I be­lieve that Ord over­states the cur­rent ca­pa­bil­ities of such tech­nol­ogy, and un­der­es­ti­mates the challenges in de­vel­op­ing pathogens of dra­mat­i­cally greater lethal­ity than ex­ist­ing nat­u­ral agents.

For ex­am­ple, Ord states that it is pos­si­ble to “cre­ate en­tire func­tional viruses from their writ­ten code” (p. 128). I be­lieve this claim is mis­lead­ing, es­pe­cially when read alongside Ord’s con­cern about ease of ob­tain­ing syn­the­sised DNA, as it can po­ten­tially be read as as­sert­ing that viruses can be cre­ated us­ing en­tirely syn­thetic means us­ing only their DNA. This is false, as the meth­ods cited by Ord de­scribe tech­niques in which syn­the­sised viral DNA is cul­tured cel­lu­lar ex­tracts, which as Ord also notes is not triv­ial and re­quires care­ful tech­nique (p. 359). This ap­proach still re­lies crit­i­cally on util­is­ing the ri­bo­somes and other cel­lu­lar ma­chin­ery to trans­late viral DNA and pro­duce the needed viral pro­teins. It does not in­volve the de­gree of con­trol or un­der­stand­ing of the pre­cise molec­u­lar pro­cesses in­volved that would be im­plied if an in­tact virus could be pro­duced from its DNA us­ing en­tirely syn­thetic means.

Ord also cites the 2012 ex­per­i­ments of Ron Fouch­ier, who con­ducted a gain-of-func­tion ex­per­i­ment with H5N1 in­fluenza in fer­rets. Ord states that “by the time it passed to the fi­nal fer­ret, his strain of H5N1 had be­come di­rectly trans­mis­si­ble be­tween mam­mals” (p. 129). While tech­ni­cally cor­rect, I be­lieve this claim is mis­lead­ing, since only a few sen­tences prior Ord states that this strain of in­fluenza had an es­ti­mated 60% mor­tal­ity rate in hu­mans, im­ply­ing that this would also ap­ply to an air­borne var­i­ant of the same virus. How­ever in Fouch­ier’s study, it is re­ported that “al­though the six fer­rets that be­came in­fected via res­pi­ra­tory droplets or aerosol also dis­played lethargy, loss of ap­petite, and ruffled fur, none of these an­i­mals died within the course of the ex­per­i­ment.” Fur­ther­more, the mere pos­si­bil­ity of air­borne trans­mis­sion says noth­ing about the effi­ciency of this trans­mis­sion mechanism. As re­ported in the pa­per:

Although our ex­per­i­ments showed that A/​H5N1 virus can ac­quire a ca­pac­ity for air­borne trans­mis­sion, the effi­ciency of this mode re­mains un­clear. Pre­vi­ous data have in­di­cated that the 2009 pan­demic A/​H1N1 virus trans­mits effi­ciently among fer­rets and that naïve an­i­mals shed high amounts of virus as early as 1 or 2 days af­ter ex­po­sure. When we com­pare the A/​H5N1 trans­mis­sion data with that of [an­other pa­per]..., the data shown in Figs. 5 and 6 sug­gest that A/​H5N1 air­borne trans­mis­sion was less ro­bust, with less and de­layed virus shed­ding com­pared with pan­demic A/​H1N1 virus.

Th­ese qual­ifi­ca­tions illus­trate the fun­da­men­tal point that most biolog­i­cal sys­tems ex­ist as a set of trade­offs and bal­ances be­tween com­pet­ing effects and con­flict­ing needs. Thus chang­ing one as­pect of a pathogen, such as its mode of trans­mis­sion, is likely to have effects on other as­pects of the pathogen, such as its lethal­ity, in­cu­ba­tion pe­riod, sus­cep­ti­bil­ity to im­mune sys­tem at­tack, or sur­vival out­side a host. In the­ory it may be pos­si­ble to de­sign a pathogen with prop­er­ties op­ti­mised to be as lethal to hu­mans as pos­si­ble, but do­ing so would re­quire far greater un­der­stand­ing of pro­tein fold­ing path­ways, pro­tein-pro­tein in­ter­ac­tions, gene ex­pres­sion, mechanisms of pathogen in­va­sion, im­mune sys­tem eva­sion strate­gies, and other such fac­tors than is cur­rently pos­sessed. Thus it is by no means clear that Ord is cor­rect when he states that “this progress in biotech­nol­ogy seems un­likely to fiz­zle out soon: there are no in­sur­mountable challenges loom­ing; no fun­da­men­tal laws block­ing fur­ther de­vel­op­ments” (p. 128). In­deed, I be­lieve there are many fun­da­men­tal challengers and gaps in our un­der­stand­ing which pre­vent the de­vel­op­ment of pathogens with ar­bi­trar­ily speci­fied prop­er­ties.

Ex­tinc­tion level agent pro­duced and delivered

Even if was tech­nolog­i­cally pos­si­ble to pro­duce a pathogen ca­pa­ble of caus­ing hu­man ex­tinc­tion, the re­search, pro­duc­tion, and dis­tri­bu­tion of such an in­fec­tious agent would still ac­tu­ally need to be car­ried out by an or­gani­sa­tion with the ca­pa­bil­ities and de­sire to do so. While Ord’s ex­am­ple of the Aum Shin­rikyo cult does demon­strate that such groups ex­ist, the very small num­ber of such at­tacks his­tor­i­cally ap­pears to in­di­cate that such groups do not ex­ist in large num­bers. Very few ide­olo­gies have an in­ter­est in bring­ing hu­man­ity to an end through vi­o­lent means. In­deed as Ord notes:

For all our flir­ta­tion with biowar­fare, there ap­pear to have been rel­a­tively few deaths from ei­ther ac­ci­dents or use… Ex­actly why this is so is un­clear. One rea­son may be that bioweapons are un­re­li­able and prone to back­firing, lead­ing states to use other weapons in prefer­ence. (p. 132)

Ord par­tially coun­ters this ob­ser­va­tion by ar­gu­ing that the sever­ity of events such as ter­ror­ist at­tacks and in­ci­dents of biowar­fare fol­low a power law dis­tri­bu­tion, with very rare, very high im­pact events mean­ing that the av­er­age size of past events will un­der­es­ti­mate the ex­pected size of fu­ture events. How­ever this re­sponse does not seem to ad­dress the core ob­ser­va­tion that bioweapons have proven very hard to con­trol, and that very few agents or or­gani­sa­tions have any in­ter­est in un­leash­ing a pathogen that kills hu­mans in­dis­crim­i­nately. This ap­pears to be re­flected in the fact that as far as is pub­li­cly known, very few at­tempts have even been made to de­ploy such weapons in mod­ern times. I thus be­lieve that we have good rea­son to think that the num­ber of peo­ple and amount of effort de­voted to de­vel­op­ing such dan­ger­ous bioweapons is likely to be low, es­pe­cially for non-state ac­tors.

Fur­ther­more, Ord fails to con­sider the prac­ti­cal difficul­ties of de­vel­op­ing and re­leas­ing a pathogen suffi­ciently deadly to cause hu­man ex­tinc­tion. In par­tic­u­lar, de­vel­op­ing a novel or­ganism would re­quire lengthy re­search and ex­ten­sive test­ing. Even if all the req­ui­site sup­plies, tech­nol­ogy, and ex­per­tise over a pe­riod of time could be ob­tained with­out arous­ing enough sus­pi­cion for the pro­ject to be in­ves­ti­gated and shut down, there still re­mains the challenge of how such a pathogen could be tested. No an­i­mal model is perfect, and so any novel pathogen would (just like vac­cines and other med­i­cal treat­ments) need to be tested on large num­bers of hu­man sub­jects, and likely ad­justed in re­sponse to re­sults. It would need to be tri­aled in differ­ent en­vi­ron­ments and cli­mates to de­ter­mine whether it would spread suffi­ciently rapidly and sur­vive out­side a host long enough. Without such tests, it is vir­tu­ally im­pos­si­ble that an untested novel pathogen would be suffi­ciently op­ti­mised to kill enough peo­ple across a wide enough range of en­vi­ron­ments to cause hu­man ex­tinc­tion. How­ever, it is hard to see how it would be pos­si­ble to carry out such wide­spread test­ing with a di­verse enough range of sub­jects with­out draw­ing the at­ten­tion of au­thor­i­ties.

A rogue state such as North Korea might be able to cir­cum­vent this par­tic­u­lar prob­lem, how­ever that raises as range of new difficul­ties, such as why it would ever be in the in­ter­est of a state ac­tor (as op­posed to a death cult ter­ror­ist group) to de­velop such a deadly, in­dis­crim­i­nate pathogen. Ord raises the pos­si­bil­ity of its use as a de­ter­rent (akin to the de­ter­rence func­tion of nu­clear weapons), but the anal­ogy does not ap­pear to hold up. Nu­clear weapons work as a de­ter­rent be­cause their pos­ses­sion can be pub­li­cly demon­strated (by test­ing), their dev­as­tat­ing im­pact is widely known, and there is no prac­ti­cal defence against them. None of these prop­er­ties are true of an ex­tremely lethal novel pathogen. A rogue state would have great difficulty prov­ing that they pos­sessed such a weapon with­out ac­tu­ally mak­ing available enough in­for­ma­tion about the pathogen, such that the world would likely be able to de­velop coun­ter­mea­sures to that par­tic­u­lar pathogen. As such, it does not ap­pear fea­si­ble to use bioweapons as effec­tive de­ter­rents, which may partly ex­plain why de­spite ex­ten­sive re­search into the pos­si­bil­ity, no states have yet used them in this man­ner. As a re­sult of these con­sid­er­a­tions, I con­clude that even if it were tech­nolog­i­cally pos­si­ble to de­velop a pathogen suffi­ciently lethal to cause hu­man ex­tinc­tion, it is un­likely that any­one would ac­tu­ally have both the de­sire and the abil­ity to suc­cess­fully pro­duce and de­liver the pathogen.

Failure of timely pub­lic policy response

The re­lease of a pathogen that has the po­ten­tial to cause hu­man ex­tinc­tion in it­self does not im­ply that hu­man ex­tinc­tion would in­evitably oc­cur. Whether this would fol­low de­pends on the ex­tent of the gov­ern­men­tal and so­cietal re­sponses to the out­break of the novel pan­demic, such as quaran­tines, wide­spread test­ing, and con­tact trac­ing. In con­sid­er­ing the bal­ance of pos­i­tive and nega­tive effects that or­gani­sa­tional and civ­i­liza­tion ad­vances have had on the abil­ity to re­spond to the risk of pathogens, Ord states that “it is hard to know whether these com­bined effects have in­creased or de­creased the ex­is­ten­tial risk from pan­demics” (p. 127). This ar­gu­ment, how­ever, seems im­plau­si­ble, since deaths from in­fec­tious dis­eases and pan­demics in par­tic­u­lar have de­creased in re­cent cen­turies, with no ma­jor pan­demics in Western Europe since the early eigh­teenth cen­tury. The dis­ap­pear­ance of plague from Western Europe, while still not well un­der­stood, plau­si­bly may have been caused at least in part by the im­prove­ment of quaran­tine and pub­lic policy re­sponses to plague. In the US, the crude death rate from in­fec­tious dis­eases fell by about 90% over the course of the twen­tieth cen­tury. Fur­ther­more, a suc­cess­ful pub­lic policy re­sponse to a pathogen out­break in even a sin­gle coun­try would likely be enough to pre­vent ex­tinc­tion, even if most coun­tries failed to en­act a suffi­cient pub­lic policy re­sponse. As such, I be­lieve it is un­likely that even an ex­tinc­tion-level novel pathogen would be able to suffi­ciently evade all pub­lic health re­sponses so as to cause hu­man ex­tinc­tion.

Failure of timely biomed­i­cal response

In ad­di­tion to the failure of pub­lic policy re­sponses, ex­tinc­tion of hu­man­ity by a novel pathogen would also re­quire the failure of any biomed­i­cal re­sponse to the pan­demic. Ord be­lieves that as biolog­i­cal tech­niques be­come eas­ier and cheaper, they be­come ac­cessible to more and more peo­ple, and hence rep­re­sent a greater and greater risk. He ar­gues:

As the pool of peo­ple with ac­cess to a tech­nique grows, so does the chance it con­tains some­one with ma­lign in­tent. (p. 134)

This ar­gu­ment, how­ever, ap­pears to only con­sider one side of the is­sue. As the pool of peo­ple with ac­cess to a tech­nique grows, so too does the num­ber of peo­ple who wish to use that tech­nique to do good. This in­cludes de­vel­op­ing tech­niques and tech­nolo­gies for more eas­ily de­tect­ing, con­trol­ling, and cur­ing in­fec­tious dis­eases. It sur­prises me that Ord never men­tions this, since the de­vel­op­ment of biomed­i­cal tech­nolo­gies does not only mean that there is greater scope for use of the tech­nol­ogy to cause dis­ease, but also greater scope for use new tech­niques to pre­vent and cure dis­ease. In­deed, since the pre­ven­tion of dis­ease re­ceives far more re­search at­ten­tion that caus­ing dis­ease, it seems rea­son­able to as­sume that our abil­ities to de­vel­op­ment treat­ments, tests, and vac­cines for dis­eases will de­velop more rapidly than our abil­ities to cause dis­ease. There are a range of emerg­ing biomed­i­cal tech­nolo­gies that promise to greater im­prove our abil­ity to fight ex­ist­ing and novel dis­eases, in­clud­ing trans­mis­si­ble vac­cines, ra­tio­nal de­sign of drugs, and re­verse vac­ci­nol­ogy. As such, I re­gard it un­likely that if biomed­i­cal tech­nol­ogy had ad­vanced suffi­ciently to be able to pro­duce an ex­tinc­tion-level pathogen, it would nev­er­the­less fail to de­velop suffi­cient coun­ter­mea­sures to the pathogen to at least pre­vent full hu­man ex­tinc­tion.

Unal­igned Ar­tifi­cial Intelligence

AI ex­perts and AI timelines

Although Ord ap­peals to sur­veys of AI re­searchers as ev­i­dence of the plau­si­bil­ity of the de­vel­op­ment of su­per­hu­man ar­tifi­cial in­tel­li­gence in the next cen­tury, ex­perts in ar­tifi­cial in­tel­li­gence do not have a good track record of pre­dict­ing fu­ture progress in AI. Mas­sively in­flated ex­pec­ta­tions of the ca­pa­bil­ities of sym­bolic AI sys­tems in the 1950s and 1960s, and of ex­pert sys­tems in the 1980s, are well-known ex­am­ples of this. More gen­er­ally, it is un­clear why we should even ex­pect AI re­searchers to have any par­tic­u­lar knowl­edge about the fu­ture tra­jec­to­ries of AI ca­pa­bil­ities. Such re­searchers study and de­velop par­tic­u­lar statis­ti­cal and com­pu­ta­tional tech­niques to solve spe­cific types of prob­lems. I am not aware of any fo­cus of their train­ing on ex­trap­o­lat­ing tech­nolog­i­cal trends, or in in­ves­ti­ga­tions his­tor­i­cal case stud­ies of tech­nolog­i­cal change. In­deed, it would seem that cog­ni­tive psy­chol­o­gists or cog­ni­tive neu­ro­scien­tists might be bet­ter placed (al­though prob­a­bly still not very well placed) to make judge­ments about the bound­aries of hu­man ca­pa­bil­ity and what would be re­quired for these to be ex­ceeded in a wide range of tasks, since AI re­searchers have no par­tic­u­lar ex­per­tise in the limits of hu­man abil­ity. AI re­searchers gen­er­ally only con­sider hu­man-level perfor­mance in the con­text of baseline lev­els of perfor­mance on well-defined tasks such as image recog­ni­tion, cat­e­gori­sa­tion, or game-play­ing. This is far re­moved from be­ing able to make judge­ments about when AIs would be able to out­perform hu­mans on ‘ev­ery task’. For ex­am­ple, do AI re­searchers re­ally have any ex­per­tise on when AIs are likely to over­take hu­man abil­ity to do philos­o­phy, serve as poli­ti­cal lead­ers, com­pose a novel, or teach high school math­e­mat­ics? Th­ese are sim­ply not ques­tions that are stud­ied by AI re­searchers, and there­fore I don’t see any rea­son why they should be re­garded as hav­ing spe­cial knowl­edge about them. Th­ese con­cerns are fur­ther em­pha­sised by the in­con­sis­tency of re­searcher re­sponses to AI timeline sur­veys:

Asked when an AI sys­tem would be ’able to ac­com­plish ev­ery task bet­ter and more cheaply than hu­man work­ers, on av­er­age they es­ti­mated a 50 per­cent change of this hap­pen­ing by 2061. (p. 141)

How­ever in a foot­note Ord notes:

Note also that this es­ti­mate may be quite un­sta­ble. A sub­set of the par­ti­ci­pants were asked a slightly differ­ent ques­tion in­stead (em­pha­sis­ing the em­ploy­ment con­se­quences by talk­ing of all oc­cu­pa­tions in­stead of all tasks). Their time by which there would be a 50% chance of this stan­dard be­ing met was 2138, with a 10% chance of it hap­pen­ing as early as 2036. (p. 362)

Another fac­tor highly per­ti­nent to es­tab­lish­ing the rele­vant set of ex­perts con­cerns how the cur­rent top­ics re­searched by AI re­searchers re­late to the even­tual set of meth­ods and tech­niques even­tu­ally used in build­ing an AGI. Ord seems to think that de­vel­op­ments of cur­rent meth­ods may be suffi­cient to de­velop AGI:

One of the lead­ing paradigms for how we might even­tu­ally cre­ate AGI com­bines deep learn­ing with an ear­lier idea called re­in­force­ment learn­ing. (p. 143)

How­ever such cur­rent meth­ods, in par­tic­u­lar deep learn­ing, are known to be sub­ject to a wide range of limi­ta­tions. Ma­jor con­cerns in­clude the ease with which ad­ver­sar­ial ex­am­ples can be used to ‘fool’ net­works into mis­clas­sify­ing ba­sic stim­uli, the lack of es­tab­lished meth­ods for in­te­grat­ing syn­tac­ti­cally-struc­tured in­for­ma­tion with neu­ral net­works, the fact that deep learn­ing is task-spe­cific and does not gen­er­al­ise well, the in­abil­ity of deep learn­ing sys­tems to de­velop hu­man-like ‘un­der­stand­ing’ that per­mits ro­bust in­fer­ences about the world, and the re­quire­ment for very large datasets for deep learn­ing al­gorithms to be trained on. While it re­mains pos­si­ble that all these limi­ta­tions may be over­come in the fu­ture, at pre­sent they rep­re­sent deep the­o­ret­i­cal limi­ta­tions of cur­rent meth­ods, and as such I see lit­tle rea­son to ex­pect they can be over­come with­out the de­vel­op­ment of sub­stan­tially new and in­no­va­tive con­cepts and tech­niques. If this is cur­rent, then there seems lit­tle rea­son to ex­pect that AI re­searchers to have any ex­per­tise in pre­dict­ing when such de­vel­op­ments are likely to take place. AI re­searchers study cur­rent tech­niques, but if (as I have ar­gued) such tech­niques are fun­da­men­tally in­ad­e­quate for the de­vel­op­ment of true AGI, then such ex­per­tise is of limited rele­vance in as­sess­ing plau­si­ble AI timelines.

One ar­gu­ment that Ord gives in ap­par­ent sup­port of the no­tion that cur­rent meth­ods may in prin­ci­ple be suffi­cient for the de­vel­op­ment of AGI re­lates to the suc­cess of us­ing deep neu­ral net­works and re­in­force­ment learn­ing to train ar­tifi­cial agents to play Atari games:

The Atari-play­ing sys­tems learn and mas­ter these games di­rectly from the score and the raw pix­els on the screen. They are a proof of con­cept for ar­tifi­cial gen­eral agents: learn­ing to con­trol the world from raw vi­sual in­put; achiev­ing their goals across a di­verse range of en­vi­ron­ments. (p. 141)

I be­lieve this is a gross over­state­ment. While these de­vel­op­ments are im­pres­sive, they in no way provide a proof of con­cept for ‘ar­tifi­cial gen­eral agents’, any­more than pro­grams de­vel­oped in the 1950s and 1960s to solve gram­mat­i­cal or ge­o­met­ric prob­lems in sim­ple en­vi­ron­ments pro­vided such a proof of con­cept. Atari games are highly sim­plified en­vi­ron­ments with com­par­a­tively few de­grees of free­dom, the num­ber of pos­si­ble ac­tions is highly limited, and where a clear mea­sure of suc­cess (score) is available. Real-world en­vi­ron­ments are ex­tremely com­pli­cated, with a vast num­ber of pos­si­ble ac­tions, and of­ten no clear mea­sure of suc­cess. Uncer­tainty also plays lit­tle di­rect role in Atari games, since a com­plete pic­ture of the cur­rent games­pace is available to the agent. In the real world, all in­for­ma­tion gained from the en­vi­ron­ment is sub­ject to er­ror, and must be care­fully in­te­grated to provide an ap­prox­i­mate model of the en­vi­ron­ment. Given these con­sid­er­a­tions, I be­lieve that Ord over­states how close we cur­rently are to achiev­ing su­per­hu­man ar­tifi­cial in­tel­li­gence, and un­der­states the difficul­ties that scal­ing up cur­rent tech­niques would face in at­tempt­ing to achieve this goal.

AI has the power to usurp humanity

Ord ar­gues that ar­tifi­cial in­tel­li­gence that was more in­tel­li­gent than hu­mans would be able to usurp hu­man­ity’s po­si­tion as the most pow­er­ful species on Earth:

What would hap­pen if some­time this cen­tury re­searchers cre­ated an ar­tifi­cial gen­eral in­tel­li­gence sur­pass­ing hu­man abil­ities in al­most ev­ery do­main? In this act of cre­ation, we would cede our sta­tus as the most in­tel­li­gent en­tities on Earth. So with­out a very good plan to keep con­trol, we should also ex­pect to cede our sta­tus as the most pow­er­ful species, and the one that con­trols its own des­tiny. (p. 143)

The as­sump­tion be­hind this claim ap­pears to be that in­tel­li­gence alone is the crit­i­cal de­ter­min­ing fac­tor be­hind which species or en­tity main­tains con­trol over Earth’s re­sources and fu­ture. This premise, how­ever, con­flicts with what Ord says ear­lier in the book:

What set us (hu­man­ity) apart was not phys­i­cal, but men­tal—our in­tel­li­gence, cre­ativity, and lan­guage...each hu­man’s abil­ity to co­op­er­ate with the dozens of other peo­ple in their band was unique among large an­i­mals. (p. 12)

Here Ord iden­ti­fies not only in­tel­li­gence, but also cre­ativity and abil­ity to co­op­er­ate with oth­ers as crit­i­cal to the suc­cess of hu­man­ity. This seems con­sis­tent with the fact that hu­man in­tel­li­gence, as far as can be de­ter­mined, has not fun­da­men­tally changed over the past 10,000 years, even while our power and ca­pa­bil­ities have dra­mat­i­cally in­creased. Ob­vi­ously, what has changed is our abil­ity to co­op­er­ate at much larger scales, and also our abil­ity to build upon the achieve­ments of pre­vi­ous gen­er­a­tions to grad­u­ally in­crease our knowl­edge, and build up more effec­tive in­sti­tu­tions and prac­tices. Given these con­sid­er­a­tions, it seems far from ob­vi­ous to me that there mere ex­is­tence of an agent more in­tel­li­gent than an in­di­vi­d­ual hu­man would have the abil­ity to usurp hu­man­ity’s po­si­tion. In­deed, Ord’s own ex­am­ples seem to fur­ther em­pha­sise this point:

His­tory already in­volves ex­am­ples of in­di­vi­d­u­als with hu­man-level in­tel­li­gence (Hitler, Stalin, Genghis Khan) scal­ing up from the power of an in­di­vi­d­ual to a sub­stan­tial frac­tion of all global power. (p. 147)

Whilst we have no clear data on the in­tel­li­gence of these three in­di­vi­d­u­als, what does seem clear is that none of them achieved the po­si­tions they did by acts of profound in­tel­lect. They were ca­pa­ble men, with Stalin in par­tic­u­lar be­ing very widely read, and Hitler known to have a sharp mem­ory for tech­ni­cal de­tails, nev­er­the­less they were far from be­ing the great­est minds of their gen­er­a­tion. Nor did they achieve their po­si­tions by ‘scal­ing up’ from an in­di­vi­d­ual to world su­per­power. I think it is more ac­cu­rate to say that they used their in­di­vi­d­ual tal­ents (mil­i­tary lead­er­ship for Genghis Khan, ad­minis­tra­tive abil­ity and poli­ti­cal schem­ing for Stalin, and or­a­tory and poli­ti­cal schem­ing for Hitler) to gain con­trol over ex­ist­ing power struc­tures (re­spec­tively Mon­gol tribes, the Soviet gov­ern­ment, and the Ger­man gov­ern­ment). They did not build these things from scratch them­selves (though Genghis Khan did es­tab­lish a unified Mon­gol state, so comes closer than the oth­ers), but were able to hi­jack ex­ist­ing sys­tems and con­vince enough peo­ple to fol­low their lead­er­ship. Th­ese skills may be re­garded as a sub­set of a very broad no­tion of in­tel­li­gence, but do not seem to cor­re­spond very closely at all to the way we nor­mally use the word ‘in­tel­li­gence’, nor do they seem likely to be the sorts of things AIs would be very good at do­ing.

Lack­ing a phys­i­cal body to in­ter­act with peo­ple, it is hard to see how an AI could in­spire the same lev­els of loy­alty and fear that these three lead­ers (and many oth­ers like then) re­lied upon in their sub­or­di­nates and fol­low­ers. Of course, AIs could ma­nipu­late hu­mans to do this job for them, but this would raise an im­mense difficulty of en­sur­ing that their hu­man pawns do not usurp their au­thor­ity, which would be very difficult if all the hu­mans that the AI is at­tempt­ing to con­trol do not ac­tu­ally have any per­sonal loy­alty for the AI it­self. Per­haps the AI could pit mul­ti­ple hu­mans against one an­other and re­tain con­trol over them in this man­ner (in­deed that is effec­tively what Hitler did with his sub­or­di­nates), how­ever do­ing so gen­er­ally re­quires some de­gree of trust and loy­alty on be­half of one’s sub­or­di­nates to be sus­tain­able. Such meth­ods are also very difficult to man­age (such as the need to pre­vent plots by sub­or­di­nates against the leader), and place clear limits on how effec­tively the cen­tral ruler can per­son­ally con­trol ev­ery­thing. Of course one could always say ‘if an AI is in­tel­li­gent enough it can solve these prob­lems’, but my ar­gu­ment is pre­cisely that it is not at all clear to me that ‘in­tel­li­gence’ is even the key fac­tor de­ter­min­ing suc­cess. A cer­tain level of in­tel­li­gence is needed, but var­i­ous forms of sub­tle in­ter­per­sonal skills dis­tinct from in­tel­li­gence seem far more im­por­tant in ac­quiring and main­tain­ing their po­si­tions, skills which a non-em­bod­ied AI would face par­tic­u­lar difficulty in ac­quiring.

Over­all, I am not con­vinced that the mere ex­is­tence of a highly-in­tel­li­gent AI would im­ply any­thing about the abil­ity of that AI to ac­quire sig­nifi­cant power over hu­man­ity. Gain­ing power re­quires much more than in­di­vi­d­ual in­tel­li­gence, but also the abil­ity to co­or­di­nate large num­bers of peo­ple, to ex­er­cise cre­ativity, to in­spire loy­alty, to build upon past achieve­ments, and many oth­ers. I am not say­ing that an AI could not do these things, only that they would not au­to­mat­i­cally be able to do these things by be­ing very in­tel­li­gent, nor would these things nec­es­sar­ily be able to be done very quickly.

AI has rea­son to usurp humanity

Although Ord’s gen­eral case for con­cern about AI does not ap­peal to any spe­cific vi­sion for what AI might look like, an anal­y­sis of the claims that he makes in­di­cates that his ar­gu­ments are mostly rele­vant to a spe­cific type of agent based on re­in­force­ment learn­ing. He says:

One of the lead­ing paradigms for how we might even­tu­ally cre­ate AGI com­bines deep learn­ing with an ear­lier idea called re­in­force­ment learn­ing… un­for­tu­nately, nei­ther of these meth­ods can be eas­ily scaled up to en­code hu­man val­ues in the agent’s re­ward func­tion. (p. 144)

While Ord pre­sents this as merely a ‘lead­ing paradigm’, sub­se­quent dis­cus­sion ap­pears to as­sume that an AI would likely em­body this paradigm. For ex­am­ple he re­marks:

An in­tel­li­gent agent would also re­sist at­tempts to change its re­ward func­tion to some­thing more al­igned with hu­man val­ues. (p. 145)

Similarly he ar­gues:

The real is­sue is that AI re­searchers don’t yet know how to make a sys­tem which, upon notic­ing this mis­al­ign­ment, up­dates its ul­ti­mate val­ues to al­ign with ours rather than up­dat­ing its in­stru­men­tal goals to over­come us. (p. 146)

While this seems plau­si­ble in the case of a re­in­force­ment learn­ing agent, it seems far less clear that it would ap­ply to an­other form of AI. In par­tic­u­lar, it is not even clear if hu­mans ac­tu­ally posses any­thing that cor­re­sponds to a ‘re­ward func­tion’, nor is it clear that such a thing is im­mutable with ex­pe­rience or over the lifes­pan. To as­sume that an AI would have such a thing there­fore is to make spe­cific as­sump­tions about the form such an AI would take. This is also ap­par­ent when Ord ar­gues:

It (the AI) would seek to ac­quire ad­di­tional re­source, com­pu­ta­tional, phys­i­cal or hu­man, as these would let it bet­ter shape the world to re­ceive higher re­ward. (p. 145)

Again, this re­mark seems ex­plic­itly to as­sume that the AI is max­imis­ing some kind of re­ward func­tion. Hu­mans of­ten act not as max­imisers but as satis­ficers, choos­ing an out­come that is good enough rather than search­ing for the best pos­si­ble out­come. Often hu­mans also act on the ba­sis of habit or fol­low­ing sim­ple rules of thumb, and are of­ten risk averse. As such, I be­lieve that to as­sume that an AI agent would be nec­es­sar­ily max­imis­ing its re­ward is to make fairly strong as­sump­tions about the na­ture of the AI in ques­tion. Ab­sent these as­sump­tions, it is not ob­vi­ous why an AI would nec­es­sar­ily have any par­tic­u­lar rea­son to usurp hu­man­ity.

Re­lated to this ques­tion about the na­ture of AI mo­ti­va­tions, I was sur­prised that (as far as I could find) Ord says noth­ing about the pos­si­ble de­vel­op­ment of ar­tifi­cial in­tel­li­gence through the av­enue of whole brain em­u­la­tion. Although cur­rently in­fea­si­ble, simu­la­tion of the neu­ral ac­tivity of an en­tire hu­man brain is po­ten­tial route to AI which re­quires only very min­i­mal the­o­ret­i­cal as­sump­tions, and no ma­jor con­cep­tual break­throughs. A low-level com­puter simu­la­tion of the brain would only re­quire suffi­cient scan­ning re­s­olu­tion to mea­sure neu­ral con­nec­tivity and pa­ram­e­ters of neu­ron phys­iol­ogy, and suffi­cient com­put­ing power to run the simu­la­tion in rea­son­able time. Plau­si­ble es­ti­mates have been made which in­di­cate that ex­trap­o­lat­ing from cur­rent trends, such tech­nolo­gies are likely to be de­vel­oped by the sec­ond half of this cen­tury. Although it is by no means cer­tain, I be­lieve it is likely that whole brain em­u­la­tion will be achiev­able be­fore it is pos­si­ble to build a gen­eral ar­tifi­cial in­tel­li­gence us­ing tech­niques that do not at­tempt to di­rectly em­u­late the biol­ogy of the brain. This po­ten­tially re­sults in a sig­nifi­cantly differ­ent anal­y­sis of the po­ten­tial risks than that pre­sented by Ord. In par­tic­u­lar, while mis­al­igned val­ues still rep­re­sent a prob­lem for em­u­lated in­tel­li­gences, we do at least pos­sess an in-prin­ci­ple method for al­ign­ing their val­ues, namely the same sort of so­cial­i­sa­tion that is used with gen­eral suc­cess in al­ign­ing the val­ues of the next gen­er­a­tion of hu­mans. As a re­sult of such con­sid­er­a­tions, I am not con­vinced that it is es­pe­cially likely that an ar­tifi­cial in­tel­li­gence would have any par­tic­u­lar rea­son or mo­ti­va­tion to usurp hu­man­ity over the next cen­tury.

AI re­tains per­ma­nent con­trol over humanity

Ord seems to as­sume that once an AI at­tained a po­si­tion of power over the des­tiny of hu­man­ity, it would in­evitably main­tain this po­si­tion in­definitely. For in­stance he states:

Such an out­come needn’t in­volve the ex­tinc­tion of hu­man­ity. But it could eas­ily be an ex­is­ten­tial catas­tro­phe nonethe­less. Hu­man­ity would have per­ma­nently ceded its con­trol over the fu­ture. Our fu­ture would be at the mercy of how a small num­ber of peo­ple set up the com­puter sys­tem that took over. If we are lucky, this could leave us with a good or de­cent out­come, or we could just as eas­ily have a deeply flawed or dystopian fu­ture locked in for­ever. (p. 148)

In this pas­sage Ord speaks of the AI as it if is sim­ply a pas­sive tool, some­thing that is cre­ated and for­ever af­ter fol­lows its origi­nal pro­gram­ming. Whilst I do not say this is im­pos­si­ble, I be­lieve that it is an un­satis­fac­tory way to de­scribe an en­tity that is sup­pos­edly a su­per­in­tel­li­gent agent, some­thing ca­pa­ble of mak­ing de­ci­sions and tak­ing ac­tions on the ba­sis of its own vo­li­tion. Here I do not mean to im­ply any­thing about the na­ture of free will, only that we do not re­gard the be­havi­our of hu­mans as sim­ply the product of what evolu­tion has ‘pro­grammed into us’. While it must be granted that evolu­tion­ary forces are pow­er­ful in shap­ing hu­man mo­ti­va­tions and ac­tions, nev­er­the­less the range of pos­si­ble sets of val­ues, so­cial ar­range­ments, per­son­al­ity types, life goals, be­liefs, and habits that is con­sis­tent with such evolu­tion­ary forces is ex­tremely broad. In­deed, this is pre­sup­posed by Ord’s claim that “hu­man­ity is cur­rently in con­trol of its own fate. We can choose our fu­ture.” (p. 142).

If hu­man­ity’s fate is in our own hands and not pre­de­ter­mined by evolu­tion, why should we not also say that the fate of a hu­man­ity dom­i­nated by an AI would be in the hands of that AI (or col­lec­tive of AIs that share con­trol), rather than in the hands of the de­sign­ers who built that AI? The rea­son I think this is im­por­tant is that it high­lights the fact that an AI-dom­i­nated fu­ture is by no means one in which the AI’s goals, be­liefs, mo­ti­va­tions, val­ues, or fo­cus is static and un­chang­ing. To as­sume oth­er­wise is to as­sume that the AI in ques­tion takes a very spe­cific form which, as I have ar­gued above, I re­gard as be­ing un­likely. This sig­nifi­cantly re­duces the like­li­hood that a cur­rent nega­tive out­come with AI rep­re­sents a per­ma­nent nega­tive out­come. Of course, this is ir­rele­vant if the AI has driven hu­mans to ex­tinc­tion, but it be­comes highly rele­vant in other situ­a­tions in which an AI has placed hu­mans in an un­de­sir­able, sub­servient po­si­tion. I am not con­vinced that such a situ­a­tion would be per­pet­u­ated in­definitely.

Prob­a­bil­ity Estimates

Tak­ing into con­sid­er­a­tion the anal­y­sis I have pre­sented above, I would like to close by pre­sent­ing some es­ti­mates of my best guess of the prob­a­bil­ity of an ex­is­ten­tial catas­tro­phe oc­cur­ring within the next cen­tury by an en­g­ineered pan­demic and un­al­igned ar­tifi­cial in­tel­li­gence. Th­ese es­ti­mates should not be taken very se­ri­ously. I do not be­lieve we have enough in­for­ma­tion to make sen­si­ble quan­ti­ta­tive es­ti­mates about these even­tu­al­ities. Nev­er­the­less, I pre­sent my es­ti­mates largely in or­der to illus­trate the ex­tent of my dis­agree­ment with Ord’s es­ti­mates, and to illus­trate the key con­sid­er­a­tions I ex­am­ine in or­der to ar­rive at an es­ti­mate.

Prob­a­bil­ity of en­g­ineered pandemics

Con­sid­er­ing the is­sue of how an en­g­ineered pan­demic could lead to the ex­tinc­tion of hu­man­ity, I iden­tify five sep­a­rate things that must oc­cur, which to a first ap­prox­i­ma­tion I will re­gard as be­ing con­di­tion­ally in­de­pen­dent of one an­other:

1. There must ex­ist a biolog­i­cal pathogen with the right bal­ance of prop­er­ties to have the po­ten­tial of lead­ing to hu­man ex­tinc­tion.

2. It must be­come tech­nolog­i­cally fea­si­ble within the next cen­tury to evolve or en­g­ineer this pathogen.

3. The ex­tinc­tion-level agent must be ac­tu­ally pro­duced and de­liv­ered by some per­son or or­gani­sa­tion.

4. The pub­lic policy re­sponse to the emerg­ing pan­demic must fail in all ma­jor world na­tions.

5. Any biomed­i­cal re­sponse to the pan­demic, such as de­vel­op­ing tests, treat­ments, or vac­cines, must fail to be de­vel­oped within suffi­cient time to pre­vent ex­tinc­tion.

On the ba­sis of the rea­son­ing pre­sented in the pre­vi­ous sec­tions, I re­gard 1) as very un­likely, 2), 4), and 5) as un­likely, and 3) as slightly less un­likely. I will op­er­a­tional­ise ‘very un­likely’ as cor­re­spond­ing to a prob­a­bil­ity of 1%, ‘un­likely’ as cor­re­spond­ing to 10%, and the ‘slightly less likely’ as 20%. Note each of these prob­a­bil­ities is taken as con­di­tional on all the pre­vi­ous el­e­ments; so for ex­am­ple my claim is that con­di­tional on an ex­tinc­tion-level pathogen be­ing pos­si­ble, there is a 10% chance that it will be tech­nolog­i­cally fea­si­ble to pro­duce this pathogen within the next cen­tury. Com­bin­ing all these el­e­ments re­sults in the fol­low­ing prob­a­bil­ity:

P(bio ex­tinc­tion) = P(ex­tinc­tion level agent ex­ists) x P(ex­tinc­tion level agent tech­nolog­i­cally fea­si­ble) x P(ex­tinc­tion level agent pro­duced and de­liv­ered) x P(failure of timely pub­lic policy re­sponse) x P(failure of timely biomed­i­cal re­sponse)

P(bio ex­tinc­tion) = 0.01×0.1×0.2×0.1×0.1 = 2×10^(-6)

In com­par­i­son, Ord’s es­ti­mated risk from en­g­ineered pan­demics is 130, or 3×10^(-2). Ord’s es­ti­mated risk is thus roughly 10,000 times larger than mine.

Prob­a­bil­ity of un­al­igned ar­tifi­cial intelligence

Con­sid­er­ing the is­sue of un­al­igned ar­tifi­cial in­tel­li­gence, I iden­tify four key stages that would need to hap­pen for this to oc­cur, which again I will re­gard to first ap­prox­i­ma­tion as be­ing con­di­tion­ally in­de­pen­dent of one an­other:

1. Ar­tifi­cial gen­eral in­tel­li­gence, or an AI which is able to out-perform hu­mans in es­sen­tially all hu­man ac­tivi­ties, is de­vel­oped within the next cen­tury.

2. This ar­tifi­cial in­tel­li­gence ac­quires the power to usurp hu­man­ity and achieve a po­si­tion of dom­i­nance on Earth.

3. This ar­tifi­cial in­tel­li­gence has a rea­son/​mo­ti­va­tion/​pur­pose to usurp hu­man­ity and achieve a po­si­tion of dom­i­nance on Earth.

4. This ar­tifi­cial in­tel­li­gence ei­ther brings about the ex­tinc­tion of hu­man­ity, or oth­er­wise re­tains per­ma­nent dom­i­nance over hu­man­ity in a man­ner so as to sig­nifi­cantly diminish our long-term po­ten­tial.

On the ba­sis of the rea­son­ing pre­sented in the pre­vi­ous sec­tions, I re­gard 1) as roughly as likely as not, and 2), 3), and 4) as be­ing un­likely. Com­bin­ing all these el­e­ments re­sults in the fol­low­ing prob­a­bil­ity:

P(AI x-risk) = P(AI of suffi­cient ca­pa­bil­ity is de­vel­oped) x P(AI gains power to usurp hu­man­ity) x P(AI has suffi­cient rea­son to usurp hu­man­ity) x P(AI re­tains per­ma­nent usurpa­tion of hu­man­ity)

P(AI x.risk) = 0.5×0.1×0.1×0.1=5×10^(-4)

In com­par­i­son, Ord’s es­ti­mated risk from un­al­igned AI is 110, or 10^-1 . Ord’s es­ti­mated risk is roughly 200 times larger than mine.

Ar­riv­ing at cred­ible estimates

Although I do not think the spe­cific num­bers I pre­sent should be taken very se­ri­ously, I would like to defend the pro­cess I have gone through in es­ti­mat­ing these risks. Speci­fi­cally, I have iden­ti­fied the key pro­cesses I be­lieve would need to oc­cur in or­der for ex­tinc­tion or other ex­is­ten­tial catas­tro­phe to oc­cur, and then as­sessed how likely each of these pro­cesses would be to oc­cur on the ba­sis of the rele­vant his­tor­i­cal, sci­en­tific, so­cial, and other con­sid­er­a­tions that I be­lieve to be rele­vant. I then com­bine these prob­a­bil­ities to pro­duce an over­all es­ti­mate.

Though far from perfect, I be­lieve this pro­cess if far more trans­par­ent than the es­ti­mates pro­vided by Ord, for which no ex­pla­na­tion is offered as to how they were de­rived. This means that it is effec­tively im­pos­si­ble to sub­ject them to crit­i­cal scrutiny. In­deed, Ord even states that his prob­a­bil­ities “aren’t sim­ply an en­cap­su­la­tion of the in­for­ma­tion and ar­gu­men­ta­tion in the chap­ters on the risks” (p. 167), which seems to im­ply that it is not even pos­si­ble to sub­ject them to crit­i­cal anal­y­sis on the ba­sis of the in­for­ma­tion pre­sent in this book. While he defends this on the ba­sis that what he knows about the risks “goes be­yond what can be dis­til­led into a few pages” (p. 167), I do not find this a very satis­fac­tory re­sponse given the to­tal lack of ex­pla­na­tion of these num­bers in a book of over 400 pages.


In this es­say I have ar­gued that in his book The Precipice, Toby Ord has failed to provide a com­pel­ling ar­gu­ment that hu­man­ity faces a ‘precipice’ with un­prece­dent­edly high and clearly un­sus­tain­able lev­els of ex­is­ten­tial risk. My main ob­jec­tive was to pre­sent an al­ter­na­tive anal­y­sis of the risks as­so­ci­ated with en­g­ineered pan­demics and un­al­igned ar­tifi­cial in­tel­li­gence, high­light­ing is­sues and con­sid­er­a­tions that I be­lieve Ord does not grant suffi­cient at­ten­tion. Fur­ther­more, on the ba­sis of this anal­y­sis I pre­sented an al­ter­na­tive set of prob­a­bil­ity es­ti­mates for these two risks, both of which are con­sid­er­ably lower than those pre­sented by Ord. While far from com­pre­hen­sive or free from de­bat­able premises, I hope that the ap­proach I have out­lined here pro­vides a differ­ent per­spec­tive on the de­bate, and helps in the de­vel­op­ment of a nu­anced un­der­stand­ing of these im­por­tant is­sues.


The value
is not of type