Defining Meta Existential Risk

(This ar­ti­cle was origi­nally posted on my web­site here. I’d love to get an Effec­tive Altru­ist cri­tique of it. In gen­eral, I think long-ter­mist EAs should give meta x-risk more at­ten­tion. I’m sorry the images are so small.)

“Since Auschwitz we know what man is ca­pa­ble of. And since Hiroshima we know what is at stake.” —Vik­tor Frankl, Man’s Search for Meaning
“The great­est short­com­ing of the hu­man race is our in­abil­ity to un­der­stand the ex­po­nen­tial func­tion.” —Al Bartlett, Pro­fes­sor at Univer­sity of Colorado

The goal of this ar­ti­cle is to help read­ers un­der­stand the con­cept of Meta Ex­is­ten­tial Risk. I’m sur­prised that (es­sen­tially) noth­ing has been writ­ten on it yet.

Be­fore un­der­stand­ing Meta Ex­is­ten­tial Risk, it’s helpful to un­der­stand Ex­is­ten­tial Risk (x-risks). Ex­is­ten­tial Risks are pos­si­ble fu­ture events that would “wipe out” all sen­tient life on Earth. A mas­sive as­ter­oid im­pact, for ex­am­ple. Ex­is­ten­tial Risks can be con­trasted with non-ex­is­ten­tial Global Catas­trophic Risks, which don’t wipe out all life but pro­duce lots of in­sta­bil­ity and harm. A (smaller) as­ter­oid im­pact, for ex­am­ple :).

Meta Ex­is­ten­tial Risk gets at at the ques­tion: why do ex­is­ten­tial risks oc­cur? The an­swer: ex­po­nen­tial curves com­bined with bad game the­ory. Let’s look at our 4 pri­mary ex­is­ten­tial risks. As you can see, each one is an ex­po­nen­tial curve.

Cli­mate Change


Ar­tifi­cial Intelligence


Biolog­i­cal Technology


Nu­clear Technology

https://​​our­wor­ld­in­​​nu­clear-weapons (this ex­po­nen­tial curve peaked at the end of the Cold War)

Ex­po­nen­tial Curves Gen­er­ate X-Risks

Th­ese x-risks fall into two pri­mary cat­e­gories:

  1. Mag­ni­tude for Destruc­tion (AI, BioTech, Nukes): In this case, the x-risk is deadly sim­ply be­cause it rep­re­sents a mas­sive in­crease in power. There is a lot of power (de­struc­tive or not) in 60,000 nukes, su­per cheap ge­nomic se­quenc­ing, or ro­bust AI. And, as the Orthog­o­nal­ity Th­e­sis states: power is or­thog­o­nal to al­ign­ment with hu­man val­ues. More power doesn’t nec­es­sar­ily mean more re­spon­si­bil­ity.

  2. Nega­tive Ex­ter­nal­ities (Cli­mate Change): In this case, the x-risk is deadly not be­cause of “the tech it­self”, but be­cause of the tech’s nega­tive ex­ter­nal­ities. The In­dus­trial Revolu­tion (in­ad­ver­tently) pumped a lot of CO2 into the at­mo­sphere, which has led to cli­mate change. The prob­lem is not fac­to­ries or planes or what­ever, but rather the ig­nored nega­tive ex­ter­nal­ity.

(Daniel Sch­macht­en­berger defines the 1st is­sue as: “Ri­valrous (win-lose) games mul­ti­plied by ex­po­nen­tial tech­nol­ogy self ter­mi­nate.” And the 2nd is­sue as: “The need to learn how to build closed loop sys­tems that don’t cre­ate de­ple­tion and ac­cu­mu­la­tion, don’t re­quire con­tinued growth, and are in har­mony with the com­plex sys­tems they de­pend on.”)

Th­ese risks might not be such a big deal if we were able to look at the nega­tive im­pact of the ex­po­nen­tial curve and say “oh, let’s just fix it”. The prob­lem is that we can’t be­cause we’re stuck in two kinds of bad game the­ory prob­lems (of­ten called “co­or­di­na­tion prob­lems”).

  1. Our “Power” Prob­lems are Arms Races: For pow­er­ful new tech, all of the play­ers are in­cen­tivized to make it more pow­er­ful with­out think­ing enough about safety. If one player thinks about safety but goes slower, then the other player “wins” the arms race. e.g. Rus­sia and the U.S. both build more nukes, Google and Baidu cre­ate pow­er­ful AI as fast as pos­si­ble (not as safe as pos­si­ble), and CRISPR com­pa­nies do the same.

  2. Nega­tive Ex­ter­nal­ities are a Tragedy of the Com­mons: It is in each coun­try’s best in­ter­est to ig­nore/​ex­ploit the global com­mons be­cause ev­ery­one else is do­ing the same. If they try to car­bon tax, un­der­fish, etc. and no one else does, then they “lose”.

This is the root of meta x-risk: ex­po­nen­tial curves with nega­tive im­pacts that hu­mans need meta-co­or­di­na­tion to stop.

Th­ese prob­lems don’t just hold for these four spe­cific ex­po­nen­tial curves. Those are just the four that we’re deal­ing with to­day. There will be more ex­po­nen­tial curves in the fu­ture that will move more quickly and be more pow­er­ful. This leads to our fi­nal defi­ni­tion.

Meta Ex­is­ten­tial Risk is the risk to hu­man­ity cre­ated by (cur­rent and fu­ture) ex­po­nen­tial curves, their mis­al­ign­ment with hu­man val­ues and/​or our Earth sys­tem, and our in­abil­ity to stop them given co­or­di­na­tion prob­lems.

Cat­e­gor­i­cally Solv­ing Meta X-Risk

I won’t dive deeply into my hy­pothe­ses for how to “cat­e­gor­i­cally solve” meta x-risk here. How­ever, I’ll show you what this looks like in a di­rec­tional sense. For any cur­rent/​fu­ture ex­po­nen­tial:

  1. To solve the nega­tive ex­ter­nal­ity prob­lem, we need to turn ex­po­nen­tials into S-curves that cre­ate closed loop sys­tems with our biolog­i­cal sub­strate.

  2. To solve the power prob­lem, we need to en­sure our ex­po­nen­tials al­ign with hu­man val­ues.

Hope this ar­ti­cle was helpful in un­der­stand­ing meta x-risk. Com­ment be­low with your thoughts/​feed­back!


  • No­tice how similar the two solves for meta x-risk are. Both are through-time vec­tors that al­ign the ex­po­nen­tial with its sub­strate. In case 1, it’s al­ign­ing the ex­po­nen­tial with the phys­i­cal world (Na­ture 1.0). i.e. Not go­ing too fast and re­plen­ish­ing the com­plex biolog­i­cal sys­tem. In case 2, it’s al­ign­ing the ex­po­nen­tial with hu­man val­ues (Na­ture 2.0). (Na­ture 3.0 is AI/​tech btw.) I’m not to­tally sure, but this feels very con­nected to For­rest Landry’s idea of con­ti­nu­ity.

  • I wanted to note this more ex­plic­itly in the vi­su­als, but there’s an­other key way to con­cep­tu­al­ize of meta x-risk: as a nat­u­ral con­clu­sion of en­tropy—it’s eas­ier to de­stroy than to cre­ate. (1000 years to make a city, a sin­gle a-bomb to de­stroy it in min­utes.)

  • In the ar­ti­cle above I make it seem like all Nega­tive Ex­ter­nal­ities are a Tragedy of the Com­mons, and all Power Prob­lems are Arms Races. I don’t think this is nec­es­sar­ily true. Or at least, I haven’t for­mal­ized why this might be true. Thoughts?

  • In these pos­si­bly in­tense times, I like to re­mind my­self of thing like “we’re in­cred­ibly in­signifi­cant”. I mean damn, the uni­verse can only har­bor life for like 1 quin­til­lionth of it’s ex­is­tence. (See Melodysheep’s Time­lapse of the Fu­ture.)

  • I linked to it above as well, but the best cur­rent writ­ings on meta x-risk are from Daniel Sch­macht­en­berger: Solv­ing the Gen­er­a­tor Func­tions of Ex­is­ten­tial Risk. Check out my pod­cast with him here.

For more on meta x-risk, see the Frag­ility sec­tion in my book. Some high­lights:

  • Re­silience, An­tifrag­ility, and the Pre­cau­tion­ary Prin­ci­ple: We should prob­a­bly just as­sume lots of crazy stuff is go­ing to hap­pen, then make our­selves able to han­dle it.

  • Ok, so we want to evolve a new The­ory of Choice that keeps our ex­po­nen­tial re­al­ity from self-ter­mi­nat­ing. What else do we need? I’d also ar­gue that we need “Ab­stract Em­pa­thy”.