Which World Gets Saved

It is com­mon to ar­gue for the im­por­tance of x-risk re­duc­tion by em­pha­siz­ing the im­mense value that may ex­ist over the course of the fu­ture, if the fu­ture comes. Fa­mously, for in­stance, Nick Bostrom calcu­lates that, even un­der rel­a­tively sober es­ti­mates of the num­ber and size of fu­ture hu­man gen­er­a­tions, “the ex­pected value of re­duc­ing ex­is­ten­tial risk by a mere one mil­lionth of one per­centage point is at least a hun­dred times the value of a mil­lion hu­man lives”.

[Note: Peo­ple some­times use the term “x-risk” to re­fer to slightly differ­ent things. Here, I use the term to re­fer to an event that would bring the value of fu­ture hu­man civ­i­liza­tion to roughly zero—an ex­tinc­tion event, a war in which we bomb our­selves back to the Stone Age and get stuck there for­ever, or some­thing along those lines.]

Among those who take such ar­gu­ments for x-risk re­duc­tion se­ri­ously, there seem to be two coun­ter­ar­gu­ments com­monly (e.g. here) raised in re­sponse. First, the fu­ture may con­tain more pain than plea­sure. If we think that this is likely, then, at least from the util­i­tar­ian per­spec­tive, x-risk re­duc­tion stops look­ing so great. Se­cond, we may have op­por­tu­ni­ties to im­prove the tra­jec­tory of the fu­ture, such as by im­prov­ing the qual­ity of global in­sti­tu­tions or by speed­ing eco­nomic growth, and such efforts may have even higher ex­pected value than (im­me­di­ate) x-risk re­duc­tion. “Mun­dane” in­sti­tu­tion-build­ing efforts may also have the benefit of re­duc­ing fu­ture catas­trophic risks, should they arise.

It seems to me that there is an­other im­por­tant con­sid­er­a­tion which com­pli­cates the case for x-risk re­duc­tion efforts, which peo­ple cur­rently ne­glect. The con­sid­er­a­tion is that, even if we think the value of the fu­ture is pos­i­tive and large, the value of the fu­ture con­di­tional on the fact that we marginally averted a given x-risk may not be. And in any event, these val­ues are bound to de­pend on the x-risk in ques­tion.

For ex­am­ple: There are things we cur­rently do not know about hu­man psy­chol­ogy, some of which bear on how in­clined we are to­ward peace and co­op­er­a­tion. Per­haps Steven Pinker is right, and vi­o­lence will con­tinue its steady de­cline, un­til one evening sees the world’s last bar fight and hu­man­ity is at peace for­ever af­ter. Or per­haps he’s wrong—per­haps a cer­tain mea­sure of im­pul­sive­ness and anger will always re­main, how­ever fa­vor­able the en­vi­ron­ment, and these im­pulses are bound to crop up pe­ri­od­i­cally in fights and mass tor­tures and world wars. In the ex­treme case, if we think that the ex­pected value of the fu­ture (if it comes) is large and pos­i­tive un­der the former hy­poth­e­sis but large and nega­tive un­der the lat­ter, then the pos­si­bil­ity that hu­man rage may end the world is a source of con­so­la­tion, not worry. It means that the ex­is­ten­tial risk posed by world war is serv­ing as a sort of “fuse”, turn­ing off the lights rather than let­ting the fam­ily burn.

As an ap­pli­ca­tion: if we think the peace­ful-psy­chol­ogy hy­poth­e­sis is more likely than the vi­o­lent-psy­chol­ogy hy­poth­e­sis, we might think that the fu­ture has high ex­pected value. We might thus con­sider it im­por­tant to avert ex­tinc­tion events like as­ter­oid im­pacts, which would knock out wor­lds “on av­er­age”. But we might op­pose efforts like the Nu­clear Threat Ini­ti­a­tive, which dis­pro­por­tionately save vi­o­lent-psy­chol­ogy wor­lds. Or we might think that the sign of the value of the fu­ture is pos­i­tive in ei­ther sce­nario, but judge that one x-risk is worth de­vot­ing more effort to than an­other, all else equal.

Once we start think­ing along these lines, we open var­i­ous cans of worms. If our x-risk re­duc­tion effort starts far “up­stream”, e.g. with an effort to make peo­ple more co­op­er­a­tive and peace-lov­ing in gen­eral, to what ex­tent should we take the suc­cess of the in­ter­me­di­ate steps (which must suc­ceed for the x-risk re­duc­tion effort to suc­ceed) as ev­i­dence that the saved world would go on to a great fu­ture? Should we in­cor­po­rate the fact of our own choice to pur­sue x-risk re­duc­tion it­self into our es­ti­mate of the ex­pected value of the fu­ture, as recom­mended by ev­i­den­tial de­ci­sion the­ory, or should we ex­clude it, as recom­mended by causal? How should we gen­er­ate all these con­di­tional ex­pected val­ues, any­way?

Some of these ques­tions may be worth the time to an­swer care­fully, and some may not. My goal here is just to raise the broad con­di­tional-value con­sid­er­a­tion which, though ob­vi­ous once stated, so far seems to have re­ceived too lit­tle at­ten­tion. (For refer­ence: on dis­cussing this con­sid­er­a­tion with Will MacAskill and Toby Ord, both said that they had not thought of it, and thought that it was a good point.) In short, “The util­i­tar­ian im­per­a­tive ‘Max­i­mize ex­pected ag­gre­gate util­ity!‘” might not re­ally, as Bostrom (2002) puts it, “be sim­plified to the maxim ‘Min­i­mize ex­is­ten­tial risk’“. It may not enough to do our best to save the world. We may also have to con­sider which world gets saved.