How can we influence the long-term future?

Link post

Hu­man his­tory can be viewed as the con­tin­u­ous evolu­tion of the rel­a­tive power of differ­ent value sys­tems, and the in­ter­ac­tions be­tween agents that em­body them. This pro­cess may reach a steady state (also some­times called a “lock-in”) at some point, rep­re­sent­ing the end of his­tory.

As­sum­ing that a steady state oc­curs even­tu­ally – which is un­clear – it can take one of the fol­low­ing forms1:

  1. Sin­gle­ton: A sin­gle ac­tor holds all power and shapes the uni­verse ac­cord­ing to its val­ues. (Note that these val­ues can still be a com­pro­mise over many value sys­tems, e.g. in a world gov­ern­ment that im­ple­ments a ne­go­ti­ated agree­ment.)

  2. Mul­tipo­lar out­come: Power is dis­tributed be­tween sev­eral sta­ble cen­tres of power with differ­ent val­ues. We can fur­ther dis­t­in­guish two cases:

    1. A fully co­op­er­a­tive mul­ti­po­lar out­come where all ac­tors agree to op­ti­mise a com­pro­mise value sys­tem to reap gains from trade. (This is equiv­a­lent to a sin­gle­ton with com­pro­mise val­ues.)

    2. A not fully co­op­er­a­tive set­ting that en­tails co­op­er­a­tion prob­lems or even con­flicts be­tween differ­ent ac­tors.

  3. Ex­tinc­tion: No ac­tors are able to shape the uni­verse ac­cord­ing to their val­ues, e.g. be­cause Earth-origi­nat­ing in­tel­li­gence goes ex­tinct.

In cases 1 and 2, we can fur­ther ask whether (and to what de­gree) the pow­er­ful ac­tors face nat­u­ral con­straints in their at­tempt to shape the uni­verse. For in­stance, large-scale space colon­i­sa­tion is not fea­si­ble with to­day’s level of tech­nol­ogy, re­gard­less of the dis­tri­bu­tion of power and val­ues. How­ever, the tech­nolog­i­cal ca­pac­ity of hu­man civil­i­sa­tion will likely ex­pand fur­ther in the fu­ture – just as there has been fairly con­sis­tent (ac­cel­er­at­ing) tech­nolog­i­cal progress and eco­nomic growth over the course of hu­man his­tory.

From the per­spec­tive of long-ter­mist con­se­quen­tial­ism, and s-risk re­duc­tion in par­tic­u­lar, there are pre­cau­tion­ary rea­sons to as­sume:

  • that a steady state other than ex­tinc­tion will even­tu­ally be reached. In­fluenc­ing that re­sult­ing steady state is most rele­vant as it will per­sist over cos­mic timescales – billions of years – and will there­fore af­fect an as­tro­nom­i­cal num­ber of sen­tient be­ings.2

  • that the steady state is tech­nolog­i­cally ma­ture: the level of tech­nolog­i­cal ca­pac­ity that’s available to ac­tors is close to the the­o­ret­i­cal max­i­mum. This is be­cause pow­er­ful tech­nol­ogy is a risk fac­tor for s-risks as it al­lows ac­tors to op­ti­mise the uni­verse to a much greater de­gree, for bet­ter or for worse.

The ques­tion is, how can we in­fluence the re­sult­ing steady state? It is hard to ac­cu­rately pre­dict the fu­ture tra­jec­tory of hu­man civil­i­sa­tion over long times­pans. And even if you can pre­dict the fu­ture, it may still be hard to in­fluence it. For in­stance, an effec­tive al­tru­ist in the year 1700 would strug­gle to find spe­cific ac­tions to al­ter the re­sult of the in­dus­trial rev­olu­tion, even if armed with the (highly un­usual) knowl­edge of what will hap­pen.

One pos­si­ble an­swer is to pre­vent hu­man ex­tinc­tion, which is a lever to change the re­sult­ing steady state from ex­tinc­tion to some dis­tri­bu­tion over non-ex­tinc­tion steady states. Whether this is a pri­or­ity de­pends on one’s view on pop­u­la­tion ethics, since an empty uni­verse would, while not con­tain­ing any­thing of value, also not con­tain any suffer­ing.3 (For two differ­ent per­spec­tives on this, see here and here.)

I’d like to em­pha­size that even if one were to be­lieve that ex­tinc­tion is prefer­able to what hap­pens in the fu­ture, it would be clearly wrong to try and in­crease ex­is­ten­tial risk. Any such ac­tion would be ex­tremely ad­ver­sar­ial to­ward other value sys­tems and would likely lead to in­creased po­lari­sa­tion and con­flict – which is an­other risk fac­tor for s-risks. There are many good rea­sons to be nice to other value sys­tems and even pure con­se­quen­tial­ists should adopt some quasi-de­on­tolog­i­cal rules, such as not us­ing or ad­vo­cat­ing vi­o­lence of any form.

Still, from my per­spec­tive – hav­ing suffer­ing-fo­cused val­ues – it is bet­ter to try and im­prove the re­sult­ing steady state if it’s not ex­tinc­tion, rather than at­tempt­ing to switch prob­a­bil­ity mass be­tween ex­tinc­tion and non-ex­tinc­tion steady states.


This brings us back to the ques­tion of when such a steady state is reached. It is pos­si­ble that this hap­pens soon, in which case we can have a di­rect and last­ing im­pact on what the re­sult­ing steady state will be. (A con­crete ex­am­ple that comes up fre­quently is that ar­tifi­cial gen­eral in­tel­li­gence will be built in this cen­tury and will soon achieve a de­ci­sive strate­gic ad­van­tage and form a sin­gle­ton.)

How­ever, I worry about a po­ten­tial bias to over­es­ti­mate our im­pact, as this re­quires the be­lief that our gen­er­a­tion is in a unique po­si­tion to have a dis­pro­por­tionate im­pact, and such be­liefs should, at least a pri­ori, be pe­nal­ised. It seems more likely that the end of his­tory is not near. If I had to guess, I’d say it’s less than 50% likely that a steady state will be at­tained within 1000 years, as­sum­ing no ex­tinc­tion.

This could lead to the frus­trat­ing con­clu­sion that our in­fluence over what hap­pens in the long-term fu­ture may be limited, since the val­ues of fu­ture ac­tors will mu­tate in many ways be­fore a steady state is reached. While our ac­tions have rip­ple effects on the fu­ture, they are hard (if not im­pos­si­ble) to pre­dict, thwart­ing at­tempts to de­liber­ately al­ter the long-term fu­ture in a par­tic­u­lar di­rec­tion. (Robin Han­son has also ar­gued that fu­ture in­fluence is hard and that value drift seems un­avoid­able at this point.)

But, since our im­pact as effec­tive al­tru­ists is much smaller in this case, there are pre­cau­tion­ary rea­sons to as­sume that we do have a rea­son­able chance to in­fluence the re­sult­ing steady state:

  • Our im­pact is larger if a steady state is very close af­ter all. (But I don’t see a very plau­si­ble sce­nario for this – in par­tic­u­lar, I’m scep­ti­cal about the spe­cific claim that ad­vanced ar­tifi­cial in­tel­li­gence is around the cor­ner.)

  • It is pos­si­ble that the gen­eral pace of change will speed up dras­ti­cally in the near fu­ture (e.g. later this cen­tury) due to tech­nolog­i­cal ad­vances, such as whole-brain em­u­la­tion or pow­er­ful biolog­i­cal en­hance­ment. In this case, the equiv­a­lent of thou­sands years of his­tory at the cur­rent pace may hap­pen within a rel­a­tively short phys­i­cal times­pan, which al­lows con­tem­po­rary hu­mans to have a larger in­fluence.

  • We can hope that there are ways to re­li­ably in­fluence the re­sult even if a steady state is far away. For in­stance, if we can effect changes in val­ues that are fairly sticky over longer timescales, then moral ad­vo­cacy is a promis­ing lever.

This may seem like a Pas­calian wa­ger, and there are good rea­sons to not put too much weight on such ar­gu­ments. But, while some form of the “frus­trat­ing con­clu­sion” is fairly likely in my opinion, the listed al­ter­na­tives are not too far-fetched ei­ther, so I think this is an ac­cept­able kind of pre­cau­tion­ary rea­son­ing.