Eric Drexler: Paretotopian Goal Alignment

What if hu­man­ity sud­denly had a thou­sand times as many re­sources at our dis­posal? It might make fight­ing over them seem like a silly idea, since co­op­er­at­ing would be a safer way to be re­li­ably bet­ter off. In this talk from EA Global 2018: Lon­don, Eric Drexler ar­gues that when emerg­ing tech­nolo­gies make our pro­duc­tive ca­pac­ity sky­rocket, we should be able to make the world much bet­ter for ev­ery­one.

A tran­script of Eric’s talk is be­low, which CEA has lightly ed­ited for clar­ity. You can also watch this talk on YouTube, or read its tran­script on effec­tivealtru­

The Talk

This talk is go­ing to be a bit con­densed and crowded. I will plow for­ward at a brisk pace, but this is a mar­velous group of peo­ple who I trust to keep up won­der­fully.

Paretotopia Slide 1

For Pare­to­topian Goal Align­ment, a key con­cept is Pareto-preferred fu­tures, mean­ing fu­tures that would be strongly ap­prox­i­mately preferred by more or less ev­ery­one. If we can have fu­tures like that, that are part of the agenda, that are be­ing se­ri­ously dis­cussed, that peo­ple are plan­ning for, then per­haps we can get there. Strong goal al­ign­ment can make a lot of out­comes hap­pen that would not work in the mid­dle of con­flict.

So, when does goal al­ign­ment mat­ter. It could mat­ter for chang­ing per­cep­tions. There’s the con­cept of an Over­ton win­dow, the range of what can be dis­cussed within a given com­mu­nity at a given time, and what can be dis­cussed and taken se­ri­ously and re­garded as rea­son­able changes over time. Over­ton win­dows also vary by com­mu­nity. The Over­ton win­dow for dis­course in the EA com­mu­nity is differ­ent from that in, for ex­am­ple, the pub­lic sphere in Coun­try X.

Paretotopia Slide 2

AI seems likely to play a pivotal role. To­day we can ask ques­tions we couldn’t ask be­fore about AI, back when AI was a very ab­stract con­cept. We can ask ques­tions like, “Where do AI sys­tems come from?” Be­cause they’re now be­ing de­vel­oped. They come from re­search and de­vel­op­ment pro­cesses. What do they do? Broadly speak­ing, they provide ser­vices; they perform tasks in bounded time with bounded re­sources. What will they be able to do? Well, if you take AI se­ri­ously, you ex­pect AI sys­tems to be able to au­to­mate more or less any hu­man task. And more than that.

So now we ask, what is re­search and de­vel­op­ment? Well, it’s a bunch of tasks to au­to­mate. In­creas­ingly, we see AI re­search and de­vel­op­ment be­ing au­to­mated, us­ing soft­ware and AI tools. And where that leads is to­ward what one can call re­cur­sive tech­nol­ogy im­prove­ment. There’s a clas­sic view of AI sys­tems build­ing bet­ter AI sys­tems. This view has been as­so­ci­ated with agents. What we see emerg­ing is re­cur­sive tech­nolog­i­cal im­prove­ment in as­so­ci­a­tion with a tech­nol­ogy base. There’s an on­go­ing in­put of hu­man in­sights, but hu­man in­sights are lev­er­aged more and more, and be­come higher and higher level. Per­haps they also be­come less and less nec­es­sary. So at some point, we have AI de­vel­op­ment with AI speed.

Where that leads is to­ward what I de­scribe here as com­pre­hen­sive AI ser­vices. Ex­pand­ing the range of ser­vices, in­creas­ing their level to­ward this asymp­totic no­tion of “com­pre­hen­sive.” What does “com­pre­hen­sive” mean? Well, it in­cludes the ser­vice of de­vel­op­ing new ser­vices, and that’s where gen­er­al­ity comes from.

So, I just note that the C in CAIS does the work of G in AGI. So if you ask, “But in the CAIS model, can you do X? Do you need AGI to do X?” Then I say, “What part of ‘com­pre­hen­sive’ do you not un­der­stand?” I won’t be quite that rude, but if you ask, the an­swer is “Well, it’s com­pre­hen­sive. What is it you want to do? Let’s talk about that.”

Paretotopia Slide 3

For this talk, I think there are some key con­sid­er­a­tions for for­ward-look­ing EA strat­egy. This set of con­sid­er­a­tions is an­chored in AI, in an im­por­tant sense. Some­day, I think it’s rea­son­able to ex­pect that AI will be visi­bly, to a bunch of rele­vant com­mu­ni­ties, poised to be on the verge of ex­plo­sive growth. That it will be slid­ing into the Over­ton win­dow of pow­er­ful de­ci­sion-mak­ers.

Not to­day, but in­creas­ingly and very strongly at some point down­stream, when big changes are hap­pen­ing be­fore their eyes and more and more ex­perts are say­ing, “Look at what’s hap­pen­ing.” As a con­se­quence of that, we will be on the verge of enor­mous ex­pan­sion in pro­duc­tive ca­pac­ity. That’s one of the ap­pli­ca­tions of AI: fast, highly effec­tive au­toma­tion.

Also, this is a harder story to tell, but it fol­lows: if the right ground­work has been laid, we could have sys­tems—se­cu­rity sys­tems, mil­i­tary sys­tems, do­mes­tic se­cu­rity sys­tems, et cetera—that are be­nign in a strong sense, as viewed by al­most all par­ties, and effec­tive with re­spect to x-risk, mil­i­tary con­flict, and so on.

A fi­nal key con­sid­er­a­tion is that these facts are out­side the Over­ton win­dow of policy dis­course. One can­not have se­ri­ous policy dis­cus­sions based on these as­sump­tions. The other facts make pos­si­ble an ap­prox­i­mately strongly Pareto-preferred world. And the fi­nal fact con­strains strate­gies by which we might ac­tu­ally move in that di­rec­tion and get there.

And that con­flict is es­sen­tial to the lat­ter part of the talk, but first I would like to talk about re­source com­pe­ti­tion, be­cause that’s of­ten seen as the “hard ques­tion.” Re­sources are bounded at any par­tic­u­lar time, and peo­ple com­pete over them. Isn’t that a rea­son why things look like a zero-sum game? And re­source com­pe­ti­tion does not al­ign goals, but in­stead makes goals op­pose each other.

So, here’s a graph called “quan­tity of stuff that party A has,” ver­ti­cally, “quan­tity of stuff that B has,” hori­zon­tally. There’s a con­straint; there’s one unit of “stuff,” and so the trade-off curve here is a straight line, and changes are on that line, and goals are op­posed. Zero-sum game.

Paretotopia Slide 4

In fact, re­sources in­crease over time, but the no­tion of in­creas­ing by a mod­er­ate num­ber of per­cent per year is what peo­ple have in mind, and the time hori­zon in which you have a 50% in­crease is con­sid­ered very large.

But even with a 50% in­crease, shown here, if ei­ther A or B takes a very large pro­por­tion dur­ing the shift, like 90%, the other one is sig­nifi­cantly worse off than where they started.

Or­di­nar­ily, when we’re think­ing about util­ity, we don’t re­gard util­ity as lin­ear in re­sources, but as some­thing more like the log­a­r­ithm of re­sources. We’ll adopt that for illus­tra­tive pur­poses here. If we plot the same curves on a log scale, the lines be­come curved. So there’s the same unit con­straint. Here’s the 50% ex­pan­sion plot­ted log­a­r­ith­mi­cally.

Paretotopia Slide 5

Qual­i­ta­tively, looks rather similar. Well, the topolog­i­cal re­la­tion­ships are the same, it’s just re-plot­ting the same lines on a log scale. But on a log scale, we can now rep­re­sent large ex­pan­sions, and have the ge­om­e­try re­flect util­ity in a di­rect vi­sual sense. So there’s the same di­a­gram, with cur­rent hold­ings and 50% ex­pan­sion. And here’s what a thou­sand­fold ex­pan­sion looks like.

Paretotopia Slide 6

Tak­ing all the gains and tak­ing 90% of the to­tal have now switched po­si­tion. Some­one could ac­tu­ally take a large share of re­sources and ev­ery­one would still be way ahead. What mat­ters is that there be some rea­son­able di­vi­sion of gains. That’s a differ­ent situ­a­tion than the 50% in­crease, where one per­son tak­ing the vast ma­jor­ity was ac­tively bad for the other.

The key point here is that the differ­ence be­tween tak­ing ev­ery­thing ver­sus hav­ing a rea­son­able share is not very large, in terms of util­ity. So it’s a very differ­ent situ­a­tion than the stan­dard zero-sum game over re­sources. The rea­son for this is that we’re look­ing for­ward to a de­ci­sion time hori­zon that spans this large change, which is his­tor­i­cally not what we have seen.

So, let’s con­sider the case for when some party tries to take ev­ery­thing, or tries to take 90%. How far do you get? Well, greed brings risk. This is go­ing to cre­ate con­flict that is not cre­ated by at­tempt­ing to do that. So the ar­gu­ment here is that not only is there a small in­cre­ment of gain if you suc­ceed, but, al­low­ing for risk, the gains from at­tempt­ing to grab ev­ery­thing are nega­tive. Risk-ad­justed util­ity is bad. Your op­ti­mum is in fact to aim for some out­come that looks at least rea­son­ably fair to all of the other par­ties who are in this game, in this pro­cess of mu­tu­ally ad­just­ing poli­cies.

Paretotopia Slide 7

And, so, this re­gion, la­beled “Pare­to­topia”, this is a re­gion of out­comes (just look­ing at re­sources, al­though there are many other con­sid­er­a­tions) in which all par­ties see very large gains. So, that’s a differ­ent kind of fu­ture to aim at. It’s a strongly goal-al­ign­ing fu­ture, if one can make var­i­ous other con­sid­er­a­tions work. The prob­lem is, of course, that this is not in­side the win­dow of dis­cus­sion that one can have in the se­ri­ous world to­day.

The first thing to con­sider is what one can do with re­sources plus strong AI. It could elimi­nate poverty while pre­serv­ing rel­a­tive wealth. The billion­aires would re­main on top, and build star­ships. The global poor re­main on the bot­tom. They only have or­bital space­craft. And I’m se­ri­ous about that, if you have good pro­duc­tive ca­pa­bil­ity. They ex­pand to­tal wealth while rol­ling back en­vi­ron­men­tal harms. That’s some­thing one can work through, just start look­ing at the en­g­ineer­ing and what one can do with ex­panded pro­duc­tive ca­pa­bil­ities.

A more challeng­ing task is pre­serv­ing rel­a­tive sta­tus po­si­tions while also miti­gat­ing op­pres­sion. Why do we ob­ject to oth­ers hav­ing a whole lot of re­sources and se­cu­rity? Be­cause those tend to be used at our ex­pense. But one can de­scribe situ­a­tions in which op­pres­sion is miti­gated in a sta­ble way.

Struc­ture trans­parency is a con­cept I will not delve into here, but is re­lated to be­ing able to have in­her­ently defen­sive sys­tems that cir­cum­vent the se­cu­rity dilemma, “se­cu­rity dilemma” be­ing the pat­tern where two par­ties de­velop “defen­sive” weapons that seem ag­gres­sive to each other, and so you have an arms race. But if one were able to build truly effec­tive, gen­uinely defen­sive sys­tems, it would provide an exit door from that arms race pro­cess.

Again, these op­por­tu­ni­ties are out­side the Over­ton win­dow of cur­rent policy dis­course. So, where are we to­day? Well, for tech­nolog­i­cal per­cep­tions, on the one hand we have “cred­ible tech­nolo­gies,” and on the other we have “re­al­is­tic tech­nolo­gies,” given what en­g­ineer­ing and physics tell us is pos­si­ble. The prob­lem is that these sets do not over­lap. “Cred­ible” and “re­al­is­tic” are dis­joint sets. It’s a lit­tle hard to plan for the fu­ture and get peo­ple al­igned to­ward the fu­ture in that situ­a­tion. So, that’s a prob­lem. How can one at­tempt to ad­dress it? Well, first we note that at pre­sent we have what are called “track one poli­cies,” or “busi­ness-as-usual poli­cies.” What is re­al­is­tic is not even in the sphere of what is dis­cussed.

Paretotopia Slide 8

Now, I would ar­gue that we, in this com­mu­nity, are in a po­si­tion to dis­cuss re­al­is­tic pos­si­bil­ities more. We are, in fact, tak­ing ad­vanced AI se­ri­ously. Peo­ple also take se­ri­ously the con­cept of the “cos­mic en­dow­ment.” So, we’re will­ing to look at this. But how can we make progress in bridg­ing be­tween the world of what is cred­ible, in “track one,” and what’s re­al­is­tic?

Paretotopia Slide 9

I think, by find­ing tech­nolo­gies that are plau­si­ble, that are within the Over­ton win­dow in the sense that dis­cussing con­tin­gen­cies and pos­si­ble fu­tures like that is con­sid­ered rea­son­able. The con­cepts are not ex­otic, they’re sim­ply be­yond what we’re fa­mil­iar with, maybe in di­rec­tions that peo­ple are start­ing to ex­pect be­cause of AI. And so if this plau­si­ble range of tech­nolo­gies cor­re­sponds to re­al­is­tic tech­nolo­gies, the same kinds of op­por­tu­ni­ties, the same kinds of risks, there­fore the same kinds of poli­cies, and also cor­re­sponds to what is within the sphere of dis­course to­day… like ex­pand­ing au­toma­tion, high pro­duc­tion… well, that’s known to be a prob­lem and an op­por­tu­nity to­day. And so on.

Then, per­haps, one can have a dis­cus­sion that amounts to what’s called “track two,” where we have a com­mu­nity that is dis­cussing ex­plor­ing po­ten­tial goals and poli­cies, with an eye on what’s re­al­is­tic. Ex­plicit dis­cus­sion of poli­cies that are both in the “plau­si­ble” range and the “re­al­is­tic” range. Hav­ing the plau­si­ble poli­cies, the plau­si­ble pre­con­di­tions, be for­ward in dis­cus­sion. So, now you have some toe­hold in the world of what se­ri­ous peo­ple are will­ing to con­sider. And in­creas­ingly move these kinds of poli­cies, which will tend to be al­igned poli­cies that we’re ex­plor­ing, into the range of con­tin­gency plan­ning, for na­tions, for in­sti­tu­tions, where peo­ple will say, “Well, we’re fo­cus­ing of course on the real world and what we ex­pect, but if this crazy stuff hap­pens, who knows.”

They’ll say, “Peo­ple are think­ing AI might be a big deal, you folks are tel­ling us that AI will ex­pand re­sources, will make pos­si­ble change in the se­cu­rity en­vi­ron­ment, and so on. Well, that’s nice. You think about that. And if it hap­pens, maybe we’ll take your ad­vice. We’ll see.”

So, in this en­deavor, one has to work on as­sump­tions and poli­cies that are both plau­si­ble and would, if im­ple­mented, be broadly at­trac­tive. So, that’s a bunch of in­tel­lec­tual work. I’ll get into the strate­gic con­text next, but I’ll spend a cou­ple mo­ments on work­ing within the Over­ton win­dow of plau­si­bil­ity first.

Paretotopia Slide 10

So, re­al­is­tic: su­per­in­tel­li­gent-level AI ser­vices. Cred­ible: ex­ten­sive ap­pli­ca­tions of high-end AI. Peo­ple are talk­ing about that. Physics-limited pro­duc­tion. Truly sci­ence fic­tion in qual­ity. Well, a lot of the same is­sues arise from strong scal­able au­toma­tion, of the sort that peo­ple are already wor­ried about in the con­text of jobs. So­lar sys­tem scale en­ergy, 10^26 watts. Well, how about break­ing con­straints on ter­res­trial en­ergy prob­lems by hav­ing re­ally in­ex­pen­sive so­lar en­ergy? It can ex­pand power out­put, de­crease en­vi­ron­men­tal foot­print, and ac­tu­ally do di­rect car­bon cap­ture, if you have that amount of en­ergy. So­lar sys­tem scale re­sources, kind of off the table, but peo­ple are be­gin­ning to talk about as­ter­oid min­ing. Re­source effi­ciency, and one can ar­gue that re­sources are not bind­ing on eco­nomic growth in the near term, and that’s enough to break out of some of the zero-sum men­tal­ity. Ab­solute defen­sive sta­bil­ity is re­al­is­tic but not some­thing that is cred­ible, but mov­ing to­ward greater defen­sive sta­bil­ity is.

And note, it’s okay to be on the right side of this slide. You don’t nec­es­sar­ily, here in this room, have to take se­ri­ously su­per­in­tel­li­gent-level AI ser­vices, alert sys­tems, scale re­sources and so on, to be play­ing the game of work­ing within the range of what is plau­si­ble in the more gen­eral com­mu­nity, and work­ing through ques­tions that would con­sti­tute “Pare­to­topian goal-al­ign­ing poli­cies” in that frame­work. So the ar­gu­ment is that even­tu­ally, re­al­ity will give a hard shove. Busi­ness-as-usual sce­nar­ios, at least the as­sump­tions be­hind them, will be dis­cred­ited and, if we’ve done our work prop­erly, so will the poli­cies that are based on those as­sump­tions. The poli­cies that lead to the idea that maybe we should be fight­ing over re­sources in the South China Sea just look ab­surd, be­cause ev­ery­one knows that in a fu­ture of great abun­dance, fight­ing over some­thing is worth­less.

Paretotopia Slide 11

So, if the right in­tel­lec­tual ground­work has been done, then, when there’s a hard shove from re­al­ity that is to­ward a fu­ture that has Pare­to­topian po­ten­tial, there will be a co­her­ent policy pic­ture that is co­her­ent across many differ­ent in­sti­tu­tions, with ev­ery­one know­ing that ev­ery­one else knows that it would be good to move in this di­rec­tion. Draft agree­ments worked out in track two diplo­macy, sce­nario plan­ning that sug­gests it would be re­ally stupid to pur­sue busi­ness as usual in arms races. If that kind of work is in place, then with a hard shove from re­al­ity, we might see a shift. Track one poli­cies are dis­cred­ited, and so peo­ple ask, “What should we do? What do we do? The world is chang­ing.”

Paretotopia Slide 12

Well, we could try these new Pare­to­topian poli­cies. They look good. If you fight over stuff, you prob­a­bly lose. And if you fight over it, you don’t get much if you win, so why not go along with the bet­ter op­tion, which has been thought through in some depth and looks at­trac­tive?

So that is the ba­sic Pare­to­topian strate­gic idea. We look at these great ad­vances, back off to plau­si­ble as­sump­tions that can be dis­cussed in that frame­work, work through in­ter­ac­tions with many, many differ­ent groups, re­flect­ing di­verse con­cerns that, in many cases, will seem op­posed but can be rec­on­ciled given greater re­sources and the abil­ity to make agree­ments that couldn’t be made in the ab­sence of, for ex­am­ple, strong AI im­ple­men­ta­tion abil­ities. And so, fi­nally, we end up in a differ­ent sort of world.

Paretotopia Slide 13

Now, this says “ro­bust.” Ro­bust against what? All of the ca­pa­bil­ities that are not within the range of dis­cus­sion or that are sim­ply sur­pris­ing. “Com­pat­i­ble.” Well, Pare­to­topian poli­cies aren’t about hav­ing one pat­tern on the world, it re­ally means many differ­ent poli­cies that are com­pat­i­ble in the sense that the out­comes are sta­ble and at­trac­tive.

And with that, the task at hand, at least in one of the many di­rec­tions that the EA com­mu­nity can push, and a set of con­sid­er­a­tions that I think are use­ful back­ground and con­text for many other EA ac­tivi­ties, is for­mu­lat­ing and pur­su­ing Pare­to­topian meta-strate­gies, and the frame­work for think­ing about those strate­gies. That means un­der­stand­ing re­al­is­tic and cred­ible ca­pa­bil­ities, and then bridg­ing the two. There’s a bunch of work on both un­der­stand­ing what’s re­al­is­tic and what is cred­ible, and the re­la­tion­ships be­tween these. There’s work on un­der­stand­ing and ac­com­mo­dat­ing di­verse con­cerns. One would like to have poli­cies that seem in­sti­tu­tion­ally ac­cept­able to the U.S. mil­i­tary, and the Com­mu­nist Party of China, and to billion­aires, and also make the ru­ral poor well-off, and so on, and have those be com­pat­i­ble goals. And to re­ally un­der­stand the con­cerns of those com­mu­ni­ties, in their own con­cep­tual and idio­matic lan­guages. That’s a key di­rec­tion to pur­sue. And that means deep­en­ing and ex­pand­ing the cir­cle of dis­course that I’m out­lin­ing.

Paretotopia Slide 14

And so, this is a lot of hard in­tel­lec­tual work and, down­stream, in­creas­ing or­ga­ni­za­tional work. I think that pretty much ev­ery­thing one might want to pur­sue in the world that is good fits broadly within this frame­work, and can per­haps be bet­ter ori­ented with some at­ten­tion to this meta-strate­gic frame­work for think­ing about goal al­ign­ment. And so, thank you.

No comments.