Tobias_Baumann

How can we in­fluence the long-term fu­ture?

• Tobias_Baumann 17 Feb 2019 21:18 UTC

  1 point

  in reply to: JanBrauner's comment on: Risk fac­tors for s-risks

  Thank you – great to hear that you’ve found it use­ful!

Risk fac­tors for s-risks

• Tobias_Baumann 2 Jan 2019 12:31 UTC

  1 point

  in reply to: Wei_Dai's comment on: Why I ex­pect suc­cess­ful (nar­row) alignment

  Thanks for the de­tailed com­ments!

  (Also, BTW, I would have preferred the word “nar­row” or some­thing like it in the post ti­tle, be­cause some peo­ple use “al­ign­ment” in a broad sense and as a re­sult may mis­in­ter­pret you as be­ing more op­ti­mistic than you ac­tu­ally are.)

  Good point – changed the ti­tle.

  Also, dis­tributed emer­gence of AI is likely not safer than cen­tral­ized AI, be­cause an “econ­omy” of AIs would be even harder to con­trol and har­ness to­wards hu­man val­ues than a sin­gle or small num­ber of AI agents.

  As long as we con­sider only nar­row al­ign­ment, it does seem safer to me in that lo­cal mis­al­ign­ment or safety is­sues in in­di­vi­d­ual sys­tems would not im­me­di­ately cause ev­ery­thing to break down, be­cause such a sys­tem would (ar­guably) not be able to ob­tain a de­ci­sive strate­gic ad­van­tage and take over the world. So there’d be time to re­act.

  But I agree with you that an econ­omy-like sce­nario en­tails other safety is­sues, and al­ign­ing the en­tire “econ­omy” with hu­man (com­pro­mise) val­ues might be very difficult. So I don’t think this is safer over­all, or at least it’s not ob­vi­ous. (From my suffer­ing-fo­cused per­spec­tive, dis­tributed emer­gence of AI ac­tu­ally seems worse than a sce­nario of the form “a sin­gle sys­tem quickly takes over and forms a sin­gle­ton”, as the lat­ter seems less likely to lead to con­flict-re­lated dis­value.)

  This as­sumes that al­ign­ment work is highly par­alleliz­able. If it’s not, then do­ing more al­ign­ment work now can shift the whole al­ign­ment timeline for­ward, in­stead of just adding to the to­tal amount of al­ign­ment work in a marginal way.

  Yeah, I do think that al­ign­ment work is fairly par­alleliz­able, and fu­ture work also has a (po­ten­tially very big) in­for­ma­tion ad­van­tage over cur­rent work be­cause they will know more about what AI tech­niques look like. Is there any prece­dent of a new tech­nol­ogy where work on safety is­sues was highly se­rial and where it was there­fore cru­cial to start work­ing on safety a long time in ad­vance?

  This only ap­plies to short-term “al­ign­ment” and not to long-term /​ scal­able al­ign­ment. That is, I have an eco­nomic in­cen­tive to build an AI that I can har­ness to give me short-term prof­its, even if that’s at the ex­pense of the long term value of the uni­verse to hu­man­ity or hu­man val­ues. This could be done for ex­am­ple by cre­at­ing an AI that is not at all al­igned with my val­ues and just giv­ing it re­wards/​pun­ish­ments so that it has a near-term in­stru­men­tal rea­son to help me (similar to how other hu­mans are use­ful to us even if they are not value al­igned to us).

  I think there are two differ­ent cases:

  • If the hu­man ac­tu­ally cares only about short-term self­ish gain, pos­si­bly at the ex­pense of oth­ers, then this isn’t a nar­row al­ign­ment failure, it’s a co­op­er­a­tion prob­lem. (But I agree that it could be a se­ri­ous is­sue).

  • If the hu­man ac­tu­ally cares about the long term, then it ap­pears that she’s mak­ing a mis­take by buy­ing an AI sys­tem that is only al­igned in the short term. So it comes down to hu­man in­ad­e­quacy – given suffi­cient in­for­ma­tion she’d buy a long-term al­igned AI sys­tem in­stead, and AI com­pa­nies would have in­cen­tive to provide long-term al­igned AI sys­tems. Though of course the “suffi­cient in­for­ma­tion” part is cru­cial, and is a fairly strong as­sump­tion as it may be hard to dis­t­in­guish be­tween “short-term al­ign­ment” and “real” al­ign­ment. I agree that this is an­other po­ten­tially se­ri­ous prob­lem.

  I think we our­selves don’t know how to re­li­ably dis­t­in­guish be­tween “at­tempts to ma­nipu­late” and “at­tempts to help” so it would be hard to AIs to learn this. One prob­lem is, our own ma­nipu­late/​help clas­sifier was trained on a nar­row set of in­puts (i.e., of other hu­mans ma­nipu­lat­ing/​helping) and will likely fail when ap­plied to AIs due to dis­tri­bu­tional shift.

  In­ter­est­ing point. I think I still have an in­tu­ition that there’s a fairly sim­ple core to it, but I’m not sure how to best ar­tic­u­late this in­tu­ition.

• Tobias_Baumann 30 Dec 2018 12:14 UTC

  2 points

  in reply to: Wei_Dai's comment on: Why I ex­pect suc­cess­ful (nar­row) alignment

  Work­ing on these prob­lems makes a lot of sense, and I’m not say­ing that the philo­soph­i­cal is­sues around what “hu­man val­ues” means will likely be solved by de­fault.

  I think in­creas­ing philo­soph­i­cal so­phis­ti­ca­tion (or “moral un­cer­tainty ex­pan­sion”) is a very good idea from many per­spec­tives. (A di­rect com­par­i­son to moral cir­cle ex­pan­sion would also need to take rel­a­tive tractabil­ity and im­por­tance into ac­count, which seems un­clear to me.)

• Tobias_Baumann 30 Dec 2018 11:51 UTC

  3 points

  in reply to: Wei_Dai's comment on: Why I ex­pect suc­cess­ful (nar­row) alignment

  Great point – I agree that it would be value to have a com­mon scale.

  I’m a bit sur­prised by the 1-10% es­ti­mate. This seems very low, es­pe­cially given that “se­ri­ous catas­tro­phe caused by ma­chine in­tel­li­gence” is broader than nar­row al­ign­ment failure. If we in­clude pos­si­bil­ities like se­ri­ous value drift as new tech­nolo­gies emerge, or difficult AI-re­lated co­op­er­a­tion and se­cu­rity prob­lems, or eco­nomic dy­nam­ics rid­ing roughshod over hu­man val­ues, then I’d put much more than 10% (plau­si­bly more than 50%) on some­thing not go­ing well.

  Re­gard­ing the “other thought­ful peo­ple” in my 80% es­ti­mate: I think it’s very un­clear who ex­actly one should up­date to­wards. What I had in mind is that many EAs who have thought about this ap­pear to not have high con­fi­dence in suc­cess­ful nar­row al­ign­ment (not clear if the me­dian is >50%?), judg­ing based on my im­pres­sions from in­ter­act­ing with peo­ple (which is ob­vi­ously not rep­re­sen­ta­tive). I felt that my opinion is quite con­trar­ian rel­a­tive to this, which is why I felt that I should be less con­fi­dent than the in­side view sug­gests, al­though as you say it’s quite hard to grasp what peo­ple’s opinions ac­tu­ally are.

  On the other hand, one pos­si­ble in­ter­pre­ta­tion (but not the only one) of the rel­a­tively low level of con­cern for AI risk among the larger AI com­mu­nity and so­cietal elites is that peo­ple are quite op­ti­mistic that “we’ll know how to cross that bridge once we get to it”.

Why I ex­pect suc­cess­ful (nar­row) alignment

A ty­pol­ogy of s-risks

Thoughts on short timelines

• Tobias_Baumann 6 Aug 2018 8:53 UTC

  7 points

  in reply to: Benjamin_Todd's comment on: Prob­lems with EA rep­re­sen­ta­tive­ness and how to solve it

  Agreed. As some­one who pri­ori­tises s-risk re­duc­tion, I find it odd that long-ter­mism is some­times con­sid­ered equiv­a­lent to x-risk re­duc­tion. It is le­gi­t­i­mate if peo­ple think that x-risk re­duc­tion is the best way to im­prove the long-term, but it should be made clear that this is based on ad­di­tional be­liefs about ethics (re­ject­ing suffer­ing-fo­cused views and not be­ing very con­cerned about value drift), about how likely x-risks in this cen­tury are, and about how tractable it is to re­duce them, rel­a­tive to other ways of im­prov­ing the long-term. I for one think that none of these points is ob­vi­ous.

  So I feel that there is a rep­re­sen­ta­tive­ness prob­lem be­tween x-risk re­duc­tion and other ways of im­prov­ing the long-term fu­ture (not nec­es­sar­ily only s-risk re­duc­tion), in ad­di­tion to an un­der­rep­re­sen­ta­tion of near-term causes.

• Tobias_Baumann 2 Nov 2017 16:25 UTC

  3 points

  in reply to: jpaddison's comment on: Mul­ti­verse-wide co­op­er­a­tion in a nutshell

  Thanks for writ­ing this up!

  I think the idea is in­trigu­ing, and I agree that this is pos­si­ble in prin­ci­ple, but I’m not con­vinced of your take on its prac­ti­cal im­pli­ca­tions. Apart from heuris­tic rea­sons to be scep­ti­cal of a new idea on this level of ab­stract­ness and spec­u­la­tive­ness, my main ob­jec­tion is that a high de­gree of similar­ity with re­spect to rea­son­ing (which is re­quired for the de­ci­sions to be en­tan­gled) prob­a­bly goes along with at least some de­gree of similar­ity with re­spect to val­ues. (And if the val­ues of the agents that cor­re­late with me are similar to mine, then the re­sult of tak­ing them into ac­count is also closer to my own val­ues than the com­pro­mise value sys­tem of all agents.)

  You write:

  Su­per­ra­tional­ity only mo­ti­vates co­op­er­a­tion if one has good rea­son to be­lieve that an­other party’s de­ci­sion al­gorithm is in­deed ex­tremely similar to one’s own. Hu­man rea­son­ing pro­cesses differ in many ways, and sym­pa­thy to­wards su­per­ra­tional­ity rep­re­sents only one small di­men­sion of one’s rea­son­ing pro­cess. It may very well be ex­tremely rare that two peo­ple’s rea­son­ing is suffi­ciently similar that, hav­ing com­mon knowl­edge of this similar­ity, they should ra­tio­nally co­op­er­ate in a pris­oner’s dilemma.

  Con­di­tional on this ex­tremely high de­gree of similar­ity to me, isn’t it also more likely that their val­ues are also similar to mine? For in­stance, if my rea­son­ing is shaped by the ex­pe­riences I’ve made, my ge­netic makeup, or the set of all ideas I’ve read about over the course of my life, then an agent with iden­ti­cal or highly similar rea­son­ing would also share a lot of these char­ac­ter­is­tics. But of course, my ex­pe­riences, genes, etc. also de­ter­mine my val­ues, so similar­ity with re­spect to these fac­tors im­plies similar­ity with re­spect to val­ues.

  This is not the same as claiming that a given char­ac­ter­is­tic X that’s rele­vant to de­ci­sion-mak­ing is gen­er­ally linked to val­ues, in the sense that peo­ple with X have sys­tem­at­i­cally differ­ent val­ues. It’s a sub­tle differ­ence: I’m not say­ing that cer­tain as­pects of rea­son­ing gen­er­ally go along with cer­tain val­ues across the en­tire pop­u­la­tion; I’m say­ing that a high de­gree of similar­ity re­gard­ing rea­son­ing goes along with similar­ity re­gard­ing val­ues.

S-risk FAQ

• Tobias_Baumann 20 Jul 2017 8:40 UTC

  12 points

  in reply to: jpaddison's comment on: An Ar­gu­ment for Why the Fu­ture May Be Good

  Thanks for writ­ing this up! I agree that this is a rele­vant ar­gu­ment, even though many steps of the ar­gu­ment are (as you say your­self) not air­tight. For ex­am­ple, con­scious­ness or suffer­ing may be re­lated to learn­ing, in which case point 3) is much less clear.

  Also, the fu­ture may con­tain vastly larger pop­u­la­tions (e.g. be­cause of space coloniza­tion), which, all else be­ing equal, may im­ply (vastly) more suffer­ing. Even if your ar­gu­ment is valid and the frac­tion of suffer­ing de­creases, it’s not clear whether the ab­solute amount will be higher or lower (as you claim in 7.).

  Fi­nally, I would ar­gue we should fo­cus on the bad sce­nar­ios any­way – given suffi­cient un­cer­tainty – be­cause there’s not much to do if the fu­ture will “au­to­mat­i­cally” be good. If s-risks are likely, my ac­tions mat­ter much more.

  (This is from a suffer­ing-fo­cused per­spec­tive. Other value sys­tems may ar­rive at differ­ent con­clu­sions.)

• Tobias_Baumann 8 Jul 2017 8:31 UTC

  0 points

  in reply to: Daniel_Dewey's comment on: My cur­rent thoughts on MIRI’s “highly re­li­able agent de­sign” work

  Do you mean more promis­ing than other tech­ni­cal safety re­search (e.g. con­crete prob­lems, Paul’s di­rec­tions, MIRI’s non-HRAD re­search)?

  Yeah, and also (differ­en­tially) more promis­ing than AI strat­egy or AI policy work. But I’m not sure how strong the effect is.

  If so, I’d be in­ter­ested in hear­ing why you think hard /​ un­ex­pected take­off differ­en­tially fa­vors HRAD.

  In a hard /​ un­ex­pected take­off sce­nario, it’s more plau­si­ble that we need to get ev­ery­thing more or less ex­actly right to en­sure al­ign­ment, and that we have only one shot at it. This might fa­vor HRAD be­cause a less prin­ci­pled ap­proach makes it com­par­a­tively un­likely that we get all the fun­da­men­tals right when we build the first ad­vanced AI sys­tem.

  In con­trast, if we think there’s no such dis­con­ti­nu­ity and AI de­vel­op­ment will be grad­ual, then AI con­trol may be at least some­what more similar (but surely not en­tirely com­pa­rable) to how we “al­ign” con­tem­po­rary soft­ware sys­tems. That is, it would be more plau­si­ble that we could test ad­vanced AI sys­tems ex­ten­sively with­out risk­ing catas­trophic failure or that we could iter­a­tively try a va­ri­ety of safety ap­proaches to see what works best.

  It would also be more likely that we’d get warn­ing signs of po­ten­tial failure modes, so that it’s com­par­a­tively more vi­able to work on con­crete prob­lems when­ever they arise, or to fo­cus on mak­ing the solu­tions to such prob­lems scal­able – which, to my un­der­stand­ing, is a key com­po­nent of Paul’s ap­proach. In this pic­ture, suc­cess­ful al­ign­ment with­out un­der­stand­ing the the­o­ret­i­cal fun­da­men­tals is more likely, which makes non-HRAD ap­proaches more promis­ing.

  My per­sonal view is that I find a hard and un­ex­pected take­off un­likely, and ac­cord­ingly fa­vor other ap­proaches than HRAD, but of course I can’t jus­tify high con­fi­dence in this given ex­pert dis­agree­ment. Similarly, I’m not highly con­fi­dent that the above dis­tinc­tion is ac­tu­ally mean­ingful.

  I’d be in­ter­ested in hear­ing your thoughts on this!

• Tobias_Baumann 7 Jul 2017 14:49 UTC

  1 point

  in reply to: jpaddison's comment on: My cur­rent thoughts on MIRI’s “highly re­li­able agent de­sign” work

  Great post! I agree with your over­all as­sess­ment that other ap­proaches may be more promis­ing than HRAD.

  I’d like to add that this may (in part) de­pend on our out­look on which AI sce­nar­ios are likely. Con­di­tional on MIRI’s view that a hard or un­ex­pected take­off is likely, HRAD may be more promis­ing (though it’s still un­clear). If the take­off is soft or AI will be more like the econ­omy, then I per­son­ally think HRAD is un­likely to be the best way to shape ad­vanced AI.

  (I wrote a re­lated piece on strate­gic im­pli­ca­tions of AI sce­nar­ios.)

Strate­gic im­pli­ca­tions of AI scenarios

• Tobias_Baumann 10 Mar 2017 10:00 UTC

  11 points

  in reply to: jpaddison's comment on: The asym­me­try and the far future

  Thanks for your post! I agree that work on pre­vent­ing risks of fu­ture suffer­ing is highly valuable.

  It’s tempt­ing to say that it im­plies that the ex­pected value of a minis­cule in­crease in ex­is­ten­tial risk to all sen­tient life is as­tro­nom­i­cal.

  Even if the fu­ture is nega­tive ac­cord­ing to your val­ues, there are strong rea­sons not to in­crease ex­is­ten­tial risk. This would be ex­tremely un­co­op­er­a­tive to­wards other value sys­tems, and there are many good rea­sons to be nice to other value sys­tems. It is bet­ter to pull the rope side­ways by work­ing to im­prove the fu­ture (i.e. re­duc­ing risks of as­tro­nom­i­cal suffer­ing) con­di­tional on there be­ing a fu­ture.

  In ad­di­tion, I think it makes sense for util­i­tar­i­ans to adopt a quasi-de­on­tolog­i­cal rule against us­ing vi­o­lence, re­gard­less of whether one is a clas­si­cal util­i­tar­ian or suffer­ing-fo­cused. This ob­vi­ously pro­hibits some­thing like in­creas­ing risks of ex­tinc­tion.

• Tobias_Baumann 17 Oct 2016 0:35 UTC

  3 points

  in reply to: jpaddison's comment on: Stu­dents for High Im­pact Char­ity: Re­view and $10K Grant

  Thanks a lot, Peter, for tak­ing the time to eval­u­ate SHIC! I agree that their work seems to be very promis­ing.

  In par­tic­u­lar, it seems that stu­dents and fu­ture lead­ers are one of the most im­por­tant tar­get groups of effec­tive al­tru­ism.

• Tobias_Baumann 17 Oct 2016 0:31 UTC

  3 points

  in reply to: Tobias_Baumann's comment on: The map of or­ga­ni­za­tions, sites and peo­ple in­volved in x-risks prevention

  A minor de­tail: It’s a bit in­ac­cu­rate to say that the Foun­da­tional Re­search In­sti­tute works on gen­eral x-risks. This text ex­plains that FRI fo­cuses on re­duc­ing risks of as­tro­nom­i­cal suffer­ing, which is re­lated to, but not the same, as x-risk re­duc­tion.

• Tobias_Baumann 17 Oct 2016 0:27 UTC

  1 point

  in reply to: jpaddison's comment on: The map of or­ga­ni­za­tions, sites and peo­ple in­volved in x-risks prevention

  Thanks for this great map!