Lukas_Gloor

• Lukas_Gloor 15 Aug 2019 12:14 UTC

  4 points

  in reply to: SoerenMind's comment on: Log­a­r­ith­mic Scales of Plea­sure and Pain: Rat­ing, Rank­ing, and Com­par­ing Peak Ex­pe­riences Suggest the Ex­is­tence of Long Tails for Bliss and Suffering

  Hm.. I’m some­what new to this “RL per­spec­tive on an­i­mal be­hav­ior,” but from what I un­der­stand about it, I see a few caveats:

  • Prob­a­bly not all suffer­ing is re­lated to learn­ing in the same way. De­pres­sion cer­tainly comes with a sub­jec­tive wish for bet­ter­ment, but of­ten lacks any mo­ti­va­tion to seek bet­ter­ment.

  • Prob­a­bly the an­i­mal first needs to ex­pe­rience trau­matic pain for it to be­come pre­oc­cu­pied with it? This means that if ex­treme pain is rare, the claim in the OP could still be com­pat­i­ble with your ob­ser­va­tion that most an­i­mals aren’t pre­oc­cu­pied with avoid­ing it.

  You could ar­gue that the learn­ing from ex­treme pain doesn’t track the sub­jec­tive in­ten­sity of pain. But then peo­ple would be choos­ing e. g. a sub­jec­tively 10x worse pain over a <10x longer pain. In this cause I’d prob­a­bly say that the sub­jec­tive im­pres­sion is mis­guided or eth­i­cally ir­rele­vant, though that’s an eth­i­cal judg­ment.

  I share your in­tu­ition for very clear-cut choice situ­a­tions about two painful ex­pe­riences. But you could imag­ine cases where a per­son chooses one thing (i.e., dis­play some re­vealed prefer­ence), but feels like there’s an im­por­tant sense in which they’d rather be the sort of per­son who chooses the other thing. I’m not sure this ex­am­ple ap­plies to pure pain-vs.-pain com­par­i­sons, but it’s a rea­son I’m not on board with nor­ma­tive eval­u­a­tions that fo­cus solely on de­ci­sions taken af­ter hav­ing be­come ac­quainted with cer­tain ex­pe­riences. For ex­am­ple, if I’m pre­sented with ei­ther stay­ing in bed or leav­ing bed + be­ing sub­jected to elec­tro shock + get­ting re­warded, I’m sure you can make the re­ward high enough that, af­ter a few forced tri­als, I’ll start vol­un­tar­ily choos­ing “leav­ing the bed” over “stay­ing” ev­ery time. In this new situ­a­tion, I’d now be wak­ing up with in­tense long­ings for re­ward, long­ings painful enough that I’d pre­fer elec­tro shocks fol­lowed by satis­fac­tion over con­tinu­a­tion of those long­ings. Note that this is an al­to­gether differ­ent thought ex­per­i­ment com­pared to the origi­nal situ­a­tion where I was wak­ing up with­out long­ings. As you in­di­cate, it seems like a fur­ther ques­tion whether, in the newer ver­sion (af­ter ac­quain­tance with shock + re­ward), we want to look at this as choos­ing the thing we learned is bet­ter, or as choos­ing some­thing other than we would have cho­sen ini­tially be­cause we de­vel­oped some type of ad­dic­tion.

  More gen­er­ally, the ob­ser­va­tion I’d like to add to this (and similar) dis­cus­sions is that hu­mans seem to have two very differ­ent “modes” for se­lect­ing ac­tions. The mode where I’m lay­ing in bed com­fortably but alert and agenty enough to de­cide how to spend my morn­ing is a differ­ent, more “sys­tem-2-heavy” mode from one where I’m hav­ing hard-to-con­trol long­ings (or aver­sions). In the more sys­tem-2-heavy mode, peo­ple may care about things that are very differ­ent from max­i­miza­tion of ex­pected ex­pe­rienced re­ward. This can skew one’s re­vealed prefer­ences about pain (or plea­sure) trade­offs in all kinds of ways, mak­ing it com­pli­cated (to say the least) to take this RL per­spec­tive (which I view as be­ing pri­mar­ily fo­cused – at least in the sense that it’s “purer” there – on the more sys­tem-1-like mode) as the ba­sis of nor­ma­tive eval­u­a­tion.

  As you say, such nor­ma­tive eval­u­a­tion (as­sum­ing we are right about the de­scrip­tive fea­tures that make up the op­tion space) comes down to sub­jec­tive judg­ment calls, and I can see why you might have differ­ent in­tu­itions from me.

  (BTW I also found the claims in the OP sur­pris­ing; and I’m not sure yet whether I agree with them.)

• Lukas_Gloor 24 Jul 2019 12:00 UTC

  12 points

  on: What posts you are plan­ning on writ­ing?

  A se­quence on moral anti-re­al­ism and its implications

  I pub­lished the first­post “What is moral re­al­ism?” last year and have about five half-finished drafts stored some­where, but then I got side­tracked mas­sively. Ten­ta­tive ti­tles were:

  1. What is moral re­al­ism? [pub­lished]

  2. Against ir­re­ducible normativity

  3. Is there a wa­ger for moral re­al­ism?

  4. Me­taeth­i­cal fa­nat­i­cism (di­alogue about the strange im­pli­ca­tions of an in­finite “moral re­al­ism wa­ger”)

  5. [Un­ti­tled – some­thing about “Peo­ple aren’t born con­se­quen­tial­ists; peo­ple live their lives in differ­ent modes; vo­ca­tions are not just dis­cov­ered but also cho­sen”]

  6. In­tro­spec­tion-based moral realism

  7. Why I’m a moral anti-re­al­ist (se­quence sum­mary)

  8. Anti-re­al­ism is not nihilistic

  9. Anti-re­al­ism: What changes?

  • Less bul­let bit­ing?

  • Treat­ing peer dis­agree­ments about val­ues differently

  • Mo­ral un­cer­tainty vs. moral underdetermination

  I might find some time later this year to finish more of the posts, but I’m not sure I still want to do the en­tire se­quence. I con­sid­ered just skip­ping to posts 7. − 9. be­cause that used to be my origi­nal plan, but then the pro­ject some­how took on a much larger scale. I’d be cu­ri­ous to what de­gree there’s in­ter­est on the fol­low­ing top­ics:

  (a) What are the ar­gu­ments against (var­i­ous an­gles of) moral re­al­ism?

  (b) What is it that peo­ple are even do­ing when they do moral philos­o­phy?

  (c) What do anti-re­al­ists think they’re do­ing; why do they care?

  (d) Im­pli­ca­tions for moral rea­son­ing if anti-re­al­ism is correct

• Lukas_Gloor 14 Mar 2019 11:58 UTC

  1 point

  in reply to: agdfoster's comment on: Pri­mates vs birds: Is one brain ar­chi­tec­ture bet­ter than the other?

  The full anal­y­sis on AI im­pacts is also re­ally clearly ex­plained, or at least I got that im­pres­sion when I was read­ing it (I nor­mally can’t fol­low these types of pro­to­cols be­cause I get bored or lose track of the rele­vance of differ­ent things). I re­ally en­joyed read­ing it!

• Lukas_Gloor 5 Feb 2019 20:24 UTC

  10 points

  on: What we talk about when we talk about life satisfaction

  I feel like the is­sues with “How satis­fied are you with your life, on a scale of 0 to 10?” run even a bit deeper than in­di­cated. The se­ries “The Amer­i­cans” is about a mar­ried cou­ple of Soviet spies liv­ing a painful and difficult life in Amer­ica dur­ing the Cold War. The wife still be­lieves in the mis­sion, her hus­band not so much (but con­tinues do­ing the job for the sake of his wife). Let’s say the wife rates her life satis­fac­tion ⁹⁄₁₀ be­cause she con­vinced her­self that she’s bravely do­ing highly im­por­tant work. The hus­band rates his life satis­fac­tion ¹⁄₁₀. They’d both score about the same in terms of more ob­jec­tive met­rics like so­cioe­co­nomic sta­tus, how much time they spend on hob­bies or with their kids, how much stress they have, etc. But they in­ter­pret things differ­ently be­cause the wife as­cribes mean­ing to her hard­ships and is proud of her ac­com­plish­ments, while the hus­band feels trapped and like he wasted his life and en­dan­gered his chil­dren for no good rea­son.

• Lukas_Gloor 30 Dec 2018 4:30 UTC

  6 points

  in reply to: Wei_Dai's comment on: Why I ex­pect suc­cess­ful (nar­row) alignment

  “Between 1 and 10%” also feels sur­pris­ingly low to me for gen­eral AI-re­lated catas­tro­phes. I at least would have thought that ex­perts are less op­ti­mistic than that.

  But pend­ing clar­ifi­ca­tion, I wouldn’t put much weight on this es­ti­mate given that the in­ter­views men­tioned in the 80k prob­lem area pro­file you link to seemed to be about in­form­ing the en­tire prob­lem pro­file rather than this es­ti­mate speci­fi­cally. So it’s not clear e.g. whether the in­ter­views in­cluded a ques­tion about all-things-con­sid­ered risk for AI-re­lated catas­tro­phe that was asked to Nick Bostrom, an anony­mous lead­ing pro­fes­sor of com­puter sci­ence, Jaan Tal­linn, Jan Leike, Miles Brundage, Nate Soares, and Daniel Dewey.

• Lukas_Gloor 25 Dec 2018 23:24 UTC

  6 points

  on: Why I pri­ori­tize moral cir­cle ex­pan­sion over ar­tifi­cial in­tel­li­gence alignment

  Next to the coun­ter­points men­tioned by Gre­gory Lewis, I think there is an ad­di­tional rea­son why MCE seems less effec­tive than more tar­geted in­ter­ven­tions to im­prove the qual­ity of the long-term fu­ture: Gains from trade be­tween hu­mans with differ­ent val­ues be­come eas­ier to im­ple­ment as the reach of tech­nol­ogy in­creases. As long as a non-triv­ial frac­tion of hu­mans end up car­ing about an­i­mal wellbe­ing or digi­tal minds, it seems likely it would be cheap for other coal­i­tions to offer trades. So whether 10% of fu­ture peo­ple end up with an ex­panded moral cir­cle or 100% may not make much of a differ­ence to the out­come: It will be rea­son­ably good ei­ther way if peo­ple reap the gains from trade.

  One might ob­ject that it is un­likely that hu­mans would be able to co­op­er­ate effi­ciently, given that we don’t see this type of co­op­er­a­tion hap­pen­ing to­day. How­ever, I think it’s rea­son­able to as­sume that stay­ing in con­trol of tech­nolog­i­cal progress be­yond the AGI tran­si­tion re­quires a de­gree of wis­dom and fore­sight that is very far away from where most so­cietal groups are at to­day. And if hu­mans do stay in con­trol, then find­ing a good solu­tion for value dis­agree­ments may be the eas­ier prob­lem, or at worst similarly hard. So it feels to me that most likely, we ei­ther we get a fu­ture that goes badly for rea­sons re­lated to lack of co­or­di­na­tion and so­phis­ti­ca­tion in the pre-AGI stage, or we get a fu­ture where hu­mans set things up wisely enough to ac­tu­ally de­sign an out­come that is nice (or at least not amongst the 10% of worst out­comes) by the lights of nearly ev­ery­one.

  Brian To­masik made the point that con­di­tional on hu­man val­ues stay­ing in con­trol, we may be very un­likely to get some­thing like broad moral re­flec­tion. In­stead, val­ues could be de­ter­mined by a very small group of in­di­vi­d­u­als who hap­pened to be in power by the time AGI ar­rives (as op­posed to in­di­vi­d­u­als end­ing up there be­cause they were un­usu­ally fore­sighted and also morally mo­ti­vated). This feels pos­si­ble too, but it seems to not be the likely de­fault to me be­cause I sus­pect that you’d need to nec­es­sar­ily in­crease your philo­soph­i­cal so­phis­ti­ca­tion in or­der to stay in con­trol of AGI, and that prob­a­bly gives you more pleas­ant out­comes (cor­re­la­tional claim). Iter­ated am­plifi­ca­tion for in­stance, as an ap­proach to AI al­ign­ment, has sev­eral uses for hu­mans: Hu­mans are not only where the re­sult­ing val­ues come from, but they’re also in charge of keep­ing the boot­strap­ping pro­cess on track and cor­rigible. And as this post on fac­tored cog­ni­tion illus­trates, this re­quires so­phis­ti­ca­tion to set up. So if that’s the bar that AGI cre­ators need to pass be­fore they can de­ter­mine how “hu­man val­ues” are to be ex­trap­o­lated, maybe we shouldn’t be too pes­simistic about the out­come. It seems kind of un­likely that some­one would go through all of that only to be like “I’m go­ing to im­ple­ment my per­sonal best guess about what mat­ters to me, with lit­tle fur­ther re­flec­tion, and no other hu­mans get a say here.” Similarly, it also feels un­likely that peo­ple would go through with all that and not find a way to make sub­parts of the pop­u­la­tion rea­son­ably con­tent about how sen­tient sub­rou­tines are go­ing to be used.

  Now, I feel a bit con­fused about the fea­si­bil­ity of AI al­ign­ment if you were to do it some­what slop­pily and with lower stan­dards. I think that there’s a spec­trum from “it just wouldn’t work at all and not be com­pet­i­tive” (and then peo­ple would have to try some other ap­proach) to “it would pro­duce a ca­pa­ble AGI but it would be vuln­er­a­ble to failure modes like ad­ver­sar­ial ex­ploits or op­ti­miza­tion dae­mons, and so it would end up with not hu­man val­ues”. Th­ese failure modes, to the very small de­gree I cur­rently un­der­stand them, sound like they would not be sen­si­tive to whether the hu­man whose ap­proval you tried to ap­prox­i­mate had an ex­panded moral cir­cle or not. I might be wrong about that. If peo­ple mostly want so­phis­ti­cated al­ign­ment pro­ce­dures be­cause they care about pre­serv­ing the op­tion for philo­soph­i­cal re­flec­tion, rather than be­cause they also think that you sim­ply run into large failure modes oth­er­wise, then it seems like (con­di­tional on some kind of value al­ign­ment) whether we get an out­come with broad moral re­flec­tion is not so clear. If it’s tech­ni­cally eas­ier to build value-al­igned AI with very parochial val­ues, then MCE could make a rele­vant differ­ence to these non-re­flec­tion out­comes.

  But all in all my ar­gu­ment is that it’s some­what strange to as­sume that a group of peo­ple could suc­ceed at build­ing an AGI op­ti­mized for its cre­ators’ val­ues, with­out hav­ing to put in so much think­ing about how to get this out­come right that they’d al­most can’t help but be­come rea­son­ably philo­soph­i­cally so­phis­ti­cated in the pro­cess. And sure, philo­soph­i­cally so­phis­ti­cated peo­ple can still have fairly strange val­ues by your own lights, but it seems like there’s more con­ver­gence. Plus I’d at least be op­ti­mistic about their propen­sity to strive to­wards pos­i­tive-sum out­comes, given how lit­tle scarcity you’d have if the tran­si­tion does go well.

  Of course, maybe value-al­ign­ment is go­ing to work very differ­ently from what peo­ple cur­rently think. The main way I’d crit­i­cize my above points is that they’re based on heavy-handed in­side-view think­ing about how difficult I (and oth­ers I’m up­dat­ing to­wards) ex­pect the AGI tran­si­tion to be. If AGI will be more like the In­dus­trial Revolu­tion rather than some­thing that is even more difficult to stay re­motely in con­trol of, or if some other tech­nol­ogy proves to be more con­se­quen­tial than AGI, then my ar­gu­ment has less force. I mainly see this as yet an­other rea­son to caveat that the ex ante plau­si­ble-seem­ing po­si­tion that MCE can have a strong im­pact on AGI out­comes starts to feel more and more con­junc­tive the more you zoom in and try to iden­tify con­crete path­ways.

• Lukas_Gloor 23 Dec 2018 0:41 UTC

  5 points

  in reply to: Ben Pace's comment on: Should donor lot­tery win­ners write re­ports?

  Other rea­sons why some­one com­pe­tent at pick­ing grants may not feel com­fortable with the thought of hav­ing to write a re­port might be be­cause writ­ing speci­fi­cally isn’t their strength or be­cause ex­pos­ing their think­ing to pub­lic scrutiny might be anx­iety-in­duc­ing.

• Lukas_Gloor 12 Dec 2018 13:41 UTC

  5 points

  in reply to: Ben_West's comment on: Why I’m fo­cus­ing on in­ver­te­brate sentience

  There’s some re­lated dis­cus­sion here.

  Lock in can also ap­ply to “value-pre­cur­sors” that de­ter­mine how one goes about moral re­flec­tion, or which types of ap­peals one ends up find­ing con­vinc­ing. I think these would get locked in to some de­gree (be­cause some­thing has to be fixed for it to be mean­ingful to talk about goal­posts at all), and by af­fect­ing the pre­cur­sors, moral or meta-philo­soph­i­cal re­flec­tion be­fore al­igned AGI can plau­si­bly af­fect the out­comes post-AGI. It’s not very clear how­ever whether that’s im­por­tant, and from whose per­spec­tive it is im­por­tant, be­cause some of the things that mask as moral un­cer­tainty might be hu­mans hav­ing un­der­de­ter­mined val­ues.

• Lukas_Gloor 29 Nov 2018 0:18 UTC

  8 points

  in reply to: Ozzie Gooen's comment on: Launch­ing the EAF Fund

  Yeah. I put it the fol­low­ing way in an­other post:

  Espe­cially when it comes to the pre­ven­tion of s-risks af­fect­ing fu­tures that oth­er­wise con­tain a lot of hap­piness, it mat­ters a great deal how the risk in ques­tion is be­ing pre­vented. For in­stance, if we en­vi­sion a fu­ture that is utopian in many re­spects ex­cept for a small por­tion of the pop­u­la­tion suffer­ing be­cause of prob­lem x, it is in the in­ter­est of vir­tu­ally all value sys­tems to solve prob­lem x in highly tar­geted ways that move prob­a­bil­ity mass to­wards even bet­ter fu­tures. By con­trast, only few value sys­tems (ones that are strongly or ex­clu­sively about re­duc­ing suffer­ing/​bad things) would con­sider it over­all good if prob­lem x was “solved” in a way that not only pre­vented the suffer­ing due to prob­lem x, but also pre­vented all the hap­piness from the fu­ture sce­nario this suffer­ing was em­bed­ded in.

  So it’d be to­tally fine to ad­dress all sources of un­nec­es­sary suffer­ing (and even “small” s-risks em­bed­ded in an oth­er­wise pos­i­tive fu­ture) if there are tar­geted ways to bring about un­con­tro­ver­sial im­prove­ments. :) In prac­tice, it’s some­times hard to find in­ter­ven­tions that are tar­geted enough be­cause af­fect­ing the fu­ture is very very difficult and we only have crude lev­ers. Hav­ing said that, I think many things that we’re go­ing to sup­port with the fund are ac­tu­ally quite pos­i­tive for pos­i­tive-fu­ture-ori­ented value sys­tems as well. So there cer­tainly are some more tar­geted lev­ers.

  There are in­stances where it does feel jus­tified to me to also move some prob­a­bil­ity mass away from s-risks to­wards ex­tinc­tion (or pa­per­clip sce­nar­ios), but that should be re­served ei­ther for un­con­tro­ver­sially ter­rible fu­tures, or for those fu­tures where most of the dis­value for down­side-fo­cused value sys­tems comes from. I doubt that this in­cludes fu­tures where 10^10x more peo­ple are happy than un­happy.

  And of course pos­i­tive-fu­ture-ori­ented EAs face analo­gous trade­offs of co­op­er­a­tion with other value sys­tems.

• Lukas_Gloor 28 Nov 2018 23:13 UTC

  22 points

  in reply to: Jan_Kulveit's comment on: Launch­ing the EAF Fund

  At least for me, this would be a pretty amaz­ing out­come, and not some­thing which should be pre­vented.

  Yeah, we’re go­ing to change the part that equates “worst case” with “s-risks”. Your view is com­mon and re­flects many eth­i­cal per­spec­tives.

  We were already think­ing about chang­ing the defi­ni­tion of “s-risk” based on similar feed­back, to make it more in­tu­itive and co­op­er­a­tive in the way you de­scribe. It prob­a­bly makes more sense to have it re­fer to only the few % of sce­nar­ios where most of the fu­ture’s ex­pected suffer­ing comes from (as­sum­ing s-risks are in­deed heavy-tailed). Th­ese ac­tual worst cases are what we want to fo­cus on with the fund.

  do I in­ter­pret it cor­rectly that in the eth­i­cal sys­tem held by the fund hu­man ex­tinc­tion is com­par­a­tively be­nign out­come in com­par­i­son with risks like cre­ation of 10^25 un­happy minds even if they are offset by much larger [say 10^10x larger] num­ber of happy minds?

  No, that’s in­cor­rect. In­so­far as some fund man­agers hold this view per­son­ally (e.g., I do, while Jonas would agree with you that the lat­ter out­come is vastly bet­ter), it won’t af­fect de­ci­sions be­cause in any case, we want to avoid do­ing things that are weakly pos­i­tive on some plau­si­ble moral views and very nega­tive on oth­ers. But I can see why you were con­cerned, and thanks for rais­ing this is­sue!

• Lukas_Gloor 5 Jun 2018 18:27 UTC

  1 point

  in reply to: MagnusVinding's comment on: 1. What Is Mo­ral Real­ism?

  To turn the ta­bles a bit here, I would say that to re­ject moral re­al­ism, on my ac­count, one would need to say that there is no gen­uine nor­ma­tive force or prop­erty in, say, a state of ex­treme suffer­ing (con­sider be­ing fried in a brazen bull for con­crete­ness).

  Cool! I think the clos­est I’ll come to dis­cussing this view is in foot­note 18. I plan to have a post on moral re­al­ism via in­tro­spec­tion about the in­trin­sic good­ness (or bad­ness) of cer­tain con­scious states.

  I agree with re­duc­tion­ism about per­sonal iden­tity and I also find this to be one of the most per­sua­sive ar­gu­ments in fa­vor of al­tru­is­tic life goals. I would not call my­self an open in­d­vi­d­u­al­ist though be­cause I’m not sure what the po­si­tion is ex­actly say­ing. For in­stance, I don’t un­der­stand how it differs from empty in­di­vi­d­u­al­ism. I’d un­der­stand if these are differ­ent fram­ings or differ­ent metaphores, but if we as­sume that we’re talk­ing about po­si­tions that can be true or false, I don’t un­der­stand what we’re ar­gu­ing about when ask­ing whether open in­di­vi­d­u­al­ism or true, or when dis­cussing open vs. empty in­di­vi­d­u­al­ism.
  Also, I think it’s perfectly co­her­ent to have ego­is­tic goals even un­der a re­duc­tion­ist view of per­sonal iden­tity. (It just turns out that ego­ism is not a well-defined con­cept ei­ther, and one has to make some judg­ment calls if one ever ex­pects to en­counter edge-cases for which our in­tu­itions give no ob­vi­ous an­swers about whether some­thing is still “me.”)

  With re­spect to the One Com­pel­ling Ax­iol­ogy you men­tion, Lukas, I am not sure why you would set the bar so high in terms of speci­fic­ity in or­der to ac­cept a re­al­ist view. I mean, if “all philoso­phers or philo­soph­i­cally-in­clined rea­son­ers” found plau­si­ble a sim­ple, yet in­ex­haus­tive prin­ci­ple like “re­duce un­nec­es­sary suffer­ing” why would that not be good enough to demon­strate its “re­al­ism” (on your ac­count) when a more spe­cific one would? It is un­clear to me why greater speci­fic­ity should be im­por­tant, es­pe­cially since even such an un­spe­cific prin­ci­ple still would have plenty of prac­ti­cal rele­vance (many peo­ple can ad­mit that they are not liv­ing in ac­cor­dance with this prin­ci­ple, even as they do ac­cept it).

  Yeah, fair point. I mean, even Rail­ton’s own view has plenty of prac­ti­cal rele­vance in the sense that it high­lights that cer­tain so­cietal ar­range­ments lead to more over­all well-be­ing or life satis­fac­tion than oth­ers. (That’s also a point that Sam Har­ris makes.) But if that’s all we mean by “moral re­al­ism” then it would be rather triv­ial. Maybe my crite­ria are a bit too strict, and I would in­deed already re­gard it as ex­tremely sur­pris­ing if you get some­thing like One Com­pel­ling Ax­iol­ogy that agrees on pop­u­la­tion ethics while leav­ing a few other things un­der­de­ter­mined.

• Lukas_Gloor 31 May 2018 9:17 UTC

  1 point

  in reply to: LanceSBush's comment on: 1. What Is Mo­ral Real­ism?

  In­spired by an­other mes­sage of yours, there’s at least one im­por­tant link here that I failed to men­tion: If moral dis­course is about a, b, and c, and philoso­phers then say they want to make it about q and ar­gue for re­al­ism about q, we can ob­ject that what­ever they may have shown us re­gard­ing re­al­ism about q, it’s cer­tainly not moral re­al­ism. And it looks like the Loeb pa­per also ar­gues that if moral dis­course is about mu­tu­ally in­com­pat­i­ble things, that looks quite bad for moral re­al­ism? Those are good points!

• Lukas_Gloor 31 May 2018 8:53 UTC

  1 point

  in reply to: kbog's comment on: 1. What Is Mo­ral Real­ism?

  Do you think your ar­gu­ment also works against Rail­ton’s moral nat­u­ral­ism, or does my One Com­pel­ling Ax­iol­ogy (OTA) pro­posal in­tro­duce some­thing that breaks the idea? The way I meant it, OTA is just a more ex­treme ver­sion of Rail­ton’s view.

  I think I can see what you’re point­ing to though. I wrote:

  Note that this pro­posal makes no claims about the lin­guis­tic level: I’m not say­ing that or­di­nary moral dis­course let’s us define moral­ity as con­ver­gence in peo­ple’s moral views af­ter philo­soph­i­cal re­flec­tion un­der ideal con­di­tions. (This would be a cir­cu­lar defi­ni­tion.) In­stead, I am fo­cus­ing on the as­pect that such con­ver­gence would be prac­ti­cally rele­vant: [...]

  So yes, this would be a bad pro­posal for what moral dis­course is about. But it’s meant like this: Rail­ton claims that moral­ity is about do­ing things that are “good for oth­ers from an im­par­tial per­spec­tive.” I like this and wanted to work with that, so I adopt this as­sump­tion, and fur­ther add that I only want to call a view moral re­al­ism if “do­ing what is good for oth­ers from an im­par­tial per­spec­tive” is well-speci­fied. Then I give some ac­count of what it would mean for it to be well-speci­fied.

  In my pro­posal, moral facts are not defined as that which peo­ple ar­rive at af­ter re­flec­tion. Mo­ral facts are still defined as the same thing Rail­ton means. I’m just adding that maybe there are no moral facts in the way Rail­ton means if we in­tro­duce the ad­di­tional re­quire­ment that (strong) un­der­de­ter­mi­na­tion is not al­lowed.

• Lukas_Gloor 25 May 2018 16:18 UTC

  1 point

  in reply to: konrad's comment on: 1. What Is Mo­ral Real­ism?

  Prob­a­bly in­tu­itions about this is­sue de­pend on which type of moral or re­li­gious dis­course one is used to. As some­one who spent a year at a Chris­tian pri­vate school in Texas where cre­ation­ism was taught in Biol­ogy class and God and Je­sus were a very tan­gible part of at least some peo­ple’s lives, I definitely got a strong sense that the meta­phys­i­cal ques­tions are ex­tremely im­por­tant.

  By con­trast, if the only type of re­li­gious claims I’d ever came into con­tact with had been mod­er­ate (pic­ture the av­er­age level of re­li­gios­ity of a per­son in, say, Zurich), then one may even con­sider it a bit of a straw­man to as­sume that re­li­gious claims are to be taken liter­ally.

  I think this con­cern is some­what rele­vant to the broader dis­cus­sion, too, be­cause you seem to im­ply that we can’t (or even shouldn’t?) make any ad­vances on non-meta­phys­i­cal claims be­fore we haven’t figured out the meta­phys­i­cal ones.

  Just to be clear, all I’m say­ing is that I think it’s go­ing to be less use­ful to dis­cuss “what are moral claims usu­ally about.” What we should in­stead do is in­stead what Chalmers de­scribes (see the quote in foot­note 4). Dis­cussing what moral claims are usu­ally about is not the same as mak­ing up one’s mind about nor­ma­tive ethics. I think it’s very use­ful to dis­cuss nor­ma­tive ethics, and I’d even say that dis­cussing whether anti-re­al­ism or re­al­ism is true might be slightly less im­por­tant than mak­ing up one’s mind about nor­ma­tive ethics. Sure, it in­forms to some ex­tent how to rea­son about moral­ity, but as has been pointed out, you can make some progress about moral ques­tions also from a lens of ag­nos­ti­cism about re­al­ism vs. anti-re­al­ism.

  To go back to the re­li­gion anal­ogy, what I’m recom­mend­ing is to first figure out whether you be­lieve in a God or an af­ter­life that would rele­vantly in­fluence your pri­ori­ties now, and not worry much about whether re­li­gious claims are “usu­ally” or “best” to be taken liter­ally or taken metaphor­i­cally(?).

• Lukas_Gloor 23 May 2018 13:13 UTC

  3 points

  in reply to: jayquigley's comment on: 1. What Is Mo­ral Real­ism?

  Why should we care whether or not moral re­al­ism is true?

  I plan to ad­dress this more in a fu­ture post, but the short an­swer is this that for some ways in which moral re­al­ism has been defined, it re­ally doesn’t mat­ter (much). But there are some ver­sions of moral re­al­ism that would “change the game” for those peo­ple who cur­rently re­ject them. And vice-versa, if one cur­rently en­dorses a view that cor­re­sponds to the two ver­sions of “strong moral re­al­ism” de­scribed in the last sec­tion of my post, one’s pri­ori­ties could change no­tice­ably if one changes one’s mind to­wards anti-re­al­ism.

  What do you think are the im­pli­ca­tions of moral anti-re­al­ism for choos­ing al­tru­is­tic ac­tivi­ties?

  It’s hard to sum­ma­rize this suc­cinctly be­cause for most of the things that are straight­for­wardly im­por­tant un­der moral re­al­ism (such as moral un­cer­tainty or defer­ring judg­ment to fu­ture peo­ple who are more knowl­edge­able about moral­ity), you can also make good ar­gu­ments in fa­vor of them go­ing from anti-re­al­ist premises. Some quick thoughts:

  – The main differ­ence is that things be­come more “messy” with anti-re­al­ism.

  – I think anti-re­al­ists should, all else equal, be more re­luc­tant to en­gage in “bul­let bit­ing” where you aban­don some of your moral in­tu­itions in fa­vor of mak­ing your moral view “sim­pler” or “more el­e­gant.” The sim­plic­ity/​el­e­gance ap­peal is that if you have a view with many pa­ram­e­ters that are fine-tuned for your per­sonal in­tu­itions, it seems ex­tremely un­likely that other peo­ple would come up with the same pa­ram­e­ters if they only thought about moral­ity more. Mo­ral re­al­ists may think that the cor­rect an­swer to moral­ity is one that ev­ery­one who is knowl­edge­able enough would en­dorse, whereas anti-re­al­ists may con­sider this a po­ten­tially im­pos­si­ble de­mand and there­fore place more weight on find­ing some­thing that feels very in­tu­itively com­pel­ling on the in­di­vi­d­ual level. Hav­ing said that, I think there are a num­ber of ar­gu­ments why even an anti-re­al­ist might want to adopt moral views that are “sim­ple and el­e­gant.” For in­stance, peo­ple may care about do­ing some­thing mean­ingful that is “greater than their own petty lit­tle in­tu­itions” – I think this is an in­tu­ition that we can try to cash out some­how even if moral re­al­ism turns out to be false (it’s just that it can be cashed out in differ­ent ways).

  – “Mo­ral un­cer­tainty” works differ­ently un­der anti-re­al­ism, be­cause you have to say what you are un­cer­tain about (it can­not be the one true moral­ity be­cause anti-re­al­ism says there is no such thing). One can be un­cer­tain about what one would value af­ter moral re­flec­tion un­der ideal con­di­tions. This kind of “valu­ing moral re­flec­tion” seems like a very use­ful anti-re­al­ist al­ter­na­tive to moral un­cer­tainty. The differ­ence is that “valu­ing re­flec­tion” may be un­der­defined, so anti-re­al­ists have to think about how to dis­t­in­guish hav­ing un­der­defined val­ues from be­ing un­cer­tain about their val­ues. This part can get tricky.

  – There was re­cently a dis­cus­sion about “goal drift” in the EA fo­rum. I think it’s a big­ger prob­lem with anti-re­al­ism all else equal (un­less one’s anti-re­al­ist moral view is ego­ism-re­lated.) But again, there are con­sid­er­a­tions that go into both di­rec­tions. :)

• Lukas_Gloor 22 May 2018 22:49 UTC

  1 point

  in reply to: jayquigley's comment on: 1. What Is Mo­ral Real­ism?

  The idea is that moral­ity is the set of rules that im­par­tial, ra­tio­nal peo­ple would ad­vo­cate as a pub­lic sys­tem.

  Yes, this sounds like con­struc­tivism. I think this is definitely a use­ful frame­work for think­ing about some moral/​moral­ity-re­lated ques­tions. I don’t think all of moral dis­course is best con­strued as be­ing about this type of hy­po­thet­i­cal rule-mak­ing, but like I say in the post, I don’t think in­ter­pret­ing moral dis­course should be the pri­mary fo­cus.

  Ra­tion­al­ity is un­der­stood, roughly speak­ing, as the set of things that vir­tu­ally all ra­tio­nal agents would be averse to. This ends up be­ing a list of ba­sic harms—things like pain, death, dis­abil­ity, in­jury, loss of free­dom, loss of plea­sure.

  Hm, this sounds like you’re talk­ing about a sub­stan­tive con­cept of ra­tio­nal­ity, as op­posed to a merely “pro­ce­du­ral” or “in­stru­men­tal” con­cept of ra­tio­nal­ity (such as it’s com­mon on Less­wrong and with anti-re­al­ist philoso­phers like Bernard Willi­ams). Sub­stan­tive con­cepts of ra­tio­nally always go un­der moral non-nat­u­ral­ism, I think.

  My post is a lit­tle con­fus­ing with re­spect to the dis­tinc­tion here, be­cause you can be a con­struc­tivist in two differ­ent ways: Pri­mar­ily as an in­ter­sub­jec­tivist metaeth­i­cal po­si­tion, and “sec­on­dar­ily” as a form of non-nat­u­ral­ism. (See my com­ments on Thomas Sit­tler’s chart.)

  Peo­ple can be in­cor­rect about whether a thing is harm­ful, just as they can be in­cor­rect about whether a flower is red. But there’s noth­ing much more ob­jec­tive or “facty” about whether the plant is red than that or­di­nary hu­man lan­guage users on earth are dis­posed to see and la­bel it as red.

  Yeah, it should be noted that “strong” ver­sions of moral re­al­ism are not com­mit­ted to silly views such as moral­ity ex­ist­ing in some kind of su­per­nat­u­ral realm. I of­ten find it difficult to ex­plain moral non-nat­u­ral­ism in a way that makes it sound as non-weird as when ac­tual moral non-nat­u­ral­ists write about it, so I have to be care­ful to not straw­man these po­si­tions. But what you de­scribe may still qual­ify as “strong” be­cause you’re talk­ing about ra­tio­nal­ity as a sub­stan­tive con­cept. (Clas­sify­ing some­thing as a “harm” is one thing if done in a de­scrip­tive sense, but prob­a­bly you’re talk­ing about clas­sify­ing things as a harm in a sense that has moral con­no­ta­tions – and that gets into more con­tro­ver­sial ter­ri­tory.)

  The book ti­tle “nor­ma­tive bedrock” also sounds rele­vant be­cause my next post will talk about “bedrock con­cepts” (Chalmers) at length, and speci­fi­cally about “ir­re­ducible nor­ma­tivity” as a bedrock con­cept, which I think makes up the core of moral non-nat­u­ral­ism.

• Lukas_Gloor 22 May 2018 22:10 UTC

  4 points

  in reply to: [deleted]'s comment on: 1. What Is Mo­ral Real­ism?

  Cool! I think this is helpful both in it­self as well as here as a com­ple­ment to my post. I also thought about mak­ing a chart but was too lazy in the end. If I may, I’ll add some com­ments about how this chart re­lates to the dis­tinc­tions I made/​kept/​found:

  “Judg­ment-de­pen­dent cog­ni­tivism” cor­re­sponds to what I la­bel­led sub­jec­tivism and in­ter­sub­jec­tivism, and “judg­ment-_in_de­pen­dent cog­ni­tivism” cor­re­sponds to “ob­jec­tivism.” (Ter­minol­ogy adopted from the Sayre-McCord es­say; but see the Scan­lon es­say for the other ter­minol­ogy.)

  I’m guess­ing “Kan­tian ra­tio­nal­ism” refers to views such as Scan­lon’s view. I’m didn’t go into de­tail in my post with ex­plain­ing the differ­ence be­tween con­struc­tivism as an in­ter­sub­jec­tivist po­si­tion and con­struc­tivism as a ver­sion of non-nat­u­ral­ism. I tried to say some­thing about that in foot­note 7 but I fear it’ll only be­come more clear in my next post. Tl;dr is that non-nat­u­ral­ists think that we can have ex­ter­nal­ist rea­sons for do­ing some­thing, rea­sons we can­not “shake off” by lack­ing in­ter­nal buy-in or in­ter­nal mo­ti­va­tion. By con­trast, some­one who merely en­dorses con­struc­tivism as an in­ter­sub­jec­tivist (or “judg­ment-de­pen­dent”) view, such as Kors­gaard for in­stance, would re­ject these ex­ter­nal­ist rea­sons.

  I agree with the way you draw the lines be­tween the re­al­ist and the anti-re­al­ist camp. The only thing I don’t like about this (and this is a crit­i­cism not about your chart, but about the way philoso­phers have drawn these cat­e­gories in the first place) is that it makes it seem as though we have to choose ex­actly one view. But by re­mov­ing the en­tire dis­cus­sion from the “lin­guis­tic level” (tak­ing a stance on how we in­ter­pret moral dis­course), we can ac­knowl­edge e.g. that sub­jec­tivism or in­ter­sub­jec­tivism rep­re­sent use­ful frame­works for think­ing about moral­ity-re­lated ques­tions, whether moral dis­course is com­pletely sub­jec­tivist or in­ter­sub­jec­tivist in na­ture or not. And even if moral dis­course was all sub­jec­tivist (which seems clearly wrong to me but let’s say it’s a hy­po­thet­i­cal), for all we’d know that could still al­low for the pos­si­bil­ity that an ob­jec­tivist moral re­al­ity ex­ists in a mean­ingful and pos­si­bly ac­tion-rele­vant sense. I like Luke’s fram­ing of “plu­ral­is­tic moral re­duc­tion­ism” be­cause it makes clear that there is more than one op­tion.

1. What Is Mo­ral Real­ism?

• Lukas_Gloor 21 May 2018 11:03 UTC

  2 points

  on: “EA” doesn’t have a tal­ent gap. Differ­ent causes have differ­ent gaps.

  Ta­lent gap—Mid­dle (~50 peo­ple)

  If the AI safety/​al­ign­ment com­mu­nity is al­to­gether around 50 peo­ple, that’s a large rel­a­tive gap. Depend­ing on how you count it might be big­ger than 50 peo­ple, but the tal­ent gap seems large in rel­a­tive terms in ei­ther case. :)

• Lukas_Gloor 24 Apr 2018 12:04 UTC

  3 points

  in reply to: Gregory_Lewis's comment on: Em­piri­cal data on value drift

  It is much less clear whether this per­son would think they’ve made a mis­take in al­lo­cat­ing more of them­selves away from EA, ei­ther at t2-now (they don’t re­gret they now have a fam­ily which takes their at­ten­tion away from EA things), or at t1-past (if their pre­vi­ous EA-self could fore­cast them be­ing in this situ­a­tion, they would not be dis­ap­pointed in them­selves). If so, these would not be op­tions that their t1-self should be try­ing to shut off, as (all things con­sid­ered) the op­tion might be on bal­ance good. I am sure there are cases where ‘life gets in the way’ in a man­ner it is rea­son­able to re­gret. But I would be chary [...]

  You dis­cuss a case where there is re­gret from the per­spec­tive of both t1 and t2, and a case where there is re­gret from nei­ther per­spec­tive. Th­ese are both plau­si­ble ac­counts. But there’s also a third op­tion that I think hap­pens a lot in prac­tice: Re­gret at t1 about the pro­jected fu­ture in ques­tion, and less/​no re­gret at t2. So the t2-self may talk about “hav­ing be­come more wise” or “hav­ing learned some­thing about my­self,” while the t1-self would not be on board with this de­scrip­tion and con­sider the fu­ture in ques­tion to be an un­for­tu­nate turn of events. (Or the t2-self could even ac­knowl­edge that some de­ci­sions in the se­quence were not ra­tio­nal, but that from their cur­rent per­spec­tive, they re­ally like the way things are.)

  The dis­tinc­tion be­tween moral in­sight and failure of goal preser­va­tion is fuzzy. Tak­ing pre­cau­tions against goal drift is a form of fa­nat­i­cism and com­mon­sense heuris­tics speak against that. OTOH, not tak­ing pre­cau­tions seems like not tak­ing the things you cur­rently care about se­ri­ously (at least in­so­far as there are things you care about that go be­yond as­pects re­lated to your per­sonal de­vel­op­ment).

  Un­for­tu­nately I don’t think there is a safe de­fault. Not tak­ing pre­cau­tions is tan­ta­mount to mak­ing the de­ci­sion to be okay with po­ten­tial value drift. And we can­not just say we are un­cer­tain about our val­ues, be­cause that could re­sult in mis­tak­ing un­cer­tainty with un­der­de­ter­mi­na­tion. There are mean­ingful ways of valu­ing fur­ther re­flec­tion about one’s own val­ues, but those types of “in­di­rect” val­ues, where one val­ues fur­ther re­flec­tion, they can also suffer from (more sub­tle) forms of goal drift.