Flodorner

• Flodorner 15 Feb 2019 17:18 UTC

  2 points

  on: Three Bi­ases That Made Me Believe in AI Risk

  “The com­bi­na­tion of these vastly differ­ent ex­pres­sions of scale to­gether with an­chor­ing makes that we should ex­pect peo­ple to over-es­ti­mate the prob­a­bil­ity of un­likely risks and hence to over-es­ti­mate the ex­pected util­ity of x-risk pre­ven­tion mea­sures. ”

  I am not en­tirely sure whether i un­der­stand this point. Is the ar­gu­ment that the an­chor­ing effect would cause an over­es­ti­ma­tion, be­cause the “per­ceived dis­tance” from an an­chor grows faster per added zero than per in­crease of one to the ex­po­nent?

• Flodorner 15 Dec 2018 20:28 UTC

  4 points

  in reply to: rohinmshah's comment on: Cri­tique of Su­per­in­tel­li­gence Part 2

  Directly rele­vant quotes from the ar­ti­cles for eas­ier refer­ence:

  Paul Chris­ti­ano:

  “This story seems con­sis­tent with the his­tor­i­cal record. Things are usu­ally pre­ceded by worse ver­sions, even in cases where there are weak rea­sons to ex­pect a dis­con­tin­u­ous jump. The best coun­terex­am­ple is prob­a­bly nu­clear weapons. But in that case there were sev­eral very strong rea­sons for dis­con­ti­nu­ity: physics has an in­her­ent gap be­tween chem­i­cal and nu­clear en­ergy den­sity, nu­clear chain re­ac­tions re­quire a large min­i­mum scale, and the dy­nam­ics of war are very sen­si­tive to en­ergy den­sity.”

  “I’m not aware of many his­tor­i­cal ex­am­ples of this phe­nomenon (and no re­ally good ex­am­ples)—to the ex­tent that there have been “key in­sights” needed to make some­thing im­por­tant work, the first ver­sion of the in­sight has al­most always ei­ther been dis­cov­ered long be­fore it was needed, or dis­cov­ered in a pre­limi­nary and weak ver­sion which is then iter­a­tively im­proved over a long time pe­riod. ”

  “Over the course of train­ing, ML sys­tems typ­i­cally go quite quickly from “re­ally lame” to “re­ally awe­some”—over the timescale of days, not months or years.

  But the train­ing curve seems al­most ir­rele­vant to take­off speeds. The ques­tion is: how much bet­ter is your AGI then the AGI that you were able to train 6 months ago?”

  AIIm­pacts:

  “Dis­con­ti­nu­ities larger than around ten years of past progress in one ad­vance seem to be rare in tech­nolog­i­cal progress on nat­u­ral and de­sir­able met­rics. We have ver­ified around five ex­am­ples, and know of sev­eral other likely cases, though have not com­pleted this in­ves­ti­ga­tion. ”

  “Sup­pos­ing that AlphaZero did rep­re­sent dis­con­ti­nu­ity on play­ing mul­ti­ple games us­ing the same sys­tem, there re­mains a ques­tion of whether that is a met­ric of suffi­cient in­ter­est to any­one that effort has been put into it. We have not in­ves­ti­gated this.

  Whether or not this case rep­re­sents a large dis­con­ti­nu­ity, if it is the only one among re­cent progress on a large num­ber of fronts, it is not clear that this raises the ex­pec­ta­tion of dis­con­ti­nu­ities in AI very much, and in par­tic­u­lar does not seem to sug­gest dis­con­ti­nu­ity should be ex­pected in any other spe­cific place.”

  “We have not in­ves­ti­gated the claims this ar­gu­ment is premised on, or ex­am­ined other AI progress es­pe­cially closely for dis­con­ti­nu­ities.”

• Flodorner 14 Dec 2018 10:55 UTC

  2 points

  on: Cri­tique of Su­per­in­tel­li­gence Part 2

  Another point against the con­tent over­hang ar­gu­ment: While more data is definitely use­ful, it is not clear, whether raw data about a world with­out a par­tic­u­lar agent in it will be similarly use­ful to this agent as data ob­tained from its own (or that of suffi­ciently similar agents) in­ter­ac­tion with the world. Depend­ing on the ac­tual im­ple­men­ta­tion of a pos­si­ble su­per­in­tel­li­gence, this raw data might be marginally helpful but far from be­ing the most rele­vant bot­tle­neck.

  “Bostrom is sim­ply mak­ing an as­sump­tion that such rapid rates of progress could oc­cur. His in­tel­li­gence spec­trum ar­gu­ment can only ever show that the rel­a­tive dis­tance in in­tel­li­gence space is small; it is silent with re­spect to likely de­vel­op­ment times­pans. ”

  It is not com­pletely silent. I would ex­pect any mean­ingful mea­sure for dis­tance in in­tel­li­gence space to at least some­what cor­re­late with times­pans nec­es­sary to bridge that dis­tance. So while the ar­gu­ment is not a de­ci­sive one re­gard­ing time spans, it also seems far from say­ing noth­ing.

  “As such it seems patently ab­surd to ar­gue that de­vel­op­ments of this mag­ni­tude could be made on the times­pan of days or weeks. We sim­ply see no ex­am­ples of any­thing like this from his­tory, and Bostrom can­not ar­gue that the ex­is­tence of su­per­in­tel­li­gence would make his­tor­i­cal par­allels ir­rele­vant, since we are pre­cisely talk­ing about the de­vel­op­ment of su­per­in­tel­li­gence in the con­text of it not already be­ing in ex­is­tence. ”

  Note that the ar­gu­ment from his­tor­i­cal par­allels is ex­tremely sen­si­tive to refer­ence class. While it seems like there has not been “any­thing like this” in sci­ence or en­g­ineer­ing (al­though progress seems to have been quite dis­con­ti­nous (but not self-re­in­forc­ing) by some met­rics at times) or re­lated to gen­eral in­tel­li­gence (here it would be in­ter­est­ing to ex­plore, whether or not the evolu­tion of hu­man in­tel­li­gence hap­pened a lot faster than an out­side ob­server would have ex­pected from look­ing at the evolu­tion of other an­i­mals, since hours and weeks seem like a some­what An­thro­pocen­tric frame of refer­ence), nar­row AI has gone from sub- to su­per­hu­man level in quite small time spans a lot re­cently (this is once again very sen­si­tive to fram­ing, so take it more as a point for the com­plex­ity of aru­ments from his­tor­i­cal par­allels, than as a di­rect ar­gu­ment for fast take-offs be­ing likely).

  “not con­sis­tent ei­ther with the slow but steady rate of progress in ar­tifi­cial in­tel­li­gence re­search over the past 60 years”

  Could you elab­o­rate? I’m not ex­tremely fa­mil­iar with the his­tory of ar­tifi­cial in­tel­li­gence, but my im­pres­sion was, that progress was quite jumpy at times, in­stead of slow and steady.

• Flodorner 13 Dec 2018 23:50 UTC

  5 points

  on: Cri­tique of Su­per­in­tel­li­gence Part 1

  Thanks for writ­ing this!

  I think you are point­ing out some im­por­tant im­pre­ci­sions, but i think that some of your ar­gu­ments aren’t as con­clu­sive as you seem to pre­sent them to be:

  “Bostrom there­fore faces a dilemma. If in­tel­li­gence is a mix of a wide range of dis­tinct abil­ities as in In­tel­li­gence(1), there is no rea­son to think it can be ‘in­creased’ in the rapidly self-re­in­forc­ing way Bostrom speaks about (in math­e­mat­i­cal terms, there is no sin­gle vari­able which we can differ­en­ti­ate and plug into the differ­en­tial equa­tion, as Bostrom does in his ex­am­ple on pages 75-76). ”

  Those vari­ables could be re­in­forc­ing each other, as one could ar­gue they had done in the evolu­tion of hu­man in­tel­li­gence. (in math­e­mat­i­cal terms, there is a run­away dy­namic similar to the one di­men­sional case for a lin­ear vec­tor-val­ued differ­en­tial equa­tion, as long as all eigen­val­ues are pos­i­tive).

  “This should be­come clear if one con­sid­ers that ‘es­sen­tially all hu­man cog­ni­tive abil­ities’ in­cludes such ac­tivi­ties as pon­der­ing moral dilem­mas, re­flect­ing on the mean­ing of life, analysing and pro­duc­ing so­phis­ti­cated liter­a­ture, for­mu­lat­ing ar­gu­ments about what con­sti­tutes a ‘good life’, in­ter­pret­ing and writ­ing po­etry, form­ing so­cial con­nec­tions with oth­ers, and crit­i­cally in­tro­spect­ing upon one’s own goals and de­sires. To me it seems ex­traor­di­nar­ily un­likely that any agent ca­pa­ble of perform­ing all these tasks with a high de­gree of profi­ciency would si­mul­ta­neously stand firm in its con­vic­tion that the only goal it had rea­sons to pur­sue was till­ing the uni­verse with pa­per­clips. To me it seems ex­traor­di­nar­ily un­likely that any agent ca­pa­ble of perform­ing all these tasks with a high de­gree of profi­ciency would si­mul­ta­neously stand firm in its con­vic­tion that the only goal it had rea­sons to pur­sue was till­ing the uni­verse with pa­per­clips.”

  Why does it seem un­likely? Also, do you mean un­likely as in “agents emerg­ing in a world similar to ours is now­prob­a­bly won’t have this prop­erty” or as in “given that some­one figured out how to con­struct a great va­ri­ety of su­per­in­tel­li­gent agents, she would still have trou­ble con­struct­ing an agent with this prop­erty?”

• Flodorner 23 Nov 2018 18:59 UTC

  1 point

  in reply to: Max_Daniel's comment on: When should EAs al­lo­cate fund­ing ran­domly? An in­con­clu­sive liter­a­ture re­view.

  Yes, ex­actly. When first read­ing your sum­mary i in­ter­preted it as the “for all” claim.

• Flodorner 22 Nov 2018 16:27 UTC

  2 points

  on: When should EAs al­lo­cate fund­ing ran­domly? An in­con­clu­sive liter­a­ture re­view.

  Very in­ter­est­ing!

  In the your liter­a­ture re­view you sum­ma­rize the Smith and Win­kler (2006) pa­per as “Prove that non­ran­dom, non-Bayesian de­ci­sion strate­gies sys­tem­at­i­cally over­es­ti­mate the value of the se­lected op­tion.”

  On first sight, this claim seems like it might be stronger than the claim i have taken away from the pa­per (which is similar to what you write later in the text): if your de­ci­sion strat­egy is to just choose the op­tion you (naively) ex­pect to be best, you will sys­tem­at­i­cally over­es­ti­mate the value of the se­lected op­tion.

  If you think the first claim is im­plied by the sec­ond (or some­thing in the pa­per i missed) in some sense, i’d love to learn about your ar­gu­ments!

  “In fact, I be­lieve that choos­ing the win­ning op­tion does max­i­mize ex­pected value if all mea­sure­ments are un­bi­ased and their re­li­a­bil­ity doesn’t vary too much.”

  I think you are ba­si­cally right, but the amount of available op­tions also plays a role here. If you con­sider a lot of non-op­ti­mal op­tions, for which your mea­sure­ments are only slightly nois­ier than for the best op­tion, you can still sys­tem­at­i­cally un­der­s­e­lect the best op­tion. (For ex­am­ple, simu­la­tions sug­gest that with 99 N(0,1.1) and 1 N(0.1,1) vari­ables, the last one will only be max­i­mal among the 100 only 0.7% of the time, de­spite hav­ing the high­est ex­pected value).

  In this case, ran­domly tak­ing one op­tion would in fact have a higher ex­pected value. (But it still seems very un­clear, how one would iden­tify similar situ­a­tions in re­al­ity, even if they ex­isted).

  Some com­bi­na­tion of mod­er­ately vary­ing noise and lots of op­tions seems like the most plau­si­ble con­di­tion, un­der which not tak­ing the win­ning op­tion might be bet­ter for some real world de­ci­sions.

• Flodorner 11 Nov 2018 13:11 UTC

  2 points

  on: Tiny Prob­a­bil­ities of Vast Utilities: De­fus­ing the Ini­tial Worry and Steel­man­ning the Problem

  I think that the as­sump­tion of the ex­is­tence of a Fun­nel shaped dis­tri­bu­tion with un­defined ex­pected value of things we care about is quite a bit stronger than as­sum­ing that there are in­finitely many pos­si­ble out­comes.

  But even if we re­strict our­selves to dis­tri­bu­tions with finite ex­pected value, our es­ti­mates can still fluc­tu­ate wildly un­til we have gath­ered huge amounts of ev­i­dence.

  So while i am scep­ti­cal of the as­sump­tion that there ex­ists a se­quence of world states with util­ities tend­ing to in­finity and even more scep­ti­cal of ex­tremely high/​low util­ity world states be­ing reach­able with suffi­cient prob­a­bil­ity for there to be un­defined ex­pected value (the ab­solute value of the util­ity of our ac­tion would have to have in­finite ex­pected value, and i’m scep­ti­cal of be­liev­ing this with­out some­thing at least close to “in­finite ev­i­dence”), i still think your post is quite valuable for start­ing a de­bate on how to deal with low prob­a­bil­ity events, cru­cial con­sid­er­a­tions and our de­ci­sion mak­ing when ex­pected val­ues fluc­tu­ate a lot.

  Also, even if my in­tu­ition about the im­pos­si­bil­ity of in­finite util­ities was true (I’m not ex­actly sure what that would ac­tu­ally mean, though), the prob­lems you men­tioned would still ap­ply to any­one who does not share this in­tu­ition.

• Flodorner 11 Nov 2018 12:12 UTC

  2 points

  in reply to: sbehmer's comment on: Is Ne­glect­ed­ness a Strong Pre­dic­tor of Marginal Im­pact?

  I think the ar­gu­ment is that ad­di­tional in­for­ma­tion show­ing that a cause has high marginal im­pact might di­vert causes away to­wards it from causes with less marginal im­pact. And get­ting this kind of in­for­ma­tion does seem more likely for causes with­out a track record al­low­ing for a some­what ro­bust es­ti­ma­tion of their (marginal) im­pact.

• Flodorner 11 Nov 2018 12:10 UTC

  1 point

  on: Is Ne­glect­ed­ness a Strong Pre­dic­tor of Marginal Im­pact?

  Com­ment moved.

• Flodorner 11 Nov 2018 11:30 UTC

  1 point

  on: Is Ne­glect­ed­ness a Strong Pre­dic­tor of Marginal Im­pact?

  For clar­ifi­ca­tion: (PITi+ui) is the “real” tractabil­ity and im­por­tance?

  The text seems to make more sense that way, but read­ing “ui is the un­known (to you) im­por­tance and tractabil­ity of the cause.“, I in­ter­preted it as ui be­ing the “real” tractabil­ity and im­por­tance in­stead of just a noise term at first.

• Flodorner 11 Nov 2018 10:23 UTC

  3 points

  on: De­bate and Effec­tive Altru­ism: Friends or Foes?

  Re­lat­edly, the im­promptu na­ture of some de­bat­ing for­mats could also help with get­ting com­fortable for­mu­lat­ing an­swers to non­triv­ial ques­tions un­der (time) pres­sure. Apart from be­ing gen­er­ally helpful, this might be es­pe­cially valuable in some types of job in­ter­views.

  I’ve been con­sid­er­ing to in­vest some time into com­pet­i­tive de­bat­ing, mostly in or­der to im­prove that skill, so if some­one has data (even anec­do­tal) on that, pleases share :)

• Flodorner 9 Nov 2018 16:36 UTC

  2 points

  on: Tiny Prob­a­bil­ities of Vast Utilities: A Prob­lem for Long-Ter­mism?

  In­ter­est­ing post!

  I am quite in­ter­ested in your other ar­gu­ments for why EV calcu­la­tions won’t work for pas­cal’s mug­ging and why they might ex­tend to x-risks. I would prob­a­bly have prefered a post already in­clud­ing all the ar­gu­ments for your case.

  About the ar­gu­ment from hy­po­thet­i­cal up­dates: My in­tu­ition is, that if you as­sign a prob­a­bil­ity of a lot more than 0.1^10^10^10 to the mug­ger ac­tu­ally be­ing able to fol­low through this might cre­ate other prob­lems (like prob­a­bil­ities of dis­tinct events adding to some­thing higher than 1 or pri­ors in­con­sis­tent with oc­cams ra­zor). If that in­tu­ition (and your ar­gu­ment) was true (my in­tu­ition might very well be wrong and seems at least slightly in­fluenced by mo­ti­vated rea­son­ing), one would ba­si­cally have to con­clude that bayesian EV rea­son­ing fails as soon as it in­volves com­bi­na­tions of ex­treme util­ities and minis­cule prob­a­bil­ities.

  How­ever, i don’t think the cre­denced for be­ing able to in­fluence x-risks are so low, that up­dat­ing be­comes im­pos­si­ble and there­fore i’m not con­vinced not to use EV to eval­u­ate them by your first ar­gu­ment. I’m quite ea­ger to see the other ar­gu­ments, though.

• Flodorner 1 Nov 2018 20:19 UTC

  2 points

  in reply to: jpaddison's comment on: Re­duc­ing ex­is­ten­tial risks or wild an­i­mal suffer­ing?

  What ex­actly do you mean with util­ity here? The Quasi-nega­tive util­i­tar­ian frame­work seems to cor­re­spond to a shift of ev­ery­one’s per­sonal util­ity, such that the shifted util­ity for each per­son is 0, when­ever this per­son’s live is nei­ther worth liv­ing, nor not worth liv­ing.

  It seems to me, like a rea­son­able no­tion of util­ity would have this prop­erty any­way (but i might just use the word differ­ently than other peo­ple, please tell me, if there is some widely used defi­ni­tion con­tra­dict­ing this!). This re­frames the dis­cus­sion into one about where the zero point of util­ity func­tions should lie, which seems eas­ier to grasp. In par­tic­u­lar, from this point of view Quasi-nega­tive util­i­tar­i­anism still gives rise to some for of the sadis­tic-re­pug­nant con­clus­sion.

  On a broader point, i sus­pect, that the re­pug­nance of repgug­nant con­clus­sions usu­ally stems from con­fu­sion/​dis­agree­ment about what “a life worth liv­ing” means. How­ever, as in your ar­ti­cle, en­ter­tain­ing this con­clus­sion still seems use­ful in or­der to sharpen our in­tu­ition about what lives are ac­tu­ally worth liv­ing.

• Flodorner 10 Sep 2018 8:59 UTC

  7 points

  in reply to: jpaddison's comment on: Ad­di­tional plans for the new EA Forum

  Are any ways of mak­ing con­tent eas­ier to filter (like for ex­am­ple tags) planned?

  I am rather new to the com­mu­nity and there have been mul­ti­ple oc­cas­sions, where i ran­domly stum­bled upon old ar­ti­cles, i haven’t read, con­cerned with top­ics i was in­ter­ested in and had pre­vi­ously made an effort to find ar­ti­cles about. This seems rather in­effi­cient.

• Flodorner 25 Jun 2018 7:25 UTC

  3 points

  in reply to: jpaddison's comment on: In­for­ma­tional haz­ards and the cost-effec­tive­ness of open dis­cus­sion of catas­trophic risks

  “to prove this ar­gu­ment I would have to pre­sent gen­eral in­for­ma­tion which may be re­garded as hav­ing in­for­ma­tional haz­ard”

  Is there any way to as­sess the cred­i­bil­ity of state­ments like this (or whether this is ac­tu­ally an ar­gu­ment worth con­sid­er­ing in a given spe­cific con­text)? It seems like you could use this as a gen­eral pur­pose ar­gu­ment for al­most ev­ery­thing.

• Flodorner 16 Jun 2018 6:48 UTC

  4 points

  in reply to: jpaddison's comment on: Don­ing with the devil

  I am not sure about whether your us­age of economies of scale already cov­ers this, but it seems to make sense to high­light, that what mat­ters is the marginal differ­ence of the money for you and your ad­ver­sary. If do­ing evil is a lot more effi­cient at low scales (Think of dis­tribut­ing highly ad­dic­tive drugs among vurn­er­a­ble pop­u­la­tions vs. Distribut­ing Malaria nets), your ad­ver­sary could be hit­ting diminish­ing re­turns already, while your marginal re­turns in­crease, and the lot­tery might still be not be worth it.

• Flodorner 11 Jun 2018 8:55 UTC

  2 points

  in reply to: Halstead's comment on: An­i­mal Equal­ity showed that ad­vo­cat­ing for diet change works. But is it cost-effec­tive?

  Are you talk­ing about the in­di­vi­d­ual level, or the mean? My es­ti­mate would be, that for the me­dian in­di­vi­d­ual, the effect will have faded out af­ter at most 6 months. How­ever, the mean might be in­fluenced by the tails quite strongly.

  Think­ing about it for a bit longer, a mean effect of 12 years does seem quite im­plau­si­ble, though. In the limit­ing case, where only the tails mat­ter, this would be equiv­a­lent to con­vinc­ing around 25% of the ini­tially in­fluenced stu­dents to stop eat­ing pork for the rest of their lives.

  The up­per bound for my 90% con­fi­dence in­ter­val for the mean seems to be around 3 years, while the lower bound is at 3 months. The prob­a­bil­ity mass within the in­ter­val is mostly cen­tered to the left.

• Flodorner 11 Jun 2018 7:49 UTC

  1 point

  in reply to: Halstead's comment on: An­i­mal Equal­ity showed that ad­vo­cat­ing for diet change works. But is it cost-effec­tive?

  The claim does not seem to be ex­actly, that there is a 10% chance of an an­i­mal ad­vo­cacy video af­fect­ing con­sump­tion de­ci­sions af­ter 12 years for a given in­di­vi­d­ual.

  I’d in­ter­pret it as: there is a 5% chance of the mean du­ra­tion of re­duc­tion, con­di­tioned on the par­ti­ci­pant re­port­ing to change their be­havi­our based on the video be­ing higher than 12 years.

  This could for ex­am­ple also be achieved by hav­ing a very long term im­pact on very few par­ti­ci­pants. This in­ter­pre­ta­tion seems a lot more plau­si­ble, al­though i am not cer­tain at all, wheter that claim cor­rect. Long term fol­low up data would cer­tainly be very helpful.

• Flodorner 29 May 2018 20:51 UTC

  1 point

  in reply to: rohinmshah's comment on: The coun­ter­fac­tual im­pact of agents act­ing in concert

  At this point, i think that to an­a­lyze the $1bn case cor­rectly, you’d have to sub­stract ev­ery­one’s op­por­tu­nity cost in the calcu­la­tion of the shap­ley value (if you want to use it here). This way, the ex­am­ple should yield what we ex­pect.

  I might do a more gen­eral writeup about shap­ley val­ues, their ad­van­tages, dis­ad­van­tages and when it makes sense to use them, if i find the time to read a bit more about the topic first.

• Flodorner 29 May 2018 12:52 UTC

  2 points

  in reply to: Emanuele_Ascani's comment on: Ex­pected cost per life saved of the TAME trial

  I think, it might be best to just re­port con­fi­dence in­ter­vals for your fi­nal es­ti­mates (guessti­mate should give you those). Then ev­ery­one can com­bine your es­ti­mates with their own pri­ors on gen­eral in­ter­ven­tion’s effec­tive­ness and thereby po­ten­tially cor­rect for the high lev­els of un­cer­tainty (at least in a crude way by es­ti­mat­ing the var­i­ance from the con­fi­dence in­ter­vals).

  The var­i­ance of X can be defined as E[X^2]-E[X]^2, which should not be hard to im­ple­ment in Guessti­mate. How­ever, i am not sure, whether or not hav­ing the var­i­ance yields to more ac­cu­rate up­dat­ing, than hav­ing a con­fi­dence in­ter­val. Op­ti­mally you’d have the full dis­tri­bu­tion, but i am not sure, whether any­one will ac­tu­ally do the maths to up­date from there. (But they could get it roughly from your guessti­mate model).

  I might com­ment more on some de­tails and the moral as­sump­tions, if i find the time for it soon.