Karma: 197

# Poor meat eater problem

10 Jul 2020 8:13 UTC
18 points
• Thanks for the an­swer.

Will MacAskill men­tioned in this com­ment that he’d ‘ex­pect that, say, a panel of su­perfore­cast­ers, af­ter be­ing ex­posed to all the ar­gu­ments, would be closer to my view than to the me­dian FHI view.’

You’re a good fore­caster right? Does it seem right to you that a panel of good fore­cast­ers would come to some­thing like Will’s view, rather than the me­dian FHI view?

• Thanks for the an­swer.

Will MacAskill men­tioned in this com­ment that he’d ‘ex­pect that, say, a panel of su­perfore­cast­ers, af­ter be­ing ex­posed to all the ar­gu­ments, would be closer to my view than to the me­dian FHI view.’

You’re a good fore­caster right? Does it seem right to you that a panel of good fore­cast­ers would come to some­thing like Will’s view, rather than the me­dian FHI view?

# [Question] Are there su­perfore­casts for ex­is­ten­tial risk?

7 Jul 2020 7:39 UTC
24 points

# AI Gover­nance Read­ing Group Guide

25 Jun 2020 10:16 UTC
16 points
• Yeah, that seems right to me.

On dou­bling con­sump­tion though, if you can sug­gest a policy that in­creases growth con­sis­tently, even­tu­ally you might cause con­sump­tion to be dou­bled (at some later time con­sump­tion un­der the faster growth will be twice as much as it would have been with the slower growth). Do you mean you don’t think you could sug­gest a policy change that would in­crease the growth rate by much?

# [Up­dated 6/​21] ‘Ex­is­ten­tial Risk and Growth’ Summary

21 Jun 2020 9:22 UTC
49 points
• Great to hear this has been use­ful!

I think if is around 1 then yes, spread­ing longter­mism prob­a­bly looks bet­ter than ac­cel­er­at­ing growth. Though I don’t know how ex­pen­sive it is to dou­ble some­one’s con­sump­tion in the long-run.

Dou­bling some­one’s con­sump­tion by just giv­ing them ex­tra money might cost $30,000 for 50 years=~$0.5mil­lion. It seems right to me that there are ways to re­duce the dis­count rate that are much cheaper than half a mil­lion dol­lars for 13 ba­sis points. Eg. some com­mu­nity build­ing prob­a­bly takes a per­son’s dis­count rate from around 2% to around 0% for less than half a mil­lion dol­lars.

I don’t know how much cheaper it might be to dou­ble some­one’s con­sump­tion by in­creas­ing growth but I sus­pect that spread­ing longter­mism still looks bet­ter for this value of .

How con­fi­dent are you that is around 1? I haven’t looked into it and don’t know how much con­sen­sus there is.

• What do you think ab­sorbers might be in cases of com­plex clue­less­ness? I see that de­lay­ing some­one on the street might just cause them to spend 30 sec­onds less pro­cras­ti­nat­ing, but how might this work for dis­tribut­ing bed­nets, or in­creas­ing eco­nomic growth?

Maybe there’s an line of ar­gu­ment around noth­ing be­ing coun­ter­fac­tual in the long-term - be­cause ev­ery time you solve a prob­lem some­one else was go­ing to solve it even­tu­ally. Eg. if you didn’t in­crease growth in some re­gion, some­one else would have 50 years later. And now you did it they won’t. But this just sounds like a weirdly sta­ble sys­tem and I guess this isn’t what you have in mind

• Thanks for writ­ing this. I hadn’t though it about this ex­plic­itly and think it’s use­ful. The bite-sized for­mat is great. A se­ries of posts would be great too.

• So you think the haz­ard rate might go from around 20% to around 1%? That’s still far from zero, and with enough cen­turies with 1% risk we’d ex­pect to go ex­tinct.

I don’t have any spe­cific sto­ries tbh, I haven’t thought about it (and maybe it’s just pretty im­plau­si­ble idk).

• Not the au­thor but I think I un­der­stand the model so can offer my thoughts:

1. Why do the time axes in many of the graphs span hun­dreds of years? In dis­cus­sions about AI x-risk, I mostly see some­thing like 20-100 years as the rele­vant timescale in which to act (i.e. by the end of that pe­riod, we will ei­ther go ex­tinct or else build an al­igned AGI and reach a tech­nolog­i­cal sin­gu­lar­ity). Look­ing at Figure 7, if we only look ahead 100 years, it seems like the risk of ex­tinc­tion ac­tu­ally goes up in the ac­cel­er­ated growth sce­nario.

The model is look­ing at gen­eral dy­nam­ics of risk from the pro­duc­tion of new goods, and isn’t try­ing to look at AI in any kind of gran­u­lar way. The timescales on which we see the in­verted U-shape de­pend on what val­ues you pick for differ­ent pa­ram­e­ters, so there are differ­ent val­ues for which the time axes would span decades in­stead of cen­turies. I guess that pick­ing a differ­ent growth rate would be one clear way to squash ev­ery­thing into a shorter time. (Maybe this is pretty con­sis­tent with short/​medium AI timelines, as they prob­a­bly cor­re­late strongly with re­ally fast growth).

I think your point about AI mess­ing up the re­sults is a good one—the model says that a boom in growth has a net effect to re­duce x-risk be­cause, while risk is in­creased in the short-term, the long-term effects can­cel that out. But if AI comes in the next 50-100 years, then the long-term benefits never ma­te­ri­al­ise.

2. What do you think of Wei Dai’s ar­gu­ment that safe AGI is harder to build than un­safe AGI and we are cur­rently putting less effort into the former, so slower growth gives us more time to do some­thing about AI x-risk (i.e. slower growth is bet­ter)?

Sure, maybe there’s a lock-in event com­ing in the next 20-200 years which we can either

• De­lay (by de­creas­ing growth) so that we have more time to de­velop safety fea­tures, or

• Make more safety-fo­cussed (by in­creas­ing growth) so it is more likely to lock in a good state

I’d think that what mat­ters is re­sources (say co­or­di­na­tion-ad­justed-IQ-per­son-hours or what­ever) spent on safety rather than time that could available to be spent on safety if we wanted. So if we’re poor and reck­less, then more time isn’t nec­es­sar­ily good. And this time spent be­ing less rich also might make other x-risks more likely. But that’s a very high level ab­strac­tion, and doesn’t re­ally en­gage with the spe­cific claim too closely so keen to hear what you think.

3. What do you think of Eliezer Yud­kowsky’s ar­gu­ment that work for build­ing an un­safe AGI par­allelizes bet­ter than work for build­ing a safe AGI, and that un­safe AGI benefits more in ex­pec­ta­tion from hav­ing more com­put­ing power than safe AGI, both of which im­ply that slower growth is bet­ter from an AI x-risk view­point?

The model doesn’t say any­thing about this kind of gran­u­lar con­sid­er­a­tion (and I don’t have strong thoughts of my own).

4. What do you think of Nick Bostrom’s urn anal­ogy for tech­nolog­i­cal de­vel­op­ments? It seems like in the anal­ogy, faster growth just means pul­ling out the balls at a faster rate with­out af­fect­ing the prob­a­bil­ity of pul­ling out a black ball. In other words, we hit the same amount of risk but ev­ery­thing just hap­pens sooner (i.e. growth is neu­tral).

In the model, risk de­pends on pro­duc­tion of con­sump­tion goods, rather than the level of con­sump­tion tech­nol­ogy. The in­tu­ition be­hind this is that tech­nolog­i­cal ideas them­selves aren’t dan­ger­ous, it’s all the stuff peo­ple do with the ideas that’s dan­ger­ous. Eg. syn­thetic biol­ogy un­der­stand­ing isn’t it­self dan­ger­ous, but a bunch of syn­thetic biol­ogy labs pro­duc­ing loads of ex­otic or­ganisms could be dan­ger­ous.

But I think it might make sense to in­stead model risk as par­tially de­pend­ing on tech­nol­ogy (as well as pro­duc­tion). Eg. once we know how to make some level of AI, the dam­age might be done, and it doesn’t mat­ter whether there are 100 of them or just one.

And the rea­son growth isn’t neu­tral in the model is that there are also safety tech­nolo­gies (which might be analo­gous to mak­ing the world more ro­bust to black balls). Growth means peo­ple value life more so they spend more on safety.

5. Look­ing at Figure 7, my “story” for why faster growth low­ers the prob­a­bil­ity of ex­tinc­tion is this: The richer peo­ple are, the less they value marginal con­sump­tion, so the more they value safety (rel­a­tive to con­sump­tion). Faster growth gets us sooner to the point where peo­ple are rich and value safety. So faster growth effec­tively gives so­ciety less time in which to mess things up (how­ever, I’m con­fused about why this hap­pens; see the next point). Does this sound right? If not, I’m won­der­ing if you could give a similar in­tu­itive story.

Sounds right to me.

6. I am con­fused why the height of the haz­ard rate in Figure 7 does not in­crease in the ac­cel­er­ated growth case. I think equa­tion (7) for δ_t might be the cause of this, but I’m not sure. My own in­tu­ition says ac­cel­er­ated growth not only con­denses along the time axis, but also stretches along the ver­ti­cal axis (so that the area un­der the curve is mostly un­af­fected).

The haz­ard rate does in­crease for the pe­riod that there is more pro­duc­tion of con­sump­tion goods, but this means that peo­ple are now richer, ear­lier than they would have been so they value safety more than they would oth­er­wise.

As an ex­treme case, sup­pose growth halted for 1000 years. It seems like in your model, the graph for haz­ard rate would be con­stant at some fixed level, ac­cu­mu­lat­ing ex­tinc­tion prob­a­bil­ity dur­ing that time. But my in­tu­ition says the haz­ard rate would first drop near zero and then stay con­stant, be­cause there are no new dan­ger­ous tech­nolo­gies be­ing in­vented. At the op­po­site ex­treme, sup­pose we sud­denly get a huge boost in growth and effec­tively reach “the end of growth” (near pe­riod 1800 in Figure 7) in an in­stant. Your model seems to say that the graph would com­press so much that we al­most cer­tainly never go ex­tinct, but my in­tu­ition says we do ex­pe­rience a lot of risk for ex­tinc­tion. Is my in­ter­pre­ta­tion of your model cor­rect, and if so, could you ex­plain why the height of the haz­ard rate graph does not in­crease?

Hmm yeah, this seems like maybe the risk de­pends in part on the rate of change of con­sump­tion tech­nolo­gies—be­cause if no new techs are be­ing dis­cov­ered, it seems like we might be safe from an­thro­pogenic x-risk.

But, even if you be­lieve that the haz­ard rate would de­cay in this situ­a­tion, maybe what’s do­ing the work is that you’re imag­in­ing that we’re still do­ing a lot of safety re­search, and think­ing about how to miti­gate risks. So that the con­sump­tion sec­tor is not grow­ing, but the safety sec­tor con­tinues to grow. In the ex­ist­ing model, the haz­ard rate could de­cay to zero in this case.

I guess I’m also not sure if I share the in­tu­ition that the haz­ard rate would de­cay to zero. Sure, we don’t have the tech­nol­ogy right now to pro­duce AGI that would con­sti­tute an ex­is­ten­tial risk but what about eg. cli­mate change, nu­clear war, biorisk, nar­row AI sys­tems be­ing used in re­ally bad ways? It seems plau­si­ble to me that if we kept our cur­rent level of tech­nol­ogy and pro­duc­tion then we’d have a non-triv­ial chance each year of kil­ling our­selves off.

What’s do­ing the work for you? Do you think the prob­a­bil­ity of an­thro­pogenic x-risk with our cur­rent tech is close to zero? Or do you think that it’s not but that if growth stopped we’d keep work­ing on safety (say de­vel­op­ing clean en­ergy, im­prov­ing re­la­tion­ships be­tween US and China etc.) so that we’d even­tu­ally be safe?

• Ah yeah that makes sense. I think they seemed dis­tinct to me be­cause one seems like ‘buy some QALYS now be­fore the sin­gu­lar­ity’ and the other seems like ‘make the sin­gu­lar­ity hap­pen sooner’ (ob­vi­ously these are big car­i­ca­tures). And the sec­ond one seems like it has a lot more value than the first if you can do it (of course I’m not say­ing you can). But yeah they are the same in that they are adding value be­fore a set time. I can imag­ine that post be­ing re­ally use­ful to send to peo­ple I talk to—look­ing for­ward to read­ing it.

• Pleased you liked it and thanks for the ques­tion. Here are my quick thoughts:

That kind of flour­ish­ing-ed­u­ca­tion sounds a bit like Bostrom’s eval­u­a­tion func­tion de­scribed here: http://​​www.staffor­ini.com/​​blog/​​bostrom/​​

Or steer­ing ca­pac­ity de­scribed here: https://​​fo­rum.effec­tivealtru­ism.org/​​posts/​​X2n6pt3uzZtxGT9Lm/​​do­ing-good-while-clueless

Un­for­tu­nately he doesn’t talk about how to con­struct the eval­u­a­tion func­tion, and steer­ing ca­pac­ity is only mo­ti­vated by an anal­ogy. I agree with you/​Bostrom/​Milan that there are prob­a­bly some things that look more ro­bustly good than oth­ers. It’s a bit un­clear how to get these but some­thing like :‘Build mod­els of how the world works by look­ing to the past and then up­dat­ing based on in­side view ar­gu­ments of the pre­sent/​fu­ture. Then take ac­tions that look good on most of your mod­els’ seems vaguely right to me. Some things that look good to me are: in­vest­ing, build­ing the EA com­mu­nity, re­duc­ing the chance of catas­trophic risks, spread­ing good val­ues, get­ting bet­ter at fore­cast­ing, build­ing mod­els of how the world works

Ad­just­ing our val­ues based on them be­ing difficult to achieve seems a bit back­ward to me, but I’m mo­ti­vated by sub­jec­tive prefer­ences, and maybe it would make more sense if you were tak­ing a more eth­i­cal/​re­al­ist ap­proach (eg. be­cause you ex­pect the cor­rect moral the­ory to ac­tu­ally be fea­si­ble to im­ple­ment).

• Fol­low­ing that pa­per, I think growth might in­crease x-risk in the near-term (say ~100-200 years), and might de­crease x-risk in the long-term (if the growth doesn’t come at the cost of later growth). I meant (1), but was think­ing about the effect of x-risk in the near-term.