I think the sci­ence value of ex­o­planet mis­sions is re­ally un­der­rated. We should build the TPF. We might be able to liter­ally find aliens or at least wor­lds where alien civ­i­liza­tions had gone ex­tinct in the past. And learn­ing about what had kil­led them might help us avoid the same fate, at least for a while. Or we might learn some­thing that we can­not fore­see.

1.) Do we need to worry about how long it takes to de­ploy cap­i­tal? If a pa­tient philan­thropist en­coun­ters a “hinge”, will there be enough pre-ex­ist­ing in­sti­tu­tions to be able to take ad­van­tage of it?

2.) How much should we value the in­for­ma­tion gained from act­ing now and “learn­ing by do­ing” about what works?

3.) I imag­ine there also should be value for a pa­tient philan­thropist to still donate con­sid­er­ably to re­search now, so we can bet­ter iden­tify “hinges” and know how and where to take ad­van­tage of them.

Also, I ex­pect to see small en­g­ineered pan­demics, but only af­ter effec­tive ge­netic en­g­ineer­ing is wide­spread. So the fact that we haven’t seen any so far is not much ev­i­dence.

Yes, that was broadly the re­sponse I had in mind as well. Same goes for most of the “un­fore­seen”/​”other” an­thro­pogenic risks; those cat­e­gories are in the chap­ter on “Fu­ture risks”, and are mostly things Ord ap­pears to think ei­ther will or may get riskier as cer­tain tech­nolo­gies are de­vel­oped/​ad­vanced.

Sleepy re­ply to To­bias’ “Ord’s es­ti­mates seem too high to me”: An im­por­tant idea in the book is that “the per-cen­tury ex­tinc­tion risks from “nat­u­ral” causes must be very low, based in part on our long his­tory of sur­viv­ing such risks” (as I phrase it in this post). The flip­side of that is roughly the ar­gu­ment that we haven’t got strong ev­i­dence of our abil­ity to sur­vive (un­col­lapsed and sans dystopia) a long pe­riod with var­i­ous tech­nolo­gies that will be de­vel­oped later, but haven’t been yet.

Of course, that doesn’t seem suffi­cient by it­self as a rea­son for a high level of con­cern, as some ver­sion of that could’ve been said at ev­ery point in his­tory when “things were chang­ing”. But if you cou­ple that gen­eral ar­gu­ment with spe­cific rea­sons to be­lieve up­com­ing tech­nolo­gies could be no­tably risky, you could per­haps rea­son­ably ar­rive at Ord’s es­ti­mates. (And there are ob­vi­ously a lot of spe­cific de­tails and ar­gu­ments and caveats that I’m omit­ting here.)

Did this end up hap­pen­ing?

What is the sig­nifi­cance of the peo­ple on the ISS? Are you sug­gest­ing that six peo­ple could re­pop­u­late the hu­man species? And what sort of dis­aster takes less time than a flight, and only kills peo­ple on the ground?

Also, I ex­pect to see small en­g­ineered pan­demics, but only af­ter effec­tive ge­netic en­g­ineer­ing is wide­spread. So the fact that we haven’t seen any so far is not much ev­i­dence.

I haven’t looked at this in much de­tail, but Ord’s es­ti­mates seem too high to me. It seems re­ally hard for hu­man­ity to go ex­tinct, con­sid­er­ing that there are peo­ple in re­mote villages, peo­ple in sub­marines, peo­ple in mid-flight at the time a dis­aster strikes, and even peo­ple on the In­ter­na­tional Space Sta­tion. (And yes, there are women on the ISS, I looked that up.) I just don’t see how e.g. a pan­demic would plau­si­bly kill all those peo­ple.

Also, if en­g­ineered pan­demics, or “un­fore­seen” and “other” an­thro­pogenic risks have a chance of 3% each of caus­ing ex­tinc­tion, wouldn’t you ex­pect to see smaller ver­sions of these risks (that kill, say, 10% of peo­ple, but don’t re­sult in ex­tinc­tion) much more fre­quently? But we don’t ob­serve that.

(I haven’t read Ord’s book so I don’t know if he ad­dresses these points.)

I find this very hard to un­der­stand.

This is use­ful feed­back. I might need to work on the word­ing.

Without that I think it is hard to say if 17% is high or not right?

I don’t think I agree with that—I think the im­por­tant con­sid­er­a­tion is the num­ber of iden­ti­fied ad­ver­tised roles of a par­tic­u­lar type rel­a­tive to the num­ber of iden­ti­fied cur­rently filled roles of the same type. Not the num­ber of ad­ver­tised roles of type A rel­a­tive to ad­ver­tised roles of type B. But FWIW the full re­port is now pub­lished.

this seems like weak ev­i­dence for bot­tle­neck claims

I agree its weak ev­i­dence; I think it’s the weak­est of the 5 bul­let points above. I find weak ev­i­dence use­ful.

Thanks for the in­put! If the above bul­let points were ev­i­dence of fund­ing con­straints, then this “more nega­tive read­ing” would be a plau­si­ble al­ter­na­tive ex­pla­na­tion. But I’m not fol­low­ing how the above bul­let points could be read in this way. Apolo­gies if I’m miss­ing some­thing.

Are you think­ing this ap­plies to all 5 of the above bul­let points? Or spe­cific bul­let points within that group?

We’re do­ing some mes­sage test­ing on this and can get back to you.

Great work, thanks for writ­ing this up!

I’m won­der­ing how this might af­fect the pub­lic de­bate on fac­tory farm­ing. An­i­mal ad­vo­cates some­times ar­gue that fac­tory farms con­tribute to an­tibiotic re­sis­tance, and this point may carry much more force in the fu­ture. So per­haps one key con­clu­sion is that ad­vo­cates should em­pha­sise this an­gle more in the fu­ture. (That said, AFAIK the coro­n­avirus doesn’t have any­thing to do with farmed an­i­mals, and my im­pres­sion from a quick Google search is that the is­sue of an­tibiotic re­sis­tance is man­age­able with the right reg­u­la­tions.)

One minor quib­ble with this post’s lan­guage, rather than any of its ac­tual claims: The ti­tle in­cludes the phrase “safety by de­fault”, and the terms “op­ti­mism” and “op­ti­mist” are re­peat­edly ap­plied to these re­searchers or their views. The ti­tle is rea­son­able in a sense, as these in­ter­views were par­tially/​mostly about whether AI would be “safe by de­fault”, or why we might be­lieve that it would be, or why these re­searchers be­lieve that that’s likely. And the use of “op­ti­mism”/​”op­ti­mist” are rea­son­able in a sense, as these re­searchers were dis­cussing why they’re rel­a­tively op­ti­mistic, com­pared to some­thing like e.g. the “typ­i­cal MIRI view”.

But it seems po­ten­tially mis­lead­ing to use those phrases here with­out em­pha­sis­ing (or at least men­tion­ing) that at least some of these re­searchers think there’s a greater than 1% chance of ex­tinc­tion or other ex­is­ten­tial catas­tro­phe as a re­sult of AI. E.g., the state­ment “Ro­hin re­ported an un­usu­ally large (90%) chance that AI sys­tems will be safe with­out ad­di­tional in­ter­ven­tion” im­plies a 10% cre­dence that that won’t be the case. Rele­vant quote from The Precipice:

In 1939, En­rico Fermi told Szilard the chain re­ac­tion was but a ‘re­mote pos­si­bil­ity’ [...]

Fermi was asked to clar­ify the ‘re­mote pos­si­bil­ity’ and ven­tured ‘ten per­cent’. Isi­dor Rabi, who was also pre­sent, replied, ‘Ten per­cent is not a re­mote pos­si­bil­ity if it means that we may die of it. If I have pneu­mo­nia and the doc­tor tells me that there is a re­mote pos­si­bil­ity that I might die, and it’s ten per­cent, I get ex­cited about it’

And in this case, the stakes are far greater (mean­ing no offence to Isi­dor Rabi).

My guess would be that a de­cent por­tion of peo­ple who (a) were more used to some­thing like the FHI/​80k/​Oxford views, and less used to the MIRI/​Bay Area views, and (b) read this with­out hav­ing read the in­ter­views in great de­tail, might think that these re­searchers be­lieve some­thing like “The chance things go wrong is too small to be worth any­one else wor­ry­ing about.” Which doesn’t seem ac­cu­rate, at least for Ro­hin and I be­lieve for Paul and Adam. (I’m less sure about Robin, and I don’t re­call if any­one but Ro­hin ex­plic­itly gave rele­vant prob­a­bil­ity es­ti­mates.)

To be clear: I don’t think you’re in­tend­ing to con­vey that mes­sage. And I definitely wouldn’t want to try shut down any state­ments about AI that don’t sound like “this is a huge deal, ev­ery­one get in here now!” I’m just a bit con­cerned about posts ac­ci­den­tally con­vey­ing an overly op­ti­mistic/​san­guine mes­sage when that wasn’t ac­tu­ally their in­tent, and when it wasn’t sup­ported by the ar­gu­ments/​ev­i­dence pro­vided.

(Some­thing in­form­ing this com­ment is my past ex­pe­rience read­ing a bunch of cog­ni­tive sci­ence work on how mis­in­for­ma­tion spreads and can be sticky. Some dis­cus­sion here, and a par­tic­u­larly rele­vant pa­per here.)

Thanks for this post!

There are lots of calls for in­di­vi­d­u­als with views around AI risk to en­gage with each other and un­der­stand the rea­son­ing be­hind fun­da­men­tal dis­agree­ments.

I strongly share this view, and have there­fore quite ap­pre­ci­ated this pro­ject by AI Im­pacts. A lot of re­sources are go­ing into this field, which I’m broadly very sup­port­ive of, but it does seem worth gain­ing a clearer un­der­stand­ing of pre­cisely why, and pre­cisely which ap­proaches should get which por­tions of those re­sources.

One other post that I per­son­ally found re­ally use­ful for un­der­stand­ing the var­i­ous views, the un­der­ly­ing as­sump­tions, and their in­ter­con­nec­tions was Clar­ify­ing some key hy­pothe­ses in AI al­ign­ment (coau­thored by Ro­hin). I’ve also col­lected here ~30 works I found that strongly re­late to this goal (I plan to up­date that col­lec­tion over time, and have now added this post to it as well).

And I’m cur­rently work­ing on a post with a very similar ob­jec­tive, but for longter­mist/​x-risk strate­gies more broadly. Hope­fully that’ll be out soon.

The link has an ex­tra ‘.’ - https://​​www.causal.app/​​

Looks neat, good luck!

In­ter­est­ing view. It seems to me like it makes sense, but I also feel like it’d be valuable for it to be fleshed out and cri­tiqued fur­ther to see how solid it is. (Per­haps this has already been done some­where—I do feel like I’ve heard vaguely similar ar­gu­ments here and there.)

Also, ar­riv­ing at this thread 5 months late, I no­tice Toby Ord makes a similar ar­gu­ment in The Precipice. He writes about:

a sub­tle form of cor­re­la­tion—not be­tween two risks, but be­tween risks and the value of the fu­ture. There might be risks that are much more likely to oc­cur in wor­lds with high po­ten­tial. For ex­am­ple, if it is pos­si­ble to cre­ate ar­tifi­cial in­tel­li­gence that far sur­passes hu­man­ity in ev­ery do­main, this would in­crease the risk from mis­al­igned AGI, but would also in­crease the value we could achieve us­ing AGI that was al­igned with hu­man val­ues. By ig­nor­ing this cor­re­la­tion, the to­tal risk ap­proach un­der­weights the value of work on this risk.

This can be use­fully un­der­stood in terms of there be­ing a com­mon cause for the risk and the benefit, pro­duc­ing the cor­re­la­tion. A high ceiling on tech­nolog­i­cal ca­pa­bil­ity might be an­other com­mon cause be­tween a va­ri­ety of risks and ex­tremely pos­i­tive fu­tures. I will set this pos­si­bil­ity aside in the rest of the book, but it is an im­por­tant is­sue for fu­ture work to ex­plore.

I think the sen­tence does make sense, though it is cer­tainly badly writ­ten!

Minor: The in-text links to end­notes link to the page for mak­ing a new fo­rum post, which pre­sum­ably is not in­tended.

Minor: The fol­low­ing sen­tence right be­fore sec­tion 4 starts seems jum­bled?

This is true for when the mag­ni­tude of warm­ing was com­pa­rable to what we are in for in the next 200 years, and when, on a re­gional ba­sis, the rate of warm­ing was com­pa­rable to what we are in for in the next 200 years.

Ex­cel­lent post. I highly value such sum­maries of known and rele­vant fac­tual in­for­ma­tion, and views on les­sons we can learn. Thank you for putting this to­gether.

I’m also cu­ri­ous what prompted you to look into this topic?

This seems like an im­por­tant con­sid­er­a­tion.

My think­ing is that down­votes should be used spar­ingly, and only with feed­back (or some cited rea­son). If some­one with a high-sta­tus/​”au­thor­i­ta­tive” view­point is be­ing down­voted, even they de­serve to know why. If some­one is brave enough to re­ply with crit­i­cism, oth­ers could up­vote that crit­i­cism anony­mously (up­votes could re­main anony­mous).

Maybe the vot­ing can re­main anony­mous, but each down­vote has to be tied to an ex­pla­na­tion? E.g. each down­vote must refer­ence a com­ment in the post as an ex­pla­na­tion. Once one per­son prop­erly crit­i­cizes with a com­ment, the oth­ers can re­fer to that com­ment anony­mously as the rea­son for their down­votes. This gives who­ever’s been down­voted a chance to know why and a chance to defend them­selves.

Awe­some! :)