Thoughts on short timelines

[Cross-posted from my web­site.]

Some ra­tio­nal­ists and effec­tive al­tru­ists have ar­gued (1, 2, 3) that there is a non-neg­ligible chance that ar­tifi­cial in­tel­li­gence will at­tain hu­man or su­per-hu­man lev­els of gen­eral in­tel­li­gence very soon.

In this post, I’d like to out­line why I’m not con­vinced that this sce­nario has non-neg­ligible prob­a­bil­ity. To clar­ify, I’m ar­gu­ing against the hy­poth­e­sis that “ar­tifi­cial gen­eral in­tel­li­gence (AGI) is 10% likely to be built in the next 10 years”, where AGI is defined as the abil­ity to suc­cess­fully perform any in­tel­lec­tual task that a hu­man is ca­pa­ble of. (My favoured defi­ni­tion of “AGI” is that au­tonomous in­tel­li­gent ma­chines con­tribute at least 50% to the global econ­omy, as out­lined here, but I don’t think the pre­cise defi­ni­tion mat­ters much for pur­poses of this post.)

The sim­plest coun­ter­ar­gu­ment is to look at the rate of progress we’re see­ing so far and ex­trap­o­late from that. Have there been any ground-break­ing re­sults over the last years? I’m not talk­ing about “nor­mal” re­sults of ma­chine learn­ing pa­pers; I’m talk­ing about mile­stones that con­sti­tute se­ri­ous progress to­wards gen­eral in­tel­li­gence. We are surely see­ing progress in the former sense – I don’t mean to be­lit­tle the efforts of ma­chine learn­ing re­searchers. (An ex­am­ple of what that I’d con­sider “ground-break­ing” is ad­vanced trans­fer be­tween differ­ent do­mains, e.g. play­ing many board or video games well af­ter train­ing on a sin­gle game.)

Some peo­ple con­sid­ered AlphaGo (and later AlphaZero) ground-break­ing in this sense. But this (the match against Lee Sedol) was in March 2016, so it’s already more than 2 years ago at the time of this writ­ing (late 2018) – and it seems that there haven’t been com­pa­rable break­throughs since then. (In my opinion, AlphaGo wasn’t that ex­cep­tional any­way – but that’s a topic for an­other post.)

Con­di­tional on short timelines, I’d ex­pect to ob­serve ground-break­ing progress all the time. So that seems to be ev­i­dence that this sce­nario is not ma­te­ri­al­iz­ing. In other words, it seems clear to me that the cur­rent rate of progress is not suffi­cient for AGI in 10 years. (See also Robin Han­son’s AI progress es­ti­mate.)

That said, we should dis­t­in­guish be­tween a) the be­lief that cur­rent rate of progress will lead to AGI within 10 years, and b) the be­lief that there will be sig­nifi­cant ac­cel­er­a­tion at some point, which will en­able AGI within 10 years. One could re­ject a) and still ex­pect a sce­nario where AGI ar­rives within 10 years, but for some rea­son we won’t see im­pres­sive re­sults un­til very near ‘the end’. In that case the lack of ground-break­ing progress we see now isn’t (strong) ev­i­dence.

But why ex­pect that? There’s an ar­gu­ment that progress will be­come dis­con­tin­u­ous as soon as re­cur­sive self-im­prove­ment be­comes pos­si­ble. But we are talk­ing about progress from the sta­tus quo to AGI, so that doesn’t ap­ply: it seems im­plau­si­ble that ar­tifi­cial in­tel­li­gences would vastly ac­cel­er­ate progress be­fore they are highly in­tel­li­gent them­selves. (I’m not fully sold on that ar­gu­ment ei­ther, but that’s an­other story for an­other time.)

Given that sig­nifi­cant re­sources have been in­vested in AI /​ ML for quite a while, it seems that dis­con­tin­u­ous progress – on the path to AGI, not dur­ing or af­ter the tran­si­tion – would be at odds with usual pat­terns of tech­nolog­i­cal progress. The refer­ence class I’m think­ing of is “im­prove­ment of a grad­ual at­tribute (like in­tel­li­gence) of a tech­nol­ogy over time, if sig­nifi­cant re­sources are in­vested”. Ex­am­ples that come to mind are the max­i­mal speed of cars, which in­creased steadily over time, or per­haps com­put­ing power and mem­ory space, which also pro­gresses very smoothly.

(See also AI Im­pact’s dis­con­tin­u­ous progress in­ves­ti­ga­tion. They ac­tu­ally con­sider new land speed records set by jet-pro­pel­led ve­hi­cles one of the few cases of (mod­er­ate) dis­con­ti­nu­ities that they’ve found so far. To me that doesn’t feel analo­gous in terms of the nec­es­sary mag­ni­tude of the dis­con­ti­nu­ity, though.)

The point is even stronger if “in­tel­li­gence” is ac­tu­ally a col­lec­tion of many dis­tinct skills and abil­ities rather than a mean­ingful, unified prop­erty (in the con­text of ma­chine in­tel­li­gence). In that case it re­quires progress on many fronts, com­pa­rable to the “over­all qual­ity” of cars or com­puter hard­ware.

It’s pos­si­ble that progress ac­cel­er­ates sim­ply due to in­creased in­ter­est – and there­fore in­creased fund­ing and other re­sources – as more peo­ple recog­nise its po­ten­tial. In­deed, while his­tor­i­cal progress in AI was fairly smooth, there may have been some ac­cel­er­a­tion over the last decade, plau­si­bly due to in­creased in­ter­est. So per­haps that could hap­pen to an even larger de­gree in the fu­ture?

There is, how­ever, already sig­nifi­cant ex­cite­ment (per­haps hype) around AI, so it seems un­likely to me that this could in­crease the rate of progress by or­ders of mag­ni­tude. In par­tic­u­lar, if highly tal­ented re­searchers are the main bot­tle­neck, you can’t scale up the field by sim­ply pour­ing more money into it. Plus, it has been ar­gued that the next AI win­ter is well on its way, i.e. we ac­tu­ally start to see a de­cline, not a fur­ther in­crease, of in­ter­est in AI.


One of the most com­mon rea­sons to nev­er­the­less as­sign a non-neg­ligible prob­a­bil­ity – say, 10% – is sim­ply that we’re so clue­less about what will hap­pen in the fu­ture that we shouldn’t be con­fi­dent ei­ther way, and should thus fa­vor a broad dis­tri­bu­tion over timelines.

But are we ac­tu­ally that ig­no­rant? It is in­deed ex­tremely hard, if not im­pos­si­ble, to pre­dict the spe­cific re­sults of com­plex pro­cesses over long times­pans – like, which memes and hash­tags will be trend­ing on Twit­ter in May 2038. How­ever, the plau­si­bil­ity or im­plau­si­bil­ity of short timelines is not a ques­tion of this type since the de­vel­op­ment of AGI would be the re­sult of a broad trend, not a spe­cific re­sult. We have rea­son­ably strong forms of ev­i­dence at our dis­posal: we can look at his­tor­i­cal and cur­rent rates of progress in AI, we can con­sider gen­eral pat­terns of in­no­va­tion and tech­nolog­i­cal progress, and we can es­ti­mate how hard gen­eral in­tel­li­gence is (e.g. whether it’s an ag­gre­ga­tion of many smart heuris­tics vs. a sin­gle in­sight).

Also, what kind of prob­a­bil­ity should an ig­no­rant prior as­sign to AGI in 10 years? 10%? But then wouldn’t you as­sign 10% to ad­vanced nan­otech­nol­ogy in 10 years be­cause of ig­no­rance? What about nu­clear risk – we’re clue­less about that too, so maybe 10% chance of a ma­jor nu­clear catas­tro­phe in the next 10 years? 10% on a com­plete break­down of the global fi­nan­cial sys­tem? But if you keep do­ing that with more and more things, you’ll end up with near cer­tainty of some­thing crazy hap­pen­ing in the next 10 years, which seems wrong given his­tor­i­cal base rates. So per­haps an ig­no­rant prior should ac­tu­ally place much lower prob­a­bil­ity on each in­di­vi­d­ual event.


But per­haps one’s own opinion shouldn’t count for much any­way, and we should in­stead defer to some set of ex­perts? Un­for­tu­nately, in­ter­pret­ing ex­pert opinion is tricky. On the one hand, in some sur­veys ma­chine learn­ing re­searchers put non-neg­ligible prob­a­bil­ity on “hu­man-level in­tel­li­gence” (what­ever that means) in 10 years. On the other hand, my im­pres­sion from in­ter­act­ing with the com­mu­nity is that the pre­dom­i­nant opinion is still to con­fi­dently dis­miss a short timeline sce­nario, to the point of not even se­ri­ously en­gag­ing with it.

Alter­na­tively, one could look at the opinions of smart peo­ple in the effec­tive al­tru­ism com­mu­nity (“EA ex­perts”), who tend to as­sign a non-neg­ligible prob­a­bil­ity to short timelines. But this (vaguely defined) set of peo­ple is sub­ject to a self-se­lec­tion bias – if you think AGI is likely to hap­pen soon, you’re much more likely to spend years think­ing and talk­ing about that – and has lit­tle ex­ter­nal val­i­da­tion of their “ex­pert” sta­tus.

A less ob­vi­ous source of “ex­pert opinion” are the fi­nan­cial mar­kets – be­cause mar­ket par­ti­ci­pants have a strong in­cen­tive to get things right – and their im­plicit opinion is to con­fi­dently dis­miss the pos­si­bil­ity of short timelines.

In any case, it’s not sur­pris­ing if some peo­ple have wrong be­liefs about this kind of ques­tion. Lots of peo­ple are wrong about lots of things. It’s not un­usual that com­mu­ni­ties (like EA or the ma­chine learn­ing com­mu­nity) have idiosyn­cratic bi­ases or suffer from group­think. The ques­tion is whether more peo­ple buy into short timelines com­pared to what you’d ex­pect con­di­tional on short timelines be­ing wrong (in which case some peo­ple will still buy into it, com­pa­rable to past AI hy­pes).

Similarly, do we see fewer or more peo­ple buy into short timelines com­pared to what you’d ex­pect if short timelines are right (in which case there will surely be a few stub­born pro­fes­sors who won’t be­lieve it un­til the very end)?

I think the an­swer to the sec­ond ques­tion is “fewer”. Per­haps the an­swer to the first ques­tion is “some­what more” but I think that’s less clear.


All things con­sid­ered, I think the prob­a­bil­ity of a short timeline sce­nario (i.e. AGI within 10 years) is not more than 1-2%. What am I miss­ing?