Eliezer’s post was less a takedown of the report, and more a takedown of the idea that the report provides a strong basis for expecting AGI in ~2050, or for discriminating scenarios like ‘AGI in 2030’, ‘AGI in 2050’, and ‘AGI in 2070’.
The report itself was quite hedged, and Holden posted a follow-up clarification emphasizing that “biological anchors” is about bounding, not pinpointing, AI timelines. So it’s not clear to me that Eliezer and Ajeya/Holden/etc. even disagree about the core question “do biological anchors provide a strong case for putting a median AGI year in ~2050?”, though maybe they disagree on the secondary question of how useful the “bounds” are.
Copying over my high-level view, which I recently wrote on Twitter:
I agree with the basic Eliezer argument in Biology-Inspired AGI Timelines that the bio-anchors stuff isn’t important or useful because AGI is a software problem, and we neither know which specific software insights are needed, nor how long it will take to get to those software insights, nor the relationship between those insights and hardware requirements.
Focusing on things like bio-anchors and hardware trends is streetlight-fallacy reasoning: it’s taking the 2% of the territory we do know about and heavily heavily focusing on that 2%, while shrugging our shoulders at the other 98%.
Like, bio-anchors reasoning might help tell you whether to expect AGI this century versus expecting it in a thousand years, but it won’t help you discriminate 2030 from 2050 from 2070 at all.
Insofar as we need to think about timelines at all, it’s true that we need some sort of prior, at least a very vague one.
The problem with the heuristic ‘look under the streetlight and anchor your prior to whatever you found under the streetlight, however marginal’ is that the info under the streetlight isn’t a random sampling from the space of relevant unknown facts about AGI; it’s a very specific and unusual kind of information.
IMO you’d be better off thinking first about that huge space of unknowns and anchoring to far fuzzier and more uncertain guesses about the whole space, rather than fixating on a very specific much-more-minor fact that’s easier to gather data about.
E.g., consider five very different a priori hypotheses about ‘what insights might be needed for AGI’, another five very different hypotheses about ‘how might different sorts of software progress relate to hardware requirements’, etc.
Think about different world-histories that might occur, and how surprised you’d be by those world-histories.
Think about worlds where things go differently than you’re expecting in 2060, and about what those worlds would genuinely retrodict about the present / past.
E.g., I think scenario analysis makes it more obvious that in worlds where AGI is 30 years away, current trends will totally break at some point on that path, radically new techniques will be developed, etc.
Think about how different the field of AI was in 1992 compared to today, or in 1962 compared to 1992.
When you’re spending most of your time looking under the streetlight — rather than grappling with how little is known, trying to painstakingly refine your instincts and intuitions about the harder-to-reason-about aspects of the problem, etc. — I think it becomes overly tempting to treat current trendlines as laws of nature that will be true forever (or that at least have a strong default of being true forever), rather than as ‘patterns that arose a few years ago and will plausibly continue for a few years more, before being replaced by new patterns and growth curves’.
Commenting on a few minor points from Scott’s post, since I meant to write a full reply at some point but haven’t had the time:
But also, there are about 10^15 synapses in the brain, each one spikes about once per second, and a synaptic spike probably does about one FLOP of computation. [...] So a human-level AI would also need to do 10^15 floating point operations per second? Unclear.
I’d say ‘clearly not, for some possible AI designs’; but maybe it will be true for the first AIs we actually build, shrug.
Or you might do what OpenPhil did and just look at a bunch of examples of evolved vs. designed systems and see which are generally better:
Why aren’t there examples like ‘amount of cargo a bird can carry compared to an airplane’, or ‘number of digits a human can multiply together in ten seconds compared to a computer’?
Seems like you’ll get a skewed number if your brainstorming process steers away from examples like these altogether.
‘AI physicist’ is less like an artificial heart (trying to exactly replicate the structure of a biological organ functioning within a specific body), more like a calculator (trying to do a certain kind of cognitive work, without any constraint at all to do it in a human-like way).
Perhaps also relevant, though it isn’t forecasting, is Eliezer’s weak (in my opinion) attempted takedown of Ajeya Cotra’s bioanchors report on AI timelines. Here’s Eliezer’s bioanchors takedown attempt, here’s Holden Karnofsky’s response to Eliezer, and here’s Scott Alexander’s response.
Eliezer’s post was less a takedown of the report, and more a takedown of the idea that the report provides a strong basis for expecting AGI in ~2050, or for discriminating scenarios like ‘AGI in 2030’, ‘AGI in 2050’, and ‘AGI in 2070’.
The report itself was quite hedged, and Holden posted a follow-up clarification emphasizing that “biological anchors” is about bounding, not pinpointing, AI timelines. So it’s not clear to me that Eliezer and Ajeya/Holden/etc. even disagree about the core question “do biological anchors provide a strong case for putting a median AGI year in ~2050?”, though maybe they disagree on the secondary question of how useful the “bounds” are.
Copying over my high-level view, which I recently wrote on Twitter:
Commenting on a few minor points from Scott’s post, since I meant to write a full reply at some point but haven’t had the time:
I’d say ‘clearly not, for some possible AI designs’; but maybe it will be true for the first AIs we actually build, shrug.
Why aren’t there examples like ‘amount of cargo a bird can carry compared to an airplane’, or ‘number of digits a human can multiply together in ten seconds compared to a computer’?
Seems like you’ll get a skewed number if your brainstorming process steers away from examples like these altogether.
‘AI physicist’ is less like an artificial heart (trying to exactly replicate the structure of a biological organ functioning within a specific body), more like a calculator (trying to do a certain kind of cognitive work, without any constraint at all to do it in a human-like way).