Ajeya Cotra: Yeah. I mean, I think I would frame it as like 12–15% by 2036, which is kind of the original question, a median of 2055, and then 70–80% chance this century. That’s how I would put the bottom line.
I’d be really interested in hearing about the discussions you have with people that have earlier median estimates, and/or what you expect those discussions would resolve around. For example, I saw that the Metaculus crowd has a median estimate of 2035 for fully general AI. Skimming their discussions, they might rely more on recent ML progress than you.
Like Linch says, some of the reason the Metaculus median is lower than mine is probably because they have a weaker definition; 2035 seems like a reasonable median for “fully general AI” as they define it, and my best guess may even be sooner.
With that said, I’ve definitely had a number of conversations with people who have shorter timelines than me for truly transformative AI; Daniel Kokotajlo articulates a view in this space here. Disagreements tend to be around the following points:
People with shorter timelines than me tend to feel that the notion of “effective horizon length” either doesn’t make sense, or that training time scales sub-linearly rather than linearly with effective horizon length, or that models with short effective horizon lengths will be transformative despite being “myopic.” They generally prefer a model where a scaled-up GPT-3 constitutes transformative AI. Since I published my draft report, Guille Costa (an intern at Open Philanthropy) released a version of the model that explicitly breaks out “scaled up GPT-3” as a hypothesis, which would imply a median of 2040 if all my other assumptions are kept intact.
They also tend to feel that extrapolations of when existing model architectures will reach human-level performance on certain benchmarks, e.g. a recently-created multitask language learning benchmark, implies that “human-level” capability would be reached at ~1e13 or 1e14 FLOP/subj sec rather than ~1e16 FLOP/subj sec as I guessed in my report. I’m more skeptical of extrapolation from benchmarks because my guess is that the benchmarks we have right now were selected to be hard-but-doable for our current generation of models, and once models start doing extremely well at these benchmarks we will likely generate harder benchmarks with more work, and there may be multiple rounds of this process.
They tend to be less skeptical on priors of sudden takeoff, which leads them to put less weight on considerations like “If transformative AI is going to be developed in only 5-10 years, why aren’t we seeing much more economic impacts from AI today?”
Some of them also feel that I underestimate the algorithmic progress that will be made over the next ~5 years: they may not disagree with my characterization of the current scaling behavior of ML systems, but they place more weight than I would on an influx of researchers (potentially working with ML-enabled tools) making new discoveries that shift us to a different scaling regime, e.g. one more like the “Lifetime Anchor” hypothesis.
Finally, some people with shorter timelines than me tend to expect that rollout of AI technologies will be faster and smoother than I do, and expect there to be less delay from “working out kinks” or making systems robust enough to deploy.
Thanks, super interesting! In my very premature thinking, the question of algorithmic progress is most load-bearing. My background is in cognitive science and my broad impression is that
human cognition is not *that* crazy complex,
that I wouldn’t be surprised at all if one of the broad architectural ideas I’ve seen floating around on human cognition could afford “significant” steps towards proper AGI
e.g. how Bayesian inference and Reinforcement Learning maybe realized in the predictive coding framework was impressive to me, for example flashed out by Steve Byrnes on LessWrong
or e.g. rough sketches of different systems that fulfill specific functions like in the further breakdown of System 2 in Stanovich’s Rationality and the Reflective Mind
when thinking about how many „significant“ steps or insights we still need until AGI, I think more on the order of less than ten
(I’ve heard the idea of “insight-based forecasting” from a Joscha Bach interview)
those insights might not be extremely expensive and, once had, cheap-ish to implement
e.g. the GANs story maybe fits this, they’re not crazy complicated, not crazy hard to implement, but very powerful
This all feels pretty freewheeling so far. Would be really interested in further thoughts or reading recommendation on algorithmic progress.
My approach to thinking about algorithmic progress has been to try to extrapolate the rate of past progress forward; I rely on two sources for this, a paper by Katja Grace and a paper by Danny Hernandez and Tom Brown. One question I’d think about when forming a view on this is whether arguments like the ones you make should lead you to expect algorithmic progress to be significantly faster than the trendline, or whether those considerations are already “priced in” to the existing trendline.
And yes, thanks, the point about thinking with trendlines in mind is really good.
Maybe those two developments could be relevant:
bigger number of recent ML/CogSci/Comp. Neuroscience graduates that academically grew up in times of noticeable AI progress and much more widespread aspirations to build AGI than the previous generation
related to my question about non-academic open-source projects: If there is a certain level of computation necessary to solve interesting general reasoning gridworld problems with new algorithms, then we might unlock a lot of work in the coming years
Thanks! :) I find Grace’s paper a little bit unsatisfying. From the outside, fields around like SAT, factoring, scheduling and linear optimization seem only weakly analogous to the fields around developing general thinking capabilities. It seems to me that the former is about hundreds of researchers going very deep into very specific problems and optimizing a ton to produce slightly more elegant and optimal solutions, whereas the latter is more about smart and creative “pioneers” having new insights how to frame the problem correctly and finding new relatively simple architectures that make a lot of progress.
What would be more informative for me?
by above logic maybe I would focus more on progress of younger fields within computer science
also maybe there is a way to measure how “random” praciticioners perceive the field to be—maybe just asking them how surprised they are by recent breakthroughs is a solid measures of how many other potential breakthroughs are still out there
also I’d be interested in solidifying my very rough impression that breakthroughs like transformers or GANs relatively simple algorithms in comparison with breakthroughs in other areas of computer science
evolution’s algorithmic progress would maybe also be informative to me, i.e. how much trial and error was roughly invested to make specific jumps
e.g. I’m reading Pearls Book of Why and he makes a tentative claim that counterfactual reasoning is something that appeared at some point, and the first sign we can report of it is the lion-man from roughly 40.000 years ago
though of course evolution did not aim at general intelligence, e.g. saying “evolution took hundreds of millions of years to develop an AGI” in this context seems disanalogous
how big of a fraction of human cognition do we actually need for TAI? E.g. we might save about an order of magnitude by ditching vision and focussing on language?
Note that the definition of “fully general AI” on that Metaculus question is considerably weaker than how Open Phil talks about “transformative AI.”
For these purposes we will thus define “an artificial general intelligence” as a single unified software system that can satisfy the following criteria, all easily completable by a typical college-educated human.
Able to reliably pass a Turing test of the type that would win the Loebner Silver Prize.
Be able to score 75th percentile (as compared to the corresponding year’s human students; this was a score of 600 in 2016) on all the full mathematics section of a circa-2015-2020 standard SAT exam, using just images of the exam pages and having less than ten SAT exams as part of the training data. (Training on other corpuses of math problems is fair game as long as they are arguably distinct from SAT exams.)
Be able to learn the classic Atari game “Montezuma’s revenge” (based on just visual inputs and standard controls) and explore all 24 rooms based on the equivalent of less than 100 hours of real-time play (see closely-related question.)
Right, to be clear I think this is (mostly) not your fault.
Unfortunately others have made this and similar mistakes before, for both other questions and this specific question.
Obviously some of the onus is on user error, but I think the rest of us (the forecasting community and the Metaculus platform) should do better on having the intuitive interpretation of the headline question match the question specifications, and vice versa.
Regarding forecasts on transformative AI:
I’d be really interested in hearing about the discussions you have with people that have earlier median estimates, and/or what you expect those discussions would resolve around. For example, I saw that the Metaculus crowd has a median estimate of 2035 for fully general AI. Skimming their discussions, they might rely more on recent ML progress than you.
Like Linch says, some of the reason the Metaculus median is lower than mine is probably because they have a weaker definition; 2035 seems like a reasonable median for “fully general AI” as they define it, and my best guess may even be sooner.
With that said, I’ve definitely had a number of conversations with people who have shorter timelines than me for truly transformative AI; Daniel Kokotajlo articulates a view in this space here. Disagreements tend to be around the following points:
People with shorter timelines than me tend to feel that the notion of “effective horizon length” either doesn’t make sense, or that training time scales sub-linearly rather than linearly with effective horizon length, or that models with short effective horizon lengths will be transformative despite being “myopic.” They generally prefer a model where a scaled-up GPT-3 constitutes transformative AI. Since I published my draft report, Guille Costa (an intern at Open Philanthropy) released a version of the model that explicitly breaks out “scaled up GPT-3” as a hypothesis, which would imply a median of 2040 if all my other assumptions are kept intact.
They also tend to feel that extrapolations of when existing model architectures will reach human-level performance on certain benchmarks, e.g. a recently-created multitask language learning benchmark, implies that “human-level” capability would be reached at ~1e13 or 1e14 FLOP/subj sec rather than ~1e16 FLOP/subj sec as I guessed in my report. I’m more skeptical of extrapolation from benchmarks because my guess is that the benchmarks we have right now were selected to be hard-but-doable for our current generation of models, and once models start doing extremely well at these benchmarks we will likely generate harder benchmarks with more work, and there may be multiple rounds of this process.
They tend to be less skeptical on priors of sudden takeoff, which leads them to put less weight on considerations like “If transformative AI is going to be developed in only 5-10 years, why aren’t we seeing much more economic impacts from AI today?”
Some of them also feel that I underestimate the algorithmic progress that will be made over the next ~5 years: they may not disagree with my characterization of the current scaling behavior of ML systems, but they place more weight than I would on an influx of researchers (potentially working with ML-enabled tools) making new discoveries that shift us to a different scaling regime, e.g. one more like the “Lifetime Anchor” hypothesis.
Finally, some people with shorter timelines than me tend to expect that rollout of AI technologies will be faster and smoother than I do, and expect there to be less delay from “working out kinks” or making systems robust enough to deploy.
Thanks, super interesting! In my very premature thinking, the question of algorithmic progress is most load-bearing. My background is in cognitive science and my broad impression is that
human cognition is not *that* crazy complex,
that I wouldn’t be surprised at all if one of the broad architectural ideas I’ve seen floating around on human cognition could afford “significant” steps towards proper AGI
e.g. how Bayesian inference and Reinforcement Learning maybe realized in the predictive coding framework was impressive to me, for example flashed out by Steve Byrnes on LessWrong
or e.g. rough sketches of different systems that fulfill specific functions like in the further breakdown of System 2 in Stanovich’s Rationality and the Reflective Mind
when thinking about how many „significant“ steps or insights we still need until AGI, I think more on the order of less than ten
(I’ve heard the idea of “insight-based forecasting” from a Joscha Bach interview)
those insights might not be extremely expensive and, once had, cheap-ish to implement
e.g. the GANs story maybe fits this, they’re not crazy complicated, not crazy hard to implement, but very powerful
This all feels pretty freewheeling so far. Would be really interested in further thoughts or reading recommendation on algorithmic progress.
My approach to thinking about algorithmic progress has been to try to extrapolate the rate of past progress forward; I rely on two sources for this, a paper by Katja Grace and a paper by Danny Hernandez and Tom Brown. One question I’d think about when forming a view on this is whether arguments like the ones you make should lead you to expect algorithmic progress to be significantly faster than the trendline, or whether those considerations are already “priced in” to the existing trendline.
And yes, thanks, the point about thinking with trendlines in mind is really good.
Maybe those two developments could be relevant:
bigger number of recent ML/CogSci/Comp. Neuroscience graduates that academically grew up in times of noticeable AI progress and much more widespread aspirations to build AGI than the previous generation
related to my question about non-academic open-source projects: If there is a certain level of computation necessary to solve interesting general reasoning gridworld problems with new algorithms, then we might unlock a lot of work in the coming years
Thanks! :) I find Grace’s paper a little bit unsatisfying. From the outside, fields around like SAT, factoring, scheduling and linear optimization seem only weakly analogous to the fields around developing general thinking capabilities. It seems to me that the former is about hundreds of researchers going very deep into very specific problems and optimizing a ton to produce slightly more elegant and optimal solutions, whereas the latter is more about smart and creative “pioneers” having new insights how to frame the problem correctly and finding new relatively simple architectures that make a lot of progress.
What would be more informative for me?
by above logic maybe I would focus more on progress of younger fields within computer science
also maybe there is a way to measure how “random” praciticioners perceive the field to be—maybe just asking them how surprised they are by recent breakthroughs is a solid measures of how many other potential breakthroughs are still out there
also I’d be interested in solidifying my very rough impression that breakthroughs like transformers or GANs relatively simple algorithms in comparison with breakthroughs in other areas of computer science
evolution’s algorithmic progress would maybe also be informative to me, i.e. how much trial and error was roughly invested to make specific jumps
e.g. I’m reading Pearls Book of Why and he makes a tentative claim that counterfactual reasoning is something that appeared at some point, and the first sign we can report of it is the lion-man from roughly 40.000 years ago
though of course evolution did not aim at general intelligence, e.g. saying “evolution took hundreds of millions of years to develop an AGI” in this context seems disanalogous
how big of a fraction of human cognition do we actually need for TAI? E.g. we might save about an order of magnitude by ditching vision and focussing on language?
Sherry et al. have a more exhaustive working paper about algorithmic progress in a wide variety of fields.
Note that the definition of “fully general AI” on that Metaculus question is considerably weaker than how Open Phil talks about “transformative AI.”
Thanks, I didn‘t read that carefully enough!
Right, to be clear I think this is (mostly) not your fault.
Unfortunately others have made this and similar mistakes before, for both other questions and this specific question.
Obviously some of the onus is on user error, but I think the rest of us (the forecasting community and the Metaculus platform) should do better on having the intuitive interpretation of the headline question match the question specifications, and vice versa.