How will AI develop over the next few centuries? Three scenarios seem particularly likely to me:
“Solving Intelligence”: Within the next 50 years, a top AI lab like Deepmind or OpenAI builds a superintelligent AI system, by using massive compute within our current ML paradigm.
“Comprehensive AI Systems”: Over the next century or few, computers keep getting better at a bunch of different domains. No one AI system is incredible at everything, each new job requires fine-tuning and domain knowledge and human-in-the-loop supervision, but soon enough we hit annual GDP growth of 25%.
“No takeoff”: Looks qualitatively similar to the above, except growth remains steady around 2% for at least several centuries. We remain in the economic paradigm of the Industrial Revolution, and AI makes an economic contribution similar to that of electricity or oil without launching us into a new period of human history. Progress continues as usual.
For clarify my beliefs about AI timelines, I found it helpful to flesh out these concrete “scenarios” by answering a set of closely related questions about how transformative AI might develop:
When do we achieve TAI? AGI? Superintelligence? How fast is takeoff? Who builds it? How much compute does it require? How much does that cost? Agent or Tool? Is machine learning the paradigm, or do we have another fundamental shift in research direction? What are the key AI Safety challenges? Who is best positioned to contribute?
The potentially useful insight here is that answering one of these questions helps you answer the others. If massive compute is necessary, then TAI will be built by a few powerful governments or corporations, not by a diverse ecosystem of small startups. If TAI isn’t achieved for another century, that affects which research agendas are most important today. Follow this exercise for awhile, and you might end up with a handful of distinct scenarios, and then you can judge the relative likelihood and timelines of each.
Here’s my rough sketch of what each of these mean. [Dumping a lot of rough notes here, which is why I’m posting as a shortform.]
Solving Intelligence: Within the next 20-50 years, a top AI lab like Deepmind or OpenAI builds a superintelligent AI system.
Machine learning is the paradigm that brings us to superintelligence. Most progress is driven by compute. Our algorithms are similar to the human brain, and therefore require similar amounts of compute.
It becomes a compute war. You’re taking the same fundamental algorithms and spending a hundred billion dollars on compute, and it works. (Informed by Ajeya’s report, IMO the most important upshot of which being that spending a truly massive amount of money can cover a sizeable portion of the difference between our current compute and the compute of the human brain. If human brain-level compute is an important threshold, then the few actors who could spend $100B+ are have an advantage of decades against against actors who can only spend millions. Would like to discuss this further.)
This is most definitely not CAIS. There would be one or two or ten superintelligent AI systems, but not two million.
Very few people can contribute effectively to AI Safety, because to contribute effectively you have to be at one of only a handful of organizations in the world. You need to be in “the room where it happens”, whether that’s the AI lab developing the superintelligence or the government attempting to monitor the project. The handful of people who can contribute are incredibly valuable.
What AI safety stuff matters?
Technical AI safety research. The people right now who are building AI that scales safely. It turns out you can do effective research now because our current methods are the methods that bring us to superintelligence, and whether or not our current research is good enough determines whether or not we survive.
Highest levels of government, for their ability to regulate AI labs. A project like this could be nationalized, or carried out under strict oversight from government regulators. Realistically I’d expect the opposite, that governments would be too slow to see the risks and rewards in such a technical domain.
People who imagine long-term policies for governing AI. I don’t know how much useful work exists here, but I have to imagine there’s some good stuff about how to run the world undersuperintelligence. What’s the game theory of multipolar scenarios? What are the points of decisive strategic advantage?
Comprehensive AI Systems: Over the next century or few, computers keep getting better at a bunch of different domains. No one AI system is incredible at everything, each new job requires fine-tuning and domain knowledge and human-in-the-loop supervision, but soon enough we hit annual GDP growth of 25%.
Governments go about international relations the same as usual, just with better weapons. There’s some strategic effects of this that Henry Kissinger and Justin Ding understand quite well, but there’s no instant collapse into one world government or anything. There’s a few outside risks here that would be terrible (a new WMD, or missile defense systems that threaten MAD), but basically we just get killer robots, which will probably be fine.
Killer robots are a key AI safety training ground. If they’re inevitable, we should be integrated within enemy lines in order to deploy safely.
We have lots of warning shots.
What are the existential risks? Nuclear war. Autonomous weapons accidents, which I suppose could turn out to be existential?? Long-term misalignment: over the next 300 years, we hand off the fate of the universe to the robots, and it’s not quite the right trajectory.
What AI Safety work is most valuable?
Run-of-the-mill AI Policy work. Accomplishing normal government objectives often unrelated to existential risk specifically, by driving forward AI progress in a technically-literate and altruistically-thoughtful way.
Driving forward AI progress. It’s a valuable technology that will help lots of people, and accelerating its arrival is a good thing.
With particular attention to safety. Building a CS culture, a Silicon Valley, a regulatory environment, and international cooperation that will sustain the three hundred year transition.
Working in military AI systems. They’re the killer robots most likely to run amok and kill some people (or 7 billion). Malfunctioning AI can also cause nuclear war by setting off geopolitical conflict. Also new WMDs would be terrible.
No takeoff: Looks qualitatively similar to the above, except growth remains steady around 2% for at least several centuries. We remain in the economic paradigm of the Industrial Revolution, and AI makes an economic contribution similar to that of electricity or oil without launching us into a new period of human history.
This seems entirely possible, maybe even the most likely outcome. I’ve been surrounded by people talking about short timelines from a pretty young age so I never really thought about this possibility, but “takeoff” is not guaranteed. The world in 500 years could resemble the world today; in fact, I’d guess most thoughtful people don’t think much about transformative AI and would assume that this is the default scenario.
Part of why I think this is entirely plausible is because I don’t see many independently strong arguments for short AI timelines:
IMO the strongest argument for short timelines is that, within the next few decades, we’ll cross the threshold for using more compute than the human brain. If this turns out to be a significant threshold and a fair milestone to anchor against, then we could hit an inflection point and rapidly see Bostrom Superintelligence-type scenarios.
I see this belief as closely associated with the entire first scenario described above: Held by OpenAI/DeepMind, the idea that we will “solve intelligence” with an agenty AI running a simple fundamental algorithm with massive compute and effectively generalizing across many domains.
IIRC, the most prominent early argument for short AI timelines, as discussed by Bostrom, Yudkowsky, and others, was recursive self-improvement. The AI will build smarter AIs, meaning we’ll eventually hit an inflection point of runaway improvement positively feeding into itself and rapidly escalating from near-human to lightyears-beyond-human intelligence. This argument seems less popular in recent years, though I couldn’t say exactly why. My only opinion would be that this seems more like an argument for “fast takeoff” (once we have near-human level AI systems for building AI systems, we’ll quickly achieve superhuman performance in that area), but does not tell you when that takeoff will occur. For all we know, this fast takeoff could happen in hundreds of years. (Or I could be misunderstanding the argument here, I’d like to think more about it.)
Surveys asking AI researchers when they expect superhuman AI have received lots of popular coverage and might be driving widespread acceptance of short timelines. My very subjective and underinformed intution puts little weight on these surveys compared to the object level arguments. The fact that people trying to build superintelligence believe it’s possible within their lifetime certainly makes me take that possibility seriously, but it doesn’t provide much of an upper bound on how long it might take. If the current consensus of AI researchers proves to be wrong about progress over the next century, I wouldn’t expect their beliefs about the next five or ten centuries to hold up—the worldview assumptions might just be entirely off-base.
These are the only three arguments for short timelines I’ve ever heard and remembered. Interested if I’m forgetting anything big here.
Compare this to the simple prior that history will continue with slow and steady single-digit growth as it has since the Industrial Revolution, and I see a significant chance that we don’t see AI takeoff for centuries, if ever. (That’s before considering object level arguments for longer timelines, which admittedly I don’t see many of, and therefore I don’t put much weight on.)
I haven’t fully thought through all of this, but would love to hear others thoughts on the probability of “no takeoff”.
This is pretty rough around the edges, but these three scenarios seem like the key possibilities for the next few centuries that I can see at this point. For the hell of it, I’ll give some very weak credences: 10% that we solve superintelligence within decades, 25% that CAIS brings double-digit growth within a century or so, maybe 50% that human progress continues as usual for at least a few centuries, and (at least) 15% that what ends up happening looks nothing like any of these scenarios.
Very interested in hearing any critiques or reactions to these scenarios or the specific arguments within.
I like the no takeoff scenario intuitive analysis, and find that I also haven’t really imagined this as a concrete possibility. Generally, I like that you have presented clearly distinct scenarios and that the logic is explicit and coherent. Two thoughts that came to mind:
Somehow in the CAIS scenario, I also expect the rapid growth and the delegation of some economic and organizational work to AI to have some weird risks that involve something like humanity getting pushed away from the economic ecosystem while many autonomous systems are self-sustaining and stuck in a stupid and lifeless revenue-maximizing loop. I couldn’t really pinpoint an x-risk scenario here.
Recursive self-improvement can also happen within long periods of time, not necessarily leading to a fast takeoff, especially if the early gains are much easier than later gains (which might make more sense if we think of AI capability development as resulting mostly from computational improvements rather than algorithmic).
Three Scenarios for AI Progress
How will AI develop over the next few centuries? Three scenarios seem particularly likely to me:
“Solving Intelligence”: Within the next 50 years, a top AI lab like Deepmind or OpenAI builds a superintelligent AI system, by using massive compute within our current ML paradigm.
“Comprehensive AI Systems”: Over the next century or few, computers keep getting better at a bunch of different domains. No one AI system is incredible at everything, each new job requires fine-tuning and domain knowledge and human-in-the-loop supervision, but soon enough we hit annual GDP growth of 25%.
“No takeoff”: Looks qualitatively similar to the above, except growth remains steady around 2% for at least several centuries. We remain in the economic paradigm of the Industrial Revolution, and AI makes an economic contribution similar to that of electricity or oil without launching us into a new period of human history. Progress continues as usual.
For clarify my beliefs about AI timelines, I found it helpful to flesh out these concrete “scenarios” by answering a set of closely related questions about how transformative AI might develop:
When do we achieve TAI? AGI? Superintelligence? How fast is takeoff? Who builds it? How much compute does it require? How much does that cost? Agent or Tool? Is machine learning the paradigm, or do we have another fundamental shift in research direction? What are the key AI Safety challenges? Who is best positioned to contribute?
The potentially useful insight here is that answering one of these questions helps you answer the others. If massive compute is necessary, then TAI will be built by a few powerful governments or corporations, not by a diverse ecosystem of small startups. If TAI isn’t achieved for another century, that affects which research agendas are most important today. Follow this exercise for awhile, and you might end up with a handful of distinct scenarios, and then you can judge the relative likelihood and timelines of each.
Here’s my rough sketch of what each of these mean. [Dumping a lot of rough notes here, which is why I’m posting as a shortform.]
Solving Intelligence: Within the next 20-50 years, a top AI lab like Deepmind or OpenAI builds a superintelligent AI system.
Machine learning is the paradigm that brings us to superintelligence. Most progress is driven by compute. Our algorithms are similar to the human brain, and therefore require similar amounts of compute.
It becomes a compute war. You’re taking the same fundamental algorithms and spending a hundred billion dollars on compute, and it works. (Informed by Ajeya’s report, IMO the most important upshot of which being that spending a truly massive amount of money can cover a sizeable portion of the difference between our current compute and the compute of the human brain. If human brain-level compute is an important threshold, then the few actors who could spend $100B+ are have an advantage of decades against against actors who can only spend millions. Would like to discuss this further.)
This is most definitely not CAIS. There would be one or two or ten superintelligent AI systems, but not two million.
Very few people can contribute effectively to AI Safety, because to contribute effectively you have to be at one of only a handful of organizations in the world. You need to be in “the room where it happens”, whether that’s the AI lab developing the superintelligence or the government attempting to monitor the project. The handful of people who can contribute are incredibly valuable.
What AI safety stuff matters?
Technical AI safety research. The people right now who are building AI that scales safely. It turns out you can do effective research now because our current methods are the methods that bring us to superintelligence, and whether or not our current research is good enough determines whether or not we survive.
Highest levels of government, for their ability to regulate AI labs. A project like this could be nationalized, or carried out under strict oversight from government regulators. Realistically I’d expect the opposite, that governments would be too slow to see the risks and rewards in such a technical domain.
People who imagine long-term policies for governing AI. I don’t know how much useful work exists here, but I have to imagine there’s some good stuff about how to run the world undersuperintelligence. What’s the game theory of multipolar scenarios? What are the points of decisive strategic advantage?
Comprehensive AI Systems: Over the next century or few, computers keep getting better at a bunch of different domains. No one AI system is incredible at everything, each new job requires fine-tuning and domain knowledge and human-in-the-loop supervision, but soon enough we hit annual GDP growth of 25%.
Governments go about international relations the same as usual, just with better weapons. There’s some strategic effects of this that Henry Kissinger and Justin Ding understand quite well, but there’s no instant collapse into one world government or anything. There’s a few outside risks here that would be terrible (a new WMD, or missile defense systems that threaten MAD), but basically we just get killer robots, which will probably be fine.
Killer robots are a key AI safety training ground. If they’re inevitable, we should be integrated within enemy lines in order to deploy safely.
We have lots of warning shots.
What are the existential risks? Nuclear war. Autonomous weapons accidents, which I suppose could turn out to be existential?? Long-term misalignment: over the next 300 years, we hand off the fate of the universe to the robots, and it’s not quite the right trajectory.
What AI Safety work is most valuable?
Run-of-the-mill AI Policy work. Accomplishing normal government objectives often unrelated to existential risk specifically, by driving forward AI progress in a technically-literate and altruistically-thoughtful way.
Driving forward AI progress. It’s a valuable technology that will help lots of people, and accelerating its arrival is a good thing.
With particular attention to safety. Building a CS culture, a Silicon Valley, a regulatory environment, and international cooperation that will sustain the three hundred year transition.
Working in military AI systems. They’re the killer robots most likely to run amok and kill some people (or 7 billion). Malfunctioning AI can also cause nuclear war by setting off geopolitical conflict. Also new WMDs would be terrible.
No takeoff: Looks qualitatively similar to the above, except growth remains steady around 2% for at least several centuries. We remain in the economic paradigm of the Industrial Revolution, and AI makes an economic contribution similar to that of electricity or oil without launching us into a new period of human history.
This seems entirely possible, maybe even the most likely outcome. I’ve been surrounded by people talking about short timelines from a pretty young age so I never really thought about this possibility, but “takeoff” is not guaranteed. The world in 500 years could resemble the world today; in fact, I’d guess most thoughtful people don’t think much about transformative AI and would assume that this is the default scenario.
Part of why I think this is entirely plausible is because I don’t see many independently strong arguments for short AI timelines:
IMO the strongest argument for short timelines is that, within the next few decades, we’ll cross the threshold for using more compute than the human brain. If this turns out to be a significant threshold and a fair milestone to anchor against, then we could hit an inflection point and rapidly see Bostrom Superintelligence-type scenarios.
I see this belief as closely associated with the entire first scenario described above: Held by OpenAI/DeepMind, the idea that we will “solve intelligence” with an agenty AI running a simple fundamental algorithm with massive compute and effectively generalizing across many domains.
IIRC, the most prominent early argument for short AI timelines, as discussed by Bostrom, Yudkowsky, and others, was recursive self-improvement. The AI will build smarter AIs, meaning we’ll eventually hit an inflection point of runaway improvement positively feeding into itself and rapidly escalating from near-human to lightyears-beyond-human intelligence. This argument seems less popular in recent years, though I couldn’t say exactly why. My only opinion would be that this seems more like an argument for “fast takeoff” (once we have near-human level AI systems for building AI systems, we’ll quickly achieve superhuman performance in that area), but does not tell you when that takeoff will occur. For all we know, this fast takeoff could happen in hundreds of years. (Or I could be misunderstanding the argument here, I’d like to think more about it.)
Surveys asking AI researchers when they expect superhuman AI have received lots of popular coverage and might be driving widespread acceptance of short timelines. My very subjective and underinformed intution puts little weight on these surveys compared to the object level arguments. The fact that people trying to build superintelligence believe it’s possible within their lifetime certainly makes me take that possibility seriously, but it doesn’t provide much of an upper bound on how long it might take. If the current consensus of AI researchers proves to be wrong about progress over the next century, I wouldn’t expect their beliefs about the next five or ten centuries to hold up—the worldview assumptions might just be entirely off-base.
These are the only three arguments for short timelines I’ve ever heard and remembered. Interested if I’m forgetting anything big here.
Compare this to the simple prior that history will continue with slow and steady single-digit growth as it has since the Industrial Revolution, and I see a significant chance that we don’t see AI takeoff for centuries, if ever. (That’s before considering object level arguments for longer timelines, which admittedly I don’t see many of, and therefore I don’t put much weight on.)
I haven’t fully thought through all of this, but would love to hear others thoughts on the probability of “no takeoff”.
Maybe the future of AI looks like this guy on the internet’s business slide deck: https://static1.squarespace.com/static/50363cf324ac8e905e7df861/t/5e45cbd35750af6b4e60ab0f/1581632599540/2017+Benedict+Evans+Ten+Year+Futures.pdf
This is pretty rough around the edges, but these three scenarios seem like the key possibilities for the next few centuries that I can see at this point. For the hell of it, I’ll give some very weak credences: 10% that we solve superintelligence within decades, 25% that CAIS brings double-digit growth within a century or so, maybe 50% that human progress continues as usual for at least a few centuries, and (at least) 15% that what ends up happening looks nothing like any of these scenarios.
Very interested in hearing any critiques or reactions to these scenarios or the specific arguments within.
I like the no takeoff scenario intuitive analysis, and find that I also haven’t really imagined this as a concrete possibility. Generally, I like that you have presented clearly distinct scenarios and that the logic is explicit and coherent. Two thoughts that came to mind:
Somehow in the CAIS scenario, I also expect the rapid growth and the delegation of some economic and organizational work to AI to have some weird risks that involve something like humanity getting pushed away from the economic ecosystem while many autonomous systems are self-sustaining and stuck in a stupid and lifeless revenue-maximizing loop. I couldn’t really pinpoint an x-risk scenario here.
Recursive self-improvement can also happen within long periods of time, not necessarily leading to a fast takeoff, especially if the early gains are much easier than later gains (which might make more sense if we think of AI capability development as resulting mostly from computational improvements rather than algorithmic).
Ah! Richard Ngo had just written something related to the CAIS scenario :)