I strongly disagree with the claim that there is a >10% chance of TAI in the next 10 years. Here are two small but meaningful pieces of why I have much longer AI timelines.
Note that TAI is here defined as one or both of: (a) any 5 year doubling of real global GDP, or (b) any catastrophic or existential AI failures.
Market Activity
Top tech companies do not believe that AI takeoff is around the corner. Mark Zuckerberg recently saw many of his top AI research scientists leave the company, as Facebook has chosen to acquire Oculus and bet on the metaverse rather than AI as the next big thing. This 2019 interview with Facebook’s VP of AI might shed some light on why.
IBM Watson just sold off their entire healthcare business. This is a strong sign of the AI’s failure to meet tremendous expectations of revolutionizing the healthcare industry. Meanwhile on LessWrong, somebody is getting lots of upvotes for predicting (in admittedly a fun, off-the-cuff manner) that “Chatbots [will be] able to provide better medical diagnoses than nearly all doctors” in 2024.
Data Constraints
Progress has been swift in areas where it is easy to generate lots of training data. ML systems are lauded for achieving human-level performance on academic competitions like ImageNet, but those performances are only possible because of the millions of labeled data points provided. NLP systems trained on self-supervised objectives leverage massive datasets, but regurgitate hate speech, fake news, and private information memorized from the internet. Reinforcement learning (RL) systems play games like chess and Atari for thousands of years of virtual time in the popular method of self play.
Many real world goals have much longer time horizons than those where AI succeeds today, and cannot be readily decomposed into smaller goals. We cannot simulate the experience of founding a startup, running an experiment, or building a relationship in the same way we can do with writing a paper or playing a game. Machines will need to learn in open-ended play with the world, where today they mostly learn from labeled examples.
What I mean is that most of these examples show a failure of narrow AIs to deliver on some economic goals. In soft takeoff, we expect to see things like broad deployment of AIs contributing to massive economic gains and GDP doublings in short periods of time well before we get to anything like AGI.
But in hard takeoff, failure to see massive success from narrow AIs could happen due to regulations and other barriers (or it could just be limitations of the narrow AI). In fact, these limitations could even point more forcefully to the massive benefits of an AI that can generalize. And having the recipe for that AGI discovered and deployed in a lab doesn’t depend on the success of prior narrow AIs in the regulated marketplace. AGI is a different breed and may also become powerful enough that it doesn’t have to play by the rules of the regulated marketplace and national legal systems.
Machines will need to learn in open-ended play with the world, where today they mostly learn from labeled examples.
Have you seen DeepMind’s Generally capable agents emerge from open-ended play? I think it is a powerful demonstration of learning from open-ended play actually working in a lab (not just a possible future approach). Though it is still in a virtual environment rather than the real physical world.
Hey Evan, these are definitely stronger points against short timelines if you believe in slow takeoff, rather than points against short-timelines in a hard takeoff world. It might come as no surprise that I think slow takeoff is much more likely than hard takeoff, with the Comprehensive AI Systems model best representing what I would expect. A short list of the key arguments there:
Discontinuities on important metrics are rare, see the AI Impacts writeup. EDIT: Dan Hendrycks and Thomas Woodside provide a great empirical survey of AI progress across several domains. It largely shows continuous progress on individual metrics, but also highlights the possibilities of emergent capabilities and discontinuity.
Much of the case for fast takeoff relies heavily on the concept of “general intelligence”. I think the history of AI progress shows that narrow progress is much more common, and I don’t expect advances in e.g. language and vision models to generalize to success in the many low-data domains required to achieve transformative AI.
Recursive self-improvement is entirely possible in theory, but far from current capabilities. AI is not currently being used to write research papers or build new models, nor is it significantly contributing to the acceleration of hardware progress. (The two most important counterexamples are OpenAI’s Codex and Google’s DL for chip placement. If these were shown to be significantly pushing the cutting edge of AI progress, I would change my views on the likelihood of recursive self-improvement in a short-timelines scenario.)
EDIT 07/2022: Here is Thomas Woodside’s list of examples of AI increasing AI progress. While it’s debatable how much of an impact these are having on the pace of progress, it’s undeniable that it’s happening to some degree and efforts are ongoing to increase capacity for recursive self-improvement. My summary above was an overstatement.
I don’t think there’s any meaningful “regulatory overhang”. I haven’t seen any good examples of industries where powerful AI systems are achieved in academic settings, but not deployed for legal reasons. Self-driving cars, maybe? But those seem like more of a regulatory success story than a failure, with most caution self-imposed by companies.
The short timelines scenarios I find most plausible are akin to those outlined by Gwern and Daniel Kokotajlo (also here), where a pretrained language model is given an RL objective function and the capacity to operate a computer, and it turns out that one smart person behind a computer can do a lot more damage than we realized. More generally, short timelines and hard takeoff can happen when continuous scaling up of inputs results in discontinuous performance on important real world objectives. But I don’t see the argument for where that discontinuity will arise—there are too many domains where a language model trained with no real world goal will be helpless.
And yeah, that paper is really cool, but is really only a proof of concept of what would have to become a superhuman science in order for our “Clippy” to take over the world. You’re pointing towards the future, but how long until it arrives?
But in hard takeoff, failure to see massive success from narrow AIs could happen due to regulations and other barriers. It could just be limitations of the narrow AIs. In fact, these limitations could even point more forcefully to the massive benefits of an AI that can generalize.
I think you’re saying that regulations/norms could mask dangerous capability and development, having the effect of eroding credibility/recourses in safety. Yet, unhindered by enforcement, bad actors continue to progress to the worse states, even using the regulations as signposts.
I’m not fully sure I understand all of the sentences in the rest of your paragraph. There’s several jumps in there?
Gwern’s writing “Clippy” lays out some potential possibilities of dislocation of safety mechanisms. If there is additional content you think is convincing (of mechanisms and enforcement) that would be good to share.
I strongly disagree with the claim that there is a >10% chance of TAI in the next 10 years. Here are two small but meaningful pieces of why I have much longer AI timelines.
Note that TAI is here defined as one or both of: (a) any 5 year doubling of real global GDP, or (b) any catastrophic or existential AI failures.
Market Activity
Top tech companies do not believe that AI takeoff is around the corner. Mark Zuckerberg recently saw many of his top AI research scientists leave the company, as Facebook has chosen to acquire Oculus and bet on the metaverse rather than AI as the next big thing. This 2019 interview with Facebook’s VP of AI might shed some light on why.
Microsoft has similarly bet heavily on entertainment over AI with their acquisition of Activision Blizzard. Microsoft purchased Activision for $68B, which is 68 times more than they invested into OpenAI three years ago, after which they have not followed up with more public investments.
IBM Watson just sold off their entire healthcare business. This is a strong sign of the AI’s failure to meet tremendous expectations of revolutionizing the healthcare industry. Meanwhile on LessWrong, somebody is getting lots of upvotes for predicting (in admittedly a fun, off-the-cuff manner) that “Chatbots [will be] able to provide better medical diagnoses than nearly all doctors” in 2024.
Data Constraints
Progress has been swift in areas where it is easy to generate lots of training data. ML systems are lauded for achieving human-level performance on academic competitions like ImageNet, but those performances are only possible because of the millions of labeled data points provided. NLP systems trained on self-supervised objectives leverage massive datasets, but regurgitate hate speech, fake news, and private information memorized from the internet. Reinforcement learning (RL) systems play games like chess and Atari for thousands of years of virtual time in the popular method of self play.
Many real world goals have much longer time horizons than those where AI succeeds today, and cannot be readily decomposed into smaller goals. We cannot simulate the experience of founding a startup, running an experiment, or building a relationship in the same way we can do with writing a paper or playing a game. Machines will need to learn in open-ended play with the world, where today they mostly learn from labeled examples.
See Andrew Ng on the incredible challenge of data sparse domains. Perhaps this is why radiologists have not been replaced by machines, as Geoffrey Hinton so confidently predicted back in 2016.
These are thoughtful data points, but consider that they may just be good evidence for hard takeoff rather than soft takeoff.
What I mean is that most of these examples show a failure of narrow AIs to deliver on some economic goals. In soft takeoff, we expect to see things like broad deployment of AIs contributing to massive economic gains and GDP doublings in short periods of time well before we get to anything like AGI.
But in hard takeoff, failure to see massive success from narrow AIs could happen due to regulations and other barriers (or it could just be limitations of the narrow AI). In fact, these limitations could even point more forcefully to the massive benefits of an AI that can generalize. And having the recipe for that AGI discovered and deployed in a lab doesn’t depend on the success of prior narrow AIs in the regulated marketplace. AGI is a different breed and may also become powerful enough that it doesn’t have to play by the rules of the regulated marketplace and national legal systems.
Have you seen DeepMind’s Generally capable agents emerge from open-ended play? I think it is a powerful demonstration of learning from open-ended play actually working in a lab (not just a possible future approach). Though it is still in a virtual environment rather than the real physical world.
Hey Evan, these are definitely stronger points against short timelines if you believe in slow takeoff, rather than points against short-timelines in a hard takeoff world. It might come as no surprise that I think slow takeoff is much more likely than hard takeoff, with the Comprehensive AI Systems model best representing what I would expect. A short list of the key arguments there:
Discontinuities on important metrics are rare, see the AI Impacts writeup. EDIT: Dan Hendrycks and Thomas Woodside provide a great empirical survey of AI progress across several domains. It largely shows continuous progress on individual metrics, but also highlights the possibilities of emergent capabilities and discontinuity.
Much of the case for fast takeoff relies heavily on the concept of “general intelligence”. I think the history of AI progress shows that narrow progress is much more common, and I don’t expect advances in e.g. language and vision models to generalize to success in the many low-data domains required to achieve transformative AI.
Recursive self-improvement is entirely possible in theory, but far from current capabilities. AI is not currently being used to write research papers or build new models, nor is it significantly contributing to the acceleration of hardware progress. (The two most important counterexamples are OpenAI’s Codex and Google’s DL for chip placement. If these were shown to be significantly pushing the cutting edge of AI progress, I would change my views on the likelihood of recursive self-improvement in a short-timelines scenario.)
EDIT 07/2022: Here is Thomas Woodside’s list of examples of AI increasing AI progress. While it’s debatable how much of an impact these are having on the pace of progress, it’s undeniable that it’s happening to some degree and efforts are ongoing to increase capacity for recursive self-improvement. My summary above was an overstatement.
I don’t think there’s any meaningful “regulatory overhang”. I haven’t seen any good examples of industries where powerful AI systems are achieved in academic settings, but not deployed for legal reasons. Self-driving cars, maybe? But those seem like more of a regulatory success story than a failure, with most caution self-imposed by companies.
The short timelines scenarios I find most plausible are akin to those outlined by Gwern and Daniel Kokotajlo (also here), where a pretrained language model is given an RL objective function and the capacity to operate a computer, and it turns out that one smart person behind a computer can do a lot more damage than we realized. More generally, short timelines and hard takeoff can happen when continuous scaling up of inputs results in discontinuous performance on important real world objectives. But I don’t see the argument for where that discontinuity will arise—there are too many domains where a language model trained with no real world goal will be helpless.
And yeah, that paper is really cool, but is really only a proof of concept of what would have to become a superhuman science in order for our “Clippy” to take over the world. You’re pointing towards the future, but how long until it arrives?
I think you’re saying that regulations/norms could mask dangerous capability and development, having the effect of eroding credibility/recourses in safety. Yet, unhindered by enforcement, bad actors continue to progress to the worse states, even using the regulations as signposts.
I’m not fully sure I understand all of the sentences in the rest of your paragraph. There’s several jumps in there?
Gwern’s writing “Clippy” lays out some potential possibilities of dislocation of safety mechanisms. If there is additional content you think is convincing (of mechanisms and enforcement) that would be good to share.
You’re right, that paragraph was confusing. I just edited it to try and make it more clear.