Hey Evan, these are definitely stronger points against short timelines if you believe in slow takeoff, rather than points against short-timelines in a hard takeoff world. It might come as no surprise that I think slow takeoff is much more likely than hard takeoff, with the Comprehensive AI Systems model best representing what I would expect. A short list of the key arguments there:
Discontinuities on important metrics are rare, see the AI Impacts writeup. EDIT: Dan Hendrycks and Thomas Woodside provide a great empirical survey of AI progress across several domains. It largely shows continuous progress on individual metrics, but also highlights the possibilities of emergent capabilities and discontinuity.
Much of the case for fast takeoff relies heavily on the concept of “general intelligence”. I think the history of AI progress shows that narrow progress is much more common, and I don’t expect advances in e.g. language and vision models to generalize to success in the many low-data domains required to achieve transformative AI.
Recursive self-improvement is entirely possible in theory, but far from current capabilities. AI is not currently being used to write research papers or build new models, nor is it significantly contributing to the acceleration of hardware progress. (The two most important counterexamples are OpenAI’s Codex and Google’s DL for chip placement. If these were shown to be significantly pushing the cutting edge of AI progress, I would change my views on the likelihood of recursive self-improvement in a short-timelines scenario.)
EDIT 07/2022: Here is Thomas Woodside’s list of examples of AI increasing AI progress. While it’s debatable how much of an impact these are having on the pace of progress, it’s undeniable that it’s happening to some degree and efforts are ongoing to increase capacity for recursive self-improvement. My summary above was an overstatement.
I don’t think there’s any meaningful “regulatory overhang”. I haven’t seen any good examples of industries where powerful AI systems are achieved in academic settings, but not deployed for legal reasons. Self-driving cars, maybe? But those seem like more of a regulatory success story than a failure, with most caution self-imposed by companies.
The short timelines scenarios I find most plausible are akin to those outlined by Gwern and Daniel Kokotajlo (also here), where a pretrained language model is given an RL objective function and the capacity to operate a computer, and it turns out that one smart person behind a computer can do a lot more damage than we realized. More generally, short timelines and hard takeoff can happen when continuous scaling up of inputs results in discontinuous performance on important real world objectives. But I don’t see the argument for where that discontinuity will arise—there are too many domains where a language model trained with no real world goal will be helpless.
And yeah, that paper is really cool, but is really only a proof of concept of what would have to become a superhuman science in order for our “Clippy” to take over the world. You’re pointing towards the future, but how long until it arrives?
Hey Evan, these are definitely stronger points against short timelines if you believe in slow takeoff, rather than points against short-timelines in a hard takeoff world. It might come as no surprise that I think slow takeoff is much more likely than hard takeoff, with the Comprehensive AI Systems model best representing what I would expect. A short list of the key arguments there:
Discontinuities on important metrics are rare, see the AI Impacts writeup. EDIT: Dan Hendrycks and Thomas Woodside provide a great empirical survey of AI progress across several domains. It largely shows continuous progress on individual metrics, but also highlights the possibilities of emergent capabilities and discontinuity.
Much of the case for fast takeoff relies heavily on the concept of “general intelligence”. I think the history of AI progress shows that narrow progress is much more common, and I don’t expect advances in e.g. language and vision models to generalize to success in the many low-data domains required to achieve transformative AI.
Recursive self-improvement is entirely possible in theory, but far from current capabilities. AI is not currently being used to write research papers or build new models, nor is it significantly contributing to the acceleration of hardware progress. (The two most important counterexamples are OpenAI’s Codex and Google’s DL for chip placement. If these were shown to be significantly pushing the cutting edge of AI progress, I would change my views on the likelihood of recursive self-improvement in a short-timelines scenario.)
EDIT 07/2022: Here is Thomas Woodside’s list of examples of AI increasing AI progress. While it’s debatable how much of an impact these are having on the pace of progress, it’s undeniable that it’s happening to some degree and efforts are ongoing to increase capacity for recursive self-improvement. My summary above was an overstatement.
I don’t think there’s any meaningful “regulatory overhang”. I haven’t seen any good examples of industries where powerful AI systems are achieved in academic settings, but not deployed for legal reasons. Self-driving cars, maybe? But those seem like more of a regulatory success story than a failure, with most caution self-imposed by companies.
The short timelines scenarios I find most plausible are akin to those outlined by Gwern and Daniel Kokotajlo (also here), where a pretrained language model is given an RL objective function and the capacity to operate a computer, and it turns out that one smart person behind a computer can do a lot more damage than we realized. More generally, short timelines and hard takeoff can happen when continuous scaling up of inputs results in discontinuous performance on important real world objectives. But I don’t see the argument for where that discontinuity will arise—there are too many domains where a language model trained with no real world goal will be helpless.
And yeah, that paper is really cool, but is really only a proof of concept of what would have to become a superhuman science in order for our “Clippy” to take over the world. You’re pointing towards the future, but how long until it arrives?