Thanks for this critique! I agree this is an important subject that is relatively understudied compared to other aspects of the problem. As far as I can tell there just isn’t a science of takeover; there’s military science and there’s the science of how to win elections in a democracy and there’s a bit of research and a few books on the topic of how to seize power in a dictatorship… but for such an important subject when you think about it it’s unfortunate that there isn’t a general study of how agents in multi-agent environments accumulate influence and achieve large-scale goals over long time periods.
I’m going to give my reactions below as I read:
These passages seem to imply that the rate of scientific progress is primarily limited by the number and intelligence level of those working on scientific research. It is not clear, however, that the evidence supports this.
I mean it’s clearly more than JUST the number and intelligence of the people involved, but surely those are major factors! Piece of evidence: Across many industries performance on important metrics (e.g. price) seems to predictably improve exponentially with investment/effort (this is called experience curve effect). Another piece of evidence: AlphaFold 2.
Later you mention the gradual accumulation of ideas and cite the common occurrence of repeated independent discoveries. I think this quite plausible. But note that a society of AIs would be thinking and communicating much faster than a society of humans, so the process of ideas gradually accumulating in their society would also be sped up.
Frist, though the actual model training was rapid, the entire process of developing Alpha Zero was far more protracted. Focusing on the day of training presents a highly misleading picture of the actual rate of progress of this particular example.
Sure, and similarly if AI R&D ability is like AI Go ability, there’ll be a series of better and better AIs over the course of many years that gradually get better at various aspects of R&D, until one day an AI is trained that is better than the most brilliant genius scientists. I actually expect things to be slower and more smoothed out than this, probably, because training will take more like a year. This is all part of the standard picture of AI takeover, not an objection to it.
Second, Go is a fully-observable, discrete-time, zero-sum, two-player board game.
I agree that the real world is more complex etc. and that just doing the same sort of self-play won’t work. There may be more sophsiticated forms of self-play that work though. Also you don’t need self-play to be superhuman at something, e.g. you could use decision transformers + imitation learning.
These all take time to develop and put into place, which is why the development of novel technologies takes a long time. For example, the Lockheed Martin F-35 took about fifteen years from initial design to scale production. The Gerald R. Ford aircraft carrier took about ten years to build and fit out. Semiconductor fabrication plants cost billions of dollars, and the entire process from the design of a chip to manufacturing takes years. Given such examples, it seems reasonable to expect that even a nascent AGI would require years to design and build a functioning nanofactory. Doing so in secret or without outside interference would be even more difficult given all the specialised equipment, raw materials, and human talent that would be needed. A bunch of humans hired online cannot simply construct a nanofactory from nothing in a few months, regardless of how advanced is the AGI overseeing the process.
I’d be interested to hear your thoughts on this post which details a combination of “near-future” military technologies. Perhaps you’ll agree that the technologies on this list could be built in a few months or years by a developed nation with the help of superintelligent AI? Then the crux would be whether this tech would allow that nation to take over the world. I personally think that military takeover scenarios are unlikely because there are much easier and safer methods, but I still think military takeover is at least on the table—crazier things have happened in history.
That said, I don’t concede the point—You are right that it would take modern humans many years to build nanofactories etc. but I don’t think this is strong evidence that a superintelligence would also take many years. Consider video games and speedrunning. Even if speedrunners don’t allow themselves to use bugs/exploits, they still usually go significantly faster than reasonably good players. Consider also human engineers building something that is well-understood already how to build vs. building something for the first time ever. The point is, if you are really smart and know what you are doing, you can do stuff much faster. You said that a lot of experimentation and experience is necessary—well, maybe it’s not. In general there’s a tradeoff between smarts and experimentation/experience; if you have more of one you need less of the other to reach the same level of performance. Maybe if you crank up smarts to superintelligence level—so intelligent that the best human geniuses seem a rounding error away from the average—you can get away with orders of magnitude less experimentation/experience. Not for everything perhaps, but for some things. Suppose there are N crazy sci-fi technologies that an AI could use to get a huge advantage: nanofactories, fusion, quantum shenanigans, bioengineering … All it takes if for 1 of them to be such that you can mostly substitute superintelligence for experimentation. And also you can still do experimentation, and you can do it much faster than humans do it too because you know what you are doing. Instead of toying around until hypotheses gradually coalesce in your brain, you can begin with a million carefully crafted hypotheses consistent with all the evidence you’ve seen so far and an experiment regime designed to optimally search through the space of hypotheses as fast as possible.
I expect it to take somewhere between a day and five years to go from what you might call human-level AI to nanobot swarms. Perhaps this isn’t that different from what you think? (Maybe you’d say something like 3 to 10 years?)
Relying on a ‘front man’ to serve as the face of the AGI would be highly dangerous, as the AGI would become dependent on this person for ensuring the loyalty of its followers. Of course one might argue that a combination of bribery and threats could be sufficient, but this is not the primary means by which successful leaders in history have obtained obedience and popularity, so an AGI limited to these tools would be at a significant disadvantage. Furthermore, an AGI reliant on control over money is susceptible to intervention by government authorities to freeze assets and hamper the transfer of funds. This would not be an issue if the AGI had control over its own territory, but then it would be subject to blockade and economic sanctions. For instance, it would take an AGI considerable effort to acquire the power of Vladimir Putin, and yet he is still facing considerable practical difficulties in exerting his will on his own (and neighbouring) populations without the intervention of the rest of the world. While none of these problems are necessarily insuperable, I believe they are significant issues that must be considered in an assessment of the plausibility of various AI takeover scenarios.
History has many examples of people ruling from behind the throne, so to speak. Often they have no official title whatsoever, but the people with the official titles are all loyal to them. Sometimes the people with the official titles do rebel and stop listening to the power behind the throne, and then said power behind the throne loses power. Other times, this doesn’t happen.
AGI need not rule from behind the scenes though. If it’s charismatic enough it can rule over a group of Blake Lemoines. Have you seen the movie Her? Did you find the behavior of the humans super implausible in that movie—no way they would form personal relationships with an AI, no way they would trust it?
It is also unclear how an AGI would gain the skills needed to manipulate and manage large numbers of humans in the first place. It is by no means evident why an AGI would be constructed with this capability, or how it would even be trained for this task, which does not seem very amenable to traditional reinforcement learning approaches. In many discussions, an AGI is simply defined as having such abilities, but it is not explained why such skills would be expected to accompany general problem-solving or planning skills. Even if a generally competent AGI had instrumental reasons to develop such skills, would it have the capability of doing so? Humans learn social skills through years of interaction with other humans, and even then, many otherwise intelligent and wealthy humans possess such skills only to a minimal degree. Unless a credible explanation can be given as to how such an AI would acquire such skills or they why should necessarily follow from broader capabilities, I do not think it is reasonable to simply define an AGI as possessing them, and then assuming this as part of a broader takeover narrative. This presents a major issue for takeover scenarios which rely on an AGI engaging large numbers of humans in its employment for the development of weapons or novel technologies.
It currently looks like most future AIs, and in particular AGIs, will have been trained on reading the whole internet & chatting to millions of humans over the course of several months. So, that’s how they’ll gain those skills.
(But also, if you are really good at generalizing to new tasks/situations, maybe manipulation of humans is one of the things you can generalize to. And if you aren’t really good at generalizing to new tasks/situations, maybe you don’t count as AGI.)
So far all I’ve done is critique your arguments but hopefully one day I’ll have assembled some writing laying out my own arguments on this subject.
Anyhow, thanks again for writing this! I strongly disagree with your conclusions but I’m glad to see this topic getting serious & thoughtful attention.
Thanks for this critique! I agree this is an important subject that is relatively understudied compared to other aspects of the problem. As far as I can tell there just isn’t a science of takeover; there’s military science and there’s the science of how to win elections in a democracy and there’s a bit of research and a few books on the topic of how to seize power in a dictatorship… but for such an important subject when you think about it it’s unfortunate that there isn’t a general study of how agents in multi-agent environments accumulate influence and achieve large-scale goals over long time periods.
I’m going to give my reactions below as I read:
I mean it’s clearly more than JUST the number and intelligence of the people involved, but surely those are major factors! Piece of evidence: Across many industries performance on important metrics (e.g. price) seems to predictably improve exponentially with investment/effort (this is called experience curve effect). Another piece of evidence: AlphaFold 2.
Later you mention the gradual accumulation of ideas and cite the common occurrence of repeated independent discoveries. I think this quite plausible. But note that a society of AIs would be thinking and communicating much faster than a society of humans, so the process of ideas gradually accumulating in their society would also be sped up.
Sure, and similarly if AI R&D ability is like AI Go ability, there’ll be a series of better and better AIs over the course of many years that gradually get better at various aspects of R&D, until one day an AI is trained that is better than the most brilliant genius scientists. I actually expect things to be slower and more smoothed out than this, probably, because training will take more like a year. This is all part of the standard picture of AI takeover, not an objection to it.
I agree that the real world is more complex etc. and that just doing the same sort of self-play won’t work. There may be more sophsiticated forms of self-play that work though. Also you don’t need self-play to be superhuman at something, e.g. you could use decision transformers + imitation learning.
I’d be interested to hear your thoughts on this post which details a combination of “near-future” military technologies. Perhaps you’ll agree that the technologies on this list could be built in a few months or years by a developed nation with the help of superintelligent AI? Then the crux would be whether this tech would allow that nation to take over the world. I personally think that military takeover scenarios are unlikely because there are much easier and safer methods, but I still think military takeover is at least on the table—crazier things have happened in history.
That said, I don’t concede the point—You are right that it would take modern humans many years to build nanofactories etc. but I don’t think this is strong evidence that a superintelligence would also take many years. Consider video games and speedrunning. Even if speedrunners don’t allow themselves to use bugs/exploits, they still usually go significantly faster than reasonably good players. Consider also human engineers building something that is well-understood already how to build vs. building something for the first time ever. The point is, if you are really smart and know what you are doing, you can do stuff much faster. You said that a lot of experimentation and experience is necessary—well, maybe it’s not. In general there’s a tradeoff between smarts and experimentation/experience; if you have more of one you need less of the other to reach the same level of performance. Maybe if you crank up smarts to superintelligence level—so intelligent that the best human geniuses seem a rounding error away from the average—you can get away with orders of magnitude less experimentation/experience. Not for everything perhaps, but for some things. Suppose there are N crazy sci-fi technologies that an AI could use to get a huge advantage: nanofactories, fusion, quantum shenanigans, bioengineering … All it takes if for 1 of them to be such that you can mostly substitute superintelligence for experimentation. And also you can still do experimentation, and you can do it much faster than humans do it too because you know what you are doing. Instead of toying around until hypotheses gradually coalesce in your brain, you can begin with a million carefully crafted hypotheses consistent with all the evidence you’ve seen so far and an experiment regime designed to optimally search through the space of hypotheses as fast as possible.
I expect it to take somewhere between a day and five years to go from what you might call human-level AI to nanobot swarms. Perhaps this isn’t that different from what you think? (Maybe you’d say something like 3 to 10 years?)
History has many examples of people ruling from behind the throne, so to speak. Often they have no official title whatsoever, but the people with the official titles are all loyal to them. Sometimes the people with the official titles do rebel and stop listening to the power behind the throne, and then said power behind the throne loses power. Other times, this doesn’t happen.
AGI need not rule from behind the scenes though. If it’s charismatic enough it can rule over a group of Blake Lemoines. Have you seen the movie Her? Did you find the behavior of the humans super implausible in that movie—no way they would form personal relationships with an AI, no way they would trust it?
It currently looks like most future AIs, and in particular AGIs, will have been trained on reading the whole internet & chatting to millions of humans over the course of several months. So, that’s how they’ll gain those skills.
(But also, if you are really good at generalizing to new tasks/situations, maybe manipulation of humans is one of the things you can generalize to. And if you aren’t really good at generalizing to new tasks/situations, maybe you don’t count as AGI.)
So far all I’ve done is critique your arguments but hopefully one day I’ll have assembled some writing laying out my own arguments on this subject.
Anyhow, thanks again for writing this! I strongly disagree with your conclusions but I’m glad to see this topic getting serious & thoughtful attention.