It seems that even in a relatively slow takeoff, you wouldn’t need that big of a discontinuity to result in a singleton AI scenario. If the first AGI that’s significantly more generally intelligent than a human is created in a world where lots of powerful narrow AIs exist, wouldn’t having a super smart thing at the center of control of a bunch of narrow AI tools plausibly be way more powerful than having human brains at the center of that control?
It seems plausible that in a “smooth” scenario the time between when the first group created AGI and the second group creating an equally powerful one could be months apart. Do you think a months-long discontinuity is not enough for an AGI to pull sufficiently ahead?
I would say that, in a scenario with relatively “smooth” progress, there’s not really a clean distinction between “narrow” AI systems and “general” AI systems; the line between “we have AGI” and “we don’t have AGI” is either a bit blurry or a bit arbitarily drawn. Even if the management/control of large collections of AI systems is eventually automated, I would also expect this process of automation to unfold over time rather than happening in single go.
In general, the smoother things are, the harder it is to tell a story where one group gets out way ahead of others. Although I’m unsure just how “unsmooth” things need to be for this outcome to be plausible.
Even if multiple groups create AGIs within a short time, isn’t having a bunch of unaligned AGIs all trying to get power at the same time also an existential risk? It doesn’t seem clear that they’d automatically keep each other in check. One might simply be better at growing or better at sabotaging other AIs. Or if they reach a stalemate they might start cooperating with each other to achieve unaligned goals as a compromise.
I think that if there were multiple AGI or AGI-ish systems in the world, and most of them were badly misaligned (e.g. willing to cause human extinction for instrumental reasons), this would present an existential risk. I wouldn’t count on them balancing each other out, in the same way that endangered gorilla populations shouldn’t count on warring communities to balance each other out.
I think the main benefits of smoothness have to do with risk awareness (e.g. by observing less catastrophic mishaps) and, especially, with opportunities for trial-and-error learning. At least when the concern is misalignment risk, I don’t think of the decentralization of power as a really major benefit in its own right: the systems in this decentralized world still mostly need to be safe.
My model is: if you have a central control unit (a human brain, or group of human brains) who is deciding how to use a bunch of narrow AIs, then if you replace that central control unit with one that it more intelligent / fast acting, the whole system will be more effective.
The only way I can think of where that wouldn’t be true would be if the general AI required so many computational resources that the narrow AIs that were acting as tools of the AGI were crippled by lack of resources. Is that what you’re imagining?
I think it’s plausible that especially general systems would be especially useful for managing the development, deployment, and interaction of other AI systems. I’m not totally sure this is the case, though. For example, at least in principle, I can imagine an AI system that is good at managing the training of other AI systems—e.g. deciding how much compute to devote to different ongoing training processes—but otherwise can’t do much else.
I would say that, in a scenario with relatively “smooth” progress, there’s not really a clean distinction between “narrow” AI systems and “general” AI systems; the line between “we have AGI” and “we don’t have AGI” is either a bit blurry or a bit arbitarily drawn. Even if the management/control of large collections of AI systems is eventually automated, I would also expect this process of automation to unfold over time rather than happening in single go.
In general, the smoother things are, the harder it is to tell a story where one group gets out way ahead of others. Although I’m unsure just how “unsmooth” things need to be for this outcome to be plausible.
I think that if there were multiple AGI or AGI-ish systems in the world, and most of them were badly misaligned (e.g. willing to cause human extinction for instrumental reasons), this would present an existential risk. I wouldn’t count on them balancing each other out, in the same way that endangered gorilla populations shouldn’t count on warring communities to balance each other out.
I think the main benefits of smoothness have to do with risk awareness (e.g. by observing less catastrophic mishaps) and, especially, with opportunities for trial-and-error learning. At least when the concern is misalignment risk, I don’t think of the decentralization of power as a really major benefit in its own right: the systems in this decentralized world still mostly need to be safe.
I think it’s plausible that especially general systems would be especially useful for managing the development, deployment, and interaction of other AI systems. I’m not totally sure this is the case, though. For example, at least in principle, I can imagine an AI system that is good at managing the training of other AI systems—e.g. deciding how much compute to devote to different ongoing training processes—but otherwise can’t do much else.