I think you make some good points about the assumption an AGI will be a goal-directed agent, but I wouldn’t be so certain that this makes Doom Scenarios less probable, only opens new doors that currently aren’t being researched enough.
In terms of AGI that are just beyond human-level not being much of a threat, I think there are a lot of key assumptions that misunderstand the radical scope of change this would cause. One is speed. Such an intelligence would probably be several orders of magnitude faster than any human intelligence. A second is the ability to replicate. Such a breakthrough would spark radical economic incentive to ramp up computational ability. Even if the first one takes a huge amount of space, the radical amount of investment to scale it I think would quickly change this in a matter of a few years. This would enable a vast number of copies of the AGI to be created. The third is coherence. These AGI copies could all work together in a far more coherent way than any corporation. Corporations are not unified entities. There is a huge amount of disorder within each, and the key decisions are still normally made by just a few individuals, radically slowing progress they can make in terms of company-wide direction and planning. The fourth change that seems very likely is the one that you credited humanity’s power for: communication. These copies could share and communicate with each other with extremely high bandwidth. Humans have to talk, write, read, and listen to share information. This is very low-bandwidth. AGIs can just share their weights with each other. Imagine if every person working on the Manhattan Project had access to all of Von Neumann’s insights, skills, and knowledge. And Einstein’s. And that of the most experienced mechanical engineers, chemists, etc. How long do you think it would have taken them to develop the atom bomb? And given this large scale of new mental power, I don’t see why no one would try to tweak it so that the AGIs start working on self-optimization. The massive incentive for outcompeting other AGIs and mere humans seems far, far too high for this not to be attempted, and I don’t see any reason why this would somehow be impossible or even extremely difficult once you have already created AGIs. Most of the progress in current capabilities of AI have come from a few basic insights from a small number of individuals. In the scope of all of humanity’s available mental power, this was unbelievably low-hanging fruit. If anything, creating more efficient and effective copies seems too easy for an AGI to do. I suspect that this will be achievable before we create AGIs that can even do everything a human can do. In other words, I expect we’ll cross into the danger/weird zone of AI before we even realize it.
I think you make some good points about the assumption an AGI will be a goal-directed agent, but I wouldn’t be so certain that this makes Doom Scenarios less probable, only opens new doors that currently aren’t being researched enough.
In terms of AGI that are just beyond human-level not being much of a threat, I think there are a lot of key assumptions that misunderstand the radical scope of change this would cause. One is speed. Such an intelligence would probably be several orders of magnitude faster than any human intelligence. A second is the ability to replicate. Such a breakthrough would spark radical economic incentive to ramp up computational ability. Even if the first one takes a huge amount of space, the radical amount of investment to scale it I think would quickly change this in a matter of a few years. This would enable a vast number of copies of the AGI to be created. The third is coherence. These AGI copies could all work together in a far more coherent way than any corporation. Corporations are not unified entities. There is a huge amount of disorder within each, and the key decisions are still normally made by just a few individuals, radically slowing progress they can make in terms of company-wide direction and planning. The fourth change that seems very likely is the one that you credited humanity’s power for: communication. These copies could share and communicate with each other with extremely high bandwidth. Humans have to talk, write, read, and listen to share information. This is very low-bandwidth. AGIs can just share their weights with each other. Imagine if every person working on the Manhattan Project had access to all of Von Neumann’s insights, skills, and knowledge. And Einstein’s. And that of the most experienced mechanical engineers, chemists, etc. How long do you think it would have taken them to develop the atom bomb? And given this large scale of new mental power, I don’t see why no one would try to tweak it so that the AGIs start working on self-optimization. The massive incentive for outcompeting other AGIs and mere humans seems far, far too high for this not to be attempted, and I don’t see any reason why this would somehow be impossible or even extremely difficult once you have already created AGIs. Most of the progress in current capabilities of AI have come from a few basic insights from a small number of individuals. In the scope of all of humanity’s available mental power, this was unbelievably low-hanging fruit. If anything, creating more efficient and effective copies seems too easy for an AGI to do. I suspect that this will be achievable before we create AGIs that can even do everything a human can do. In other words, I expect we’ll cross into the danger/weird zone of AI before we even realize it.