You don’t need campaigners if AGI will be a better campaigner than you. You don’t need policy expertise if AGI will know more about policy than you. This passage treats AGI as a machine that accelerates scientific R&D, but that’s not what AGI is. AGI is intelligence.
I think you’re conflating “Transformative AI” with “Artificial General Intelligence”. It seems very possible (though perhaps not very probable) that progress could slow down and preserve existing jaggedness: one can easily imagine a scenario in which increasingly capable AI replaces all coders (and is basically ubiquitous as an assistant in math research) but can’t manage to replace other types of knowledge work due to lack of generality. Maybe it never gets good enough to replace boots on the ground investigative journalism, or maybe robotics doesn’t advance fast enough for AI to automate wet-lab work, or maybe medicine and personal care will always require a human touch, yadda yadda.
But I haven’t seen that many people grapple with the world of difference between the theoretical and observed AI coverage markers—not to mention the fact that usage doesn’t mean replacement. It’s possible that this relationship will not hold in the future, but it’s also possible that Moravec’s Paradox will hold for the next couple of decades, and that computers and humans will continue to have distinct comparative advantages (or perhaps even complementary ones).
I technically agree with your point about there only being two possible futures. But I only think so because your first future covers far too many possible outcomes, including some in which AI is “transformative” but not necessarily superintelligent (or even “generally” intelligent).
Jagged progress is conceivable, but it’s virtually impossible that AI could replace all coders and accelerate math research but not replace other jobs, because coding and math research (specifically ML-type math) are exactly the skills needed to accelerate AI development. If AI can accelerate AI development, then the timeline to getting an AI that can replace humans on all tasks becomes much shorter.
This is true if we assume that AI development has no cap. What if we find out that human intelligence is beyond AI? Maybe acceleration just leads to us hitting the “ceiling” sooner. (Again, not making a probability judgment here, just pointing out that this is a plausible outcome.)
I think you’re conflating “Transformative AI” with “Artificial General Intelligence”. It seems very possible (though perhaps not very probable) that progress could slow down and preserve existing jaggedness: one can easily imagine a scenario in which increasingly capable AI replaces all coders (and is basically ubiquitous as an assistant in math research) but can’t manage to replace other types of knowledge work due to lack of generality. Maybe it never gets good enough to replace boots on the ground investigative journalism, or maybe robotics doesn’t advance fast enough for AI to automate wet-lab work, or maybe medicine and personal care will always require a human touch, yadda yadda.
I’ve seen many people throw around the Anthropic labor market graph (as pictured below):
But I haven’t seen that many people grapple with the world of difference between the theoretical and observed AI coverage markers—not to mention the fact that usage doesn’t mean replacement. It’s possible that this relationship will not hold in the future, but it’s also possible that Moravec’s Paradox will hold for the next couple of decades, and that computers and humans will continue to have distinct comparative advantages (or perhaps even complementary ones).
I technically agree with your point about there only being two possible futures. But I only think so because your first future covers far too many possible outcomes, including some in which AI is “transformative” but not necessarily superintelligent (or even “generally” intelligent).
Jagged progress is conceivable, but it’s virtually impossible that AI could replace all coders and accelerate math research but not replace other jobs, because coding and math research (specifically ML-type math) are exactly the skills needed to accelerate AI development. If AI can accelerate AI development, then the timeline to getting an AI that can replace humans on all tasks becomes much shorter.
This is true if we assume that AI development has no cap. What if we find out that human intelligence is beyond AI? Maybe acceleration just leads to us hitting the “ceiling” sooner. (Again, not making a probability judgment here, just pointing out that this is a plausible outcome.)