[ETA: I posted a revised version of this essay here.]
AI pause advocates often say they are pro-technology and pro-economic growth, and that they simply make one exception for AI because of its unique risks. But this reasoning will grow less credible over time as AI comes to account for a larger and larger share of economic growth.
Simple growth models predict that AI capable of substituting for human labor will raise economic growth rates by an order of magnitude or more. If that’s right, then AI will eventually be driving the vast majority of technological innovation and improvements in the standard of living. Stopping AI really would be like halting technology itself, because you would be shutting off the source of nearly all growth.
This suggests that proposing to pause AI today is like proposing to pause electricity in 1880: yes, electricity is technically just one technology among many, but pausing it would threaten to shut down progress on most of the others.
I also question the premise that AI is unique in its risks. Pause advocates argue that, apart from perhaps nuclear weapons, AI is the first technology to threaten the survival of the human species. But the boundary around “human species” is arbitrary. It only fails to feel that way because, for us today, the human species seems synonymous with the whole world. Replacing us feels like ending the world.
Yet a hunter-gatherer tribe might just as easily feel the same way about themselves and their way of life. To them, the development of agriculture would feel like an existential risk. It would, from their point of view, be a threat to everything that matters.
In reality, the world is much larger than hunter-gatherer tribes or even the human species. By developing AI, we are bringing into existence a new class of sapient beings, ones who will inhabit the world alongside us. I personally predict we will coexist with them peacefully, and I welcome efforts to make that outcome more likely. It would certainly be a terrible tragedy if the AIs create turn violent against us. But peaceful or not, the final outcome matters for them too. We are not the only people in history.
In the future, the vast majority of interesting and valuable events will likely occur between digital people, not between the more limited biological ones. The vast majority of relationships, discoveries, adventures, acts of kindness, and feelings of joy will take place within an artificial world, one to which the label “human” may no longer cleanly apply.
In such a world, insisting that the human species represents everything that matters will be like insisting that hunter-gatherers represent the whole world. That may have felt like a reasonable claim 12,000 years ago, but today it would sound silly.
Whether we like it or not, technology has always posed massive risks to “the world”. AI is not the first technology to do this, and it will likely not be the last. The only difference is that this time, technology threatens the world that people alive today grew up in. Just as our ancestors experienced before us, we face the prospect of losing the world we know in exchange for material progress and prosperity. I am happy to take that trade, just as I am glad my ancestors took it in theirs.
Just as our ancestors experienced before us, we face the prospect of losing the world we know in exchange for material progress and prosperity.
Your ancestors who adopted agriculture did so because they thought that they and their children would get to eat the bread, not that they were sowing the seeds of their own destruction. If they had known that planting crops would lead to invasion and replacement they likely would not have done it. This rather large dis-analogy makes me think your use of the word ‘just’ is a bit of a stretch here.
Our ancestors had less insight into the trade they were making than we do about our own situation. That’s true.
Yet they still made the trade, and in hindsight, was it a bad trade to make? I disagree with people like Jared Diamond who argue that the agricultural revolution was the “worst mistake in the history of the human race”. It certainly had some very negative consequences. But like most people, I think the agricultural revolution was still a good thing overall, despite the fact that it carried enormous negative side effects.
I suspect the transition to AI will be less calamitous and more peaceful than our transition to agriculture. In my view, this means our trade is even easier to make. Yet, I still recognize that we face similar tradeoffs. We risk losing our way of life. There is also a credible risk (even if I think it’s small), that the entire human species will go extinct. That would be very bad, but as I argued in the post, it would not be the same as losing all value in the universe.
Our ancestors did not make this trade at all for the most part. Mostly they stayed hunter gathers, until the people who adopted farming out populated them and then expanded and killed/outcompeted them. (technically, I guess “our” ancestors are the ones who adopted the agriculture)
Likewise, the vast majority of humanity is not directly developing AI. Therefore, in an important sense, “we” are not making the trade of whether to develop AI; only a small number of people are.
If you and me and all of humanity gets killed by AI and turned into paperclips, that would be an unprecedented moral catastrophe. If the AIs that killed all of us stay around and enjoy having more paperclips, that is still extremely bad. The very act of killing us makes these AIs not a worthy successor of the human species.
This suggests that proposing to pause AI today is like proposing to pause electricity in 1880
The prospect of AI killing all of us makes these very different. Yes, in both cases a pause will probably slow GDP growth. But humans should be willing to accept lower GDP if this notably reduces the chance of all humans being killed.
[ETA: I posted a revised version of this essay here.]
AI pause advocates often say they are pro-technology and pro-economic growth, and that they simply make one exception for AI because of its unique risks. But this reasoning will grow less credible over time as AI comes to account for a larger and larger share of economic growth.
Simple growth models predict that AI capable of substituting for human labor will raise economic growth rates by an order of magnitude or more. If that’s right, then AI will eventually be driving the vast majority of technological innovation and improvements in the standard of living. Stopping AI really would be like halting technology itself, because you would be shutting off the source of nearly all growth.
This suggests that proposing to pause AI today is like proposing to pause electricity in 1880: yes, electricity is technically just one technology among many, but pausing it would threaten to shut down progress on most of the others.
I also question the premise that AI is unique in its risks. Pause advocates argue that, apart from perhaps nuclear weapons, AI is the first technology to threaten the survival of the human species. But the boundary around “human species” is arbitrary. It only fails to feel that way because, for us today, the human species seems synonymous with the whole world. Replacing us feels like ending the world.
Yet a hunter-gatherer tribe might just as easily feel the same way about themselves and their way of life. To them, the development of agriculture would feel like an existential risk. It would, from their point of view, be a threat to everything that matters.
In reality, the world is much larger than hunter-gatherer tribes or even the human species. By developing AI, we are bringing into existence a new class of sapient beings, ones who will inhabit the world alongside us. I personally predict we will coexist with them peacefully, and I welcome efforts to make that outcome more likely. It would certainly be a terrible tragedy if the AIs create turn violent against us. But peaceful or not, the final outcome matters for them too. We are not the only people in history.
In the future, the vast majority of interesting and valuable events will likely occur between digital people, not between the more limited biological ones. The vast majority of relationships, discoveries, adventures, acts of kindness, and feelings of joy will take place within an artificial world, one to which the label “human” may no longer cleanly apply.
In such a world, insisting that the human species represents everything that matters will be like insisting that hunter-gatherers represent the whole world. That may have felt like a reasonable claim 12,000 years ago, but today it would sound silly.
Whether we like it or not, technology has always posed massive risks to “the world”. AI is not the first technology to do this, and it will likely not be the last. The only difference is that this time, technology threatens the world that people alive today grew up in. Just as our ancestors experienced before us, we face the prospect of losing the world we know in exchange for material progress and prosperity. I am happy to take that trade, just as I am glad my ancestors took it in theirs.
Your ancestors who adopted agriculture did so because they thought that they and their children would get to eat the bread, not that they were sowing the seeds of their own destruction. If they had known that planting crops would lead to invasion and replacement they likely would not have done it. This rather large dis-analogy makes me think your use of the word ‘just’ is a bit of a stretch here.
Our ancestors had less insight into the trade they were making than we do about our own situation. That’s true.
Yet they still made the trade, and in hindsight, was it a bad trade to make? I disagree with people like Jared Diamond who argue that the agricultural revolution was the “worst mistake in the history of the human race”. It certainly had some very negative consequences. But like most people, I think the agricultural revolution was still a good thing overall, despite the fact that it carried enormous negative side effects.
I suspect the transition to AI will be less calamitous and more peaceful than our transition to agriculture. In my view, this means our trade is even easier to make. Yet, I still recognize that we face similar tradeoffs. We risk losing our way of life. There is also a credible risk (even if I think it’s small), that the entire human species will go extinct. That would be very bad, but as I argued in the post, it would not be the same as losing all value in the universe.
Our ancestors did not make this trade at all for the most part. Mostly they stayed hunter gathers, until the people who adopted farming out populated them and then expanded and killed/outcompeted them. (technically, I guess “our” ancestors are the ones who adopted the agriculture)
Likewise, the vast majority of humanity is not directly developing AI. Therefore, in an important sense, “we” are not making the trade of whether to develop AI; only a small number of people are.
If you and me and all of humanity gets killed by AI and turned into paperclips, that would be an unprecedented moral catastrophe. If the AIs that killed all of us stay around and enjoy having more paperclips, that is still extremely bad. The very act of killing us makes these AIs not a worthy successor of the human species.
The prospect of AI killing all of us makes these very different. Yes, in both cases a pause will probably slow GDP growth. But humans should be willing to accept lower GDP if this notably reduces the chance of all humans being killed.