Bostrom may have talked about this elsewhere since I’ve heard other people say this, but he doesn’t make this point in the paper. He only mentions AI briefly as a tool the panopticon government could use to analyze the video and audio coming in from their surveillance. He also says:
“Being even further removed from individuals and culturally cohesive ‘peoples’ than are typical state governments, such an institution might by some be perceived as less legitimate, and it may be more susceptible to agency problems such as bureaucratic sclerosis or political drift away from the public interest.”
He also considers what might be required for a global state to bring other world governments to heel. So I don’t think he is assuming that the state can completely ignore all dissent or resistance because it FOOMs into an all powerful AI.
Either way I think that is a really bad argument. It’s basically just saying “if we had aligned superintelligence running the world everything would be fine” which is almost tautologically true. But what are we supposed to conclude from that? I don’t think that tells us anything about increasing state power on the margin. Also, aligning the interests of powerful AI with a powerful global state is not sufficient for alignment of AI with humanity more generally. Powerful global states are not very well aligned with the interests of their constituents.
My reading is that Bostrom is making arguments about how human governance would need to change to address risks from some types of technology. The arguments aren’t explicitly contingent on any AI technology that isn’t available today.
Bostrom may have talked about this elsewhere since I’ve heard other people say this, but he doesn’t make this point in the paper. He only mentions AI briefly as a tool the panopticon government could use to analyze the video and audio coming in from their surveillance. He also says:
“Being even further removed from individuals and culturally cohesive ‘peoples’ than are typical state governments, such an institution might by some be perceived as less legitimate, and it may be more susceptible to agency problems such as bureaucratic sclerosis or political drift away from the public interest.”
He also considers what might be required for a global state to bring other world governments to heel. So I don’t think he is assuming that the state can completely ignore all dissent or resistance because it FOOMs into an all powerful AI.
Either way I think that is a really bad argument. It’s basically just saying “if we had aligned superintelligence running the world everything would be fine” which is almost tautologically true. But what are we supposed to conclude from that? I don’t think that tells us anything about increasing state power on the margin. Also, aligning the interests of powerful AI with a powerful global state is not sufficient for alignment of AI with humanity more generally. Powerful global states are not very well aligned with the interests of their constituents.
My reading is that Bostrom is making arguments about how human governance would need to change to address risks from some types of technology. The arguments aren’t explicitly contingent on any AI technology that isn’t available today.