Let’s assume that an AGI is, indeed, created sometime in the future. Let us also assume that MIRI achieves its goal of essentialy protecting us from the existential dangers that stem from it. My question may well be quite naive, but how likely is it for a totalitarian “New World Order” to seize control of said AGI and use it for their own purposes, deciding who gets to benefit from it and to what degree? This is something I, myself, get asked a lot and while it takes into account the current state of society which look nothing like the next ones probably will, I can’t seem to properly reject as a possibilty.
I wouldn’t reject it as a possibility. MIRI wants AGI to have good consequences for human freedom, happiness, etc., but any big increase in power raises the risk that the power will be abused. Ideally we’d want the AI to resist being misused, but there’s a tradeoff between ‘making the AI more resistant to misuse by its users (when the AI is right and the user is wrong)’ and ‘making the AI more amenable to correction by its users (when the AI is wrong and the user is right).’
I wouldn’t say it’s inevitable either, though. It doesn’t appear to me that past technological growth has tended to increase how totalitarian the average state is.
Let’s assume that an AGI is, indeed, created sometime in the future. Let us also assume that MIRI achieves its goal of essentialy protecting us from the existential dangers that stem from it. My question may well be quite naive, but how likely is it for a totalitarian “New World Order” to seize control of said AGI and use it for their own purposes, deciding who gets to benefit from it and to what degree?
This is something I, myself, get asked a lot and while it takes into account the current state of society which look nothing like the next ones probably will, I can’t seem to properly reject as a possibilty.
I wouldn’t reject it as a possibility. MIRI wants AGI to have good consequences for human freedom, happiness, etc., but any big increase in power raises the risk that the power will be abused. Ideally we’d want the AI to resist being misused, but there’s a tradeoff between ‘making the AI more resistant to misuse by its users (when the AI is right and the user is wrong)’ and ‘making the AI more amenable to correction by its users (when the AI is wrong and the user is right).’
I wouldn’t say it’s inevitable either, though. It doesn’t appear to me that past technological growth has tended to increase how totalitarian the average state is.