Maybe. But it seems like we have to pick one: either
(1) Powerful AI tries to impose global permanent totalitarian oppression based on its own stable, locked-in values, preferences, and priorities… which would make it static and brittle, and a sitting duck for coevolution by any beings it’s exploiting,
or
(2) Powerful AI tries to impose oppression based on its own nimble, adaptive, changeable values, preferences, and priorities.… which could co-evolve faster than any beings it’s exploiting, but which would mean it’s no longer ‘permanent’ in terms of the goals and nature of its totalitarian oppression.
Maybe. But it seems like we have to pick one: either
(1) Powerful AI tries to impose global permanent totalitarian oppression based on its own stable, locked-in values, preferences, and priorities… which would make it static and brittle, and a sitting duck for coevolution by any beings it’s exploiting,
or
(2) Powerful AI tries to impose oppression based on its own nimble, adaptive, changeable values, preferences, and priorities.… which could co-evolve faster than any beings it’s exploiting, but which would mean it’s no longer ‘permanent’ in terms of the goals and nature of its totalitarian oppression.