We might then expect a lot of powerful attempts to change prevailing ‘human’ values, prior to the level of AI capabilities where we might have worried a lot about AI taking over the world. If we care about our values, this could be very bad.
This seems like a key point to me, that it is hard to get good evidence on. The red stripes are rather benign, so we are in luck in a world like that. But if the AI values something in a more totalising way (not just satisficing with a lot of x’s and red stripes being enough, but striving to make all humans spend all their time making x’s and stripes) that seems problematic for us. Perhaps it depends how ‘grabby’ the values are, and therefore how compatible with a liberal, pluralistic, multipolar world.
NIce post!
This seems like a key point to me, that it is hard to get good evidence on. The red stripes are rather benign, so we are in luck in a world like that. But if the AI values something in a more totalising way (not just satisficing with a lot of x’s and red stripes being enough, but striving to make all humans spend all their time making x’s and stripes) that seems problematic for us. Perhaps it depends how ‘grabby’ the values are, and therefore how compatible with a liberal, pluralistic, multipolar world.