I don’t think you need to commit yourself to including everyone. If it is true for any subset of people, then the point you gesture at in your post goes through. I have had similar thoughts to those you suggest in the post. If we gave the AI the goal of ‘do what Barack Obama would do if properly informed and at his most lucid’, I don’t really get why we would have high confidence in a treacherous turn or of the AI misbehaving in a catastrophic way. The main response to this seems to be to point to examples of AI not doing what we intend from limited computer games. I agree something similar might happen with advanced AI but don’t get why it is guaranteed to do so or why any of the arguments I have seen lend weight to any particular probability estimate of catastrophe.
It also seems like increased capabilities would in a sense increased alignment (with Obama) because the more advanced AIs would have a better idea of what Obama would do.
I don’t think you need to commit yourself to including everyone. If it is true for any subset of people, then the point you gesture at in your post goes through. I have had similar thoughts to those you suggest in the post. If we gave the AI the goal of ‘do what Barack Obama would do if properly informed and at his most lucid’, I don’t really get why we would have high confidence in a treacherous turn or of the AI misbehaving in a catastrophic way. The main response to this seems to be to point to examples of AI not doing what we intend from limited computer games. I agree something similar might happen with advanced AI but don’t get why it is guaranteed to do so or why any of the arguments I have seen lend weight to any particular probability estimate of catastrophe.
It also seems like increased capabilities would in a sense increased alignment (with Obama) because the more advanced AIs would have a better idea of what Obama would do.