I worry that the values that people want to put into a singleton are badly wrong, e.g. creating hedonium. I want a singleton that will protect us from other AI. Other than that, I’d be wary of trying to maximize a value right now. At most I’d tell the AI “hold until future orders”.
“Hold until future orders” is one approach but it might turn out to be much more difficult than actually creating an AI with correct values. This is because the formal specification of metaethics (that is a mathematical procedure that takes humans as input and produces a utility function as output) should be of much lower complexity than specifying what it means to “protect from other AI but do nothing else.”
I worry that the values that people want to put into a singleton are badly wrong, e.g. creating hedonium. I want a singleton that will protect us from other AI. Other than that, I’d be wary of trying to maximize a value right now. At most I’d tell the AI “hold until future orders”.
“Hold until future orders” is one approach but it might turn out to be much more difficult than actually creating an AI with correct values. This is because the formal specification of metaethics (that is a mathematical procedure that takes humans as input and produces a utility function as output) should be of much lower complexity than specifying what it means to “protect from other AI but do nothing else.”