I completely agree that many conceivable post-human future have low value. See also “unhumanity” scenario in my analysis. I think that term “existential risk” might be somewhat misleading since what we’re really aiming it as “existence of beings and experiences that we value” rather than just existence of “something.” That is, I view your reasoning not as an argument for caring less about existential risk but as an argument for working towards a valuable far future.
Regarding MIRI, I think their position is completely adequate since once we create a singleton which endorses our values it will guard us from all sorts of bad futures, not only from extinction.
Regarding “consciousness as similarity”, I think it’s a useful heuristic but it’s not necessarily universally applicable. I consider certain futures in which I gradually evolve into something much more complex than my current self as positive, but one must be very careful about which trajectories to endorse. Building an FAI will save us from doing irreversible mistakes, but if for some reason constructing a singleton turns out to be intractable we will have to think of other solutions.
I worry that the values that people want to put into a singleton are badly wrong, e.g. creating hedonium. I want a singleton that will protect us from other AI. Other than that, I’d be wary of trying to maximize a value right now. At most I’d tell the AI “hold until future orders”.
“Hold until future orders” is one approach but it might turn out to be much more difficult than actually creating an AI with correct values. This is because the formal specification of metaethics (that is a mathematical procedure that takes humans as input and produces a utility function as output) should be of much lower complexity than specifying what it means to “protect from other AI but do nothing else.”
I completely agree that many conceivable post-human future have low value. See also “unhumanity” scenario in my analysis. I think that term “existential risk” might be somewhat misleading since what we’re really aiming it as “existence of beings and experiences that we value” rather than just existence of “something.” That is, I view your reasoning not as an argument for caring less about existential risk but as an argument for working towards a valuable far future.
Regarding MIRI, I think their position is completely adequate since once we create a singleton which endorses our values it will guard us from all sorts of bad futures, not only from extinction.
Regarding “consciousness as similarity”, I think it’s a useful heuristic but it’s not necessarily universally applicable. I consider certain futures in which I gradually evolve into something much more complex than my current self as positive, but one must be very careful about which trajectories to endorse. Building an FAI will save us from doing irreversible mistakes, but if for some reason constructing a singleton turns out to be intractable we will have to think of other solutions.
I worry that the values that people want to put into a singleton are badly wrong, e.g. creating hedonium. I want a singleton that will protect us from other AI. Other than that, I’d be wary of trying to maximize a value right now. At most I’d tell the AI “hold until future orders”.
“Hold until future orders” is one approach but it might turn out to be much more difficult than actually creating an AI with correct values. This is because the formal specification of metaethics (that is a mathematical procedure that takes humans as input and produces a utility function as output) should be of much lower complexity than specifying what it means to “protect from other AI but do nothing else.”