I mean that the end of the world isn’t a bad outcome to someone who only values the absence of suffering, and who is perfectly indifferent between all ‘positive’ states. (This is Ord’s definition of absolute NU, so I don’t think I’m straw-manning that kind.) And if something isn’t bad (and doesn’t prevent any good), a utilitarian ‘doesn’t have to work on it’ in the sense that there’s no moral imperative to.
(1) That makes sense. But there’s an escalation problem: worse risk is better to ANU (see below).
(2) One dreadful idea is that self-replicators would do the anti-suffering work, obviating the need for sentient guardians, but I see what you’re saying. Again though, this uncertainty about moral patients licences ANU work on x-risks to humans… but only while moving the degenerate ‘solution’ upward, to valuing risks that destroy more classes of candidate moral patients. At the limit, the end of the entire universe is indisputably optimal to an ANU. So you’re right about Earth x-risks (which is mostly all people talk about) but not for really farout scifi ones, which ANU seems to value.
Actually this degenerate motion might change matters practically: it seems improbable that it’d be harder to remove suffering with biotechnology than to destroy everything. Up to you if you’re willing to bite the bullet on the remaining theoretical repugnance.
(To clarify, I think basically no negative utilitarian wants this, including those who identify with absolute NU. But that suggests that their utility function is more complex than they let on. You hint at this when you mention valuing an ‘infinite game’ of suffering alleviation. This doesn’t make sense on the ANU account, because each iteration can only break even (not increase suffering) or lose (increase suffering).)
Most ethical views have degenerate points in them, but valuing the greatest destruction equal to the greatest hedonic triumph is unusually repugnant, even among repugnant conclusions.
I don’t think instrumentally valuing positive states helps with the x-risk question, because they get trumped by a sufficiently large amount of terminal value, again e.g. the end of all things.
I mean that the end of the world isn’t a bad outcome to someone who only values the absence of suffering, and who is perfectly indifferent between all ‘positive’ states. (This is Ord’s definition of absolute NU, so I don’t think I’m straw-manning that kind.) And if something isn’t bad (and doesn’t prevent any good), a utilitarian ‘doesn’t have to work on it’ in the sense that there’s no moral imperative to.
(1) That makes sense. But there’s an escalation problem: worse risk is better to ANU (see below).
(2) One dreadful idea is that self-replicators would do the anti-suffering work, obviating the need for sentient guardians, but I see what you’re saying. Again though, this uncertainty about moral patients licences ANU work on x-risks to humans… but only while moving the degenerate ‘solution’ upward, to valuing risks that destroy more classes of candidate moral patients. At the limit, the end of the entire universe is indisputably optimal to an ANU. So you’re right about Earth x-risks (which is mostly all people talk about) but not for really farout scifi ones, which ANU seems to value.
Actually this degenerate motion might change matters practically: it seems improbable that it’d be harder to remove suffering with biotechnology than to destroy everything. Up to you if you’re willing to bite the bullet on the remaining theoretical repugnance.
(To clarify, I think basically no negative utilitarian wants this, including those who identify with absolute NU. But that suggests that their utility function is more complex than they let on. You hint at this when you mention valuing an ‘infinite game’ of suffering alleviation. This doesn’t make sense on the ANU account, because each iteration can only break even (not increase suffering) or lose (increase suffering).)
Most ethical views have degenerate points in them, but valuing the greatest destruction equal to the greatest hedonic triumph is unusually repugnant, even among repugnant conclusions.
I don’t think instrumentally valuing positive states helps with the x-risk question, because they get trumped by a sufficiently large amount of terminal value, again e.g. the end of all things.
(I’m not making claims about other kinds of NU.)