**Would some moral/virtue intuitions prefer small but non-zero x-risk?**
[Me trying to lean into a kind of philosophical reasoning I don’t find plausible. Not important, except perhaps as cautionary tale for what kind of things could happen if you wanted to base the case for reducing x-risk on purely non-consequentialist reasons.]
(Inspired by a conversation with Toby Newberry about something else.)
The basic observation: we sometimes think that a person achieving some success is particularly praiseworthy, remarkable, virtuous, or similar if they could have failed. (Or if they needed to expend a lot of effort, suffered through hardship, etc.)
Could this mean that we removed one source of value if we reduced x-risk to zero? Achieving our full potential would then no longer constitute a contingent achievement—it would be predetermined, with failure no longer on the table.
We can make the thought more precise in a toy model: Suppose that at some time t_0 x-risk is permanently reduced to zero. The worry is that acts happening after t_0 (or perhaps acts of agents born after t_0), even if they produce value, are less valuable in one respect: In their role of being a part of us eventually achieving our full potentially, they can no longer fail. More broadly, humanity’s great generation-spanning project (whatever that is) can no longer fail. Those humans living after t_0 therefore have a less valuable part in that project. They merely help push along a wagon that was set firmly in its tracks by their ancestors. Their actions may have various valuable consequences, but they no longer increase the probability of humanity’s grand project succeeding.
(Similarly, if we’re inclined to regard humanity as a whole, or generations born after t_0, as moral agents in their own right we might worry that zero x-risk detracts from the value of their actions.)
Some intuition pumps:
Suppose that benevolent parents arrange things (perhaps with the help of fantastic future technology) in such a way that their child will with certainty have a highly successful career. (They need not remove that child’s agency or autonomy: perhaps the child can choose if she’ll become a world-class composer, scientist, political leader, etc. - but no matter what she does, she’ll excel and will be remembered for their great achievements.) We can even suppose that it will seem to the child as if it could have failed, and that she did experience small-scale setbacks: perhaps she went rock climbing, slipped, and broke her leg; but unbeknownst to her she was watched by invisible drones that would have caught her if she would have otherwise died. And so on for every single moment of her life.
Suppose that the matriarch of a family dynasty arranges things in such a way that the family business’s success will henceforth be guaranteed. Stock price, cash flow, and so on may still go up and down, but the firm can no longer go bankrupt and will broadly be the most profitable company in its sector, forever. Her heirs might on some occasions think that their management has turned things around, and prevented the company’s downfall. But really the shots were called by a great grandmother generations ago.
We may think that there is something objectionable, even dystopian about these situations. At the very least, we may think that the apparent successes of the child prodigy or the family business leaders count for less because, in one respect, they could not have failed.
If we give a lot of weight to such worries we may not want to eliminate x-risk. Instead, perhaps, we’d conclude that it’s best to carefully manage x-risk: at any point, it should not be so high that we run an irresponsible risk of squandering all future potential—but it also should not be so low that our children are robbed of more value than we protect.
--
Some reasons why I think this is either implausible or irrelevant:
I think the intuitions underlying the basic observation aren’t about axiology, probably not even about normative ethics in any sense. I think they are best explained away by evolutionary debunking arguments, or taken to be heuristics about instrumental value, or (if one wants to take them as seriously as plausibly defensible) about some primitive notion of ‘virtue’ or ‘praiseworthiness’ that’s different from axiology.
This at least partly relies on intuitions about the import of “could have done otherwise” claims, which may be confused or deceptive anyway. For example, we might think that Frankfurt-style examples show that such claims aren’t actually relevant for moral responsibility, despite appearing to be at first glance. And we might think this is a reason to also not take them seriously in this context.
Even if there’s something to this, on plausible views the loss of value from removing x-risk would be overcompensated by the gains.
Reducing x-risk to exactly zero, as opposed to something tiny, is implausible anyway. (Though we might worry that all of this applies if x-risk falls below some positive threshold, or that there is some gradual loss of value as x-risk falls.)
The worry straightforwardly only applies in scenarios where we can permanently reduce x-risk to zero, no matter what future agents do. This seems extremely implausible. Even if we could reduce x-risk to zero, most likely it would require the continuing effort of future agents to keep it at zero.
Interesting post. I think I have a couple of thoughts, please forgive the uneditted nature.
One issue is whether more than one person can get credit for the same event. If this is the case, then both the climber girl and the parents can get credit for her surviving the climb (after all, both their actions were sufficient). Similarly, both we and the future people can get credit for saving the world.
If not, then only one person can get the credit for every instance of world saving. Either we can harvest them now, or we can leave them for other people to get. But the latter strategy involves the risk that they will remain unharvested, leading to a reduction in the total quantity of creditworthiness mankind accrues. So from the point of view of an impartial maximiser of humanity’s creditworthiness, we should seize as many as we can, leaving as little as possible for the future.
Secondly, as a new parent I see the appeal of the invisible robots of deliverance! I am keen to let the sproglet explore and stake out her own achievements, but I don’t think she loses much when I keep her from dying. She can get plenty of moral achievement from ascending to new heights, even if I have sealed off the depths.
Finally, there is of course the numerical consideration that even if facing a 1% risk of extinction carried some inherent moral glory, it would also reduce the value of all subsequent things by 1% (in expectation). Unless you think the benefit from our children, rather than us, overcoming that risk is large compared to the total value of the future of humanity, it seems like we should probably deny them it.
Thanks, this all makes sense to me. Just one quick comment:
So from the point of view of an impartial maximiser of humanity’s creditworthiness, we should seize as many as we can, leaving as little as possible for the future.
If I understand you correctly, your argument for this conclusion assumed that the total number of world-saving instances is fixed independently of anyone’s actions. But I think in practice this is wrong, i.e. the number of world-saving opportunities is endogenous to people’s actions including in particular whether they reap current world-saving opportunities.
Oversimplified example: perhaps currently there is one world-saving instance per year from Petrov-style incidents, i.e. countries not launching a nuclear strike in response to a false alarm of a nuclear attack. But if there was a breakthrough in nuclear disarmament that reduced nuclear stockpiles to zero this would also eliminate these future world-saving opportunities.
[Oversimplified b/c in fact a nuclear exchange isn’t clearly an x-risk.]
Hey, yes—I would count that nuclear disarmament breakthrough as being equal to the sum of those annual world-saving instances. So you’re right that the number of events isn’t fixed, but their measure (as in the % of the future of humanity saved) is bounded.
Nice post. I’m reminded of this Bertrand Russell passage:
“all the labours of the ages, all the devotion, all the inspiration, all the noonday brightness of human genius, are destined to extinction in the vast death of the solar system, and that the whole temple of Man’s achievement must inevitably be buried beneath the debris of a universe in ruins … Only within the scaffolding of these truths, only on the firm foundation of unyielding despair, can the soul’s habitation henceforth be safely built.”
—A Free Man’s Worship, 1903
I take Russell as arguing that the inevitability (as he saw it) of extinction undermines the possibility of enduring achievement, and that we must therefore either ground life’s meaning in something else, or accept nihilism.
At a stretch, maybe you could run your argument together with Russell’s — if we ground life’s meaning in achievement, then avoiding nihilism requires that humanity neither go extinct nor achieve total existential security.
Related: Relationships in a post-singularity future can also be set up to work well, so that the setup overdetermines any efforts by the individuals in them.
To me, that takes away the whole point. I don’t think this would feel less problematic if somehow future people decided to add some noise to the setup, such that relationships occasionally fail.
The reason I find any degree of “setup” problematic is because this seems like emphasizing the self-oriented benefits one gets out of relationships, and de-emphasizing the from-you-independent identity of the other person. It’s romantic to think that there’s a soulmate out there who would be just as happy to find you as you are about finding them. It’s not that romantic to think about creating your soulmate with the power of future technology (or society doing this for you).
This is the “person-affecting intuition for thinking about soulmates.” If the other person exists already, I’d be excited to meet them, and would be motivated to put in a lot of effort to make things work, as opposed to just giving up on myself in the face of difficulties. By contrast, if the person doesn’t exist yet or won’t exist in a way independent of my actions, I feel like there’s less of a point/appeal to it.
**Would some moral/virtue intuitions prefer small but non-zero x-risk?**
[Me trying to lean into a kind of philosophical reasoning I don’t find plausible. Not important, except perhaps as cautionary tale for what kind of things could happen if you wanted to base the case for reducing x-risk on purely non-consequentialist reasons.]
(Inspired by a conversation with Toby Newberry about something else.)
The basic observation: we sometimes think that a person achieving some success is particularly praiseworthy, remarkable, virtuous, or similar if they could have failed. (Or if they needed to expend a lot of effort, suffered through hardship, etc.)
Could this mean that we removed one source of value if we reduced x-risk to zero? Achieving our full potential would then no longer constitute a contingent achievement—it would be predetermined, with failure no longer on the table.
We can make the thought more precise in a toy model: Suppose that at some time t_0 x-risk is permanently reduced to zero. The worry is that acts happening after t_0 (or perhaps acts of agents born after t_0), even if they produce value, are less valuable in one respect: In their role of being a part of us eventually achieving our full potentially, they can no longer fail. More broadly, humanity’s great generation-spanning project (whatever that is) can no longer fail. Those humans living after t_0 therefore have a less valuable part in that project. They merely help push along a wagon that was set firmly in its tracks by their ancestors. Their actions may have various valuable consequences, but they no longer increase the probability of humanity’s grand project succeeding.
(Similarly, if we’re inclined to regard humanity as a whole, or generations born after t_0, as moral agents in their own right we might worry that zero x-risk detracts from the value of their actions.)
Some intuition pumps:
Suppose that benevolent parents arrange things (perhaps with the help of fantastic future technology) in such a way that their child will with certainty have a highly successful career. (They need not remove that child’s agency or autonomy: perhaps the child can choose if she’ll become a world-class composer, scientist, political leader, etc. - but no matter what she does, she’ll excel and will be remembered for their great achievements.) We can even suppose that it will seem to the child as if it could have failed, and that she did experience small-scale setbacks: perhaps she went rock climbing, slipped, and broke her leg; but unbeknownst to her she was watched by invisible drones that would have caught her if she would have otherwise died. And so on for every single moment of her life.
Suppose that the matriarch of a family dynasty arranges things in such a way that the family business’s success will henceforth be guaranteed. Stock price, cash flow, and so on may still go up and down, but the firm can no longer go bankrupt and will broadly be the most profitable company in its sector, forever. Her heirs might on some occasions think that their management has turned things around, and prevented the company’s downfall. But really the shots were called by a great grandmother generations ago.
We may think that there is something objectionable, even dystopian about these situations. At the very least, we may think that the apparent successes of the child prodigy or the family business leaders count for less because, in one respect, they could not have failed.
If we give a lot of weight to such worries we may not want to eliminate x-risk. Instead, perhaps, we’d conclude that it’s best to carefully manage x-risk: at any point, it should not be so high that we run an irresponsible risk of squandering all future potential—but it also should not be so low that our children are robbed of more value than we protect.
--
Some reasons why I think this is either implausible or irrelevant:
I think the intuitions underlying the basic observation aren’t about axiology, probably not even about normative ethics in any sense. I think they are best explained away by evolutionary debunking arguments, or taken to be heuristics about instrumental value, or (if one wants to take them as seriously as plausibly defensible) about some primitive notion of ‘virtue’ or ‘praiseworthiness’ that’s different from axiology.
This at least partly relies on intuitions about the import of “could have done otherwise” claims, which may be confused or deceptive anyway. For example, we might think that Frankfurt-style examples show that such claims aren’t actually relevant for moral responsibility, despite appearing to be at first glance. And we might think this is a reason to also not take them seriously in this context.
Even if there’s something to this, on plausible views the loss of value from removing x-risk would be overcompensated by the gains.
Reducing x-risk to exactly zero, as opposed to something tiny, is implausible anyway. (Though we might worry that all of this applies if x-risk falls below some positive threshold, or that there is some gradual loss of value as x-risk falls.)
The worry straightforwardly only applies in scenarios where we can permanently reduce x-risk to zero, no matter what future agents do. This seems extremely implausible. Even if we could reduce x-risk to zero, most likely it would require the continuing effort of future agents to keep it at zero.
Interesting post. I think I have a couple of thoughts, please forgive the uneditted nature.
One issue is whether more than one person can get credit for the same event. If this is the case, then both the climber girl and the parents can get credit for her surviving the climb (after all, both their actions were sufficient). Similarly, both we and the future people can get credit for saving the world.
If not, then only one person can get the credit for every instance of world saving. Either we can harvest them now, or we can leave them for other people to get. But the latter strategy involves the risk that they will remain unharvested, leading to a reduction in the total quantity of creditworthiness mankind accrues. So from the point of view of an impartial maximiser of humanity’s creditworthiness, we should seize as many as we can, leaving as little as possible for the future.
Secondly, as a new parent I see the appeal of the invisible robots of deliverance! I am keen to let the sproglet explore and stake out her own achievements, but I don’t think she loses much when I keep her from dying. She can get plenty of moral achievement from ascending to new heights, even if I have sealed off the depths.
Finally, there is of course the numerical consideration that even if facing a 1% risk of extinction carried some inherent moral glory, it would also reduce the value of all subsequent things by 1% (in expectation). Unless you think the benefit from our children, rather than us, overcoming that risk is large compared to the total value of the future of humanity, it seems like we should probably deny them it.
Thanks, this all makes sense to me. Just one quick comment:
If I understand you correctly, your argument for this conclusion assumed that the total number of world-saving instances is fixed independently of anyone’s actions. But I think in practice this is wrong, i.e. the number of world-saving opportunities is endogenous to people’s actions including in particular whether they reap current world-saving opportunities.
Oversimplified example: perhaps currently there is one world-saving instance per year from Petrov-style incidents, i.e. countries not launching a nuclear strike in response to a false alarm of a nuclear attack. But if there was a breakthrough in nuclear disarmament that reduced nuclear stockpiles to zero this would also eliminate these future world-saving opportunities.
[Oversimplified b/c in fact a nuclear exchange isn’t clearly an x-risk.]
Hey, yes—I would count that nuclear disarmament breakthrough as being equal to the sum of those annual world-saving instances. So you’re right that the number of events isn’t fixed, but their measure (as in the % of the future of humanity saved) is bounded.
Nice post. I’m reminded of this Bertrand Russell passage:
I take Russell as arguing that the inevitability (as he saw it) of extinction undermines the possibility of enduring achievement, and that we must therefore either ground life’s meaning in something else, or accept nihilism.
At a stretch, maybe you could run your argument together with Russell’s — if we ground life’s meaning in achievement, then avoiding nihilism requires that humanity neither go extinct nor achieve total existential security.
Related: Relationships in a post-singularity future can also be set up to work well, so that the setup overdetermines any efforts by the individuals in them.
To me, that takes away the whole point. I don’t think this would feel less problematic if somehow future people decided to add some noise to the setup, such that relationships occasionally fail.
The reason I find any degree of “setup” problematic is because this seems like emphasizing the self-oriented benefits one gets out of relationships, and de-emphasizing the from-you-independent identity of the other person. It’s romantic to think that there’s a soulmate out there who would be just as happy to find you as you are about finding them. It’s not that romantic to think about creating your soulmate with the power of future technology (or society doing this for you).
This is the “person-affecting intuition for thinking about soulmates.” If the other person exists already, I’d be excited to meet them, and would be motivated to put in a lot of effort to make things work, as opposed to just giving up on myself in the face of difficulties. By contrast, if the person doesn’t exist yet or won’t exist in a way independent of my actions, I feel like there’s less of a point/appeal to it.