Hey Jack, I think this is a great question and I dedicate a portion of my MA philosophy thesis to this. Here are some general points:
It is likely that the expected moral value of the future is dominated by futures in which there is optimization for moral (dis)value. Since we would expect it to be much more likely there will be optimization for value than for disvalue, the expected moral value of the future seems positive (unless you adhere to strict/lexical negative utilitarianism). This claim depends on the difference between possible worlds that are optimized for (dis)value vs. states that are subject to other pressures, like competition: this difference is not obviously large (see Price of Anarchy).
There seems substantial convergence between improving the quality of the long-term future and reducing extinction risk. Things that can bring humanity to extinction (superintelligence, virus, nuclear winter, extreme climate change) can also very bad for the long-term future of humanity if they do not lead to extinction. Therefore, reducing extinction risk also has very positive effects on the quality of the long-term future. Potential suffering risk from misaligned AI is one candidate. In addition, I think global catastrophes, if they don’t lead to extinction, create a negative trajectory change in expectation. Either because civilizational collapse puts us on a worse trajectory, but the most likely outcome is what I call general global disruption: civilization doesn’t quite collapse, but think are shaken up a lot. From my thesis:
Should we expect global disruption to be (in expectation) good or bad for the value of the future? This is speculative, but Beckstead lays out some reasons to expect that global disruption will put humanity most certainly on a worse trajectory: it may reverse social progress, limit the ability to adequately regulate the development of dangerous technologies, open an opportunity for authoritarian regimes to take hold, or increase inter-state conflict (Beckstead, 2015). We can also approach the issue abstractly: disruption can be seen as injecting more noise into a previously more stable global system, increasing the probability that the world settles into a different semi-stable configuration. If there are many more undesirable configurations of the world than desirable ones, increasing randomness is more likely to lead to an undesirable state of the world. I am convinced that, unless we are currently in a particularly bad state of the world, global disruption would have a very negative effect (in expectation) on the value of the long-term future.
I find it unlikely that we would export wild-animal suffering beyond our solar system. It takes a lot of time to move to different solar systems, and I don’t think future civilizations will require a lot of wilderness: it’s a very inefficient use of resources. So I believe the amount of suffering is relatively small from that source. However, I think some competitive dynamics between digital beings could create astronomical amounts of suffering, and this could come about if we focus only on reducing extinction risk.
Whether you want to focus on the quality of the future also depends on your moral views. Some people weigh preventing future suffering much more heavily than enabling the creation of future happiness. For them, part of the value of reducing extinction risk is taken away, and they will have stronger reasons to focus on the quality of the future.
I found the post by Brauner & Grosse-Holz and the post by Beckstead most helpful. I know that Haydn Belfield (CSER) is currently working on a longer article about the long-term significance of reducing Global Catastrophic Risks.
In conclusion, I think reducing extinction risk is a very positive in terms of expected value, even if one expects the future to be negative! However, depending on different parameters, there might be better options than focusing on extinction risk. Candidates involve particular parts of moral circle expansion and suffering risks from AI.
I can send you the current draft of my thesis in case you’re interested, and will post it online once I have finished it.
We can also approach the issue abstractly: disruption can be seen as injecting more noise into a previously more stable global system, increasing the probability that the world settles into a different semi-stable configuration. If there are many more undesirable configurations of the world than desirable ones, increasing randomness is more likely to lead to an undesirable state of the world. I am convinced that, unless we are currently in a particularly bad state of the world, global disruption would have a very negative effect (in expectation) on the value of the long-term future.
If there are many more undesirable configurations of the world than desirable ones, then we should, a priori, expect that our present configuration is an undesirable one. Also, if the only effect of disruption was to re-randomize the world order, then the only thing you’d need for disruption to be positive is for the current state to be worse than the average civilisation from the distribution. Maybe this is what you mean with “particularly bad state”, but intuitively, I interpret that more like the bottom 15 %.
There are certainly arguments to make for our world being better than average. But I do think that you actually have to make those arguments, and that without them, this abstract model won’t tell you if disruption is good or bad.
Hmm, I have not phrased my idea clearly, so thank you for your comment, because now I am improving my concepts :)
If there are many more undesirable configurations of the world than desirable ones, then we should, a priori, expect that our present configuration is an undesirable one.
I agree with this. But that does not imply that disruption would not have a negative effect on expectation.
I don’t see disruption as ‘re-randomization’ and picking any new configuration out of the space of all possible futures. Rather, I see disruption as a ‘random departure’ from a current state, and not each possible future is equally close to the current state. And because I expect there to almost always be more ways to go ‘down’ than ‘up’, I expect this random departure to be (highly) negative.
I find it unlikely that we would export wild-animal suffering beyond our solar system. It takes a lot of time to move to different solar systems, and I don’t think future civilizations will require a lot of wilderness: it’s a very inefficient use of resources. So I believe the amount of suffering is relatively small from that source. However, I think some competitive dynamics between digital beings could create astronomical amounts of suffering, and this could come about if we focus only on reducing extinction risk.
Agreed, but the other possibility is that there will be simulations of wild animals in the future. So I think spreading the meme that wild animals can suffer to the AI community could be valuable.
Hey Jack, I think this is a great question and I dedicate a portion of my MA philosophy thesis to this. Here are some general points:
It is likely that the expected moral value of the future is dominated by futures in which there is optimization for moral (dis)value. Since we would expect it to be much more likely there will be optimization for value than for disvalue, the expected moral value of the future seems positive (unless you adhere to strict/lexical negative utilitarianism). This claim depends on the difference between possible worlds that are optimized for (dis)value vs. states that are subject to other pressures, like competition: this difference is not obviously large (see Price of Anarchy).
There seems substantial convergence between improving the quality of the long-term future and reducing extinction risk. Things that can bring humanity to extinction (superintelligence, virus, nuclear winter, extreme climate change) can also very bad for the long-term future of humanity if they do not lead to extinction. Therefore, reducing extinction risk also has very positive effects on the quality of the long-term future. Potential suffering risk from misaligned AI is one candidate. In addition, I think global catastrophes, if they don’t lead to extinction, create a negative trajectory change in expectation. Either because civilizational collapse puts us on a worse trajectory, but the most likely outcome is what I call general global disruption: civilization doesn’t quite collapse, but think are shaken up a lot. From my thesis:
I find it unlikely that we would export wild-animal suffering beyond our solar system. It takes a lot of time to move to different solar systems, and I don’t think future civilizations will require a lot of wilderness: it’s a very inefficient use of resources. So I believe the amount of suffering is relatively small from that source. However, I think some competitive dynamics between digital beings could create astronomical amounts of suffering, and this could come about if we focus only on reducing extinction risk.
Whether you want to focus on the quality of the future also depends on your moral views. Some people weigh preventing future suffering much more heavily than enabling the creation of future happiness. For them, part of the value of reducing extinction risk is taken away, and they will have stronger reasons to focus on the quality of the future.
I found the post by Brauner & Grosse-Holz and the post by Beckstead most helpful. I know that Haydn Belfield (CSER) is currently working on a longer article about the long-term significance of reducing Global Catastrophic Risks.
In conclusion, I think reducing extinction risk is a very positive in terms of expected value, even if one expects the future to be negative! However, depending on different parameters, there might be better options than focusing on extinction risk. Candidates involve particular parts of moral circle expansion and suffering risks from AI.
I can send you the current draft of my thesis in case you’re interested, and will post it online once I have finished it.
If there are many more undesirable configurations of the world than desirable ones, then we should, a priori, expect that our present configuration is an undesirable one. Also, if the only effect of disruption was to re-randomize the world order, then the only thing you’d need for disruption to be positive is for the current state to be worse than the average civilisation from the distribution. Maybe this is what you mean with “particularly bad state”, but intuitively, I interpret that more like the bottom 15 %.
There are certainly arguments to make for our world being better than average. But I do think that you actually have to make those arguments, and that without them, this abstract model won’t tell you if disruption is good or bad.
Hmm, I have not phrased my idea clearly, so thank you for your comment, because now I am improving my concepts :)
I agree with this. But that does not imply that disruption would not have a negative effect on expectation.
I don’t see disruption as ‘re-randomization’ and picking any new configuration out of the space of all possible futures. Rather, I see disruption as a ‘random departure’ from a current state, and not each possible future is equally close to the current state. And because I expect there to almost always be more ways to go ‘down’ than ‘up’, I expect this random departure to be (highly) negative.
Agreed, but the other possibility is that there will be simulations of wild animals in the future. So I think spreading the meme that wild animals can suffer to the AI community could be valuable.
Thanks! Would be very interesting to see you thesis once it is finished