Great work. A few notes in descending order or importance which I’d love to see addressed at least in brief:
1) This seems not to engage with the questions about short-term versus long-term prioritization and discount rates. I’d think that the implicit assumptions should be made clearer.
2) It doesn’t seem obvious to me that, given the universalist assumptions about the value of animal or other non-human species, the long term future is affected nearly as much by the presence or absence of humans. Depending on uncertainties about the Fermi hypothesis and the viability of non-human animals developing sentience over long time frames, this might greatly matter.
3) Reducing the probability of technological existential risks may require increasing the probability of human stagnation.
4) S-risks are plausibly more likely if moral development is outstripped by growth in technological power over relatively short time frames, and existential catastrophe has a comparatively limited downside.
1) This seems not to engage with the questions about short-term versus long-term prioritization and discount rates. I’d think that the implicit assumptions should be made clearer.
Yes, the article does not deal with considerations for and against caring about the long-term. This is discussed elsewhere. Instead, the article assumes that we care about the long-term (e.g. that we don’t discount the value of future lives strongly), and analyses what implications follow from that view.
We tried to make that explicit. E.g., the first point under “Moral assumptions” reads:
Throughout this article, we base our considerations on two assumptions:
1. That it morally matters what happens in the billions of years to come. From this very long-term view, making sure the future plays out well is a primary moral concern.
2) It doesn’t seem obvious to me that, given the universalist assumptions about the value of animal or other non-human species, the long term future is affected nearly as much by the presence or absence of humans. Depending on uncertainties about the Fermi hypothesis and the viability of non-human animals developing sentience over long time frames, this might greatly matter.
I think this point matters. Part 2.1 of the article deals with the implications of potential future non-human animal civilizations and extraterrestrials. I think the implications are somewhat complicated and depend quite a bit on your values, so I won’t try to summarize them here.
4) S-risks are plausibly more likely if moral development is outstripped by growth in technological power over relatively short time frames, and existential catastrophe has a comparatively limited downside.
We don’t try to argue for increasing the speed of technological progress.
Apart from that, it is not clear to me that extinction has “comparatively little downside” (compared to S-risks, you probably mean). It, of course, depends on your moral values. But even from a suffering-focused perspective, it may well be that we would—with more moral and empirical insight—come to realize that the universe is already filled with suffering. I personally would not be surprised if “S-risks by omission” (*) weighed pretty heavily in the overall calculus. This topic is discussed in part 2.2.
I don’t have anything useful to say regarding your point 3).
I’d agree with your points regarding limited scope for the first and second points, but I don’t understand how anyone can make prioritization decisions when we have no discounting—it’s nearly always better to conserve resources. If we have discounting for costs but not benefits, however, I worry the framework is incoherent. This is a much more general confusion I have, and the fact that you didn’t address or resolve it is unsurprising.
Re: S-Risks, I’m wondering whether we need to be concerned about value misalignment leading to arbitrarily large negative utility, given some perspectives. I’m concerned that human values are incoherent, and any given maximization is likely to cause arbitrarily large “suffering” for some values—and if there are multiple groups with different values, this might mean any maximization imposes maximal suffering on the large majority of people’s values.
For example, if 1⁄3 of humanity feels that human liberty is a crucial value, without which human pleasure is worse than meaningless, another 1⁄3 views earning reward as critical, and the last 1⁄3 views bliss/pure hedonium as optimal, we would view tiling the universe with human brains maxed out for any one of these as a hugely negative outcome for 2⁄3 of humanity, much worse than extinction.
Regarding your second point, just a few thoughts: First of all, an important point is how you think values and morality work. If two-thirds of humanity, after thorough reflection, disagree with your values, does this give you a reason to become less certain about your values as well? Maybe adopt their values to a degree? …
Secondly, I am also uncertain how coherent/convergent human values will be. There seem to be good arguments for both sides, see e.g. this blog post by Paul Christiano (and the discussion with Brian Tomasik in the comments of that post): https://rationalaltruist.com/2013/06/13/against-moral-advocacy/
Third: In a situation like the one you described above, at least there would be huge room for compromise/gains from trade/… So if future humanity would be split into the three factions you suggested, they would not necessarily fight a war until only one faction remains that can then tile the universe with their preferred version. Indeed, they probably would not, as cooperation seems better for everyone in expectation.
1) I agree that there is some confusion on my part, and on the part of most others I have spoken to, about how terminal values and morality do or do not get updated.
2) Agreed.
3) I will point to a maybe forthcoming paper / idea of Eric Drexler at FHI that makes this point, which he called “pareto-topia”. Despite the wonderful virtues of the idea, I’m unclear if there is a stable game-theoretic mechanism that prevents a race to the bottom outcome when fundamentally different values are being traded off. Specifically in this case, it’s possible that different values lead to an inability to truthfully/reliably cooperate—a paved road to pareto-topia seems not to exist, and there might be no path at all.
Great work. A few notes in descending order or importance which I’d love to see addressed at least in brief:
1) This seems not to engage with the questions about short-term versus long-term prioritization and discount rates. I’d think that the implicit assumptions should be made clearer.
2) It doesn’t seem obvious to me that, given the universalist assumptions about the value of animal or other non-human species, the long term future is affected nearly as much by the presence or absence of humans. Depending on uncertainties about the Fermi hypothesis and the viability of non-human animals developing sentience over long time frames, this might greatly matter.
3) Reducing the probability of technological existential risks may require increasing the probability of human stagnation.
4) S-risks are plausibly more likely if moral development is outstripped by growth in technological power over relatively short time frames, and existential catastrophe has a comparatively limited downside.
Hi David, thanks for your comments.
Yes, the article does not deal with considerations for and against caring about the long-term. This is discussed elsewhere. Instead, the article assumes that we care about the long-term (e.g. that we don’t discount the value of future lives strongly), and analyses what implications follow from that view.
We tried to make that explicit. E.g., the first point under “Moral assumptions” reads:
I think this point matters. Part 2.1 of the article deals with the implications of potential future non-human animal civilizations and extraterrestrials. I think the implications are somewhat complicated and depend quite a bit on your values, so I won’t try to summarize them here.
We don’t try to argue for increasing the speed of technological progress.
Apart from that, it is not clear to me that extinction has “comparatively little downside” (compared to S-risks, you probably mean). It, of course, depends on your moral values. But even from a suffering-focused perspective, it may well be that we would—with more moral and empirical insight—come to realize that the universe is already filled with suffering. I personally would not be surprised if “S-risks by omission” (*) weighed pretty heavily in the overall calculus. This topic is discussed in part 2.2.
I don’t have anything useful to say regarding your point 3).
(*) term coined by Lukas Gloor, I think.
Thanks for replying.
I’d agree with your points regarding limited scope for the first and second points, but I don’t understand how anyone can make prioritization decisions when we have no discounting—it’s nearly always better to conserve resources. If we have discounting for costs but not benefits, however, I worry the framework is incoherent. This is a much more general confusion I have, and the fact that you didn’t address or resolve it is unsurprising.
Re: S-Risks, I’m wondering whether we need to be concerned about value misalignment leading to arbitrarily large negative utility, given some perspectives. I’m concerned that human values are incoherent, and any given maximization is likely to cause arbitrarily large “suffering” for some values—and if there are multiple groups with different values, this might mean any maximization imposes maximal suffering on the large majority of people’s values.
For example, if 1⁄3 of humanity feels that human liberty is a crucial value, without which human pleasure is worse than meaningless, another 1⁄3 views earning reward as critical, and the last 1⁄3 views bliss/pure hedonium as optimal, we would view tiling the universe with human brains maxed out for any one of these as a hugely negative outcome for 2⁄3 of humanity, much worse than extinction.
Regarding your second point, just a few thoughts:
First of all, an important point is how you think values and morality work. If two-thirds of humanity, after thorough reflection, disagree with your values, does this give you a reason to become less certain about your values as well? Maybe adopt their values to a degree? …
Secondly, I am also uncertain how coherent/convergent human values will be. There seem to be good arguments for both sides, see e.g. this blog post by Paul Christiano (and the discussion with Brian Tomasik in the comments of that post): https://rationalaltruist.com/2013/06/13/against-moral-advocacy/
Third: In a situation like the one you described above, at least there would be huge room for compromise/gains from trade/… So if future humanity would be split into the three factions you suggested, they would not necessarily fight a war until only one faction remains that can then tile the universe with their preferred version. Indeed, they probably would not, as cooperation seems better for everyone in expectation.
1) I agree that there is some confusion on my part, and on the part of most others I have spoken to, about how terminal values and morality do or do not get updated.
2) Agreed.
3) I will point to a maybe forthcoming paper / idea of Eric Drexler at FHI that makes this point, which he called “pareto-topia”. Despite the wonderful virtues of the idea, I’m unclear if there is a stable game-theoretic mechanism that prevents a race to the bottom outcome when fundamentally different values are being traded off. Specifically in this case, it’s possible that different values lead to an inability to truthfully/reliably cooperate—a paved road to pareto-topia seems not to exist, and there might be no path at all.