Suffering risks. S-risks are arguably a far more serious issue than reducing the risk of extinction, as the scope of the suffering could be infinite. The fact that there is a risk of a maligned superintelligence creating a hellish dystopia on a cosmic scale with more intense suffering than has ever existed in history means that even if the risk of this happening is small, this is balanced by its extreme disutility. S-risks are also highly neglected, relative to their potential extreme disutility. It could even be argues that it would be rational to completely dedicate your life to reducing S-risks because of this. The only organizations I’m aware of that are directly working on reducing S-risks are the Center on Long-Term Risk and the Center for Reducing Suffering. One possible way AI could lead to astronomical suffering is if there is a “near miss” in AI alignment, where the AI alignment problem is partially solved, but not entirely. Other potential sources of S-risks may include malevolence, or an AI that includes religious hells when aligned to reflect the values of humanity.
Other S-risks that may or may not sound more plausible are suffering simulations (maybe an AI comes to the conclusion that a good way to study humans is to simulate earth at the time of the Black Death) or suffering subroutines (maybe reinforcement learners that are able to suffer enable faster or more efficient algorithms).
FWIW, infinities could go either way if you recognize moral goods that can aggregate by summing. I think where infinities seem more likely for suffering than goods are if your views are ethically asymmetric and assign more weight to suffering, especially some kinds of suffering being infinitely bad, but no goods being infinitely good (or no goods at all), or goods only being able to offset but not outweigh bads.
To preface my criticism I’ll say I think concrete ways that AI may cause great suffering do deserve attention.
But:
the scope is surely not infinite. The heat death of the universe and the finite number of atoms in it pose a limit.
Unless you think unaligned AIs will somehow be inclined to not only ignore what people want, but actually keep them alive and torture them—which sounds implausible to me—how’s this not Pascal’s mugging?
the scope is surely not infinite. The heat death of the universe and the finite number of atoms in it pose a limit.
We can’t say for certain that travel to other universes is impossible, so we can’t rule it out as a theoretical possibility. As for the heat death if the universe, Alexey Turchin created thischart of theoretical ways that the heat death of the universe could be survivable by our descendants.
Unless you think unaligned AIs will somehow be inclined to not only ignore what people want, but actually keep them alive and torture them—which sounds implausible to me—how’s this not Pascal’s mugging?
The entities that are being subjected to the torture wouldn’t necessarily be “people” per se. I am talking about conscious entities in general. Solving the alignment problem from the perspective of hedonistic utilitarianism would involve the superintelligence having consciousness-centric values and the ability to create and preserve conscious states with high levels of valence. If a superintelligence with consciousness-centric values that can create large amounts of bliss is realistically possible, the possibility of a consciousness-centric superintelligence that creates large amounts of suffering isn’t necessarily that much less realistic. If you believe that a superintelligence causing torture is implausible, you also have to accept that a superintelligence creating a utopia is also implausible.
It should be mentioned that all (or at least most) ideas to survive the heat death of the universe involve speculative physics. Moreover, you have to deal with infinities. If everyone is suffering but there is one sentient being that experiences a happy moment every million years, does this mean that there is an infinite amount of suffering and an infite amount of happiness and the future is of neutral value? If any future with an infinite amount of suffering is bad, does this mean that it is good if sentient life does not exists forever? There is no obvious answer to these questions.
We can’t say for certain that travel to other universes is impossible, so we can’t rule it out as a theoretical possibility.
How’s this argument different from saying, for example, that we can’t rule out God’s existence so we should take him into consideration? Or that we can’t rule out the possibility of the universe being suddenly magically replaced with a utilitarian optional one?
Alexey Turchin created this chart of theoretical ways that the heat death of the universe could be survivable by our descendants.
The linked post is basically a definition of what “survival” means, without any argument on how any of it is at all plausible.
If you believe that a superintelligence causing torture is implausible, you also have to accept that a superintelligence creating a utopia is also implausible.
How’s this argument different from saying, for example, that we can’t rule out God’s existence so we should take him into consideration? Or that we can’t rule out the possibility of the universe being suddenly magically replaced with a utilitarian optional one?
If you want to reduce the risk of going to some form of hell as much as possible, you ought to determine what sorts of “hells” have the highest probability of existing, and to what extent avoiding said hells is tractable. As far as I can tell, the “hells” that seem to be the most realistic are hells resulting from bad AI alignment, and hells resulting from living in a simulation. Hells resulting from bad AI alignment can be plausibly avoided by contributing in some way to solving the AI alignment problem. It’s not clear how hells resulting from living in a simulation could be avoided, but it’s possible that ways to avoid these sorts of hells could be discovered with further analysis of different theoretical types of simulations we may be living in, such as in thismap. Robin Hanson explored some of the potential utilitarian implications of the simulation hypothesis in his article How To Live In A Simulation. Furthermore, mind enhancement could potentially reduce S-risks. If you manage to improve your general thinking abilities, you could potentially discover a new way to reduce S-risks.
A Christian or a Muslim could argue that you ought to convert to their religions in order to avoid going to hell. But a problem with Pascal’s Wager-type arguments is the issue of tradeoffs. It’s not clear that practicing a religion is the most optimal way to avoid hell/S-risks. The time spent going to church, praying, and otherwise being dedicated to your religion is time not spent thinking about AI safety and strategizing ways to avoid S-risks. Working on AI safety, strategizing ways to avoid S-risks, and trying to improve your thinking abilities would probably be more effective at reducing your risk of going to some sort of hell than, say, converting to Christianity would.
The linked post is basically a definition of what “survival” means, without any argument on how any of it is at all plausible.
It mentions finding ways to travel to other universes, send information to other universes, creating a superintelligence to figure out ways to avoid heat death, convincing the creators of the simulation to not turn it off, etc. While these hypothetical ways to survive heat death do involve a lot of speculative physics, they are more than just “defining survival”.
I believe neither is plausible by mistake.
Yet we live in a reality where happiness and suffering exist seemingly by mistake. Your nervous system is the result of millions of years of evolution, not the result of an intelligent designer.
Suffering risks. S-risks are arguably a far more serious issue than reducing the risk of extinction, as the scope of the suffering could be infinite. The fact that there is a risk of a maligned superintelligence creating a hellish dystopia on a cosmic scale with more intense suffering than has ever existed in history means that even if the risk of this happening is small, this is balanced by its extreme disutility. S-risks are also highly neglected, relative to their potential extreme disutility. It could even be argues that it would be rational to completely dedicate your life to reducing S-risks because of this. The only organizations I’m aware of that are directly working on reducing S-risks are the Center on Long-Term Risk and the Center for Reducing Suffering. One possible way AI could lead to astronomical suffering is if there is a “near miss” in AI alignment, where the AI alignment problem is partially solved, but not entirely. Other potential sources of S-risks may include malevolence, or an AI that includes religious hells when aligned to reflect the values of humanity.
Other S-risks that may or may not sound more plausible are suffering simulations (maybe an AI comes to the conclusion that a good way to study humans is to simulate earth at the time of the Black Death) or suffering subroutines (maybe reinforcement learners that are able to suffer enable faster or more efficient algorithms).
FWIW, infinities could go either way if you recognize moral goods that can aggregate by summing. I think where infinities seem more likely for suffering than goods are if your views are ethically asymmetric and assign more weight to suffering, especially some kinds of suffering being infinitely bad, but no goods being infinitely good (or no goods at all), or goods only being able to offset but not outweigh bads.
To preface my criticism I’ll say I think concrete ways that AI may cause great suffering do deserve attention.
But:
the scope is surely not infinite. The heat death of the universe and the finite number of atoms in it pose a limit.
Unless you think unaligned AIs will somehow be inclined to not only ignore what people want, but actually keep them alive and torture them—which sounds implausible to me—how’s this not Pascal’s mugging?
We can’t say for certain that travel to other universes is impossible, so we can’t rule it out as a theoretical possibility. As for the heat death if the universe, Alexey Turchin created this chart of theoretical ways that the heat death of the universe could be survivable by our descendants.
The entities that are being subjected to the torture wouldn’t necessarily be “people” per se. I am talking about conscious entities in general. Solving the alignment problem from the perspective of hedonistic utilitarianism would involve the superintelligence having consciousness-centric values and the ability to create and preserve conscious states with high levels of valence. If a superintelligence with consciousness-centric values that can create large amounts of bliss is realistically possible, the possibility of a consciousness-centric superintelligence that creates large amounts of suffering isn’t necessarily that much less realistic. If you believe that a superintelligence causing torture is implausible, you also have to accept that a superintelligence creating a utopia is also implausible.
It should be mentioned that all (or at least most) ideas to survive the heat death of the universe involve speculative physics. Moreover, you have to deal with infinities. If everyone is suffering but there is one sentient being that experiences a happy moment every million years, does this mean that there is an infinite amount of suffering and an infite amount of happiness and the future is of neutral value? If any future with an infinite amount of suffering is bad, does this mean that it is good if sentient life does not exists forever? There is no obvious answer to these questions.
How’s this argument different from saying, for example, that we can’t rule out God’s existence so we should take him into consideration? Or that we can’t rule out the possibility of the universe being suddenly magically replaced with a utilitarian optional one?
The linked post is basically a definition of what “survival” means, without any argument on how any of it is at all plausible.
I believe neither is plausible by mistake.
If you want to reduce the risk of going to some form of hell as much as possible, you ought to determine what sorts of “hells” have the highest probability of existing, and to what extent avoiding said hells is tractable. As far as I can tell, the “hells” that seem to be the most realistic are hells resulting from bad AI alignment, and hells resulting from living in a simulation. Hells resulting from bad AI alignment can be plausibly avoided by contributing in some way to solving the AI alignment problem. It’s not clear how hells resulting from living in a simulation could be avoided, but it’s possible that ways to avoid these sorts of hells could be discovered with further analysis of different theoretical types of simulations we may be living in, such as in this map. Robin Hanson explored some of the potential utilitarian implications of the simulation hypothesis in his article How To Live In A Simulation. Furthermore, mind enhancement could potentially reduce S-risks. If you manage to improve your general thinking abilities, you could potentially discover a new way to reduce S-risks.
A Christian or a Muslim could argue that you ought to convert to their religions in order to avoid going to hell. But a problem with Pascal’s Wager-type arguments is the issue of tradeoffs. It’s not clear that practicing a religion is the most optimal way to avoid hell/S-risks. The time spent going to church, praying, and otherwise being dedicated to your religion is time not spent thinking about AI safety and strategizing ways to avoid S-risks. Working on AI safety, strategizing ways to avoid S-risks, and trying to improve your thinking abilities would probably be more effective at reducing your risk of going to some sort of hell than, say, converting to Christianity would.
It mentions finding ways to travel to other universes, send information to other universes, creating a superintelligence to figure out ways to avoid heat death, convincing the creators of the simulation to not turn it off, etc. While these hypothetical ways to survive heat death do involve a lot of speculative physics, they are more than just “defining survival”.
Yet we live in a reality where happiness and suffering exist seemingly by mistake. Your nervous system is the result of millions of years of evolution, not the result of an intelligent designer.