“There are more things in heaven and earth, Horatio, Than are dreamt of in your philosophy”
One thing I’ve been floating about for a while, and haven’t really seen anybody else deeply explore[1], is what I call “further moral goods”: further axes of moral value as yet inaccessible to us, that is qualitatively not just quantitatively different from anything we’ve observed to date.
The physical world: matter, energy, atoms, stars, cells. An detached external observer might think that’s all there is to our universe.
The mathematical world. Mathematics, logic, abstract structure, rationality, “natural laws.” Even many otherwise-strict “materialists” can see how the mathematical world is conceptually distinct from the physical one: mathematical truths seem conceptually different and perhaps deeper than mere physical facts. And if you’re a robot/present-day LLM, you might just live in the first two worlds[2]. Some Kantians try to ground morality entirely within this world, in the logic of cooperation and strategic interaction.
The world of consciousness. The experiential realm. Qualia, subjective experience, “what it’s like to be me.” Most secular moral philosophers treat this as where the real moral action is. A pure hedonic utilitarian might think conscious experience is the only thing that matters, but even other moral philosophies would consider conscious experience extremely important (usually the most important).
For the purposes of this post, I’m not that interested in the delineating between whether these worlds are truly different or just conceptually interesting ways to talk about things (ie I’m not positing a strong position on mathematical platonism or consciousness dualism)
But what’s interesting to me is how these different worlds ground morality/value, what some philosophers would call “axiology.” When people try to solely ground morality in the first two worlds, and even more so when people try to ground morality in the first world alone[3], deep believers in all three worlds (which I think is most people, and most philosophers) think they’re entirely missing the point! It seems almost self-evident that conscious experience is much more important than the arrangement of mere rocks, or bloodless abstract game theory of feeling-less zombies!
But are these the only 3 worlds? Is it possible to have other morally relevant worlds, and in particular worlds that will self-evidently seem so much more important than subjective experience if only we know about them?
Perhaps.
For example, (most) religious people believe they have an answer, :
The supernatural world. The world of spirits, Gods, heavens and hells. Religious traditions often claim that divine or transcendent value is qualitatively, not just quantitatively, superior to natural goods. Saying that “heaven is infinite bliss” is a secular/materialist approximation of something much deeper. (Other handles: the ineffable, the sublime)
Now I think the religious people are wrong about the world as we see it today. But do we have strong reason to think that the three worlds as we know them are the only ones left? I think no.
In particular, we have two distinct reasons to think future intelligences can discover other worlds:
A. AIs, including future AIs, will be a distinct type of mind(s) than human mind(s). Just as most people today believe that humans (and other animals) have qualia that present-day AIs do not have, we should also think it’s plausible that different mental architectures in AI will allow them to have moral goods that we cannot experience or perhaps even conceive.
B. Superintelligences (likely digital intelligences, though in theory could also be our posthuman descendants) will be able to search for further moral goods. At some point in the future (if we don’t all die first), it will become trivial to spend more brainpower than has ever existed in all of human science and philosophy combined to search for other sources of moral value. This can come from engineering unique environmental arrangements of matter, unique structures of minds, or something else entirely.
So one day our descendants may discover worlds five, six, and so on: sources of moral value qualitatively distinct and superior to what we have access to, in the same way that grounding morality purely in game theory or entropy feels foolish to most experiencing humans today.
If true, this is a big deal! [4]
This seems overall quite possible to me. But is it probable?
I don’t have a good sense of high likely this all is. Trying to estimate it feels beyond my forecasting or philosophical competence. But it seems plausible enough, and interesting enough, that I wanted to bring it to people’s attention, in case other people have ideas on how to extend it.
Appendix A:
Existing literature: This concept is widespread but undertheorized. Mill’s qualitative distinction among pleasures can point us in this direction; Bostrom’s “Letter from Utopia” is the most vivid articulation (“What I feel is as far beyond feelings as what I think is beyond thoughts”); Danaher (2021) coined “axiological possibility space”; Ord’s The Precipice argues we have “barely begun the ascent” and our investigations of flourishing may be “like astronomy before telescopes.” According to a search from Claude, Nagel, Jackson, and Chalmers “collectively demonstrate that the space of possible conscious experiences vastly exceeds human experience.” Banks’s concept of Subliming: where “the very ideas, the actual concepts of good, of fairness and of justice just ceased to matter”, is the most philosophically precise depiction I’ve seen in science fiction.
[1] Though I’ve seen shades of it in academic philosophy, EA/longtermist writing, science fiction/fantasy, and discussions of religion
[2] This is disputed.
[3] eg entropy as the guiding factor of morality, a la Beff Jezos.
[4] And if false, but convincing enough to be an attractor state for our descendants, this will sadly also be a very big deal.
One reason to think we might not find anything morally valuable that distinct from what we already know about is that our concept of morality is made to fit with the stuff we already know about.
Agreed. It’s possible that we/our descendants won’t see much value for extending past blissful experiences even when other axes of value are theoretically possible, in the same way that aliens without conscious experiences would not see any particular reason to privilege qualia (even if they could be convinced that it’s real).
“There are more things in heaven and earth, Horatio, Than are dreamt of in your philosophy”
One thing I’ve been floating about for a while, and haven’t really seen anybody else deeply explore[1], is what I call “further moral goods”: further axes of moral value as yet inaccessible to us, that is qualitatively not just quantitatively different from anything we’ve observed to date.
For background, I think normal, secular, humans live in 3 conceptually distinct but overlapping worlds:
The physical world: matter, energy, atoms, stars, cells. An detached external observer might think that’s all there is to our universe.
The mathematical world. Mathematics, logic, abstract structure, rationality, “natural laws.” Even many otherwise-strict “materialists” can see how the mathematical world is conceptually distinct from the physical one: mathematical truths seem conceptually different and perhaps deeper than mere physical facts. And if you’re a robot/present-day LLM, you might just live in the first two worlds[2]. Some Kantians try to ground morality entirely within this world, in the logic of cooperation and strategic interaction.
The world of consciousness. The experiential realm. Qualia, subjective experience, “what it’s like to be me.” Most secular moral philosophers treat this as where the real moral action is. A pure hedonic utilitarian might think conscious experience is the only thing that matters, but even other moral philosophies would consider conscious experience extremely important (usually the most important).
For the purposes of this post, I’m not that interested in the delineating between whether these worlds are truly different or just conceptually interesting ways to talk about things (ie I’m not positing a strong position on mathematical platonism or consciousness dualism)
But what’s interesting to me is how these different worlds ground morality/value, what some philosophers would call “axiology.” When people try to solely ground morality in the first two worlds, and even more so when people try to ground morality in the first world alone[3], deep believers in all three worlds (which I think is most people, and most philosophers) think they’re entirely missing the point! It seems almost self-evident that conscious experience is much more important than the arrangement of mere rocks, or bloodless abstract game theory of feeling-less zombies!
But are these the only 3 worlds? Is it possible to have other morally relevant worlds, and in particular worlds that will self-evidently seem so much more important than subjective experience if only we know about them?
Perhaps.
For example, (most) religious people believe they have an answer, :
The supernatural world. The world of spirits, Gods, heavens and hells. Religious traditions often claim that divine or transcendent value is qualitatively, not just quantitatively, superior to natural goods. Saying that “heaven is infinite bliss” is a secular/materialist approximation of something much deeper. (Other handles: the ineffable, the sublime)
Now I think the religious people are wrong about the world as we see it today. But do we have strong reason to think that the three worlds as we know them are the only ones left? I think no.
In particular, we have two distinct reasons to think future intelligences can discover other worlds:
A. AIs, including future AIs, will be a distinct type of mind(s) than human mind(s). Just as most people today believe that humans (and other animals) have qualia that present-day AIs do not have, we should also think it’s plausible that different mental architectures in AI will allow them to have moral goods that we cannot experience or perhaps even conceive.
B. Superintelligences (likely digital intelligences, though in theory could also be our posthuman descendants) will be able to search for further moral goods. At some point in the future (if we don’t all die first), it will become trivial to spend more brainpower than has ever existed in all of human science and philosophy combined to search for other sources of moral value. This can come from engineering unique environmental arrangements of matter, unique structures of minds, or something else entirely.
So one day our descendants may discover worlds five, six, and so on: sources of moral value qualitatively distinct and superior to what we have access to, in the same way that grounding morality purely in game theory or entropy feels foolish to most experiencing humans today.
If true, this is a big deal! [4]
This seems overall quite possible to me. But is it probable?
I don’t have a good sense of high likely this all is. Trying to estimate it feels beyond my forecasting or philosophical competence. But it seems plausible enough, and interesting enough, that I wanted to bring it to people’s attention, in case other people have ideas on how to extend it.
Appendix A:
Existing literature: This concept is widespread but undertheorized. Mill’s qualitative distinction among pleasures can point us in this direction; Bostrom’s “Letter from Utopia” is the most vivid articulation (“What I feel is as far beyond feelings as what I think is beyond thoughts”); Danaher (2021) coined “axiological possibility space”; Ord’s The Precipice argues we have “barely begun the ascent” and our investigations of flourishing may be “like astronomy before telescopes.” According to a search from Claude, Nagel, Jackson, and Chalmers “collectively demonstrate that the space of possible conscious experiences vastly exceeds human experience.” Banks’s concept of Subliming: where “the very ideas, the actual concepts of good, of fairness and of justice just ceased to matter”, is the most philosophically precise depiction I’ve seen in science fiction.
[1] Though I’ve seen shades of it in academic philosophy, EA/longtermist writing, science fiction/fantasy, and discussions of religion
[2] This is disputed.
[3] eg entropy as the guiding factor of morality, a la Beff Jezos.
[4] And if false, but convincing enough to be an attractor state for our descendants, this will sadly also be a very big deal.
wrote it up in more detail on substack: https://linch.substack.com/p/further-moral-goods
One reason to think we might not find anything morally valuable that distinct from what we already know about is that our concept of morality is made to fit with the stuff we already know about.
Agreed. It’s possible that we/our descendants won’t see much value for extending past blissful experiences even when other axes of value are theoretically possible, in the same way that aliens without conscious experiences would not see any particular reason to privilege qualia (even if they could be convinced that it’s real).