Most intelligent beings in the multiverse share similar preferences. This came about because there are facts about what preferences one should have, just like there exist facts about what decision theory one should use or what prior one should have, and species that manage to build intergalactic civilizations (or the equivalent in other universes) tend to discover all of these facts. There are occasional paperclip maximizers that arise, but they are a relatively minor presence or tend to be taken over by more sophisticated minds.
I think in this post you’re not giving enough attention to the possibility that there’s something that we call “doing philosophy” that can be used to discover all kinds of philosophical truths, and that you can’t become a truly powerful civilization without being able to “do philosophy” and be generally motivated by the results. Consider that philosophy seems to have helped the West become the dominant civilization on Earth, for example by inventing logic and science, and more recently have led to the discovery of ideas like acausal extortion/trade (which seem promising albeit still highly speculative). Of course I’m very uncertain of this and have little idea what “doing philosophy” actually consists of, but I’ve written a few more words on this topic if you’re interested.
Very interesting, Wei! Thanks a lot for the comment and the links.
TL;DR of my response: Your argument assumes that the first two conditions I list are met by default, which is I think a strong assumption (Part 1). Assuming that is the case, however, your point suggests there might be a selection effect favoring agents that act in accordance with the moral truth, which might be stronger than the selection effect I depict for values that are more expansion-conducive than the moral truth. This is something I haven’t seriously considered and this made me update! Nonetheless, for your argument to be valid and strong, theorthogonality thesis has to be almost completely false, and I think we need more solid evidence to challenge that thesis (Part 2).
Part 1: Strong assumption
This came about because there are facts about what preferences one should have, just like there exist facts about what decision theory one should use or what prior one should have, and species that manage to build intergalactic civilizations (or the equivalent in other universes) tend to discover all of these facts.
My understanding is that this scenario says the seven conditions I listed are met because it is actually trivial for a super-capable intergalactic civilization to meet those (or even required for it to become intergalactic in the first place, as you suggest later).
I think this is plausible for the following conditions:
#3 They find something they recognize as a moral truth.
#4 They (unconditionally) accept it, even if it is highly counterintuitive.
#5 The thing they found is actually the moral truth. No normative mistake.
#6 They succeed at acting in accordance with it. No practical mistake.
#7 They stick to this forever. No value drift.
You might indeed expect that the most powerful civs figure out how to overcome these challenges, and that those who don’t are left behind.[1] This is something I haven’t seriously considered before, so thanks!
However, recall the first two conditions:
There is a moral truth.
It is possible to “find it” and recognize it as such.
How capable a civilization is doesn’t matter when it comes to how likely these two are to be met. And while most metaethical debates focus only on 1, saying 1 is true is a much weaker claim than saying 1&2 is true (see, e.g., the naturalism vs non-naturalism controversy, which is I think only one piece of the puzzle).
Part 2: Challenging the orthogonality thesis
Then, you say that in this scenario you depict
There are occasional paperclip maximizers that arise, but they are a relatively minor presence or tend to be taken over by more sophisticated minds.
Maybe, but what I argue I that they are (occasional) “sophisticated minds” with values that are more expansion-conducive than the (potential) moral truth (e.g., because they have simple unconstrained goals such as “let’s just maximize for more life” or “for expansion itself”), and that they’re the ones who tend to take over.
But then you make this claim, which, if true, seems to sort of debunk my argument:
you can’t become a truly powerful civilization without being able to “do philosophy” and be generally motivated by the results.
(Given the context in your comment, I assume that by “being able to do philosophy”, you mean “being able to do things like finding the moral truth”.)
But I don’t think this claim is true.[1] However, you made me update and I might update more once I read the posts of yours that you linked! :)
I remain skeptical because this would imply the orthogonality thesis is almost completely false. Assuming there is a moral truth and that it is possible to “find” it and recognize it as such, I tentatively still believe that extremely powerful agents/civs with motivations misaligned with the moral truth are very plausible and not rare. You can at least imagine scenarios where they started aligned but then value drifted (without that making them significantly less powerful).
I remain skeptical because this would imply the orthogonality thesis is almost completely false.
The orthogonality thesis could be (and I think almost certainly is) false with respect to some agent-generating processes (e.g., natural selection) and true with respect to others (e.g. Q-learning).
My impression was that philosophers tended to disagree a lot on what moral truths are?
Consider that philosophy seems to have helped the West become the dominant civilization on Earth, for example by inventing logic and science
I’d argue that the process of Western civilization dominating the Earth was not a very moral process, and was actually pretty immoral, despite the presence of logic and science. It involved several genocides (in the Americas), colonization, the Two World Wars… In the process, some good things definitely happened (medicine progressing, for instance), but mostly for humans. The status of farmed animals seems to have consistently worsened with factory farming stepping in.
So I’d argue that was the West dominating the world happened because it was more powerful, not because it was more moral (see The End of the Magemachine by Fabian Scheidler for more on this topic).
In that view, science and logic matter because they allow you to have more power. They allow to have a more truthful picture of how the universe works, which allows making stuff like firearms and better boats and antibiotics and nuclear bombs. But this is the process of “civilizations competing with each other” described above. It’s not a comparison based on “who is acting closer to what is morally good”?
In Six Plausible Meta-Ethical Alternatives, I wrote (as one of the six alternatives):
I think in this post you’re not giving enough attention to the possibility that there’s something that we call “doing philosophy” that can be used to discover all kinds of philosophical truths, and that you can’t become a truly powerful civilization without being able to “do philosophy” and be generally motivated by the results. Consider that philosophy seems to have helped the West become the dominant civilization on Earth, for example by inventing logic and science, and more recently have led to the discovery of ideas like acausal extortion/trade (which seem promising albeit still highly speculative). Of course I’m very uncertain of this and have little idea what “doing philosophy” actually consists of, but I’ve written a few more words on this topic if you’re interested.
Very interesting, Wei! Thanks a lot for the comment and the links.
TL;DR of my response: Your argument assumes that the first two conditions I list are met by default, which is I think a strong assumption (Part 1). Assuming that is the case, however, your point suggests there might be a selection effect favoring agents that act in accordance with the moral truth, which might be stronger than the selection effect I depict for values that are more expansion-conducive than the moral truth. This is something I haven’t seriously considered and this made me update! Nonetheless, for your argument to be valid and strong, the orthogonality thesis has to be almost completely false, and I think we need more solid evidence to challenge that thesis (Part 2).
Part 1: Strong assumption
My understanding is that this scenario says the seven conditions I listed are met because it is actually trivial for a super-capable intergalactic civilization to meet those (or even required for it to become intergalactic in the first place, as you suggest later).
I think this is plausible for the following conditions:
#3 They find something they recognize as a moral truth.
#4 They (unconditionally) accept it, even if it is highly counterintuitive.
#5 The thing they found is actually the moral truth. No normative mistake.
#6 They succeed at acting in accordance with it. No practical mistake.
#7 They stick to this forever. No value drift.
You might indeed expect that the most powerful civs figure out how to overcome these challenges, and that those who don’t are left behind.[1] This is something I haven’t seriously considered before, so thanks!
However, recall the first two conditions:
There is a moral truth.
It is possible to “find it” and recognize it as such.
How capable a civilization is doesn’t matter when it comes to how likely these two are to be met. And while most metaethical debates focus only on 1, saying 1 is true is a much weaker claim than saying 1&2 is true (see, e.g., the naturalism vs non-naturalism controversy, which is I think only one piece of the puzzle).
Part 2: Challenging the orthogonality thesis
Then, you say that in this scenario you depict
Maybe, but what I argue I that they are (occasional) “sophisticated minds” with values that are more expansion-conducive than the (potential) moral truth (e.g., because they have simple unconstrained goals such as “let’s just maximize for more life” or “for expansion itself”), and that they’re the ones who tend to take over.
But then you make this claim, which, if true, seems to sort of debunk my argument:
(Given the context in your comment, I assume that by “being able to do philosophy”, you mean “being able to do things like finding the moral truth”.)
But I don’t think this claim is true.[1] However, you made me update and I might update more once I read the posts of yours that you linked! :)
I remain skeptical because this would imply the orthogonality thesis is almost completely false. Assuming there is a moral truth and that it is possible to “find” it and recognize it as such, I tentatively still believe that extremely powerful agents/civs with motivations misaligned with the moral truth are very plausible and not rare. You can at least imagine scenarios where they started aligned but then value drifted (without that making them significantly less powerful).
The orthogonality thesis could be (and I think almost certainly is) false with respect to some agent-generating processes (e.g., natural selection) and true with respect to others (e.g. Q-learning).
Do you have any reading to suggest on that topic? I’d be curious to understand that position more :)
My impression was that philosophers tended to disagree a lot on what moral truths are?
I’d argue that the process of Western civilization dominating the Earth was not a very moral process, and was actually pretty immoral, despite the presence of logic and science. It involved several genocides (in the Americas), colonization, the Two World Wars… In the process, some good things definitely happened (medicine progressing, for instance), but mostly for humans. The status of farmed animals seems to have consistently worsened with factory farming stepping in.
So I’d argue that was the West dominating the world happened because it was more powerful, not because it was more moral (see The End of the Magemachine by Fabian Scheidler for more on this topic).
In that view, science and logic matter because they allow you to have more power. They allow to have a more truthful picture of how the universe works, which allows making stuff like firearms and better boats and antibiotics and nuclear bombs. But this is the process of “civilizations competing with each other” described above. It’s not a comparison based on “who is acting closer to what is morally good”?