This is missing a very important point, which is that I think humans have morally relevant experience and I’m not confident that misaligned AIs would. When the next generation replaces the current one this is somewhat ok because those new humans can experience joy, wonder, adventure etc. My best guess is that AIs that take over and replace humans would not have any morally relevant experience, and basically just leave the universe morally empty. (Note that this might be an ok outcome if by default you expect things to be net negative)
I also think that there is way more overlap in the “utility functions” between humans, than between humans and misaligned AIs. Most humans feel empathy and don’t want to cause others harm. I think humans would generally accept small costs to improve the lives of others, and a large part of why people don’t do this is because people have cognitive biases or aren’t thinking clearly. This isn’t to say that any random human would reflectively become a perfectly selfless total utilitarian, but rather that most humans do care about the wellbeing of other humans. By default, I don’t think misaligned AIs will really care about the wellbeing of humans at all.
My best guess is that AIs that take over and replace humans would not have any morally relevant experience, and basically just leave the universe morally empty.
I don’t think that’s particularly likely, but I can understand if you think this is an important crux.
For what it’s worth, I don’t think it matters as much whether the AIs themselves are sentient, but rather whether they care about sentience. For example, from the perspective of sentience, humans weren’t necessarily a great addition to the world, because of their contribution to suffering in animal agriculture (although I’m not giving a confident take here).
Even if AIs are not sentient, they’ll still be responsible for managing the world, and creating structures in the universe. When this happens, there’s a lot of ways for sentience to come about, and I care more about the lower level sentience that the AI manages than the actual AIs at the top who may or may not be sentient.
This is missing a very important point, which is that I think humans have morally relevant experience and I’m not confident that misaligned AIs would. When the next generation replaces the current one this is somewhat ok because those new humans can experience joy, wonder, adventure etc. My best guess is that AIs that take over and replace humans would not have any morally relevant experience, and basically just leave the universe morally empty. (Note that this might be an ok outcome if by default you expect things to be net negative)
I also think that there is way more overlap in the “utility functions” between humans, than between humans and misaligned AIs. Most humans feel empathy and don’t want to cause others harm. I think humans would generally accept small costs to improve the lives of others, and a large part of why people don’t do this is because people have cognitive biases or aren’t thinking clearly. This isn’t to say that any random human would reflectively become a perfectly selfless total utilitarian, but rather that most humans do care about the wellbeing of other humans. By default, I don’t think misaligned AIs will really care about the wellbeing of humans at all.
I don’t think that’s particularly likely, but I can understand if you think this is an important crux.
For what it’s worth, I don’t think it matters as much whether the AIs themselves are sentient, but rather whether they care about sentience. For example, from the perspective of sentience, humans weren’t necessarily a great addition to the world, because of their contribution to suffering in animal agriculture (although I’m not giving a confident take here).
Even if AIs are not sentient, they’ll still be responsible for managing the world, and creating structures in the universe. When this happens, there’s a lot of ways for sentience to come about, and I care more about the lower level sentience that the AI manages than the actual AIs at the top who may or may not be sentient.