I believe that artificial intelligence and other resources will have diminishing marginal utility. If that’s the case, then it should be possible to do so that both humans and future digital minds are close to their maximum level of utility/happiness by splitting resources between them. The idea of “both digital minds and humans are important” would be therefore significantly more attractive for humans and realistic to accomplish than the idea of “digital minds are all of what matters”.
On moral patience:
Morality is subjective.
The criteria for moral patience are subjective. You can say that a human is a moral patient, you can say that AI agent is a moral patient, or you can say that a stone is a moral patient. There’s no objective truth about that.
However, we can define moral patience as “if I want to maximize my happiness/utility, whose happiness/utility should I care about when choosing my actions”? That is objective.
If a human wants to maximize their happiness, who should they care about?
Firstly, we need to answer what are the reasons why a human would have an interest to care about someone/something else.
The reasons are:
Game-theoretic—especially direct and indirect reciprocity.
Importantly, I think that agents have interest to care about weaker agents (ask me if you want to know why).
I also think that it’s in the interest of a human to care about other sentient life, at least to some extent, because:
It’s a norm that is useful—if we live by that norm, then we have that guarantee that we won’t end up terribly, because as long as we are sentient, someone will care about us.
It’s a norm that has a historical precedent and it’s easier to sustain a norm than to establish a new one. But the historical precedent is that humans care about existing sentient life, not potential sentient life.
If the norm of respecting sentient life implied that humans have to commit all their resources for the benefit of future digital minds / AI agents, then that norm would stop to be useful to humans, so that norm would either become weaker to the point where human lives matter, or humans would stop creating new AI agents.
Emotional—you care about someone because you simply want them to be happy.
I think it’s likely that there are game-theoretic and/or emotional reasons why we should care about AI agents / digital beings, but not to the point that they are all that we should care about and that we should ignore other agents or beings.
On number of agents:
How to count the number of agents? If we have multiple agents on different computers that cooperate and exchange knowledge, and then we have one agent that is distributed among many computers, then what’s the difference between them? Does that count as one agent or many?
Let’s suppose that we define agents by their goals (i.e. two computers with AI on it are considered different agents, if they aim to achieve different goals). If there are multiple agents with different goals (either human or AIs), then the best thing that they can do is to create one AI agent that will be optimized for the cumulative utility/happiness all of them. So, I expect that there won’t be many separate agents with different goals in the future.
My thoughts.
On implications of diminishing marginal utility:
I believe that artificial intelligence and other resources will have diminishing marginal utility. If that’s the case, then it should be possible to do so that both humans and future digital minds are close to their maximum level of utility/happiness by splitting resources between them. The idea of “both digital minds and humans are important” would be therefore significantly more attractive for humans and realistic to accomplish than the idea of “digital minds are all of what matters”.
On moral patience:
Morality is subjective.
The criteria for moral patience are subjective. You can say that a human is a moral patient, you can say that AI agent is a moral patient, or you can say that a stone is a moral patient. There’s no objective truth about that.
However, we can define moral patience as “if I want to maximize my happiness/utility, whose happiness/utility should I care about when choosing my actions”? That is objective.
If a human wants to maximize their happiness, who should they care about?
Firstly, we need to answer what are the reasons why a human would have an interest to care about someone/something else.
The reasons are:
Game-theoretic—especially direct and indirect reciprocity.
Importantly, I think that agents have interest to care about weaker agents (ask me if you want to know why).
I also think that it’s in the interest of a human to care about other sentient life, at least to some extent, because:
It’s a norm that is useful—if we live by that norm, then we have that guarantee that we won’t end up terribly, because as long as we are sentient, someone will care about us.
It’s a norm that has a historical precedent and it’s easier to sustain a norm than to establish a new one. But the historical precedent is that humans care about existing sentient life, not potential sentient life.
If the norm of respecting sentient life implied that humans have to commit all their resources for the benefit of future digital minds / AI agents, then that norm would stop to be useful to humans, so that norm would either become weaker to the point where human lives matter, or humans would stop creating new AI agents.
Emotional—you care about someone because you simply want them to be happy.
I think it’s likely that there are game-theoretic and/or emotional reasons why we should care about AI agents / digital beings, but not to the point that they are all that we should care about and that we should ignore other agents or beings.
On number of agents:
How to count the number of agents? If we have multiple agents on different computers that cooperate and exchange knowledge, and then we have one agent that is distributed among many computers, then what’s the difference between them? Does that count as one agent or many?
Let’s suppose that we define agents by their goals (i.e. two computers with AI on it are considered different agents, if they aim to achieve different goals). If there are multiple agents with different goals (either human or AIs), then the best thing that they can do is to create one AI agent that will be optimized for the cumulative utility/happiness all of them. So, I expect that there won’t be many separate agents with different goals in the future.