This is an interesting idea. I’m trying to think of it in terms of analogues: you could feasibly replace “digital minds” with “animals” and achieve a somewhat similar conclusion. It doesn’t seem that hard to create vast amounts of animal suffering (the animal agriculture industry has this figured out quite well), so some agent could feasibly threaten all vegans with large-scale animal suffering. And as you say, occasionally following through might help make that threat more credible.
Perhaps the reason we don’t see this happening is that nobody really wants to influence vegans alone. There aren’t many strategic reasons to target an unorganized group of people whose sole common characteristic is that they care about animals. There isn’t much that an agent could gain from a threat.
I imagine the same might be true of digital minds. If it’s anything similar to the animal case, moral circle expansion to digital minds will likely occur in the same haphazard, unorganized way—and so there wouldn’t be much of a reason to specifically target people who care about digital minds. That said, if this moral circle expansion caught on predominantly in one country (or maybe within one powerful company), a competitor or opponent might then have a real use for threatening the digital mind-welfarists. Such an unequal distribution of digital mind-welfarists seems quite unlikely, though.
At any rate, this might be a relevant consideration for other types of moral circle expansion, too.
I think it is useful to think about something like this happening in the current world like you did here because we have better intuitions about the current world. Someone could say that they will torture animals unless vegans give them money, I guess. I think this doesn’t happen for multiple reasons. One of them is that it would be irrational for vegans to agree to give money because then other people would continue exploiting them with this simple trick.
I think that the same applies to far future scenarios. If an agent allows itself to be manipulated this easily, it won’t become powerful. It’s more rational to just make it publicly known that you refuse to engage with such threats. This is one of the reasons why most Western countries have a publicly declared policy to not negotiate with terrorists. So yeah, thinking about it this way, I am no longer concerned about this threats thing.
Someone could say that they will torture animals unless vegans give them money, I guess. I think this doesn’t happen for multiple reasons.
Interestingly, there is at least one instance where this apparently has happened. (It’s possible it was just a joke, though.) There was even a law review article about the incident.
I think this is an interesting point but I’m not convinced that it’s true with high enough probability that the alternative isn’t worth considering.
In particular, I can imagine luck/happenstance to shake out enough that arbitrarily powerful agents on one dimension are less powerful/rational on other dimensions.
Another issue is the nature of precommitments[1]. It seems that under most games/simple decision theories for playing those games (eg “Chicken” in CDT), being the first to credibly precommit gives you a strategic edge under most circumstances. But if you’re second in those situations, it’s not clear whether “I don’t negotiate with terrorists” is a better or worse stance than swerving.
(And in the former case, with both sides precommitting, a lot of torture will still happen).
[1] using what I assume is the technical definition of precommitment
Other analogies might be human rights and carbon emissions, as used in politics. Say that Party A cares about reducing emissions, then the opposing Party B has an incentive to appear as though they don’t care about it at all and even propose actions that would increase emissions so that they could trade “not doing that” with some concession from Party A. I’m sure that we could find lots of real-world examples of that.
Similarly, some (totalitarian?) regimes might have some incentive to make major parts of the population politically conceived as unworthy and let them have a very poor lifestyle, so that other countries who care about that population would be open to trade where helping those people would be considered a benefit for those other countries.
This is an interesting idea. I’m trying to think of it in terms of analogues: you could feasibly replace “digital minds” with “animals” and achieve a somewhat similar conclusion. It doesn’t seem that hard to create vast amounts of animal suffering (the animal agriculture industry has this figured out quite well), so some agent could feasibly threaten all vegans with large-scale animal suffering. And as you say, occasionally following through might help make that threat more credible.
Perhaps the reason we don’t see this happening is that nobody really wants to influence vegans alone. There aren’t many strategic reasons to target an unorganized group of people whose sole common characteristic is that they care about animals. There isn’t much that an agent could gain from a threat.
I imagine the same might be true of digital minds. If it’s anything similar to the animal case, moral circle expansion to digital minds will likely occur in the same haphazard, unorganized way—and so there wouldn’t be much of a reason to specifically target people who care about digital minds. That said, if this moral circle expansion caught on predominantly in one country (or maybe within one powerful company), a competitor or opponent might then have a real use for threatening the digital mind-welfarists. Such an unequal distribution of digital mind-welfarists seems quite unlikely, though.
At any rate, this might be a relevant consideration for other types of moral circle expansion, too.
I think it is useful to think about something like this happening in the current world like you did here because we have better intuitions about the current world. Someone could say that they will torture animals unless vegans give them money, I guess. I think this doesn’t happen for multiple reasons. One of them is that it would be irrational for vegans to agree to give money because then other people would continue exploiting them with this simple trick.
I think that the same applies to far future scenarios. If an agent allows itself to be manipulated this easily, it won’t become powerful. It’s more rational to just make it publicly known that you refuse to engage with such threats. This is one of the reasons why most Western countries have a publicly declared policy to not negotiate with terrorists. So yeah, thinking about it this way, I am no longer concerned about this threats thing.
Interestingly, there is at least one instance where this apparently has happened. (It’s possible it was just a joke, though.) There was even a law review article about the incident.
I think this is an interesting point but I’m not convinced that it’s true with high enough probability that the alternative isn’t worth considering.
In particular, I can imagine luck/happenstance to shake out enough that arbitrarily powerful agents on one dimension are less powerful/rational on other dimensions.
Another issue is the nature of precommitments[1]. It seems that under most games/simple decision theories for playing those games (eg “Chicken” in CDT), being the first to credibly precommit gives you a strategic edge under most circumstances. But if you’re second in those situations, it’s not clear whether “I don’t negotiate with terrorists” is a better or worse stance than swerving.
(And in the former case, with both sides precommitting, a lot of torture will still happen).
[1] using what I assume is the technical definition of precommitment
Interesting!
Other analogies might be human rights and carbon emissions, as used in politics. Say that Party A cares about reducing emissions, then the opposing Party B has an incentive to appear as though they don’t care about it at all and even propose actions that would increase emissions so that they could trade “not doing that” with some concession from Party A. I’m sure that we could find lots of real-world examples of that.
Similarly, some (totalitarian?) regimes might have some incentive to make major parts of the population politically conceived as unworthy and let them have a very poor lifestyle, so that other countries who care about that population would be open to trade where helping those people would be considered a benefit for those other countries.