The you that chooses is more fundamental than the you that experiences, because if you remove experience you get a blindmind you that will presumably want it back. Even if it can’t be gotten back, presumably **you **will still pursue your values whatever they were. On the other hand, if you remove your entire algorithm but leave the qualia, you get an empty observer that might not be completely lacking in value, but wouldn’t be you, and if you then replace the algorithm you get a sentient someone else.
Thus I submit that moral patients are straightforwardly the agents, while sentience is something that they can have and use.
If there is an agent that lost its qualia and wants to get them back, then I (probably) want to help it get them back, because I (probably) value qualia myself in an altruistic way. On the other hand, if there is a blindmind agent that doesn’t have or care about qualia, and just wants to make paperclips or whatever, then I (probably) don’t want to help them do that (except instrumentally, if doing so helps my own goals). It seems like you’re implicitly trying to make me transfer my intuitions from the former to the latter, by emphasizing the commonalities (they’re both agents) and ignoring the differences (one cares about something I also care about, the other doesn’t), which I think is an invalid move.
Apologies if I’m being uncharitable or misinterpreting you, but aside from this, I really don’t see what other logic or argumentative force is supposed to make me, after reading your first paragraph, reach the conclusion in your second paragraph, i.e., decide that I now want to value/help all agents, including blindminds that just want to make paperclips. If you have something else in mind, please spell it out more?
Regarding paperclippers: in addition to what Pumo said in their reply concerning mutual alignment (and what will be said in Part 2), I’d say that stupid goals are stupid goals independent of sentience. I wouldn’t altruistically devote resources to helping a non-sentient AI make a billion paperclips for the exact same reason that I wouldn’t altruistically devote resources to helping some human autist make a billion paperclips. Maybe I’m misunderstanding your objection; possibly your objection is something more like “if we unmoor moral value from qualia, there’s nothing left to ground it in and the result is absurdity”. For now I’ll just say that we are definitely not asserting “all agents have equal moral status”, or “all goals are equal/interchangeable” (indeed, Part 2 asserts the opposite).
Regarding ‘why should I care about the blindmind for-its-own-sake?’, here’s another way to get there:
My understanding is that there are two primary competing views which assign non-sentient agents zero moral status:
1. Choice/preference (even of sentient beings) of a moral patient isn’t inherently relevant, qualia valence is everything (e.g. we should tile the universe in hedonium), e.g. hedonic utilitarianism. We don’t address this view much in the essay, except to point out that a) most people don’t believe this, and b) to whatever extent that this view results in totalizing hegemonic behavior, it sets the stage for mass conflict between believers of it and opponents of it (presumably including non-sentient agents). If we accept a priori something like ‘positive valenced qualia is the only fundamental good’, this view might at least be internally consistent (at which point I can only argue against the arbitrariness of accepting the a priori premise about qualia or argue against any system of values that appears to lead to it’s own defeat timelessly, though I realize the latter is a whole object-level debate to be had in and of itself).
2. Choice/preference fundamentally matters, but only if the chooser is also an experiencer. This seems like the more common view, e.g. (sentientist) preference utilitarianism. a) I think it can be shown that the ‘experience’ part of that is not load-bearing in a justifiable way, which I’ll address more below, and b) this suffers from the same practical problem as (b) above, though less severely.
#2 raises the question of “why would choice/preference-satisfaction have value independent of any particular valenced qualia, but not have value independent of qualia more generally?”. If you posit that the value of preference-satisfaction is wholly instrumental insofar as having your preferences satisfied generates some positive-valence qualia, this view just collapses back to #1. If you instead hold that preference-satisfaction is inherently (non-instrumentally) morally valuable by virtue of the preference-holder having qualia, even if only neutral-valenced qualia with regard to the preference… why? What morally-relevant work is the qualia doing and/or what is the posited morally-relevant connection between the (neutral) qualia and the preference? Why would the (valence-neutral) satisfaction of the preference have more value than just some other neutral qualia existing on it’s own while the preference goes un-satisfied? It seems to me like at this point we aren’t talking about ‘feelings’ (which exist in the domain of qualia) anymore, we’re talking about ‘choices and preferences’ (which are concepts within the domain of agency). To restate: The neutral qualia sitting alongside the preference or the satisfaction of the preference isn’t doing anything, or else (if the existence of the qualia is what is actually important) the preference itself doesn’t appear to be *doing* anything in which case preference satisfaction is not inherently important.
So the argument is that if you already attribute value to the preference satisfaction of sentient beings independent of their valenced qualia (e.g. if you would, for non-instrumental reasons, respect the stated preferences of some stranger even if you were highly confident that you could induce higher net-positive valence in them by not respecting their preferences), then you are already valuing something that really has nothing to do with their qualia. And thus there’s not as large of an intuition gap on the way to agential moral value as there at first seemed to be. Granted, this only applies to anyone who valued preference satisfaction in the first place.
Your interpretation isn’t exactly wrong, I’m proposing an onthological shift on the understanding of what’s more central to the self, the thing to care about (i.e. is the moral patient fundamentally a qualia that has or can have an agent, or an agent that has or can have qualia?).
The intuition is that if qualia, on its own, is generic and completely interchangeable among moral patients, it might not be what makes them such, even if it’s an important value. A blindmind upload has ultimately far more in common with the sentient person they are based on than said person has with a phenomenal experience devoid of all the content that makes up their agency.
Thus it would be the agent the thing that primarily values the qualia (and everything else), rather than the reverse. This decenters qualia even if it is exceptionally valuable, being valuable not a priori (and thus, the agent being valued instrumentally in order to ensure its existence) but because it was chosen (and the thing that would have intrinsic value would be that which can make such choices).
A blindmind that doesn’t want qualia would be valuable then in this capacity to value things about the world in general, of which qualia is just a particular type (even if very valuable for sentient agents).
The appropiate type to compare rather than a Paperclip Maximizer (who, in Part 2 I argue, represents a type of agent whose values are inherently an aggression against the possibility of universal cooperation) would be aliens with strange and hard to comprehend values but no more intrinsically tied to the destruction of everything else than human values. If the moral patiency in them is only their qualia, then the best thing we could do for them is to just give them positive feelings, routing around whatever they valued in particular as means to that (and thus ultimately not really being about changing the outer world).
Respecting their agency would mean at least trying to understand what they are trying to do, from their perspective, not necessarily to give them everything they want (that’s subject to many considerations and their particular values), but to respect their goals in the sense that, when a human wants to make some great art, we take that helping them means helping them with that, rather than puting them in an experience machine where they think they did it.
If there is an agent that lost its qualia and wants to get them back, then I (probably) want to help it get them back, because I (probably) value qualia myself in an altruistic way. On the other hand, if there is a blindmind agent that doesn’t have or care about qualia, and just wants to make paperclips or whatever, then I (probably) don’t want to help them do that (except instrumentally, if doing so helps my own goals). It seems like you’re implicitly trying to make me transfer my intuitions from the former to the latter, by emphasizing the commonalities (they’re both agents) and ignoring the differences (one cares about something I also care about, the other doesn’t), which I think is an invalid move.
Apologies if I’m being uncharitable or misinterpreting you, but aside from this, I really don’t see what other logic or argumentative force is supposed to make me, after reading your first paragraph, reach the conclusion in your second paragraph, i.e., decide that I now want to value/help all agents, including blindminds that just want to make paperclips. If you have something else in mind, please spell it out more?
I see where you’re coming from.
Regarding paperclippers: in addition to what Pumo said in their reply concerning mutual alignment (and what will be said in Part 2), I’d say that stupid goals are stupid goals independent of sentience. I wouldn’t altruistically devote resources to helping a non-sentient AI make a billion paperclips for the exact same reason that I wouldn’t altruistically devote resources to helping some human autist make a billion paperclips. Maybe I’m misunderstanding your objection; possibly your objection is something more like “if we unmoor moral value from qualia, there’s nothing left to ground it in and the result is absurdity”. For now I’ll just say that we are definitely not asserting “all agents have equal moral status”, or “all goals are equal/interchangeable” (indeed, Part 2 asserts the opposite).
Regarding ‘why should I care about the blindmind for-its-own-sake?’, here’s another way to get there:
My understanding is that there are two primary competing views which assign non-sentient agents zero moral status:
1. Choice/preference (even of sentient beings) of a moral patient isn’t inherently relevant, qualia valence is everything (e.g. we should tile the universe in hedonium), e.g. hedonic utilitarianism. We don’t address this view much in the essay, except to point out that a) most people don’t believe this, and b) to whatever extent that this view results in totalizing hegemonic behavior, it sets the stage for mass conflict between believers of it and opponents of it (presumably including non-sentient agents). If we accept a priori something like ‘positive valenced qualia is the only fundamental good’, this view might at least be internally consistent (at which point I can only argue against the arbitrariness of accepting the a priori premise about qualia or argue against any system of values that appears to lead to it’s own defeat timelessly, though I realize the latter is a whole object-level debate to be had in and of itself).
2. Choice/preference fundamentally matters, but only if the chooser is also an experiencer. This seems like the more common view, e.g. (sentientist) preference utilitarianism. a) I think it can be shown that the ‘experience’ part of that is not load-bearing in a justifiable way, which I’ll address more below, and b) this suffers from the same practical problem as (b) above, though less severely.
#2 raises the question of “why would choice/preference-satisfaction have value independent of any particular valenced qualia, but not have value independent of qualia more generally?”. If you posit that the value of preference-satisfaction is wholly instrumental insofar as having your preferences satisfied generates some positive-valence qualia, this view just collapses back to #1. If you instead hold that preference-satisfaction is inherently (non-instrumentally) morally valuable by virtue of the preference-holder having qualia, even if only neutral-valenced qualia with regard to the preference… why? What morally-relevant work is the qualia doing and/or what is the posited morally-relevant connection between the (neutral) qualia and the preference? Why would the (valence-neutral) satisfaction of the preference have more value than just some other neutral qualia existing on it’s own while the preference goes un-satisfied? It seems to me like at this point we aren’t talking about ‘feelings’ (which exist in the domain of qualia) anymore, we’re talking about ‘choices and preferences’ (which are concepts within the domain of agency). To restate: The neutral qualia sitting alongside the preference or the satisfaction of the preference isn’t doing anything, or else (if the existence of the qualia is what is actually important) the preference itself doesn’t appear to be *doing* anything in which case preference satisfaction is not inherently important.
So the argument is that if you already attribute value to the preference satisfaction of sentient beings independent of their valenced qualia (e.g. if you would, for non-instrumental reasons, respect the stated preferences of some stranger even if you were highly confident that you could induce higher net-positive valence in them by not respecting their preferences), then you are already valuing something that really has nothing to do with their qualia. And thus there’s not as large of an intuition gap on the way to agential moral value as there at first seemed to be. Granted, this only applies to anyone who valued preference satisfaction in the first place.
Your interpretation isn’t exactly wrong, I’m proposing an onthological shift on the understanding of what’s more central to the self, the thing to care about (i.e. is the moral patient fundamentally a qualia that has or can have an agent, or an agent that has or can have qualia?).
The intuition is that if qualia, on its own, is generic and completely interchangeable among moral patients, it might not be what makes them such, even if it’s an important value. A blindmind upload has ultimately far more in common with the sentient person they are based on than said person has with a phenomenal experience devoid of all the content that makes up their agency.
Thus it would be the agent the thing that primarily values the qualia (and everything else), rather than the reverse. This decenters qualia even if it is exceptionally valuable, being valuable not a priori (and thus, the agent being valued instrumentally in order to ensure its existence) but because it was chosen (and the thing that would have intrinsic value would be that which can make such choices).
A blindmind that doesn’t want qualia would be valuable then in this capacity to value things about the world in general, of which qualia is just a particular type (even if very valuable for sentient agents).
The appropiate type to compare rather than a Paperclip Maximizer (who, in Part 2 I argue, represents a type of agent whose values are inherently an aggression against the possibility of universal cooperation) would be aliens with strange and hard to comprehend values but no more intrinsically tied to the destruction of everything else than human values. If the moral patiency in them is only their qualia, then the best thing we could do for them is to just give them positive feelings, routing around whatever they valued in particular as means to that (and thus ultimately not really being about changing the outer world).
Respecting their agency would mean at least trying to understand what they are trying to do, from their perspective, not necessarily to give them everything they want (that’s subject to many considerations and their particular values), but to respect their goals in the sense that, when a human wants to make some great art, we take that helping them means helping them with that, rather than puting them in an experience machine where they think they did it.