Why only conscious preferences matter
I did not use an LLM in writing this post.
Plants, generative AI models and thermostats have preferences or goals, but they (probably) do not have conscious experiences. Their preferences are (probably) unconscious. But in my moral theory (called mild welfarism), only conscious preferences matter. A conscious preference is a preference (value or goal) that can be consciously experienced by a conscious (sentient) being. Although just a moment ago you may not have been consciously aware of your preference to stay alive, you could have been made aware of that preference, for example when you were in danger and you felt fear, or when you now reflect on that preference by reading this sentence. Hence, your preference to stay alive is now and was a moment ago a conscious preference.
The reason why preferences matter is straightforward: if you prefer a moral theory according to which preferences do not matter, your preference for that moral theory cannot matter either. But why should we only take into consideration conscious preferences (and not unconscious preferences) in our moral deliberations?
The theory of mild welfarism starts with the premise that only welfare matters. By definition, the welfare of a conscious, sentient being in a situation or outcome measures that individualâs strength of preference for that situation or outcome. In other words, welfare measures how good that situation is according to the individual itself. Hence, welfare is tightly linked to preferences. But an individual can only measure the strength of its preferences when it is consciously aware of its preferences.
The ability to pay attention is crucial: a preference only matters when there is someone (a sentient being) who can pay attention to that preference. And the ability to pay attention is the basis of consciousness, according to Michael Grazianoâs Attention Schema Theory (AST) of consciousness. According to that theory, consciousness is the brainâs simplified, schematic description of its own data-processing focus. To manage its own limited processing power, the brain uses attention to focus on specific signals. Signals can be perceptions or memories, but also goals or preferences. The brain builds an internal map or schema of that attention process in order to control its own internal focus. This attention schema is what we experience as awareness. We feel like we have conscious, subjective awareness because our brains are constantly building a model of our own attention.
The strength of a preference measures how strongly the entity tends to pay attention to the preference. A preference is a signal (in particular information about a goal function), and a strong preference is a signal that is (or tends to be) strongly amplified in order to gain or attract attention when the preference is at stake. Your preference to stay alive is strong, because when your live is at stake in a dangerous situation, the perception of the danger, the fear and the preference to stay alive all grab your attention. The information that staying alive is one of your goals, gets amplified. Without an attention mechanism, all preferences of the unconscious entity have zero strength, as with complete indifference. In the moral calculation, they count as zero.
In summary: a preference is only relevant in our moral considerations when the entity that has that preference is able to care about that preference, by being able to pay attention to that preference. In order to pay attention, such an entity needs to have an attention control mechanism or attention schema. Such a mechanism automatically and unavoidably generates a consciousness. That is why only conscious preferences matter. For unconscious entities, their preferences have zero strength and hence zero moral relevance.
What do you think about AI systems that donât have the kinds of limitations we do, and that might not need to restrict focus as much, or might be able to devote cognitive resources to many different problems in parallel. It is plausible that they wouldnât count as having attention in the way that Graziano thinks we do, and that they wouldnât therefore satisfy his theory of consciousness. But it seems weird to suggest that the reason our preferences matter is because of our cognitive limitations, no?
Ha, nice argument :-) It reminds me of a famous defense of predation argument in animal rights ethics: lions do not violate the rights of their prey, because lions do not have moral agency. In other words: lions have a privilege to harm, kill and use other animals, because they are too stupid to understand morality.
To the point: although it may seem weird, I think it makes sense: if due to cognitive limitations an organism was forced to generate an attention schema that generates consciousness, that organism is more concious than e.g. an AI that does not have those cognitive limitations that cause the development of an attention schema. Furthermore, in terms of the ability to pay attention, that unfocused AI (i.e. it is not able to focus), has a cognitive limitation. It lacks a cognitive ability. Although I consider it possible and even likely that a strong AI also develops an attention schema, because that is so beneficial and efficient. Our brains are also able to run processes in parallel, devoting resources to many (unconcious) parallel processes. Yet, we benefit from having the cognitive capacity to focus, to pay attention, using an attention schema. Hence, even with our cognitive ability of parallel processing, we developed an attention schema.
Perhaps a good analogy: some animals are not able to run fast on their legs to escape from predators. They have a physical limitation. Due to that limitation, those animals developped wings with which they can fly and escape from predators. Does it seem weird to suggest that the reason a bird can fly is because of its physical limitation to run fast?
Thought-provoking argument! However, I see some gaps:
1.
I think this mixes up two senses of âmatteringâ.
I might not think my preference for hedonic utilitarianism is terminally valuable but still think that believing true things is instrumentally valuable to achieve utility.
2. I share your intuition that âpreferences need to have a weightâ but I donât think thatâs the same thing as âbeing represented within an attention schemaâ. I think plants can weigh their preferences (e.g. tolerate drier soil if the luminosity is sufficient) without having explicit preferences (i.e. language?) or a comprehensive world model, let alone a meta-model.
While I agree with the thesis from the title, I think it might be better anchored in ~Sharon H. Rawletteâs argument that conscious sensations of â(un)desirabilityâ construct/âdefine what we mean by morality.
Sorry, I donât see the mixing up of two senses of mattering and donât understand your example.
Good point about weighing preferences. Yes, you can have a multidimensional goal function, as a landscape, where different directions (preferences) have different slopes (strengths). (Picture a consumer choice diagram used in microeconomics, with indifference curves.) But in this context, with strength I ment the amount of amplification of a signal. The preference to avoid a strong pain is stronger than the preference to avoid a slight pain, because the pain signal of the strong pain is more strongly amplified to get my attention.
Maybe I misunderstood it as an argument for seeing preferences as a terminal value. I think almost all theories would agree âthis theory matters in some senseâ but I can imagine many ethical theories that do not see âbelieving in this moral theoryâ as either good or bad, each for different reasons. Hedonic utilitarianism, as an example of a consequentialist theory, does not see âbelieving in hedonic utilitarianismâ as inherently valuableâdepending on the consequences of believing it, it might even recommend not believing it in some contextsâwhile still positing hedonism itself is true. For a nihilist, ânihilism itselfâ probably does not matter morally but it does âmatterâ in terms of its alleged explanatory power.
You could see the luminosity itself as a factor that moderates the intensity of the signal to grow. But there also seem to be many internal processes that moderate the intensity of signals processed by plants: AI points me to a barley study where stress-related calcium signals varied by stimulus, dose, and tissue (pmc.ncbi.nlm.nih.gov). It still doesnât require an attention schemaâi.e. weighted preferences donât imply consciousness (at least under your conception of preferences and Grazianoâs of consciousness, as I understand them).
Good points, I think I know what you are trying to say. My idea is incomplete or perhaps wrong. Perhaps having a representation of the attention becomes important, but if so, why? Iâll have to think further about this. Thanks