Ok, so here’s my take away from the answers so far:
Most flavors of utilitarianism (except for preference utilitarianism) don’t consider any goal-having agent achieving those goals as utility. Instead there assumed to be some metric of similarity between the goals and/or mental states of the agent and those of humans, and the agent’s achievement of its goals counts the less toward total utility the lower this similarity metric is, so completely alien agents achieving their alien goals and [non-]experiencing alien non-joy about it don’t register as adding utility.
How exactly this metric should be formulated is disputed and fuzzy, and quite often a lot of this fuzziness and uncertainty is swept under the rug with the word “sentience” (or something similar) written on it.
Additionally, the proportion of EAs who would seriously consider “all humans replaced by [particular kind of] AIs” as an acceptable outcome may be not as trivial as I assumed.
Please let me know if I’m grossly misunderstanding or misrepresenting something, and thank you everyone for your explanations!
Ok, so here’s my take away from the answers so far:
Most flavors of utilitarianism (except for preference utilitarianism) don’t consider any goal-having agent achieving those goals as utility. Instead there assumed to be some metric of similarity between the goals and/or mental states of the agent and those of humans, and the agent’s achievement of its goals counts the less toward total utility the lower this similarity metric is, so completely alien agents achieving their alien goals and [non-]experiencing alien non-joy about it don’t register as adding utility.
How exactly this metric should be formulated is disputed and fuzzy, and quite often a lot of this fuzziness and uncertainty is swept under the rug with the word “sentience” (or something similar) written on it.
Additionally, the proportion of EAs who would seriously consider “all humans replaced by [particular kind of] AIs” as an acceptable outcome may be not as trivial as I assumed.
Please let me know if I’m grossly misunderstanding or misrepresenting something, and thank you everyone for your explanations!