I forgot to mention that your post did help to clarify points and alleviate some of my confusion. Particularly the idea that an ultra-powerful AI tool (which may or may not be sentient) “would still permit one human to wield power over all others.”
The hypothetical of an AI wiping out all of humanity because it figures out (or thinks it figures out) that it will increase overall utility by doing so is just one extreme possibility. There must be a lot of credible seeming scenarios opposed to this one in which an AI could be used to increase overall suffering. (Unless the assumption was that a super intelligent being or device couldn’t help but come around to a utilitarian perspective, no matter how it was initially programmed!)
Also, like Scott Alexander wrote on his post about this, x-risk reduction is not all about AI.
Still, from a utilitarian perspective, it seems like talking about “AI friendliness” should mean friendliness to overall utility, which won’t automatically mean friendliness to humanity or human rights. But again, I imagine plenty of EAs do make that distinction, and I’m just not aware of it because I haven’t looked that far into it. And anyway, that’s not a critique of AI risk being a concern for EAs; at most, it’s a critique of some instances of rhetoric.
I haven’t explored the debate over AI risk in the EA movement in depth, so I’m not informed enough to take a strong position. But Kosta’s comment gets at one of the things that has puzzled me—as basically an interested outsider—about the concern for x-risk in EA. A very strong fear of human extinction seems to treat humanity as innately important. But in a hedonic utilitarian framework, humanity is only contingently important to the extent that the continuation of humanity improves overall utility. If an AI or AIs could improve overall utility by destroying humanity (perhaps after determining that humans feel more suffering than pleasure overall, or that humans cause more suffering than pleasure overall, or that AIs feel more pleasure and less suffering than humans and so should use all space and resources to sustain as many AIs as possible), then hedonic utilitarians (and EAs, to the extent that they are hedonic utilitarians) should presumably want AIs to do this.
I’m sure there are arguments that an AI that destroys humanity would end up lowering utility, but I don’t get the impression that x-risk-centered EAs only oppose the destruction of humanity if it turns out humanity adds more pleasure to the aggregate. I would have expected to see EAs arguing something more like, “Let’s make sure an AI only destroys us if destroying us turns out to raise aggregate good,” but instead the x-risk EAs seem to be saying something more like, “Let’s make sure an AI doesn’t destroy us.”
But maybe the x-risk-centered EAs aren’t hedonic utilitarians, or they mostly tend to think an AI destroying humanity would lower overall utility and that’s why they oppose it, or there’s something else that I’m missing – which is probably the case, since I haven’t investigated the debate in detail.