I forgot to mention that your post did help to clarify points and alleviate some of my confusion. Particularly the idea that an ultra-powerful AI tool (which may or may not be sentient) “would still permit one human to wield power over all others.”
The hypothetical of an AI wiping out all of humanity because it figures out (or thinks it figures out) that it will increase overall utility by doing so is just one extreme possibility. There must be a lot of credible seeming scenarios opposed to this one in which an AI could be used to increase overall suffering. (Unless the assumption was that a super intelligent being or device couldn’t help but come around to a utilitarian perspective, no matter how it was initially programmed!)
Also, like Scott Alexander wrote on his post about this, x-risk reduction is not all about AI.
Still, from a utilitarian perspective, it seems like talking about “AI friendliness” should mean friendliness to overall utility, which won’t automatically mean friendliness to humanity or human rights. But again, I imagine plenty of EAs do make that distinction, and I’m just not aware of it because I haven’t looked that far into it. And anyway, that’s not a critique of AI risk being a concern for EAs; at most, it’s a critique of some instances of rhetoric.
Brian Tomasik is ab self-described “negative-leaning” hedonic utilitarian who is a prominent thinker for effective altruism. He’s written about how humanity might have values which lead us to generating much suffering in the future, but also worries a machine superintelligence might end up doing the same. They’re myriad reasons he thinks this I can’t do justice to here. I believe right now he thinks the best course of action is to try steering values of present-day humanity, as much of it or as crucially an influential subset as possible, towards neglecting suffering less. He also believes doing foundational research into ascertaining better the chances of a singleton to promulgate suffering throughout space in the future. To this end he both does research with and funds colleagues at the Foundational Research Institute.
His whole body of work concerning future suffering is referred to as “astronomical suffering” considerations, sort of complementary utilitarian consideration to Dr
Bostrom’s astronomical waste argument. You can read more of Mr.
Tomasik’s work on the far future and related topics here. Note some of it is advanced and may require you to read beforehand to understand all premises in some of his essays, but he also usually provides citations for all this.
I forgot to mention that your post did help to clarify points and alleviate some of my confusion. Particularly the idea that an ultra-powerful AI tool (which may or may not be sentient) “would still permit one human to wield power over all others.”
The hypothetical of an AI wiping out all of humanity because it figures out (or thinks it figures out) that it will increase overall utility by doing so is just one extreme possibility. There must be a lot of credible seeming scenarios opposed to this one in which an AI could be used to increase overall suffering. (Unless the assumption was that a super intelligent being or device couldn’t help but come around to a utilitarian perspective, no matter how it was initially programmed!)
Also, like Scott Alexander wrote on his post about this, x-risk reduction is not all about AI.
Still, from a utilitarian perspective, it seems like talking about “AI friendliness” should mean friendliness to overall utility, which won’t automatically mean friendliness to humanity or human rights. But again, I imagine plenty of EAs do make that distinction, and I’m just not aware of it because I haven’t looked that far into it. And anyway, that’s not a critique of AI risk being a concern for EAs; at most, it’s a critique of some instances of rhetoric.
Brian Tomasik is ab self-described “negative-leaning” hedonic utilitarian who is a prominent thinker for effective altruism. He’s written about how humanity might have values which lead us to generating much suffering in the future, but also worries a machine superintelligence might end up doing the same. They’re myriad reasons he thinks this I can’t do justice to here. I believe right now he thinks the best course of action is to try steering values of present-day humanity, as much of it or as crucially an influential subset as possible, towards neglecting suffering less. He also believes doing foundational research into ascertaining better the chances of a singleton to promulgate suffering throughout space in the future. To this end he both does research with and funds colleagues at the Foundational Research Institute.
His whole body of work concerning future suffering is referred to as “astronomical suffering” considerations, sort of complementary utilitarian consideration to Dr Bostrom’s astronomical waste argument. You can read more of Mr. Tomasik’s work on the far future and related topics here. Note some of it is advanced and may require you to read beforehand to understand all premises in some of his essays, but he also usually provides citations for all this.
Worth noting that the negative-learning position is pretty fringe though, especially in mainstream philosophy. Personally, I avoid it.