‘...the thought of a superpowerful AI that shares the value system of e.g. LessWrong is slightly terrifying to me.‘
Old post, but I’ve meant to say this for several months: Whilst I am not a fan of Yudkowsky, I do think that his stuff about this showed a fair amount of sensitivity to the idea that it would be unfair if a particular group of people just programed their values into the AI, taking no heed of the fact that humans disagree. (Not that that means there is no reason to worry about the proposal to build a “good” AI that runs everything).
His original (since abandoned I think) proposal, was that we would get the AI to have goal like ‘maximizes things all or pretty much all fully informed humans would agree are good, minimizes things all or almost all fully informed would humans agree are bad, and where humans would disagree on whether something is good or bad even after being fully informed of all relevant facts, try and minimize your impact on that thing, and leave it up to humans to sort out amongst themselves.’ (Not an exact rendition, but close enough for present purposes.) Of course, there’s a sense in which that still embodies liberal democratic values about what is fair, but I’m guessing if your a contemporary person with a humanities degree, you probably share those very broad and abstract values.
‘...the thought of a superpowerful AI that shares the value system of e.g. LessWrong is slightly terrifying to me.‘
Old post, but I’ve meant to say this for several months: Whilst I am not a fan of Yudkowsky, I do think that his stuff about this showed a fair amount of sensitivity to the idea that it would be unfair if a particular group of people just programed their values into the AI, taking no heed of the fact that humans disagree. (Not that that means there is no reason to worry about the proposal to build a “good” AI that runs everything).
His original (since abandoned I think) proposal, was that we would get the AI to have goal like ‘maximizes things all or pretty much all fully informed humans would agree are good, minimizes things all or almost all fully informed would humans agree are bad, and where humans would disagree on whether something is good or bad even after being fully informed of all relevant facts, try and minimize your impact on that thing, and leave it up to humans to sort out amongst themselves.’ (Not an exact rendition, but close enough for present purposes.) Of course, there’s a sense in which that still embodies liberal democratic values about what is fair, but I’m guessing if your a contemporary person with a humanities degree, you probably share those very broad and abstract values.