Why should we expect AI systems to develop bizarre and alien preferences when virtually all biological organisms have extremely normal preferences? (For instance, humans like to eat ice cream, but they don’t like to eat, as you mention, jet engine fuel.)
Isn’t this backwards? We call eating ice cream normal because humans like to do it, not the other way around. That isn’t evidence that AIs will have normal (i.e. similar to human) preferences.
I see. Yeah, I expressed what I was thinking incorrectly here. What I meant was that our preferences are slightly abnormal compared to what you would expect but not extraordinarily alien.
In our ancestral environment, we developed a taste for sugary things because berries, which are sugary, are good for us. Knowing this, you would expect that we would eat berries but not ice cream. The fact that we eat ice cream is somewhat abnormal. That said, it is not extraordinarily alien. If we, for instance, devoted our entire lives to eating as much sugar as is physiologically possible without dying, that would be very alien.
I should have said that their argument implies that AIs might have weird preferences but not extremely bizarre ones.
Isn’t this backwards? We call eating ice cream normal because humans like to do it, not the other way around. That isn’t evidence that AIs will have normal (i.e. similar to human) preferences.
I see. Yeah, I expressed what I was thinking incorrectly here. What I meant was that our preferences are slightly abnormal compared to what you would expect but not extraordinarily alien.
In our ancestral environment, we developed a taste for sugary things because berries, which are sugary, are good for us. Knowing this, you would expect that we would eat berries but not ice cream. The fact that we eat ice cream is somewhat abnormal. That said, it is not extraordinarily alien. If we, for instance, devoted our entire lives to eating as much sugar as is physiologically possible without dying, that would be very alien.
I should have said that their argument implies that AIs might have weird preferences but not extremely bizarre ones.
Thanks for catching that!