I see. Yeah, I expressed what I was thinking incorrectly here. What I meant was that our preferences are slightly abnormal compared to what you would expect but not extraordinarily alien.
In our ancestral environment, we developed a taste for sugary things because berries, which are sugary, are good for us. Knowing this, you would expect that we would eat berries but not ice cream. The fact that we eat ice cream is somewhat abnormal. That said, it is not extraordinarily alien. If we, for instance, devoted our entire lives to eating as much sugar as is physiologically possible without dying, that would be very alien.
I should have said that their argument implies that AIs might have weird preferences but not extremely bizarre ones.
I see. Yeah, I expressed what I was thinking incorrectly here. What I meant was that our preferences are slightly abnormal compared to what you would expect but not extraordinarily alien.
In our ancestral environment, we developed a taste for sugary things because berries, which are sugary, are good for us. Knowing this, you would expect that we would eat berries but not ice cream. The fact that we eat ice cream is somewhat abnormal. That said, it is not extraordinarily alien. If we, for instance, devoted our entire lives to eating as much sugar as is physiologically possible without dying, that would be very alien.
I should have said that their argument implies that AIs might have weird preferences but not extremely bizarre ones.
Thanks for catching that!