If “human-compatible” means anything non-speciesistic, then I agree that it is an unfortunate phrase, since it is misleading. I also think it is misleading to call idealized preferences for “human values,” since humans don’t actually hold those preferences, as you correctly point out.
You write that
“Which ethical system is correct?” isn’t written in the stars or in Plato’s heaven; it seems like if the answer is encoded anywhere in the universe, it must be encoded in our brains (or in logical constructs out of brains).
Let X be the claims, which you deny in this quote. If X is taken litterally, then it is a straw man, since no one believes in it. If X is metaphorical, then it is very unclear what its supposed to mean or whether it means anything. The claim that “ethics is encoded somewhere in the universe” is also unclear. My best attempt to ascribe meaning to it is as follows “there is some entity in the universe, which constitutes all of ethics,” but claims seems false. The most basic ethical principles is, I believe, in some ways like logical principles. The validity of the argument “p and q, therefore p” is not constituted by any feature of the universe. To see this, imagine an alternative universe, which differs from the real in basically any way you like. It’s governed by different laws of nature, contains different lifeforms (or perhaps no life at all) has a different cosmological history etc. If this universe had been real, then “p and q, therefore p” would still be valid. Basic ethical principles like the claim that the suffering is bad, seems just like this. If human preferences (or other features of the universe) where to be different, then suffering would still be bad.
What explains why “suffering is bad” is true in all universes? How could an agent realistically discover this truth—how do we filter out the false moral claims and zero in on the true ones, and how could an alien do the same?
If “human-compatible” means anything non-speciesistic, then I agree that it is an unfortunate phrase, since it is misleading. I also think it is misleading to call idealized preferences for “human values,” since humans don’t actually hold those preferences, as you correctly point out.
You write that
Let X be the claims, which you deny in this quote. If X is taken litterally, then it is a straw man, since no one believes in it. If X is metaphorical, then it is very unclear what its supposed to mean or whether it means anything. The claim that “ethics is encoded somewhere in the universe” is also unclear. My best attempt to ascribe meaning to it is as follows “there is some entity in the universe, which constitutes all of ethics,” but claims seems false. The most basic ethical principles is, I believe, in some ways like logical principles. The validity of the argument “p and q, therefore p” is not constituted by any feature of the universe. To see this, imagine an alternative universe, which differs from the real in basically any way you like. It’s governed by different laws of nature, contains different lifeforms (or perhaps no life at all) has a different cosmological history etc. If this universe had been real, then “p and q, therefore p” would still be valid. Basic ethical principles like the claim that the suffering is bad, seems just like this. If human preferences (or other features of the universe) where to be different, then suffering would still be bad.
I agree that suffering is bad in all universes, for the reasons described in https://www.lesswrong.com/posts/zqwWicCLNBSA5Ssmn/by-which-it-may-be-judged. I’d say that “ethics… is not constituted by any feature of the universe” in the sense you note, but I’d point to our human brains if we were asking any question like:
What explains why “suffering is bad” is true in all universes? How could an agent realistically discover this truth—how do we filter out the false moral claims and zero in on the true ones, and how could an alien do the same?