I’m very pleased more thinking is being done on this – thank you.
I’m not sure I follow this:
Pushing for “animal-friendly” values may be harmful if it skews trajectories that are good for animals
As an intuition pump, imagine that animal farming (or other functional-animal mistreatment by humans) will be eradicated by default (e.g. because it will stop being economically valuable). If we manage to instill strong animal-related concerns that are not perfectly “wise” (e.g. specific ~beliefs on what is good or bad for farmed animals), then the AI(s) may perpetuate farming in some form even if that choice is unnecessary and harmful.
Would this be an example: we instill a goal in a powerful AI system along the lines of “reduce the suffering of animals who are being farmed”. Then the AI system prevents the abolition of animal farming on the grounds that it can’t achieve that goal if animal farming ends?
(Lizka and I have slightly different views here, speaking only for myself.)
This is a good question. The basic point is that, just as lock-in can prevent things from getting worse, it can also prevent things from getting better.
For example, the Universal Declaration of the Rights of Mother Earth says that all beings have the right to “not have its genetic structure modified or disrupted in a manner that threatens its integrity or vital and healthy functioning”.
Even though I think this right could reasonably be interpreted as “animal-friendly”, my guess is that it would be bad to lock it in because it would prevent us from e.g. genetically modifying farmed animals to feel less pain.
Not OP but I would say that if we end up with an ASI that can misunderstand values in that kind of way, then it will almost certainly wipe out humanity anyway.
That is the same category of mistake as “please maximize the profit of this paperclip factory” getting interpreted as “convert all available matter into paperclip machines”.
I’m very pleased more thinking is being done on this – thank you.
I’m not sure I follow this:
Would this be an example: we instill a goal in a powerful AI system along the lines of “reduce the suffering of animals who are being farmed”. Then the AI system prevents the abolition of animal farming on the grounds that it can’t achieve that goal if animal farming ends?
(Lizka and I have slightly different views here, speaking only for myself.)
This is a good question. The basic point is that, just as lock-in can prevent things from getting worse, it can also prevent things from getting better.
For example, the Universal Declaration of the Rights of Mother Earth says that all beings have the right to “not have its genetic structure modified or disrupted in a manner that threatens its integrity or vital and healthy functioning”.
Even though I think this right could reasonably be interpreted as “animal-friendly”, my guess is that it would be bad to lock it in because it would prevent us from e.g. genetically modifying farmed animals to feel less pain.
Not OP but I would say that if we end up with an ASI that can misunderstand values in that kind of way, then it will almost certainly wipe out humanity anyway.
That is the same category of mistake as “please maximize the profit of this paperclip factory” getting interpreted as “convert all available matter into paperclip machines”.
Yes, my example and the paperclip one both seem like a classic case of outer misalignment / reward misspecification.