We would lose a great deal of value by optimizing the universe according to current moral uncertainty, without the opportunity to reflect and become less uncertain over time.
There’s a great deal of reflection necessary to figure out what actions moral theory X recommends, e.g. to figure out which minds exist or what implicit promises people have made to each other. I don’t see this reflection as distinct from reflection about moral uncertainty; if we’re going to defer to a reflection process anyway for making decisions, we might as well let that reflection process decide on issues of moral theory.
What if AI exploring moral uncertainty finds that there is provably no correct moral theory or right moral facts? It that case, there is no moral uncertainty between moral theories, as they are all false. Could it escape this obstacle just by aggregating human’s opinion about possible situations?
What if AI exploring moral uncertainty finds that there is provably no correct moral theory or right moral facts?
In that case it would be exploring traditional metaethics, not moral uncertainty.
But if moral uncertainty is used as a solution then we just bake in some high level criteria for the appropriateness of a moral theory, and the credences will necessarily sum to 1. This is little different from baking in coherent extrapolated volition. In either case the agent is directly motivated to do whatever it is that satisfies our designated criteria, and it will still want to do it regardless of what it thinks about moral realism.
Those criteria might be very vague and philosophical, or they might be very specific and physical (like ‘would a simulation of Bertrand Russell say “a-ha, that’s a good theory”?’), but either way they will be specified.
What about moral uncertainty as an alternative to CEV? (https://nebula.wsimg.com/1cc278bf0e7470c060032c9624508149?AccessKeyId=07941C4BD630A320288F&disposition=0&alloworigin=1)
I expect:
We would lose a great deal of value by optimizing the universe according to current moral uncertainty, without the opportunity to reflect and become less uncertain over time.
There’s a great deal of reflection necessary to figure out what actions moral theory X recommends, e.g. to figure out which minds exist or what implicit promises people have made to each other. I don’t see this reflection as distinct from reflection about moral uncertainty; if we’re going to defer to a reflection process anyway for making decisions, we might as well let that reflection process decide on issues of moral theory.
What if AI exploring moral uncertainty finds that there is provably no correct moral theory or right moral facts? It that case, there is no moral uncertainty between moral theories, as they are all false. Could it escape this obstacle just by aggregating human’s opinion about possible situations?
In that case it would be exploring traditional metaethics, not moral uncertainty.
But if moral uncertainty is used as a solution then we just bake in some high level criteria for the appropriateness of a moral theory, and the credences will necessarily sum to 1. This is little different from baking in coherent extrapolated volition. In either case the agent is directly motivated to do whatever it is that satisfies our designated criteria, and it will still want to do it regardless of what it thinks about moral realism.
Those criteria might be very vague and philosophical, or they might be very specific and physical (like ‘would a simulation of Bertrand Russell say “a-ha, that’s a good theory”?’), but either way they will be specified.