I don’t think CEV or similar reflection processes reliably lead to wide moral circles. I think they can still be heavily influenced by their initial set-up (e.g. what the values of humanity when reflection begins).
Why do you think this is the case?
Do you think there is an alternative reflection process (either implemented by an AI, by a human society, or combination of both) that could be defined that would reliably lead to wide moral circles? Do you have any thoughts on what would it look like?
If we go through some kind of reflection process to determine our values, I would much rather have a reflection process that wasn’t dependent on whether or not MCE occurred before hand, and I think not leading to a wide moral circle should be considered a serious bug in any definition of a reflection process. It seems to me that working on producing this would be a plausible alternative or at least parallel path to directly performing MCE.
I think that there’s an inevitable tradeoff between wanting a reflection process to have certain properties and worries about this violating goal preservation for at least some people. This blogpost is not about MCE directly, but if you think of “BAAN thought experiment” as “we do moral reflection and the outcome is such a wide circle that most people think it is extremely counterintuitive” then the reasoning in large parts of the blogpost should apply perfectly to the discussion here.
That is not to say that trying to fine tune reflection processes is pointless: I think it’s very important to think about what our desiderata should be for a CEV-like reflection process. I’m just saying that there will be tradeoffs between certain commonly mentioned desiderata that people don’t realize are there because they think there is such a thing as “genuinely free and open-ended deliberation.”
Thanks for commenting, Lukas. I think Lukas, Brian Tomasik, and others affiliated with FRI have thought more about this, and I basically defer to their views here, especially because I haven’t heard any reasonable people disagree with this particular point. Namely, I agree with Lukas that there seems to be an inevitable tradeoff here.
Why do you think this is the case? Do you think there is an alternative reflection process (either implemented by an AI, by a human society, or combination of both) that could be defined that would reliably lead to wide moral circles? Do you have any thoughts on what would it look like?
If we go through some kind of reflection process to determine our values, I would much rather have a reflection process that wasn’t dependent on whether or not MCE occurred before hand, and I think not leading to a wide moral circle should be considered a serious bug in any definition of a reflection process. It seems to me that working on producing this would be a plausible alternative or at least parallel path to directly performing MCE.
I think that there’s an inevitable tradeoff between wanting a reflection process to have certain properties and worries about this violating goal preservation for at least some people. This blogpost is not about MCE directly, but if you think of “BAAN thought experiment” as “we do moral reflection and the outcome is such a wide circle that most people think it is extremely counterintuitive” then the reasoning in large parts of the blogpost should apply perfectly to the discussion here.
That is not to say that trying to fine tune reflection processes is pointless: I think it’s very important to think about what our desiderata should be for a CEV-like reflection process. I’m just saying that there will be tradeoffs between certain commonly mentioned desiderata that people don’t realize are there because they think there is such a thing as “genuinely free and open-ended deliberation.”
Thanks for commenting, Lukas. I think Lukas, Brian Tomasik, and others affiliated with FRI have thought more about this, and I basically defer to their views here, especially because I haven’t heard any reasonable people disagree with this particular point. Namely, I agree with Lukas that there seems to be an inevitable tradeoff here.