I’d also agree that designing trustworthy reflection procedures is important. My intuitions here are:
(1) value-drift is a big potential problem with FRI’s work (even if they “lock in” caring about suffering, if their definition of ‘suffering’ drifts, their tacit values do too);
(2) value-drift will be a problem for any system of ethics that doesn’t cleanly ‘compile to physics’. (This is a big claim, centering around my Objection 6, above.)
Perhaps we could generalize this latter point as “if information is physical, and value is informational, then value is physical too.”
I really enjoyed your linked piece on meta-ethics. Short but insightful. I believe I’d fall into the second bucket.
If you’re looking for what (2) might look like in practice, and how we might try to relate it to the human brain’s architecture/drives, you might enjoy this: http://opentheory.net/2017/05/why-we-seek-out-pleasure-the-symmetry-theory-of-homeostatic-regulation/
I’d also agree that designing trustworthy reflection procedures is important. My intuitions here are: (1) value-drift is a big potential problem with FRI’s work (even if they “lock in” caring about suffering, if their definition of ‘suffering’ drifts, their tacit values do too); (2) value-drift will be a problem for any system of ethics that doesn’t cleanly ‘compile to physics’. (This is a big claim, centering around my Objection 6, above.)
Perhaps we could generalize this latter point as “if information is physical, and value is informational, then value is physical too.”