I posed a puzzle in Is the potential astronomical waste in our universe too small to care about?, which seems to arise in any model of moral uncertainty where the representatives of different moral theories can trade resources, power, or influence with each other, like here. (Originally my post assumed a Moral Parliament where the delegates can trade votes with each other.)
(There was some previous discussion of my puzzle on this forum, when the Greaves and Cotton-Barratt preprint came out.)
This worry about internal bargaining & moral parliament approaches strikes me as entirely well-taken. The worry basically being that on the most obvious way of implementing these proposals for dealing with moral uncertainty, it looks as though a decision maker will wind up governed by the dead hand of her past empirical evidence. Suppose outcome X looks unlikely, and my subagents A and B strike a deal that benefits A (in which I have low credence) just in case X obtains. Now I find out that outcome X only looked unlikely because my old evidence wasn’t so great. In fact, X now obtains! Should I now do as A recommends, even though I have much higher credence in the moral theory that B represents?
What I think this worry shows us is that we should not implement the internal bargaining proposal in the most obvious way. Instead, we should implement it in a slightly less obvious way. I outline what I have in mind more formally in work in progress that I’d be very happy to share with anyone interested, but here’s the basic idea: the ‘contracts’ between subagents / parliamentarians that a decision maker should regard herself as bound by in the present choice situation are not the contracts that those subagents / parliamentarians agreed to earlier in the decision maker’s lifetime based on the evidence that she had then. Instead, the decision maker should regard herself as bound by the contracts that those subagents would have agreed earlier in the decision maker’s lifetime if they had then had the empirical evidence that they have now. This should resolve Wei_Dai’s puzzle.
I posed a puzzle in Is the potential astronomical waste in our universe too small to care about?, which seems to arise in any model of moral uncertainty where the representatives of different moral theories can trade resources, power, or influence with each other, like here. (Originally my post assumed a Moral Parliament where the delegates can trade votes with each other.)
(There was some previous discussion of my puzzle on this forum, when the Greaves and Cotton-Barratt preprint came out.)
This worry about internal bargaining & moral parliament approaches strikes me as entirely well-taken. The worry basically being that on the most obvious way of implementing these proposals for dealing with moral uncertainty, it looks as though a decision maker will wind up governed by the dead hand of her past empirical evidence. Suppose outcome X looks unlikely, and my subagents A and B strike a deal that benefits A (in which I have low credence) just in case X obtains. Now I find out that outcome X only looked unlikely because my old evidence wasn’t so great. In fact, X now obtains! Should I now do as A recommends, even though I have much higher credence in the moral theory that B represents?
What I think this worry shows us is that we should not implement the internal bargaining proposal in the most obvious way. Instead, we should implement it in a slightly less obvious way. I outline what I have in mind more formally in work in progress that I’d be very happy to share with anyone interested, but here’s the basic idea: the ‘contracts’ between subagents / parliamentarians that a decision maker should regard herself as bound by in the present choice situation are not the contracts that those subagents / parliamentarians agreed to earlier in the decision maker’s lifetime based on the evidence that she had then. Instead, the decision maker should regard herself as bound by the contracts that those subagents would have agreed earlier in the decision maker’s lifetime if they had then had the empirical evidence that they have now. This should resolve Wei_Dai’s puzzle.