“Various theories of moral uncertainty exist, outlining how this aggregation works; but none of them actually escape the issue. The theories of moral uncertainty that Effective Altruists rely on are themselves frameworks for commensurating values and systematically ranking options, and (as such) they are also vulnerable to ‘value dictatorship’, where after some point the choices recommended by utilitarianism come to swamp the recommendations of other theories. In the literature, this phenomenon is well-known as ‘fanaticism’.[10]”
This seems too strong. Doesn’t this only apply to maximizing expected choiceworthiness with intertheoretic comparisons, among popular approaches? My impression is that none of the other popular approaches are fanatical. You mention moral parliament in the footnote as a non-fanatical exception, but there’s also MEC variance voting, formal bargain-theoretic approaches and my favourite theory.
Also, theories with value lexicality can swamp utilitarianism, although this also seems fanatical.
Hi Michael, thanks for your comment—since it overlapped quite significantly with Leonard’s comment, I have tried to address both of your concerns together in one of my replies to Leonard. Let me know if I failed to reply sufficiently to some of your points!
I think the quote literally mischaracterizes approaches to normative uncertainty that EAs rely on as all leading to fanaticism, when almost none do, only really MEC with intertheoretical comparisons. Whether the ones that deny intertheoretical comparisons also reject universal domain is a separate issue.
Or maybe I misunderstood, and you’re only referring to versions of MEC with intertheoretical comparisons?
I think that approaches other than MEC + ITT aren’t typically clear enough about universal domain, and can be interpreted in several different ways because they’re not completely formally specified. (Not itself a criticism!) But (a corollary of the argument in this post) these frameworks only actually avoid fanaticism if they deny universal domain. So it isn’t clear whether they fall victim too fanaticism, pace your claim that ‘almost none do’.
I think—purely personal impression—that most people who think very hard about moral uncertainty are committed to quite a rationalistic view of ethics, and that this leads quite naturally to universal domain; so I said that, interpreting this into their frameworks, they will fall victim to fanaticism. But I mentioned the moral parliament as an example of an approach where the proponents have explicitly denied universal domain, as an example of the other possibility. However you interpret it, though, it’s an either/or situation.
(Lexical orders / deontological constraints are a separate issue, as you mention.)
are basically completely formally specified (at least as much as any other approach I’ve seen, like MEC+intertheoretical comparisons).
are never fanatical with respect to your credences in theories iff you normalize theories based on the options you actually have available and so violate the independence of irrelevant alternatives (IIA).
Maybe violating IIA conflicts with how you think about “universal domain”, and IIA is usually assumed without much comment. If you think the quality of an outcome is based entirely on the features of that outcome, and this internal quality is all that matters when comparing outcoms, then IIA should hold.
don’t require or make intertheoretical comparisons.
rank all options (outcome distributions, including fixed outcomes) for any finite set of available options when each moral theory ranks them all, too. Maybe you can extend to arbitrary size sets by normalizing each theory relative to the infimum and supremum value across options according to that theory. So, I suspect they satisfy something that can reasonably be called universal domain.
I think formal bargain-theoretic approaches (here and here) satisfy 1 and 3, aren’t fanatical, but I’m not sure about 4 and violations of IIA.
On variance voting: yeah, I think 4 is the point here. I don’t think you can extend this approach to unbounded choice sets. I’m travelling atm so can’t be more formal, but hopefully tomorrow I can write up something a bit more detailed on this.
On bargaining-theory approaches, it actually isn’t clear that they avoid fanaticism: see pp.24-26 of the Greaves and Cotton-Barratt paper you link, especially their conclusion that ‘the Nash approach is not completely immune from fanaticism’. Again, I think constructive ambiguity in the way these theories get described often helps obscure their relationship to fanatical conclusions; but there’s an impossibility result here, so ultimately there’s no avoiding the choice.
“Various theories of moral uncertainty exist, outlining how this aggregation works; but none of them actually escape the issue. The theories of moral uncertainty that Effective Altruists rely on are themselves frameworks for commensurating values and systematically ranking options, and (as such) they are also vulnerable to ‘value dictatorship’, where after some point the choices recommended by utilitarianism come to swamp the recommendations of other theories. In the literature, this phenomenon is well-known as ‘fanaticism’.[10]”
This seems too strong. Doesn’t this only apply to maximizing expected choiceworthiness with intertheoretic comparisons, among popular approaches? My impression is that none of the other popular approaches are fanatical. You mention moral parliament in the footnote as a non-fanatical exception, but there’s also MEC variance voting, formal bargain-theoretic approaches and my favourite theory.
Also, theories with value lexicality can swamp utilitarianism, although this also seems fanatical.
Hi Michael, thanks for your comment—since it overlapped quite significantly with Leonard’s comment, I have tried to address both of your concerns together in one of my replies to Leonard. Let me know if I failed to reply sufficiently to some of your points!
I think the quote literally mischaracterizes approaches to normative uncertainty that EAs rely on as all leading to fanaticism, when almost none do, only really MEC with intertheoretical comparisons. Whether the ones that deny intertheoretical comparisons also reject universal domain is a separate issue.
Or maybe I misunderstood, and you’re only referring to versions of MEC with intertheoretical comparisons?
I think that approaches other than MEC + ITT aren’t typically clear enough about universal domain, and can be interpreted in several different ways because they’re not completely formally specified. (Not itself a criticism!) But (a corollary of the argument in this post) these frameworks only actually avoid fanaticism if they deny universal domain. So it isn’t clear whether they fall victim too fanaticism, pace your claim that ‘almost none do’.
I think—purely personal impression—that most people who think very hard about moral uncertainty are committed to quite a rationalistic view of ethics, and that this leads quite naturally to universal domain; so I said that, interpreting this into their frameworks, they will fall victim to fanaticism. But I mentioned the moral parliament as an example of an approach where the proponents have explicitly denied universal domain, as an example of the other possibility. However you interpret it, though, it’s an either/or situation.
(Lexical orders / deontological constraints are a separate issue, as you mention.)
Assuming all your moral theories are at least interval-scale, I think variance voting and similar normalization-based methods
are basically completely formally specified (at least as much as any other approach I’ve seen, like MEC+intertheoretical comparisons).
are never fanatical with respect to your credences in theories iff you normalize theories based on the options you actually have available and so violate the independence of irrelevant alternatives (IIA).
Maybe violating IIA conflicts with how you think about “universal domain”, and IIA is usually assumed without much comment. If you think the quality of an outcome is based entirely on the features of that outcome, and this internal quality is all that matters when comparing outcoms, then IIA should hold.
don’t require or make intertheoretical comparisons.
rank all options (outcome distributions, including fixed outcomes) for any finite set of available options when each moral theory ranks them all, too. Maybe you can extend to arbitrary size sets by normalizing each theory relative to the infimum and supremum value across options according to that theory. So, I suspect they satisfy something that can reasonably be called universal domain.
I think formal bargain-theoretic approaches (here and here) satisfy 1 and 3, aren’t fanatical, but I’m not sure about 4 and violations of IIA.
On variance voting: yeah, I think 4 is the point here. I don’t think you can extend this approach to unbounded choice sets. I’m travelling atm so can’t be more formal, but hopefully tomorrow I can write up something a bit more detailed on this.
On bargaining-theory approaches, it actually isn’t clear that they avoid fanaticism: see pp.24-26 of the Greaves and Cotton-Barratt paper you link, especially their conclusion that ‘the Nash approach is not completely immune from fanaticism’. Again, I think constructive ambiguity in the way these theories get described often helps obscure their relationship to fanatical conclusions; but there’s an impossibility result here, so ultimately there’s no avoiding the choice.