âVarious theories of moral uncertainty exist, outlining how this aggregation works; but none of them actually escape the issue. The theories of moral uncertainty that Effective Altruists rely on are themselves frameworks for commensurating values and systematically ranking options, and (as such) they are also vulnerable to âvalue dictatorshipâ, where after some point the choices recommended by utilitarianism come to swamp the recommendations of other theories. In the literature, this phenomenon is well-known as âfanaticismâ.[10]â
This seems too strong. Doesnât this only apply to maximizing expected choiceworthiness with intertheoretic comparisons, among popular approaches? My impression is that none of the other popular approaches are fanatical. You mention moral parliament in the footnote as a non-fanatical exception, but thereâs also MEC variance voting, formal bargain-theoretic approaches and my favourite theory.
Also, theories with value lexicality can swamp utilitarianism, although this also seems fanatical.
Hi Michael, thanks for your commentâsince it overlapped quite significantly with Leonardâs comment, I have tried to address both of your concerns together in one of my replies to Leonard. Let me know if I failed to reply sufficiently to some of your points!
I think the quote literally mischaracterizes approaches to normative uncertainty that EAs rely on as all leading to fanaticism, when almost none do, only really MEC with intertheoretical comparisons. Whether the ones that deny intertheoretical comparisons also reject universal domain is a separate issue.
Or maybe I misunderstood, and youâre only referring to versions of MEC with intertheoretical comparisons?
I think that approaches other than MEC + ITT arenât typically clear enough about universal domain, and can be interpreted in several different ways because theyâre not completely formally specified. (Not itself a criticism!) But (a corollary of the argument in this post) these frameworks only actually avoid fanaticism if they deny universal domain. So it isnât clear whether they fall victim too fanaticism, pace your claim that âalmost none doâ.
I thinkâpurely personal impressionâthat most people who think very hard about moral uncertainty are committed to quite a rationalistic view of ethics, and that this leads quite naturally to universal domain; so I said that, interpreting this into their frameworks, they will fall victim to fanaticism. But I mentioned the moral parliament as an example of an approach where the proponents have explicitly denied universal domain, as an example of the other possibility. However you interpret it, though, itâs an either/âor situation.
(Lexical orders /â deontological constraints are a separate issue, as you mention.)
are basically completely formally specified (at least as much as any other approach Iâve seen, like MEC+intertheoretical comparisons).
are never fanatical with respect to your credences in theories iff you normalize theories based on the options you actually have available and so violate the independence of irrelevant alternatives (IIA).
Maybe violating IIA conflicts with how you think about âuniversal domainâ, and IIA is usually assumed without much comment. If you think the quality of an outcome is based entirely on the features of that outcome, and this internal quality is all that matters when comparing outcoms, then IIA should hold.
donât require or make intertheoretical comparisons.
rank all options (outcome distributions, including fixed outcomes) for any finite set of available options when each moral theory ranks them all, too. Maybe you can extend to arbitrary size sets by normalizing each theory relative to the infimum and supremum value across options according to that theory. So, I suspect they satisfy something that can reasonably be called universal domain.
I think formal bargain-theoretic approaches (here and here) satisfy 1 and 3, arenât fanatical, but Iâm not sure about 4 and violations of IIA.
On variance voting: yeah, I think 4 is the point here. I donât think you can extend this approach to unbounded choice sets. Iâm travelling atm so canât be more formal, but hopefully tomorrow I can write up something a bit more detailed on this.
On bargaining-theory approaches, it actually isnât clear that they avoid fanaticism: see pp.24-26 of the Greaves and Cotton-Barratt paper you link, especially their conclusion that âthe Nash approach is not completely immune from fanaticismâ. Again, I think constructive ambiguity in the way these theories get described often helps obscure their relationship to fanatical conclusions; but thereâs an impossibility result here, so ultimately thereâs no avoiding the choice.
âVarious theories of moral uncertainty exist, outlining how this aggregation works; but none of them actually escape the issue. The theories of moral uncertainty that Effective Altruists rely on are themselves frameworks for commensurating values and systematically ranking options, and (as such) they are also vulnerable to âvalue dictatorshipâ, where after some point the choices recommended by utilitarianism come to swamp the recommendations of other theories. In the literature, this phenomenon is well-known as âfanaticismâ.[10]â
This seems too strong. Doesnât this only apply to maximizing expected choiceworthiness with intertheoretic comparisons, among popular approaches? My impression is that none of the other popular approaches are fanatical. You mention moral parliament in the footnote as a non-fanatical exception, but thereâs also MEC variance voting, formal bargain-theoretic approaches and my favourite theory.
Also, theories with value lexicality can swamp utilitarianism, although this also seems fanatical.
Hi Michael, thanks for your commentâsince it overlapped quite significantly with Leonardâs comment, I have tried to address both of your concerns together in one of my replies to Leonard. Let me know if I failed to reply sufficiently to some of your points!
I think the quote literally mischaracterizes approaches to normative uncertainty that EAs rely on as all leading to fanaticism, when almost none do, only really MEC with intertheoretical comparisons. Whether the ones that deny intertheoretical comparisons also reject universal domain is a separate issue.
Or maybe I misunderstood, and youâre only referring to versions of MEC with intertheoretical comparisons?
I think that approaches other than MEC + ITT arenât typically clear enough about universal domain, and can be interpreted in several different ways because theyâre not completely formally specified. (Not itself a criticism!) But (a corollary of the argument in this post) these frameworks only actually avoid fanaticism if they deny universal domain. So it isnât clear whether they fall victim too fanaticism, pace your claim that âalmost none doâ.
I thinkâpurely personal impressionâthat most people who think very hard about moral uncertainty are committed to quite a rationalistic view of ethics, and that this leads quite naturally to universal domain; so I said that, interpreting this into their frameworks, they will fall victim to fanaticism. But I mentioned the moral parliament as an example of an approach where the proponents have explicitly denied universal domain, as an example of the other possibility. However you interpret it, though, itâs an either/âor situation.
(Lexical orders /â deontological constraints are a separate issue, as you mention.)
Assuming all your moral theories are at least interval-scale, I think variance voting and similar normalization-based methods
are basically completely formally specified (at least as much as any other approach Iâve seen, like MEC+intertheoretical comparisons).
are never fanatical with respect to your credences in theories iff you normalize theories based on the options you actually have available and so violate the independence of irrelevant alternatives (IIA).
Maybe violating IIA conflicts with how you think about âuniversal domainâ, and IIA is usually assumed without much comment. If you think the quality of an outcome is based entirely on the features of that outcome, and this internal quality is all that matters when comparing outcoms, then IIA should hold.
donât require or make intertheoretical comparisons.
rank all options (outcome distributions, including fixed outcomes) for any finite set of available options when each moral theory ranks them all, too. Maybe you can extend to arbitrary size sets by normalizing each theory relative to the infimum and supremum value across options according to that theory. So, I suspect they satisfy something that can reasonably be called universal domain.
I think formal bargain-theoretic approaches (here and here) satisfy 1 and 3, arenât fanatical, but Iâm not sure about 4 and violations of IIA.
On variance voting: yeah, I think 4 is the point here. I donât think you can extend this approach to unbounded choice sets. Iâm travelling atm so canât be more formal, but hopefully tomorrow I can write up something a bit more detailed on this.
On bargaining-theory approaches, it actually isnât clear that they avoid fanaticism: see pp.24-26 of the Greaves and Cotton-Barratt paper you link, especially their conclusion that âthe Nash approach is not completely immune from fanaticismâ. Again, I think constructive ambiguity in the way these theories get described often helps obscure their relationship to fanatical conclusions; but thereâs an impossibility result here, so ultimately thereâs no avoiding the choice.