I have a bunch of comments on the specific decision theories, some adding to what you have here, and some corrections and recommendations, ~all of them minor and not detracting from the larger point of this post, but I think worth making anyway. Some of these are comments I’ve made elsewhere on the sequence.
Expectational total utilitarianism (and expected utility maximization with unbounded utility generally) also violates Continuity, because of St Petersburg prospects and/or infinities. If we’re counting it against discounting, we have some reason to do so for expectational total utilitarianism. I could imagine expectational total utilitarianism’s violations of Continuity being more intuitively acceptable, though.
Violating Continuity per se doesn’t seem instrumentally irrational or a big deal to me. The problem of attenuated fanaticism at the threshold you point out seems intuitively worse, but normative uncertainty over the threshold should smooth things out somewhat.
Discounting small probabilities or small probability differences also violates Independence, and is vulnerable to money pumps in theory. (I think it also doesn’t respect statewise dominance or stochastic dominance, which would be worse, but maybe fixable, by just ordering by dominance first and then filling in the rest of the missing comparisons with discounted expected utility?)
Expectational total utilitarianism (and expected utility maximization with an unbounded utility function generally) violates countable extensions of Independence and is vulnerable to similar money pumps and Dutch books in theory, which should also count against the theory. Maybe not as bad as how WLU and REU violate Independence or are vulnerable to money pumps, but still worth pointing out.
Expected utility maximization can be guaranteed to avoid fanaticism while satisfying the standard EUT axioms (and countable extensions), with a bounded utility function and the bounds small enough or marginal returns decreasing fast enough, in relative terms. Or, at least, say, a bounded function applied to differences in a difference-making version, but at the cost of stochastic dominance (specifically stochastic equivalence, not statewise dominance) and EUT. In my view, expected utility maximization with a bounded utility function (not difference-making) is the most instrumentally rational of the options, and it and boundedness with respect to differences seem the most promising, but have barely been discussed in the sequence (if it all?). I would recommend exploring these options more.
WLU can be guaranteed to avoid fanaticism, with the right choice of weighting function (or, in the difference-making version, at least). If your utility function is u, I think just picking your weighting function w so that w(y)*y is bounded and increasing in y(=u(x)) works, e.g. w(y) = f(y) / y, where f is bounded and increasing, with the bounds small enough. Then WLU looks a lot like maximizing the expected value of f(u(x)), just with some renormalization. But then you could just maximize the expected value of f(u(x)) instead (or some other bounded u instead directly), like in the previous point, to avoid violating Independence.
Expected utility maximization can be guaranteed to avoid fanaticism while satisfying the standard EUT axioms (and countable extensions), with a bounded utility function and the bounds small enough or marginal returns decreasing fast enough, in relative terms…
In my view, expected utility with a bounded utility function (not difference-making) is the most instrumentally rational of the options, and it and boundedness with respect to differences seem the most promising, but have barely been discussed in the sequence (if it all?). I would recommend exploring these options more.
I’m definitely in for exploring a variety of more options. We didn’t explore all possible options in this series, and I think we could, in theory, spend a lot more time investigating possible options including some of the combinations of theories, and more edge case versions of particular views like WLU you lay out.
However, I think while it is plausible EV could avoid some version of fanaticism that way, it still seems vulnerable to a very related issue like the following.
It seems there are actually two places for EV where rounding down or bound setting needs to happen to avoid issues with particularly risky gambles. (1) For really low probabilities (i.e. 1 in 100 trillion) with really high outcomes and (2) around the 50% line distinguishing actions that lean net positive from those that are neutral or negative in expectation. Conceptually, these are very similar but practically there may be different implications for doing them.
While it seems a bounded EV function with a function that assigns marginal returns a really steep decline could avoid the fanaticism of (1) (though this itself creates counterintuitive results), it doesn’t seem like this type of solution alone would resolve the issue where the the decision point is whether something is lean net positive but possibly only barely of (2).That is, there are many choices about actions where the sign of the action is uncertain and this applies, among other things, to x-risk interventions that have the possibility of having a very large expected utility if the action succeeds. Practically, it seems these types of choices are likely very common for charitable actors.
If despite a really large expected utility in your bounded function, you don’t think we should always take an action that is only, say, 50.0001% positive in expectation you wind up in a very similar place with regard to being “mugged” by high value outcomes that are not just unlikely to pay out but almost equally as likely to cause harm, then you think something has gone awry in EV. And it doesn’t seem reasonable bounds designed for avoiding really low probabilities but high EV outcomes will help you avoid this.
To be clear, I haven’t reasoned this out entirely, and I will just preemptively grant it’s possible you could create a different “bound” that would act on not just small probabilities, but also on these edge-cases where EU suggests taking these types of gambles. But if you do that this looks a lot like what you are doing is introducing a difference-making criteria to your decision theory. To the extent you may think this type of modified EU is viable, it is because it mimics the aversion of these other theories to certain types of uncertainty.
Basically, I’m actually not confident that this type of modification should matter much for us. The axiom choices matter here for which theory to put the most weight in but I’m unsure this type of distinction is buying you much practically if, say, after you make them you still end up with a set of theoretical options that look in practice like pure EV vs EV with rounding down vs something like WLU vs something like REU.
I agree it would be hard to avoid something like (2) with views that respect stochastic dominance with respect to the total welfare of outcomes, including background value (not difference-making). That includes maximizing the EV of a bounded increasing function of total welfare, as well as REU and WLU for total welfare, all with respect to outcomes including background value and not difference-making. Tarsney, 2020 makes it hard, and following it, x-risk reduction might be best across those views (Tarsney, 2023, footnote 43, although he says it could depend on the probabilities). See the following footnote for another possible exception with outcome risk aversion, relevant for extinction risk reduction.[1]
If you change the underlying order on outcomes from total welfare, you can also avoid nearly 50-50 actions from dominating things that are more likely to make a positive difference. A steep enough geometric discounting of future welfare[2] or a low enough future cutoff for consideration (a kind of view RP considered here) + excluding invertebrates might work.
I also think difference-making views, as you suggest, would avoid (2).
Basically, I’m actually not confident that this type of modification should matter much for us. The axiom choices matter here for which theory to put the most weight in but I’m unsure this type of distinction is buying you much practically if, say, after you make them you still end up with a set of theoretical options that look in practice like pure EV vs EV with rounding down vs something like WLU vs something like REU.
Tarsney, 2020 requires a lot of very uncertain background value that’s statistically independent from the effects of the intervention. Too little background value could be statistically independent, because a lot of things are jointly determined or correlated across the universe, e.g. sentience, moral weights, and, perhaps most importantly, (the sign of) the average welfare across the universe.
Conditional on generally horrible welfare across aliens (non-Earth-originating moral patients, generally), we should worry more that our descendants (or Earth-originating moral patients) will have horrible welfare if we don’t go extinct.
Then you just need to be sufficiently risk-averse, and something slighly better than 50-50 that could make things far worse could look bad overall.
I don’t know if this actually works in practice, though. It’ll depend on the particulars, and I’ve ignored our descendants’ possible effects on aliens.
Difference-making risk aversion (the accounts RP has considered, other than rounding/discounting) doesn’t necessarily avoid generalizations of (2), the 50-50 problem. It can
just shift the 50-50 problem to a different place, e.g. 70% good vs 30% bad being neutral in expectation but 70.0001% being extremely good in expectation, or
still have the 50-50 problem, but with unequal payoffs for good and bad, so be neutral at 50-50, but 50.0001% being extremely good in expectation.
To avoid these more general problems within standard difference-making accounts, I think you’d need to bound the differences you make from above. For example, apply a function that’s bounded above to the difference, or assume differences in value are bounded above).
On the other hand, maybe having the problem at 50-50 with equal magnitude but opposite sign payoffs is much worse, because our uninformed prior for the value of a random action is generally going to be symmetric around 0 net value.
Proofs below.
Assume you have an action with positive payoff x (compared to doing nothing) with probability p=50.0001%, and negative payoff y=-x otherwise, with x very large. Then
Holding the conditional payoffs x and -x constant, but changing the probabilities at 100% x and 0% y=-x, the act would be good overall. OTOH, it’s bad at 0% x and 100% y=-x. By Continuity (or the Intermediate Value Theorem), there has to be some p so that the act that’s x with probability p and y=-x with probability 1-p is neutral in expectation. Then we get the same problem at p, and a small probability like 0.0001% over p instead of p can make the action extremely good in expectation, if x was chosen to be large enough.
Holding the probability p=50% constant, if the negative payoff y were actually 0, and the positive payoff still x and large, the act would be good overall. It’s bad for y<0 low enough.[1] Then, by the Intermediate Value Theorem, there’s some y so that the act that’s x with probability 50% and y with probability 50% is neutral in expectation. And again, 50.0001% x and otherwise y can be extremely good in expectation, if x was chosen to be large enough.
Each can be avoided if the adjusted value of x is bounded and the bound is low enough, or x itself is bounded above with a low enough bound.
I think the same would apply to difference-making ambiguity aversion, too.
y=-x if difference-making risk averse, any y< -x if difference-making risk neutral, and generally for some y<0 if the disvalue of net harm isn’t bounded and the function is continuous.
I have a bunch of comments on the specific decision theories, some adding to what you have here, and some corrections and recommendations, ~all of them minor and not detracting from the larger point of this post, but I think worth making anyway. Some of these are comments I’ve made elsewhere on the sequence.
Expectational total utilitarianism (and expected utility maximization with unbounded utility generally) also violates Continuity, because of St Petersburg prospects and/or infinities. If we’re counting it against discounting, we have some reason to do so for expectational total utilitarianism. I could imagine expectational total utilitarianism’s violations of Continuity being more intuitively acceptable, though.
Violating Continuity per se doesn’t seem instrumentally irrational or a big deal to me. The problem of attenuated fanaticism at the threshold you point out seems intuitively worse, but normative uncertainty over the threshold should smooth things out somewhat.
Discounting small probabilities or small probability differences also violates Independence, and is vulnerable to money pumps in theory. (I think it also doesn’t respect statewise dominance or stochastic dominance, which would be worse, but maybe fixable, by just ordering by dominance first and then filling in the rest of the missing comparisons with discounted expected utility?)
Expectational total utilitarianism (and expected utility maximization with an unbounded utility function generally) violates countable extensions of Independence and is vulnerable to similar money pumps and Dutch books in theory, which should also count against the theory. Maybe not as bad as how WLU and REU violate Independence or are vulnerable to money pumps, but still worth pointing out.
Expected utility maximization can be guaranteed to avoid fanaticism while satisfying the standard EUT axioms (and countable extensions), with a bounded utility function and the bounds small enough or marginal returns decreasing fast enough, in relative terms. Or, at least, say, a bounded function applied to differences in a difference-making version, but at the cost of stochastic dominance (specifically stochastic equivalence, not statewise dominance) and EUT. In my view, expected utility maximization with a bounded utility function (not difference-making) is the most instrumentally rational of the options, and it and boundedness with respect to differences seem the most promising, but have barely been discussed in the sequence (if it all?). I would recommend exploring these options more.
WLU can be guaranteed to avoid fanaticism, with the right choice of weighting function (or, in the difference-making version, at least). If your utility function is u, I think just picking your weighting function w so that w(y)*y is bounded and increasing in y(=u(x)) works, e.g. w(y) = f(y) / y, where f is bounded and increasing, with the bounds small enough. Then WLU looks a lot like maximizing the expected value of f(u(x)), just with some renormalization. But then you could just maximize the expected value of f(u(x)) instead (or some other bounded u instead directly), like in the previous point, to avoid violating Independence.
Thanks for the engagement, Michael.
I largely agree with your notes and caveats.
However, on this:
I’m definitely in for exploring a variety of more options. We didn’t explore all possible options in this series, and I think we could, in theory, spend a lot more time investigating possible options including some of the combinations of theories, and more edge case versions of particular views like WLU you lay out.
However, I think while it is plausible EV could avoid some version of fanaticism that way, it still seems vulnerable to a very related issue like the following.
It seems there are actually two places for EV where rounding down or bound setting needs to happen to avoid issues with particularly risky gambles. (1) For really low probabilities (i.e. 1 in 100 trillion) with really high outcomes and (2) around the 50% line distinguishing actions that lean net positive from those that are neutral or negative in expectation. Conceptually, these are very similar but practically there may be different implications for doing them.
While it seems a bounded EV function with a function that assigns marginal returns a really steep decline could avoid the fanaticism of (1) (though this itself creates counterintuitive results), it doesn’t seem like this type of solution alone would resolve the issue where the the decision point is whether something is lean net positive but possibly only barely of (2).That is, there are many choices about actions where the sign of the action is uncertain and this applies, among other things, to x-risk interventions that have the possibility of having a very large expected utility if the action succeeds. Practically, it seems these types of choices are likely very common for charitable actors.
If despite a really large expected utility in your bounded function, you don’t think we should always take an action that is only, say, 50.0001% positive in expectation you wind up in a very similar place with regard to being “mugged” by high value outcomes that are not just unlikely to pay out but almost equally as likely to cause harm, then you think something has gone awry in EV. And it doesn’t seem reasonable bounds designed for avoiding really low probabilities but high EV outcomes will help you avoid this.
To be clear, I haven’t reasoned this out entirely, and I will just preemptively grant it’s possible you could create a different “bound” that would act on not just small probabilities, but also on these edge-cases where EU suggests taking these types of gambles. But if you do that this looks a lot like what you are doing is introducing a difference-making criteria to your decision theory. To the extent you may think this type of modified EU is viable, it is because it mimics the aversion of these other theories to certain types of uncertainty.
Basically, I’m actually not confident that this type of modification should matter much for us. The axiom choices matter here for which theory to put the most weight in but I’m unsure this type of distinction is buying you much practically if, say, after you make them you still end up with a set of theoretical options that look in practice like pure EV vs EV with rounding down vs something like WLU vs something like REU.
EDIT: grammar fix.
I agree it would be hard to avoid something like (2) with views that respect stochastic dominance with respect to the total welfare of outcomes, including background value (not difference-making). That includes maximizing the EV of a bounded increasing function of total welfare, as well as REU and WLU for total welfare, all with respect to outcomes including background value and not difference-making. Tarsney, 2020 makes it hard, and following it, x-risk reduction might be best across those views (Tarsney, 2023, footnote 43, although he says it could depend on the probabilities). See the following footnote for another possible exception with outcome risk aversion, relevant for extinction risk reduction.[1]
If you change the underlying order on outcomes from total welfare, you can also avoid nearly 50-50 actions from dominating things that are more likely to make a positive difference. A steep enough geometric discounting of future welfare[2] or a low enough future cutoff for consideration (a kind of view RP considered here) + excluding invertebrates might work.
I also think difference-making views, as you suggest, would avoid (2).
Fair. This seems right to me.
Tarsney, 2020 requires a lot of very uncertain background value that’s statistically independent from the effects of the intervention. Too little background value could be statistically independent, because a lot of things are jointly determined or correlated across the universe, e.g. sentience, moral weights, and, perhaps most importantly, (the sign of) the average welfare across the universe.
Conditional on generally horrible welfare across aliens (non-Earth-originating moral patients, generally), we should worry more that our descendants (or Earth-originating moral patients) will have horrible welfare if we don’t go extinct.
Then you just need to be sufficiently risk-averse, and something slighly better than 50-50 that could make things far worse could look bad overall.
I don’t know if this actually works in practice, though. It’ll depend on the particulars, and I’ve ignored our descendants’ possible effects on aliens.
And far away moral patients, if you accept acausal influence.
Difference-making risk aversion (the accounts RP has considered, other than rounding/discounting) doesn’t necessarily avoid generalizations of (2), the 50-50 problem. It can
just shift the 50-50 problem to a different place, e.g. 70% good vs 30% bad being neutral in expectation but 70.0001% being extremely good in expectation, or
still have the 50-50 problem, but with unequal payoffs for good and bad, so be neutral at 50-50, but 50.0001% being extremely good in expectation.
To avoid these more general problems within standard difference-making accounts, I think you’d need to bound the differences you make from above. For example, apply a function that’s bounded above to the difference, or assume differences in value are bounded above).
On the other hand, maybe having the problem at 50-50 with equal magnitude but opposite sign payoffs is much worse, because our uninformed prior for the value of a random action is generally going to be symmetric around 0 net value.
Proofs below.
Assume you have an action with positive payoff x (compared to doing nothing) with probability p=50.0001%, and negative payoff y=-x otherwise, with x very large. Then
Holding the conditional payoffs x and -x constant, but changing the probabilities at 100% x and 0% y=-x, the act would be good overall. OTOH, it’s bad at 0% x and 100% y=-x. By Continuity (or the Intermediate Value Theorem), there has to be some p so that the act that’s x with probability p and y=-x with probability 1-p is neutral in expectation. Then we get the same problem at p, and a small probability like 0.0001% over p instead of p can make the action extremely good in expectation, if x was chosen to be large enough.
Holding the probability p=50% constant, if the negative payoff y were actually 0, and the positive payoff still x and large, the act would be good overall. It’s bad for y<0 low enough.[1] Then, by the Intermediate Value Theorem, there’s some y so that the act that’s x with probability 50% and y with probability 50% is neutral in expectation. And again, 50.0001% x and otherwise y can be extremely good in expectation, if x was chosen to be large enough.
Each can be avoided if the adjusted value of x is bounded and the bound is low enough, or x itself is bounded above with a low enough bound.
I think the same would apply to difference-making ambiguity aversion, too.
y=-x if difference-making risk averse, any y< -x if difference-making risk neutral, and generally for some y<0 if the disvalue of net harm isn’t bounded and the function is continuous.