To me it seems the main concern is with using expected value maximization, not with longtermism. Rather than being rationally required to take an action with the highest expected value, I think you are probably only rationally required not to take any action resulting in a world that is worse than an alternative at every percentile of the probability distribution. So in this case you would not have to take the bet because at the 0.1st percentile of the probability distribution taking the bet has a lower value than status quo, while at the 99th percentile it has a higher value.
In practice, this still ends up looking approximately like expected value maximization for most EA decisions because of the huge background uncertainty about what the world will look like. (My current understanding is that you can think of this as an extended version of “if everyone in EA took risky high EV options, then the aggregate result will pretty consistently/with low risk be near the total expected value”)
(Note: I’ve made several important additions to this comment within the first ~30 minutes of posting it, plus some more minor edits after.)
I think this is an important point, so I’ve given you a strong upvote. Still, I think total utilitarians aren’t rationally required to endorse EV maximization or longtermism, even approximately except under certain other assumptions.
Tarsney has also written that stochastic dominance doesn’t lead to EV maximization or longtermism under total utilitarianism, if the probabilities (probability differences) are low enough, and has said it’s plausible the probabilities are in fact that low (not that he said it’s his best guess they’re that low). See “The epistemic challenge to longtermism”, and especially footnote 41.
It’s also not clear to me that we shouldn’t just ignore background noise that’s unaffected by our actions or generally balance other concerns against stochastic dominance, like risk aversion or ambiguity aversion, particularly with respect to the difference one makes, as discussed in “The case for strong longtermism” by Greaves and MacAskill in section 7.5. Greaves and MacAskill do argue that ambiguity aversion with respect to the outcomes doesn’t point against existential risk reduction, and if I recall correctly from following citations, that ambiguity aversion with respect to the difference one makes is too agent-relative.
On the other hand, using your own precise subjective probabilities to define rational requirement seems pretty agent-relative to me, too. Surely, if the correct ethics is fully agent-neutral, you should be required to do what actually maximizes value among available options, regardless of your own particular beliefs about what’s best. Or, at least, precise subjective probabilities seem hard to defend as agent-neutral, when different rational agents could have different beliefs even with access to the same information, due to different priors or because they weigh evidence differently.
Plus, without separability (ignoring what’s unaffected) in the first place, the case for utilitarianism itself seems much weaker, since the representation theorems that imply utilitarianism, like Harsanyi’s (and generalization here) and the deterministic ones like the one here, require separability or something similar.
FWIW, stochastic dominance is a bit stronger than you write here, since you can allow A to strictly beat B at only some quantiles, but equality at the rest, and then A dominates B.
There have of course been arguments people have raised against some of the axioms—I think most commonly people argue against axioms 3 and 4 from the link.
Thank you for pointing me to that and getting me to think critically about it. I think I agree with all the axioms.
a rational agent should act as to maximize expected value of their value function
I think this is misleading. The VNM theorem only says that there exists a function u such that a rational agent’s actions maximize E[u]. But u does not have to be “their value function.”
Consider a scenario in which there are 3 possible outcomes: A1 = enormous suffering, A2 = neutral, A3= mild joy. Let’s say my value function is v(A1)=−9,v(A2)=0, and v(A3)=1, in the intuitive sense of the word “value.”
When I work through the proof you sent in this example, I am forced to prefer pA1+(1−p)A3 for some probability p, but this probability does not have to be 0.1, so I don’t have to maximize my expected value. In reality, I would be “risk averse” and assign p=0.05 or something. See 4.1Automatic consideration of risk aversion.
More details of how I filled in the proof:
We normalize my value function so u(A1)=0 and u(A3)=1. Then we define u(A2)=q2.
Let M=1⋅A2, then M′=q2A3+(1−q2)A1, and I am indifferent between M andM′. However, nowhere did I specify what q2 is, so “there exists a function u such that I’m maximizing the expectation of it” is not that meaningful, because it does not have to align with the value I assign to the event.
I think you are probably only rationally required not to take any action resulting in a world that is worse than an alternative at every percentile of the probability distribution
I think this is probably wrong, and I view stochastic dominance as a backup decision rule, not as a total replacement for expected value. Some thoughts here.
I think Dutch book/money pump arguments require you to rank unrealistic hypotheticals (e.g. where your subjective probabilities in, say, extinction risk are predictably manipulated by an adversary), and the laws of large numbers and central limit theorems can have limited applicability, if there are too few statistically independent outcomes.
Even much of our uncertainty should be correlated across agents in a multiverse, e.g. uncertainty about logical implications, facts or tendencies about the world. We can condition on some of those uncertain possibilities separately, apply the LLN or CLT to each across the multiverse, and then aggregate over the conditions, but I’m not convinced this works out to give you EV maximization.
To me it seems the main concern is with using expected value maximization, not with longtermism. Rather than being rationally required to take an action with the highest expected value, I think you are probably only rationally required not to take any action resulting in a world that is worse than an alternative at every percentile of the probability distribution. So in this case you would not have to take the bet because at the 0.1st percentile of the probability distribution taking the bet has a lower value than status quo, while at the 99th percentile it has a higher value.
In practice, this still ends up looking approximately like expected value maximization for most EA decisions because of the huge background uncertainty about what the world will look like. (My current understanding is that you can think of this as an extended version of “if everyone in EA took risky high EV options, then the aggregate result will pretty consistently/with low risk be near the total expected value”)
See this episode of the 80,000 hours podcast for a good description of this “stochastic dominance” framework: https://80000hours.org/podcast/episodes/christian-tarsney-future-bias-fanaticism/.
(Note: I’ve made several important additions to this comment within the first ~30 minutes of posting it, plus some more minor edits after.)
I think this is an important point, so I’ve given you a strong upvote. Still, I think total utilitarians aren’t rationally required to endorse EV maximization or longtermism, even approximately except under certain other assumptions.
Tarsney has also written that stochastic dominance doesn’t lead to EV maximization or longtermism under total utilitarianism, if the probabilities (probability differences) are low enough, and has said it’s plausible the probabilities are in fact that low (not that he said it’s his best guess they’re that low). See “The epistemic challenge to longtermism”, and especially footnote 41.
It’s also not clear to me that we shouldn’t just ignore background noise that’s unaffected by our actions or generally balance other concerns against stochastic dominance, like risk aversion or ambiguity aversion, particularly with respect to the difference one makes, as discussed in “The case for strong longtermism” by Greaves and MacAskill in section 7.5. Greaves and MacAskill do argue that ambiguity aversion with respect to the outcomes doesn’t point against existential risk reduction, and if I recall correctly from following citations, that ambiguity aversion with respect to the difference one makes is too agent-relative.
On the other hand, using your own precise subjective probabilities to define rational requirement seems pretty agent-relative to me, too. Surely, if the correct ethics is fully agent-neutral, you should be required to do what actually maximizes value among available options, regardless of your own particular beliefs about what’s best. Or, at least, precise subjective probabilities seem hard to defend as agent-neutral, when different rational agents could have different beliefs even with access to the same information, due to different priors or because they weigh evidence differently.
Plus, without separability (ignoring what’s unaffected) in the first place, the case for utilitarianism itself seems much weaker, since the representation theorems that imply utilitarianism, like Harsanyi’s (and generalization here) and the deterministic ones like the one here, require separability or something similar.
FWIW, stochastic dominance is a bit stronger than you write here, since you can allow A to strictly beat B at only some quantiles, but equality at the rest, and then A dominates B.
Relevant: The von Neumann-Morgenstern utility theorem shows that under certain reasonable seeming axioms, a rational agent should act as to maximize expected value of their value function: https://en.wikipedia.org/wiki/Von_Neumann%E2%80%93Morgenstern_utility_theorem
There have of course been arguments people have raised against some of the axioms—I think most commonly people argue against axioms 3 and 4 from the link.
Thank you for pointing me to that and getting me to think critically about it. I think I agree with all the axioms.
I think this is misleading. The VNM theorem only says that there exists a function u such that a rational agent’s actions maximize E[u]. But u does not have to be “their value function.”
Consider a scenario in which there are 3 possible outcomes: A1 = enormous suffering, A2 = neutral, A3= mild joy. Let’s say my value function is v(A1)=−9,v(A2)=0, and v(A3)=1, in the intuitive sense of the word “value.”
When I work through the proof you sent in this example, I am forced to prefer pA1+(1−p)A3 for some probability p, but this probability does not have to be 0.1, so I don’t have to maximize my expected value. In reality, I would be “risk averse” and assign p=0.05 or something. See 4.1Automatic consideration of risk aversion.
More details of how I filled in the proof:
We normalize my value function so u(A1)=0 and u(A3)=1. Then we define u(A2)=q2.
Let M=1⋅A2, then M′=q2A3+(1−q2)A1, and I am indifferent between M andM′. However, nowhere did I specify what q2 is, so “there exists a function u such that I’m maximizing the expectation of it” is not that meaningful, because it does not have to align with the value I assign to the event.
I think this is probably wrong, and I view stochastic dominance as a backup decision rule, not as a total replacement for expected value. Some thoughts here.
Why try to maximize EV at all, though?
I think Dutch book/money pump arguments require you to rank unrealistic hypotheticals (e.g. where your subjective probabilities in, say, extinction risk are predictably manipulated by an adversary), and the laws of large numbers and central limit theorems can have limited applicability, if there are too few statistically independent outcomes.
Even much of our uncertainty should be correlated across agents in a multiverse, e.g. uncertainty about logical implications, facts or tendencies about the world. We can condition on some of those uncertain possibilities separately, apply the LLN or CLT to each across the multiverse, and then aggregate over the conditions, but I’m not convinced this works out to give you EV maximization.