Another use of “consequentialism” in decision theory is in dynamic choice settings (i.e. where an agent makes several choices over time, and future choices and payoffs typically depend on past choices). Consequentialist decision rules depend only on the future choices and payoffs and decision rules that violate consequentialism in this sense sometimes depend on past choices.
An example: suppose an agent is deciding whether to take a pleasurable but addictive drug. If the agent takes the drug, they then decide whether to stop taking it or to continue taking it. Suppose the agent initially judges taking the drug once to have the highest payoff, continuing to take the drug to have the lowest payoff and never taking it to be in between. Suppose further though, that if the agent takes the drug, they will immediately become addicted and will then prefer to carry on taking it to stopping. One decision rule in the dynamic choice literature is called “resolute choice” and requires the agent to take the drug once and then stop, because this brings about the highest payoff, as judged initially. This is a non-consequentialist decision rule because at the second choice point (carry on taking the drug or stop), the agent follows their previously made plan and stops, even though it goes against their current preference to carry on.
I don’t know how, if at all, this relates to what Yudkowsky means by “consequentialism”, but this seems sufficiently different from what you described as “decision consequentialism” that I thought it was worth adding, in case it’s a further source of confusion.
Another use of “consequentialism” in decision theory is in dynamic choice settings (i.e. where an agent makes several choices over time, and future choices and payoffs typically depend on past choices). Consequentialist decision rules depend only on the future choices and payoffs and decision rules that violate consequentialism in this sense sometimes depend on past choices.
An example: suppose an agent is deciding whether to take a pleasurable but addictive drug. If the agent takes the drug, they then decide whether to stop taking it or to continue taking it. Suppose the agent initially judges taking the drug once to have the highest payoff, continuing to take the drug to have the lowest payoff and never taking it to be in between. Suppose further though, that if the agent takes the drug, they will immediately become addicted and will then prefer to carry on taking it to stopping. One decision rule in the dynamic choice literature is called “resolute choice” and requires the agent to take the drug once and then stop, because this brings about the highest payoff, as judged initially. This is a non-consequentialist decision rule because at the second choice point (carry on taking the drug or stop), the agent follows their previously made plan and stops, even though it goes against their current preference to carry on.
I don’t know how, if at all, this relates to what Yudkowsky means by “consequentialism”, but this seems sufficiently different from what you described as “decision consequentialism” that I thought it was worth adding, in case it’s a further source of confusion.