A common mistake I see people make in their consequentialist analysis is to only consider one level of counterfactuals. Whereas in reality, to figure out correct counterfactual utility requires you to, in some sense, chain counterfactuals all the way through. And only looking at first-level counterfactuals can in some cases be worse than not looking at counterfactuals at all.
Toy examples:
Ali is trying to figure out what job to get for the next year. He can choose to be a mechanic at the Utility Factory, an independent researcher of Goodness Studies, or earning-to-give.
Ali is trying to be cooperative, and asks each org what the nearest counterfactual is, and several charities on the value of money. He concludes that:
As a mechanic at the Utility Factory, Ali can naively produce 30 utility. However, Barbara (the Utility Factory’s counterfactual hire) can produce 25 utility. So Ali’s counterfactual value-add at Utility Factory is 30-25 utility = 5 utility.
As an independent researcher of Goodness Studies, Ali can produce 10 utility. He asked the Altruism Fund, and their marginal grant (at Ali’s stipend) produces 4 utility. So Ali’s counterfactual value at Goodness Studies is only 6 utility.
If he does earning-to-give, Ali can produce 10 utility. So Ali decides to earn-to-give.
Casey is trying to figure out which projects to donate to. They figure that a donation to cover the Optimal Charity’s funding gap provides 10 utility. However, they think that if they don’t fund Optimal Charity, there’s an 80% chance that the Altruism Fund will cover it, whereas the Kinda Good Charity can produce 3 utility for the same money. Casey concludes that (100%-80%) * 10 =2 utility, and 2<3, so they can produce more utility by donating to the Kinda Good charity than the Optimal Charity.
Dina is trying to decide whether to volunteer at the Cosmopolitan Conference. She asks the organizers about counterfactuality, and the organizers say that they almost always find volunteers eventually, and her volunteering skills are not unusually in high demand at the Cosmopolitan Conference. Dina concludes that her marginal value as a volunteer at the Cosmopolitan Conference is ~0, and thus she can better use her time elsewhere.
I claim that all the people above are making (similar) key mistakes in their moral math.
In Ali’s case, he models his value-add over Barbara, but neglects to consider that Barbara’s counterfactual value not working at the Utility Factory isn’t 0. Instead Barbara can probably do good elsewhere if she’s not working at the Utility Factory. But Ali can’t just account for Barbara’s counterfactual either, as maybe Barbara’s nearby job might free up Eden...and so forth.
in theory the chain can last forever, in practice I think there is a reasonable stopping point once you reach someone whose counterfactuals are roughly “just contribute to the world by being a normal person participating in the global economy”
Casey considers that his marginal donation to Optimal Charity might not have a large impact as the Altruism Fund can with high probability cover it, but neglects to consider that freeing up the Altruism Fund’s $s can also lead to the Altruism Fund funding other projects.
Dina considers her own time not spent volunteering to locally have more marginal impact than her time volunteering, but neglects to consider that other volunteers might also counterfactually produce impact if they aren’t volunteering.
In addition to the problem of insufficent chaining of the direct counterfactuals, the altruists above are also failing to track the relevant transaction costs.
Hiring, grantmaking, finding volunteers etc, takes time and usually money. It’s not free to wait for a perfect replacement for someone if work needs to be done right now.
I don’t have a good way to avoid this. In the meantime, some ideas:
Thinking about impact in terms of “impact equity” instead of counterfactuals might be fruitful
If that’s too hard, one heuristic is to weight 0th level impact (eg how much impact your job directly produces) more than you did previously, rather than immediately jump to looking at the 1st level counterfactual.
Another heuristic is to weight common sense morality as well as non-moral considerations more than you would otherwise endorse.
I also think that another important issue is that reasoning on counterfactuals makes people more prone to do things that are unusual AND is more prone to errors (e.g. by not taking into account some other effects).
Both combined make counterfactual reasoning without empirical data pretty perilous on average IMO.
In the case of Ali in your example above for instance, Ali could neglect that the performance he’ll have will determine the opportunities & impact he has 5y down the line and so that being excited/liking the job is a major variable. Without counterfactual reasoning, Ali would have intuitively relied much more on excitement to pick the job but by doing counterfactual reasoning which seemed convincing, he neglected this important variable and made a bad choice.
I think that counterfactual reasoning makes people very prone to ignoring Chesterton’s fence.
On the topic of coordination, readers may want to check the concept of Shapley value. It is an extension of counterfactual value, and is uniquely determined by the following properties:
Property 1: Sum of the values adds up to the total value (Efficiency)
Property 2: Equal agents have equal value (Symmetry)
Property 3: Order indifference: it doesn’t matter which order you go in (Linearity). Or, in other words, if there are two steps, Value(Step1 + Step2) = Value(Step1) + Value(Step2).
This resonates with my experience in the financial markets especially in the derivative markets. Quite often the financial market is quite efficient in the first order, but less efficient in second and third order where it is more neglected. And further down the orders the significance diminishes even if neglected.
I wrote about this in Impact above Replacement, and suggested that a better way of thinking about counterfactual impact is via what I called the replacement view, “where your impact is the value you produced using some amount of resources, minus the value the ‘replacement-level person’ of that reference group would’ve produced using those resources”.
This is an effective altruist variant of what sabermetricians call Wins Above Replacement, a stat that aims to measure how many more wins a baseball player contributes to a team than the replacement-level player. The replacement-level player is the substitute who would’ve been called up to join the team had the player being measured not participated.
Still, there are issues with that way of looking at things too, e.g., it’s somewhat unclear which reference group to use, also I’m not sure it’s conceptually sound (though it seems better than naive alternatives).
Be careful with naive counterfactuals
A common mistake I see people make in their consequentialist analysis is to only consider one level of counterfactuals. Whereas in reality, to figure out correct counterfactual utility requires you to, in some sense, chain counterfactuals all the way through. And only looking at first-level counterfactuals can in some cases be worse than not looking at counterfactuals at all.
Toy examples:
Ali is trying to figure out what job to get for the next year. He can choose to be a mechanic at the Utility Factory, an independent researcher of Goodness Studies, or earning-to-give.
Ali is trying to be cooperative, and asks each org what the nearest counterfactual is, and several charities on the value of money. He concludes that:
As a mechanic at the Utility Factory, Ali can naively produce 30 utility. However, Barbara (the Utility Factory’s counterfactual hire) can produce 25 utility. So Ali’s counterfactual value-add at Utility Factory is 30-25 utility = 5 utility.
As an independent researcher of Goodness Studies, Ali can produce 10 utility. He asked the Altruism Fund, and their marginal grant (at Ali’s stipend) produces 4 utility. So Ali’s counterfactual value at Goodness Studies is only 6 utility.
If he does earning-to-give, Ali can produce 10 utility. So Ali decides to earn-to-give.
Casey is trying to figure out which projects to donate to. They figure that a donation to cover the Optimal Charity’s funding gap provides 10 utility. However, they think that if they don’t fund Optimal Charity, there’s an 80% chance that the Altruism Fund will cover it, whereas the Kinda Good Charity can produce 3 utility for the same money. Casey concludes that (100%-80%) * 10 =2 utility, and 2<3, so they can produce more utility by donating to the Kinda Good charity than the Optimal Charity.
Dina is trying to decide whether to volunteer at the Cosmopolitan Conference. She asks the organizers about counterfactuality, and the organizers say that they almost always find volunteers eventually, and her volunteering skills are not unusually in high demand at the Cosmopolitan Conference. Dina concludes that her marginal value as a volunteer at the Cosmopolitan Conference is ~0, and thus she can better use her time elsewhere.
I claim that all the people above are making (similar) key mistakes in their moral math.
In Ali’s case, he models his value-add over Barbara, but neglects to consider that Barbara’s counterfactual value not working at the Utility Factory isn’t 0. Instead Barbara can probably do good elsewhere if she’s not working at the Utility Factory. But Ali can’t just account for Barbara’s counterfactual either, as maybe Barbara’s nearby job might free up Eden...and so forth.
in theory the chain can last forever, in practice I think there is a reasonable stopping point once you reach someone whose counterfactuals are roughly “just contribute to the world by being a normal person participating in the global economy”
Casey considers that his marginal donation to Optimal Charity might not have a large impact as the Altruism Fund can with high probability cover it, but neglects to consider that freeing up the Altruism Fund’s $s can also lead to the Altruism Fund funding other projects.
Dina considers her own time not spent volunteering to locally have more marginal impact than her time volunteering, but neglects to consider that other volunteers might also counterfactually produce impact if they aren’t volunteering.
In addition to the problem of insufficent chaining of the direct counterfactuals, the altruists above are also failing to track the relevant transaction costs.
Hiring, grantmaking, finding volunteers etc, takes time and usually money. It’s not free to wait for a perfect replacement for someone if work needs to be done right now.
I don’t have a good way to avoid this. In the meantime, some ideas:
Thinking about impact in terms of “impact equity” instead of counterfactuals might be fruitful
If that’s too hard, one heuristic is to weight 0th level impact (eg how much impact your job directly produces) more than you did previously, rather than immediately jump to looking at the 1st level counterfactual.
Another heuristic is to weight common sense morality as well as non-moral considerations more than you would otherwise endorse.
I agree with the general underlying point.
I also think that another important issue is that reasoning on counterfactuals makes people more prone to do things that are unusual AND is more prone to errors (e.g. by not taking into account some other effects).
Both combined make counterfactual reasoning without empirical data pretty perilous on average IMO.
In the case of Ali in your example above for instance, Ali could neglect that the performance he’ll have will determine the opportunities & impact he has 5y down the line and so that being excited/liking the job is a major variable. Without counterfactual reasoning, Ali would have intuitively relied much more on excitement to pick the job but by doing counterfactual reasoning which seemed convincing, he neglected this important variable and made a bad choice.
I think that counterfactual reasoning makes people very prone to ignoring Chesterton’s fence.
Nice point, Linch!
On the topic of coordination, readers may want to check the concept of Shapley value. It is an extension of counterfactual value, and is uniquely determined by the following properties:
This resonates with my experience in the financial markets especially in the derivative markets. Quite often the financial market is quite efficient in the first order, but less efficient in second and third order where it is more neglected. And further down the orders the significance diminishes even if neglected.
Ben Todd’s articles The value of coordination and its more updated version Doing good together: how to coordinate effectively and avoid single-player thinking over at 80,000 Hours seem relevant.
I wrote about this in Impact above Replacement, and suggested that a better way of thinking about counterfactual impact is via what I called the replacement view, “where your impact is the value you produced using some amount of resources, minus the value the ‘replacement-level person’ of that reference group would’ve produced using those resources”.
Still, there are issues with that way of looking at things too, e.g., it’s somewhat unclear which reference group to use, also I’m not sure it’s conceptually sound (though it seems better than naive alternatives).