Note that risk-weighted utility theories (REUT) are not the only way of rationalising Sheila’s decision-making. For example, it might be the case that when Sheila goes to Leighton Moss and sees no birds, she feels an immense regret, because post-hoc she knows she would have been better off if she’d gone to Shapwick Heath. So the decision matrix looks like this, where δ is the loss of utility due to Sheila’s regret at the decision she made.
That is, I have re-described the situation such that even using expected utility theory, for large enough δ, Sheila is still being rational to go to Shapwick Heath. Of course, where Sheila would not feel such regret (δ= 0), or the regret would not be large enough to change the result, then on expected utility theory (EUT) Sheila is not acting rationally.
It’s not clear to me how to choose between EUT and REUT for describing situations like this. My intuition is that, if upon making the decision I could immediately forget that I had done so, then I would not succumb to the Allais paradox (though I probably would succumb to it right now). So I’m weakly inclined to stick with EU. But I haven’t thought about this long and hard.
The relevance to AI strategy
Considerations about risk-aversion are relevant for evaluating different AGI outcomes and therefore are relevant for determining AI strategy. See Holden Karnofsky’s Questions about AI strategy: ‘insights about how we should value “paperclipping” vs. other outcomes could be as useful as insights about how likely we should consider paperclipping to be’. In this case “other outcomes” would include the long miserable and the long happy future.
Pushing back on the second conclusion
While I agree with the first of the conclusions (“if you are sufficiently averse to risk, and you wish to donate some money to do good, then you should donate it to organisations working to hasten human extinction rather than ones working to prevent it”), I don’t agree with the second (“whether or not you are averse to risk, this is what you should do”).
In essence, I think the total hedonist utilitarian has no intrinsic reason to take into account the risk-preferences of other people, even if those people would be affected by the decision.
To see this, imagine that Sheila is facing a classic scenario in moral philosophy. A trolley is on track to run over five people, but Sheila has the chance to divert the trolley such that it only runs over one person. Sheila is a total hedonist utilitarian and believes the world would be better off if she diverted the trolley. We can easily imagine Sheila’s friend Zero having different moral views and coming to the opposite conclusion. Sheila is on the phone to Zero while she is making this decision. Zero explains the reasons he has for his moral values and why he thinks she should not divert the trolley. Sheila listens, but by the end is sure that she does not agree with Zero’s axiology. Her decision theory suggests she should act so as to maximise the amount of utility in the world and divert the trolley. Given her moral values, to do otherwise would be to act irrationally by her own lights.
If Sheila is risk-neutral and is the leader in the rock-climbing example above, then she has no intrinsic reason to respect the risk preferences of those she affects. That is, the total hedonist utilitarian must bite the bullet and generally reject Pettigrew’s conclusion from the rock-climbing example that “she should descend.” Of course, in this specific case, imposing risks on others could annoy them and this annoyance could plausibly outweigh the value to the total hedonist utilitarian of carrying on up the mountain.
I think there are good deontological reasons to intrinsically respect the risk-preferences of those around you. But to the extent that the second argument is aimed at the total hedonist utilitarian, I do not think it goes through. For a longer explanation of this argument, see this essay where I responded to an earlier draft of this paper. And I’d love to hear where people think my response goes wrong.
Thanks for the comment, Alejandro, and apologies for the delay responding—a vacation and then a bout of covid.
On the first: yes, it’s definitely true that there are other theories besides Buchak’s that capture the Allais preferences, and a regret averse theory that incorporates the regret into the utilities would do that. So we’d have to look at a particular regret-averse theory and then look to cases in which it and Buchak’s theory come apart and see what we make of them. Buchak herself does offer an explicit argument in favour of her theory that goes beyond just appeal to the intuitive response to Allais. But it’s quite involved and it might not have the same force.
On the second: thanks for the link!
On the third: I’m not so sure about the analogy with Zero. It’s true that Sheila needn’t defer to Zero’s axiology, since we might think that which axiology is correct is a matter of fact and so Sheila might just think she’s right and Zero is wrong. But risk attitudes aren’t like that. They’re not matters of fact, but something more like subjective preferences. But I can see that it is certainly a consistent moral view to be total hedonist utilitarian and think that you have no obligation to take the risk attitudes of others into account. I suppose I just think that it’s not the correct moral view, even for someone whose axiology is total hedonist utilitarian. For them, that axiology should supply their utility function for moral decisions, but their risk attitudes should be supplied in the way suggested by the Risk Principle*. But I’m not clear how to adjudicate this.
Welcome to the EA forum Richard!
Three comments:
Should we use risk-averse decision theories?
Note that risk-weighted utility theories (REUT) are not the only way of rationalising Sheila’s decision-making. For example, it might be the case that when Sheila goes to Leighton Moss and sees no birds, she feels an immense regret, because post-hoc she knows she would have been better off if she’d gone to Shapwick Heath. So the decision matrix looks like this, where δ is the loss of utility due to Sheila’s regret at the decision she made.
That is, I have re-described the situation such that even using expected utility theory, for large enough δ, Sheila is still being rational to go to Shapwick Heath. Of course, where Sheila would not feel such regret (δ= 0), or the regret would not be large enough to change the result, then on expected utility theory (EUT) Sheila is not acting rationally.
It’s not clear to me how to choose between EUT and REUT for describing situations like this. My intuition is that, if upon making the decision I could immediately forget that I had done so, then I would not succumb to the Allais paradox (though I probably would succumb to it right now). So I’m weakly inclined to stick with EU. But I haven’t thought about this long and hard.
The relevance to AI strategy
Considerations about risk-aversion are relevant for evaluating different AGI outcomes and therefore are relevant for determining AI strategy. See Holden Karnofsky’s Questions about AI strategy: ‘insights about how we should value “paperclipping” vs. other outcomes could be as useful as insights about how likely we should consider paperclipping to be’. In this case “other outcomes” would include the long miserable and the long happy future.
Pushing back on the second conclusion
While I agree with the first of the conclusions (“if you are sufficiently averse to risk, and you wish to donate some money to do good, then you should donate it to organisations working to hasten human extinction rather than ones working to prevent it”), I don’t agree with the second (“whether or not you are averse to risk, this is what you should do”).
In essence, I think the total hedonist utilitarian has no intrinsic reason to take into account the risk-preferences of other people, even if those people would be affected by the decision.
To see this, imagine that Sheila is facing a classic scenario in moral philosophy. A trolley is on track to run over five people, but Sheila has the chance to divert the trolley such that it only runs over one person. Sheila is a total hedonist utilitarian and believes the world would be better off if she diverted the trolley. We can easily imagine Sheila’s friend Zero having different moral views and coming to the opposite conclusion. Sheila is on the phone to Zero while she is making this decision. Zero explains the reasons he has for his moral values and why he thinks she should not divert the trolley. Sheila listens, but by the end is sure that she does not agree with Zero’s axiology. Her decision theory suggests she should act so as to maximise the amount of utility in the world and divert the trolley. Given her moral values, to do otherwise would be to act irrationally by her own lights.
If Sheila is risk-neutral and is the leader in the rock-climbing example above, then she has no intrinsic reason to respect the risk preferences of those she affects. That is, the total hedonist utilitarian must bite the bullet and generally reject Pettigrew’s conclusion from the rock-climbing example that “she should descend.” Of course, in this specific case, imposing risks on others could annoy them and this annoyance could plausibly outweigh the value to the total hedonist utilitarian of carrying on up the mountain.
I think there are good deontological reasons to intrinsically respect the risk-preferences of those around you. But to the extent that the second argument is aimed at the total hedonist utilitarian, I do not think it goes through. For a longer explanation of this argument, see this essay where I responded to an earlier draft of this paper. And I’d love to hear where people think my response goes wrong.
Thanks for the comment, Alejandro, and apologies for the delay responding—a vacation and then a bout of covid.
On the first: yes, it’s definitely true that there are other theories besides Buchak’s that capture the Allais preferences, and a regret averse theory that incorporates the regret into the utilities would do that. So we’d have to look at a particular regret-averse theory and then look to cases in which it and Buchak’s theory come apart and see what we make of them. Buchak herself does offer an explicit argument in favour of her theory that goes beyond just appeal to the intuitive response to Allais. But it’s quite involved and it might not have the same force.
On the second: thanks for the link!
On the third: I’m not so sure about the analogy with Zero. It’s true that Sheila needn’t defer to Zero’s axiology, since we might think that which axiology is correct is a matter of fact and so Sheila might just think she’s right and Zero is wrong. But risk attitudes aren’t like that. They’re not matters of fact, but something more like subjective preferences. But I can see that it is certainly a consistent moral view to be total hedonist utilitarian and think that you have no obligation to take the risk attitudes of others into account. I suppose I just think that it’s not the correct moral view, even for someone whose axiology is total hedonist utilitarian. For them, that axiology should supply their utility function for moral decisions, but their risk attitudes should be supplied in the way suggested by the Risk Principle*. But I’m not clear how to adjudicate this.