I’m not sure naive total utility maximization [in a static framework] is the best framework to be thinking about dealing with existential risk over time.[1]
Assuming the number of risks and error bars are not trivially small, the universal outcome of concentrating all your risk mitigations on one is that most risks continue to be a high as they could possibly be. The modal outcome is that the risks ignored includes at least one risk greater than the one all efforts are concentrated on mitigating. Some reasonable assumptions in the article above show this can hold even where the actual biggest risk is orders of magnitude greater than the one targeted. In the diversified approach, less money are devoted to reducing the perceived biggest risk, but the rest is apportioned to reducing other risks. This seems more robust to conventional assumptions like uncertainty and some risks being easier to mitigate than others.
And tbh I’m not even seeing an average utility boost from concentrating on the single largest risk as opposed to mitigating lots of risks without ancillary assumptions like increasing returns to risk reduction expenditure or the actual value of many risks under consideration being 0.
Yeah I agree—expected utility maximisation really starts to fall apart in this existential risk regime, even over trajectories rather than applied statically, and it only makes sense “locally” and at the margin.
Personally I’m very happy to bite the bullet and not be rigorously utilitarian, but I’m also a global health focussed “old school EA” thinking about how much to diversify donations across charities ;)
I’m not sure naive total utility maximization [in a static framework] is the best framework to be thinking about dealing with existential risk over time.[1]
Assuming the number of risks and error bars are not trivially small, the universal outcome of concentrating all your risk mitigations on one is that most risks continue to be a high as they could possibly be. The modal outcome is that the risks ignored includes at least one risk greater than the one all efforts are concentrated on mitigating. Some reasonable assumptions in the article above show this can hold even where the actual biggest risk is orders of magnitude greater than the one targeted. In the diversified approach, less money are devoted to reducing the perceived biggest risk, but the rest is apportioned to reducing other risks. This seems more robust to conventional assumptions like uncertainty and some risks being easier to mitigate than others.
And tbh I’m not even seeing an average utility boost from concentrating on the single largest risk as opposed to mitigating lots of risks without ancillary assumptions like increasing returns to risk reduction expenditure or the actual value of many risks under consideration being 0.
Yeah I agree—expected utility maximisation really starts to fall apart in this existential risk regime, even over trajectories rather than applied statically, and it only makes sense “locally” and at the margin.
Personally I’m very happy to bite the bullet and not be rigorously utilitarian, but I’m also a global health focussed “old school EA” thinking about how much to diversify donations across charities ;)