I was more referring to the diversification as implied by “don’t focus the vast majority of efforts on one cause”, which to me meant more “if you’re a decision maker over some amount of resources, you should diversify the allocation across cause areas”. Which I agree with, but it’s quite hard to really justify.
Yes, via nonlinearity you can get to diversification, but this means making additional assumptions beyond sampling error/publication bias/optimiser curse type effects. The nonlinearity you’re describing matters on a movement level, but not on a individual decision makers level. ”Impact risk aversion” is another mechanism to get diversification which I think can be reasonable in cases where eg low impact reduces the probability of future donations or similar.
One channel I think is under explored and might work well as a justification for diversification in practice is something like this (I haven’t thought about this rigorously though): if I predictably optimise and my objective function is known to others, they will (in the worst case, possibly thru misaligned incentives) feed me biased information to influence my decision, and optimisation is very sensitive to noise, therefore I subject myself to adverse selection. So basically by not optimising but diversifying across good options you reduce the negative impact of this type of adverse selection. In this case, the “errors” are not iid. Hard to say how much diversification that yields.
I think the conclusion that diversification is a good strategy follows trivially from the optimizers’ curse: if you focus all your efforts on the apparent biggest threat, you’ve probably just focused on the cause with the largest risk assessment error and entirely neglected the actual biggest threat. A more diverse allocation is more likely to address the actual biggest threat. If there are diminishing returns to resources allocated to mitigate particular risk areas that makes diversification look better (complex nonlinear returns complicate it). As does the possibility that larger errors in risk assessment for a particular type of risk are inversely correlated with ability to invest in the best mitigation strategy for that type of risk.[1]
But your point about adverse selection is a good one too. Metrics are gameable, and there are stronger incentives to do so when funding is “winner takes all” rather than “we disburse funds to a wide selection of causes and value rigour and disclosure of uncertainties”
I think there are probably exceptions to this, but I think it’s generally true. Good understanding of celestial mechanics and early warning systems, for example, are absolutely essential to potentially preventing hypothetical large space rocks colliding with earth, but also mean that we are less likely to overestimate the imminence of destruction by a rogue asteroid than we are for more unpredictable phenomenon.
While this sounds intuitively right, I think in the simplest utility maximising setting (iid additive errors with mean zero) your first claim does not seem true? The best looking noisy option is still most likely to be the best?
(I need to think more about the maths, but at least you need some kind of shrinkage to a prior that can change the ranking, which you’re unlikely to get, and if you’re maximising utility the solution is always fully concentrated?)
I’m not sure naive total utility maximization [in a static framework] is the best framework to be thinking about dealing with existential risk over time.[1]
Assuming the number of risks and error bars are not trivially small, the universal outcome of concentrating all your risk mitigations on one is that most risks continue to be a high as they could possibly be. The modal outcome is that the risks ignored includes at least one risk greater than the one all efforts are concentrated on mitigating. Some reasonable assumptions in the article above show this can hold even where the actual biggest risk is orders of magnitude greater than the one targeted. In the diversified approach, less money are devoted to reducing the perceived biggest risk, but the rest is apportioned to reducing other risks. This seems more robust to conventional assumptions like uncertainty and some risks being easier to mitigate than others.
And tbh I’m not even seeing an average utility boost from concentrating on the single largest risk as opposed to mitigating lots of risks without ancillary assumptions like increasing returns to risk reduction expenditure or the actual value of many risks under consideration being 0.
Yeah I agree—expected utility maximisation really starts to fall apart in this existential risk regime, even over trajectories rather than applied statically, and it only makes sense “locally” and at the margin.
Personally I’m very happy to bite the bullet and not be rigorously utilitarian, but I’m also a global health focussed “old school EA” thinking about how much to diversify donations across charities ;)
Interesting. Who might these people be who deliberately feed you biased information? How do they benefit from you focusing on cause area y instead of cause area z?
I was more referring to the diversification as implied by “don’t focus the vast majority of efforts on one cause”, which to me meant more “if you’re a decision maker over some amount of resources, you should diversify the allocation across cause areas”. Which I agree with, but it’s quite hard to really justify.
Yes, via nonlinearity you can get to diversification, but this means making additional assumptions beyond sampling error/publication bias/optimiser curse type effects.
The nonlinearity you’re describing matters on a movement level, but not on a individual decision makers level.
”Impact risk aversion” is another mechanism to get diversification which I think can be reasonable in cases where eg low impact reduces the probability of future donations or similar.
One channel I think is under explored and might work well as a justification for diversification in practice is something like this (I haven’t thought about this rigorously though): if I predictably optimise and my objective function is known to others, they will (in the worst case, possibly thru misaligned incentives) feed me biased information to influence my decision, and optimisation is very sensitive to noise, therefore I subject myself to adverse selection. So basically by not optimising but diversifying across good options you reduce the negative impact of this type of adverse selection. In this case, the “errors” are not iid. Hard to say how much diversification that yields.
I think the conclusion that diversification is a good strategy follows trivially from the optimizers’ curse: if you focus all your efforts on the apparent biggest threat, you’ve probably just focused on the cause with the largest risk assessment error and entirely neglected the actual biggest threat. A more diverse allocation is more likely to address the actual biggest threat. If there are diminishing returns to resources allocated to mitigate particular risk areas that makes diversification look better (complex nonlinear returns complicate it). As does the possibility that larger errors in risk assessment for a particular type of risk are inversely correlated with ability to invest in the best mitigation strategy for that type of risk.[1]
But your point about adverse selection is a good one too. Metrics are gameable, and there are stronger incentives to do so when funding is “winner takes all” rather than “we disburse funds to a wide selection of causes and value rigour and disclosure of uncertainties”
I think there are probably exceptions to this, but I think it’s generally true. Good understanding of celestial mechanics and early warning systems, for example, are absolutely essential to potentially preventing hypothetical large space rocks colliding with earth, but also mean that we are less likely to overestimate the imminence of destruction by a rogue asteroid than we are for more unpredictable phenomenon.
While this sounds intuitively right, I think in the simplest utility maximising setting (iid additive errors with mean zero) your first claim does not seem true? The best looking noisy option is still most likely to be the best?
(I need to think more about the maths, but at least you need some kind of shrinkage to a prior that can change the ranking, which you’re unlikely to get, and if you’re maximising utility the solution is always fully concentrated?)
I’m not sure naive total utility maximization [in a static framework] is the best framework to be thinking about dealing with existential risk over time.[1]
Assuming the number of risks and error bars are not trivially small, the universal outcome of concentrating all your risk mitigations on one is that most risks continue to be a high as they could possibly be. The modal outcome is that the risks ignored includes at least one risk greater than the one all efforts are concentrated on mitigating. Some reasonable assumptions in the article above show this can hold even where the actual biggest risk is orders of magnitude greater than the one targeted. In the diversified approach, less money are devoted to reducing the perceived biggest risk, but the rest is apportioned to reducing other risks. This seems more robust to conventional assumptions like uncertainty and some risks being easier to mitigate than others.
And tbh I’m not even seeing an average utility boost from concentrating on the single largest risk as opposed to mitigating lots of risks without ancillary assumptions like increasing returns to risk reduction expenditure or the actual value of many risks under consideration being 0.
Yeah I agree—expected utility maximisation really starts to fall apart in this existential risk regime, even over trajectories rather than applied statically, and it only makes sense “locally” and at the margin.
Personally I’m very happy to bite the bullet and not be rigorously utilitarian, but I’m also a global health focussed “old school EA” thinking about how much to diversify donations across charities ;)
Interesting. Who might these people be who deliberately feed you biased information? How do they benefit from you focusing on cause area y instead of cause area z?
To be clear, it need not be deliberate and they need not benefit personally!