though a corollary of it is “don’t assume that just because you’ve picked direct work that your career choice is maximally good and stuff like donations and helping others is just a distraction”. This is arguably true for speculative career choices even if the optimal cause is the correct one (i.e. even if AI x-risk really does dominate everything, lots of the promising approaches to resolving it that people might choose will have no impact)
I like diversification as a reaction to this type of uncertainty, but it does not trivially follow? I might be missing something—do you have a favourite minimal set of assumptions that rigorously yield diversification as a function of this?
the x-risk coming from each cause decreases with each additional person working on this cause
Whether diversification is better (in expectation) depends on how a cause’s x-risk decreases as additional people work on this cause. If x-risk decreases linearly (the 1000th person makes the same marginal contribution as the 1st), then diversification is not better in expectation. But if the contribution to x-risk prevention is marginally decreasing in people, diversification is better.
(By diversification I mean each person choosing their top estimated x-risk cause individually. But it can also mean that some people deliberately do not work on the cause with the highest aggregated risk estimate.)
I was more referring to the diversification as implied by “don’t focus the vast majority of efforts on one cause”, which to me meant more “if you’re a decision maker over some amount of resources, you should diversify the allocation across cause areas”. Which I agree with, but it’s quite hard to really justify.
Yes, via nonlinearity you can get to diversification, but this means making additional assumptions beyond sampling error/publication bias/optimiser curse type effects. The nonlinearity you’re describing matters on a movement level, but not on a individual decision makers level. ”Impact risk aversion” is another mechanism to get diversification which I think can be reasonable in cases where eg low impact reduces the probability of future donations or similar.
One channel I think is under explored and might work well as a justification for diversification in practice is something like this (I haven’t thought about this rigorously though): if I predictably optimise and my objective function is known to others, they will (in the worst case, possibly thru misaligned incentives) feed me biased information to influence my decision, and optimisation is very sensitive to noise, therefore I subject myself to adverse selection. So basically by not optimising but diversifying across good options you reduce the negative impact of this type of adverse selection. In this case, the “errors” are not iid. Hard to say how much diversification that yields.
I think the conclusion that diversification is a good strategy follows trivially from the optimizers’ curse: if you focus all your efforts on the apparent biggest threat, you’ve probably just focused on the cause with the largest risk assessment error and entirely neglected the actual biggest threat. A more diverse allocation is more likely to address the actual biggest threat. If there are diminishing returns to resources allocated to mitigate particular risk areas that makes diversification look better (complex nonlinear returns complicate it). As does the possibility that larger errors in risk assessment for a particular type of risk are inversely correlated with ability to invest in the best mitigation strategy for that type of risk.[1]
But your point about adverse selection is a good one too. Metrics are gameable, and there are stronger incentives to do so when funding is “winner takes all” rather than “we disburse funds to a wide selection of causes and value rigour and disclosure of uncertainties”
I think there are probably exceptions to this, but I think it’s generally true. Good understanding of celestial mechanics and early warning systems, for example, are absolutely essential to potentially preventing hypothetical large space rocks colliding with earth, but also mean that we are less likely to overestimate the imminence of destruction by a rogue asteroid than we are for more unpredictable phenomenon.
While this sounds intuitively right, I think in the simplest utility maximising setting (iid additive errors with mean zero) your first claim does not seem true? The best looking noisy option is still most likely to be the best?
(I need to think more about the maths, but at least you need some kind of shrinkage to a prior that can change the ranking, which you’re unlikely to get, and if you’re maximising utility the solution is always fully concentrated?)
I’m not sure naive total utility maximization [in a static framework] is the best framework to be thinking about dealing with existential risk over time.[1]
Assuming the number of risks and error bars are not trivially small, the universal outcome of concentrating all your risk mitigations on one is that most risks continue to be a high as they could possibly be. The modal outcome is that the risks ignored includes at least one risk greater than the one all efforts are concentrated on mitigating. Some reasonable assumptions in the article above show this can hold even where the actual biggest risk is orders of magnitude greater than the one targeted. In the diversified approach, less money are devoted to reducing the perceived biggest risk, but the rest is apportioned to reducing other risks. This seems more robust to conventional assumptions like uncertainty and some risks being easier to mitigate than others.
And tbh I’m not even seeing an average utility boost from concentrating on the single largest risk as opposed to mitigating lots of risks without ancillary assumptions like increasing returns to risk reduction expenditure or the actual value of many risks under consideration being 0.
Yeah I agree—expected utility maximisation really starts to fall apart in this existential risk regime, even over trajectories rather than applied statically, and it only makes sense “locally” and at the margin.
Personally I’m very happy to bite the bullet and not be rigorously utilitarian, but I’m also a global health focussed “old school EA” thinking about how much to diversify donations across charities ;)
Interesting. Who might these people be who deliberately feed you biased information? How do they benefit from you focusing on cause area y instead of cause area z?
Above all it implies don’t focus the vast majority of efforts on one cause.
That might not be practical for career choices,[1] but it’s certainly possible for a funder or movement
though a corollary of it is “don’t assume that just because you’ve picked direct work that your career choice is maximally good and stuff like donations and helping others is just a distraction”. This is arguably true for speculative career choices even if the optimal cause is the correct one (i.e. even if AI x-risk really does dominate everything, lots of the promising approaches to resolving it that people might choose will have no impact)
I like diversification as a reaction to this type of uncertainty, but it does not trivially follow? I might be missing something—do you have a favourite minimal set of assumptions that rigorously yield diversification as a function of this?
One simple model is:
each person can choose only one cause area
errors are iid across people
the x-risk coming from each cause decreases with each additional person working on this cause
Whether diversification is better (in expectation) depends on how a cause’s x-risk decreases as additional people work on this cause. If x-risk decreases linearly (the 1000th person makes the same marginal contribution as the 1st), then diversification is not better in expectation. But if the contribution to x-risk prevention is marginally decreasing in people, diversification is better.
(By diversification I mean each person choosing their top estimated x-risk cause individually. But it can also mean that some people deliberately do not work on the cause with the highest aggregated risk estimate.)
I was more referring to the diversification as implied by “don’t focus the vast majority of efforts on one cause”, which to me meant more “if you’re a decision maker over some amount of resources, you should diversify the allocation across cause areas”. Which I agree with, but it’s quite hard to really justify.
Yes, via nonlinearity you can get to diversification, but this means making additional assumptions beyond sampling error/publication bias/optimiser curse type effects.
The nonlinearity you’re describing matters on a movement level, but not on a individual decision makers level.
”Impact risk aversion” is another mechanism to get diversification which I think can be reasonable in cases where eg low impact reduces the probability of future donations or similar.
One channel I think is under explored and might work well as a justification for diversification in practice is something like this (I haven’t thought about this rigorously though): if I predictably optimise and my objective function is known to others, they will (in the worst case, possibly thru misaligned incentives) feed me biased information to influence my decision, and optimisation is very sensitive to noise, therefore I subject myself to adverse selection. So basically by not optimising but diversifying across good options you reduce the negative impact of this type of adverse selection. In this case, the “errors” are not iid. Hard to say how much diversification that yields.
I think the conclusion that diversification is a good strategy follows trivially from the optimizers’ curse: if you focus all your efforts on the apparent biggest threat, you’ve probably just focused on the cause with the largest risk assessment error and entirely neglected the actual biggest threat. A more diverse allocation is more likely to address the actual biggest threat. If there are diminishing returns to resources allocated to mitigate particular risk areas that makes diversification look better (complex nonlinear returns complicate it). As does the possibility that larger errors in risk assessment for a particular type of risk are inversely correlated with ability to invest in the best mitigation strategy for that type of risk.[1]
But your point about adverse selection is a good one too. Metrics are gameable, and there are stronger incentives to do so when funding is “winner takes all” rather than “we disburse funds to a wide selection of causes and value rigour and disclosure of uncertainties”
I think there are probably exceptions to this, but I think it’s generally true. Good understanding of celestial mechanics and early warning systems, for example, are absolutely essential to potentially preventing hypothetical large space rocks colliding with earth, but also mean that we are less likely to overestimate the imminence of destruction by a rogue asteroid than we are for more unpredictable phenomenon.
While this sounds intuitively right, I think in the simplest utility maximising setting (iid additive errors with mean zero) your first claim does not seem true? The best looking noisy option is still most likely to be the best?
(I need to think more about the maths, but at least you need some kind of shrinkage to a prior that can change the ranking, which you’re unlikely to get, and if you’re maximising utility the solution is always fully concentrated?)
I’m not sure naive total utility maximization [in a static framework] is the best framework to be thinking about dealing with existential risk over time.[1]
Assuming the number of risks and error bars are not trivially small, the universal outcome of concentrating all your risk mitigations on one is that most risks continue to be a high as they could possibly be. The modal outcome is that the risks ignored includes at least one risk greater than the one all efforts are concentrated on mitigating. Some reasonable assumptions in the article above show this can hold even where the actual biggest risk is orders of magnitude greater than the one targeted. In the diversified approach, less money are devoted to reducing the perceived biggest risk, but the rest is apportioned to reducing other risks. This seems more robust to conventional assumptions like uncertainty and some risks being easier to mitigate than others.
And tbh I’m not even seeing an average utility boost from concentrating on the single largest risk as opposed to mitigating lots of risks without ancillary assumptions like increasing returns to risk reduction expenditure or the actual value of many risks under consideration being 0.
Yeah I agree—expected utility maximisation really starts to fall apart in this existential risk regime, even over trajectories rather than applied statically, and it only makes sense “locally” and at the margin.
Personally I’m very happy to bite the bullet and not be rigorously utilitarian, but I’m also a global health focussed “old school EA” thinking about how much to diversify donations across charities ;)
Interesting. Who might these people be who deliberately feed you biased information? How do they benefit from you focusing on cause area y instead of cause area z?
To be clear, it need not be deliberate and they need not benefit personally!