At risk of sounding foolish, this seems odd “There can’t be too many things that reduce the expected value of the future by 10%; if there were, there would be no expected value left. So, the prior that any particular thing has such an impact should be quite low.”
But, if we lived in a world where there are 10 death-stars lurking around in the heavens. And, all of these are very likely to obliterate the earth and reduce the expected value of the Earth significantly. Then can’t the EV detraction of each individual death star be (say) 90% & EV of the future remains above zero. One way of looking at it would be that the Death Stars are deployed in a staggered fashion. The impact of the deployment of the first death star being that earth’s EV falls from 100 to 10, the deployment of the second would imply if falls from 10 to 1, and the third from 1 to 0.1 etc.
Apologies for the terrible explanation of my point.
I also found that passage odd, though I think for a somewhat different reason (or at least with a different framing).
For me, the passage reminded me of the O-ring theory of economic development, “which proposes that tasks of production must be executed proficiently together in order for any of them to be of high value”.
For the sake of the argument, let’s make the very extreme and unlikely assumption that, if no longtermists worked on them, each of AI risk, biorisk, and nuclear war would by themselves be enough to guarantee an existential catastrophe that reduces the value of the future to approximately 0. In that case, we might say that the EV of the future is ~0, and even if we were to totally fix one of those problems, or even two of them, the EV would still be ~0. Therefore, the EV of working on any one or two of the problems, viewed in isolation, is 0. But the EV of fixing all three would presumably be astronomical.
We could maybe say that existential catastrophe in this scenario is overdetermined, and so we need to remove multiple risks in order for catastrophe to actually not happen. This might naively make it look like many individual prevention efforts were totally worthless, and it might indeed mean that they are worthless if the other efforts don’t happen, but it’s still the case that, altogether, that collection of efforts is extremely valuable.
This also sort-of reminds me of some of 80k/Ben Todd’s comments on attributing impact, e.g.:
A common misconception is that the credit for an action can’t add up to more than 100%. But it’s perfectly possible for both people to be responsible. Suppose Amy and Bob see someone drowning. Amy performs CPR while Bob calls the ambulance. If Amy wasn’t there to perform CPR, the person would have died while Bob called the ambulance. If Bob wasn’t there, the ambulance wouldn’t have arrived in time. Both Amy and Bob saved a life, and it wouldn’t have happened if either was there. So, both are 100% responsible.
I haven’t taken the time to work through how well this point holds up when instead each x-risk causes less than 100%, e.g. 10%, chance of existential catastrophe if there were no longtermists working on it. But it seems plausible that there could be more than 100% worth of x-risks, if we add it up across the centuries/millenia, such that, naively, any specified effort to reduce x-risks that doesn’t by itself reduce the total risk to less than 100% appears worthless.
So I think the point that, in a sense, only so many things can reduce the EV of the future by 10% does highlight that work on AI risk might be less valuable that one would naively think, if we expect other people/generations to drop the ball on other issues. But if we instead view this like a situation where there’s a community of people working on AI risk, another working on biorisk, another working on nuclear, future generations working on whatever risks come up then, etc., then it seems totally possible that each community has to fix their problem. And so then it makes sense for our generation to work on the issues we can work on now, and more specifically for people with a comparative advantage for AI to work on that, those with a comparative advantage for general great power war reduction to work on that, etc.
So Paul’s statement there seems to sort-of hold up as a partial argument for reducing focus on AI risk, but in a specific way (to ensure we also patch all the other leaks too). It doesn’t seem like it holds up as an argument that AI safety work is less valuable than we thought in a simple sense, such that we should redirect efforts to non-longtermist work.
(I’m not saying Paul specifically intended to have the latter implication, but I think it’d be easy to make that inference from what he said, at least as quoted here. I’m also not sure if I’m right or if I’ve expressed myself well, and I’d be interested in other people’s thoughts.)
Btw, there’s a section on “Comparing and combining risks” in Chapter 6 of The Precipice(pages 171-173 in my version), which is very relevant to this discussion. Appendix D expands on that further. I’d recommend interested people check that out.
At risk of sounding foolish, this seems odd “There can’t be too many things that reduce the expected value of the future by 10%; if there were, there would be no expected value left. So, the prior that any particular thing has such an impact should be quite low.”
But, if we lived in a world where there are 10 death-stars lurking around in the heavens. And, all of these are very likely to obliterate the earth and reduce the expected value of the Earth significantly. Then can’t the EV detraction of each individual death star be (say) 90% & EV of the future remains above zero. One way of looking at it would be that the Death Stars are deployed in a staggered fashion. The impact of the deployment of the first death star being that earth’s EV falls from 100 to 10, the deployment of the second would imply if falls from 10 to 1, and the third from 1 to 0.1 etc.
Apologies for the terrible explanation of my point.
I also found that passage odd, though I think for a somewhat different reason (or at least with a different framing).
For me, the passage reminded me of the O-ring theory of economic development, “which proposes that tasks of production must be executed proficiently together in order for any of them to be of high value”.
For the sake of the argument, let’s make the very extreme and unlikely assumption that, if no longtermists worked on them, each of AI risk, biorisk, and nuclear war would by themselves be enough to guarantee an existential catastrophe that reduces the value of the future to approximately 0. In that case, we might say that the EV of the future is ~0, and even if we were to totally fix one of those problems, or even two of them, the EV would still be ~0. Therefore, the EV of working on any one or two of the problems, viewed in isolation, is 0. But the EV of fixing all three would presumably be astronomical.
We could maybe say that existential catastrophe in this scenario is overdetermined, and so we need to remove multiple risks in order for catastrophe to actually not happen. This might naively make it look like many individual prevention efforts were totally worthless, and it might indeed mean that they are worthless if the other efforts don’t happen, but it’s still the case that, altogether, that collection of efforts is extremely valuable.
This also sort-of reminds me of some of 80k/Ben Todd’s comments on attributing impact, e.g.:
I haven’t taken the time to work through how well this point holds up when instead each x-risk causes less than 100%, e.g. 10%, chance of existential catastrophe if there were no longtermists working on it. But it seems plausible that there could be more than 100% worth of x-risks, if we add it up across the centuries/millenia, such that, naively, any specified effort to reduce x-risks that doesn’t by itself reduce the total risk to less than 100% appears worthless.
So I think the point that, in a sense, only so many things can reduce the EV of the future by 10% does highlight that work on AI risk might be less valuable that one would naively think, if we expect other people/generations to drop the ball on other issues. But if we instead view this like a situation where there’s a community of people working on AI risk, another working on biorisk, another working on nuclear, future generations working on whatever risks come up then, etc., then it seems totally possible that each community has to fix their problem. And so then it makes sense for our generation to work on the issues we can work on now, and more specifically for people with a comparative advantage for AI to work on that, those with a comparative advantage for general great power war reduction to work on that, etc.
So Paul’s statement there seems to sort-of hold up as a partial argument for reducing focus on AI risk, but in a specific way (to ensure we also patch all the other leaks too). It doesn’t seem like it holds up as an argument that AI safety work is less valuable than we thought in a simple sense, such that we should redirect efforts to non-longtermist work.
(I’m not saying Paul specifically intended to have the latter implication, but I think it’d be easy to make that inference from what he said, at least as quoted here. I’m also not sure if I’m right or if I’ve expressed myself well, and I’d be interested in other people’s thoughts.)
Thanks v much—your third para was a much better explanation of what I was driving at!
Btw, there’s a section on “Comparing and combining risks” in Chapter 6 of The Precipice (pages 171-173 in my version), which is very relevant to this discussion. Appendix D expands on that further. I’d recommend interested people check that out.
See this comment thread.