I also found that passage odd, though I think for a somewhat different reason (or at least with a different framing).
For me, the passage reminded me of the O-ring theory of economic development, āwhich proposes that tasks of production must be executed proficiently together in order for any of them to be of high valueā.
For the sake of the argument, letās make the very extreme and unlikely assumption that, if no longtermists worked on them, each of AI risk, biorisk, and nuclear war would by themselves be enough to guarantee an existential catastrophe that reduces the value of the future to approximately 0. In that case, we might say that the EV of the future is ~0, and even if we were to totally fix one of those problems, or even two of them, the EV would still be ~0. Therefore, the EV of working on any one or two of the problems, viewed in isolation, is 0. But the EV of fixing all three would presumably be astronomical.
We could maybe say that existential catastrophe in this scenario is overdetermined, and so we need to remove multiple risks in order for catastrophe to actually not happen. This might naively make it look like many individual prevention efforts were totally worthless, and it might indeed mean that they are worthless if the other efforts donāt happen, but itās still the case that, altogether, that collection of efforts is extremely valuable.
This also sort-of reminds me of some of 80k/āBen Toddās comments on attributing impact, e.g.:
A common misconception is that the credit for an action canāt add up to more than 100%. But itās perfectly possible for both people to be responsible. Suppose Amy and Bob see someone drowning. Amy performs CPR while Bob calls the ambulance. If Amy wasnāt there to perform CPR, the person would have died while Bob called the ambulance. If Bob wasnāt there, the ambulance wouldnāt have arrived in time. Both Amy and Bob saved a life, and it wouldnāt have happened if either was there. So, both are 100% responsible.
I havenāt taken the time to work through how well this point holds up when instead each x-risk causes less than 100%, e.g. 10%, chance of existential catastrophe if there were no longtermists working on it. But it seems plausible that there could be more than 100% worth of x-risks, if we add it up across the centuries/āmillenia, such that, naively, any specified effort to reduce x-risks that doesnāt by itself reduce the total risk to less than 100% appears worthless.
So I think the point that, in a sense, only so many things can reduce the EV of the future by 10% does highlight that work on AI risk might be less valuable that one would naively think, if we expect other people/āgenerations to drop the ball on other issues. But if we instead view this like a situation where thereās a community of people working on AI risk, another working on biorisk, another working on nuclear, future generations working on whatever risks come up then, etc., then it seems totally possible that each community has to fix their problem. And so then it makes sense for our generation to work on the issues we can work on now, and more specifically for people with a comparative advantage for AI to work on that, those with a comparative advantage for general great power war reduction to work on that, etc.
So Paulās statement there seems to sort-of hold up as a partial argument for reducing focus on AI risk, but in a specific way (to ensure we also patch all the other leaks too). It doesnāt seem like it holds up as an argument that AI safety work is less valuable than we thought in a simple sense, such that we should redirect efforts to non-longtermist work.
(Iām not saying Paul specifically intended to have the latter implication, but I think itād be easy to make that inference from what he said, at least as quoted here. Iām also not sure if Iām right or if Iāve expressed myself well, and Iād be interested in other peopleās thoughts.)
Btw, thereās a section on āComparing and combining risksā in Chapter 6 of The Precipice(pages 171-173 in my version), which is very relevant to this discussion. Appendix D expands on that further. Iād recommend interested people check that out.
I also found that passage odd, though I think for a somewhat different reason (or at least with a different framing).
For me, the passage reminded me of the O-ring theory of economic development, āwhich proposes that tasks of production must be executed proficiently together in order for any of them to be of high valueā.
For the sake of the argument, letās make the very extreme and unlikely assumption that, if no longtermists worked on them, each of AI risk, biorisk, and nuclear war would by themselves be enough to guarantee an existential catastrophe that reduces the value of the future to approximately 0. In that case, we might say that the EV of the future is ~0, and even if we were to totally fix one of those problems, or even two of them, the EV would still be ~0. Therefore, the EV of working on any one or two of the problems, viewed in isolation, is 0. But the EV of fixing all three would presumably be astronomical.
We could maybe say that existential catastrophe in this scenario is overdetermined, and so we need to remove multiple risks in order for catastrophe to actually not happen. This might naively make it look like many individual prevention efforts were totally worthless, and it might indeed mean that they are worthless if the other efforts donāt happen, but itās still the case that, altogether, that collection of efforts is extremely valuable.
This also sort-of reminds me of some of 80k/āBen Toddās comments on attributing impact, e.g.:
I havenāt taken the time to work through how well this point holds up when instead each x-risk causes less than 100%, e.g. 10%, chance of existential catastrophe if there were no longtermists working on it. But it seems plausible that there could be more than 100% worth of x-risks, if we add it up across the centuries/āmillenia, such that, naively, any specified effort to reduce x-risks that doesnāt by itself reduce the total risk to less than 100% appears worthless.
So I think the point that, in a sense, only so many things can reduce the EV of the future by 10% does highlight that work on AI risk might be less valuable that one would naively think, if we expect other people/āgenerations to drop the ball on other issues. But if we instead view this like a situation where thereās a community of people working on AI risk, another working on biorisk, another working on nuclear, future generations working on whatever risks come up then, etc., then it seems totally possible that each community has to fix their problem. And so then it makes sense for our generation to work on the issues we can work on now, and more specifically for people with a comparative advantage for AI to work on that, those with a comparative advantage for general great power war reduction to work on that, etc.
So Paulās statement there seems to sort-of hold up as a partial argument for reducing focus on AI risk, but in a specific way (to ensure we also patch all the other leaks too). It doesnāt seem like it holds up as an argument that AI safety work is less valuable than we thought in a simple sense, such that we should redirect efforts to non-longtermist work.
(Iām not saying Paul specifically intended to have the latter implication, but I think itād be easy to make that inference from what he said, at least as quoted here. Iām also not sure if Iām right or if Iāve expressed myself well, and Iād be interested in other peopleās thoughts.)
Thanks v muchāyour third para was a much better explanation of what I was driving at!
Btw, thereās a section on āComparing and combining risksā in Chapter 6 of The Precipice (pages 171-173 in my version), which is very relevant to this discussion. Appendix D expands on that further. Iād recommend interested people check that out.