I used to think along the lines you described, but I believe I was wrong.
The expected value contained >1 year in the future is:
p * 0 + (1-p) * U
One can simply consider these 2 outcomes, but I find it more informative to consider many potential futures, and how an intervention changes the probability of each of them. I posted a comment about how I think this would work out for a binary future. To illustrate, assume there are 101 possible futures with value U_i = 10^i â 1 (for i between 0 and 100), and that future 0 corresponds to human extinction in 2025 (U_0 = 0). The increase in expected values equals to Delta = sum_i dp_i*U_i, where dp_i is the variation in the probability of future i (sum_i dp_i = 0).
Decreasing the probability of human extinction over 2025 by 100 % does not imply Delta is astronomically large. For example, all the probability mass of future 0 could be moved to future 1 (dp_1 = -dp_0, and dp_i = 0 for i between 2 to 100). In this case, Delta = 9*|dp_0| would be at most 9 (dp_i â 1), i.e. far from astronomically large.
So if U is astronomically big, and dp is not astronomically small, then the expected value of reducing nearterm extinction risk must be astronomically big as well.
For Delta to be astronomically large, non-negligible probability mass has to be moved from future 0 to astronomically valuable futures. dp = sum_(i >= 1) dp_i not being astronomically small is not enough for that. I think the easiest way to avoid human extinction in 2025 is postponing it to 2026, which would not make the future astronomically valuable.
The impact of nearterm extinction risk on future expected value doesnât need to be âdirectly guessedâ, as I think you are suggesting? The relationship can be worked out precisely. It is given by the above formula (as long as you have an estimate of U to begin with).
I agree the impact of decreasing the nearterm risk of human extinction does not have to be directly guessed. I just meant it has traditionally been guessed. In any case, I think one had better use standard cost-effectiveness analyses.
Thanks for the explanation, I have a clearer understanding of what you are arguing for now! Sorry I didnât appreciate this properly when reading the post.
So youâre claiming that if we intervene to reduce the probability of extinction in 2025, then that increases the probability of extinction in 2026, 2027, etc, even after conditioning on not going extinct earlier? The increase is such that the chance of reaching the far future is unchanged?
My next question is: why should we expect something like that to be true???
It seems very unlikely to me that reducing near term extinction risk in 2025 then increases P(extinction in 2026 | not going extinct in 2025). If anything, my prior expectation is that the opposite would be true. If we get better at mitigating existential risks in 2025, why would we expect that to make us worse at mitigating them in 2026?
If I understand right, youâre basing this on a claim that we should expect the impact of any intervention to decay exponentially as we go further and further into the future, and youâre then looking at what has to happen in order to make this true. I can sympathise with the intuition here. But I donât agree with how itâs being applied.
I think the correct way of applying this intuition is to say that itâs these quantities which will only be changed negligibly in the far future by interventions we take today:
P(going extinct in far future year X | we reach far future year X) (1)
E(utility in far future year X | we reach year X) (2)
In a world where the future has astronomical value, we obviously can astronomically change the expected value of the future by adjusting near-term extinction risk. To take an extreme example: if we make near-term extinction risk 100%, then expected future value becomes zero, however far into the future X is.
I think asserting that (1) and (2) are unchanged is the correct way of capturing the idea that the effect of interventions tends to wash out over time. That then leads to the conclusion from my original comment.
I think your life-expectancy example is helpful. But I think the conclusion is the opposite of what youâre claiming. If I play Russian Roulette and take an instantaneous risk of death, p, and my current life expectancy is L, then my life expectancy will decrease by pL. This is certainly non-negligible for non-negligible p, even though the time I take the risk over is minuscule in comparison to the duration of my life.
Of course I have changed your example here. You were talking about reducing the risk of death in a minuscule time period, rather than increasing it. Itâs true that that doesnât meaningfully change your life expectancy, but thatâs not because the duration of time is small in relation to your life, itâs because the risk of death in such a minuscule time period is already minuscule!
If we translate this back to existential risk, it does become a good argument against the astronomical cost-effectiveness claim, but itâs now a different argument. Itâs not that near-term extinction isnât important for someone who thinks the future has astronomical value. Itâs that: if you believe the future has astronomical value, then you are committed to believing that the extinction risk in most centuries is astronomically low, in which case interventions to reduce it stop looking so attractive. The only way to rescue the âastronomical cost-effectiveness claimâ is to argue for something like the âtime of perilsâ hypothesis. Essentially that we are doing the equivalent of playing Russian Roulette right now, but that we will stop doing so soon, if we survive.
Thanks for the explanation, I have a clearer understanding of what you are arguing for now! Sorry I didnât appreciate this properly when reading the post.
No worries; you are welcome!
So youâre claiming that if we intervene to reduce the probability of extinction in 2025, then that increases the probability of extinction in 2026, 2027, etc, even after conditioning on not going extinct earlier? The increase is such that the chance of reaching the far future is unchanged?
Yes, I think so.
It seems very unlikely to me that reducing near term extinction risk in 2025 then increases P(extinction in 2026 | not going extinct in 2025). If anything, my prior expectation is that the opposite would be true. If we get better at mitigating existential risks in 2025, why would we expect that to make us worse at mitigating them in 2026?
It is not that I expect us to get worse at mitigation. I just expect it is way easier to move probability mass from the words with human extinction in 2025 to the ones with human extinction in 2026 than to ones with astronomically large value. The cost of moving physical mass increases with distance, and I guess the cost of moving probability mass increases (maybe exponentially) with value-distance (difference between the value of the worlds).
If I understand right, youâre basing this on a claim that we should expect the impact of any intervention to decay exponentially as we go further and further into the future, and youâre then looking at what has to happen in order to make this true.
Yes.
In a world where the future has astronomical value, we obviously can astronomically change the expected value of the future by adjusting near-term extinction risk.
Correct me if I am wrong, but I think you are suggesting something like the following. If there is a 99 % chance we are in future 100 (U_100 = 10^100), and a 1 % (= 1 â 0.99) chance we are in future 0 (U_0 = 0), i.e. if it is very likely we are in an astronomically valuable world[1], we can astronomically increase the expected value of the future by decreasing the chance of future 0. I do not agree. Even if the chance of future 0 is decreased by 100 %, I would say all its probability mass (1 pp) would be moved to nearby worlds whose value is not astronomical. For example, the expected value of the future would only increase by 0.09 (= 0.01*9) if all the probability mass was moved to future 1 (U_1 = 9).
The only way to rescue the âastronomical cost-effectiveness claimâ is to argue for something like the âtime of perilsâ hypothesis. Essentially that we are doing the equivalent of playing Russian Roulette right now, but that we will stop doing so soon, if we survive.
The time of perils hypothesis implies the probability mass is mostly distributed across worlds with tiny and astronomical value. However, the conclusion I reached above does not depend on the initial probabilities of futures 0 and 100. It works just as well for a probability of future 0 of 50 %, and a probability of future 100 of 50 %. My conclusion only depends on my assumption that decreasing the probability of future 0 overwhelmingly increases the chance of nearby non-astronomically valuable worlds, having a negligible effect on the probability of astronomically valuable worlds.
hroughCorrect me if I am wrong, but I think you are suggesting something like the following. If there is a 99 % chance we are in future 100 (U_100 = 10^100), and a 1 % (= 1 â 0.99) chance we are in future 0 (U_0 = 0), i.e. if it is very likely we are in an astronomically valuable world[1], we can astronomically increase the expected value of the future by decreasing the chance of future 0. I do not agree. Even if the chance of future 0 is decreased by 100 %, I would say all its probability mass (1 pp) would be moved to nearby worlds whose value is not astronomical. For example, the expected value of the future would only increase by 0.09 (= 0.01*9) if all the probability mass was moved to future 1 (U_1 = 9).
The claim you quoted here was a lot simpler than this.
I was just pointing out that if we take an action to increase near-term extinction risk to 100% (i.e. we deliberately go extinct), then we reduce the expected value of the future to zero. Thatâs an undeniable way that a change to near-term extinction risk can have an astronomical effect on the expected value of the future, provided only that the future has astronomical expected value before we make the intervention.
It is not that I expect us to get worse at mitigation.
But this is more or less a consequence of your claims isnât it?
The cost of moving physical mass increases with distance, and I guess the cost of moving probability mass increases (maybe exponentially) with value-distance (difference between the value of the worlds).
I donât see any basis for this assumption. For example, it is contradicted by my example above, where we deliberately go extinct, and therefore move all of the probability weight from U_100 to U_0, despite their huge value difference.
Or I suppose maybe I do agree with your assumption (as canât think of any counter-examples I would actually endorse in practice) I just disagree with how youâre explaining its consequences. I would say it means the future does not have astronomical expected value, not that it does have astronomical value but that we canât influence it (since it seems clear we can if it does).
(If I remember our exchange on the Toby Ord post correctly, I think you made some claim along the lines of: there are no conceivable interventions which would allow us to increase extinction risk to ~100%. This seems like an unlikely claim to me, but itâs also I think a different argument to the one youâre making in this post anyway.)
Hereâs another way of explaining it. In this case the probability p_100 of U_100 is given by the huge product:
P(making it through next year) X P(making it through the year after given we make it through year 1) X âŚ..⌠etc
Changing near-term extinction risk is influencing the first factor in this product, so it would be weird if it didnât change p_100 as well. The same logic doesnât apply to the global health interventions that youâre citing as an analogy, and makes existential risk special.
In fact I would say it is your claim (that the later factors get modified too in just such a special way as to cancel out the drop in the first factor) which involves near-term interventions having implausible effects on the future that we shouldnât a priori expect them to have.
I was just pointing out that if we take an action to increase near-term extinction risk to 100% (i.e. we deliberately go extinct), then we reduce the expected value of the future to zero. Thatâs an undeniable way that a change to near-term extinction risk can have an astronomical effect on the expected value of the future, provided only that the future has astronomical expected value before we make the intervention.
Agreed. However, I would argue that increasing the nearterm risk of human extinction to 100 % would be astronomically difficult/âcostly. In the framework of my previous comment, that would eventually require moving probability mass from world 100 to 0, which I believe is as super hard as moving mass world 0 to 100.
Hereâs another way of explaining it. In this case the probability p_100 of U_100 is given by the huge product:
P(making it through next year) X P(making it through the year after given we make it through year 1) X âŚ..⌠etc
Changing near-term extinction risk is influencing the first factor in this product, so it would be weird if it didnât change p_100 as well. The same logic doesnât apply to the global health interventions that youâre citing as an analogy, and makes existential risk special.
One can make a similar argument for the effect size of global health and development interventions. Assuming the effect size is strictly decreasing, denoting by Xi the effect size at year i, P(XN>x)=P(X1>x)P(X2>x|X1>x)...P(XN>x|XNâ1>x). Ok, P(XN>x) increases with P(X1>x) on priors. However, it could still be the case that the effect size will decay to practically 0 within a few decades or centuries.
It is not a strict requirement, but it is an arguably reasonable assumption. Are there any interventions whose estimates of (posterior) counterfactual impact, in terms of expected total hedonistic utility (not e.g. preventing the extinction of a random species), do not decay to 0 in at most a few centuries? From my perspective, their absence establishes a strong prior against persistent/âincreasing effects.
Sure, but once youâve assumed that already, you donât need to rely any more on an argument about shifts to P(X_1 > x) being cancelled out by shifts to P(X_n > x) for larger n (which if I understand correctly is the argument youâre making about existential risk).
If P(X_N > x) is very small to begin with for some large N, then it will stay small, even if we adjust P(X_1 > x) by a lot (we canât make it bigger than 1!) So we can safely say under your assumption that adjusting the P(X_1 > x) factor by a large amount does influence P(X_N > x) as well, itâs just that it canât make it not small.
The existential risk set-up is fundamentally different. We are assuming the future has astronomical value to begin with, before we intervene. That now means non-tiny changes to P(Making it through the next year) must have astronomical value too (unless there is some weird conspiracy among the probability of making it through later years which precisely cancels this out, but that seems very weird, and not something you can justify by pointing to global health as an analogy).
Thanks for the discussion, Toby. I do not plan to follow up further, but, for reference/âtransparency, I maintain my guess that the future is astronomically valuable, but that no interventions are astronomically cost-effective.
Thanks for looking into the post, Toby!
I used to think along the lines you described, but I believe I was wrong.
One can simply consider these 2 outcomes, but I find it more informative to consider many potential futures, and how an intervention changes the probability of each of them. I posted a comment about how I think this would work out for a binary future. To illustrate, assume there are 101 possible futures with value U_i = 10^i â 1 (for i between 0 and 100), and that future 0 corresponds to human extinction in 2025 (U_0 = 0). The increase in expected values equals to Delta = sum_i dp_i*U_i, where dp_i is the variation in the probability of future i (sum_i dp_i = 0).
Decreasing the probability of human extinction over 2025 by 100 % does not imply Delta is astronomically large. For example, all the probability mass of future 0 could be moved to future 1 (dp_1 = -dp_0, and dp_i = 0 for i between 2 to 100). In this case, Delta = 9*|dp_0| would be at most 9 (dp_i â 1), i.e. far from astronomically large.
For Delta to be astronomically large, non-negligible probability mass has to be moved from future 0 to astronomically valuable futures. dp = sum_(i >= 1) dp_i not being astronomically small is not enough for that. I think the easiest way to avoid human extinction in 2025 is postponing it to 2026, which would not make the future astronomically valuable.
I agree the impact of decreasing the nearterm risk of human extinction does not have to be directly guessed. I just meant it has traditionally been guessed. In any case, I think one had better use standard cost-effectiveness analyses.
Thanks for the explanation, I have a clearer understanding of what you are arguing for now! Sorry I didnât appreciate this properly when reading the post.
So youâre claiming that if we intervene to reduce the probability of extinction in 2025, then that increases the probability of extinction in 2026, 2027, etc, even after conditioning on not going extinct earlier? The increase is such that the chance of reaching the far future is unchanged?
My next question is: why should we expect something like that to be true???
It seems very unlikely to me that reducing near term extinction risk in 2025 then increases P(extinction in 2026 | not going extinct in 2025). If anything, my prior expectation is that the opposite would be true. If we get better at mitigating existential risks in 2025, why would we expect that to make us worse at mitigating them in 2026?
If I understand right, youâre basing this on a claim that we should expect the impact of any intervention to decay exponentially as we go further and further into the future, and youâre then looking at what has to happen in order to make this true. I can sympathise with the intuition here. But I donât agree with how itâs being applied.
I think the correct way of applying this intuition is to say that itâs these quantities which will only be changed negligibly in the far future by interventions we take today:
P(going extinct in far future year X | we reach far future year X) (1)
E(utility in far future year X | we reach year X) (2)
In a world where the future has astronomical value, we obviously can astronomically change the expected value of the future by adjusting near-term extinction risk. To take an extreme example: if we make near-term extinction risk 100%, then expected future value becomes zero, however far into the future X is.
I think asserting that (1) and (2) are unchanged is the correct way of capturing the idea that the effect of interventions tends to wash out over time. That then leads to the conclusion from my original comment.
I think your life-expectancy example is helpful. But I think the conclusion is the opposite of what youâre claiming. If I play Russian Roulette and take an instantaneous risk of death, p, and my current life expectancy is L, then my life expectancy will decrease by pL. This is certainly non-negligible for non-negligible p, even though the time I take the risk over is minuscule in comparison to the duration of my life.
Of course I have changed your example here. You were talking about reducing the risk of death in a minuscule time period, rather than increasing it. Itâs true that that doesnât meaningfully change your life expectancy, but thatâs not because the duration of time is small in relation to your life, itâs because the risk of death in such a minuscule time period is already minuscule!
If we translate this back to existential risk, it does become a good argument against the astronomical cost-effectiveness claim, but itâs now a different argument. Itâs not that near-term extinction isnât important for someone who thinks the future has astronomical value. Itâs that: if you believe the future has astronomical value, then you are committed to believing that the extinction risk in most centuries is astronomically low, in which case interventions to reduce it stop looking so attractive. The only way to rescue the âastronomical cost-effectiveness claimâ is to argue for something like the âtime of perilsâ hypothesis. Essentially that we are doing the equivalent of playing Russian Roulette right now, but that we will stop doing so soon, if we survive.
No worries; you are welcome!
Yes, I think so.
It is not that I expect us to get worse at mitigation. I just expect it is way easier to move probability mass from the words with human extinction in 2025 to the ones with human extinction in 2026 than to ones with astronomically large value. The cost of moving physical mass increases with distance, and I guess the cost of moving probability mass increases (maybe exponentially) with value-distance (difference between the value of the worlds).
Yes.
Correct me if I am wrong, but I think you are suggesting something like the following. If there is a 99 % chance we are in future 100 (U_100 = 10^100), and a 1 % (= 1 â 0.99) chance we are in future 0 (U_0 = 0), i.e. if it is very likely we are in an astronomically valuable world[1], we can astronomically increase the expected value of the future by decreasing the chance of future 0. I do not agree. Even if the chance of future 0 is decreased by 100 %, I would say all its probability mass (1 pp) would be moved to nearby worlds whose value is not astronomical. For example, the expected value of the future would only increase by 0.09 (= 0.01*9) if all the probability mass was moved to future 1 (U_1 = 9).
The time of perils hypothesis implies the probability mass is mostly distributed across worlds with tiny and astronomical value. However, the conclusion I reached above does not depend on the initial probabilities of futures 0 and 100. It works just as well for a probability of future 0 of 50 %, and a probability of future 100 of 50 %. My conclusion only depends on my assumption that decreasing the probability of future 0 overwhelmingly increases the chance of nearby non-astronomically valuable worlds, having a negligible effect on the probability of astronomically valuable worlds.
If there was a 100 % of us being in an astronomically valuable world, there would be no nearterm extinction risk to be decreased.
The claim you quoted here was a lot simpler than this.
I was just pointing out that if we take an action to increase near-term extinction risk to 100% (i.e. we deliberately go extinct), then we reduce the expected value of the future to zero. Thatâs an undeniable way that a change to near-term extinction risk can have an astronomical effect on the expected value of the future, provided only that the future has astronomical expected value before we make the intervention.
But this is more or less a consequence of your claims isnât it?
I donât see any basis for this assumption. For example, it is contradicted by my example above, where we deliberately go extinct, and therefore move all of the probability weight from U_100 to U_0, despite their huge value difference.
Or I suppose maybe I do agree with your assumption (as canât think of any counter-examples I would actually endorse in practice) I just disagree with how youâre explaining its consequences. I would say it means the future does not have astronomical expected value, not that it does have astronomical value but that we canât influence it (since it seems clear we can if it does).
(If I remember our exchange on the Toby Ord post correctly, I think you made some claim along the lines of: there are no conceivable interventions which would allow us to increase extinction risk to ~100%. This seems like an unlikely claim to me, but itâs also I think a different argument to the one youâre making in this post anyway.)
Hereâs another way of explaining it. In this case the probability p_100 of U_100 is given by the huge product:
P(making it through next year) X P(making it through the year after given we make it through year 1) X âŚ..⌠etc
Changing near-term extinction risk is influencing the first factor in this product, so it would be weird if it didnât change p_100 as well. The same logic doesnât apply to the global health interventions that youâre citing as an analogy, and makes existential risk special.
In fact I would say it is your claim (that the later factors get modified too in just such a special way as to cancel out the drop in the first factor) which involves near-term interventions having implausible effects on the future that we shouldnât a priori expect them to have.
Agreed. However, I would argue that increasing the nearterm risk of human extinction to 100 % would be astronomically difficult/âcostly. In the framework of my previous comment, that would eventually require moving probability mass from world 100 to 0, which I believe is as super hard as moving mass world 0 to 100.
One can make a similar argument for the effect size of global health and development interventions. Assuming the effect size is strictly decreasing, denoting by Xi the effect size at year i, P(XN>x)=P(X1>x)P(X2>x|X1>x)...P(XN>x|XNâ1>x). Ok, P(XN>x) increases with P(X1>x) on priors. However, it could still be the case that the effect size will decay to practically 0 within a few decades or centuries.
I donât see why the same argument holds for global health interventions....?
Why should X_N > x require X_1 > x....?
It is not a strict requirement, but it is an arguably reasonable assumption. Are there any interventions whose estimates of (posterior) counterfactual impact, in terms of expected total hedonistic utility (not e.g. preventing the extinction of a random species), do not decay to 0 in at most a few centuries? From my perspective, their absence establishes a strong prior against persistent/âincreasing effects.
Sure, but once youâve assumed that already, you donât need to rely any more on an argument about shifts to P(X_1 > x) being cancelled out by shifts to P(X_n > x) for larger n (which if I understand correctly is the argument youâre making about existential risk).
If P(X_N > x) is very small to begin with for some large N, then it will stay small, even if we adjust P(X_1 > x) by a lot (we canât make it bigger than 1!) So we can safely say under your assumption that adjusting the P(X_1 > x) factor by a large amount does influence P(X_N > x) as well, itâs just that it canât make it not small.
The existential risk set-up is fundamentally different. We are assuming the future has astronomical value to begin with, before we intervene. That now means non-tiny changes to P(Making it through the next year) must have astronomical value too (unless there is some weird conspiracy among the probability of making it through later years which precisely cancels this out, but that seems very weird, and not something you can justify by pointing to global health as an analogy).
Thanks for the discussion, Toby. I do not plan to follow up further, but, for reference/âtransparency, I maintain my guess that the future is astronomically valuable, but that no interventions are astronomically cost-effective.