It’s quite likely the extinction/existential catastrophe rate approaches zero within a few centuries if civilization survives, because:
Riches and technology make us comprehensively immune to natural disasters.
Cheap ubiquitous detection, barriers, and sterilization make civilization immune to biothreats
Advanced tech makes neutral parties immune to the effects of nuclear winter.
Local cheap production makes for small supply chains that can regrow from disruption as industry becomes more like information goods.
Space colonization creates robustness against local disruption.
Aligned AI blocks threats from misaligned AI (and many other things).
Advanced technology enables stable policies (e.g. the same AI police systems enforce treaties banning WMD war for billions of years), and the world is likely to wind up in some stable situation (bouncing around until it does).
If we’re more than 50% likely to get to that kind of robust state, which I think is true, and I believe Toby does as well, then the life expectancy of civilization is very long, almost as long on a log scale as with 100%.
Your argument depends on 99%+++ credence that such safe stable states won’t be attained, which is doubtful for 50% credence, and quite implausible at that level. A classic paper by the climate economist Martin Weitzman shows that the average discount rate over long periods is set by the lowest plausible rate (as the possibilities of high rates drop out after a short period and you get a constant factor penalty for the probability of low discount rates, not exponential decay).
A classic paper by the climate economist Martin Weitzman shows that the average discount rate over long periods is set by the lowest plausible rate (as the possibilities of high rates drop out after a short period).
Weitzman did not, however, supply any fundamental justification for his assumption that the effective discount factor is the expectation value of the true discount factor. That assumption is equivalent to the claim that a project should be adopted iff its expected net present value (ENPV) is positive. Gollier (2004) pointed out that if one adopts instead a criterion of positive expected net future value (ENFV), one obtains precisely the opposite of Weitzman’s result: the effective discount rate increases over time, tending as t → ∞ to its maximum possible value. Since neither criterion is obviously inferior to the other, the difference in results generated the so-called ‘Gollier-Weitzman puzzle’.
...
In the lively post-2004 debate over the Gollier-Weitzman puzzle (Hepburn & Groom, 2007; Buchholz & Schumacher, 2008; Gollier & Weitzman, 2010; Freeman, 2010; Gollier, 2010; Traeger, 2012a; K. J. Arrow et al., 2014), there is a widespread consensus that something like Weitzman’s original conclusion is correct, although participants to the debate continue to differ significantly over the reasons for this conclusion and the precise conditions under which it holds.
Barring simulation shutdown sorts of things or divine intervention I think more like 1 in 1 million per century, on the order of magnitude of encounters with alien civilizations. Simulation shutdown is a hole in the argument that we could attain such a state, and I think a good reason not to say things like ‘the future is in expectation 50 orders of magnitude more important than the present.’
Whether simulation shutdown is a good reason not to say such things would seem to depend on how you model the possibility of simulation shutdown.
One naive model would say that there is a 1/n chance that the argument that such a risk exists is correct, and if so, there is 1/m annual risk, otherwise, there is 0 annual risk from simulation shutdown. In such a model, the value of the future endowment would only be decreased n-fold. Whereas if you thought that there was definitely a 1/(mn) annual risk (i.e. the annual risk is IID) then that risk would diminish the value of the cosmic endowment by many OoM.
Simulation shutdown would end value in our reality, but in that of the simulators it would presumably continue, such that future expected value would be greater than the suggested by an extinction risk of 10^-6 per century? On the other hand, even if this is true, it would not matter because it would be quite hard to influence the eventual simulators?
This seems like an important claim, but I could also quite plausibly see misaligned AIs destabilising a world where an aligned AI exists. What reasons do we have to think that an aligned AI would be able to very consistently (e.g. 99.9%+ of the time) ward off attacks from misaligned AI from bad actors? Given all the uncertainty around these scenarios, I think the extinction risk per century from this alone could in the 1-5% range and create a nontrivial discount rate.
Today there is room for an intelligence explosion and explosive reproduction of AGI/robots (the Solar System can support trillions of AGIs for every human alive today). If aligned AGI undergoes such intelligence explosion and reproduction there is no longer free energy for rogue AGI to grow explosively. A single rogue AGI introduced to such a society would be vastly outnumbered and would lack special advantages, while superabundant AGI law enforcement would be well positioned to detect or prevent such an introduction in any case.
Already today states have reasonably strong monopolies on the use of force. If all military equipment (and an AI/robotic infrastructure that supports it and most of the economy) is trustworthy (e.g. can be relied on not to engage in military coup, to comply with and enforce international treaties via AIs verified by all states, etc) then there could be trillions of aligned AGIs per human, plenty to block violent crime or WMD terrorism.
For war between states, that’s point #7. States can make binding treaties to renounce WMD war or protect human rights or the like, enforced by AGI systems jointly constructed/inspected by the parties.
One possibility would be that these misaligned AIs are quickly defeated or contained, and future ones are also severely resource-constrained by the aligned AI, which has a large resource advantage. So there are possible worlds where there aren’t really any powerful misaligned AIs (nearby), and those worlds have vast futures.
[Epistemic status: I have just read the summary of the post by the author and by Zoe Williams]
Doesn’t it make more sense to just set the discount rate dynamically accounting for the current (at the present) estimations?
If/when we reach a point where the long-term existential catastrophe rate approaches zero, then it will be the moment to set the discount rate to zero. Now it is not, so the discount rate should be higher as the author proposes.
Is there a reason for not using in a dynamic discount rate?
If you think the annual rate of catastrophic risk is X this century but only 0.5X next century, 0.25X the century afterwards etc, then you’d vastly underestimate expected value of the future if you use current x-risk levels to set the long-term discount rate.
Whether this is practically relevant or not is of course significantly more debatable.
It’s quite likely the extinction/existential catastrophe rate approaches zero within a few centuries if civilization survives, because:
Riches and technology make us comprehensively immune to natural disasters.
Cheap ubiquitous detection, barriers, and sterilization make civilization immune to biothreats
Advanced tech makes neutral parties immune to the effects of nuclear winter.
Local cheap production makes for small supply chains that can regrow from disruption as industry becomes more like information goods.
Space colonization creates robustness against local disruption.
Aligned AI blocks threats from misaligned AI (and many other things).
Advanced technology enables stable policies (e.g. the same AI police systems enforce treaties banning WMD war for billions of years), and the world is likely to wind up in some stable situation (bouncing around until it does).
If we’re more than 50% likely to get to that kind of robust state, which I think is true, and I believe Toby does as well, then the life expectancy of civilization is very long, almost as long on a log scale as with 100%.
Your argument depends on 99%+++ credence that such safe stable states won’t be attained, which is doubtful for 50% credence, and quite implausible at that level. A classic paper by the climate economist Martin Weitzman shows that the average discount rate over long periods is set by the lowest plausible rate (as the possibilities of high rates drop out after a short period and you get a constant factor penalty for the probability of low discount rates, not exponential decay).
Not really disagreeing with anything you’ve said, but I just wanted to point out that this sentence doesn’t quite acknowledge the whole story. To quote from Hilary Greaves’ “Discounting for public policy: A survey” paper:
The link you just posted seems broken? Hilary Greaves’ Discounting for public policy: A survey is available in full at the following URL.
What would be the per-century risk in such a state?
Also, does the >50% of such a state account for the possibility of alien civilizations destroying us or otherwise limiting our expansion?
Barring simulation shutdown sorts of things or divine intervention I think more like 1 in 1 million per century, on the order of magnitude of encounters with alien civilizations. Simulation shutdown is a hole in the argument that we could attain such a state, and I think a good reason not to say things like ‘the future is in expectation 50 orders of magnitude more important than the present.’
Whether simulation shutdown is a good reason not to say such things would seem to depend on how you model the possibility of simulation shutdown.
One naive model would say that there is a 1/n chance that the argument that such a risk exists is correct, and if so, there is 1/m annual risk, otherwise, there is 0 annual risk from simulation shutdown. In such a model, the value of the future endowment would only be decreased n-fold. Whereas if you thought that there was definitely a 1/(mn) annual risk (i.e. the annual risk is IID) then that risk would diminish the value of the cosmic endowment by many OoM.
I’d use reasoning like this, so simulation concerns don’t have to be ~certain to drastically reduce EV gaps between local and future oriented actions.
What is the source of the estimate of the frequency of encounters with alien civilizations?
Here’s a good piece.
Hi Carl,
Simulation shutdown would end value in our reality, but in that of the simulators it would presumably continue, such that future expected value would be greater than the suggested by an extinction risk of 10^-6 per century? On the other hand, even if this is true, it would not matter because it would be quite hard to influence the eventual simulators?
You could have acausal influence over others outside the simulation you find yourself in, perhaps especially others very similar to you in other simulations. See also https://longtermrisk.org/how-the-simulation-argument-dampens-future-fanaticism
Thanks for sharing that article, Michael!
On the topic of acausal influence, I liked this post, and this clarifying comment.
This seems like an important claim, but I could also quite plausibly see misaligned AIs destabilising a world where an aligned AI exists. What reasons do we have to think that an aligned AI would be able to very consistently (e.g. 99.9%+ of the time) ward off attacks from misaligned AI from bad actors? Given all the uncertainty around these scenarios, I think the extinction risk per century from this alone could in the 1-5% range and create a nontrivial discount rate.
Today there is room for an intelligence explosion and explosive reproduction of AGI/robots (the Solar System can support trillions of AGIs for every human alive today). If aligned AGI undergoes such intelligence explosion and reproduction there is no longer free energy for rogue AGI to grow explosively. A single rogue AGI introduced to such a society would be vastly outnumbered and would lack special advantages, while superabundant AGI law enforcement would be well positioned to detect or prevent such an introduction in any case.
Already today states have reasonably strong monopolies on the use of force. If all military equipment (and an AI/robotic infrastructure that supports it and most of the economy) is trustworthy (e.g. can be relied on not to engage in military coup, to comply with and enforce international treaties via AIs verified by all states, etc) then there could be trillions of aligned AGIs per human, plenty to block violent crime or WMD terrorism.
For war between states, that’s point #7. States can make binding treaties to renounce WMD war or protect human rights or the like, enforced by AGI systems jointly constructed/inspected by the parties.
One possibility would be that these misaligned AIs are quickly defeated or contained, and future ones are also severely resource-constrained by the aligned AI, which has a large resource advantage. So there are possible worlds where there aren’t really any powerful misaligned AIs (nearby), and those worlds have vast futures.
Could you say more about what you mean by this?
[Epistemic status: I have just read the summary of the post by the author and by Zoe Williams]
Doesn’t it make more sense to just set the discount rate dynamically accounting for the current (at the present) estimations?
If/when we reach a point where the long-term existential catastrophe rate approaches zero, then it will be the moment to set the discount rate to zero. Now it is not, so the discount rate should be higher as the author proposes.
Is there a reason for not using in a dynamic discount rate?
If you think the annual rate of catastrophic risk is X this century but only 0.5X next century, 0.25X the century afterwards etc, then you’d vastly underestimate expected value of the future if you use current x-risk levels to set the long-term discount rate.
Whether this is practically relevant or not is of course significantly more debatable.
But the underestimation would be temporal.
I’m also not clear on whether this is practically relevant or not.