However, an absolute reduction of cumulative risk by 10-8 requires (by definition) driving cumulative risk at least below 1-10-8. Again, you say, that must be easy. Not so. Driving cumulative risk this low requires driving per-century risk to about 1.6*10-6, barely one in a million.
I’m unclear on what this means. I currently think that humanity has better than a 10-8 chance of surviving the next billion years, so can I just say that “driving cumulative risk at least below 1-10-8” is already done? Is the 1.6*10-6 per-century risk some sort of average of 10 million different per-century numbers (such that my views on the cumulative risk imply that this risk is similarly already below that number), or is this trying to force our thinking into an implausible-to-me model where the per-century risk is the same in every century, or is this talking about the first future century in which risk drops below that level?
On the whole, this decay-rate framing of the problem feels more confusing to me than something like a two-stage framing where there is some short-term risk of extinction (over the next 100 or 1000 years or similar) and then some probability of long-term survival conditional on surviving the first stage.
e.g., Suppose that someone thinks that humanity has a 10^-2 chance (1%) of surviving the next thousand years, and a 10^-4 chance (.01%) of surviving the next billion years conditional on surviving the next thousand years, and that our current actions can only affect the first of those two probabilities. Then increasing humanity’s chances of surviving a billion years by 10^-8 (in absolute terms) requires adding 10^-4 to our 10^-2 chance of surviving the next thousand years (an absolute .01% increase), or, equivalently, multiplying our chances of surviving the next thousand years by x1.01 (a 1% relative increase).
Thanks Dan! As mentioned, to think that cumulative risk is below 1-(10^-8) is to make a fairly strong claim about per-century risk. If you think we’re already there, that’s great!
Bostrom was actually considering something slightly stronger: the prospect of reducing cumulative risk by a further 10^(-8) from wherever it is at currently. That’s going to be hard even if you think that cumulative risk is already lower than I do. So for example, you can ask what changes you’d have to make to per-century risk to drop cumulative risk from N to r-(10^-8) for any r in [0,1). Honestly, that’s a more general and interesting way to do the math here. The only reason I didn’t do this is that (a) it’s slightly harder, and (b) most academic readers will already find per-century risk of ~one-in-a-million relatively implausible, and (c) my general aim was to illustrate the importance of carefully distinguishing between per-century risk and cumulative risk.
It might be a good idea, in rough terms, to think of a constant hazard rate as an average across all centuries. I suspect that if the variance of risk across centuries is low-ish, this is a good idea, whereas if the variance of risk across centuries is high-ish, it’s a bad idea. In particular, on a time of perils view, focusing on average (mean) risk rather than explicit distributions of risk across centuries will strongly over-value the future, since a future in which much of the risk is faced early on is lower-value than a future in which risk is spread out.
Strong declining trends in hazard rates induce a time-of-perils like structure, except that on some models they might make a bit weaker assumptions about risk than leading time of perils models do. At least one leading time of perils model (Aschenbrenner) has a declining hazard structure. In general, the question will be how to justify a declining hazard rate, given a standard story on which (a) technology drives risk, and (b) technology is increasing rapidly. I think that some of the arguments against the time of perils hypothesis made in my paper “Existential risk pessimism and the time of perils” against the time of perils hypothesis will be relevant here, whereas others may be less relevant, depending on your view.
In general, I’d like to emphasize the importance of arguing for views about future rates of existential risk. Sometimes effective altruists are very quick to produce models and assign probabilities to models. Models are good (they make things clear!) but they don’t reduce the need to support models with arguments, and assignments of probability are not arguments, but rather statements in need of argument.
I’m unclear on what this means. I currently think that humanity has better than a 10-8 chance of surviving the next billion years, so can I just say that “driving cumulative risk at least below 1-10-8” is already done? Is the 1.6*10-6 per-century risk some sort of average of 10 million different per-century numbers (such that my views on the cumulative risk imply that this risk is similarly already below that number), or is this trying to force our thinking into an implausible-to-me model where the per-century risk is the same in every century, or is this talking about the first future century in which risk drops below that level?
On the whole, this decay-rate framing of the problem feels more confusing to me than something like a two-stage framing where there is some short-term risk of extinction (over the next 100 or 1000 years or similar) and then some probability of long-term survival conditional on surviving the first stage.
e.g., Suppose that someone thinks that humanity has a 10^-2 chance (1%) of surviving the next thousand years, and a 10^-4 chance (.01%) of surviving the next billion years conditional on surviving the next thousand years, and that our current actions can only affect the first of those two probabilities. Then increasing humanity’s chances of surviving a billion years by 10^-8 (in absolute terms) requires adding 10^-4 to our 10^-2 chance of surviving the next thousand years (an absolute .01% increase), or, equivalently, multiplying our chances of surviving the next thousand years by x1.01 (a 1% relative increase).
Thanks Dan! As mentioned, to think that cumulative risk is below 1-(10^-8) is to make a fairly strong claim about per-century risk. If you think we’re already there, that’s great!
Bostrom was actually considering something slightly stronger: the prospect of reducing cumulative risk by a further 10^(-8) from wherever it is at currently. That’s going to be hard even if you think that cumulative risk is already lower than I do. So for example, you can ask what changes you’d have to make to per-century risk to drop cumulative risk from N to r-(10^-8) for any r in [0,1). Honestly, that’s a more general and interesting way to do the math here. The only reason I didn’t do this is that (a) it’s slightly harder, and (b) most academic readers will already find per-century risk of ~one-in-a-million relatively implausible, and (c) my general aim was to illustrate the importance of carefully distinguishing between per-century risk and cumulative risk.
It might be a good idea, in rough terms, to think of a constant hazard rate as an average across all centuries. I suspect that if the variance of risk across centuries is low-ish, this is a good idea, whereas if the variance of risk across centuries is high-ish, it’s a bad idea. In particular, on a time of perils view, focusing on average (mean) risk rather than explicit distributions of risk across centuries will strongly over-value the future, since a future in which much of the risk is faced early on is lower-value than a future in which risk is spread out.
Strong declining trends in hazard rates induce a time-of-perils like structure, except that on some models they might make a bit weaker assumptions about risk than leading time of perils models do. At least one leading time of perils model (Aschenbrenner) has a declining hazard structure. In general, the question will be how to justify a declining hazard rate, given a standard story on which (a) technology drives risk, and (b) technology is increasing rapidly. I think that some of the arguments against the time of perils hypothesis made in my paper “Existential risk pessimism and the time of perils” against the time of perils hypothesis will be relevant here, whereas others may be less relevant, depending on your view.
In general, I’d like to emphasize the importance of arguing for views about future rates of existential risk. Sometimes effective altruists are very quick to produce models and assign probabilities to models. Models are good (they make things clear!) but they don’t reduce the need to support models with arguments, and assignments of probability are not arguments, but rather statements in need of argument.