Error

I think a key point of contention here with many people (including me) who would endorse the argument for working to mitigate existential risks might be your background assumption that the per-unit risk rate is constant over time. I would put substantial probability (at least $5\%$ at a conservative minimum) on us being in a relatively short period of heightened existential risk, which will then be followed by a much longer and safer period. I think if you put substantial credence on this, then small relative reductions of the risk rate in this century still end up looking very good in expected value.

To make this concrete, consider the following simplified model. Suppose that we will face 10 centuries with a $10\%$ chance that we go extinct in each one, followed by a reduction of the per-century risk to ${10}^{-6}$. (Let’s suppose all these probabilities are independent for simplicity.) Under this model, the value of the future would be approximately $3.5\times {10}^{14}$ human lives.

Then, if we could decrease the relative risk of extinction this century by 1 in 100 million, this would be equivalent in expected value to saving approximately $1.4\times {10}^6$ lives. (To calculate this, consider the probability that we would have gone extinct this century, but our intervention prevented this, and that we would not have gone extinct in the following dangerous centuries.) Discounting by a further factor of 20 to account for my $5\%$ credence that this model is reasonable, this would give a lower bound on the value of our intervention of approximately $7\times {10}^4$ lives.

These are somewhat smaller than the numbers Bostrom gets (partly due to my conservative discounting for model uncertainty) but even so are large enough that I think his core point still stands.

I expect our key disagreement might be in whether we should assign non-negligible credence to humanity driving the background risk down to a very low level. While this might seem unlikely from our current perspective, I find it hard to justify putting a credence below $5\%$ on this happening. This largely comes from seeing several plausible pathways for this to happen (space colonisation, lock-in driven by TAI, etc.) plus quite a lot of epistemic modesty because predicting the future is hard.

I really appreciated this post and it’s sequel (and await the third in the sequence)! The “second mistake” was totally new to me, and I hadn’t grasped the significance of the “first mistake”. The post did persuade me that the case for existential risk reduction is less robust than I had previously thought.

One tiny thing. I think this should read “from 20% to 10% risk”:

More rarely, we talk about absolute reductions, which subtract an absolute amount from the current level of risk. It is in this sense that a 10% reduction in risk takes us from 80% to 70% risk, from 20% to 18% risk, or from 10% to 0% risk. (Formally, relative risk reduction by f takes us from risk r to risk r – f).

However, an absolute reduction of cumulative risk by 10^-8 requires (by definition) driving cumulative risk at least below 1-10^-8. Again, you say, that must be easy. Not so. Driving cumulative risk this low requires driving per-century risk to about 1.6*10^-6, barely one in a million.

I’m unclear on what this means. I currently think that humanity has better than a 10^-8 chance of surviving the next billion years, so can I just say that “driving cumulative risk at least below 1-10^-8” is already done? Is the 1.6*10^-6 per-century risk some sort of average of 10 million different per-century numbers (such that my views on the cumulative risk imply that this risk is similarly already below that number), or is this trying to force our thinking into an implausible-to-me model where the per-century risk is the same in every century, or is this talking about the first future century in which risk drops below that level?

On the whole, this decay-rate framing of the problem feels more confusing to me than something like a two-stage framing where there is some short-term risk of extinction (over the next 100 or 1000 years or similar) and then some probability of long-term survival conditional on surviving the first stage.

e.g., Suppose that someone thinks that humanity has a 10^-2 chance (1%) of surviving the next thousand years, and a 10^-4 chance (.01%) of surviving the next billion years conditional on surviving the next thousand years, and that our current actions can only affect the first of those two probabilities. Then increasing humanity’s chances of surviving a billion years by 10^-8 (in absolute terms) requires adding 10^-4 to our 10^-2 chance of surviving the next thousand years (an absolute .01% increase), or, equivalently, multiplying our chances of surviving the next thousand years by x1.01 (a 1% relative increase).