A model about the effect of total existential risk on career choice

[Jonas Moss]

Which existential risk cause should you focus on? The cause where you have the largest impact on decreasing total existential risk. That’s not the same as working on the cause where you have the largest impact when seen in isolation.

Model

Suppose there are $K$existential risks, each with its probability $p_k$ of ending the world. For each cause $k$ you can reduce the probability of the world ending from that cause by $d_k$, but only if you spend your whole career doing it.

For instance, suppose the risks are AI, biorisk, and asteroids. They have associated probabilities $p_1=0.9,p_2=0.10$ and $p_3=0.01$.^[1] How much could you decrease the probability of extinction for each cause? You’re pretty good at deflecting asteroids and killing viruses escaping from labs, but not that good at making humans lovable for AIs. Your probabilities are, say, $d_3={10}^{-7}$, $d_2={10}^{-7}$, $d_3=5\cdot {10}^{-8}$.

Risk type	Probability ($p_k$)	Probability reduction ($d_k$)
AI	0.9	$5\cdot {10}^{-8}$
Biorisk	0.1	${10}^{-7}$
Asteroids	0.01	${10}^{-7}$

Which career should you choose? It sounds plausible that you should be agnostic between the biorisk and asteroids path. That’s where you’ll reduce the probability of extinction the most, after all. But we should do a decision-theoretic analysis of the problem to make sure.

Let’s use the utility function where the world survives has utility $1$ and the world ceases to exist has utility $0$. Let $a=(a_1,\dots ,a_K)$ be a $0-1$ vector with $a_k=1$ if you choose action $k$ and $0$ otherwise. Then you ought to solve the total utility maximization problem

$$\underset{a\in A}{max}\prod _{k=1}^K(1-p_k+d_k{a}_k).$$

Why? Because you don’t care which event causes extinction, only that it doesn’t happen. And the total probability of no extinction equals ${\prod }_{k=1}^K(1-p_k+d_k{a}_k)$.

Anyway, we can show that

1. the optimal action, i.e., career path, is the one with the highest $d_k/(1-p_k)$;

2. the multiplicative improvement you’re causing by choosing action $k$ is $1+d_k/(1-p_k)$.

Proof

Define

$$\Pi =\prod _{k=1}^K(1-p_k).$$

The utility when taking action $k$ equals

$$\Pi \frac{1-p_k+d_k}{1-p_k}=\Pi (1+\frac{d_k}{1-p_k}),$$

which is clearly maximized in $k$ that maximises $d_k/(1-p_k)$.

Consequences

You need to take both the probability of extinction by cause $k$ and your ability to reduce the probability into account when you choose your career. If, for instance, the probability of AI ending the world ($p_1$) is higher than biorisk ending the world ($p_2$), you need to be at least $\frac{1-p_1}{1-p_2}$ times better at biorisk than AI risk (in terms of reducing the probability) to justify working on biorisk. If the probability of bio extinction is $0.01$ and the probability of AI extinction is $0.90$, you need to be $0.99/0.10=9.9\approx 10$ better at biorisk to justify doing biorisk instead of AI.

We can expand the table above to include the benefit of taking each action:

Risk type	Probability ($p_k$)	Probability reduction ($d_k$)	Benefit ($\approx$)
AI	0.9	$5\cdot {10}^{-8}$	$1+5\cdot {10}^{-7}$
Biorisk	0.1	${10}^{-7}$	$1+1.11\cdot {10}^{-7}$
Asteroids	0.01	${10}^{-6}$	$1+{10}^{-7}$

So, the AI safety career is better than the asteroid career. But not by a lot, as the number $(1+5\cdot {10}^{-7})/(1+{10}^{-7})$ is virtually indistinguishable from $1$. But of course, a higher number is a higher number, and they do add up. If we only care about the part $d_k/(1-p_k)$, which might be reasonable, doing the AI career is $5$ times better than the asteroids career. Which is more impressive.

A model with uncertainty

So, you say you have epistemic uncertainty about the probabilities of extinction from each cause? Perhaps you think your choice of entering a fiend may remove the risk entirely, not reduce it by a small number? (E.g., either you solve AI alignment, or you don’t).

That turns out not to matter. For the problem doesn’t change much when you allow for uncertainty. Provided $p_j,d_k$, $p_j,p_k$ and $d_j,d_k$ are independent when $j\ne k$ we find that

$$\begin{array}{ccc}{\text{argmax}}_{a\in A}E\left[\prod _{k=1}^K\right(1-p_k+d_k{a}_k\left)\right]& =& {\text{argmax}}_{a\in A}\prod _{k=1}^{K}E[1-p_k+d_k{a}_k],\\ & =& {\text{argmax}}_{a\in A}\Pi \frac{E(1-p_k)+E\left(d_k\right)a_k}{E(1-p_k)},\\ & =& {\text{argmax}}_{a\in A}\Pi [1+\frac{E\left(d_k\right)a_k}{1-E\left(p_k\right)}],\end{array}$$

where $\Pi ={\prod }_{k=1}^{K}E(1-p_k).$ The problem is maximized in the action with the highest $E\left[d_k\right]/(1-E[p_k\left]\right).$

Footnotes

1. ^

   These probabilities sum to more than $1$, but that doesn’t matter for our purposes. Think about them as the probabilities of independent events and the event “the world ends” as an event that happens if at least one of them occurs.

[MathiasKB🔸]

Thanks for writing this Jonas. As someone much below the lesswrong average at math, I would be grateful for a clarification of this sentence:

Provided $p_j,d_k$, $p_j,p_k$ and $d_j,d_k$ are independent when $j\ne k$

What does $j$ and $k$ refer to here? Moreover is it a reasonable assumption, that the uncertainties of existential risks are independent? It seems to me that many uncertainties run across risk types, such as chance of recovery after civilisations collapse.

[Jonas Moss]

$j$ and $k$ are indices for the causes. I wrote $j\ne k$ because you don’t have to assume that $d_k$ and $p_k$ are independent for the math to work. But everything else will have to be independent.

Maybe the uncertainties shouldn’t be independent, but often they will. Our uncertainty about the probability of AI doom is probably not related to our uncertainty about the probability of pandemics doom, for instance.

[JakubK]

I don’t understand what $d_k$ is. What do you mean by a “probability reduction” of $5\cdot {10}^{-8}$?

[Jonas Moss]

If the probability of extinction by cause $k$ is $p_k$ and the probability reduction for that cause is $d_k$, the probability of extinction becomes $p_k-d_k$ if you choose to focus on cause $k$.