‘Existential Risk and Growth’ Deep Dive #1 - Summary of the Paper

[Alex HT]

This is the first of a three-part series of posts from me and Ben Snodin with input from Phil Trammell summarising, critiquing, and suggesting variations on Leopold Aschenbrenner’s Existential Risk and Growth (2019).

• This first post gives an overview of the model and results of the paper.

• The second post will discuss how seriously to take the conclusions that follow from the model, highlight some of the more robust conclusions, and discuss objections to the model.

• The third post will outline directions of further research that we believe to be most promising.

The series should be useful to anyone interested in understanding existential risk, in particular:

• Those interested in how economic growth influences existential risk;

• Those thinking about whether we can find useful insights about the complex systems that determine existential risk through relatively simple, abstract models.

Hopefully we’ll demonstrate that this kind of research is a fruitful way to get insights about existential risk, and others will be motivated to improve on the existing paper. The series assumes understanding of basic calculus but no formal economics knowledge.

Summary Outline

Section 1 - Setup introduces a model of an economy containing a consumption sector and a safety sector. Section 2 - Allocations explores the very long-term behaviour of this economy for two different cases: a fixed allocation of labour across different sectors, and a variable allocation determined by the preferences of the population. Section 3 - Transition Dynamics looks at how this economy behaves at intermediate times when the allocation between sectors is determined by the preferences of the population.

The results of the model are that civilisation may be going through a ‘time of perils’ in which it is rich enough to have the advanced technologies that threaten its survival, but is not yet rich enough to be willing to spend sufficiently on mitigating these risks. It might then be prudent to minimise time spent in this time of perils—by preventing economic stagnation, or accelerating economic growth. In light of this result, Section 4 - Patience vs. Growth compares the impact on existential risk of making the population more patient with the value of making the population wealthier.

1. Setup

This section builds a model of an economy including population, safety, consumption, and existential risk. We will then see how this model evolves over time.

Population

We model the economy as containing two sectors—a consumption sector, and a safety sector. There are consumption goods which people buy and consume to increase their welfare. There are safety goods (such as pollution mitigation, nonproliferation treaties, or pandemic preparedness programs), which reduce the risk of catastrophe. Each person in the economy works in either the consumption sector or the safety sector. People employed in a given sector can be workers or scientists. Workers produce goods for their respective sector. Scientists produce technologies for their respective sectors.

We can define labour allocation variables for this system:

Where $N_t$ is the total population, $t$ is time and the subscript $t$ indicates that the variable is a function of time.

Labour Allocation Variables

There are three questions we can ask for this model:

1. What is the fraction $s_t$ of scientists at time t working in the consumption
sector? (The other scientists are in the safety sector.)

2. What is the fraction $l_t$ of workers at time t working in the consumption sector?
(The other workers are in the safety sector.)

3. What is the fraction ${\sigma }_t$ of people at time t working as scientists? (The other
people are workers.)

So our allocation variables are $s_t$, $l_t$, and ${\sigma }_t$. These are all the variables that can be controlled in the model.

Production Functions

We assume that the production of consumption technologies, $A_t$, and safety technologies, $B_t$, are governed by these equations:

$\frac{dA}{dt}=S_{at}^{\lambda }{A}_t^{\varphi }$ ,

$\frac{dB}{dt}=S_{bt}^{\lambda }{B}_t^{\varphi }$ .

Where $\varphi$ and $\lambda$ are constants, $\varphi <1$ and $0<\lambda <1$. This is just to say that technological progress has diminishing returns in scientists and existing technologies. We assume that the production of consumption goods, $C_t$, and safety goods, $H_t$, are governed by these equations:

$C_t=A_t^{\alpha }{L}_{at}$

$H_t=B_t^{\alpha }{L}_{bt}$

Where $0<\alpha <1$. (In the current version of the paper, there are slightly different production functions. These give similar results but add unnecessary complication. My understanding is that the will be changed to the functions presented in this summary.)

Similarly to the labour allocation variables, we can define per capita variables for production in each sector $c_t=\frac{C_t}{N_t}$ and $h\left(t\right)=\frac{H_t}{N_t}$.. We assume a rate of population growth $\overline{n}$ that is exogenous to the model so that: $\frac{dN}{dt}=\overline{n}{N}_t$.

Existential Risk

Consumption goods increase welfare -which is great. But they can also increase the probability of existential risk -for example, by more carbon emissions, more powerful AI systems, or more dangerous biotechnology. This risk depends on the total amount of consumption goods being produced, and the risk mitigation depends on the total amount of safety goods being produced.

Risk of catastrophe at a time $t$ is given by:

${\delta }_t=\overline{\delta }{C}_t^{ϵ}{H}_t^{-\beta }$

Where $\overline{\delta }$ is a constant, $C_t$ is the production in the consumption sector, $H_t$ is the production in the safety sector, $ϵ$ is the potency of consumption goods on increasing existential risk, and $\beta$ is the potency of safety technologies on reducing existential risk. This formalizes the idea of `differential technological development’: existential risk depends on the rate of development of potentially dangerous technologies relative to technologies that ameliorate these hazards.

Therefore, the probability of civilisation surviving to time $t$ is given by:

$M_t=e^{-{\int }_0^t{\delta }_t^{\prime }d{t}^{\prime }}$

This is useful to consider for two reasons. Firstly, by considering the limit as $t\to \infty$, we can see the probability that civilisation never destroys itself:

$M_{\infty }=e^{-{\int }_0^{\infty }{\delta }_t^{\prime }d{t}^{\prime }}$.

This probability is only non-zero if ${\int }_0^{\infty }{\delta }_t^{\prime }d{t}^{\prime }$ is finite. So instantaneous existential risk has to decrease quickly enough over time or civilisation will destroy itself eventually. The second reason this is useful is because of the role it plays in agents’ actions. This is discussed in the next section.

Agent utility

If we first find the utility function of a representative agent in this model, we can then see how that agent would allocate labour to maximise her own utility. Then, if people in the world collectively allocated labour in accordance with their preferences, the results of this model indicate real world labour allocation. A representative agent cares about their own instantaneous utility ut which depends on their consumption at time at that time ct. However, they discount this utility by two things:

1. A survival probability $M_t$ - future consumption is less valuable than present
consumption because of uncertainty about whether civilisation will survive;

2. A discount factor $e^{-\rho t}$ - people behave as is they value welfare now more than
welfare later.

So agents will want to maximise this utility function:

$U={\int }_0^{\infty }{e}^{-\rho t}u\left(c_t\right)M_{t}dt.$

Form of the Utility Function

We assume a utility function of the following form:

$u\left(c_t\right)=\overline{u}+\frac{c_t^{1-\gamma }}{1-\gamma }$

$\overline{u}$ is a constant normalising the utility of death to zero. The second term hasconstantelasticity in consumption (unless $\gamma =1$). So increasing $c$ by some fraction $a$ always increases $u-\overline{u}$ by some other fraction $b$.

This means that the total utility, including $\overline{u}$, has constant elasticity of marginal utility of consumption. So multiplying $c$ by some small fraction $a$ always multiplies $\frac{du}{dc}$by some other fraction $b$.

$\gamma$ is a parameter that measures people’s preferences about risk (the coefficient of relative risk aversion). If $\gamma =0$ then the preferences associated with the utility function are risk neutral (so twice as much consumption means twice as much as utility). For higher values of $\gamma$, the value of life relative to the value of marginal consumption is higher. For these cases, an increase in wealth will create more demand for safety relative to consumption.

[Edit: this section incorrectly said the the whole utility function has constant elasticity in consumption, which isn’t true—only the second term has constant elasticity in consumption. The whole utility function has constant elasticity of marginal utility in consumption.]

2. Allocations

The allocation variables $s_t$, $l_t$, and ${\sigma }_t$ could either be set at some fixed levels, or they could be allowed to vary to represent agent’s shifting priorities as the system evolves. The first case isn’t particularly realistic and, in short, in the fixed allocation, we find that:

1. If $ϵ<\beta$, the hazard rate ${\delta }_t$ falls exponentially and $M_{\infty }>0$,

2. If $ϵ>\beta$, the hazard rate δ increases exponentially and $M_{\infty }=0$,

3. If $ϵ=\beta$, the hazard rate converges to a constant and $M_{\infty }=0$.

This highlights the crucial role of the value $ϵ-\beta$ - it represents the scale effect of existential risk, or the fragility of civilisation. If $ϵ-\beta <0$ then increasing the size of the economy while keeping labour allocations constant will reduce existential risk. And if $ϵ-\beta <0$ then increasing the size of the economy while keeping labour allocations constant will increase existential risk.

Optimal Impatient Allocation

As people grow wealthier, the marginal utility of consumption declines, and it becomes relatively more valuable to purchase more life and spend on avoiding death. In this model, agents are able to do this through spending on the safety sector. This section explores how agents might allocate variables for different values of $ϵ-\beta$ .

Constrained Optimisation

[This section is a bit technical, so feel free to skip to the results, starting with ‘Robust World’. Though if you are interested, I’ve written up an explanation of how to use Hamiltonians for systems like these here.]

We want to find the path of $s_t$, $l_t$, and ${\sigma }_t$that maximises

$U={\int }_0^{\infty }{e}^{-\rho t}u\left(c_t\right)M_{t}dt$

subject to the constraints of the technology production functions, and the survival probability:

$\dot{A}=[{\sigma }_t{s}_t{N}_t{]}^{\lambda }{A}_t^{\varphi },$

$\dot{B}=\left[{\sigma }_t\right(1-s_t)N_t{]}^{\lambda }{B}_t^{\varphi }$ $,$

$\dot{M}=-\delta c_t^{ϵ}{h}_t^{-\beta }{M}_t$.

Where the overdot indicates the derivative with respect to time. To find the path that optimises U subject to our constraints we can define a Hamiltonian H:

$H=M_{t}u\left(c_t\right)+{\mu }_{At}({\sigma }_t{s}_t{N}_t{)}^{\lambda }{A}_t^{\varphi }+{\mu }_{Bt}({\sigma }_t(1-s_t)N_t{)}^{\lambda }{B}_t^{\varphi }-{\mu }_{Mt}{\delta }_t{c}_t^{ϵ}{h}_t^{-\beta }{M}_t$

Where ${\mu }_i$ are Lagrange multipliers. Then the optimal path of $s_t$, $l_t$, and ${\sigma }_t$ is characterised by the following conditions holding at all times:

$\frac{\partial H}{\partial s}=0$,

$\frac{\partial H}{\partial l}=0$,

$\frac{\partial H}{\partial \sigma }=0$,

$\frac{\partial H}{\partial A}=-{\dot{\mu }}_A+\rho {\mu }_A$,

$\frac{\partial H}{\partial B}=-{\dot{\mu }}_B+\rho {\mu }_B$,

$\frac{\partial H}{\partial M}=-{\dot{\mu }}_M+\rho {\mu }_M$.

subject to the boundary conditions:

${lim}_{t\to \infty }\left(e^{-\rho t}{\mu }_{At}{A}_t\right)=0$,

${lim}_{t\to \infty }\left(e^{-\rho t}{\mu }_{Bt}{B}_t\right)=0$,

${lim}_{t\to \infty }\left(e^{-\rho t}{\mu }_{Mt}{M}_t\right)=0$.

These are the conditions solved in the paper, and the results are presented below.

Robust World

If $ϵ\le \beta$, so that existential risk falls with scale, then as long as there is growth, and the proportion of resources going to safety does not decline exponentially or faster (which it will not, even along the impatient-optimal path), risk decreases. Furthermore, exponential growth will produce an exponential fall in the hazard rate, and a positive probability of avoiding existential catastrophe.

Fragile World

If $ϵ>\beta$, so that existential risk grows with scale:

1.$\gamma >1$ the relative value of life rises as consumption grows. So does risk, but
labour is moved from consumption to safety quickly enough that ${\delta }_t$ declines
quickly enough. Unlike in the fixed allocation, there is a positive, nonzero
probability that humanity does not succumb to an existential catastrophe - $M_{\infty }>0$

2. If $\gamma \le 1$, the relative value of life is slightly less. Labour is still shifted towards
safety, but not quickly enough for the hazard rate to decay - $M_{\infty }=0$.

Hyper-Fragile World

If $ϵ\gg \beta$ then there is no safe allocation on the optimal path. Even if almost everyone works on safety, civilisation will destroy itself eventually. The condition for this is that $\frac{ϵ-\beta }{\beta }>\frac{\alpha \lambda }{1-\varphi }$, which can be interpreted as the scale effect of existential risk being greater than the scale effect of ideas.

Note that, if $ϵ\gg \beta$ then there is no desirable allocation, not even from a fully patient perspective, that causes δ to decay quickly enough for $M_{\infty }>0$. To lower existential risk quickly enough, labour must be shifted so far from consumption to safety that instantaneous utility falls below $\overline{u}$ -rendering death preferable to life.

3. Transition Dynamics

So far, we have looked at what state the system tends to for large values of time. But we are probably far from this long-run result. To understand the relationship between growth and risk today, we need to look at how the system changes as it moves towards the steady state. We’ll look at the impatient optimal allocation for the case $ϵ>\beta$ because this seems like the most action-relevant case. If $ϵ\gg \beta$ then there is no safe feasible allocation of labour, and if $ϵ\le \beta$ then, as long as there is some growth and safety spending, $M_{\infty }>0$. Also, the case $ϵ>\beta$ is perhaps the most likely. Too see this, consider that both anthropogenic existential risk and safety spending seem to have increased over time, suggesting the scale effect of existential risk is positive.

To get a sense of the qualitative dynamics of the system we have to pick some parameter values. These values, and evidence for them, are discussed in the paper, which also says that other values produce similar results. We’re particularly interested in the behaviour of the instantaneous hazard rate ${\delta }_t$ over time. The hazard rate follows an inverted U-shape. It starts at a relatively low level, but then grows with scale. At the time representing today, ${\delta }_t$ is much higher than it was hundreds of years ago. However, as consumption grows, the value of extra life increases and so resources are shifted to safety. This slows the growth of ${\delta }_t$ and eventually causes it to decrease. Note that this is analogous to other empirical trends we’ve observed, like environmental degradation (and perhaps meat consumption) where, when people are poorer they are more willing to tolerate some bad thing if it makes increasing consumption easier. Then, as they become wealthier, they value reducing the bad thing relatively more than they did before.

Acceleration

Imagine we increase growth for a period of 30 years and then it goes back to its previous steady level. E.g. imagine annual population growth goes from its usual value of 1% to a higher rate of 2% for 30 years, before going back to 1%.

The effect of accelerated growth for a period of time is to make people richer earlier, so they sooner value life relatively more, so they sooner spend more on safety. This means that the overall area under the hazard rate curve is lower, so the total probability of existential catastrophe over time is smaller. The reverse happens when growth decelerates. Society is stuck with higher levels of risk for longer. So slower growth, even just for a while, might not just mean lower living standards, but perhaps also significantly smaller chance of civilisation’s survival.

Transitory Boom

Imagine we increase growth for a period of 30 years and then it goes back to a lower value for some time such that the overall growth was the same as if the boom hadn’t happened. In particular, imagine that annual population growth goes from its usual value of 1% to a higher rate of 2% for 30 years, before going all the way down to 0% for 30 years, and then eventually back to 1%. What will the effect on hazard rate be? In this case, growth rates of technologies and consumption are the same after the boom as they were before. The allocation between sectors is the same after the boom as the relative value of life is the same after the boom.

The trajectory of the hazard rate after the boom is the same as without the boom. But in the process of the boom there was increased existential risk. Unlike the acceleration case, this temporary increase in risk came with no long-term decrease in risk. Simply “juicing” growth for a while may actually backfire, reducing the probability of humanity’s survival. The opposite occurs for a transitory bust—the hazard rate curve is perturbed downwards and then returns to its previous trajectory.

4. Patience vs. Growth

Safety can be increased by increasing consumption (so people are richer, and value life relatively more), or by reducing their rate of pure time preference (so they value future welfare relatively more). If we want civilisation to demand more existential risk mitigation, should we prioritise economic growth, or moral longtermism? This depends on γ and the relative value of life. For a higher relative value of life it needs more and more consumption to increase safety-focus so economic growth becomes less effective at reducing existential risk. The table below shows, for different values of $\gamma$, what the elasticity of concern for safety in ρ is (i.e. how sensitive concern for safety is to changes in the discount rate). For comparison, the third column shows what ρ would have to be reduced to from its original value of 2% to match the increase in the concern for life from doubling consumption.

Conclusion

From the paper:

‘Technological development can create or mitigate existential risks...humanity may be in a critical “time of perils.” We may be economically advanced enough to be able to destroy ourselves, but not economically advanced enough that we care about this existential risk and spend on safety. This “time of perils” implies that working on reducing existential risk now could be very impactful from an altruistic perspective. Faster economic growth, while initially increasing risk, can help us get through this “time of perils” more quickly and thus increases the long-run probability of humanity’s survival. Conversely, short-term economic stagnation could substantially curtail the future of human civilization. Even if you care only about the long-term future of humanity, the pace of economic growth in the short run could be key to whether we make it there.’

[Pablo]

Leopold has now published a popular article discussing this topic. Highly recommended.

An excerpt:

Philosophers like Nick Bostrom, Derek Parfit, and Toby Ord have become increasingly concerned about such so-called “existential risks.” An unrecoverable collapse of civilization wouldn’t just be tragic for the billions who would suffer and die. Perhaps the greatest tragedy would be the foreclosing of all of humanity’s potential. Humanity could flourish for billions of years and enable trillions of happy human lives—if only we do not destroy ourselves beforehand.

This line of thinking has led some to question whether “progress”—in particular, technological progress—is as straightforwardly beneficial as commonly assumed. Nick Bostrom imagines the process of technological development as “pulling balls out of a giant urn.” So far, we’ve been lucky, pulling out a great many “white” balls that are broadly beneficial. But someday, we might pull out a “black” ball: a new technology that destroys humanity. Before that first nuclear test, some of the physicists worried that the nuclear bomb would ignite the atmosphere and end the world. Their calculations ultimately deemed it “extremely unlikely,” and so they proceeded with the test—which, as it turns out, did not end the world. Perhaps the next time, we don’t get so lucky.

The same technological progress that creates these risks is also what drives economic growth. Does that mean economic growth is inherently risky? Economic growth has brought about extraordinary prosperity. But for the sake of posterity, must we choose safe stagnation instead? This view is arguably becoming ever-more popular, particularly amongst those concerned about climate change; Greta Thunberg recently denounced “fairy tales of eternal economic growth” at the United Nations.

I argue that the opposite is the case. It is not safe stagnation and risky growth that we must choose between; rather, it is stagnation that is risky and it is growth that leads to safety.

We might indeed be in “time of perils”: we might be advanced enough to have developed the means for our destruction, but not advanced enough to care sufficiently about safety. But stagnation does not solve the problem: we would simply stagnate at this high level of risk. Eventually, a nuclear war or environmental catastrophe would doom humanity regardless.

Faster economic growth could initially increase risk, as feared. But it will also help us get past this time of perils more quickly. When people are poor, they can’t focus on much beyond ensuring their own livelihoods. But as people grow richer, they start caring more about things like the environment and protecting against risks to life. And so, as economic growth makes people richer, they will invest more in safety, protecting against existential catastrophes. As technological innovation and our growing wealth has allowed us to conquer past threats to human life like smallpox, so can faster economic growth, in the long run, increase the overall chances of humanity’s survival.

[Aaron Gertler 🔸]

Thank you for producing this! I’d love to see more such summaries of EA research papers, even if the authors don’t go so far as to use LaTeX (though that’s a nice bonus).

[Ben_West🔸]

Thanks for writing this up! It sparked some good discussion here at CEA.

This paper claims that CRRA (γ) is around 1. According to the table in the article (p13 in your summary), if γ=1.1, then we are indifferent between decreasing the discount rate from 2% to 1.87% versus doubling consumption.

I admit that I have a very poor model of how hard it is to decrease someone’s discount rate, but it seems massively easier to decrease people’s discount rates by 13 basis points versus doubling consumption. Are my intuitions really off about that?

If I’m right, it seems like this model pretty clearly argues for spreading longtermism versus economic growth.

[Alex HT]

Great to hear this has been useful!

I think if $\gamma$ is around 1 then yes, spreading longtermism probably looks better than accelerating growth. Though I don’t know how expensive it is to double someone’s consumption in the long-run.

Doubling someone’s consumption by just giving them extra money might cost $30,000 for 50 years=~$0.5million. It seems right to me that there are ways to reduce the discount rate that are much cheaper than half a million dollars for 13 basis points. Eg. some community building probably takes a person’s discount rate from around 2% to around 0% for less than half a million dollars.

I don’t know how much cheaper it might be to double someone’s consumption by increasing growth but I suspect that spreading longtermism still looks better for this value of $\gamma$.

How confident are you that $\gamma$ is around 1? I haven’t looked into it and don’t know how much consensus there is.

[Ben_West🔸]

I just looked at the one paper. I’m not sure if other sources disagree.

Even if $\gamma$ was 2 though, I still feel like it would be cheaper?

It’s hard for me to think about changing individuals, but if I think of governments: there some government department which is responsible for the welfare of some population. We have 2 options:

1. Convince them to update their models to move their discount rate from 2% to 1%, or

2. Convince them to adopt some policy which doubles the consumption of everyone within that population

Surely the first one is easier? If only because it’s at least in principle possible – even if the US government would magically do whatever I said, I don’t know if I could suggest a policy change that would double consumption.

[Alex HT]

Yeah, that seems right to me.

On doubling consumption though, if you can suggest a policy that increases growth consistently, eventually you might cause consumption to be doubled (at some later time consumption under the faster growth will be twice as much as it would have been with the slower growth). Do you mean you don’t think you could suggest a policy change that would increase the growth rate by much?

[Ben_West🔸]

That’s a fair point. I guess the thought experiment could be something like: the department is responsible for some set of people’s welfare 50 years from now. We have to either convince that department to have a lower discount rate 50 years from now, or adopt some measures such that the people born 50 years from now will have doubled the consumption (relative to the counterfactual)?

If that’s right, the discount rate thing still seems easier. It seems hard to double consumption over a 50 year period, though definitely easier than doubling it immediately.