I think this model is kind of misleading, and that the original astronomical waste argument is still strong. It seems to me that a ton of the work in this model is being done by the assumption of constant risk, even in post-peril worlds. I think this is pretty strange. Here are some brief comments:
If you’re talking about the probability of a universal quantifier, such as “for all humans x, x will die”, then it seems really weird to say that this remains constant, even when the thing you’re quantifying over grows larger.
For instance, it seems clear that if there were only 100 humans, the probability of x-risk would be much higher than if there were 10^6 humans. So it seems like if there are 10^20 humans, it should be harder to cause extinction than 10^10 humans.
Assuming constant risk has the implication that human extinction is guaranteed to happen at some point in the future, which puts sharp bounds on the goodness of existential risk reduction.
It’s not that hard to get exponentially decreasing probability on universal quantifiers if you assume independence in survival amongst some “unit” of humanity. In computing applications, it’s not that hard to drive down the probability of error exponentially in the resources allocated, because each unit of resource can ~halve the probability of error. Naively, each human doesn’t want to die, so there are # humans rolls for surviving/solving x-risk.
It seems like the probability of x-risk ought to be inversely proportional to the current estimated amount of value at stake. This seems to follow if you assume that civilization acts as a “value maximizer” and it’s not that hard to reduce x-risk. Haven’t worked it out, so wouldn’t be surprised if I was making some basic error here.
Generally, it seems like most of the risk is going to come from worlds where the chance of extinction isn’t actually a universal quantifier, and there’s some correlation amongst seemingly independent roles for survival. In particularly bad cases, humans go extinct if there exists someone that wants to destroy the universe, so we actually see an extremely rapid increasing probability of extinction as we get more humans. These worlds would require extremely strong coordination and governance solutions.
These worlds are also slightly physically impossible because parts of humanity will rapidly become causally isolated from each other. I don’t know enough cosmology to have an intuition for which way the functional form will ultimately go.
Generally, it seems like the naive view is that as humans get richer/smarter, they’ll allocate more and more resources towards not dying. At equilibrium, it seems reasonable to first-order-assume we’ll drive existential risk down until the marginal cost equals the marginal benefit, so the key question is how this equilibrium behaves. It seems like my guess is that it will depend heavily on the total amount of value available in the future, determined by physical constraints (and potentially more galaxy-brained considerations).
This view seems to allow you to recover more the more naive astronomical waste perspective.
This makes me feel like the model makes kind of strong assumptions about the amount it will ultimately cost to drive down existential risk. E.g. you seem to imply that rl = 0.0001 is small, but an independent chance that large each century suggests that the probability humanity survives for ~10^10 years is ~0. This feels quite absurd to me.
The sentence: “Note that for the Pessimist, this is a reduction of 200,000%”, but humans routinely reduce the probabilities of failures by more than 200,000% via engineering efforts and produce highly complex and artifacts like computers, airplanes, rockets, satellites, etc. It feels like you should naively expect “breaking” human civilization to be harder than breaking an airplane, especially when civilization is actively trying to ensure that it doesn’t go extinct.
Also, you seem to assume each century has some constant value v eventually, which seems reasonable to me, but the implication “Warming (slightly) on short-termist cause areas” relies on an assumption that the current century is close to value v, when it seems like even pretty naive bounds (e.g. percent of sun’s energy), suggest that the current century is not even within a factor of 10^9 of the long-run value-per-century humanity could reach.
Assuming that value grows quadratically seems also quite weird, because of analysis like eternity in 6 hours, which seems to imply that a resource-maximizing civilization will undergo a period of incredibly rapid expansion to achieve per-century rates of value much higher than the current century, and then have nowhere else to go. A better model from my perspective is logistic growth of value, with the upper bound given by some weak proxy like “suppose that value is linear in the amount of energy a civilization uses, then take the total amount of value in the year 2020”, with the ultimate unit being “value in 2020″. This would produce much higher numbers, and give a more intuitive sense of “astronomical waste.”
I like the process of proposing concrete models for things as a substrate for disagreement, and I appreciate that you wrote this. It feels much better to articulate objections like “I don’t think this particular parameter should be constant in your model” than to have abstract arguments. I also like how it’s now more clear that if you do believe that risk in post-peril worlds is constant, then the argument for longtermism is much weaker (although I think still quite strong because of my comments about v).
I agree that it’s important to look in detail at models to see what is going on. We can’t settle debates about value from the armchair.
I’ll try to type up some thoughts in a few edits, since I want to make sure to think about what to say.
Population growth: It’s definitely possible to decompose the components of the Ord/Adamczewski/Thorstad model into their macroeconomic determinants (population, capital, technology, etc.). Economists like to do this. For example, Leopold does this.
It can also be helpful to decompose the model in other ways. Sociologists might want to split up things like v and r into a more fine-grained model of their social/political determinants, for example.
I tend to think that population growth is not going to be enough to substantially reverse the conclusions of the model, although I’d be really interested to see if you wanted to work through the conclusions here. For example, my impression with Leopold’s model is that if you literally cut population growth out of the model, the long-term qualitative behavior and conclusions wouldn’t change that much: it would still be the case that the driving assumptions are (a) research in safety technologies exponentially decreases risk, (b) the rate at which safety research exponentially decreases risk is “fast”, in the sense specified by Leopold’s main theorem.
(I didn’t check this explicitly. I’d be curious if someone wanted to verify this).
I also think it is important to think about some ways in which population growth can be bad for the value of existential risk mitigation. For example, the economist Maya Eden is looking at non-Malthusian models of population growth on which there is room for the human population to grow quite substantially for quite some time. She thinks that these models can often make it very important to do things that will kickstart population growth now, like staying at home to raise children or investing in economic growth. Insofar as investments and research into existential risk mitigation take us away from those activities, it might turn out that existential risk mitigation is relatively less valuable—quite substantially so, on some of Maya’s models. We just managed to talk Maya into contributing a paper to an anthology on longtermism that GPI is putting together, so I hope that some of these models will be in print within 1-2 years.
In the meantime, maybe some people would like to play around with population in Leopold’s model and report back on what happens?
First edit: Constant risk and guaranteed extinction: A nitpick: constant risk doesn’t assume guaranteed extinction, but rather extinction with probability 1. Probability 0 events are maximally unlikely, but not impossible. (Example: for every point on 3-D space there’s probability 0 that a dart lands centered around that point. But it does land centered around one).
More to the point, it’s not quite the assumption of constant risk that gives probability 1 to (eventual) extinction. To get probability <1 of eventual extinction, in most models you something stronger like (a) indefinitely sustained decay in extinction risk of exponential or higher speed, and (b) a “fast” rate of decay in your chosen functional form. This is pretty clear, for example, in Leopold’s model.
I think that claims like (a) and (b) can definitely be true, but that it is important to think carefully about why they are true. For example, Leopold generates (a) explicitly through a specified mechanism: producing safety technologies exponentially decreases risk. I’ve argued that (a) might not be true. Leopold doesn’t generate (b). Leopold just put (b) as part of his conclusion: if (b) isn’t true, we’re toast.
Second edit: Allocating resources towards safety: I think it would be very valuable to do more explicit modelling about the effect of resources on safety.
Leopold’s model discusses one way to do this. If you assume that safety research drives down risk exponentially at a reasonably quick rate, then we can just spend a lot more on safety as we get richer. I discuss this model in Section 5.
That’s not the only mechanism you could propose for using resources to provide safety. Did you have another mechanism in mind?
I think it would be super-useful to have an alternative to Leopold’s model. More models are (almost) always better.
In the spirit of our shared belief in the importance of models, I would encourage you to write down your model of choice explicitly and study it a bit. I think it could be very valuable for readers to work through the kinds of modelling assumptions needed to get a Time of Perils Hypothesis of the needed form, and why we might believe/disbelieve these assumptions.
Third edit: Constant value: It would definitely be bad to assume that all centuries have constant value. I think the exact word I used for that assumption was “bonkers”! My intention with the value growth models in Section 3 was to show that the conclusions are robust to some models of value growth (linear, quadratic) and in another comment I’ve sketched some calculations to suggest that my the model’s conclusions might even be robust to cubic, quartic (!) or maybe even quintic (!!) growth.
I definitely didn’t consider logistic growth explicitly. It could be quite interesting to think about the case of logistic growth. Do you want to write this up and see what happens?
I don’t think that the warming on short-termist cause areas should rely on an assumption of constant v. This is the case for a number of reasons. (a) Anything we can do to decrease the value of longtermist causes translates into a warming in our (relative) attitudes towards short-termist cause areas; (b) Short-termist causes can often have quite good longtermist implications, as for example in the Maya Eden models I mentioned earlier, or in some of Tyler Cowen’s work on economic growth. (c) As we saw in the model, probabilities matter. Even if you think there is an astronomical amount of value out there to be gained, if you think we’re sufficiently unlikely to gain it then short-termist causes can look relatively more attractive.
A concluding thought: I do very much like your emphasis on concrete models as a substitute for disgareement. Perhaps I could interest you in working out some of the models you raised in more detail, for example by modeling the effects of population growth using a standard macroeconomic model (Leopold uses a Solow-style model, but you could use another, and maybe go more endogenous)? I’d be curious to see what you find!
Another risk is replacement by aliens (life or AI), either they get to where we want to expand to first and we’re prevented from generating much value there or we have to leave regions we previously occupied, or even have it all taken over. If they are expansive like “grabby aliens”, we might not be left with much or anything. We might expect aliens from multiple directions effectively boxing us in a bounded region of space.
A nonzero lower bound for the existential risk rate would be reasonable on this account, although I still wouldn’t assign full weight to this model, and might still assign some weight to decreasing risk models with risks approaching 0. Maybe we’re so far from aliens that we will practically never encounter them.
On the other hand, there seem to be some pretty hard limits on the value we can generate set by the accelerating expansion of the universe, but this is probably better captured with a bound on the number of terms in the sum and not an existential risk rate. This would prevent the sum from becoming infinite with high probability, although we might want to allow exotic possibilities of infinities.
I think this model is kind of misleading, and that the original astronomical waste argument is still strong. It seems to me that a ton of the work in this model is being done by the assumption of constant risk, even in post-peril worlds. I think this is pretty strange. Here are some brief comments:
If you’re talking about the probability of a universal quantifier, such as “for all humans x, x will die”, then it seems really weird to say that this remains constant, even when the thing you’re quantifying over grows larger.
For instance, it seems clear that if there were only 100 humans, the probability of x-risk would be much higher than if there were 10^6 humans. So it seems like if there are 10^20 humans, it should be harder to cause extinction than 10^10 humans.
Assuming constant risk has the implication that human extinction is guaranteed to happen at some point in the future, which puts sharp bounds on the goodness of existential risk reduction.
It’s not that hard to get exponentially decreasing probability on universal quantifiers if you assume independence in survival amongst some “unit” of humanity. In computing applications, it’s not that hard to drive down the probability of error exponentially in the resources allocated, because each unit of resource can ~halve the probability of error. Naively, each human doesn’t want to die, so there are # humans rolls for surviving/solving x-risk.
It seems like the probability of x-risk ought to be inversely proportional to the current estimated amount of value at stake. This seems to follow if you assume that civilization acts as a “value maximizer” and it’s not that hard to reduce x-risk. Haven’t worked it out, so wouldn’t be surprised if I was making some basic error here.
Generally, it seems like most of the risk is going to come from worlds where the chance of extinction isn’t actually a universal quantifier, and there’s some correlation amongst seemingly independent roles for survival. In particularly bad cases, humans go extinct if there exists someone that wants to destroy the universe, so we actually see an extremely rapid increasing probability of extinction as we get more humans. These worlds would require extremely strong coordination and governance solutions.
These worlds are also slightly physically impossible because parts of humanity will rapidly become causally isolated from each other. I don’t know enough cosmology to have an intuition for which way the functional form will ultimately go.
Generally, it seems like the naive view is that as humans get richer/smarter, they’ll allocate more and more resources towards not dying. At equilibrium, it seems reasonable to first-order-assume we’ll drive existential risk down until the marginal cost equals the marginal benefit, so the key question is how this equilibrium behaves. It seems like my guess is that it will depend heavily on the total amount of value available in the future, determined by physical constraints (and potentially more galaxy-brained considerations).
This view seems to allow you to recover more the more naive astronomical waste perspective.
This makes me feel like the model makes kind of strong assumptions about the amount it will ultimately cost to drive down existential risk. E.g. you seem to imply that rl = 0.0001 is small, but an independent chance that large each century suggests that the probability humanity survives for ~10^10 years is ~0. This feels quite absurd to me.
The sentence: “Note that for the Pessimist, this is a reduction of 200,000%”, but humans routinely reduce the probabilities of failures by more than 200,000% via engineering efforts and produce highly complex and artifacts like computers, airplanes, rockets, satellites, etc. It feels like you should naively expect “breaking” human civilization to be harder than breaking an airplane, especially when civilization is actively trying to ensure that it doesn’t go extinct.
Also, you seem to assume each century has some constant value v eventually, which seems reasonable to me, but the implication “Warming (slightly) on short-termist cause areas” relies on an assumption that the current century is close to value v, when it seems like even pretty naive bounds (e.g. percent of sun’s energy), suggest that the current century is not even within a factor of 10^9 of the long-run value-per-century humanity could reach.
Assuming that value grows quadratically seems also quite weird, because of analysis like eternity in 6 hours, which seems to imply that a resource-maximizing civilization will undergo a period of incredibly rapid expansion to achieve per-century rates of value much higher than the current century, and then have nowhere else to go. A better model from my perspective is logistic growth of value, with the upper bound given by some weak proxy like “suppose that value is linear in the amount of energy a civilization uses, then take the total amount of value in the year 2020”, with the ultimate unit being “value in 2020″. This would produce much higher numbers, and give a more intuitive sense of “astronomical waste.”
I like the process of proposing concrete models for things as a substrate for disagreement, and I appreciate that you wrote this. It feels much better to articulate objections like “I don’t think this particular parameter should be constant in your model” than to have abstract arguments. I also like how it’s now more clear that if you do believe that risk in post-peril worlds is constant, then the argument for longtermism is much weaker (although I think still quite strong because of my comments about v).
Thanks Mark! This is extremely helpful.
I agree that it’s important to look in detail at models to see what is going on. We can’t settle debates about value from the armchair.
I’ll try to type up some thoughts in a few edits, since I want to make sure to think about what to say.
Population growth: It’s definitely possible to decompose the components of the Ord/Adamczewski/Thorstad model into their macroeconomic determinants (population, capital, technology, etc.). Economists like to do this. For example, Leopold does this.
It can also be helpful to decompose the model in other ways. Sociologists might want to split up things like v and r into a more fine-grained model of their social/political determinants, for example.
I tend to think that population growth is not going to be enough to substantially reverse the conclusions of the model, although I’d be really interested to see if you wanted to work through the conclusions here. For example, my impression with Leopold’s model is that if you literally cut population growth out of the model, the long-term qualitative behavior and conclusions wouldn’t change that much: it would still be the case that the driving assumptions are (a) research in safety technologies exponentially decreases risk, (b) the rate at which safety research exponentially decreases risk is “fast”, in the sense specified by Leopold’s main theorem.
(I didn’t check this explicitly. I’d be curious if someone wanted to verify this).
I also think it is important to think about some ways in which population growth can be bad for the value of existential risk mitigation. For example, the economist Maya Eden is looking at non-Malthusian models of population growth on which there is room for the human population to grow quite substantially for quite some time. She thinks that these models can often make it very important to do things that will kickstart population growth now, like staying at home to raise children or investing in economic growth. Insofar as investments and research into existential risk mitigation take us away from those activities, it might turn out that existential risk mitigation is relatively less valuable—quite substantially so, on some of Maya’s models. We just managed to talk Maya into contributing a paper to an anthology on longtermism that GPI is putting together, so I hope that some of these models will be in print within 1-2 years.
In the meantime, maybe some people would like to play around with population in Leopold’s model and report back on what happens?
First edit: Constant risk and guaranteed extinction: A nitpick: constant risk doesn’t assume guaranteed extinction, but rather extinction with probability 1. Probability 0 events are maximally unlikely, but not impossible. (Example: for every point on 3-D space there’s probability 0 that a dart lands centered around that point. But it does land centered around one).
More to the point, it’s not quite the assumption of constant risk that gives probability 1 to (eventual) extinction. To get probability <1 of eventual extinction, in most models you something stronger like (a) indefinitely sustained decay in extinction risk of exponential or higher speed, and (b) a “fast” rate of decay in your chosen functional form. This is pretty clear, for example, in Leopold’s model.
I think that claims like (a) and (b) can definitely be true, but that it is important to think carefully about why they are true. For example, Leopold generates (a) explicitly through a specified mechanism: producing safety technologies exponentially decreases risk. I’ve argued that (a) might not be true. Leopold doesn’t generate (b). Leopold just put (b) as part of his conclusion: if (b) isn’t true, we’re toast.
Second edit: Allocating resources towards safety: I think it would be very valuable to do more explicit modelling about the effect of resources on safety.
Leopold’s model discusses one way to do this. If you assume that safety research drives down risk exponentially at a reasonably quick rate, then we can just spend a lot more on safety as we get richer. I discuss this model in Section 5.
That’s not the only mechanism you could propose for using resources to provide safety. Did you have another mechanism in mind?
I think it would be super-useful to have an alternative to Leopold’s model. More models are (almost) always better.
In the spirit of our shared belief in the importance of models, I would encourage you to write down your model of choice explicitly and study it a bit. I think it could be very valuable for readers to work through the kinds of modelling assumptions needed to get a Time of Perils Hypothesis of the needed form, and why we might believe/disbelieve these assumptions.
Third edit: Constant value: It would definitely be bad to assume that all centuries have constant value. I think the exact word I used for that assumption was “bonkers”! My intention with the value growth models in Section 3 was to show that the conclusions are robust to some models of value growth (linear, quadratic) and in another comment I’ve sketched some calculations to suggest that my the model’s conclusions might even be robust to cubic, quartic (!) or maybe even quintic (!!) growth.
I definitely didn’t consider logistic growth explicitly. It could be quite interesting to think about the case of logistic growth. Do you want to write this up and see what happens?
I don’t think that the warming on short-termist cause areas should rely on an assumption of constant v. This is the case for a number of reasons. (a) Anything we can do to decrease the value of longtermist causes translates into a warming in our (relative) attitudes towards short-termist cause areas; (b) Short-termist causes can often have quite good longtermist implications, as for example in the Maya Eden models I mentioned earlier, or in some of Tyler Cowen’s work on economic growth. (c) As we saw in the model, probabilities matter. Even if you think there is an astronomical amount of value out there to be gained, if you think we’re sufficiently unlikely to gain it then short-termist causes can look relatively more attractive.
A concluding thought: I do very much like your emphasis on concrete models as a substitute for disgareement. Perhaps I could interest you in working out some of the models you raised in more detail, for example by modeling the effects of population growth using a standard macroeconomic model (Leopold uses a Solow-style model, but you could use another, and maybe go more endogenous)? I’d be curious to see what you find!
Another risk is replacement by aliens (life or AI), either they get to where we want to expand to first and we’re prevented from generating much value there or we have to leave regions we previously occupied, or even have it all taken over. If they are expansive like “grabby aliens”, we might not be left with much or anything. We might expect aliens from multiple directions effectively boxing us in a bounded region of space.
A nonzero lower bound for the existential risk rate would be reasonable on this account, although I still wouldn’t assign full weight to this model, and might still assign some weight to decreasing risk models with risks approaching 0. Maybe we’re so far from aliens that we will practically never encounter them.
On the other hand, there seem to be some pretty hard limits on the value we can generate set by the accelerating expansion of the universe, but this is probably better captured with a bound on the number of terms in the sum and not an existential risk rate. This would prevent the sum from becoming infinite with high probability, although we might want to allow exotic possibilities of infinities.