The idea is that by speeding through you increase risk initially, but the total risk is lower—i.e. smaller area under the grey curve here:
I think this probably breaks down if the peak is high enough though (here I’m thinking of AGI x-risk). Aschenbrenner gives the example of:
On the other extreme, humanity is extremely fragile. No matter how high a fraction of our resources we dedicate to safety, we cannot prevent an unrecoverable catastrophe. ..there is nothing we can do regardless. An existential catastrophe is inevitable, and it is impossible for us to survive to reach a grand future.
And argues that
even if there is some probability we do live in this world, to maximize the moral value of the future, we should act as if we live in the other scenarios where a long and flourishing future is possible.
I’m not sure if this applies if there is some possibility of “pulling the curve sideways” to flatten it—i.e. increase the fraction of resources spent on safety whilst keeping consumption (or growth) constant. This seems to be what those concerned with x-risk are doing for the most part (rather than trying to slow down growth).
That graph represents the comparison of increased growth (gray line) to some baseline (black).
Instead, I’m talking about the case where the dot is at the far left side of the graph: we’re currently at a low level of risk, and continuing at the baseline growth rate means ‘climbing the risk mountain’ before we get to the other side with lower risk. If the peak is very high (relative to the dot), then it’s not clear that continued (or accelerated) growth is optimal; stagnation might be better.
The idea is that by speeding through you increase risk initially, but the total risk is lower—i.e. smaller area under the grey curve here:
I think this probably breaks down if the peak is high enough though (here I’m thinking of AGI x-risk). Aschenbrenner gives the example of:
And argues that
I’m not sure if this applies if there is some possibility of “pulling the curve sideways” to flatten it—i.e. increase the fraction of resources spent on safety whilst keeping consumption (or growth) constant. This seems to be what those concerned with x-risk are doing for the most part (rather than trying to slow down growth).
That graph represents the comparison of increased growth (gray line) to some baseline (black).
Instead, I’m talking about the case where the dot is at the far left side of the graph: we’re currently at a low level of risk, and continuing at the baseline growth rate means ‘climbing the risk mountain’ before we get to the other side with lower risk. If the peak is very high (relative to the dot), then it’s not clear that continued (or accelerated) growth is optimal; stagnation might be better.