What’s doing the work for you? Do you think the probability of anthropogenic x-risk with our current tech is close to zero? Or do you think that it’s not but that if growth stopped we’d keep working on safety (say developing clean energy, improving relationships between US and China etc.) so that we’d eventually be safe?
I think the first option (low probability of x-risk with current technology) is driving my intuition.
Just to take some reasonable-seeming numbers (since I don’t have numbers of my own): in The Precipice, Toby Ord estimates ~19% chance of existential catastrophe from anthropogenic risks within the next 100 years. If growth stopped now, I would take out unaligned AI and unforeseen/other (although “other” includes things like totalitarian regimes so maybe some of the probability mass should be kept), and would also reduce engineered pandemics (not sure by how much), which would bring the chance down to 0.3% to 4%. (Of course, this is a naive analysis since if growth stopped a bunch of other things would change, etc.)
My intuitions depend a lot on when growth stopped. If growth stopped now I would be less worried, but if it stopped after some dangerous-but-not-growth-promoting technology was invented, I would be more worried.
but what about eg. climate change, nuclear war, biorisk, narrow AI systems being used in really bad ways?
I’m curious what kind of story you have in mind for current narrow AI systems leading to existential catastrophe.
So you think the hazard rate might go from around 20% to around 1%? That’s still far from zero, and with enough centuries with 1% risk we’d expect to go extinct.
I don’t have any specific stories tbh, I haven’t thought about it (and maybe it’s just pretty implausible idk).
Hey, thanks for engaging with this, and sorry for not noticing your original comment for so many months. I agree that in reality the hazard rate at t depends not just on the level of output and safety measures maintained at t but also on “experiments that might go wrong” at t. The model is indeed a simplification in this way.
Just to make sure something’s clear, though (and sorry if this was already clear): Toby’s 20% hazard rate isn’t the current hazard rate; it’s the hazard rate this century, but most of that is due to developments he projects occurring later this century. Say the current (instantaneous) hazard rate is 1% per century; my guess is that most of this consists of (instantaneous) risk imposed by existing stockpiles of nuclear weapons, existing climate instability, and so on, rather than (instantaneous) risk imposed by research currently ongoing. So if stopping growth would lower the hazard rate, it would be a matter of moving from 1% to 0.8% or something, not from 20% to 1%.
I’m just putting numbers to the previous sentence: “Say the current (instantaneous) hazard rate is 1% per century; my guess is that most of this consists of (instantaneous) risk imposed by existing stockpiles of nuclear weapons, existing climate instability, and so on, rather than (instantaneous) risk imposed by research currently ongoing.”
If “most” means “80%” there, then halting growth would lower the hazard rate from 1% to 0.8%.
I think the first option (low probability of x-risk with current technology) is driving my intuition.
Just to take some reasonable-seeming numbers (since I don’t have numbers of my own): in The Precipice, Toby Ord estimates ~19% chance of existential catastrophe from anthropogenic risks within the next 100 years. If growth stopped now, I would take out unaligned AI and unforeseen/other (although “other” includes things like totalitarian regimes so maybe some of the probability mass should be kept), and would also reduce engineered pandemics (not sure by how much), which would bring the chance down to 0.3% to 4%. (Of course, this is a naive analysis since if growth stopped a bunch of other things would change, etc.)
My intuitions depend a lot on when growth stopped. If growth stopped now I would be less worried, but if it stopped after some dangerous-but-not-growth-promoting technology was invented, I would be more worried.
I’m curious what kind of story you have in mind for current narrow AI systems leading to existential catastrophe.
So you think the hazard rate might go from around 20% to around 1%? That’s still far from zero, and with enough centuries with 1% risk we’d expect to go extinct.
I don’t have any specific stories tbh, I haven’t thought about it (and maybe it’s just pretty implausible idk).
I’m not attached to those specific numbers, but I think they are reasonable.
Right, maybe I shouldn’t have said “near zero”. But I still think my basic point (of needing to lower the hazard rate if growth stops) stands.
Hey, thanks for engaging with this, and sorry for not noticing your original comment for so many months. I agree that in reality the hazard rate at t depends not just on the level of output and safety measures maintained at t but also on “experiments that might go wrong” at t. The model is indeed a simplification in this way.
Just to make sure something’s clear, though (and sorry if this was already clear): Toby’s 20% hazard rate isn’t the current hazard rate; it’s the hazard rate this century, but most of that is due to developments he projects occurring later this century. Say the current (instantaneous) hazard rate is 1% per century; my guess is that most of this consists of (instantaneous) risk imposed by existing stockpiles of nuclear weapons, existing climate instability, and so on, rather than (instantaneous) risk imposed by research currently ongoing. So if stopping growth would lower the hazard rate, it would be a matter of moving from 1% to 0.8% or something, not from 20% to 1%.
Can you say how you came up with the “moving from 1% to 0.8%” part? Everything else in your comment makes sense to me.
I’m just putting numbers to the previous sentence: “Say the current (instantaneous) hazard rate is 1% per century; my guess is that most of this consists of (instantaneous) risk imposed by existing stockpiles of nuclear weapons, existing climate instability, and so on, rather than (instantaneous) risk imposed by research currently ongoing.”
If “most” means “80%” there, then halting growth would lower the hazard rate from 1% to 0.8%.