Instead it’s saying that it may be more natural to have the object-level conversations about transitions rather than about risk-per-century.
Hmm, I definitely think there’s an object level disagreement about the structure of risk here.
Take the invention of nuclear weapons for example. This was certainly a “transition” in society relevant to existential risk. But it doesn’t make sense to me to analogise it to a risk in putting up a sail. Instead, nuclear weapons are just now a permanent risk to humanity, which goes up or down depending on geopolitical strategy.
I don’t see why future developments wouldn’t work the same way. It seems that since early humanity the state risk has only been increasing further and further as technology develops. I know there are arguments for why it could suddenly drop, but I agree with the linked Thorstadt analysis that this seems unlikely.
Ok, I agree with you that state risk is also an important part of the picture. I basically agree that nuclear risk is better understood as a state risk. I think the majority of AI risk is better understood as a transition risk, which was why I was emphasising that.
I guess at a very high level, I think: either there are accessible arrangements for society at some level of technological advancement which drive risk very low, or there aren’t. If there aren’t, it’s very unlikely that the future will be very large. If there are, then there’s a question of whether the world can reach such a state before an existential catastrophe. If risk now is lower than risk we’re likely to incur on the path to such an arrangement, it can be thought of as a transition risk (whether we manage to bear the increased exposure on the way)
… by analogy, maybe there’s a part of putting up the sail where you’re exposed to being washed overboard by a freak wave, which can be thought of as a state risk which forms part of a transition risk.
If there are accessible arrangements, even if we can’t identify them now, I expect some significant effort to go into searching and steering for them, so a nontrivial chance of reaching one. An argument that we won’t reach such a state seems like it’s either going to need to argue that there are no such states (seems unlikely to me; I think my intuition is informed in part by the existence of error correcting codes), or that it’s vanishingly unlikely that we could reach one that does exist (doesn’t seem impossible to me but I find it hard to see how we could hope to get confidence on this point).
(With apologies, I think this comment is kind of dense. Some better version of it would give the arguments more cleanly.)
I guess at a very high level, I think: either there are accessible arrangements for society at some level of technological advancement which drive risk very low, or there aren’t. If there aren’t, it’s very unlikely that the future will be very large. If there are, then there’s a question of whether the world can reach such a state before an existential catastrophe.
This reasoning seems off. Why would it have to drive thing to very low risk, rather than to a low but significant level of risk, like we have today with nuclear weapons? Why would it be impossible to find arrangements that keep the level of state risk at like 1%?
AI risk thinking seems to have a lot of “all or nothing” reasoning that seems completely unjustified to me.
I’m worried we’re talking past each other here. We totally might find arrangements that keep the state risk at like 1% -- and in that case then (as Thorstad points out) we expect not to have a very large future (though it could still be decently large compared to the world today).
But if your axiology is (in part) totalist, you’ll care a lot whether we actually get to very large futures. I’m saying (agreeing with Thorstad) that these are dependent on finding some arrangement which drives risk very low. Then I’m saying (disagreeing with Thorstad?) that the decision-relevant question is more like “have we got any chance of getting to such a state?” rather than “are we likely to reach such a state?”
I think the talk of transition risks and sail metaphors aren’t actually that relevant to your argument here? Wouldn’t a gradual and continuous decrease to state risk, like Kuznets curve shown in Thorstadt’s paper here, have the same effect?
Yeah it totally has the same effect. It can just be less natural to analyse, if you think the risk will (or might) decrease a lot following some transition (which is also when the risk will mostly be incurred), but you’re less confident about when the transition will occur.
But it doesn’t make sense to me to analogise it to a risk in putting up a sail.
I think this depends on the timeframe. Over a longer one, looking into the estimated destroyable area by nuclear weapons, nuclear risk looks like a transition risk (see graph below). In addition, I think the nuclear extinction risk has decreased even more than the destroyable area, since I believe greater wealth has made society more resilient to the effects of nuclear war and nuclear winter. For reference, I estimated the current annual nuclear extinction risk is 5.93*10^-12.
Hmm, I definitely think there’s an object level disagreement about the structure of risk here.
Take the invention of nuclear weapons for example. This was certainly a “transition” in society relevant to existential risk. But it doesn’t make sense to me to analogise it to a risk in putting up a sail. Instead, nuclear weapons are just now a permanent risk to humanity, which goes up or down depending on geopolitical strategy.
I don’t see why future developments wouldn’t work the same way. It seems that since early humanity the state risk has only been increasing further and further as technology develops. I know there are arguments for why it could suddenly drop, but I agree with the linked Thorstadt analysis that this seems unlikely.
Ok, I agree with you that state risk is also an important part of the picture. I basically agree that nuclear risk is better understood as a state risk. I think the majority of AI risk is better understood as a transition risk, which was why I was emphasising that.
I guess at a very high level, I think: either there are accessible arrangements for society at some level of technological advancement which drive risk very low, or there aren’t. If there aren’t, it’s very unlikely that the future will be very large. If there are, then there’s a question of whether the world can reach such a state before an existential catastrophe. If risk now is lower than risk we’re likely to incur on the path to such an arrangement, it can be thought of as a transition risk (whether we manage to bear the increased exposure on the way) … by analogy, maybe there’s a part of putting up the sail where you’re exposed to being washed overboard by a freak wave, which can be thought of as a state risk which forms part of a transition risk.
If there are accessible arrangements, even if we can’t identify them now, I expect some significant effort to go into searching and steering for them, so a nontrivial chance of reaching one. An argument that we won’t reach such a state seems like it’s either going to need to argue that there are no such states (seems unlikely to me; I think my intuition is informed in part by the existence of error correcting codes), or that it’s vanishingly unlikely that we could reach one that does exist (doesn’t seem impossible to me but I find it hard to see how we could hope to get confidence on this point).
(With apologies, I think this comment is kind of dense. Some better version of it would give the arguments more cleanly.)
This reasoning seems off. Why would it have to drive thing to very low risk, rather than to a low but significant level of risk, like we have today with nuclear weapons? Why would it be impossible to find arrangements that keep the level of state risk at like 1%?
AI risk thinking seems to have a lot of “all or nothing” reasoning that seems completely unjustified to me.
I’m worried we’re talking past each other here. We totally might find arrangements that keep the state risk at like 1% -- and in that case then (as Thorstad points out) we expect not to have a very large future (though it could still be decently large compared to the world today).
But if your axiology is (in part) totalist, you’ll care a lot whether we actually get to very large futures. I’m saying (agreeing with Thorstad) that these are dependent on finding some arrangement which drives risk very low. Then I’m saying (disagreeing with Thorstad?) that the decision-relevant question is more like “have we got any chance of getting to such a state?” rather than “are we likely to reach such a state?”
Okay, that makes a lot more sense, thank you.
I think the talk of transition risks and sail metaphors aren’t actually that relevant to your argument here? Wouldn’t a gradual and continuous decrease to state risk, like Kuznets curve shown in Thorstadt’s paper here, have the same effect?
Yeah it totally has the same effect. It can just be less natural to analyse, if you think the risk will (or might) decrease a lot following some transition (which is also when the risk will mostly be incurred), but you’re less confident about when the transition will occur.
Nice discussion, Owen and titotal!
I think this depends on the timeframe. Over a longer one, looking into the estimated destroyable area by nuclear weapons, nuclear risk looks like a transition risk (see graph below). In addition, I think the nuclear extinction risk has decreased even more than the destroyable area, since I believe greater wealth has made society more resilient to the effects of nuclear war and nuclear winter. For reference, I estimated the current annual nuclear extinction risk is 5.93*10^-12.