I would be really interested in a post that outlined 1-3 different scenarios for post-AGI x-risk based on increasingly strict assumptions. So the first one would assume that misaligned superintelligent AI would almost instantly emerge from AGI, and describe the x-risks associated with that. Then the assumptions become stricter and stricter, like AGI would only be able to improve itself slowly, we would be able to align it to our goals etc.
I think this could be a valuable post to link people to, as a lot of debates around whether AI poses an x-risk seem to fall on accepting or rejecting potential scenarios, but it’s usually unproductive because everyone has different assumptions about what AI will be capable of.
So with this post, to say that AI x-risk is not tangible, then for each AI development scenario (with increasingly strict assumptions), you have to either:
reject at least one of the listed assumptions (e.g. argue that computer chips are a limit on exponential intelligence increases)
or argue that all proposed existential risks in that scenario are so impossible that even an AI wouldn’t be able to make any of them work.
If you can’t do either of those, you accept AI is an x-risk. If you can, you move on to the next scenario with stricter assumptions. Eventually you find the assumptions you agree with, and have to reject all proposed x-risks in that scenario to say that AI x-risk isn’t real.
The post might also help with planning for different scenarios if it’s more detailed than I’m anticipating.
I would be really interested in a post that outlined 1-3 different scenarios for post-AGI x-risk based on increasingly strict assumptions. So the first one would assume that misaligned superintelligent AI would almost instantly emerge from AGI, and describe the x-risks associated with that. Then the assumptions become stricter and stricter, like AGI would only be able to improve itself slowly, we would be able to align it to our goals etc.
I think this could be a valuable post to link people to, as a lot of debates around whether AI poses an x-risk seem to fall on accepting or rejecting potential scenarios, but it’s usually unproductive because everyone has different assumptions about what AI will be capable of.
So with this post, to say that AI x-risk is not tangible, then for each AI development scenario (with increasingly strict assumptions), you have to either:
reject at least one of the listed assumptions (e.g. argue that computer chips are a limit on exponential intelligence increases)
or argue that all proposed existential risks in that scenario are so impossible that even an AI wouldn’t be able to make any of them work.
If you can’t do either of those, you accept AI is an x-risk. If you can, you move on to the next scenario with stricter assumptions. Eventually you find the assumptions you agree with, and have to reject all proposed x-risks in that scenario to say that AI x-risk isn’t real.
The post might also help with planning for different scenarios if it’s more detailed than I’m anticipating.