Thank you for taking the time to write this—I think it is a clear and concise entry point into the AGI ruin arguments.
I want to voice an objection / point out an omission to point 2: I agree that any plan towards a sufficiently complicated goal will include “acquire resources” as a sub-goal, and that “getting rid of all humans” might be a by-product of some ways to achieve this sub-goal. I’m also willing to grant that if all we now about the plan is that it achieves the end (sufficiently complicated) goal, it is likely that the plan might lead to the destruction of all humans.
However I don’t see why we can’t infer more about the plans. Specifically I think an ASI plan for a sufficiently complicated goal should be 1) feasible and 2) efficient (at least in some sense). If the ASI doesn’t believe that it can overpower humanity, then it’s plans will not include overpowering humanity. Even more, if the ASI ascribes a high enough cost to overpowering humanity, it would instead opt to acquire resources in another way.
It seems that for point 2 to hold you must think that an ASI can overpower humanity with 1) close to a 100% certainty and 2) at negligible cost to the ASI. However I don’t think this is (explicitly) argued for in the article. Or maybe I’m missing something?
Thank you for taking the time to write this—I think it is a clear and concise entry point into the AGI ruin arguments.
I want to voice an objection / point out an omission to point 2: I agree that any plan towards a sufficiently complicated goal will include “acquire resources” as a sub-goal, and that “getting rid of all humans” might be a by-product of some ways to achieve this sub-goal. I’m also willing to grant that if all we now about the plan is that it achieves the end (sufficiently complicated) goal, it is likely that the plan might lead to the destruction of all humans.
However I don’t see why we can’t infer more about the plans. Specifically I think an ASI plan for a sufficiently complicated goal should be 1) feasible and 2) efficient (at least in some sense). If the ASI doesn’t believe that it can overpower humanity, then it’s plans will not include overpowering humanity. Even more, if the ASI ascribes a high enough cost to overpowering humanity, it would instead opt to acquire resources in another way.
It seems that for point 2 to hold you must think that an ASI can overpower humanity with 1) close to a 100% certainty and 2) at negligible cost to the ASI. However I don’t think this is (explicitly) argued for in the article. Or maybe I’m missing something?