did you end up doing this? If it’s still upcoming, I’d also be interested
demirev
Karma: 8
Thank you for taking the time to write this—I think it is a clear and concise entry point into the AGI ruin arguments.
I want to voice an objection / point out an omission to point 2: I agree that any plan towards a sufficiently complicated goal will include “acquire resources” as a sub-goal, and that “getting rid of all humans” might be a by-product of some ways to achieve this sub-goal. I’m also willing to grant that if all we now about the plan is that it achieves the end (sufficiently complicated) goal, it is likely that the plan might lead to the destruction of all humans.
However I don’t see why we can’t infer more about the plans. Specifically I think an ASI plan for a sufficiently complicated goal should be 1) feasible and 2) efficient (at least in some sense). If the ASI doesn’t believe that it can overpower humanity, then it’s plans will not include overpowering humanity. Even more, if the ASI ascribes a high enough cost to overpowering humanity, it would instead opt to acquire resources in another way.
It seems that for point 2 to hold you must think that an ASI can overpower humanity with 1) close to a 100% certainty and 2) at negligible cost to the ASI. However I don’t think this is (explicitly) argued for in the article. Or maybe I’m missing something?
I also think this was a not-so-good and somewhat misleading analogy—the association between Novik and Caroline in the example is strictly one-way (Caroline likes Novik, Novik has no idea who Caroline is), whereas the association between FTX and EA is clearly two-way (e.g. various EA orgs endorsing and promoting SBF, SBF choosing to earn-to-give after talking with 80k etc).
Why would this notion be obvious? Seems just as likely, if not more, that in a more complicated domain your ability to achieve your goals hits diminishing returns w.r.t intelligence more quickly than in simple environments. If that is the case the ‘disadvantage of weaker minds’ will be smaller, not larger.
(I don’t find the presented analogy very convincing, especially in the ‘complicated domain’ example of modern Russia overpowering medieval Earth, since it is only obvious if we imagine the entire economy/infrastructure/resources of Russia. Consider instead the thought experiment of say 200 people armed with 21st century knowledge but nothing else, and it seems not so obvious they’ll be easily able to ‘roll-over 11th-century Earth’).