Do you think that there are relatively probably existential catastrophes from rogue AI that would allow for alien colonization of Earth?
Yes, that’s what I think.
First, consider the ‘classic’ Bostrom/Yudkowsky catastrophe scenario in which a single superintelligent agent with misaligned goals kills everyone and then, for instrumental reasons, expands into the universe. I agree that this would be a significant obstacle for alien civilization (though not totally impossible—e.g. there’s some, albeit perhaps tiny, chance that an expanding alien civilization could be a more powerful adversary, or there could be some kind of trade, or …).
However, I don’t think we can be highly confident that this is how an existential catastrophe due to AI would look like. Cf. Christiano’s What failure looks like,Drexler’s Reframing Superintelligence, and also recent posts on AI risk arguments/scenarios by Tom Sittler and Richard Ngo. On some of the scenarios discussed there, I think it’s hard to see whether they’d result in an obstacle to alien civilizations or not.
More broadly, I’d be wary to assign very high confidence to any feature of a post-AI catastrophe world. AI that could cause an existential catastrophe is a technology we don’t currently possess and cannot anticipate in all its details—therefore, I think it’s quite likely that an actual catastrophe based on such AI would in at least some ways have unanticipated properties, i.e., would at least not completely fall into any category of catastrophe we currently anticipate. Relatively robust high-level considerations such as Omohundro’s convergent instrumental goal argument can give us good reasons to nevertheless assign significant credence to some properties (e.g., a superintelligent AI agent seems likely to acquire resources), but I don’t think they suffice for >90% credence in anything.
Yes, that’s what I think.
First, consider the ‘classic’ Bostrom/Yudkowsky catastrophe scenario in which a single superintelligent agent with misaligned goals kills everyone and then, for instrumental reasons, expands into the universe. I agree that this would be a significant obstacle for alien civilization (though not totally impossible—e.g. there’s some, albeit perhaps tiny, chance that an expanding alien civilization could be a more powerful adversary, or there could be some kind of trade, or …).
However, I don’t think we can be highly confident that this is how an existential catastrophe due to AI would look like. Cf. Christiano’s What failure looks like, Drexler’s Reframing Superintelligence, and also recent posts on AI risk arguments/scenarios by Tom Sittler and Richard Ngo. On some of the scenarios discussed there, I think it’s hard to see whether they’d result in an obstacle to alien civilizations or not.
More broadly, I’d be wary to assign very high confidence to any feature of a post-AI catastrophe world. AI that could cause an existential catastrophe is a technology we don’t currently possess and cannot anticipate in all its details—therefore, I think it’s quite likely that an actual catastrophe based on such AI would in at least some ways have unanticipated properties, i.e., would at least not completely fall into any category of catastrophe we currently anticipate. Relatively robust high-level considerations such as Omohundro’s convergent instrumental goal argument can give us good reasons to nevertheless assign significant credence to some properties (e.g., a superintelligent AI agent seems likely to acquire resources), but I don’t think they suffice for >90% credence in anything.
That sounds reasonable.