I agree with your post in principle, we should take currently unknown, non-human moral agents into the calculation of X-risks.
On the other hand, I personally think leaving behind an AGI (which, afterall, is still an “agent” influenced by our thoughts and values and carries it on in some manner) is a preferable end game for human civilisation compared to a lot of other scenarios, even if the impact of an AGI catastrophe is probably going to span the entire galaxy and beyond.
Grey goo and other “AS-catastrophes” are definitely very bad.
From “worst” to “less bad”, I think the scenarios would line up something like this:
2: High velocity (relativistic) grey goo with no AGI. Potentially obliterates our entire light-cone, although advanced ailen civilisations might survive.
3: Low velocity grey goo with no AGI, sterlises the Earth with ease and potentially spreads to other solar systems or the entire galaxy but probably not beyond (the intergalatic travel time would probably be too long for the goo to maintain their function). Technological ailen civilisations might survive.
4: End of all life on Earth from other disasters.
5: AGI catastrophe with spill over in our light cone. I think an AGI’s encounter with intelligent alien life is not guaranteed to follow the same calculus as its relationship with humans, so even if an AGI destroys humanity, it is not necessarily going to destroy (or even be hostile to) some ailen civilisation it encounters.
For a world without humans, I am a bit uncertain about whether the Earth has enough “time left” (about ~500 million years before the sun’s increasing luminosity makes the Earth significantly less habitable) for another intelligent species to emerge after a major extinction event (say, large mammals took the same hit as dinosaurs) that included humanity. And whether the Earth would have enough accessible fossil fuel for them to develop technological civilisation.
I agree with your post in principle, we should take currently unknown, non-human moral agents into the calculation of X-risks.
On the other hand, I personally think leaving behind an AGI (which, afterall, is still an “agent” influenced by our thoughts and values and carries it on in some manner) is a preferable end game for human civilisation compared to a lot of other scenarios, even if the impact of an AGI catastrophe is probably going to span the entire galaxy and beyond.
Grey goo and other “AS-catastrophes” are definitely very bad.
From “worst” to “less bad”, I think the scenarios would line up something like this:
1: False vacuum decay, obliterates our light-cone.
2: High velocity (relativistic) grey goo with no AGI. Potentially obliterates our entire light-cone, although advanced ailen civilisations might survive.
3: Low velocity grey goo with no AGI, sterlises the Earth with ease and potentially spreads to other solar systems or the entire galaxy but probably not beyond (the intergalatic travel time would probably be too long for the goo to maintain their function). Technological ailen civilisations might survive.
4: End of all life on Earth from other disasters.
5: AGI catastrophe with spill over in our light cone. I think an AGI’s encounter with intelligent alien life is not guaranteed to follow the same calculus as its relationship with humans, so even if an AGI destroys humanity, it is not necessarily going to destroy (or even be hostile to) some ailen civilisation it encounters.
For a world without humans, I am a bit uncertain about whether the Earth has enough “time left” (about ~500 million years before the sun’s increasing luminosity makes the Earth significantly less habitable) for another intelligent species to emerge after a major extinction event (say, large mammals took the same hit as dinosaurs) that included humanity. And whether the Earth would have enough accessible fossil fuel for them to develop technological civilisation.