Closely related, and also important, is the question of “which world gets precluded”. Different possibilities include:
By reducing extinction risk from a (hypothetical) scenario in which Earth explodes and falls into pieces, we preclude a world in which there’s no life (and therefore no powerful agent) on what previously was Earth.
By reducing extinction risk from pandemics, we preclude a world in which there’s no human on Earth, but possibly other intelligent species that have evolved to fill the niche previously occupied by humans.
By reducing extinction risk from unaligned AGI, we preclude a world in which there’s no human on Earth, but powerful unaligned AGIs, who may then start its conquest to the broader universe.
By reducing non-extinction existential risks (e.g. disappointing futures), we preclude a world in which humans continue to exist, but fail to realize their potential in doing good.
What will the rankings be like, if we sort the four precluded worlds in decreasing order of badness? I’m highly unsure either, but I would guess something like 4>2>3>1 (larger means worse).
Closely related, and also important, is the question of “which world gets precluded”. Different possibilities include:
By reducing extinction risk from a (hypothetical) scenario in which Earth explodes and falls into pieces, we preclude a world in which there’s no life (and therefore no powerful agent) on what previously was Earth.
By reducing extinction risk from pandemics, we preclude a world in which there’s no human on Earth, but possibly other intelligent species that have evolved to fill the niche previously occupied by humans.
By reducing extinction risk from unaligned AGI, we preclude a world in which there’s no human on Earth, but powerful unaligned AGIs, who may then start its conquest to the broader universe.
By reducing non-extinction existential risks (e.g. disappointing futures), we preclude a world in which humans continue to exist, but fail to realize their potential in doing good.
What will the rankings be like, if we sort the four precluded worlds in decreasing order of badness? I’m highly unsure either, but I would guess something like 4>2>3>1 (larger means worse).