Interesting, I thought p(doom) was about literal extinction? If it also refers to unrecoverable collapse, then Iâm really surprised that takes up 15-30% of your potential scenarios! I always saw that part of the existential risk definition as negligible.
p(doom) is about doom. For AI, I think this can mean a few things:
Literal human extinction
Humans lose power over their future but are still alive (and potentially even have nice lives), either via stable totalitarianism or gradual disempowerment or other means
I checked parts of the study, and the 0.12% figure is for P(AI-caused existential catastrophe by 2100) according to the âAI skepticsâ. This is what is written about the definition of existential catastrophe just before it:
Participants made an initial forecast on the core question they disagreed about (weâll call this U, for âultimate questionâ): by 2100, will AI cause an existential catastrophe? We defined âexistential catastropheâ as an event in which at least one of the following occurs:
Humanity goes extinct
Humanity experiences âunrecoverable collapse,â which means either:
<$1 trillion global GDP annually [in 2022 dollars] for at least a million years (continuously), beginning before 2100; or
Human population remains below 1 million for at least a million years (continuously), beginning before 2100.
That sounds similar to the classic existential risk definition?
(Another thing thatâs important to note is that the study specifically sought forecasters skeptical of AI. So it doesnât tell us much if anything about what a group of random superforecasters would actually predict!)
I am very very surprised your âsecond bucketâ contains the possibility of humans potentially having nice lives! I suspect if you had asked me the definition of p(doom) before I read your initial comment, I would actually have mentioned the definition of existential risks that includes the permanent destruction of future potential. But I simply never took that second part seriously? Hence my initial confusion. I just assumed disempowerment or a loss of control would lead to literal extinction anyway, and that most people shared this assumption. In retrospect, that was probably naive of me. Now Iâm genuinely curious how much of peopleâs p(doom) estimates actually comes from actual extinction versus other scenarios...
That sounds similar to the classic existential risk definition?
Bostrom defines existential risk as âOne where an adverse outcome would either annihilate Earth-originating intelligent life or permanently and drastically curtail its potential.â Thereâs tons of events that could permanently and drastically curtail potential without reducing population or GDP that much. For example, AI could very plausibly seize total power, and still choose to keep >1 million humans alive. Keeping humans alive seems very cheap on a cosmic scale, so it could be justified by caring about humans a tiny bit, or maybe justified by thinking that aliens might care about humans and the AI wanting to preserve the option of trading with aliens, or something else. It seems very plausible that this could still have curtailed our potential, in the relevant sense. (E.g. if our potential required us to have control over a non-trivial fraction of resources.)
I think this is more likely than extinction, conditional on (what I would call) doom from misaligned AI. You can also compare with Paul Christianoâs more detailed views.
Interesting, I thought p(doom) was about literal extinction? If it also refers to unrecoverable collapse, then Iâm really surprised that takes up 15-30% of your potential scenarios! I always saw that part of the existential risk definition as negligible.
p(doom) is about doom. For AI, I think this can mean a few things:
Literal human extinction
Humans lose power over their future but are still alive (and potentially even have nice lives), either via stable totalitarianism or gradual disempowerment or other means
The second bucket is pretty big
I checked parts of the study, and the 0.12% figure is for P(AI-caused existential catastrophe by 2100) according to the âAI skepticsâ. This is what is written about the definition of existential catastrophe just before it:
That sounds similar to the classic existential risk definition?
(Another thing thatâs important to note is that the study specifically sought forecasters skeptical of AI. So it doesnât tell us much if anything about what a group of random superforecasters would actually predict!)
I am very very surprised your âsecond bucketâ contains the possibility of humans potentially having nice lives! I suspect if you had asked me the definition of p(doom) before I read your initial comment, I would actually have mentioned the definition of existential risks that includes the permanent destruction of future potential. But I simply never took that second part seriously? Hence my initial confusion. I just assumed disempowerment or a loss of control would lead to literal extinction anyway, and that most people shared this assumption. In retrospect, that was probably naive of me. Now Iâm genuinely curious how much of peopleâs p(doom) estimates actually comes from actual extinction versus other scenarios...
Bostrom defines existential risk as âOne where an adverse outcome would either annihilate Earth-originating intelligent life or permanently and drastically curtail its potential.â Thereâs tons of events that could permanently and drastically curtail potential without reducing population or GDP that much. For example, AI could very plausibly seize total power, and still choose to keep >1 million humans alive. Keeping humans alive seems very cheap on a cosmic scale, so it could be justified by caring about humans a tiny bit, or maybe justified by thinking that aliens might care about humans and the AI wanting to preserve the option of trading with aliens, or something else. It seems very plausible that this could still have curtailed our potential, in the relevant sense. (E.g. if our potential required us to have control over a non-trivial fraction of resources.)
I think this is more likely than extinction, conditional on (what I would call) doom from misaligned AI. You can also compare with Paul Christianoâs more detailed views.