I haven’t put serious thought into probabilities and I notice how AI experts’ probabilities are all over the map, which makes me think nobody has a great model. That said, my off-the-cuff probabilities are something like
if humanity doesn’t get its act together, 75% chance that AI causes extinction
unconditional 30% chance that AI causes extinction
when that happens depends on AI timelines, and I defer to Metaculus / AI Impacts / etc. on predicting timelines. maybe 20% chance of superintelligent AI within 10 years
Qualitatively, I think the prosaic alignment folks are too optimistic about alignment being easy so my P(doom) is higher than theirs.
Thanks, Michael. I think your numbers suggest your unconditional probability of human extinction over the next 10 years is 6 % (= 0.3*0.2). Power laws fit to battle deaths per war have a mean tail index of 1.60, such that battle deaths have a probability of 2.51 % (= 0.1^1.60) of becoming 10 times as large. Applying this to your estimate would suggest a probability of a 10 % population drop over the next 10 years of 2.39 (= 0.06/0.0251), i.e. impossibly high. What is your guess for an AI catastrophe killing 10 % of the population in the next 10 years? I suspect it is not that different from your guess for the probability of extinction, whereas this should be much lower according to typical power laws describing catastrophe deaths?
For reference, I think the probability of human extinction over the next 10 years is lower than 10^-6. Somewhat relatedly, fitting a power law to Metaculus’ community predictions about small AI catastrophes, I estimated a probability of human extinction before 2100 due to an AI malfunction of 0.004 %.
Applying this to your estimate would suggest a probability of a 10 % population drop over the next 10 years of 2.39
Tell me if I’m understanding this correctly:
My (rough) numbers suggest a 6% chance that 100% of people die
According to a fitted power law, that implies a 239% chance that 10% of people die
I disagree but I like your model and I think it’s a pretty good way of thinking about things.
On my plurality model (i.e. the model to which I assign the plurality of subjective probability), superintelligent AI (SAI) either kills 100% of people or it kills no people. I don’t think the outcome of SAI fits a power law.
A power law is typically generated by a combination of exponentials, which might be a good description of battle deaths, but I don’t think it’s a good description of AI. I think power laws are often a decent fit for combinations of heterogeneous events (such as mass deaths from all causes combined), but maybe not a great fit, so I wouldn’t put too much credence in the power law model in this case.
I think it’s very unlikely that an AI catastrophe kills 10% of the population in the next 10 years (not 10^-6 unlikely, more like 10^-3 unlikely). I can think of a few ways this could happen (e.g., a country gives an autonomous AI control over its nuclear arsenal and the AI decides to nuke a bunch of cities), but they seem much less likely than an SAI deciding to completely extinguish humanity.
I estimated a probability of human extinction before 2100 due to an AI malfunction of 0.004 %.
Even if you put 99% credence in this model, surely P(extinction) will be dominated by other models? Even within the model, P(extinction) should be higher than that based on uncertainty about the value of the alpha parameter.
My (rough) numbers suggest a 6% chance that 100% of people die
According to a fitted power law, that implies a 239% chance that 10% of people die
On 1, yes, and over the next 10 years or so (20 % chance of superintelligent AI over the next 10 years, times 30 % chance of extinction quickly after superintelligent AI)? On 2, yes, for a power law with a tail index of 1.60, which is the mean tail index of the power laws fitted to battle deaths per war here.
I think it’s very unlikely that an AI catastrophe kills 10% of the population in the next 10 years (not 10^-6 unlikely, more like 10^-3 unlikely).
I meant to ask about the probability of human population becoming less than (not around) 90 % as large as now over the next 10 years, which has to be higher than the probability of human extinction. Since 10^-3 << 6 %, I guess your probability of a population loss of 10 % or more is just slighly higher than your probability of human extinction.
Even if you put 99% credence in this model, surely P(extinction) will be dominated by other models? Even within the model, P(extinction) should be higher than that based on uncertainty about the value of the alpha parameter.
I think using a power law will tend to overestimate the probability of human extinction, as my sense is that tail distributions usually starts to decay faster as severity increases. This is the case for the annual conflict deaths as a fraction of the global population, and arguably annual epidemic/pandemic deaths as a fraction of the global population. The reason is that the tail distribution has to reach 0 for a 100 % population loss, whereas a power law will predict that going from 8 billion to 16 billion deaths is as likely as going from 4 billion to 8 billion deaths.
I haven’t put serious thought into probabilities and I notice how AI experts’ probabilities are all over the map, which makes me think nobody has a great model. That said, my off-the-cuff probabilities are something like
if humanity doesn’t get its act together, 75% chance that AI causes extinction
unconditional 30% chance that AI causes extinction
when that happens depends on AI timelines, and I defer to Metaculus / AI Impacts / etc. on predicting timelines. maybe 20% chance of superintelligent AI within 10 years
Qualitatively, I think the prosaic alignment folks are too optimistic about alignment being easy so my P(doom) is higher than theirs.
Thanks, Michael. I think your numbers suggest your unconditional probability of human extinction over the next 10 years is 6 % (= 0.3*0.2). Power laws fit to battle deaths per war have a mean tail index of 1.60, such that battle deaths have a probability of 2.51 % (= 0.1^1.60) of becoming 10 times as large. Applying this to your estimate would suggest a probability of a 10 % population drop over the next 10 years of 2.39 (= 0.06/0.0251), i.e. impossibly high. What is your guess for an AI catastrophe killing 10 % of the population in the next 10 years? I suspect it is not that different from your guess for the probability of extinction, whereas this should be much lower according to typical power laws describing catastrophe deaths?
For reference, I think the probability of human extinction over the next 10 years is lower than 10^-6. Somewhat relatedly, fitting a power law to Metaculus’ community predictions about small AI catastrophes, I estimated a probability of human extinction before 2100 due to an AI malfunction of 0.004 %.
Tell me if I’m understanding this correctly:
My (rough) numbers suggest a 6% chance that 100% of people die
According to a fitted power law, that implies a 239% chance that 10% of people die
I disagree but I like your model and I think it’s a pretty good way of thinking about things.
On my plurality model (i.e. the model to which I assign the plurality of subjective probability), superintelligent AI (SAI) either kills 100% of people or it kills no people. I don’t think the outcome of SAI fits a power law.
A power law is typically generated by a combination of exponentials, which might be a good description of battle deaths, but I don’t think it’s a good description of AI. I think power laws are often a decent fit for combinations of heterogeneous events (such as mass deaths from all causes combined), but maybe not a great fit, so I wouldn’t put too much credence in the power law model in this case.
I think it’s very unlikely that an AI catastrophe kills 10% of the population in the next 10 years (not 10^-6 unlikely, more like 10^-3 unlikely). I can think of a few ways this could happen (e.g., a country gives an autonomous AI control over its nuclear arsenal and the AI decides to nuke a bunch of cities), but they seem much less likely than an SAI deciding to completely extinguish humanity.
Even if you put 99% credence in this model, surely P(extinction) will be dominated by other models? Even within the model, P(extinction) should be higher than that based on uncertainty about the value of the alpha parameter.
On 1, yes, and over the next 10 years or so (20 % chance of superintelligent AI over the next 10 years, times 30 % chance of extinction quickly after superintelligent AI)? On 2, yes, for a power law with a tail index of 1.60, which is the mean tail index of the power laws fitted to battle deaths per war here.
I meant to ask about the probability of human population becoming less than (not around) 90 % as large as now over the next 10 years, which has to be higher than the probability of human extinction. Since 10^-3 << 6 %, I guess your probability of a population loss of 10 % or more is just slighly higher than your probability of human extinction.
I think using a power law will tend to overestimate the probability of human extinction, as my sense is that tail distributions usually starts to decay faster as severity increases. This is the case for the annual conflict deaths as a fraction of the global population, and arguably annual epidemic/pandemic deaths as a fraction of the global population. The reason is that the tail distribution has to reach 0 for a 100 % population loss, whereas a power law will predict that going from 8 billion to 16 billion deaths is as likely as going from 4 billion to 8 billion deaths.