I’m highly sceptical of point probability estimates for events for which we have virtually no information—that’s exactly why I made these tools.
I mentioned point/mean probability estimates, but my upper bounds (e.g. 90th percentile) are quite close, as they are strongly limited by the means. For example, if one’s mean probability is 10^-10, the 90th percentile probability cannot be higher than 10^-9, otherwise the mean probability would be higher than 10^-10 (= (1 − 0.90)*10^-9), which is the mean. So my point remains as long as you think my point/mean estimates are reasonable.
Per Dan Schwarz’s recent post, it seems much more important to me to give an interactive model into which people can put their own credences, so that we can then debate the input rather than the output.
Makes sense. I liked that post. I think my comment was probably overly crictical, and not related specifically to your series. I was not clear, but I meant to point to the greater value of using standard cost-effectiveness analyses (relative to using a model like yours) given my current empirical beliefs (astronomically low non-TAI extinction risk).
More importantly, much of the point of the calculators is that one can still have very low credence of direct extinction from any of the sources you mentioned and still believe that such events substantially reduce the chance of us becoming interstellar by two basic mechanisms
Fair! I suspect the number of lives saved, maybe weighted by the reciprocal of the population size, would still be a good proxy for the benefits of interventions affecting civilisational collapse. When I tried to look into this quantitatively in the context of nuclear war, improving worst case outcomes did not appear to be the driver of the overall expected value. So I am guessing using standard cost-effectiveness analyses, based on a metric like lives saved per dollar, would continue to be a fair way of assessing interventions.
In any case, I assume your model would still be useful for people with different views!
If by ‘recovery’ you mean ‘reaching modern technology’ this is compatible with what I’ve just written above—it might turn out to be relatively trivial to rereach modern technology, but increasingly implausible that we can ever progress beyond it.
I meant full recovery in the sense of reaching the same state we are in now, with roughly the same chances of becoming a benevolent interstellar civilisation going forward.
If I understand you right, you’re getting most of your confidence in recovery from situations where other intelligent species evolve?
For my estimate of a probability of 0.0513 % of not fully recovering, yes, because I was assuming human extinction in my calculation. If the disaster is less severe, my probability of not fully recovering would be even lower.
as with shorter term recoveries, if it happened to us, it could happen again to the new species
If one thinks the probability of extinction or permanent collapse without TAI is astronomically low (as I do), the probability of a double catastrophe is astronomically low to the power of 2, i.e. it presents negligible risk. So I believe a single catastrophe has to be somewhat plausible for the possibility of further catastrophes to matter.
on the timescale that could take, we have to think seriously about the amount of loss of future value due to expansion of the universe. I think Anders Sandberg is working on modelling this.
I think considerations related the astronomical waste argument are applicable not only to saving lives in catastrophes, but also in normal time. To bring everything into the same framework in a simple way, I would run a standard cost-effectiveness analysis, but weighting saving lives by a function of the population size (e.g. 1/”population size” as I suggested above).
If one is concerned that our current civilisation’s values are contingent and counterfactually positive, it seems still less likely that a new species would share them.
On priors, I would say one should expect the values of a new similarly capable species to be as good as those of humans.
intelligence might just not re-evolve to the degree needed to develop advanced technology. There are a number of theories about why we developed it to such a degree—despite abstract reasoning having virtually no survival value to hunter gatherer or even agricultural societies—and if something like e.g. the sexual selection theory is correct, it might have been a one-off evolutionary accident, never to be repeated.
I imagine different theories make significantly different predictions about the difficulty of going from e.g. monkeys to humans, but I have the impression there is often little data to validate them, and therefore think significant weight should be given to a prior simply informed by how long a given transition took. This is why I got my estimate for the probability of not fully recovering relying on the time since the last mass extinction.
even after tens of millions of years, most of the resources I described above would still be gone if they depleted them. Fossil fuels accumulated throughout the period of complex life on this planet which we’re about halfway through—so we could expect at best to have approximately the same number once more, if no civilisation ever used any in the meantime. Fissile materials will never recover, and it’s hard to see how eg. a thin layer of rock phosphorus scattered across the bottom of the ocean would ever return to an economically viable state, even on geological timelines. So the new species might evolve only to find themselves trapped on the surface.
Even assuming we could only recover with pretty high likelihood once, we would need 2 events with astronomically low chances to go from where we are to extinction or permanent collapse. So the overall risk of these would be negligible if one agrees with my estimates. At the same time, I think you are raising great points. I do not think they have much strenght, but this is just because of my view that catastrophes plausibly leading to extinction or global collapse are very unlikely.
I explicitly aimed to capture this concern in the OP description ‘human descendants (or whatever class of life we think has value)’. If you think TAI replacing humans would be as good or better, you can treat scenarios where it does so as transitioning directly from whatever state we’d be in at the time to an interstellar/existentially secure state.
Fwiw in the Matthew Barnett post you linked to, I replied that I strongly support that position philosophically—I think my take was even more pro-conscious-AI than his.
Makes sense. I think human extinction caused by TAI would be bad if it happened in the next few years, as I suppose there would not be enough time to plan a good transition in this case. Nevertheless, in a future dominated by sentient benevolent advanced AIs, human extinction would not be obviously bad.
Yeah, it sounds like this might not be appropriate for someone with your credences, though I’m confused by what you say here:
I mentioned point/mean probability estimates, but my upper bounds (e.g. 90th percentile) are quite close, as they are strongly limited by the means. For example, if one’s mean probability is 10^-10, the 90th percentile probability cannot be higher than 10^-9, otherwise the mean probability would be higher than 10^-10 (= (1 − 0.90)*10^-9), which is the mean. So my point remains as long as you think my point/mean estimates are reasonable.
I’m not sure what you mean by this. What are you taking the mean of, and which type of mean, and why? It sounds like maybe you’re talking about the arithmetic mean? If so that isn’t how I would think about unknown probabilities fwiw. IMO it seems more appropriate to use a geometric mean to express this kind of uncertainty, or explicitly model the distribution of possible probabilities. I don’t think either approach should limit your high-end credences.
Makes sense. I liked that post. I think my comment was probably overly crictical, and not related specifically to your series. I was not clear, but I meant to point to the greater value of using standard cost-effectiveness analyses (relative to using a model like yours) given my current empirical beliefs (astronomically low non-TAI extinction risk).
Yeah, fair enough :)
If one thinks the probability of extinction or permanent collapse without TAI is astronomically low (as I do)
Have you written somewhere about why you think permanent collapse is so unlikely? The more I think about it, the higher my credence seems to get :\
I have the impression there is often little data to validate them, and therefore think significant weight should be given to a prior simply informed by how long a given transition took.
I’m not saying the sexual selection theory is strongly likely to be correct. But it seems to be taken seriously by evolutionary psychologists, and if you’re finding that other theories of human intelligence give ultra-high credence of a new species evolving, it seems like that credence should be substantially lowered by even a modest belief in the plausibility of such theories.
What are you taking the mean of, and which type of mean, and why? It sounds like maybe you’re talking about the arithmetic mean?
Yes, I was referring to the arithmetic mean of a probability distribution. To illustrate, if I thought the probability of a given event was uniformly distributed between 0 and 1, the mean (best guess) probability would be 50 % (= (0 + 1)/2).
IMO it seems more appropriate to use a geometric mean to express this kind of uncertainty, or explicitly model the distribution of possible probabilities.
I agree the median, geometric mean, or geometric mean of odds are usually better than the mean to aggregate forecasts[1]. However, if we aggregated multiple probability distributions from various models/forecasters, we would end up with a final probability distribution, and I am saying our final point estimate corresponds to the mean of this distribution. Jaime Sevilla illustrated this here.
I don’t think either approach should limit your high-end credences.
Maybe it helps to think about this in the context of a distribution which is not over a probability. If we have a distribution over possible profits, and our expected profit is 100 $, it cannot be the case that the 90th percentile profit is 1 M$, because in this case the expected profit would be at least 100 k$ (= (1 − 0.90)*1*10^6), which is much larger than 100 $.
You may want to check Joe Carlsmith’s thoughts on this topic in the context of AI risk.
Have you written somewhere about why you think permanent collapse is so unlikely? The more I think about it, the higher my credence seems to get :\
No, at least not in any depth. I think permanent collapse would require very large population and infrastructure losses, but I see these as very unlikely, at least in the absence of TAI. I estimated a probability of 3.29*10^-6 of the climatic effects of nuclear war before 2050 killing 50 % of the global population (based on the distribution I defined here for the famine death rate). Pandemics would not directly cause infrastructure loss. Indirectly, there could be infrastructure loss due people stopping maintenance activities out of fear of being infected, but I guess this requires a level of lethality which makes the pandemic very unlikely.
Besides more specific considerations like the above, I have consistently ended up arriving to tail risk estimates much lower than canonical ones from the effective altruism community. So, instead of regarding these as a prior as I used to do, now I immediately start from a lower prior, as I should not expect by risk estimates to go down/up[2]. For context on me arriving to lower tail risk estimates, you can check the posts I linked at the start of my 1st comment. Here are 2 concrete examples I discussed elsewhere:
Luisa Rodriguez’s analyses, which are arguably somewhat canonical in the effective altruism community, imply 630 M expected deaths before 2050 from nuclear wars between the United States and Russia, whereas I estimated just 2 % of that.
Denkenberger 2022impliesthe value of the future would decrease by 12.0 % given a 10 % agricultural shortfall, which corresponds to an injection of soot into the stratosphere of around 5 Tg[3], whereas I think the longterm impact of this would be negligible. Even given human extinction, I guess the value of the future would only decrease by 0.0513 % (see rough calculation in my 1st comment).
Relatedly, my extinction risk estimatesaremuch lower than Toby Ord’s existential risk estimates given inThe Precipice.
I aggregated probabilities using the median to estimate my prior extinction risk for wars and terrorist attacks, and using the geometric mean to obtain my nuclear war extinction risk.
Thanks for the follow up! I strongly upvoted it.
I mentioned point/mean probability estimates, but my upper bounds (e.g. 90th percentile) are quite close, as they are strongly limited by the means. For example, if one’s mean probability is 10^-10, the 90th percentile probability cannot be higher than 10^-9, otherwise the mean probability would be higher than 10^-10 (= (1 − 0.90)*10^-9), which is the mean. So my point remains as long as you think my point/mean estimates are reasonable.
Makes sense. I liked that post. I think my comment was probably overly crictical, and not related specifically to your series. I was not clear, but I meant to point to the greater value of using standard cost-effectiveness analyses (relative to using a model like yours) given my current empirical beliefs (astronomically low non-TAI extinction risk).
Fair! I suspect the number of lives saved, maybe weighted by the reciprocal of the population size, would still be a good proxy for the benefits of interventions affecting civilisational collapse. When I tried to look into this quantitatively in the context of nuclear war, improving worst case outcomes did not appear to be the driver of the overall expected value. So I am guessing using standard cost-effectiveness analyses, based on a metric like lives saved per dollar, would continue to be a fair way of assessing interventions.
In any case, I assume your model would still be useful for people with different views!
I meant full recovery in the sense of reaching the same state we are in now, with roughly the same chances of becoming a benevolent interstellar civilisation going forward.
For my estimate of a probability of 0.0513 % of not fully recovering, yes, because I was assuming human extinction in my calculation. If the disaster is less severe, my probability of not fully recovering would be even lower.
If one thinks the probability of extinction or permanent collapse without TAI is astronomically low (as I do), the probability of a double catastrophe is astronomically low to the power of 2, i.e. it presents negligible risk. So I believe a single catastrophe has to be somewhat plausible for the possibility of further catastrophes to matter.
I think considerations related the astronomical waste argument are applicable not only to saving lives in catastrophes, but also in normal time. To bring everything into the same framework in a simple way, I would run a standard cost-effectiveness analysis, but weighting saving lives by a function of the population size (e.g. 1/”population size” as I suggested above).
On priors, I would say one should expect the values of a new similarly capable species to be as good as those of humans.
I imagine different theories make significantly different predictions about the difficulty of going from e.g. monkeys to humans, but I have the impression there is often little data to validate them, and therefore think significant weight should be given to a prior simply informed by how long a given transition took. This is why I got my estimate for the probability of not fully recovering relying on the time since the last mass extinction.
Even assuming we could only recover with pretty high likelihood once, we would need 2 events with astronomically low chances to go from where we are to extinction or permanent collapse. So the overall risk of these would be negligible if one agrees with my estimates. At the same time, I think you are raising great points. I do not think they have much strenght, but this is just because of my view that catastrophes plausibly leading to extinction or global collapse are very unlikely.
Makes sense. I think human extinction caused by TAI would be bad if it happened in the next few years, as I suppose there would not be enough time to plan a good transition in this case. Nevertheless, in a future dominated by sentient benevolent advanced AIs, human extinction would not be obviously bad.
Yeah, it sounds like this might not be appropriate for someone with your credences, though I’m confused by what you say here:
I’m not sure what you mean by this. What are you taking the mean of, and which type of mean, and why? It sounds like maybe you’re talking about the arithmetic mean? If so that isn’t how I would think about unknown probabilities fwiw. IMO it seems more appropriate to use a geometric mean to express this kind of uncertainty, or explicitly model the distribution of possible probabilities. I don’t think either approach should limit your high-end credences.
Yeah, fair enough :)
Have you written somewhere about why you think permanent collapse is so unlikely? The more I think about it, the higher my credence seems to get :\
I’m not saying the sexual selection theory is strongly likely to be correct. But it seems to be taken seriously by evolutionary psychologists, and if you’re finding that other theories of human intelligence give ultra-high credence of a new species evolving, it seems like that credence should be substantially lowered by even a modest belief in the plausibility of such theories.
Yes, I was referring to the arithmetic mean of a probability distribution. To illustrate, if I thought the probability of a given event was uniformly distributed between 0 and 1, the mean (best guess) probability would be 50 % (= (0 + 1)/2).
I agree the median, geometric mean, or geometric mean of odds are usually better than the mean to aggregate forecasts[1]. However, if we aggregated multiple probability distributions from various models/forecasters, we would end up with a final probability distribution, and I am saying our final point estimate corresponds to the mean of this distribution. Jaime Sevilla illustrated this here.
Maybe it helps to think about this in the context of a distribution which is not over a probability. If we have a distribution over possible profits, and our expected profit is 100 $, it cannot be the case that the 90th percentile profit is 1 M$, because in this case the expected profit would be at least 100 k$ (= (1 − 0.90)*1*10^6), which is much larger than 100 $.
You may want to check Joe Carlsmith’s thoughts on this topic in the context of AI risk.
No, at least not in any depth. I think permanent collapse would require very large population and infrastructure losses, but I see these as very unlikely, at least in the absence of TAI. I estimated a probability of 3.29*10^-6 of the climatic effects of nuclear war before 2050 killing 50 % of the global population (based on the distribution I defined here for the famine death rate). Pandemics would not directly cause infrastructure loss. Indirectly, there could be infrastructure loss due people stopping maintenance activities out of fear of being infected, but I guess this requires a level of lethality which makes the pandemic very unlikely.
Besides more specific considerations like the above, I have consistently ended up arriving to tail risk estimates much lower than canonical ones from the effective altruism community. So, instead of regarding these as a prior as I used to do, now I immediately start from a lower prior, as I should not expect by risk estimates to go down/up[2]. For context on me arriving to lower tail risk estimates, you can check the posts I linked at the start of my 1st comment. Here are 2 concrete examples I discussed elsewhere:
Luisa Rodriguez’s analyses, which are arguably somewhat canonical in the effective altruism community, imply 630 M expected deaths before 2050 from nuclear wars between the United States and Russia, whereas I estimated just 2 % of that.
Denkenberger 2022implies the value of the future would decrease by 12.0 % given a 10 % agricultural shortfall, which corresponds to an injection of soot into the stratosphere of around 5 Tg[3], whereas I think the longterm impact of this would be negligible. Even given human extinction, I guess the value of the future would only decrease by 0.0513 % (see rough calculation in my 1st comment).
Relatedly, my extinction risk estimates are much lower than Toby Ord’s existential risk estimates given in The Precipice.
I aggregated probabilities using the median to estimate my prior extinction risk for wars and terrorist attacks, and using the geometric mean to obtain my nuclear war extinction risk.
If I expected my best guess to go up/down, I should just update my best guess now to the value I expect it will converge to.
Xia 2022 predicts a shortfall of 7.0 % for 5 Tg without adaptation (see last row of Table S2).