The x-risk model in particular seems to make bake-in quite conservative assumptions (medium-high confidence)
Note less conservative assumptions for existential risk interventions make them even less comparable with neartermist ones. Extending the time horizon beyond 3023 increases the cost-effectiveness of existential risk interventions, but not that of neartermist ones. Under a longtermist view where longterm effects dominate, it is crucial to model the longterm effects of neartermist interventions, but these are not in the model. So as of now I do not think it is that useful to compare longtermist with neartermist interventions.
Thatās fair, though I personally would be happy to just factor in neartermist interventions to marginal changes in economic growth (which in most cases I expect to be negligible) in the absence of some causal mechanism by which I should expect some neartermist intervention to have an outsized influence on the long-run future.
Thanks for following up! How about assessing the benefits of both global catastrophic risk (GCR) and neartermist interventions in terms of lives saved, but weighting these by a function which increases as population size decreases? Saving lives is an output in both types of intervention, but neartermist interventions save lives at a higher population size than GCR ones. For reference:
Carl Shulman seemed to suggest in a post on the flow-through effects of saving a life that the respective indirect longterm effects, in terms of the time by which humanityās trajectory is advanced, are inversely proportional to the population when the life is saved[1].
Based on the above, and assuming a power law for the ratio between the pre- and post-catastrophe (human) population, and a constant cost to save a life as a function of such ratio, it looks like saving lives in normal times is better to improve the longterm future than doing so in catastrophes.
For example, suppose one saved a drowning child 10,000 years ago, when the human population was estimated to be only in the millions. For convenience, weāll posit a little over 7 million, 1/ā1000th of the current population. Since the child would add to population pressures on food supplies and disease risk, the effective population/āeconomic boost could range from a fraction of a lifetime to a couple of lifetimes (via children), depending on the frequency of famine conditions. Famines were not annual and population fluctuated on a time scale of decades, so I will use 20 years of additional life expectancy.
So, for ~ 20 years the ancient population would be 1ā7,000,000th greater, and economic output/ātechnological advance. We might cut this to 1ā10,000,000 to reflect reduced availability of other inputs, although increasing returns could cut the other way. Using 1ā10,000,000 cumulative world economic output would reach the same point ~ 1ā500,000th of a year faster. An extra 1ā500,000th of a year with around our current population of ~7 billion would amount to an additional ~14,000 life -years, 700 times the contemporary increase in life years lived. Moreover, those extra lives on average have a higher standard of living than their ancient counterparts.
Readers familiar with Nick Bostromās paper on astronomical waste will see that this is a historical version of the same logic: when future populations will be far larger, expediting that process even slightly can affect the existence of many people. We cut off our analysis with current populations, but the greater the population this growth process will reach, the greater long-run impact of technological speedup from saving ancient lives.
it looks like saving lives in normal times is better to improve the longterm future than doing so in catastrophes.
Both seem negligible in the effect on the long term future without some more specific causal mechanism other than āthings go fasterā right?
Like I would guess that the vast majority of risk (now) is anthropogenic risk and anthropogenic risk should be unaffected by just speeding things up (or plausibly higher if it causes things to go faster at critical point rather than just getting to the critical point sooner).
And astronomical waste itself is also negligible (about 1ā10 billion per year).
As far as I can tell, Carl doesnāt overcome this basic argument in his post and it is very unclear to me if he is even trying to argue āsaving lives substantially improves the long run futureā. I think he is mostly just using the past as an analogy for the current case for longtermism?
Both seem negligible in the effect on the long term future without some more specific causal mechanism other than āthings go fasterā right?
I guess you are thinking that multiplying a non-negligible reduction in the nearterm risk of human extinction per cost (e.g. 2.17*10^-13 per dollar[1]) by an astronomical expected value of the future (e.g. 1.40*10^52 human lives[2]) will result in an astronomically high cost-effectiveness (e.g. 3.04*10^39 life/ā$). However, this calculation assumes the reduction in nearterm extinction risk equals the relative increase in the expected value of the future, whereas I think the latter is astronomically lower.
And astronomical waste itself is also negligible (about 1ā10 billion per year).
It is unclear to me whether existential risk is higher than 10^-10 per year. Iamopento best guesses of an annual extinction risk of 10^-8, and a probability of 1 % of extinction being an existential catastrophe[3], which would lead to an annual existential risk of 10^-10.
As far as I can tell, Carl doesnāt overcome this basic argument in his post and it is very unclear to me if he is even trying to argue āsaving lives substantially improves the long run futureā. I think he is mostly just using the past as an analogy for the current case for longtermism?
I agree Carl was not trying to argue for saving lives in normal times being among the most cost-effective ways of improving the longterm future.
Median cost-effectiveness bar for mitigating existential risk I collected. The bar does not respect extinction risk, but I assume the people who provided the estimates would have guessed similar values for extinction risk.
Mean of a loguniform distribution with minimum and maximum of 10^23 and 10^54 lives. The minimum is the estimate for āan extremely conservative readerā obtained in Table 3 of Newberry 2021. The maximum is the largest estimate in Table 1 of Newberry 2021, determined for the case where all the resources of the affectable universe support digital persons. The upper bound can be 10^30 times as high if civilization āaestivate[s] until the far future in order to exploit the low temperature environmentā, in which computations are more efficient. Using a higher bound does not qualitatively change my point.
I estimated a 0.0513 % chance of not fully recovering from a repetition of the last mass extinction 66 M years ago, the CretaceousāPaleogene extinction event. If biological humans go extinct because of advanced AI, I guess it is very likely they will have suitable successors then, either in the form of advanced AI or some combinations between it and humans.
and a probability of 1 % of extinction being an existential catastrophe
I think you should probably have a higher probability on some unknown filter making it less likely that intelligent civilization re-evolves. (Given anthropics.)
Iād say 20% chance that intelligent life doesnāt re-evolve on earth due to this mechanism.
There are also potentially aliens, which is perhaps a factor of 2 getting me to 10% chance of no group capable of using resources conditional on literal extinction of all intelligent civilization on earth. (Which is 10x higher than your estimate.)
I also think that Iād prefer human control than the next evolved life and than aliens by a moderate amount due to similarity of values arguments.
Iāve now updated toward a higher chance life re-evolves and a lower chance on some unknown filter because we can see that the primate to intelligent civilization time gap is quite small.
That makes sense. It looks like humans branched off chimpanzees just 5.5 M years (= (5 + 6)/ā2*10^6) ago. Assuming the time from chimpanzees to a species similar to humans follows an exponential distribution with a mean equal to that time, the probability of not recovering after human extinction in the 1 billion years during which Earth will remain habitable would be only 1.09*10^-79 (= e^(-10^9/ā(5.5*10^6))). The probability of not recovering is higher due to model uncertainty. The time to recover may follow a different distribution.
In addition, recovery can be harder for other risks:
Catastrophes wiping out more species in humansā evolutionary past (e.g. the impact of a large comet) would have a longer expected recovery time, and therefore imply a lower chance of recovery during the time Earth will remain habitable.
The estimates I provided in my comment were mainly illustrative. However, my 1 % chance of existential catastrophe conditional on human extinction was coming from my expectation that humans will be on board with going extinct in the vast majority of worlds where they go extinct in the next few centuries because their AI or posthuman descendents would live on.
whereas I think the latter is astronomically lower.
Your argument doesnāt seem clearly laid out in the doc, but it sounds to me like your view is that there isnāt a ātime of perilsā and then sufficient technology for long run robustness.
I think you might find it useful to more clearly state your argument which seems very opaque in that linked document.
I disagree and think a time of perils seems quite likely given the potential for a singularity.
There is a bunch of discussion making this exact point in response to āMistakes in the moral mathematics of existential riskā (which seems mostly mistaken to me via the mechanism implicitly putting astronomically low probability on robust intersteller civilizations).
It is unclear to me whether existential risk is higher than 10^-10 per year.
Causes of X-risk which seem vastly higher than this include:
AI takeover supposing you grant that AI control is less valuable.
Autocratic control supposing you grant that autocratic control of the long run future is less valuable.
I mostly think x-risk is mostly non-extinction and almost all the action is in changing which entities have control over resources rather than reducing astronomical waste.
Perhaps you adopt a view in which you donāt care at all what happens with long run resources so long as any group hypothetically has the ability to utilize these resources? Otherwise, given the potential for lock in, it seems like influencing who has control is vastly more important than you seem to be highlighting.
(My guess is that āno-entity ends up being in a position where they could hypothetically utilize long run resourcesā is about 300x lower than other x-risk (perhaps 0.1% vs 30% all cause x-risk) which is vastly higher than your estimate.)
I also put vaster higher probability than you on extinction due to incredibly powerful bioweapons or other future technology, but this isnāt most of my concern.
Your argument doesnāt seem clearly laid out in the doc, but it sounds to me like your view is that there isnāt a ātime of perilsā and then sufficient technology for long run robustness.
I am mainly sceptical of the possibility of making worlds with astronomical value significantly more likely, regardless of whether the longterm annual probability of value dropping a lot tends to 0 or not.
I think you might find it useful to more clearly state your argument which seems very opaque in that linked document.
I agree what I shared is not very clear, although I will probably post it roughly as is one of these days, and then eventually follow up.
I disagree and think a time of perils seems quite likely given the potential for a singularity.
It is unclear to me whether faster economic growth or technological progress imply a higher extinction risk. I would say this has generally been going down until now, except maybe from around 1939 (start of World War 2) to 1986 (when nuclear warheads peaked), although the fraction of people living in democracies increased 21.6 pp (= 0.156 + 0.183 - (0.0400 + 0.0833)) during this period.
There is a bunch of discussion making this exact point in response to āMistakes in the moral mathematics of existential riskā (which seems mostly mistaken to me via the mechanism implicitly putting astronomically low probability on robust intersteller civilizations).
I agree the probability of intersteller civilizations and astronomically valuable futures more broadly should not be astronomically low. For example, I guess it is fine to assume a 1 % chance on each order of magnitude between 1 and 10^100 human lives of future value. This is not my best guess, but it is just to give you a sense than I think astronomically valuable futures are plausible. However, I guess it is very hard to increase the probability of the astronomically valuable worlds.
Causes of X-risk which seem vastly higher than this include:
AI takeover supposing you grant that AI control is less valuable.
Autocratic control supposing you grant that autocratic control of the long run future is less valuable.
I mostly think x-risk is mostly non-extinction and almost all the action is in changing which entities have control over resources rather than reducing astronomical waste.
I guess the probability of something like a global dictactorship by 2100 is many orders of magnitude higher than 10^-10, but I do not think it would be permanent. If it was, then I would guess the alternative would be worse.
Perhaps you adopt a view in which you donāt care at all what happens with long run resources so long as any group hypothetically has the ability to utilize these resources? Otherwise, given the potential for lock in, it seems like influencing who has control is vastly more important than you seem to be highlighting.
(My guess is that āno-entity ends up being in a position where they could hypothetically utilize long run resourcesā is about 300x lower than other x-risk (perhaps 0.1% vs 30% all cause x-risk) which is vastly higher than your estimate.)
There are many concept of existential risk, so I prefer to focus on probabilities of clearly defined situations. One could think about existential risk from risk R as the relative increase in the expected value of the future if risk R was totally mitigated, but this is super hard to estimate in a way that the results are informative. I currently think it is better to assess interventions based on standard cost-effectiveness analyses.
It is unclear to me whether faster economic growth or technological progress imply a higher extinction risk. I would say this has generally been going down until now
My view is that the majority of bad-things-happen-with the cosmic endowment risk is downstream of AI takeover.
I generally donāt think looking at historical case studies will be super informative here.
I agree that doing the singularity faster doesnāt make things worse, Iām just noting that youāll go through a bunch of technology in a small amount of wall clock time.
Sure, but is the probability of it being permanent more like 0.05 or 10^-6? I would guess more like 0.05. (Given modern technology and particularly the possibility of AI and the singularity.)
It depends on the specific definition of global dictactorship and the number of years. However, the major problem is that I have very little to say about what will happen further than 100 years into the future other than thinking that whatever is happening will continue to change, and is not determined by what we do now.
By āpermanentā, I mean >10 billion years. By āglobalā, I mean āit ācontrolsā >80% of resources under earth originating civilization controlā. (Where control evolves with the extent to which technology allows for control.)
Thanks for clarifying! Based on that, and Wikipediaās definition of dictactorship as āan autocratic form of government which is characterized by a leader, or a group of leaders, who hold governmental powers with few to no limitationsā, I would say more like 10^-6. However, I do not think this matters, because that far into the future I would no longer be confident to say which form of government is better or worse.
I am mainly sceptical of the possibility of making worlds with astronomical value significantly more likely, regardless of whether the longterm annual probability of value dropping a lot tends to 0 or not.
As, in your argument is that you are skeptical on priors? I think Iām confused what the argument is here.
Separately, my view is that due to acausal trade, itās very likely that changing from human control to AI control looks less like āmaking worlds with astronomical value more likelyā and looks more like āshifting some resources across the entire continuous measureā. But, this mostly adds up to the same thing as creating astronomical value.
As, in your argument is that you are skeptical on priors? I think Iām confused what the argument is here.
Yes, mostly that. As far as I can tell, the (posterior) counterfactual impact of interventions whose effects can be accurately measured, like ones in global health and development, decays to 0 as time goes by, and can be modelled as increasing the value of the world for a few years or decades, far from astronomically.
Separately, my view is that due to acausal trade, itās very likely that changing from human control to AI control looks less like āmaking worlds with astronomical value more likelyā and looks more like āshifting some resources across the entire continuous measureā. But, this mostly adds up to the same thing as creating astronomical value.
I personally do not think acausal trade considerations are action relevant, but, if I was to think along those lines, I would assume there is way more stuff to be acausally influenced which is weakly correlated with what humans do than that is strongly correlated. So the probability of influencing more stuff acausally should still decrease with value, and I guess the decrease in the probability density would be faster than the increase in value, such that value density decreases with value. In this case, the expected value from astronomical acausal trades would still be super low.
Thanks for sharing, Caleb.
Note less conservative assumptions for existential risk interventions make them even less comparable with neartermist ones. Extending the time horizon beyond 3023 increases the cost-effectiveness of existential risk interventions, but not that of neartermist ones. Under a longtermist view where longterm effects dominate, it is crucial to model the longterm effects of neartermist interventions, but these are not in the model. So as of now I do not think it is that useful to compare longtermist with neartermist interventions.
Thatās fair, though I personally would be happy to just factor in neartermist interventions to marginal changes in economic growth (which in most cases I expect to be negligible) in the absence of some causal mechanism by which I should expect some neartermist intervention to have an outsized influence on the long-run future.
Thanks for following up! How about assessing the benefits of both global catastrophic risk (GCR) and neartermist interventions in terms of lives saved, but weighting these by a function which increases as population size decreases? Saving lives is an output in both types of intervention, but neartermist interventions save lives at a higher population size than GCR ones. For reference:
Carl Shulman seemed to suggest in a post on the flow-through effects of saving a life that the respective indirect longterm effects, in terms of the time by which humanityās trajectory is advanced, are inversely proportional to the population when the life is saved[1].
Based on the above, and assuming a power law for the ratio between the pre- and post-catastrophe (human) population, and a constant cost to save a life as a function of such ratio, it looks like saving lives in normal times is better to improve the longterm future than doing so in catastrophes.
Here is the relevant excerpt:
Both seem negligible in the effect on the long term future without some more specific causal mechanism other than āthings go fasterā right?
Like I would guess that the vast majority of risk (now) is anthropogenic risk and anthropogenic risk should be unaffected by just speeding things up (or plausibly higher if it causes things to go faster at critical point rather than just getting to the critical point sooner).
And astronomical waste itself is also negligible (about 1ā10 billion per year).
As far as I can tell, Carl doesnāt overcome this basic argument in his post and it is very unclear to me if he is even trying to argue āsaving lives substantially improves the long run futureā. I think he is mostly just using the past as an analogy for the current case for longtermism?
Thanks for the comment, Ryan!
I guess you are thinking that multiplying a non-negligible reduction in the nearterm risk of human extinction per cost (e.g. 2.17*10^-13 per dollar[1]) by an astronomical expected value of the future (e.g. 1.40*10^52 human lives[2]) will result in an astronomically high cost-effectiveness (e.g. 3.04*10^39 life/ā$). However, this calculation assumes the reduction in nearterm extinction risk equals the relative increase in the expected value of the future, whereas I think the latter is astronomically lower.
It is unclear to me whether existential risk is higher than 10^-10 per year. I am open to best guesses of an annual extinction risk of 10^-8, and a probability of 1 % of extinction being an existential catastrophe[3], which would lead to an annual existential risk of 10^-10.
I agree Carl was not trying to argue for saving lives in normal times being among the most cost-effective ways of improving the longterm future.
Median cost-effectiveness bar for mitigating existential risk I collected. The bar does not respect extinction risk, but I assume the people who provided the estimates would have guessed similar values for extinction risk.
Mean of a loguniform distribution with minimum and maximum of 10^23 and 10^54 lives. The minimum is the estimate for āan extremely conservative readerā obtained in Table 3 of Newberry 2021. The maximum is the largest estimate in Table 1 of Newberry 2021, determined for the case where all the resources of the affectable universe support digital persons. The upper bound can be 10^30 times as high if civilization āaestivate[s] until the far future in order to exploit the low temperature environmentā, in which computations are more efficient. Using a higher bound does not qualitatively change my point.
I estimated a 0.0513 % chance of not fully recovering from a repetition of the last mass extinction 66 M years ago, the CretaceousāPaleogene extinction event. If biological humans go extinct because of advanced AI, I guess it is very likely they will have suitable successors then, either in the form of advanced AI or some combinations between it and humans.
I think you should probably have a higher probability on some unknown filter making it less likely that intelligent civilization re-evolves. (Given anthropics.)
Iād say 20% chance that intelligent life doesnāt re-evolve on earth due to this mechanism.
There are also potentially aliens, which is perhaps a factor of 2 getting me to 10% chance of no group capable of using resources conditional on literal extinction of all intelligent civilization on earth. (Which is 10x higher than your estimate.)
I also think that Iād prefer human control than the next evolved life and than aliens by a moderate amount due to similarity of values arguments.
Iāve now updated toward a higher chance life re-evolves and a lower chance on some unknown filter because we can see that the primate to intelligent civilization time gap is quite small.
That makes sense. It looks like humans branched off chimpanzees just 5.5 M years (= (5 + 6)/ā2*10^6) ago. Assuming the time from chimpanzees to a species similar to humans follows an exponential distribution with a mean equal to that time, the probability of not recovering after human extinction in the 1 billion years during which Earth will remain habitable would be only 1.09*10^-79 (= e^(-10^9/ā(5.5*10^6))). The probability of not recovering is higher due to model uncertainty. The time to recover may follow a different distribution.
In addition, recovery can be harder for other risks:
Catastrophes wiping out more species in humansā evolutionary past (e.g. the impact of a large comet) would have a longer expected recovery time, and therefore imply a lower chance of recovery during the time Earth will remain habitable.
As I said above, I estimated a 0.0513 % chance of not fully recovering from a repetition of the last mass extinction 66 M years ago, the CretaceousāPaleogene extinction event.
A rogue AI would not allow another species to take control.
The estimates I provided in my comment were mainly illustrative. However, my 1 % chance of existential catastrophe conditional on human extinction was coming from my expectation that humans will be on board with going extinct in the vast majority of worlds where they go extinct in the next few centuries because their AI or posthuman descendents would live on.
Your argument doesnāt seem clearly laid out in the doc, but it sounds to me like your view is that there isnāt a ātime of perilsā and then sufficient technology for long run robustness.
I think you might find it useful to more clearly state your argument which seems very opaque in that linked document.
I disagree and think a time of perils seems quite likely given the potential for a singularity.
There is a bunch of discussion making this exact point in response to āMistakes in the moral mathematics of existential riskā (which seems mostly mistaken to me via the mechanism implicitly putting astronomically low probability on robust intersteller civilizations).
Causes of X-risk which seem vastly higher than this include:
AI takeover supposing you grant that AI control is less valuable.
Autocratic control supposing you grant that autocratic control of the long run future is less valuable.
I mostly think x-risk is mostly non-extinction and almost all the action is in changing which entities have control over resources rather than reducing astronomical waste.
Perhaps you adopt a view in which you donāt care at all what happens with long run resources so long as any group hypothetically has the ability to utilize these resources? Otherwise, given the potential for lock in, it seems like influencing who has control is vastly more important than you seem to be highlighting.
(My guess is that āno-entity ends up being in a position where they could hypothetically utilize long run resourcesā is about 300x lower than other x-risk (perhaps 0.1% vs 30% all cause x-risk) which is vastly higher than your estimate.)
I also put vaster higher probability than you on extinction due to incredibly powerful bioweapons or other future technology, but this isnāt most of my concern.
I am mainly sceptical of the possibility of making worlds with astronomical value significantly more likely, regardless of whether the longterm annual probability of value dropping a lot tends to 0 or not.
I agree what I shared is not very clear, although I will probably post it roughly as is one of these days, and then eventually follow up.
It is unclear to me whether faster economic growth or technological progress imply a higher extinction risk. I would say this has generally been going down until now, except maybe from around 1939 (start of World War 2) to 1986 (when nuclear warheads peaked), although the fraction of people living in democracies increased 21.6 pp (= 0.156 + 0.183 - (0.0400 + 0.0833)) during this period.
I agree the probability of intersteller civilizations and astronomically valuable futures more broadly should not be astronomically low. For example, I guess it is fine to assume a 1 % chance on each order of magnitude between 1 and 10^100 human lives of future value. This is not my best guess, but it is just to give you a sense than I think astronomically valuable futures are plausible. However, I guess it is very hard to increase the probability of the astronomically valuable worlds.
I guess the probability of something like a global dictactorship by 2100 is many orders of magnitude higher than 10^-10, but I do not think it would be permanent. If it was, then I would guess the alternative would be worse.
I strongly endorse expected total hedonistic utilitarianism.
There are many concept of existential risk, so I prefer to focus on probabilities of clearly defined situations. One could think about existential risk from risk R as the relative increase in the expected value of the future if risk R was totally mitigated, but this is super hard to estimate in a way that the results are informative. I currently think it is better to assess interventions based on standard cost-effectiveness analyses.
My view is that the majority of bad-things-happen-with the cosmic endowment risk is downstream of AI takeover.
I generally donāt think looking at historical case studies will be super informative here.
I agree that doing the singularity faster doesnāt make things worse, Iām just noting that youāll go through a bunch of technology in a small amount of wall clock time.
Sure, but is the probability of it being permanent more like 0.05 or 10^-6? I would guess more like 0.05. (Given modern technology and particularly the possibility of AI and the singularity.)
It depends on the specific definition of global dictactorship and the number of years. However, the major problem is that I have very little to say about what will happen further than 100 years into the future other than thinking that whatever is happening will continue to change, and is not determined by what we do now.
By āpermanentā, I mean >10 billion years. By āglobalā, I mean āit ācontrolsā >80% of resources under earth originating civilization controlā. (Where control evolves with the extent to which technology allows for control.)
Thanks for clarifying! Based on that, and Wikipediaās definition of dictactorship as āan autocratic form of government which is characterized by a leader, or a group of leaders, who hold governmental powers with few to no limitationsā, I would say more like 10^-6. However, I do not think this matters, because that far into the future I would no longer be confident to say which form of government is better or worse.
As, in your argument is that you are skeptical on priors? I think Iām confused what the argument is here.
Separately, my view is that due to acausal trade, itās very likely that changing from human control to AI control looks less like āmaking worlds with astronomical value more likelyā and looks more like āshifting some resources across the entire continuous measureā. But, this mostly adds up to the same thing as creating astronomical value.
Yes, mostly that. As far as I can tell, the (posterior) counterfactual impact of interventions whose effects can be accurately measured, like ones in global health and development, decays to 0 as time goes by, and can be modelled as increasing the value of the world for a few years or decades, far from astronomically.
I personally do not think acausal trade considerations are action relevant, but, if I was to think along those lines, I would assume there is way more stuff to be acausally influenced which is weakly correlated with what humans do than that is strongly correlated. So the probability of influencing more stuff acausally should still decrease with value, and I guess the decrease in the probability density would be faster than the increase in value, such that value density decreases with value. In this case, the expected value from astronomical acausal trades would still be super low.