Here are some very brief takes on the CCM web app now that RP has had a chance to iron out any initial bugs. I’m happy to elaborate more on any of these comments.
Some praise
This is an extremely ambitious project, and it’s very surprising that this is the first unified model of this type I’ve seen (though I’m sure various people have their own private models).
I have a bunch of quantitative models on cause prio sub-questions, but I don’t like to share these publicly because of the amount of context that’s required to interpret them (and because the methodology is often pretty unrefined) - props to RP for sharing theirs!
I could see this product being pretty useful to new funders who have a lot of flexibility over where donations go.
I think the intra-worldview models (e.g. comparing animal welfare interventions) seem reasonable to me (though I only gave these a quick glance)
I see this as a solid contribution to cause prioritisation efforts and I admire the focus on trying to do work that people might actually use—rather than just producing a paper with no accompanying tool.
Some critiques
I think RP underrates the extent to which their default values will end up being the defaults for model users (particularly some of the users they most want to influence)
I think the default values are (in my personal view) pretty far from my values or the mean of values for people who have thought hard about these topics in the EA community.
The x-risk model in particular seems to make bake-in quite conservative assumptions (medium-high confidence)
I found it difficult to provide very large numbers on future population per star—I think with current rates of economic and compute growth, the number of digital people could be extremely high very quickly.
I think some x-risk interventions could plausibly have very long run effects on x-risk (e.g. by building an aligned super intelligence)
The x-risk model seems to confuse existential risk and extinction risk (medium confidence—maybe this was explained somewhere, and I missed it)
Using the model felt clunky to me, it didn’t handle extremely large values well and it made iterating on values difficult, it’s not the kind of thing that you can “play with” imo.
Some improvements I’d like to see
I’d be interested in seeing RP commission some default values from researchers/EAs who can explain their suggested values well.
I would like for the overall app to feel more polished/responsive/usable—idk how much this would cost. I’d guess it’s at least a month’s work for a competent dev, maybe more.
I think RP underrates the extent to which their default values will end up being the defaults for model users (particularly some of the users they most want to influence)
This is a fair criticism: we started this project with the plan of providing somewhat authoritative numbers but discovered this to be more difficult than we initially expected and instead opted to express significant skepticism about the default choices. Where there was controversy (for instance, in how many years forward we should look), we opted for middle-of-the-road choices. I agree that it would add a lot of value to get reasonable and well-thought-out defaults. Maybe the best way to approach controversy would be to opt for different sets of parameter defaults that users could toggle between based on what different people in the community think.
I found it difficult to provide very large numbers on future population per star—I think with current rates of economic and compute growth, the number of digital people could be extremely high very quickly.
The ability to try to represent digital people with populations per star was a last-minute choice. We originally just aimed for that parameter to represent human populations. (It isn’t even completely obvious to me that stars are the limiting factor on the number of digital people.) However, I also think these things don’t matter since the main aim of the project isn’t really affected by exactly how valuable x-risk projects are in expectation. If you think there may be large populations, the model is going to imply incredibly high rates of return on extinction risk work. Whether those are the obvious choice or not depends not on exactly how high the return, but on how you feel about the risk, and the risks won’t change with massively higher populations.
I think some x-risk interventions could plausibly have very long run effects on x-risk (e.g. by building an aligned super intelligence)
If you think we’ll likely have an aligned super-intelligence within 100 years, then you might try to model this by setting risks very low after the next century and treating your project as just a small boost on its eventual discovery. However, you might not think that either superaligned AI or extinction is inevitable. One thing we don’t try to do is model trajectory changes, and those seem potentially hugely significant, but also rather difficult to model with any degree of confidence.
The x-risk model seems to confuse existential risk and extinction risk (medium confidence—maybe this was explained somewhere, and I missed it)
We distinguish extinction risk from risks of sub-extinction catastrophes, but we don’t model any kind of as-bad-as-extinction risks.
The x-risk model in particular seems to make bake-in quite conservative assumptions (medium-high confidence)
Note less conservative assumptions for existential risk interventions make them even less comparable with neartermist ones. Extending the time horizon beyond 3023 increases the cost-effectiveness of existential risk interventions, but not that of neartermist ones. Under a longtermist view where longterm effects dominate, it is crucial to model the longterm effects of neartermist interventions, but these are not in the model. So as of now I do not think it is that useful to compare longtermist with neartermist interventions.
That’s fair, though I personally would be happy to just factor in neartermist interventions to marginal changes in economic growth (which in most cases I expect to be negligible) in the absence of some causal mechanism by which I should expect some neartermist intervention to have an outsized influence on the long-run future.
Thanks for following up! How about assessing the benefits of both global catastrophic risk (GCR) and neartermist interventions in terms of lives saved, but weighting these by a function which increases as population size decreases? Saving lives is an output in both types of intervention, but neartermist interventions save lives at a higher population size than GCR ones. For reference:
Carl Shulman seemed to suggest in a post on the flow-through effects of saving a life that the respective indirect longterm effects, in terms of the time by which humanity’s trajectory is advanced, are inversely proportional to the population when the life is saved[1].
Based on the above, and assuming a power law for the ratio between the pre- and post-catastrophe (human) population, and a constant cost to save a life as a function of such ratio, it looks like saving lives in normal times is better to improve the longterm future than doing so in catastrophes.
For example, suppose one saved a drowning child 10,000 years ago, when the human population was estimated to be only in the millions. For convenience, we’ll posit a little over 7 million, 1/1000th of the current population. Since the child would add to population pressures on food supplies and disease risk, the effective population/economic boost could range from a fraction of a lifetime to a couple of lifetimes (via children), depending on the frequency of famine conditions. Famines were not annual and population fluctuated on a time scale of decades, so I will use 20 years of additional life expectancy.
So, for ~ 20 years the ancient population would be 1⁄7,000,000th greater, and economic output/technological advance. We might cut this to 1⁄10,000,000 to reflect reduced availability of other inputs, although increasing returns could cut the other way. Using 1⁄10,000,000 cumulative world economic output would reach the same point ~ 1⁄500,000th of a year faster. An extra 1⁄500,000th of a year with around our current population of ~7 billion would amount to an additional ~14,000 life -years, 700 times the contemporary increase in life years lived. Moreover, those extra lives on average have a higher standard of living than their ancient counterparts.
Readers familiar with Nick Bostrom’s paper on astronomical waste will see that this is a historical version of the same logic: when future populations will be far larger, expediting that process even slightly can affect the existence of many people. We cut off our analysis with current populations, but the greater the population this growth process will reach, the greater long-run impact of technological speedup from saving ancient lives.
it looks like saving lives in normal times is better to improve the longterm future than doing so in catastrophes.
Both seem negligible in the effect on the long term future without some more specific causal mechanism other than “things go faster” right?
Like I would guess that the vast majority of risk (now) is anthropogenic risk and anthropogenic risk should be unaffected by just speeding things up (or plausibly higher if it causes things to go faster at critical point rather than just getting to the critical point sooner).
And astronomical waste itself is also negligible (about 1⁄10 billion per year).
As far as I can tell, Carl doesn’t overcome this basic argument in his post and it is very unclear to me if he is even trying to argue “saving lives substantially improves the long run future”. I think he is mostly just using the past as an analogy for the current case for longtermism?
Both seem negligible in the effect on the long term future without some more specific causal mechanism other than “things go faster” right?
I guess you are thinking that multiplying a non-negligible reduction in the nearterm risk of human extinction per cost (e.g. 2.17*10^-13 per dollar[1]) by an astronomical expected value of the future (e.g. 1.40*10^52 human lives[2]) will result in an astronomically high cost-effectiveness (e.g. 3.04*10^39 life/$). However, this calculation assumes the reduction in nearterm extinction risk equals the relative increase in the expected value of the future, whereas I think the latter is astronomically lower.
And astronomical waste itself is also negligible (about 1⁄10 billion per year).
It is unclear to me whether existential risk is higher than 10^-10 per year. Iamopento best guesses of an annual extinction risk of 10^-8, and a probability of 1 % of extinction being an existential catastrophe[3], which would lead to an annual existential risk of 10^-10.
As far as I can tell, Carl doesn’t overcome this basic argument in his post and it is very unclear to me if he is even trying to argue “saving lives substantially improves the long run future”. I think he is mostly just using the past as an analogy for the current case for longtermism?
I agree Carl was not trying to argue for saving lives in normal times being among the most cost-effective ways of improving the longterm future.
Median cost-effectiveness bar for mitigating existential risk I collected. The bar does not respect extinction risk, but I assume the people who provided the estimates would have guessed similar values for extinction risk.
Mean of a loguniform distribution with minimum and maximum of 10^23 and 10^54 lives. The minimum is the estimate for “an extremely conservative reader” obtained in Table 3 of Newberry 2021. The maximum is the largest estimate in Table 1 of Newberry 2021, determined for the case where all the resources of the affectable universe support digital persons. The upper bound can be 10^30 times as high if civilization “aestivate[s] until the far future in order to exploit the low temperature environment”, in which computations are more efficient. Using a higher bound does not qualitatively change my point.
I estimated a 0.0513 % chance of not fully recovering from a repetition of the last mass extinction 66 M years ago, the Cretaceous–Paleogene extinction event. If biological humans go extinct because of advanced AI, I guess it is very likely they will have suitable successors then, either in the form of advanced AI or some combinations between it and humans.
and a probability of 1 % of extinction being an existential catastrophe
I think you should probably have a higher probability on some unknown filter making it less likely that intelligent civilization re-evolves. (Given anthropics.)
I’d say 20% chance that intelligent life doesn’t re-evolve on earth due to this mechanism.
There are also potentially aliens, which is perhaps a factor of 2 getting me to 10% chance of no group capable of using resources conditional on literal extinction of all intelligent civilization on earth. (Which is 10x higher than your estimate.)
I also think that I’d prefer human control than the next evolved life and than aliens by a moderate amount due to similarity of values arguments.
I’ve now updated toward a higher chance life re-evolves and a lower chance on some unknown filter because we can see that the primate to intelligent civilization time gap is quite small.
That makes sense. It looks like humans branched off chimpanzees just 5.5 M years (= (5 + 6)/2*10^6) ago. Assuming the time from chimpanzees to a species similar to humans follows an exponential distribution with a mean equal to that time, the probability of not recovering after human extinction in the 1 billion years during which Earth will remain habitable would be only 1.09*10^-79 (= e^(-10^9/(5.5*10^6))). The probability of not recovering is higher due to model uncertainty. The time to recover may follow a different distribution.
In addition, recovery can be harder for other risks:
Catastrophes wiping out more species in humans’ evolutionary past (e.g. the impact of a large comet) would have a longer expected recovery time, and therefore imply a lower chance of recovery during the time Earth will remain habitable.
The estimates I provided in my comment were mainly illustrative. However, my 1 % chance of existential catastrophe conditional on human extinction was coming from my expectation that humans will be on board with going extinct in the vast majority of worlds where they go extinct in the next few centuries because their AI or posthuman descendents would live on.
whereas I think the latter is astronomically lower.
Your argument doesn’t seem clearly laid out in the doc, but it sounds to me like your view is that there isn’t a “time of perils” and then sufficient technology for long run robustness.
I think you might find it useful to more clearly state your argument which seems very opaque in that linked document.
I disagree and think a time of perils seems quite likely given the potential for a singularity.
There is a bunch of discussion making this exact point in response to “Mistakes in the moral mathematics of existential risk” (which seems mostly mistaken to me via the mechanism implicitly putting astronomically low probability on robust intersteller civilizations).
It is unclear to me whether existential risk is higher than 10^-10 per year.
Causes of X-risk which seem vastly higher than this include:
AI takeover supposing you grant that AI control is less valuable.
Autocratic control supposing you grant that autocratic control of the long run future is less valuable.
I mostly think x-risk is mostly non-extinction and almost all the action is in changing which entities have control over resources rather than reducing astronomical waste.
Perhaps you adopt a view in which you don’t care at all what happens with long run resources so long as any group hypothetically has the ability to utilize these resources? Otherwise, given the potential for lock in, it seems like influencing who has control is vastly more important than you seem to be highlighting.
(My guess is that “no-entity ends up being in a position where they could hypothetically utilize long run resources” is about 300x lower than other x-risk (perhaps 0.1% vs 30% all cause x-risk) which is vastly higher than your estimate.)
I also put vaster higher probability than you on extinction due to incredibly powerful bioweapons or other future technology, but this isn’t most of my concern.
Your argument doesn’t seem clearly laid out in the doc, but it sounds to me like your view is that there isn’t a “time of perils” and then sufficient technology for long run robustness.
I am mainly sceptical of the possibility of making worlds with astronomical value significantly more likely, regardless of whether the longterm annual probability of value dropping a lot tends to 0 or not.
I think you might find it useful to more clearly state your argument which seems very opaque in that linked document.
I agree what I shared is not very clear, although I will probably post it roughly as is one of these days, and then eventually follow up.
I disagree and think a time of perils seems quite likely given the potential for a singularity.
It is unclear to me whether faster economic growth or technological progress imply a higher extinction risk. I would say this has generally been going down until now, except maybe from around 1939 (start of World War 2) to 1986 (when nuclear warheads peaked), although the fraction of people living in democracies increased 21.6 pp (= 0.156 + 0.183 - (0.0400 + 0.0833)) during this period.
There is a bunch of discussion making this exact point in response to “Mistakes in the moral mathematics of existential risk” (which seems mostly mistaken to me via the mechanism implicitly putting astronomically low probability on robust intersteller civilizations).
I agree the probability of intersteller civilizations and astronomically valuable futures more broadly should not be astronomically low. For example, I guess it is fine to assume a 1 % chance on each order of magnitude between 1 and 10^100 human lives of future value. This is not my best guess, but it is just to give you a sense than I think astronomically valuable futures are plausible. However, I guess it is very hard to increase the probability of the astronomically valuable worlds.
Causes of X-risk which seem vastly higher than this include:
AI takeover supposing you grant that AI control is less valuable.
Autocratic control supposing you grant that autocratic control of the long run future is less valuable.
I mostly think x-risk is mostly non-extinction and almost all the action is in changing which entities have control over resources rather than reducing astronomical waste.
I guess the probability of something like a global dictactorship by 2100 is many orders of magnitude higher than 10^-10, but I do not think it would be permanent. If it was, then I would guess the alternative would be worse.
Perhaps you adopt a view in which you don’t care at all what happens with long run resources so long as any group hypothetically has the ability to utilize these resources? Otherwise, given the potential for lock in, it seems like influencing who has control is vastly more important than you seem to be highlighting.
(My guess is that “no-entity ends up being in a position where they could hypothetically utilize long run resources” is about 300x lower than other x-risk (perhaps 0.1% vs 30% all cause x-risk) which is vastly higher than your estimate.)
There are many concept of existential risk, so I prefer to focus on probabilities of clearly defined situations. One could think about existential risk from risk R as the relative increase in the expected value of the future if risk R was totally mitigated, but this is super hard to estimate in a way that the results are informative. I currently think it is better to assess interventions based on standard cost-effectiveness analyses.
It is unclear to me whether faster economic growth or technological progress imply a higher extinction risk. I would say this has generally been going down until now
My view is that the majority of bad-things-happen-with the cosmic endowment risk is downstream of AI takeover.
I generally don’t think looking at historical case studies will be super informative here.
I agree that doing the singularity faster doesn’t make things worse, I’m just noting that you’ll go through a bunch of technology in a small amount of wall clock time.
Sure, but is the probability of it being permanent more like 0.05 or 10^-6? I would guess more like 0.05. (Given modern technology and particularly the possibility of AI and the singularity.)
It depends on the specific definition of global dictactorship and the number of years. However, the major problem is that I have very little to say about what will happen further than 100 years into the future other than thinking that whatever is happening will continue to change, and is not determined by what we do now.
By “permanent”, I mean >10 billion years. By “global”, I mean “it ‘controls’ >80% of resources under earth originating civilization control”. (Where control evolves with the extent to which technology allows for control.)
Thanks for clarifying! Based on that, and Wikipedia’s definition of dictactorship as “an autocratic form of government which is characterized by a leader, or a group of leaders, who hold governmental powers with few to no limitations”, I would say more like 10^-6. However, I do not think this matters, because that far into the future I would no longer be confident to say which form of government is better or worse.
I am mainly sceptical of the possibility of making worlds with astronomical value significantly more likely, regardless of whether the longterm annual probability of value dropping a lot tends to 0 or not.
As, in your argument is that you are skeptical on priors? I think I’m confused what the argument is here.
Separately, my view is that due to acausal trade, it’s very likely that changing from human control to AI control looks less like “making worlds with astronomical value more likely” and looks more like “shifting some resources across the entire continuous measure”. But, this mostly adds up to the same thing as creating astronomical value.
As, in your argument is that you are skeptical on priors? I think I’m confused what the argument is here.
Yes, mostly that. As far as I can tell, the (posterior) counterfactual impact of interventions whose effects can be accurately measured, like ones in global health and development, decays to 0 as time goes by, and can be modelled as increasing the value of the world for a few years or decades, far from astronomically.
Separately, my view is that due to acausal trade, it’s very likely that changing from human control to AI control looks less like “making worlds with astronomical value more likely” and looks more like “shifting some resources across the entire continuous measure”. But, this mostly adds up to the same thing as creating astronomical value.
I personally do not think acausal trade considerations are action relevant, but, if I was to think along those lines, I would assume there is way more stuff to be acausally influenced which is weakly correlated with what humans do than that is strongly correlated. So the probability of influencing more stuff acausally should still decrease with value, and I guess the decrease in the probability density would be faster than the increase in value, such that value density decreases with value. In this case, the expected value from astronomical acausal trades would still be super low.
Here are some very brief takes on the CCM web app now that RP has had a chance to iron out any initial bugs. I’m happy to elaborate more on any of these comments.
Some praise
This is an extremely ambitious project, and it’s very surprising that this is the first unified model of this type I’ve seen (though I’m sure various people have their own private models).
I have a bunch of quantitative models on cause prio sub-questions, but I don’t like to share these publicly because of the amount of context that’s required to interpret them (and because the methodology is often pretty unrefined) - props to RP for sharing theirs!
I could see this product being pretty useful to new funders who have a lot of flexibility over where donations go.
I think the intra-worldview models (e.g. comparing animal welfare interventions) seem reasonable to me (though I only gave these a quick glance)
I see this as a solid contribution to cause prioritisation efforts and I admire the focus on trying to do work that people might actually use—rather than just producing a paper with no accompanying tool.
Some critiques
I think RP underrates the extent to which their default values will end up being the defaults for model users (particularly some of the users they most want to influence)
I think the default values are (in my personal view) pretty far from my values or the mean of values for people who have thought hard about these topics in the EA community.
The x-risk model in particular seems to make bake-in quite conservative assumptions (medium-high confidence)
I found it difficult to provide very large numbers on future population per star—I think with current rates of economic and compute growth, the number of digital people could be extremely high very quickly.
I think some x-risk interventions could plausibly have very long run effects on x-risk (e.g. by building an aligned super intelligence)
The x-risk model seems to confuse existential risk and extinction risk (medium confidence—maybe this was explained somewhere, and I missed it)
Using the model felt clunky to me, it didn’t handle extremely large values well and it made iterating on values difficult, it’s not the kind of thing that you can “play with” imo.
Some improvements I’d like to see
I’d be interested in seeing RP commission some default values from researchers/EAs who can explain their suggested values well.
I would like for the overall app to feel more polished/responsive/usable—idk how much this would cost. I’d guess it’s at least a month’s work for a competent dev, maybe more.
Thanks for recording these thoughts!
Here are a few responses to the criticisms.
This is a fair criticism: we started this project with the plan of providing somewhat authoritative numbers but discovered this to be more difficult than we initially expected and instead opted to express significant skepticism about the default choices. Where there was controversy (for instance, in how many years forward we should look), we opted for middle-of-the-road choices. I agree that it would add a lot of value to get reasonable and well-thought-out defaults. Maybe the best way to approach controversy would be to opt for different sets of parameter defaults that users could toggle between based on what different people in the community think.
The ability to try to represent digital people with populations per star was a last-minute choice. We originally just aimed for that parameter to represent human populations. (It isn’t even completely obvious to me that stars are the limiting factor on the number of digital people.) However, I also think these things don’t matter since the main aim of the project isn’t really affected by exactly how valuable x-risk projects are in expectation. If you think there may be large populations, the model is going to imply incredibly high rates of return on extinction risk work. Whether those are the obvious choice or not depends not on exactly how high the return, but on how you feel about the risk, and the risks won’t change with massively higher populations.
If you think we’ll likely have an aligned super-intelligence within 100 years, then you might try to model this by setting risks very low after the next century and treating your project as just a small boost on its eventual discovery. However, you might not think that either superaligned AI or extinction is inevitable. One thing we don’t try to do is model trajectory changes, and those seem potentially hugely significant, but also rather difficult to model with any degree of confidence.
We distinguish extinction risk from risks of sub-extinction catastrophes, but we don’t model any kind of as-bad-as-extinction risks.
Thanks for sharing, Caleb.
Note less conservative assumptions for existential risk interventions make them even less comparable with neartermist ones. Extending the time horizon beyond 3023 increases the cost-effectiveness of existential risk interventions, but not that of neartermist ones. Under a longtermist view where longterm effects dominate, it is crucial to model the longterm effects of neartermist interventions, but these are not in the model. So as of now I do not think it is that useful to compare longtermist with neartermist interventions.
That’s fair, though I personally would be happy to just factor in neartermist interventions to marginal changes in economic growth (which in most cases I expect to be negligible) in the absence of some causal mechanism by which I should expect some neartermist intervention to have an outsized influence on the long-run future.
Thanks for following up! How about assessing the benefits of both global catastrophic risk (GCR) and neartermist interventions in terms of lives saved, but weighting these by a function which increases as population size decreases? Saving lives is an output in both types of intervention, but neartermist interventions save lives at a higher population size than GCR ones. For reference:
Carl Shulman seemed to suggest in a post on the flow-through effects of saving a life that the respective indirect longterm effects, in terms of the time by which humanity’s trajectory is advanced, are inversely proportional to the population when the life is saved[1].
Based on the above, and assuming a power law for the ratio between the pre- and post-catastrophe (human) population, and a constant cost to save a life as a function of such ratio, it looks like saving lives in normal times is better to improve the longterm future than doing so in catastrophes.
Here is the relevant excerpt:
Both seem negligible in the effect on the long term future without some more specific causal mechanism other than “things go faster” right?
Like I would guess that the vast majority of risk (now) is anthropogenic risk and anthropogenic risk should be unaffected by just speeding things up (or plausibly higher if it causes things to go faster at critical point rather than just getting to the critical point sooner).
And astronomical waste itself is also negligible (about 1⁄10 billion per year).
As far as I can tell, Carl doesn’t overcome this basic argument in his post and it is very unclear to me if he is even trying to argue “saving lives substantially improves the long run future”. I think he is mostly just using the past as an analogy for the current case for longtermism?
Thanks for the comment, Ryan!
I guess you are thinking that multiplying a non-negligible reduction in the nearterm risk of human extinction per cost (e.g. 2.17*10^-13 per dollar[1]) by an astronomical expected value of the future (e.g. 1.40*10^52 human lives[2]) will result in an astronomically high cost-effectiveness (e.g. 3.04*10^39 life/$). However, this calculation assumes the reduction in nearterm extinction risk equals the relative increase in the expected value of the future, whereas I think the latter is astronomically lower.
It is unclear to me whether existential risk is higher than 10^-10 per year. I am open to best guesses of an annual extinction risk of 10^-8, and a probability of 1 % of extinction being an existential catastrophe[3], which would lead to an annual existential risk of 10^-10.
I agree Carl was not trying to argue for saving lives in normal times being among the most cost-effective ways of improving the longterm future.
Median cost-effectiveness bar for mitigating existential risk I collected. The bar does not respect extinction risk, but I assume the people who provided the estimates would have guessed similar values for extinction risk.
Mean of a loguniform distribution with minimum and maximum of 10^23 and 10^54 lives. The minimum is the estimate for “an extremely conservative reader” obtained in Table 3 of Newberry 2021. The maximum is the largest estimate in Table 1 of Newberry 2021, determined for the case where all the resources of the affectable universe support digital persons. The upper bound can be 10^30 times as high if civilization “aestivate[s] until the far future in order to exploit the low temperature environment”, in which computations are more efficient. Using a higher bound does not qualitatively change my point.
I estimated a 0.0513 % chance of not fully recovering from a repetition of the last mass extinction 66 M years ago, the Cretaceous–Paleogene extinction event. If biological humans go extinct because of advanced AI, I guess it is very likely they will have suitable successors then, either in the form of advanced AI or some combinations between it and humans.
I think you should probably have a higher probability on some unknown filter making it less likely that intelligent civilization re-evolves. (Given anthropics.)
I’d say 20% chance that intelligent life doesn’t re-evolve on earth due to this mechanism.
There are also potentially aliens, which is perhaps a factor of 2 getting me to 10% chance of no group capable of using resources conditional on literal extinction of all intelligent civilization on earth. (Which is 10x higher than your estimate.)
I also think that I’d prefer human control than the next evolved life and than aliens by a moderate amount due to similarity of values arguments.
I’ve now updated toward a higher chance life re-evolves and a lower chance on some unknown filter because we can see that the primate to intelligent civilization time gap is quite small.
That makes sense. It looks like humans branched off chimpanzees just 5.5 M years (= (5 + 6)/2*10^6) ago. Assuming the time from chimpanzees to a species similar to humans follows an exponential distribution with a mean equal to that time, the probability of not recovering after human extinction in the 1 billion years during which Earth will remain habitable would be only 1.09*10^-79 (= e^(-10^9/(5.5*10^6))). The probability of not recovering is higher due to model uncertainty. The time to recover may follow a different distribution.
In addition, recovery can be harder for other risks:
Catastrophes wiping out more species in humans’ evolutionary past (e.g. the impact of a large comet) would have a longer expected recovery time, and therefore imply a lower chance of recovery during the time Earth will remain habitable.
As I said above, I estimated a 0.0513 % chance of not fully recovering from a repetition of the last mass extinction 66 M years ago, the Cretaceous–Paleogene extinction event.
A rogue AI would not allow another species to take control.
The estimates I provided in my comment were mainly illustrative. However, my 1 % chance of existential catastrophe conditional on human extinction was coming from my expectation that humans will be on board with going extinct in the vast majority of worlds where they go extinct in the next few centuries because their AI or posthuman descendents would live on.
Your argument doesn’t seem clearly laid out in the doc, but it sounds to me like your view is that there isn’t a “time of perils” and then sufficient technology for long run robustness.
I think you might find it useful to more clearly state your argument which seems very opaque in that linked document.
I disagree and think a time of perils seems quite likely given the potential for a singularity.
There is a bunch of discussion making this exact point in response to “Mistakes in the moral mathematics of existential risk” (which seems mostly mistaken to me via the mechanism implicitly putting astronomically low probability on robust intersteller civilizations).
Causes of X-risk which seem vastly higher than this include:
AI takeover supposing you grant that AI control is less valuable.
Autocratic control supposing you grant that autocratic control of the long run future is less valuable.
I mostly think x-risk is mostly non-extinction and almost all the action is in changing which entities have control over resources rather than reducing astronomical waste.
Perhaps you adopt a view in which you don’t care at all what happens with long run resources so long as any group hypothetically has the ability to utilize these resources? Otherwise, given the potential for lock in, it seems like influencing who has control is vastly more important than you seem to be highlighting.
(My guess is that “no-entity ends up being in a position where they could hypothetically utilize long run resources” is about 300x lower than other x-risk (perhaps 0.1% vs 30% all cause x-risk) which is vastly higher than your estimate.)
I also put vaster higher probability than you on extinction due to incredibly powerful bioweapons or other future technology, but this isn’t most of my concern.
I am mainly sceptical of the possibility of making worlds with astronomical value significantly more likely, regardless of whether the longterm annual probability of value dropping a lot tends to 0 or not.
I agree what I shared is not very clear, although I will probably post it roughly as is one of these days, and then eventually follow up.
It is unclear to me whether faster economic growth or technological progress imply a higher extinction risk. I would say this has generally been going down until now, except maybe from around 1939 (start of World War 2) to 1986 (when nuclear warheads peaked), although the fraction of people living in democracies increased 21.6 pp (= 0.156 + 0.183 - (0.0400 + 0.0833)) during this period.
I agree the probability of intersteller civilizations and astronomically valuable futures more broadly should not be astronomically low. For example, I guess it is fine to assume a 1 % chance on each order of magnitude between 1 and 10^100 human lives of future value. This is not my best guess, but it is just to give you a sense than I think astronomically valuable futures are plausible. However, I guess it is very hard to increase the probability of the astronomically valuable worlds.
I guess the probability of something like a global dictactorship by 2100 is many orders of magnitude higher than 10^-10, but I do not think it would be permanent. If it was, then I would guess the alternative would be worse.
I strongly endorse expected total hedonistic utilitarianism.
There are many concept of existential risk, so I prefer to focus on probabilities of clearly defined situations. One could think about existential risk from risk R as the relative increase in the expected value of the future if risk R was totally mitigated, but this is super hard to estimate in a way that the results are informative. I currently think it is better to assess interventions based on standard cost-effectiveness analyses.
My view is that the majority of bad-things-happen-with the cosmic endowment risk is downstream of AI takeover.
I generally don’t think looking at historical case studies will be super informative here.
I agree that doing the singularity faster doesn’t make things worse, I’m just noting that you’ll go through a bunch of technology in a small amount of wall clock time.
Sure, but is the probability of it being permanent more like 0.05 or 10^-6? I would guess more like 0.05. (Given modern technology and particularly the possibility of AI and the singularity.)
It depends on the specific definition of global dictactorship and the number of years. However, the major problem is that I have very little to say about what will happen further than 100 years into the future other than thinking that whatever is happening will continue to change, and is not determined by what we do now.
By “permanent”, I mean >10 billion years. By “global”, I mean “it ‘controls’ >80% of resources under earth originating civilization control”. (Where control evolves with the extent to which technology allows for control.)
Thanks for clarifying! Based on that, and Wikipedia’s definition of dictactorship as “an autocratic form of government which is characterized by a leader, or a group of leaders, who hold governmental powers with few to no limitations”, I would say more like 10^-6. However, I do not think this matters, because that far into the future I would no longer be confident to say which form of government is better or worse.
As, in your argument is that you are skeptical on priors? I think I’m confused what the argument is here.
Separately, my view is that due to acausal trade, it’s very likely that changing from human control to AI control looks less like “making worlds with astronomical value more likely” and looks more like “shifting some resources across the entire continuous measure”. But, this mostly adds up to the same thing as creating astronomical value.
Yes, mostly that. As far as I can tell, the (posterior) counterfactual impact of interventions whose effects can be accurately measured, like ones in global health and development, decays to 0 as time goes by, and can be modelled as increasing the value of the world for a few years or decades, far from astronomically.
I personally do not think acausal trade considerations are action relevant, but, if I was to think along those lines, I would assume there is way more stuff to be acausally influenced which is weakly correlated with what humans do than that is strongly correlated. So the probability of influencing more stuff acausally should still decrease with value, and I guess the decrease in the probability density would be faster than the increase in value, such that value density decreases with value. In this case, the expected value from astronomical acausal trades would still be super low.