I roughly think that there simply isn’t very strong evidence for this. I.e. I think it would be mistaken to have a highly resilient large credence in extinction risk eventually falling to ~0.0000001%, humanity or its descendants surviving for a billion years, or anything like that.
[ETA: Upon rereading, I realized the above is ambiguous. With “large” I was here referring to something stronger than “non-extreme”. E.g. I do think it’s defensible to believe that, e.g. “I’m like 90% confident that over the next 10 years my credence in information-based civilization surviving for 1 billion years won’t fall below 0.1%”, and indeed that’s a statement I would endorse. I think I’d start feeling skeptical if someone claimed there is no way they’d update to a credence below 40% or something like that.]
I think this is one of several reasons for why the “naive case” for focusing on extinction risk reduction fails. (Another example of such a reason is the fact that, for most known hazards, collapse short of extinction seems way more likely than immediate extinction, that as a consequence most interventions affect both the probability of extinction and the probability and trajectory of various collapse scenarios, and that the latter effect might dominate but has unclear sign.)
I think the most convincing response is a combination of the following. Note, however, that the last two mostly argue that we should be longtermists despite the case for billion-year futures being shaky rather than defenses of that case itself.
You are correct that within fixed models we can justifiably have extreme credences, e.g. for the probability of a specific result of 30 coin flips. However, I think the case for “modesty”—i.e. not ruling out very long futures—rests largely on model uncertainty, i.e. our inability to confidently identify the ‘correct’ model for reasoning about the length of the future.
For example, suppose I produce a coin from my pocket and ask you to estimate how likely it is that in my first 30 flips I get only heads. Your all-things-considered credence will be dominated by your uncertainty over whether my coin is strongly biased toward heads. Since 30 heads are vanishingly unlikely if the coin is fair, this is the case even if your prior says that most coins someone produces from their pocket are fair: “vanishingly unlikely” here is much stronger (in this case around 10−9) than your prior can justifiably be, i.e. “most coins” might defensibly refer to 90% or 99% or 99.99% but not 99.9999999%.
This insight that extremely low credences all-things-considered are often “forbidden” by model uncertainty is basically the point from Ord, Hillerbrand, & Sandberg (2008).
Note that I think it’s still true that there is a possible epistemic state (and probably even model we can write down now) that rules out very long futures with extreme confidence. The point just is that we won’t be able to get to that epistemic state in practice.
Overall, I think the lower bound on the all-things-considered credence we should have in some speculative scenario often comes down to understanding how “fundamental” our model uncertainty is. I.e. roughly: to get to models that have practically significant credence in the scenario in question, how fundamentally would I need to revise my best-guess model of the world?
E.g. if I’m asking whether the LHC will blow up the world, or whether it’s worth looking for the philosopher’s stone, then I would need to revise extremely fundamental aspects of my world model such as fundamental physics—we are justified in having pretty high credences in those.
By contrast, very long futures seem at least plausibly consistent with fundamental physics as well as plausible theories for how cultural evolution, technological progress, economics, etc. work.
It is here, and for this reason, that points like “but it’s conceivable that superintelligent AI will reduce extinction risk to near-zero” are significant.
Therefore, model uncertainty will push me toward a higher credence in a very long future than in the LHC blowing up the world (but even for the latter my credence is plausibly dominated by model uncertainty rather than my credence in this happening conditional on my model of physics being correct).
Longtermism goes through (i.e. it looks like we can have most impact by focusing on the long-term) on much less extreme time scales than 1 billion.
Some such less extreme time scales have “more defensible” reasons behind them, e.g. outside view considerations based on the survival of other species or the amount of time humanity or civilization have survived so far. The Lindy rule prior you describe is one example.
There is a wager for long futures: we can have much more impact if the future is long, so these scenarios might dominate our decision-making even if they are unlikely.
(NB I think this is a wager that is unproblematic only if we have independently established that the probability of the relevant futures isn’t vanishingly small. This is because of the standard problems around Pascal’s wager.)
That all being said, my views on this feel reasonably but not super resilient—like it’s “only” 10% I’ll have changed my mind about this in major ways in 2 years. I also think there is room for more work on how to best think about such questions (the Ord et al. paper is a great example), e.g. checking that this kind of reasoning doesn’t “prove too much” or leads to absurd conclusions when applied to other cases.
Thanks for this. I won’t respond to your second/third bullets; as you say it’s not a defense of the claim itself, and while it’s plausible to me that many conclusions go through on much shorter timelines, I still want to understand the basis for the actual arguments made as best I can. Not least because if I can’t defend such arguments, then my personal pitches for longtermism (both to myself and to others) will not include them; they and I will focus on the next e.g. 10,000 years instead.
On your first bullet:
You are correct that within fixed models we can justifiably have extreme credences, e.g. for the probability of a specific result of 30 coin flips. However, I think the case for “modesty”—i.e. not ruling out very long futures—rests largely on model uncertainty...
...This insight that extremely low credences all-things-considered are often “forbidden” by model uncertainty is basically the point from Ord, Hillerbrand, & Sandberg (2008).
I’ll go and read the paper you mention, but flagging that my coinflip example is more general than you seem to think. Probability theory has conjunctions even outside of simple fixed models, and it’s the conjunction, not the fixed model, which is forcing you to have extreme credences. At best, we may be able to define a certain class of events where such credences are ‘forbidden’ (this could well be what the paper tries to do). We would then need to make sure that no such event can be expressed as a conjunction of a very large number of other such events.
Concretely, P(Humanity survives one billion years) is the product of one million probabilities of surviving each millenia, conditional on having survivied up to that point. As a result, we either need to set some of the intervening probabilities like P(Humanity survivies the next millenia | Humanity has survived to the year 500,000,000 AD) extremely high, or we need to set the overall product extremely low. Setting everything to the range 0.01% − 99.99% is not an option, without giving up on arithmetic or probability theory. And of course, I could break the product into a billion-fold conjunction where each component was ‘survive the next year’ if I wanted to make the requirements even more extreme.
Note I think it is plausible such extremes can be justified, since it seems like a version of humanity that has survived 500,000 millenia really should have excellent odds of surviving the next millenium. Indeed, I think that if you actually write out the model uncertainty argument mathematically, what ends up happening here is the fact that humanity has survivied 500,000 millenia is massive overwhelming Bayesian evidence that the ‘correct’ model is one of the ones that makes such a long life possible, allowing you to reach very extreme credences about the then-future. This is somewhat analagous to the intuitive extreme credence most people have that they won’t die in the next second.
my coinflip example is more general than you seem to think. Probability theory has conjunctions even outside of simple fixed models, and it’s the conjunction, not the fixed model, which is forcing you to have extreme credences. At best, we may be able to define a certain class of events where such credences are ‘forbidden’ (this could well be what the paper tries to do).
I agree with everything you say in your reply. I think I simply partly misunderstood the point you were trying to make and phrased part of my response poorly. In particular, I agree that extreme credences aren’t ‘forbidden’ in general.
(Sorry, I think it would have been better if I had flagged that I had read your comment and written mine very quickly.)
I still think that the distinction between credence/probabilities within a model and credence that a modelis correct are is relevant here, for reasons such as:
I think it’s often harder to justify an extreme credence that a particular model is right than it is to justify an extreme probability within a model.
Often when it seems we have extreme credence in a model this just holds “at a certain level of detail”, and if we looked at a richer space of models that makes more fine-grained distinctions we’d say that our credence is distributed over a (potentially very large) family of models.
There is a difference between an extreme all-things-considered credence (i.e. in this simplified way of thinking about epistemics the ‘expected credence’ across models) and being highly confident in an extreme credence;
I think the latter is less often justified than the former. And again if it seems that the latter is justified, I think it’ll often be because an extreme amount of credence is distributed among different models, but all of these models agree about some event we’re considering. (E.g. ~all models agree that I wont’t spontaneously die in the next second, or that Santa Clause isn’t going to appear in my bedroom.)
When different models agree that some event is the conjunction of many others, then each model will have an extreme credence for some event but the models might disagree about for which events the credence is extreme.
Taken together (i.e. across events/decisions) your all-things-considered credences might look therefore look “funny” or “inconsistent” (by the light of any single model). E.g. you might have non-extreme all-things-considered credence in two events based on two different models that are inconsistent with each other, and each of which rules out one of the events with extreme probability but not the other.
I acknowledge that I’m making somewhat vague claims here, and that in order to have anything close to a satisfying philosophical account of what’s going on I would need to spell out what exactly I mean by “often” etc. (Because as I said I do agree that these claims don’t always hold!)
Some fixed models also support macroscopic probabilities of indefinite survival: e.g. if in each generation each individual has a number of descendants drawn from a Poisson distribution with parameter 1.1, then there’s a finite chance of extinction in each generation but these diminish fast enough (as the population gets enormous) that if you make it through an initial rocky period you’re pretty much safe.
That model is clearly too optimistic because it doesn’t admit crises with correlated problems across all the individuals in a generation. But then there’s a question about how high is the unavoidable background rate of such crises (i.e. ones that remain even if you have a very sophisticated and well-resourced attempt to prevent them).
On current understanding I think the lower bounds for the rate of exogenous such events rely on things like false vacuum decay (and maybe GRBs while we’re local enough), and those lower bounds are really quite low, so it’s fairly plausible that the true rate is really low (though also plausible it’s higher because there are risks that aren’t observed/understood).
Bounding endogenous risk seems a bit harder to reason about. I think that you can give kind of fairytale/handwaving existence proofs of stable political systems (which might however be utterly horrific to us). Then it’s at least sort of plausible that there would be systems which are simultaneously extremely stable and also desirable.
I won’t respond to your second/third bullets; as you say it’s not a defense of the claim itself, and while it’s plausible to me that many conclusions go through on much shorter timelines, I still want to understand the basis for the actual arguments made as best I can. Not least because if I can’t defend such arguments, then my personal pitches for longtermism (both to myself and to others) will not include them; they and I will focus on the next e.g. 10,000 years instead.
To be clear, this makes a lot of sense to me, and I emphatically agree that understanding the arguments is valuable independently from whether this immediately changes a practical conclusion.
I roughly think that there simply isn’t very strong evidence for this. I.e. I think it would be mistaken to have a highly resilient large credence in extinction risk eventually falling to ~0.0000001%, humanity or its descendants surviving for a billion years, or anything like that.
[ETA: Upon rereading, I realized the above is ambiguous. With “large” I was here referring to something stronger than “non-extreme”. E.g. I do think it’s defensible to believe that, e.g. “I’m like 90% confident that over the next 10 years my credence in information-based civilization surviving for 1 billion years won’t fall below 0.1%”, and indeed that’s a statement I would endorse. I think I’d start feeling skeptical if someone claimed there is no way they’d update to a credence below 40% or something like that.]
I think this is one of several reasons for why the “naive case” for focusing on extinction risk reduction fails. (Another example of such a reason is the fact that, for most known hazards, collapse short of extinction seems way more likely than immediate extinction, that as a consequence most interventions affect both the probability of extinction and the probability and trajectory of various collapse scenarios, and that the latter effect might dominate but has unclear sign.)
I think the most convincing response is a combination of the following. Note, however, that the last two mostly argue that we should be longtermists despite the case for billion-year futures being shaky rather than defenses of that case itself.
You are correct that within fixed models we can justifiably have extreme credences, e.g. for the probability of a specific result of 30 coin flips. However, I think the case for “modesty”—i.e. not ruling out very long futures—rests largely on model uncertainty, i.e. our inability to confidently identify the ‘correct’ model for reasoning about the length of the future.
For example, suppose I produce a coin from my pocket and ask you to estimate how likely it is that in my first 30 flips I get only heads. Your all-things-considered credence will be dominated by your uncertainty over whether my coin is strongly biased toward heads. Since 30 heads are vanishingly unlikely if the coin is fair, this is the case even if your prior says that most coins someone produces from their pocket are fair: “vanishingly unlikely” here is much stronger (in this case around 10−9) than your prior can justifiably be, i.e. “most coins” might defensibly refer to 90% or 99% or 99.99% but not 99.9999999%.
This insight that extremely low credences all-things-considered are often “forbidden” by model uncertainty is basically the point from Ord, Hillerbrand, & Sandberg (2008).
Note that I think it’s still true that there is a possible epistemic state (and probably even model we can write down now) that rules out very long futures with extreme confidence. The point just is that we won’t be able to get to that epistemic state in practice.
Overall, I think the lower bound on the all-things-considered credence we should have in some speculative scenario often comes down to understanding how “fundamental” our model uncertainty is. I.e. roughly: to get to models that have practically significant credence in the scenario in question, how fundamentally would I need to revise my best-guess model of the world?
E.g. if I’m asking whether the LHC will blow up the world, or whether it’s worth looking for the philosopher’s stone, then I would need to revise extremely fundamental aspects of my world model such as fundamental physics—we are justified in having pretty high credences in those.
By contrast, very long futures seem at least plausibly consistent with fundamental physics as well as plausible theories for how cultural evolution, technological progress, economics, etc. work.
It is here, and for this reason, that points like “but it’s conceivable that superintelligent AI will reduce extinction risk to near-zero” are significant.
Therefore, model uncertainty will push me toward a higher credence in a very long future than in the LHC blowing up the world (but even for the latter my credence is plausibly dominated by model uncertainty rather than my credence in this happening conditional on my model of physics being correct).
Longtermism goes through (i.e. it looks like we can have most impact by focusing on the long-term) on much less extreme time scales than 1 billion.
Some such less extreme time scales have “more defensible” reasons behind them, e.g. outside view considerations based on the survival of other species or the amount of time humanity or civilization have survived so far. The Lindy rule prior you describe is one example.
There is a wager for long futures: we can have much more impact if the future is long, so these scenarios might dominate our decision-making even if they are unlikely.
(NB I think this is a wager that is unproblematic only if we have independently established that the probability of the relevant futures isn’t vanishingly small. This is because of the standard problems around Pascal’s wager.)
That all being said, my views on this feel reasonably but not super resilient—like it’s “only” 10% I’ll have changed my mind about this in major ways in 2 years. I also think there is room for more work on how to best think about such questions (the Ord et al. paper is a great example), e.g. checking that this kind of reasoning doesn’t “prove too much” or leads to absurd conclusions when applied to other cases.
Thanks for this. I won’t respond to your second/third bullets; as you say it’s not a defense of the claim itself, and while it’s plausible to me that many conclusions go through on much shorter timelines, I still want to understand the basis for the actual arguments made as best I can. Not least because if I can’t defend such arguments, then my personal pitches for longtermism (both to myself and to others) will not include them; they and I will focus on the next e.g. 10,000 years instead.
On your first bullet:
I’ll go and read the paper you mention, but flagging that my coinflip example is more general than you seem to think. Probability theory has conjunctions even outside of simple fixed models, and it’s the conjunction, not the fixed model, which is forcing you to have extreme credences. At best, we may be able to define a certain class of events where such credences are ‘forbidden’ (this could well be what the paper tries to do). We would then need to make sure that no such event can be expressed as a conjunction of a very large number of other such events.
Concretely, P(Humanity survives one billion years) is the product of one million probabilities of surviving each millenia, conditional on having survivied up to that point. As a result, we either need to set some of the intervening probabilities like P(Humanity survivies the next millenia | Humanity has survived to the year 500,000,000 AD) extremely high, or we need to set the overall product extremely low. Setting everything to the range 0.01% − 99.99% is not an option, without giving up on arithmetic or probability theory. And of course, I could break the product into a billion-fold conjunction where each component was ‘survive the next year’ if I wanted to make the requirements even more extreme.
Note I think it is plausible such extremes can be justified, since it seems like a version of humanity that has survived 500,000 millenia really should have excellent odds of surviving the next millenium. Indeed, I think that if you actually write out the model uncertainty argument mathematically, what ends up happening here is the fact that humanity has survivied 500,000 millenia is massive overwhelming Bayesian evidence that the ‘correct’ model is one of the ones that makes such a long life possible, allowing you to reach very extreme credences about the then-future. This is somewhat analagous to the intuitive extreme credence most people have that they won’t die in the next second.
I agree with everything you say in your reply. I think I simply partly misunderstood the point you were trying to make and phrased part of my response poorly. In particular, I agree that extreme credences aren’t ‘forbidden’ in general.
(Sorry, I think it would have been better if I had flagged that I had read your comment and written mine very quickly.)
I still think that the distinction between credence/probabilities within a model and credence that a model is correct are is relevant here, for reasons such as:
I think it’s often harder to justify an extreme credence that a particular model is right than it is to justify an extreme probability within a model.
Often when it seems we have extreme credence in a model this just holds “at a certain level of detail”, and if we looked at a richer space of models that makes more fine-grained distinctions we’d say that our credence is distributed over a (potentially very large) family of models.
There is a difference between an extreme all-things-considered credence (i.e. in this simplified way of thinking about epistemics the ‘expected credence’ across models) and being highly confident in an extreme credence;
I think the latter is less often justified than the former. And again if it seems that the latter is justified, I think it’ll often be because an extreme amount of credence is distributed among different models, but all of these models agree about some event we’re considering. (E.g. ~all models agree that I wont’t spontaneously die in the next second, or that Santa Clause isn’t going to appear in my bedroom.)
When different models agree that some event is the conjunction of many others, then each model will have an extreme credence for some event but the models might disagree about for which events the credence is extreme.
Taken together (i.e. across events/decisions) your all-things-considered credences might look therefore look “funny” or “inconsistent” (by the light of any single model). E.g. you might have non-extreme all-things-considered credence in two events based on two different models that are inconsistent with each other, and each of which rules out one of the events with extreme probability but not the other.
I acknowledge that I’m making somewhat vague claims here, and that in order to have anything close to a satisfying philosophical account of what’s going on I would need to spell out what exactly I mean by “often” etc. (Because as I said I do agree that these claims don’t always hold!)
Some fixed models also support macroscopic probabilities of indefinite survival: e.g. if in each generation each individual has a number of descendants drawn from a Poisson distribution with parameter 1.1, then there’s a finite chance of extinction in each generation but these diminish fast enough (as the population gets enormous) that if you make it through an initial rocky period you’re pretty much safe.
That model is clearly too optimistic because it doesn’t admit crises with correlated problems across all the individuals in a generation. But then there’s a question about how high is the unavoidable background rate of such crises (i.e. ones that remain even if you have a very sophisticated and well-resourced attempt to prevent them).
On current understanding I think the lower bounds for the rate of exogenous such events rely on things like false vacuum decay (and maybe GRBs while we’re local enough), and those lower bounds are really quite low, so it’s fairly plausible that the true rate is really low (though also plausible it’s higher because there are risks that aren’t observed/understood).
Bounding endogenous risk seems a bit harder to reason about. I think that you can give kind of fairytale/handwaving existence proofs of stable political systems (which might however be utterly horrific to us). Then it’s at least sort of plausible that there would be systems which are simultaneously extremely stable and also desirable.
To be clear, this makes a lot of sense to me, and I emphatically agree that understanding the arguments is valuable independently from whether this immediately changes a practical conclusion.