1. It’s a priori extremely unlikely that we’re at the hinge of history
Claim 1
I want to push back on the idea of setting the “ur-prior” at 1 in 100,000, which seems far too low to me. I also will critique the method that arrived at that number, and propose a method of determining the prior that seems superior to me.
(One note before that: I’m going to ignore the possibility that the hingiest century could be in the past and assume that we are just interested in the question of how probable it is that the current century is hingier than any future century.)
First, to argue that 1 in 100,000 is too low: The hingiest century of the future must occur before civilization goes extinct. Therefore, one’s prior that the current century is the hingiest century of the future must be at least as high as one’s credence that civilization will go extinct in the current century. I think this is already (significantly) greater than 1 in 100,000.
I’ll come back to this idea when I propose my method of determining the prior, but first to critique yours:
The method you used to come up with the 1 in 100,000 prior that our current century is hingier than any future century was to estimate the expected number of centuries that civilization will survive (1,000,000) and then to try to “[restrict] ourselves to a uniform prior over the first 10%” of that expected number of centuries because “the number of future people is decreasing every century.”
(Note that while I think the adjustment from 10^-6 to 10^-5 is an adjustment for a good reason in the right direction, I think it can be left out of the prior: You can update on the fact that “the number of future people is decreasing every century” (and other things) later after determining the prior.)
Now to critique the method Will used of arriving at the 1 in 1,000,000 prior. It basically starts with an implicit probability distribution for when civilization is going to go extinct (good), but then compresses that into an average expected number of centuries that civilization is going to survive and (mistakenly) essentially assumes that civilization is going to last precisely that long. It then computes one over the average expected number of centuries to get the base rate that a given century is the hingiest (determining a base rate is good, but this isn’t the right way).
I propose that a better method is that one should start with the same implicit probability distribution for the expected lifespan of civilization, except make it explicit, and do the same base rate calculation but for each discrete possible length of civilization (1 century, 2 centuries, etc) instead of compressing the probability distribution for the expected lifespan of civilization into an average expected number of centuries.
That is, I’d argue that one’s prior that the current century is the hingiest century of the future should be equal to one’s credence that civilization will go extinct in the current century plus 1⁄2 times one’s credence that civilization will go extinct in the second century (since there will then be two possible centuries and we are calculating a base rate), plus 1⁄3 times one’s credence that civilization will go extinct in the third century (this is the third base rate we are summing), etc.
From my “1000 Century Model”, assuming a 1% per century risk of extinction per century for 1000 centuries, the prior that the first century is the hingiest is ~4.65%.
From my “90% Likely to Survive 999 Centuries Model”, assuming a 10% chance of extinction in the first century, and a 0% chance of extinction every year thereafter until the 1000th century, and a 100% chance of extinction in the 1000th century, my method gives a prior of ~10.09% that the first century is the hingiest. On the other hand, since the expected number of centuries is ~900 years, MacAskill’s method gives an initial prior of ~0.111% and a prior of ~1.111% after “[restricting] ourselves to a uniform prior over the first 10% [of expected centuries]”. Both priors calculated using MacAskill’s method are below the 10% rate of extinction in the first century, which (I claim again) obviously means they are too low.
Using a distribution over possible futures seems important. The specific method you propose seems useful for getting a better picture of maxi{P(century i most leveraged)}. However, what we want in order to make decisions is something more akin to maxi{E[leverage of century i]}. The most obvious difference is that scenarios in which the future is short and there is little one can do about it score highly on expected ranking and low on expected value. I am unclear on whether a flat prior makes sense for expectancy, but it seems more reasonable than for probability.
Of course, even maxi{E[leverage of century i]} does not accurately reflect what we are looking for. Similarly to Gregory_Lewis’ comment, the decision-relevant thing (if ‘punting to the future’ is possible at all) is closer still to maxi{E[what we will assess the leverage of century i to be at the time]}. i.e. whether we will have higher expected leverage in some future century according to our beliefs at that time. Thinking this through, I also find it plausible that even this does not make sense when using the definitions in the post, and will make a related top-level comment.
While I agree with you that maxi(P(century i most leveraged)) is not that action relevant, it is what Will is analyzing in the post, and think that William Kiely’s suggested prior seems basically reasonable for answering that question. As Will said explicitly in another comment:
Agree that it might well be that even though one has a very low credence in HoH, one should still act in the same way. (e.g. because if one is not at HoH, one is a sim, and your actions don’t have much impact).
I do think that the focus on maxi(P(century i most leveraged)) is the part of the post that I am least satisfied by, and that makes it hardest to engage with it, since I don’t really know why we care about the question of “are we in the most influential time in history?”. What we actually care about is the effectiveness of our interventions to give resources to the future, and the marginal effectiveness of those resources in the future, both of which are quite far removed from that question (because of the difficulties of sending resources to the future, and the fact that the answer to that question makes overall only a small difference for the total magnitude of the impact of any individual’s actions).
I agree that, among other things, discussion of mechanisms for sending resources to the future would needed to make such a decision. I figured that all these other considerations were deliberately excluded from this post to keep its scope manageable.
However, I do think that one can interpret the post as making claims about a more insightful kind of probability: the odds with which the current century is the one which will have the highest leverage-evaluated-at-the-time (in contrast to an omniscient view / end-of-time evaluation, which is what this thread mostly focuses on). I think that William_MacAskill’s main arguments are broadly compatible with both of these concepts, so one could get more out of the piece by interpreting it as about the more useful concept.
Formally, one could see the thing being analysed as
P(i=0 maximises E[leverage of century i∣Fi]),
where Fi is the knowledge available at the beginning of century i. If we and all future generations may freely move resources across time, and some things that are maybe omitted from the leverage definition are held constant, this expression tells us with what odds we are correct to do ‘direct work’ today as opposed to transfer resources one century forward. (Confusion about what ‘direct work’ means noted here.)
However, you seem to be right that as soon as you don’t hold other very important factors (such as how well one can send resources to the future) constant, those additional terms go inside the maximisation evaluation, and hence the above expression still isn’t that useful. (In particular, it can’t just be multiplied by an independent factor to get to a useable expression.)
(Also, I feel like I’m mathing from the hip here, so quite possibly I’ve got this quite wrong.)
Another reason to think that MacAskill’s method of determining the prior is flawed that I forgot to write down:
If one uses the same approach to come up with a prior that the second, third, fourth, X century is the hingiest century of the future, and then adds these priors together one ought to get 100%. This is true because exactly one of the set of all future centuries must be the hingiest century of the future. Yet with MacAskill’s method of determining the priors, when one sums all the individual priors that the hingiest century is century X, one gets a number far greater than 100%. That is, MacAskill’s estimate is that there are 1 million expected centuries ahead, so he uses a prior of 1 in 1 million that the first century is the hingiest (before the arbitrary 10x adjustment). However, his model assumes that it’s possible that civilization could last as long as 10 billion centuries (1 trillion years). So what is his prior that e.g. the 2 billionth century is the hingiest? 1 in 1 million also? Surely this isn’t reasonable, for if one uses a prior of 1 in 1 million for all 10 billion possible centuries then, one’s prior expectation that one of the 10 billion centuries that civilization will possible live through is 10,000 (aka 1,000,000%). One’s credence in this ought to be 1 (100%) by definition.
My method of determining the prior doesn’t have this problem. On the contrary, as Column J of my linked spreadsheet from the previous comment shows, the prior probability that the Hingiest Century is somewhere in the Century 1-1000 range (which I calculate by summing the individual priors for those thousand centuries) approaches 100% as the probability that civilization goes extinct in those first 1000 centuries approaches 100%.
Yeah, I think I messed up this bit. I should have used the harmonic mean rather than the arithmetic mean when averaging over possibilities of how many people will be in the future. Doing this brings the chance of being among the most influential person ever close to the chance of being the most influential person ever in a small-population universe. But then we get the issue that being the most influential person ever in a small-population universe is much less important than being the most influential person in a big-population universe. And it’s only the latter that we care about.
So what I really should have said (in my too-glib argument) is: for simplicity, just assume a high-population future, which are the action-relevant futures if you’re a longtermist. Then take a uniform prior over all times (or all people) in that high-population future. So my claim is: “In the action-relevant worlds, the frequency of ‘most important time’ (or ‘most important person’) is extremely low, and so should be our prior.”
Thanks for the reply, Will. I go by Will too by the way.
for simplicity, just assume a high-population future, which are the action-relevant futures if you’re a longtermist
This assumption seems dubious to me because it seems to ignore the nontrivial possibility that there is something like a Great Filter in our future that requires direct-work to overcome (or could benefit from direct-work).
That is, maybe if we solve one challenge right in our near-term future right (e.g. hand-off the future to benevolent AGI) then it will be more or less inevitable that life will flourish for billions of years, and if we fail to overcome that challenge then we will go extinct fairly soon. As long as you put a nontrivial probability on such a challenge existing in the short-term future and it being tractable, then even longtermist altruists in the small-population worlds (possibly ours) who try punting to the future / passing the buck instead of doing direct work and thus fail to make it past the Great-Filter-like challenge can (I claim, contrary to you by my understanding) be said to be living in an action-relevant world despite living in a small-population universe. This is because they had the power (even though they didn’t exercise it) to make the future a big-population universe.
From my “1000 Century Model”, assuming a 1% per century risk of extinction every year for 1000 years
Did you mean to say “assuming a 1% risk of extinction per century for 1000 centuries”? That seems to better fit the rest of what you said, and what’s in your model, as best I can tell.
I want to push back on the idea of setting the “ur-prior” at 1 in 100,000, which seems far too low to me. I also will critique the method that arrived at that number, and propose a method of determining the prior that seems superior to me.
(One note before that: I’m going to ignore the possibility that the hingiest century could be in the past and assume that we are just interested in the question of how probable it is that the current century is hingier than any future century.)
First, to argue that 1 in 100,000 is too low: The hingiest century of the future must occur before civilization goes extinct. Therefore, one’s prior that the current century is the hingiest century of the future must be at least as high as one’s credence that civilization will go extinct in the current century. I think this is already (significantly) greater than 1 in 100,000.
I’ll come back to this idea when I propose my method of determining the prior, but first to critique yours:
The method you used to come up with the 1 in 100,000 prior that our current century is hingier than any future century was to estimate the expected number of centuries that civilization will survive (1,000,000) and then to try to “[restrict] ourselves to a uniform prior over the first 10%” of that expected number of centuries because “the number of future people is decreasing every century.”
(Note that while I think the adjustment from 10^-6 to 10^-5 is an adjustment for a good reason in the right direction, I think it can be left out of the prior: You can update on the fact that “the number of future people is decreasing every century” (and other things) later after determining the prior.)
Now to critique the method Will used of arriving at the 1 in 1,000,000 prior. It basically starts with an implicit probability distribution for when civilization is going to go extinct (good), but then compresses that into an average expected number of centuries that civilization is going to survive and (mistakenly) essentially assumes that civilization is going to last precisely that long. It then computes one over the average expected number of centuries to get the base rate that a given century is the hingiest (determining a base rate is good, but this isn’t the right way).
I propose that a better method is that one should start with the same implicit probability distribution for the expected lifespan of civilization, except make it explicit, and do the same base rate calculation but for each discrete possible length of civilization (1 century, 2 centuries, etc) instead of compressing the probability distribution for the expected lifespan of civilization into an average expected number of centuries.
That is, I’d argue that one’s prior that the current century is the hingiest century of the future should be equal to one’s credence that civilization will go extinct in the current century plus 1⁄2 times one’s credence that civilization will go extinct in the second century (since there will then be two possible centuries and we are calculating a base rate), plus 1⁄3 times one’s credence that civilization will go extinct in the third century (this is the third base rate we are summing), etc.
I’ve modeled an example of this here: https://docs.google.com/spreadsheets/d/1AqlfY47EmdcsE0D_uR4UlC3IuQbsCXLsq7YdtEqnyjg/edit?usp=sharing
From my “1000 Century Model”, assuming a 1% per century risk of extinction per century for 1000 centuries, the prior that the first century is the hingiest is ~4.65%.
From my “90% Likely to Survive 999 Centuries Model”, assuming a 10% chance of extinction in the first century, and a 0% chance of extinction every year thereafter until the 1000th century, and a 100% chance of extinction in the 1000th century, my method gives a prior of ~10.09% that the first century is the hingiest. On the other hand, since the expected number of centuries is ~900 years, MacAskill’s method gives an initial prior of ~0.111% and a prior of ~1.111% after “[restricting] ourselves to a uniform prior over the first 10% [of expected centuries]”. Both priors calculated using MacAskill’s method are below the 10% rate of extinction in the first century, which (I claim again) obviously means they are too low.
Using a distribution over possible futures seems important. The specific method you propose seems useful for getting a better picture of maxi{P(century i most leveraged)}. However, what we want in order to make decisions is something more akin to maxi{E[leverage of century i]}. The most obvious difference is that scenarios in which the future is short and there is little one can do about it score highly on expected ranking and low on expected value. I am unclear on whether a flat prior makes sense for expectancy, but it seems more reasonable than for probability.
Of course, even maxi{E[leverage of century i]} does not accurately reflect what we are looking for. Similarly to Gregory_Lewis’ comment, the decision-relevant thing (if ‘punting to the future’ is possible at all) is closer still to maxi{E[what we will assess the leverage of century i to be at the time]}. i.e. whether we will have higher expected leverage in some future century according to our beliefs at that time. Thinking this through, I also find it plausible that even this does not make sense when using the definitions in the post, and will make a related top-level comment.
While I agree with you that maxi(P(century i most leveraged)) is not that action relevant, it is what Will is analyzing in the post, and think that William Kiely’s suggested prior seems basically reasonable for answering that question. As Will said explicitly in another comment:
I do think that the focus on maxi(P(century i most leveraged)) is the part of the post that I am least satisfied by, and that makes it hardest to engage with it, since I don’t really know why we care about the question of “are we in the most influential time in history?”. What we actually care about is the effectiveness of our interventions to give resources to the future, and the marginal effectiveness of those resources in the future, both of which are quite far removed from that question (because of the difficulties of sending resources to the future, and the fact that the answer to that question makes overall only a small difference for the total magnitude of the impact of any individual’s actions).
I agree that, among other things, discussion of mechanisms for sending resources to the future would needed to make such a decision. I figured that all these other considerations were deliberately excluded from this post to keep its scope manageable.
However, I do think that one can interpret the post as making claims about a more insightful kind of probability: the odds with which the current century is the one which will have the highest leverage-evaluated-at-the-time (in contrast to an omniscient view / end-of-time evaluation, which is what this thread mostly focuses on). I think that William_MacAskill’s main arguments are broadly compatible with both of these concepts, so one could get more out of the piece by interpreting it as about the more useful concept.
Formally, one could see the thing being analysed as
P(i=0 maximises E[leverage of century i∣Fi]),
where Fi is the knowledge available at the beginning of century i. If we and all future generations may freely move resources across time, and some things that are maybe omitted from the leverage definition are held constant, this expression tells us with what odds we are correct to do ‘direct work’ today as opposed to transfer resources one century forward. (Confusion about what ‘direct work’ means noted here.)
However, you seem to be right that as soon as you don’t hold other very important factors (such as how well one can send resources to the future) constant, those additional terms go inside the maximisation evaluation, and hence the above expression still isn’t that useful. (In particular, it can’t just be multiplied by an independent factor to get to a useable expression.)
(Also, I feel like I’m mathing from the hip here, so quite possibly I’ve got this quite wrong.)
Another reason to think that MacAskill’s method of determining the prior is flawed that I forgot to write down:
If one uses the same approach to come up with a prior that the second, third, fourth, X century is the hingiest century of the future, and then adds these priors together one ought to get 100%. This is true because exactly one of the set of all future centuries must be the hingiest century of the future. Yet with MacAskill’s method of determining the priors, when one sums all the individual priors that the hingiest century is century X, one gets a number far greater than 100%. That is, MacAskill’s estimate is that there are 1 million expected centuries ahead, so he uses a prior of 1 in 1 million that the first century is the hingiest (before the arbitrary 10x adjustment). However, his model assumes that it’s possible that civilization could last as long as 10 billion centuries (1 trillion years). So what is his prior that e.g. the 2 billionth century is the hingiest? 1 in 1 million also? Surely this isn’t reasonable, for if one uses a prior of 1 in 1 million for all 10 billion possible centuries then, one’s prior expectation that one of the 10 billion centuries that civilization will possible live through is 10,000 (aka 1,000,000%). One’s credence in this ought to be 1 (100%) by definition.
My method of determining the prior doesn’t have this problem. On the contrary, as Column J of my linked spreadsheet from the previous comment shows, the prior probability that the Hingiest Century is somewhere in the Century 1-1000 range (which I calculate by summing the individual priors for those thousand centuries) approaches 100% as the probability that civilization goes extinct in those first 1000 centuries approaches 100%.
Thanks, William!
Yeah, I think I messed up this bit. I should have used the harmonic mean rather than the arithmetic mean when averaging over possibilities of how many people will be in the future. Doing this brings the chance of being among the most influential person ever close to the chance of being the most influential person ever in a small-population universe. But then we get the issue that being the most influential person ever in a small-population universe is much less important than being the most influential person in a big-population universe. And it’s only the latter that we care about.
So what I really should have said (in my too-glib argument) is: for simplicity, just assume a high-population future, which are the action-relevant futures if you’re a longtermist. Then take a uniform prior over all times (or all people) in that high-population future. So my claim is: “In the action-relevant worlds, the frequency of ‘most important time’ (or ‘most important person’) is extremely low, and so should be our prior.”
Thanks for the reply, Will. I go by Will too by the way.
This assumption seems dubious to me because it seems to ignore the nontrivial possibility that there is something like a Great Filter in our future that requires direct-work to overcome (or could benefit from direct-work).
That is, maybe if we solve one challenge right in our near-term future right (e.g. hand-off the future to benevolent AGI) then it will be more or less inevitable that life will flourish for billions of years, and if we fail to overcome that challenge then we will go extinct fairly soon. As long as you put a nontrivial probability on such a challenge existing in the short-term future and it being tractable, then even longtermist altruists in the small-population worlds (possibly ours) who try punting to the future / passing the buck instead of doing direct work and thus fail to make it past the Great-Filter-like challenge can (I claim, contrary to you by my understanding) be said to be living in an action-relevant world despite living in a small-population universe. This is because they had the power (even though they didn’t exercise it) to make the future a big-population universe.
Did you mean to say “assuming a 1% risk of extinction per century for 1000 centuries”? That seems to better fit the rest of what you said, and what’s in your model, as best I can tell.
Yes, thank you for the correction!