The outside-view argument (in response to your first argument)
In the blog post, I stated the priors-based argument quite poorly—I thought this bit wouldn’t be where the disagreement was, so I didn’t spend much time on it. How wrong I was about that! For the article version (link), I tidied it up.
The key thing is that the way I’m setting priors is as a function from populations to credences: for any property F, your prior should be such that if there are n people in a population, the probability that you are in the m most F people in that population is m/n.
This falls out of the self-sampling assumption, that a rational agent’s priors locate her uniformly at random within each possible world. If you reject this way of setting priors then, by modus tollens, you reject the self-sampling assumption. That’s pretty interesting if so!
On this set-up of the argument (which is what was in my head but I hadn’t worked through), I don’t make any claims about how likely it is that we are part of a very long future. Only that, a priori, the probability that we’re *both* in a very large future *and* one of the most influential people ever is very low. For that reason, there aren’t any implications from that argument to claims about the magnitude of extinction risk this century. We could be comparatively un-influential in many ways: if extinction risk is high this century but continues to be high for very many centuries; if extinction risk is low this century and will be higher in coming centuries; if extinction risk is any level and we can’t do anything about it, or are not yet knowledgeable enough to choose actions wisely, or if longtermism is false. (etc)
Separately, I still don’t see the case for building earliness into our priors, rather than updating on the basis of finding oneself seemingly-early. Building earliness into your prior means you’ve got to give up on the very-plausible-seeming self-sampling assumption; means you’ve got to treat the predicate ‘is most influential’ differently than other predicates; has technical challenges; and the case in favour seems to rely on a posteriori observations about how the world works, like those you give in your post.
I still don’t see the case for building earliness into our priors, rather than updating on the basis of finding oneself seemingly-early.
If we’re doing things right, it shouldn’t matter whether we’re building earliness into our prior or updating on the basis of earliness.
Let the set H=”the 1e10 (i.e. 10 billion) most influential people who will ever live” and let E=”the 1e11 (i.e. 100 billion) earliest people who will ever live”. Assume that the future will contain 1e100 people. Let X be a randomly sampled person.
For our unconditional prior P(X in H), everyone agrees that uniform probability is appropriate, i.e., P(X in H) = 1e-90. (I.e. we’re not giving up on the self-sampling assumption.)
However, for our belief over P(X in H | X in E), i.e. the probability that a randomly chosen early person is one of the most influential people, some people argue we should utilise an e.g. exponential function where earlier people are more likely to be influential (which could be called a prior over “X in H” based on how early X is). However, it seems like you’re saying that we shouldn’t assess P(X in H | X in E) directly from such a prior, but instead get it from bayesian updates. So lets do that.
P(X in H | X in E) = P(X in E | X in H) * P(X in H) / P(X in E) = P(X in E | X in H) * 1e-90 / 1e-89 = P(X in E | X in H) * 1e-1 = P(X in E | X in H) / 10
So now we’ve switched over to instead making a guess about P(X in E | X in H), i.e. the probability that one of the 1e10 most influential people also is one of the 1e11 earliest people, and dividing by 10. That doesn’t seem much easier than making a guess about P(X in H | X in E), and it’s not obvious whether our intuitions here would lead us to expect more or less influentialness.
Also, the way that 1e-90 and 1e-89 are both extraordinarily unlikely, but divide out to becoming 1e-1, illustrates Buck’s point:
if you condition on us being at an early time in human history (which is an extremely strong condition, because it has incredibly low prior probability), it’s not that surprising for us to find ourselves at a hingey time.
“If we’re doing things right, it shouldn’t matter whether we’re building earliness into our prior or updating on the basis of earliness.”
Thanks, Lukas, I thought this was very clear and exactly right.
“So now we’ve switched over to instead making a guess about P(X in E | X in H), i.e. the probability that one of the 1e10 most influential people also is one of the 1e11 earliest people, and dividing by 10. That doesn’t seem much easier than making a guess about P(X in H | X in E), and it’s not obvious whether our intuitions here would lead us to expect more or less influentialness.”
That’s interesting, thank you—this statement of the debate has helped clarify things for me. It does seem to me that doing the update - going via P(X in E | X in H) rather than directly trying to assess P(X in H | X in E) - is helpful, but I’d understand the position of someone who wanted just to assess P(X in H | X in E) directly.
I think it’s helpful to assess P(X in E | X in H) because it’s not totally obvious how one should update on the basis of earliness. The arrow of causality and the possibility of lock-in over time definitely gives reasons in favor of influential people being earlier. But there’s still the big question of how great an update that should be. And the cumulative nature of knowledge and understanding gives reasons in favor thinking that later people are more likely to be more influential.
This seems important to me because, for someone claiming that we should think that we’re at the HoH, the update on the basis of earliness is doing much more work than updates on the basis of, say, familiar arguments about when AGI is coming and what will happen when it does. To me at least, that’s a striking fact and wouldn’t have been obvious before I started thinking about these things.
This seems important to me because, for someone claiming that we should think that we’re at the HoH, the update on the basis of earliness is doing much more work than updates on the basis of, say, familiar arguments about when AGI is coming and what will happen when it does. To me at least, that’s a striking fact and wouldn’t have been obvious before I started thinking about these things.
It seems to me the object level is where the action is, and the non-simulation Doomsday Arguments mostly raise a phantom consideration that cancels out (in particular, cancelling out re whether there is an influenceable lock-in event this century).
You could say a similar thing about our being humans rather than bacteria, which cumulatively outnumber us by more than 1,000,000,000,000,000,000,000,000 times on Earth thus far according to the paleontologists.
Or you could go further and ask why we aren’t neutrinos? There are more than 1,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000 of them in the observable universe.
However extravagant the class you pick, it’s cancelled out by the knowledge that we find ourselves in our current situation. I think it’s more confusing than helpful to say that our being humans rather than neutrinos is doing more than 10^70 times as much work as object-level analysis of AI in the case for attending to x-risk/lock-in with AI. You didn’t need to think about that in the first place to understand AI or bioweapons, it was an irrelevant distraction.
The same is true for future populations that know they’re living in intergalactic societies and the like. If we compare possible world A, where future Dyson spheres can handle a population of P (who know they’re in that era), and possible world B, where future Dyson spheres can support a population of 2P, they don’t give us much different expectations of the number of people finding themselves in our circumstances, and so cancel out.
The simulation argument (or a brain-in-vats story or the like) is different and doesn’t automatically cancel out because it’s a way to make our observations more likely and common. However, for policy it does still largely cancel out, as long as the total influence of people genuinely in our apparent circumstances is a lot greater than that of all simulations with apparent circumstances like ours: a bigger future world means more influence for genuine inhabitants of important early times and also more simulations. [But our valuation winds up being bounded by our belief about the portion of all-time resources allocated to sims in apparent positions like ours.]
Another way of thinking about this is that prior to getting confused by any anthropic updating, if you were going to set a policy for humans who find ourselves in our apparent situation across nonanthropic possibilities assessed at the object level (humanity doomed, Time of Perils, early lock-in, no lock-in), you would just want to add up the consequences of the policy across genuine early humans and sims in each (non-anthropically assessed) possible world.
A vast future gives more chances for influence on lock-in later, which might win out as even bigger than this century (although this gets rapidly less likely with time and expansion), but it shouldn’t change our assessment of lock-in this century, and a substantial chance of that gives us a good chance of HoH (or simulation-adjusted HoH).
One way to frame this is that we do need extraordinarily strong evidence to update from thinking that we’re almost certainly not the most influential time to thinking that we might plausibly be the most influential time. However, we don’t need extraordinarily strong evidence pointing towards us almost certainly being the most influential (that then “averages out” to thinking that we’re plausibly the most influential). It’s sufficient to get extraordinarily strong evidence that we are at a point in history which is plausibly the most influential. And if we condition on the future being long and that we aren’t in a simulation (because that’s probably when we have the most impact), we do in fact have extraordinarily strong evidence that we are very early in history, which is a point that’s plausibly the most influential.
The question which seems important to me now is: does Will think that the probability of high influentialness conditional on birth rank (but before accounting for any empirical knowledge) is roughly the same as the negative exponential distribution Toby discussed in the comments on his original post?
I actually think the negative exponential gives too little weight to later people, because I’m not certain that late people can’t be influential. But if I had a person from the first 1e-89 of all people who’ve ever lived and a random person from the middle, I’d certainly say that the former was more likely to be one of the most influential people. They’d also be more likely to be one of the least influential people! Their position is just so special!
Maybe my prior would be like 30% to a uniform function, 40% to negative exponentials of various slopes, and 30% to other functions (e.g. the last person who ever lived seems more likely to be the most influential than a random person in the middle.)
Only using a single, simple function for something so complicated seems overconfident to me. And any mix of functions where one of them assigns decent probability to early people being the most influential is enough that it’s not super unlikely that early people are the most influential.
“Only using a single, simple function for something so complicated seems overconfident to me. And any mix of functions where one of them assigns decent probability to early people being the most influential is enough that it’s not super unlikely that early people are the most influential.”
I strongly agree with this. The fact that under a mix of distributions, it becomes not super unlikely that early people are the most influential, is really important and was somewhat buried in the original comments-discussion.
And then we’re also very distinctive in other ways: being on one planet, being at such a high-growth period, etc.
I agree that our earliness gives a dramatic update in favor of us being influential. I don’t have a stable view on the magnitude of that.
I’m not convinced that the negative exponential form of Toby’s distribution is the right one, but I don’t have any better suggestions
Like Lukas, I think that Toby’s distribution gives too much weight to early people, so the update I would make is less dramatic than Toby’s
Seeing as Toby’s prior is quite sensitive to choice of reference-class, I would want to choose the reference class of all observer-moments, where an observer is a conscious being. This means we’re not as early as we would say if we used the distribution of Homo sapiens, or of hominids. I haven’t thought about what exactly that means, though my intuition is that it means the update isn’t nearly as big.
So I guess the answer to your question is ‘no’: our earliness is an enormous update, but not as big as Toby would suggest.
for any property F, your prior should be such that if there are n people in a population, the probability that you are in the m most F people in that population is m/n.
I want to dig into this a little, because it feels like there might be some sort of selection effect going on here. Suppose I make a claim X, and it has a number of implications X1, X2, X3 and so on. Each of these might apply to a different population, and have a different prior probability as a standalone claim. But if a critic chooses the one which has the lowest prior probability (call it Xn) in order to attack X, then it is much less fishy that Xn has a low prior probability, because the critic had many degrees of freedom in how they made their choice, which means the fishiness of the implication they choose is less reflective of the strength of the original hypothesis.
I don’t know how to quantify this, but it seems very relevant to your critique of Bostrom and Yudkowsky—their hypothesis has many different implications, and so a critic can choose freely which one to criticise as fishy (in this case, the implication of current influentialness). Of course you might respond that the implication of our influentialness is the most obvious and natural one for you to evaluate. But I have two problems with this:
It’s very easy to overestimate in hindsight how natural a given criterion was. Partly that’s because hindsight bias is very strong in humans. Additionally, though, if there had been any other axis which led to a more obvious critique of B+Y, then it seems pretty plausible that we’d have prioritised that critique instead. So critics probably are making use of more degrees of freedom than they realise, because B+Y’s hypothesis has already passed muster on the most obvious axes. (Edited to add: consider for example how many possible definitions of influentialness Will could have used which would have lead to his current argument being weaker!)
Different people have different standards for which criteria/priorities are compelling to them, and therefore should evaluate exactly the same argument differently. For example, someone who isn’t altruistic and doesn’t care about their influence over the future would likely find influentialness a very unnatural axis to evaluate B+Y on, and so should have more credence in their thesis than you do.
You might say that the fishiness of us being the most influential people is so extreme that it outweighs these considerations. But my suspicion here is that as the number of implications of a hypothesis grows, it becomes exponentially more likely that we find one which has a certain level of fishiness (just because you need to multiply the probabilities of each one not being that fishy, assuming that the fishiness of the implications are independent—although ofc that’s a gross simplification). And so for far-ranging hypotheses, the fact that we can find one or two axes along which they fare very badly might provide relatively little evidence.
Note however that I feel pretty uncertain about these points, though, and it’s quite possible that they’re totally wrong.
I don’t make any claims about how likely it is that we are part of a very long future. Only that, a priori, the probability that we’re *both* in a very large future *and* one of the most influential people ever is very low. For that reason, there aren’t any implications from that argument to claims about the magnitude of extinction risk this century.
I don’t understand why there are implications from that argument to claims about the magnitude of our influentialness either.
As an analogy, suppose Alice bought a lottery ticket that will win her $100,000,000 with an extremely small probability. The lottery is over, and she is now looking at the winning numbers on her phone, comparing them one by one to the numbers on her ticket. Her excitement grows as she finds more and more of the winning numbers on her ticket. She managed to verify that she got 7 numbers right (amazing!), but before she finished comparing the rest of the numbers, her battery died. She tries to find a charger, and in the meantime she’s considering whether to donate the money to FHI if she wins. It occurs to her that the probability that *both* [a given person wins the lottery] *and* [donating $100,000,000 to FHI will reduce existential risk] is extremely small. She reasons that, sure, there are some plausible arguments that donating $100,000,000 to FHI will have a huge positive impact, but are those arguments strong enough considering her extremely small prior probability in the above conjunction?
The key thing is that the way I’m setting priors is as a function from populations to credences: for any property F, your prior should be such that if there are n people in a population, the probability that you are in the m most F people in that population is m/n.
The fact that I consider a certain property F should update me, though. This already demonstrates that F is something that I am particularly interested in, or that F is salient to me, which presumably makes it more likely that I am an outlier on F.
Also, this principle can have pretty strange implications depending on how you apply it. For instance, if I look at the population of all beings on Earth, it is extremely surprising (10^-12 or so) that I am a human rather than an insect.
On this set-up of the argument (which is what was in my head but I hadn’t worked through), I don’t make any claims about how likely it is that we are part of a very long future.
This does make a lot more sense than what you wrote in your post.
Do you agree that as written, the argument as written in your EA Forum post is quite flawed? If so, I think you should edit it to more clearly indicate that it was a mistake, given that people are still linking to it.
Yeah, I do think the priors-based argument given in the post was poorly stated, and therefore led to unnecessary confusion. Your suggestion is very reasonable, and I’ve now edited the post.
Actually, rereading my post I realize I had already made an edit similar to the one you suggest (though not linking to the article which hadn’t been finished) back in March 2020:
”[Later Edit (Mar 2020): The way I state the choice of prior in the text above was mistaken, and therefore caused some confusion. The way I should have stated the prior choice, to represent what I was thinking of, is as follows:
The prior probability of us living in the most influential century, conditional on Earth-originating civilization lasting for n centuries, is 1/n.
The unconditional prior probability over whether this is the most influential century would then depend on one’s priors over how long Earth-originating civilization will last for. However, for the purpose of this discussion we can focus on just the claim that we are at the most influential century AND that we have an enormous future ahead of us. If the Value Lock-In or Time of Perils views are true, then we should assign a significant probability to that claim. (i.e. they are claiming that, if we act wisely this century, then this conjunctive claim is probably true.) So that’s the claim we can focus our discussion on.
It’s worth noting that my proposal follows from the Self-Sampling Assumption, which is roughly (as stated by Teru Thomas (‘Self-location and objective chance’ (ms)): “A rational agent’s priors locate him uniformly at random within each possible world.” I believe that SSA is widely held: the key question in the anthropic reasoning literature is whether it should be supplemented with the self-indication assumption (giving greater prior probability mass to worlds with large populations). But we don’t need to debate SIA in this discussion, because we can simply assume some prior probability distribution over sizes over the total population—the question of whether we’re at the most influential time does not require us to get into debates over anthropics.]”
Oh man, I’m so sorry, you’re totally right that this edit fixes the problem I was complaining about. When I read this edit, I initially misunderstood it in such a way that it didn’t address my concern. My apologies.
Separately, I still don’t see the case for building earliness into our priors, rather than updating on the basis of finding oneself seemingly-early.
Do you have some other way of updating on the arrow of time? (It seems like the fact that we can influence future generations, but they can’t influence us, is pretty significant, and should be factored into the argument somewhere.)
I wouldn’t call that an update on finding ourselves early, but more like just an update on the structure of the population being sampled from.
You could make an argument that a certain kind of influence strictly decreases with time. So the hinge was at the Big Bang.
But, there (probably) weren’t any agents around to control anything then, so maybe you say there was zero influence available at that time. Everything that happened was just being determined by low level forces and fields and particles (and no collections of those could be reasonably described as conscious agents).
Today, much of what happens (on Earth) is determined by conscious agents, so in some sense the total amount of extant influence has grown.
Let’s maybe call the first kind of influence time-priority, and the second agency. So, since the Big Bang, the level of time-priority influence available in the universe has gone way down, but the level of aggregate agency in the universe has gone way up.
On a super simple model that just takes these two into account, you might multiply them together to get the total influence available at a certain time (and then divide by the number of people alive at that time to get the average person’s influence). This number will peak somewhere in the middle (assuming it’s zero both at the Big Bang and at the Heat Death).
That maybe doesn’t tell you much, but then you could start taking into account some other considerations, like how x-risk could result in a permanent drop of agency down to zero. Or how perhaps there’s an upper limit on how much agency is potentially available in the universe.
In any case, it seems like the direction of causality should be a pretty important part of the analysis (even if it points in the opposite direction of another factor, like increasing agency), either as part of the prior or as one of the first things you update on.
(Comment 2⁄5)
The outside-view argument (in response to your first argument)
In the blog post, I stated the priors-based argument quite poorly—I thought this bit wouldn’t be where the disagreement was, so I didn’t spend much time on it. How wrong I was about that! For the article version (link), I tidied it up.
The key thing is that the way I’m setting priors is as a function from populations to credences: for any property F, your prior should be such that if there are n people in a population, the probability that you are in the m most F people in that population is m/n.
This falls out of the self-sampling assumption, that a rational agent’s priors locate her uniformly at random within each possible world. If you reject this way of setting priors then, by modus tollens, you reject the self-sampling assumption. That’s pretty interesting if so!
On this set-up of the argument (which is what was in my head but I hadn’t worked through), I don’t make any claims about how likely it is that we are part of a very long future. Only that, a priori, the probability that we’re *both* in a very large future *and* one of the most influential people ever is very low. For that reason, there aren’t any implications from that argument to claims about the magnitude of extinction risk this century. We could be comparatively un-influential in many ways: if extinction risk is high this century but continues to be high for very many centuries; if extinction risk is low this century and will be higher in coming centuries; if extinction risk is any level and we can’t do anything about it, or are not yet knowledgeable enough to choose actions wisely, or if longtermism is false. (etc)
Separately, I still don’t see the case for building earliness into our priors, rather than updating on the basis of finding oneself seemingly-early. Building earliness into your prior means you’ve got to give up on the very-plausible-seeming self-sampling assumption; means you’ve got to treat the predicate ‘is most influential’ differently than other predicates; has technical challenges; and the case in favour seems to rely on a posteriori observations about how the world works, like those you give in your post.
If we’re doing things right, it shouldn’t matter whether we’re building earliness into our prior or updating on the basis of earliness.
Let the set H=”the 1e10 (i.e. 10 billion) most influential people who will ever live” and let E=”the 1e11 (i.e. 100 billion) earliest people who will ever live”. Assume that the future will contain 1e100 people. Let X be a randomly sampled person.
For our unconditional prior P(X in H), everyone agrees that uniform probability is appropriate, i.e., P(X in H) = 1e-90. (I.e. we’re not giving up on the self-sampling assumption.)
However, for our belief over P(X in H | X in E), i.e. the probability that a randomly chosen early person is one of the most influential people, some people argue we should utilise an e.g. exponential function where earlier people are more likely to be influential (which could be called a prior over “X in H” based on how early X is). However, it seems like you’re saying that we shouldn’t assess P(X in H | X in E) directly from such a prior, but instead get it from bayesian updates. So lets do that.
P(X in H | X in E) = P(X in E | X in H) * P(X in H) / P(X in E) = P(X in E | X in H) * 1e-90 / 1e-89 = P(X in E | X in H) * 1e-1 = P(X in E | X in H) / 10
So now we’ve switched over to instead making a guess about P(X in E | X in H), i.e. the probability that one of the 1e10 most influential people also is one of the 1e11 earliest people, and dividing by 10. That doesn’t seem much easier than making a guess about P(X in H | X in E), and it’s not obvious whether our intuitions here would lead us to expect more or less influentialness.
Also, the way that 1e-90 and 1e-89 are both extraordinarily unlikely, but divide out to becoming 1e-1, illustrates Buck’s point:
“If we’re doing things right, it shouldn’t matter whether we’re building earliness into our prior or updating on the basis of earliness.”
Thanks, Lukas, I thought this was very clear and exactly right.
“So now we’ve switched over to instead making a guess about P(X in E | X in H), i.e. the probability that one of the 1e10 most influential people also is one of the 1e11 earliest people, and dividing by 10. That doesn’t seem much easier than making a guess about P(X in H | X in E), and it’s not obvious whether our intuitions here would lead us to expect more or less influentialness.”
That’s interesting, thank you—this statement of the debate has helped clarify things for me. It does seem to me that doing the update - going via P(X in E | X in H) rather than directly trying to assess P(X in H | X in E) - is helpful, but I’d understand the position of someone who wanted just to assess P(X in H | X in E) directly.
I think it’s helpful to assess P(X in E | X in H) because it’s not totally obvious how one should update on the basis of earliness. The arrow of causality and the possibility of lock-in over time definitely gives reasons in favor of influential people being earlier. But there’s still the big question of how great an update that should be. And the cumulative nature of knowledge and understanding gives reasons in favor thinking that later people are more likely to be more influential.
This seems important to me because, for someone claiming that we should think that we’re at the HoH, the update on the basis of earliness is doing much more work than updates on the basis of, say, familiar arguments about when AGI is coming and what will happen when it does. To me at least, that’s a striking fact and wouldn’t have been obvious before I started thinking about these things.
It seems to me the object level is where the action is, and the non-simulation Doomsday Arguments mostly raise a phantom consideration that cancels out (in particular, cancelling out re whether there is an influenceable lock-in event this century).
You could say a similar thing about our being humans rather than bacteria, which cumulatively outnumber us by more than 1,000,000,000,000,000,000,000,000 times on Earth thus far according to the paleontologists.
Or you could go further and ask why we aren’t neutrinos? There are more than 1,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000 of them in the observable universe.
However extravagant the class you pick, it’s cancelled out by the knowledge that we find ourselves in our current situation. I think it’s more confusing than helpful to say that our being humans rather than neutrinos is doing more than 10^70 times as much work as object-level analysis of AI in the case for attending to x-risk/lock-in with AI. You didn’t need to think about that in the first place to understand AI or bioweapons, it was an irrelevant distraction.
The same is true for future populations that know they’re living in intergalactic societies and the like. If we compare possible world A, where future Dyson spheres can handle a population of P (who know they’re in that era), and possible world B, where future Dyson spheres can support a population of 2P, they don’t give us much different expectations of the number of people finding themselves in our circumstances, and so cancel out.
The simulation argument (or a brain-in-vats story or the like) is different and doesn’t automatically cancel out because it’s a way to make our observations more likely and common. However, for policy it does still largely cancel out, as long as the total influence of people genuinely in our apparent circumstances is a lot greater than that of all simulations with apparent circumstances like ours: a bigger future world means more influence for genuine inhabitants of important early times and also more simulations. [But our valuation winds up being bounded by our belief about the portion of all-time resources allocated to sims in apparent positions like ours.]
Another way of thinking about this is that prior to getting confused by any anthropic updating, if you were going to set a policy for humans who find ourselves in our apparent situation across nonanthropic possibilities assessed at the object level (humanity doomed, Time of Perils, early lock-in, no lock-in), you would just want to add up the consequences of the policy across genuine early humans and sims in each (non-anthropically assessed) possible world.
A vast future gives more chances for influence on lock-in later, which might win out as even bigger than this century (although this gets rapidly less likely with time and expansion), but it shouldn’t change our assessment of lock-in this century, and a substantial chance of that gives us a good chance of HoH (or simulation-adjusted HoH).
One way to frame this is that we do need extraordinarily strong evidence to update from thinking that we’re almost certainly not the most influential time to thinking that we might plausibly be the most influential time. However, we don’t need extraordinarily strong evidence pointing towards us almost certainly being the most influential (that then “averages out” to thinking that we’re plausibly the most influential). It’s sufficient to get extraordinarily strong evidence that we are at a point in history which is plausibly the most influential. And if we condition on the future being long and that we aren’t in a simulation (because that’s probably when we have the most impact), we do in fact have extraordinarily strong evidence that we are very early in history, which is a point that’s plausibly the most influential.
The question which seems important to me now is: does Will think that the probability of high influentialness conditional on birth rank (but before accounting for any empirical knowledge) is roughly the same as the negative exponential distribution Toby discussed in the comments on his original post?
I actually think the negative exponential gives too little weight to later people, because I’m not certain that late people can’t be influential. But if I had a person from the first 1e-89 of all people who’ve ever lived and a random person from the middle, I’d certainly say that the former was more likely to be one of the most influential people. They’d also be more likely to be one of the least influential people! Their position is just so special!
Maybe my prior would be like 30% to a uniform function, 40% to negative exponentials of various slopes, and 30% to other functions (e.g. the last person who ever lived seems more likely to be the most influential than a random person in the middle.)
Only using a single, simple function for something so complicated seems overconfident to me. And any mix of functions where one of them assigns decent probability to early people being the most influential is enough that it’s not super unlikely that early people are the most influential.
“Only using a single, simple function for something so complicated seems overconfident to me. And any mix of functions where one of them assigns decent probability to early people being the most influential is enough that it’s not super unlikely that early people are the most influential.”
I strongly agree with this. The fact that under a mix of distributions, it becomes not super unlikely that early people are the most influential, is really important and was somewhat buried in the original comments-discussion.
And then we’re also very distinctive in other ways: being on one planet, being at such a high-growth period, etc.
Thanks, I agree that this is key. My thoughts:
I agree that our earliness gives a dramatic update in favor of us being influential. I don’t have a stable view on the magnitude of that.
I’m not convinced that the negative exponential form of Toby’s distribution is the right one, but I don’t have any better suggestions
Like Lukas, I think that Toby’s distribution gives too much weight to early people, so the update I would make is less dramatic than Toby’s
Seeing as Toby’s prior is quite sensitive to choice of reference-class, I would want to choose the reference class of all observer-moments, where an observer is a conscious being. This means we’re not as early as we would say if we used the distribution of Homo sapiens, or of hominids. I haven’t thought about what exactly that means, though my intuition is that it means the update isn’t nearly as big.
So I guess the answer to your question is ‘no’: our earliness is an enormous update, but not as big as Toby would suggest.
I want to dig into this a little, because it feels like there might be some sort of selection effect going on here. Suppose I make a claim X, and it has a number of implications X1, X2, X3 and so on. Each of these might apply to a different population, and have a different prior probability as a standalone claim. But if a critic chooses the one which has the lowest prior probability (call it Xn) in order to attack X, then it is much less fishy that Xn has a low prior probability, because the critic had many degrees of freedom in how they made their choice, which means the fishiness of the implication they choose is less reflective of the strength of the original hypothesis.
I don’t know how to quantify this, but it seems very relevant to your critique of Bostrom and Yudkowsky—their hypothesis has many different implications, and so a critic can choose freely which one to criticise as fishy (in this case, the implication of current influentialness). Of course you might respond that the implication of our influentialness is the most obvious and natural one for you to evaluate. But I have two problems with this:
It’s very easy to overestimate in hindsight how natural a given criterion was. Partly that’s because hindsight bias is very strong in humans. Additionally, though, if there had been any other axis which led to a more obvious critique of B+Y, then it seems pretty plausible that we’d have prioritised that critique instead. So critics probably are making use of more degrees of freedom than they realise, because B+Y’s hypothesis has already passed muster on the most obvious axes. (Edited to add: consider for example how many possible definitions of influentialness Will could have used which would have lead to his current argument being weaker!)
Different people have different standards for which criteria/priorities are compelling to them, and therefore should evaluate exactly the same argument differently. For example, someone who isn’t altruistic and doesn’t care about their influence over the future would likely find influentialness a very unnatural axis to evaluate B+Y on, and so should have more credence in their thesis than you do.
You might say that the fishiness of us being the most influential people is so extreme that it outweighs these considerations. But my suspicion here is that as the number of implications of a hypothesis grows, it becomes exponentially more likely that we find one which has a certain level of fishiness (just because you need to multiply the probabilities of each one not being that fishy, assuming that the fishiness of the implications are independent—although ofc that’s a gross simplification). And so for far-ranging hypotheses, the fact that we can find one or two axes along which they fare very badly might provide relatively little evidence.
Note however that I feel pretty uncertain about these points, though, and it’s quite possible that they’re totally wrong.
I don’t understand why there are implications from that argument to claims about the magnitude of our influentialness either.
As an analogy, suppose Alice bought a lottery ticket that will win her $100,000,000 with an extremely small probability. The lottery is over, and she is now looking at the winning numbers on her phone, comparing them one by one to the numbers on her ticket. Her excitement grows as she finds more and more of the winning numbers on her ticket. She managed to verify that she got 7 numbers right (amazing!), but before she finished comparing the rest of the numbers, her battery died. She tries to find a charger, and in the meantime she’s considering whether to donate the money to FHI if she wins. It occurs to her that the probability that *both* [a given person wins the lottery] *and* [donating $100,000,000 to FHI will reduce existential risk] is extremely small. She reasons that, sure, there are some plausible arguments that donating $100,000,000 to FHI will have a huge positive impact, but are those arguments strong enough considering her extremely small prior probability in the above conjunction?
The fact that I consider a certain property F should update me, though. This already demonstrates that F is something that I am particularly interested in, or that F is salient to me, which presumably makes it more likely that I am an outlier on F.
Also, this principle can have pretty strange implications depending on how you apply it. For instance, if I look at the population of all beings on Earth, it is extremely surprising (10^-12 or so) that I am a human rather than an insect.
This does make a lot more sense than what you wrote in your post.
Do you agree that as written, the argument as written in your EA Forum post is quite flawed? If so, I think you should edit it to more clearly indicate that it was a mistake, given that people are still linking to it.
Yeah, I do think the priors-based argument given in the post was poorly stated, and therefore led to unnecessary confusion. Your suggestion is very reasonable, and I’ve now edited the post.
Actually, rereading my post I realize I had already made an edit similar to the one you suggest (though not linking to the article which hadn’t been finished) back in March 2020:
”[Later Edit (Mar 2020): The way I state the choice of prior in the text above was mistaken, and therefore caused some confusion. The way I should have stated the prior choice, to represent what I was thinking of, is as follows:
The prior probability of us living in the most influential century, conditional on Earth-originating civilization lasting for n centuries, is 1/n.
The unconditional prior probability over whether this is the most influential century would then depend on one’s priors over how long Earth-originating civilization will last for. However, for the purpose of this discussion we can focus on just the claim that we are at the most influential century AND that we have an enormous future ahead of us. If the Value Lock-In or Time of Perils views are true, then we should assign a significant probability to that claim. (i.e. they are claiming that, if we act wisely this century, then this conjunctive claim is probably true.) So that’s the claim we can focus our discussion on.
It’s worth noting that my proposal follows from the Self-Sampling Assumption, which is roughly (as stated by Teru Thomas (‘Self-location and objective chance’ (ms)): “A rational agent’s priors locate him uniformly at random within each possible world.” I believe that SSA is widely held: the key question in the anthropic reasoning literature is whether it should be supplemented with the self-indication assumption (giving greater prior probability mass to worlds with large populations). But we don’t need to debate SIA in this discussion, because we can simply assume some prior probability distribution over sizes over the total population—the question of whether we’re at the most influential time does not require us to get into debates over anthropics.]”
Oh man, I’m so sorry, you’re totally right that this edit fixes the problem I was complaining about. When I read this edit, I initially misunderstood it in such a way that it didn’t address my concern. My apologies.
Do you have some other way of updating on the arrow of time? (It seems like the fact that we can influence future generations, but they can’t influence us, is pretty significant, and should be factored into the argument somewhere.)
I wouldn’t call that an update on finding ourselves early, but more like just an update on the structure of the population being sampled from.
You could make an argument that a certain kind of influence strictly decreases with time. So the hinge was at the Big Bang.
But, there (probably) weren’t any agents around to control anything then, so maybe you say there was zero influence available at that time. Everything that happened was just being determined by low level forces and fields and particles (and no collections of those could be reasonably described as conscious agents).
Today, much of what happens (on Earth) is determined by conscious agents, so in some sense the total amount of extant influence has grown.
Let’s maybe call the first kind of influence time-priority, and the second agency. So, since the Big Bang, the level of time-priority influence available in the universe has gone way down, but the level of aggregate agency in the universe has gone way up.
On a super simple model that just takes these two into account, you might multiply them together to get the total influence available at a certain time (and then divide by the number of people alive at that time to get the average person’s influence). This number will peak somewhere in the middle (assuming it’s zero both at the Big Bang and at the Heat Death).
That maybe doesn’t tell you much, but then you could start taking into account some other considerations, like how x-risk could result in a permanent drop of agency down to zero. Or how perhaps there’s an upper limit on how much agency is potentially available in the universe.
In any case, it seems like the direction of causality should be a pretty important part of the analysis (even if it points in the opposite direction of another factor, like increasing agency), either as part of the prior or as one of the first things you update on.