It is great to see all your thinking on this down in one place: there are lots of great points here (and in the comments too). By explaining your thinking so clearly, it makes it much easier to see where one departs from it.
My biggest departure is on the prior, which actually does most of the work in your argument: it creates the extremely high bar for evidence, which I agree probably couldn’t be met. I’ve mentioned before that I’m quite sure the uniform prior is the wrong choice here and that this makes a big difference. I’ll explain a bit about why I think that.
As a general rule if you have a domain like this that extends indefinitely in one direction, the correct prior is one that diminishes as you move further away in that direction, rather than picking a somewhat arbitrary end point and using a uniform prior on that. People do take this latter approach in scientific papers, but I think it is usually wrong to do so. Moreover in your case in particular, there are also good reasons to suspect that the chance of a century being the most influential should diminish over time. Especially because there are important kinds of significant event (such as the value lock-in or an existential catastrophe) where early occurrence blocks out later occurrence.
This directly leads to diminishing credence over time. e.g. if there is a known constant chance of such a key event happening in any century conditional on not happening before that time then the chance it first happens in any century diminishes exponentially as time goes on. Or if this chance is unknown and could be anything between zero and one, then instead of an exponential decline, it diminishes more slowly (analogous to Weitzman discounting). The most famous model of this is Laplace’s Law of Succession, where if your prior for the unknown contstant hazard rate per time period is uniform on the interval between 0 and 1, then the chance it happens in the nth period if it hasn’t before is 1/n+2 — a hyperbola. I think hazard rates closer to zero and one are more likely than those in between, so I prefer the bucket shaped Jeffrey’s prior (= Beta(0.5, 0.5) for the maths nerds out there), which gives a different hyperbola of 1/2n+2 (and makes my case a little bit harder than if I’d settled for the uniform prior).
A raw application of this would say that since Homo sapiens has been around for 2,000 centuries (without, let us suppose, having had such a one-off critical time yet), the chance it happens this century is 1 in 2,002 (or 1 in 4,002). [Actually I’ll just say 1 in 2,000 or (1 in 4,000), as the +2 is just an artefact of how we cut up the time periods and can be seen to go to zero when we use continuous time.] This is a lot more likely than your 1 in a million or 1 in 100,000. And it gets even more so when you run it in terms of persons or person years (as I believe you should). i.e. measure time with a clock that ticks as each lifetime ends, rather than one that ticks each second. e.g. about 1/20th of all people who have ever lived are alive now, so the next century it is not really 1⁄2,000th of human history but more like 1/20th of it. On this clock and with this prior, one would expect a 1⁄20 (or 1⁄40) chance of a pivotal event (first) occurring.
Note that while your model applied a kind of principle of indifference uniformly across time, saying each century was equally likely (a kind of outside view), my model makes similar sounding assumptions. It assumes that each century is equally likely to have such a high stakes pivotal event (conditional on it not already happening), and if you do the maths, this also corresponds to each order of magnitude of time having an equal (unconditional) chance of the the pivotal event happening in it (i.e. instead of equal chance in century 1, century 2, century 3… it is equal chance in centuries 1 to 10, centuries 10 to 100, centuries 100 to 1,000), which actually seems more intuitive to me. Then there is the wrinkle that I don’t assign it across clock time, but across persons or person-years (e.g. where I say ‘century’ your could read it as ‘1 trillion person years’). All these choices are inspired by very similar motivations to how you chose your prior.
[As an interesting side-note, this kind of prior is also what you get if you apply Richard Gott’s version of the Doomsday Argument to estimate how long we will last (say, instead of the toy model you apply), and this is another famous way of doing outside-view forecasting.]
I doubt I can easily convince you that the prior I’ve chosen is objectively best, or even that it is better than the one you used. Prior-choice is a bit of an art, rather like choice of axioms. But I hope you see that it does show that the whole thing comes down to whether you choose a prior like you did, or another reasonable alternative. My prior gives a prior chance of HoH of about 5% or 2.5%, which is thousands of times more likely than yours, and can easily be bumped up by the available evidence to probabilities >10%. So your argument doesn’t do well on sensitivity analysis over prior-choice. Additionally, if you didn’t know which of these priors to use and used a mixture with mine weighted in to a non-trivial degree, this would also lead to a substantial prior probability of HoH. And this is only worse if instead of using a 1/n hyperbola like I did, you had arguments that it declined more quickly, like 1/n^2 or an exponential. So it only goes through if you are very solidly committed to a prior like the one you used.
Thanks so much for this very clear response, it was a very satisfying read, and there’s a lot for me to chew on. And thanks for locating the point of disagreement — prior to this post, I would have guessed that the biggest difference between me and some others was on the weight placed on the arguments for the Time of Perils and Value Lock-In views, rather than on the choice of prior. But it seems that that’s not true, and that’s very helpful to know. If so, it suggests (advertisement to the Forum!) that further work on prior-setting in EA contexts is very high-value.
I agree with you that under uncertainty over how to set the prior, because we’re clearly so distinctive in some particular ways (namely, that we’re so early on in civilisation, that the current population is so small, etc), my choice of prior will get washed out by models on which those distinctive features are important; I characterised these as outside-view arguments, but I’d understand if someone wanted to characterise that as prior-setting instead.
I also agree that there’s a strong case for making the prior over persons (or person-years) rather than centuries. In your discussion, you go via number of persons (or person-years) per century to the comparative importance of centuries. What I’d be inclined to do is just change the claim under consideration to: “I am among the (say) 100,000 most influential people ever”. This means we still take into account the fact that, though more populous centuries are more likely to be influential, they are also harder to influence in virtue of their larger population. If we frame the core claim in terms of being among the most influential people, rather than being at the most influential time, the core claim seems even more striking to me. (E.g. a uniform prior over the first 100 billion people would give a prior of 1 in 1 million of being in the 100,000 most influential people ever. Though of course, there would also be an extra outside-view argument for moving from this prior, which is that not many people are trying to influence the long-run future.)
However, I don’t currently feel attracted to your way of setting up the prior. In what follows I’ll just focus on the case of a values lock-in event, and for simplicity I’ll just use the standard Laplacean prior rather than your suggestion of a Jeffreys prior.
In significant part my lack of attraction is because the claims — that (i) there’s a point in time where almost everything about the fate of the universe gets decided; (ii) that point is basically now; (iii) almost no-one sees this apart from us (where ‘us’ is a very small fraction of the world) — seem extraordinary to me, and I feel I need extraordinary evidence in order to have high credence in them. My prior-setting discussion was one way of cashing out why these seem extraordinary. If there’s some way of setting priors such that claims (i)-(iii) aren’t so extraordinary after all, I feel like a rabbit is being pulled out of a hat.
Then I have some specific worries with the Laplacean approach (which I *think* would apply to the Jeffreys prior, too, but I’m yet to figure out what a Fischer information matrix is, so I don’t totally back myself here).
But before I mention the worries, I’ll note that it seems to me that you and I are currently talking about priors over different propositions. You seem to be considering the propositions, ‘there is a lock-in event this century’ or ‘there is an extinction event this century’; I’m considering the proposition ‘I am at the most influential time ever’ or ‘I am one of the most influential people ever.’ As is well-known, when it comes to using principle-of-indifference-esque reasoning, if you use that reasoning over a number of different propositions then you can end up with inconsistent probability assignments. So, at best, one should use such reasoning in a very restricted way.
The reason I like thinking about my proposition (‘are we at the most important time?’ or ‘are we one of the most influential people ever?’) for the restricted principle of indifference, is that:
(i) I know the frequency of occurrence of ‘most influential person’, for each possible total population of civilization (past, present and future). Namely, it occurs once out of the total population. So I can look at each possible population size for the future, look at my credence in each possible population occurring, and in each case know the frequency of being the most influential person (or, more naturally, in the 100,000 most influential people).
(ii) it’s the most relevant proposition for the question of what I should do. (e.g. Perhaps it’s likely that there’s a lock-in event, but we can’t do anything about it and future people could, so we should save for a later date.)
Anyway, the worries about Laplacean (and Jeffreys) prior.
First, the Laplacean prior seems to get the wrong answer for lots of similar predicates. Consider the claims: “I am the most beautiful person ever” or “I am the strongest person ever”, rather than “I am the most important person ever”. If we used the Laplacean prior in the way you suggest for these claims, the first person would assign 50% credence to being the strongest person ever, even if they knew that there was probably going to be billions of people to come. This doesn’t seem right to me.
Second, it also seems very sensitive to our choice of start date. If the proposition under question is, ‘there will be a lock-in event this century’, I’d get a very different prior depending on whether I chose to begin counting from: (i) the dawn of the information age; (ii) the beginning of the industrial revolution; (iii) the start of civilisation; (iv) the origin of homo sapiens; (v) the origin of the genus homo; (vi) the origin of mammals, etc.
Of course, the uniform prior has something similar, but I think it handles the issue gracefully. e.g. On priors, I should think it’s 1 in 5 million likely that I’m the funniest person in Scotland; 1 in 65 million that I’m the funniest person in Britain, 1 in 7.5 billion that I’m the funniest person in the world. Similarly, with whether I’m the most influential person in the post-industrial era, the post-agricultural era, etc.
Third, the Laplacean prior doesn’t add up to 1 across all people. For example, suppose you’re the first person and you know that there will be 3 people. Then, on the Laplacean prior, the total probability for being the most influential person ever is ½ + ½(⅓) + ½(⅔)(¼) = ¾. But I know that someone has to be the most influential person ever. This suggests the Laplacean prior is the wrong prior choice for the proposition I’m considering, whereas the simple frequency approach gets it right.
So even if one feels skeptical of the uniform prior, I think the Laplacean way of prior-setting isn’t a better alternative. In general: I’m sympathetic to having a model where early people are more likely to be more influential, but a model which is uniform over orders of magnitude seems too extreme to me.
(As a final thought: Doesn’t this form of prior-setting also suffer from the problem of there being too many hypotheses? E.g. consider the propositions:
A—There will be a value lock-in event this centuryB—There will be a lock-in of hedonistic utilitarian values this century
C—There will be a lock-in of preference utilitarian values this century
D—There will be a lock-in of Kantian values this century
E—There will be a lock-in of fascist values this century
On the Laplacean approach, these would all get the same probability assignment—which seems inconsistent. And then just by stacking priors over particular lock-in events, we can get a probability that it’s overwhelmingly likely that there’s some lock-in event this century. I’ve put this comment in parentheses, though, as I feel *even less* confident about my worry here than my other worries listed.)
Thanks for this very thorough reply. There are so many strands here that I can’t really hope to do justice to them all, but I’ll make a few observations.
1) There are two versions of my argument. The weak/vague one is that a uniform prior is wrong and the real prior should decay over time, such that you can’t make your extreme claim from priors. The strong/precise one is that it should decay as 1/n^2 in line with a version of LLS. The latter is more meant as an illustration. It is my go-to default for things like this, but my main point here is the weaker one. It seems that you agree that it should decay, and that the main question now is whether it does so fast enough to make your prior-based points moot. I’m not quite sure how to resolve that. But I note that from this position, we can’t reach either your argument that from priors this is way too unlikely for our evidence to overturn (and we also can’t reach my statement of the opposite of that).
2) I wouldn’t use the LLS prior for arbitrary superlative properties where you fix the total population. I’d use it only if the population over time was radically unknown (so that the first person is much more likely to be strongest than the thousandth, because there probably won’t be a thousand) or where there is a strong time dependency such that it happening at one time rules out later times.
3) You are right that I am appealing to some structural properties beyond mere superlatives, such as extinction or other permanent lock-in. This is because these things happening in a century would be sufficient for that century to have a decent chance of being the most influential (technically this still depends on the influenceability of the event, but I think most people would grant that conditional on next century being the end of humanity, it is no longer surprising at all if this or next century were the most influential). So I think that your prior setting approach proves too much, telling us that there is almost no chance of extinction or permanent lock-in next century (and even after updating on evidence). This feels fishy. A bit like Bostrom’s ‘presumptuous philosopher’ example. I think it looks even more fishy in your worked example where the prior is low precisely because of an assumption about how long we will last without extinction: especially as that assumption is compatible with, say, a 50% chance of extinction in the next century. (I don’t think this is a knockdown blow here: but I’m trying to indicate the part of your argument I think would be most likely to fall and roughly why).
4) I agree there is an issue to do with too many hypotheses . And a related issue with what is the first timescale on which to apply a 1⁄2 chance of the event occurring. I think these can be dealt with together. You modify the raw LLS prior by some other kind of prior you have for each particular type of event (which you need to have since some are sub-events of others and rationality requires you to assign lower probability to them). You could operationalise this by asking over what time frame you’d expect a 1⁄2 chance of that event occurring. Then LLS isn’t acting as an indifference principle, but rather just as a way of keeping track of how to update your ur prior in light of how many time periods have elapsed without the event occurring. I think this should work out somewhat similarly, just with a stretched PDF that still decays as 1/n^2, but am not sure. There may be a literature on this.
I appreciate your explicitly laying out issues with the Laplace prior! I found this helpful.The approach to picking a prior here which I feel least uneasy about is something like: “take a simplicity-weighted average over different generating processes for distributions of hinginess over time”. This gives a mixture with some weight on uniform (very simple), some weight on monotonically-increasing and monotonically-decreasing functions (also quite simple), some weight on single-peaked and single-troughed functions (disproportionately with the peak or trough close to one end), and so on…If we assume a big future and you just told me the number of people in each generation, I think my prior might be something like 20% that the most hingey moment was in the past, 1% that it was in the next 10 centuries, and the rest after that. After I notice that hingeyness is about influence, and causality gives a time asymmetry favouring early times, I think I might update to >50% that it was in the past, and 2% that it would be in the next 10 centuries.(I might start with some similar prior about when the strongest person lives, but then when I begin to understand something about strength the generating mechanisms which suggest that the strongest people would come early and everything would be diminishing thereafter seem very implausible, so I would update down a lot on that.)
I’m sympathetic to the mixture of simple priors approach and value simplicity a great deal. However, I don’t think that the uniform prior up to an arbitrary end point is the simplest as your comment appears to suggest. e.g. I don’t see how it is simpler than an exponential distribution with an arbitrary mean (which is the max entropy prior over R+ conditional on a finite mean). I’m not sure if there is a max entropy prior over R+ without the finite mean assumption, but 1/x^2 looks right to me for that.
Also, re having a distribution that increases over a fixed time interval giving a peak at the end, I agree that this kind of thing is simple, but note that since we are actually very uncertain over when that interval ends, that peak gets very smeared out. Enough so that I don’t think there is a peak at the end at all when the distribution is denominated in years (rather than centiles through human history or something). That said, it could turn into a peak in the middle, depending on the nature of one’s distribution over durations.
I doubt I can easily convince you that the prior I’ve chosen is objectively best, or even that it is better than the one you used. Prior-choice is a bit of an art, rather like choice of axioms. But I hope you see that it does show that the whole thing comes down to whether you choose a prior like you did, or another reasonable alternative… Additionally, if you didn’t know which of these priors to use and used a mixture with mine weighted in to a non-trivial degree, this would also lead to a substantial prior probability of HoH.
I think this point is even stronger, as your early sections suggest. If we treat the priors as hypotheses about the distribution of events in the world, then past data can provide evidence about which one is right, and (the principle of) Will’s prior would have given excessively low credence to humanity’s first million years being the million years when life traveled to the Moon, humanity becoming such a large share of biomass, the first 10,000 years of agriculture leading to the modern world, and so forth. So those data would give us extreme evidence for a less dogmatic prior being correct.
If we treat the priors as hypotheses about the distribution of events in the world, then past data can provide evidence about which one is right, and (the principle of) Will’s prior would have given excessively low credence to humanity’s first million years being the million years when life traveled to the Moon, humanity becoming such a large share of biomass, the first 10,000 years of agriculture leading to the modern world, and so forth.
On the other hand, the kinds of priors Toby suggests would also typically give excessively low credence to these events taking so long. So the data doesn’t seem to provide much active support for the proposed alternative either.
It also seems to me like different kinds of priors are probably warranted for predictions about when a given kind of event will happen for the first time (e.g. the first year in which someone is named Steve) and predictions about when a given property will achieve its maximum value (e.g. the year with the most Steves). It can therefore be consistent to expect the kinds of “firsts” you list to be relatively bunched up near the start of human history, while also expecting relevant “mosts” (such as the most hingey year) to be relatively spread out.
That being said, I find it intuitive that periods with lots of “firsts” should tend to be disproportionately hingey. I think this intuition could be used to construct a model in which early periods are especially likely to be hingey.
I don’t think I agree with this, unless one is able to make a comparative claim about the importance (from a longtermist perspective) of these events relative to future events’ importance—which is exactly what I’m questioning.
I do think that weighting earlier generations more heavily is correct, though; I don’t feel that much turns on whether one construes this as prior choice or an update from one’s prior.
A related outside-view argument for the HoH being more likely to occur in earlier centuries:
New things must happen more frequently in earlier centuries because over time, we will run out of new things to do.
HoH will probably occur due to some significant thing (or things) happening.
HoH must coincide with the first occurrence of this thing, because later occurrences of the same thing or similar things cannot be more important.
If we accept these premises, this justifies using a diminishing prior like Laplace.
As a general rule if you have a domain like this that extends indefinitely in one direction, the correct prior is one that diminishes as you move further away in that direction, rather than picking a somewhat arbitrary end point and using a uniform prior on that.
Just a quick thought on this issue: Using Laplace’s rule of succession (or any other similar prior) also requires picking a somewhat arbitrary start point. You suggest 200000BC as a start point, but one could of course pick earlier or later years and get out different numbers. So the uniform prior’s sensitivity to decisions about how to truncate the relevant time interval isn’t a special weakness; it doesn’t seem to provide grounds for prefering the Laplacian prior.
I think that for some notion of an “arbitrary superlative,” a uniform prior also makes a lot more intuitive sense than a Laplacian prior. The Laplacian prior would give very strange results, for example, if you tried to use it to estimate the hottest day on Earth, the year with the highest portion of Americans named Zach, or the year with the most supernovas.
Moreover in your case in particular, there are also good reasons to suspect that the chance of a century being the most influential should diminish over time.
I agree with this intuition, but I suppose see it as a reason to shift away from a uniform prior rather than to begin from something as lopsided as a Laplacian. I think that this intuition is also partially (but far from entirely) counterbalanced by the countervailing intuitions Will lists for expecting influence to increase over time.
Just a quick thought on this issue: Using Laplace’s rule of succession (or any other similar prior) also requires picking a somewhat arbitrary start point.
Doesn’t the uniform prior require picking an arbitrary start point and end point? If so, switching to a prior that only requires an arbitrary start point seems like an improvement, all else equal. (Though maybe still worth pointing out that all arbitrariness has not been eliminated, as you’ve done here.)
You are right that having a fuzzy starting point for when we started drawing from the urn causes problems for Laplace’s Law of Succession, making it less appropriate without modification. However, note that in terms of people who have ever lived, there isn’t that much variation as populations were so low for so long, compared to now.
I see your point re ‘arbitrary superlatives’, but am not sure it goes through technically. If I could choose a prior over the relative timescale of beginning to the final year of humanity, I would intuitively have peaks at both ends. But denominated in years, we don’t know where the final year is and have a distribution over this that smears that second peak out over a long time. This often leaves us just with the initial peak and a monotonic decline (though not necessarily of the functional form of LLS). That said, this interacts with your first point, as the beginning of humanity is also vague, smearing that peak out somewhat too.
So your prior says, unlike Will’s, that there are non-trivial probabilities of very early lock-in. That seems plausible and important. But it seems to me that your analysis not only uses a different prior but also conditions on “we live extremely early” which I think is problematic.
Will argues that it’s very weird we seem to be at an extremely hingy time. So we should discount that possibility. You say that we’re living at an extremely early time and it’s not weird for early times to be hingy. I imagine Will’s response would be “it’s very weird we seem to be living at an extremely early time then” (and it’s doubly weird if it implies we live in an extremely hingy time).
If living at an early time implies something that is extremely unlikely a priori for a random person from the timeline, then there should be an explanation. These 3 explanations seem exhaustive:
1) We’re extremely lucky.
2) We aren’t actually early: E.g. we’re in a simulation or the future is short. (The latter doesn’t necessarily imply that xrisk work doesn’t have much impact because the future might just be short in terms of people in our anthropic reference class).
3) Early people don’t actually have outsized influence: E.g. the hazard/hinge rate in your model is low (perhaps 1/N where N is the length of the future). In a Bayesian graphical model, there should be a strong update in favor of low hinge rates after observing that we live very early (unless another explanation is likely a priori).
Both 2) and 3) seem somewhat plausible a priori so it seems we don’t need to assume that a big coincidence explains how early we live.
I don’t think I’m building in any assumptions about living extremely early—in fact I think it makes as little assumption on that as possible. The prior you get from LLS or from Gott’s doomsday argument says the median number of people to follow us is as many as have lived so far (~100 billion), that we have an equal chance of being in any quantile, and so for example we only have a 1 in a million chance of living in the first millionth. (Though note that since each order of magnitude contributes an equal expected value and there are infinitely many orders of magnitude, the expected number of people is infinite / has no mean.)
If you’re just presenting a prior I agree that you’ve not conditioned on an observation “we’re very early”. But to the extent that your reasoning says there’s a non-trivial probability of [we have extremely high influence over a big future], you do condition on some observation of that kind. In fact, it would seem weird if any Copernican prior could give non-trivial mass to that proposition without an additional observation.
I continue my response here because the rest is more suitable as a higher-level comment.
What is a Copernican prior? I can’t find any google results
It’s just an informal way to say that we’re probably typical observers. It’s named after Copernicus because he found that the Earth isn’t as special as people thought.
I don’t know the history of the term or its relationship to Copernicus, but I can say how my forgotten source defined it. Suppose you want to ask, “How long will my car run?” Suppose it’s a weird car that has a different engine and manufacturer than other cars, so those cars aren’t much help. One place you could start is with how long it’s currently be running for. This is based on the prior that you’re observing it on average halfway through its life. If it’s been running for 6 months so far, you would guess 1 year. There surely exists a more rigorous definition than this, but that’s the gist.
Wikipedia gives the physicist’s version, but EAs (and maybe philosophers?) use it more broadly.
The short summary I use to describe it is that “we” are not that special, for various definitions of the word we.
Some examples on FB.
>> And it gets even more so when you run it in terms of persons or person years (as I believe you should). i.e. measure time with a clock that ticks as each lifetime ends, rather than one that ticks each second. e.g. about 1/20th of all people who have ever lived are alive now, so the next century it is not really 1⁄2,000th of human history but more like 1/20th of it.
And if you use person-years, you get something like 1⁄7 − 1/14! 
>> I doubt I can easily convince you that the prior I’ve chosen is objectively best, or even that it is better than the one you used. Prior-choice is a bit of an art, rather like choice of axioms.
I’m pretty confused about how these dramatically different priors are formed, and would really appreciate it if somebody (maybe somebody less busy than Will or Toby?) could give pointers on how to read up more on forming these sort of priors. As you allude to, this question seems to map to anthropics, and I’m curious how much the priors here necessarily maps to your views on anthropics. Eg, am I reading the post and your comment correctly that Will takes an SIA view and you take an SSA view on anthropic questions?
In general, does anybody have pointers on how best to reason about anthropic and anthropic-adjacent questions?
On your prior,
P(high influence) isn’t tiny. But if I understand correctly, that’s just because
P(high influence | short future) isn’t tiny whereas
P(high influence | long future) is still tiny. (I haven’t checked the math, correct me if I’m wrong).
So your argument doesn’t seems to save existential risk work. The only way to get a non-trivial P(high influence | long future) with your prior seems to be by conditioning on an additional observation “we’re extremely early”. As I argued here, that’s somewhat sketchy to do.
I don’t have time to get into all the details, but I think that while your intuition is reasonable (I used to share it) the maths does actually turn out my way. At least on one interpretation of what you mean. I looked into this when wondering if the doomsday argument suggested that the EV of the future must be small. Try writing out the algebra for a Gott style prior that there is an x% chance we are in the first x%, for all x. You get a Pareto distribution that is a power law with infinite mean. While there is very little chance on this prior that there is a big future ahead, the size of each possible future compensates for that, such that each order of magnitude of increasing size of the future contributes an equal expected amount of population to the future, such that the sum is infinite.
I’m not quite sure what to make of this, and it may be quite brittle (e.g. if we were somehow certain that there weren’t more than 10^100 people in the future, the expected population wouldn’t be all that high), but as a raw prior I really think it is both an extreme outside view, saying we are equally likely to live at any relative position in the sequence *and* that there is extremely high (infinite) EV in the future—not because it thinks there is any single future whose EV is high, but because the series diverges.
This isn’t quite the same as your claim (about influence), but does seem to ‘save existential risk work’ from this challenge based on priors (I don’t actually think it needed saving, but that is another story).
The diverging series seems to be a version of the St Petersburg paradox, which has fooled me before. In the original version, you have a 2^-k chance of winning 2^k for every positive integer k, which leads to infinite expected payoff. One way in which it’s brittle is that, as you say, the payoff is quite limited if we have some upper bound on the size of the population. Two other mathematical ways are 1) if the payoff is just 1.99^k or 2) if it is 2^0.99k.
On second thoughts, I think it’s worth clarifying that my claim is still true even though yours is important in its own right. On Gott’s reasoning, P(high influence | world has 2^N times the # of people who’ve already lived) is still just 2^-N (that’s 2^-(N-1) if summed over all k>=N). As you said, these tiny probabilities are balanced out by asymptotically infinite impact.
I’ll write up a separate objection to that claim but first a clarifying question: Why do you call Gott’s conditional probability a prior? Isn’t it more of a likelihood? In my model it should be combined with a prior P(number of people the world has). The resulting posterior is then the prior for further enquiries.
As you wrote, the future being short “doesn’t necessarily imply that xrisk work doesn’t have much impact because the future might just be short in terms of people in our anthropic reference class”.
Another thought that comes to mind is that there may exist many evolved civilizations that their behavior is correlated with our behavior. If so, us deciding to work hard on reducing x-risks means it’s more likely that those other civilizations would also decide—during early centuries—to work hard on reducing x-risks.
Under Toby’s prior, what is the prior probability that the most influential century ever is in the past?
Quite high. If you think it hasn’t happened yet, then this is a problem for my prior that Will’s doesn’t have.
More precisely, the argument I sketched gives a prior whose PDF decays roughly as 1/n^2 (which corresponds to the chance of it first happening in the next period after n absences decaying as ~1/n). You might be able to get some tweaks to this such that it is less likely than not to happen by now, but I think the cleanest versions predict it would have happened by now. The clean version of Laplace’s Law of Succession, measured in centuries, says there would only be a 1⁄2,001 chance it hadn’t happened before now, which reflects poorly on the prior, but I don’t think it quite serves to rule it out. If you don’t know whether it has happened yet (e.g. you are unsure of things like Will’s Axial Age argument), this would give some extra weight to that possibility.
Given this, if one had a hyperprior over different possible Beta distributions, shouldn’t 2000 centuries of no event occurring cause one to update quite hard against the (0.5, 0.5) or (1, 1) hyperparameters, and in favour of a prior that was massively skewed towards the per-century probability of no-lock-in-event being very low?
(And noting that, depending exactly on how the proposition is specified, I think we can be very confident that it hasn’t happened yet. E.g. if the proposition under consideration was ‘a values lock-in event occurs such that everyone after this point has the same values’.)
That’s interesting. Earlier I suggested that a mixture of different priors that included some like mine would give a result very different to your result. But you are right to say that we can interpret this in two ways: as a mixture of ur priors or as a mixture of priors we get after updating on the length of time so far. I was implicitly assuming the latter, but maybe the former is better and it would indeed lessen or eliminate the effect I mentioned.
Your suggestion is also interesting as a general approach, choosing a distribution over these Beta distributions instead of debating between certainty in (0,0), (0.5, 0.5), and (1,1). For some distributions over Beta parameters these the maths is probably quite tractable. That might be an answer to the right meta-rational approach rather than an answer to the right rational approach, or something, but it does seem nicely robust.
I don’t understand this. Your last comment suggests that there may be several key events (some of which may be in the past), but I read your top-level comment as assuming that there is only one, which precludes all future key events (i.e. something like lock-in or extinction). I would have interpreted your initial post as follows:
Suppose we observe 20 past centuries during which no key event happens. By Laplace’s Law of Succession, we now think that the odds are 1⁄22 in each century. So you could say that the odds that a key event “would have occurred” over the course of 20 centuries is 1 - (1-1/22)^20 = 60.6%. However, we just said that we observed no key event, and that’s what our “hazard rate” is based on, so it is moot to ask what could have been. The probability is 0.
This seems off, and I think the problem is equating “no key event” with “not hingy”, which is too simple because one can potentially also influence key events in the distant future. (Or perhaps there aren’t even any key events, or there are other ways to have a lasting impact.)
I know this is an old thread, and I’m not totally sure how this affects the debate here, but for what it’s worth I think applying principle of indifference-type reasoning here implies that the appropriate uninformative prior is an exponential distribution.
I apply the principle of indifference (or maybe of invariance, following Jaynes (1968)) as follows: If I wake up tomorrow knowing absolutely nothing about the world and am asked about the probability of 10 days into the future containing the most important time in history conditional on it being in the future, I should give the same answer as if I were to be woken up 100 years from now and were asked about the day 100 years and 10 days from now. I would need some further information (e.g. about the state of the world, of human society, etc.) to say why one would be more probable than the other, and here I’m looking for a prior from a state of total ignorance.
This invariance can be generalized as: Pr(X>t+k|X>t) = Pr(X>t’+k|X>t’) for all k, t, t’. This happens to be the memoryless property, and the exponential distribution is the only continuous distribution that has this property. Thus if we think that our priors from a state of total ignorance should satisfy this requirement, our prior needs to be an exponential distribution. I imagine there are other ways of characterizing similar indifference requirements that imply memorylessness.
This is not to say our current beliefs should follow this distribution: we have additional information about the world, and we should update on this information. It’s also possible that the principle of indifference might be applied in a different way to give a different uninformative prior as in the Bertrand paradox.
(The Jaynes paper: https://bayes.wustl.edu/etj/articles/prior.pdf)
The following is yet another perspective on which prior to use, which questions whether we should assume some kind of uniformity principle:
As has been discussed in other comments and the initial text, there are some reasons to expect later times to be hingier (e.g. better knowledge) and there are some reasons to expect earlier times to be hingier (e.g. because of smaller populations). It is plausible that these reasons skew one way or another, and this effect might outweigh other sources of variance in hinginess.
That means that the hingiest times are disproportionately likely to be either a) the earliest generation (e.g. humans in pre-historic population bottlenecks) or b) the last generation (i.e. the time just before some lock-in happens). Our time is very unlikely to be the hingiest in this perspective (unless you think that lock-in happens very soon). So this suggests a low prior for HoH; however, what matters is arguably comparing present hinginess to the future, rather than to the past. And in this perspective it would be not-very-unlikely that our time is hingier than all future times.
In other words, rather than there being anything special about our time, it could just the case that a) hinginess generally decreases over time and b) this effect is stronger than other sources of variance in hinginess. I’m fairly agnostic about both of these claims, and Will argued against a), but it’s surely likelier than 1 in 100000 (in the absense of further evidence), and arguably likelier even than 5%. (This isn’t exactly HoH because past times would be even hingier.)
At least in Will’s model, we are among the earliest human generations, so I don’t think this argument holds very much, unless you posit a very fast diminishing prior (which so far nobody has done).