”But what is your posterior? Like Buck, I’m unclear whether your view is the central estimate should be (e.g.) 0.1% or 1 / 1 million.”
I’m surprised this wasn’t clear to you, which has made me think I’ve done a bad job of expressing myself.
It’s the former, and for the reason of your explanation (2): us being early, being on a single planet, being at such a high rate of economic growth, should collectively give us an enormous update. In the blog post I describe what I call the outside-view arguments, including that we’re very early on, and say: “My view is that, in the aggregate, these outside-view arguments should substantially update one from one’s prior towards HoH, but not all the way to significant credence in HoH.[3] [3] Quantitatively: These considerations push me to put my posterior on HoH into something like the [1%, 0.1%] interval. But this credence interval feels very made-up and very unstable.”
I’m going to think more about your claim that in the article I’m ‘hiding the ball’. I say in the introduction that “there are some strong arguments for thinking that this century might be unusually influential”, discuss the arguments that I think really should massively update us in section 5 of the article, and in that context I say “We have seen that there are some compelling arguments for thinking that the present time is unusually influential. In particular, we are growing very rapidly, and civilisation today is still small compared to its potential future size, so any given unit of resources is a comparatively large fraction of the whole. I believe these arguments give us reason to think that the most influential people may well live within the next few thousand years.” Then in the conclusion I say: “There are some good arguments for thinking that our time is very unusual, if we are at the start of a very long-lived civilisation: the fact that we are so early on, that we live on a single planet, and that we are at a period of rapid economic and technological progress, are all ways in which the current time is very distinctive, and therefore are reasons why we may be highly influential too.” That seemed clear to me, but I should judge clarity by how readers interpret what I’ve written.
For my part, I’m more partial to ‘blaming the reader’, but (evidently) better people mete out better measure than I in turn.
Insofar as it goes, I think the challenge (at least for me) is qualitative terms can cover multitudes (or orders of magnitudes) of precision. I’d take ~0.3% to be ‘significant’ credence for some values of significant. ‘Strong’ ‘compelling’ or ‘good’ arguments could be an LR of 2 (after all, RCT confirmation can be ~3) or 200.
I also think quantitative articulation would help the reader (or at least this reader) better benchmark the considerations here. Taking the rough posterior of 0.1% and prior of 1 in 100 million, this implies a likelihood ratio of ~~100 000 - loosely, ultra-decisive evidence. If we partition out the risk-based considerations (which it discussion seems to set as ‘less than decisive’ so <100), the other considerations (perhaps mostly those in S5) give you a LR of > ~1000 - loosely, very decisive evidence.
Yet the discussion of the considerations in S5 doesn’t give the impression we should conclude they give us ‘massive updates’. You note there are important caveats to these considerations, you say in summing up these arguments are ‘far from watertight’, and I also inferred the sort of criticisms given in S3 around our limited reasoning ability and scepticism of informal arguments would also apply here too. Hence my presumption these other considerations, although more persuasive than object level arguments around risks, would still end up below the LR ~ 100 for ‘decisive’ evidence, rather than much higher.
Another way this would help would be illustrating the uncertainty. Given some indicative priors you note vary by ten orders of magnitude, the prior is not just astronomical but extremely uncertain. By my lights, the update doesn’t greatly reduce our uncertainty (and could compound it, given challenges in calibrating around very high LRs). If the posterior odds could be ‘out by 100 000x either way’ the central estimate being at ~0.3% could still give you (given some naive log-uniform) 20%+ mass distributed at better than even odds of HH.
The moaning about hiding the ball arises from the sense this numerical articulation reveals (I think) some powerful objections the more qualitative treatment obscures. E.g.
Typical HH proponents are including considerations around earliness/single planet/ etc. in their background knowledge/prior when discussing object level risks. Noting the prior becomes astronomically adverse when we subtract these out of background knowledge, and so the object level case for (e.g.) AI risk can’t possibly be enough to carry the day alone seems a bait-and-switch: you agree the prior becomes massively less astronomical when we include single planet etc. in background knowledge, and in fact things like ‘we live on only one planet’ are in our background knowledge (and were being assumed at least tacitly by HH proponents).
The attempt to ‘bound’ object level arguments by their LR (e.g. “Well, these are informal, and it looks fishy, etc. so it is hard to see how you can get LR >100 from these”) doesn’t seem persuasive when your view is that the set of germane considerations (all of which seem informal, have caveats attached, etc.) in concert are giving you an LR of ~100 000 or more. If this set of informal considerations can get you more than half way from the astronomical prior to significant credence, why be so sure additional ones (e.g.) articulating a given danger can’t carry you the rest of the way?
I do a lot of forecasting, and I struggle to get a sense of what priors of 1/ 100 M or decisive evidence to the tune of LR 1000 would look like in ‘real life’ scenarios. Numbers this huge (where you end up virtually ‘off the end of the tail’ of your stipulated prior) raise worries about consilience (cf. “I guess the sub-prime morgage crisis was a 10 sigma event”), but moreover pragmatic defeat: there seems a lot to distrust in an epistemic procedure along the lines of “With anthropics given stipulated subtracted background knowledge we end up with an astronomically minute prior (where we could be off by many orders of magnitude), but when we update on adding back in elements of our actual background knowledge this shoots up by many orders of magnitude (but we are likely still off by many orders of magnitude)”. Taking it face value would mean a minute update to our ‘pre theoretic prior’ on the topic before embarking on this exercise (providing these overlapped and was not as radically uncertain, varying no more than a couple rather than many orders of magnitude). If we suspect (which I think we should) this procedure of partitioning out background knowledge into update steps which approach log log variance and where we have minimal calibration is less reliable than using our intuitive gestalt over our background knowledge as whole, we should discount its deliverances still further.
Thanks Greg - I asked and it turned out I had one remaining day to make edits to the paper, so I’ve made some minor ones in a direction you’d like, though I’m sure they won’t be sufficient to satisfy you.
Going to have to get back on with other work at this point, but I think your arguments are important, though the ‘bait and switch’ doesn’t seem totally fair—e.g. the update towards living in a simulation only works when you appreciate the improbability of living on a single planet.
How much of that 0.1% comes from worlds where your outside view argument is right vs worlds where your outside view argument is wrong?
This kind of stuff is pretty complicated so I might not be making sense here, but here’s what I mean: I have some distribution over what model to be using to answer the “are we at HoH” question, and each model has some probability that we’re at HoH, and I derive my overall belief by adding up the credence in HoH that I get from each model (weighted by my credence in it). It seems like your outside view model assigns approximately zero probability to HoH, and so if now is the HoH, it’s probably because we shouldn’t be using your model, rather than because we’re in the tiny proportion of worlds in your model where now is HoH.
I think this distinction is important because it seems to me that the probability of HoH give your beliefs should be almost entirely determined by the prior and HoH-likelihood of models other than the one you proposed—if your central model is the outside-view model you proposed, and you’re 80% confident in that, then I suspect that the majority of your credence on HoH should come from the other 20% of your prior, and so the question of how much your outside-view-model updates based on evidence doesn’t seem likely to be very important.
Thanks for this, Greg.
”But what is your posterior? Like Buck, I’m unclear whether your view is the central estimate should be (e.g.) 0.1% or 1 / 1 million.”
I’m surprised this wasn’t clear to you, which has made me think I’ve done a bad job of expressing myself.
It’s the former, and for the reason of your explanation (2): us being early, being on a single planet, being at such a high rate of economic growth, should collectively give us an enormous update. In the blog post I describe what I call the outside-view arguments, including that we’re very early on, and say: “My view is that, in the aggregate, these outside-view arguments should substantially update one from one’s prior towards HoH, but not all the way to significant credence in HoH.[3]
[3] Quantitatively: These considerations push me to put my posterior on HoH into something like the [1%, 0.1%] interval. But this credence interval feels very made-up and very unstable.”
I’m going to think more about your claim that in the article I’m ‘hiding the ball’. I say in the introduction that “there are some strong arguments for thinking that this century might be unusually influential”, discuss the arguments that I think really should massively update us in section 5 of the article, and in that context I say “We have seen that there are some compelling arguments for thinking that the present time is unusually influential. In particular, we are growing very rapidly, and civilisation today is still small compared to its potential future size, so any given unit of resources is a comparatively large fraction of the whole. I believe these arguments give us reason to think that the most influential people may well live within the next few thousand years.” Then in the conclusion I say: “There are some good arguments for thinking that our time is very unusual, if we are at the start of a very long-lived civilisation: the fact that we are so early on, that we live on a single planet, and that we are at a period of rapid economic and technological progress, are all ways in which the current time is very distinctive, and therefore are reasons why we may be highly influential too.” That seemed clear to me, but I should judge clarity by how readers interpret what I’ve written.
For my part, I’m more partial to ‘blaming the reader’, but (evidently) better people mete out better measure than I in turn.
Insofar as it goes, I think the challenge (at least for me) is qualitative terms can cover multitudes (or orders of magnitudes) of precision. I’d take ~0.3% to be ‘significant’ credence for some values of significant. ‘Strong’ ‘compelling’ or ‘good’ arguments could be an LR of 2 (after all, RCT confirmation can be ~3) or 200.
I also think quantitative articulation would help the reader (or at least this reader) better benchmark the considerations here. Taking the rough posterior of 0.1% and prior of 1 in 100 million, this implies a likelihood ratio of ~~100 000 - loosely, ultra-decisive evidence. If we partition out the risk-based considerations (which it discussion seems to set as ‘less than decisive’ so <100), the other considerations (perhaps mostly those in S5) give you a LR of > ~1000 - loosely, very decisive evidence.
Yet the discussion of the considerations in S5 doesn’t give the impression we should conclude they give us ‘massive updates’. You note there are important caveats to these considerations, you say in summing up these arguments are ‘far from watertight’, and I also inferred the sort of criticisms given in S3 around our limited reasoning ability and scepticism of informal arguments would also apply here too. Hence my presumption these other considerations, although more persuasive than object level arguments around risks, would still end up below the LR ~ 100 for ‘decisive’ evidence, rather than much higher.
Another way this would help would be illustrating the uncertainty. Given some indicative priors you note vary by ten orders of magnitude, the prior is not just astronomical but extremely uncertain. By my lights, the update doesn’t greatly reduce our uncertainty (and could compound it, given challenges in calibrating around very high LRs). If the posterior odds could be ‘out by 100 000x either way’ the central estimate being at ~0.3% could still give you (given some naive log-uniform) 20%+ mass distributed at better than even odds of HH.
The moaning about hiding the ball arises from the sense this numerical articulation reveals (I think) some powerful objections the more qualitative treatment obscures. E.g.
Typical HH proponents are including considerations around earliness/single planet/ etc. in their background knowledge/prior when discussing object level risks. Noting the prior becomes astronomically adverse when we subtract these out of background knowledge, and so the object level case for (e.g.) AI risk can’t possibly be enough to carry the day alone seems a bait-and-switch: you agree the prior becomes massively less astronomical when we include single planet etc. in background knowledge, and in fact things like ‘we live on only one planet’ are in our background knowledge (and were being assumed at least tacitly by HH proponents).
The attempt to ‘bound’ object level arguments by their LR (e.g. “Well, these are informal, and it looks fishy, etc. so it is hard to see how you can get LR >100 from these”) doesn’t seem persuasive when your view is that the set of germane considerations (all of which seem informal, have caveats attached, etc.) in concert are giving you an LR of ~100 000 or more. If this set of informal considerations can get you more than half way from the astronomical prior to significant credence, why be so sure additional ones (e.g.) articulating a given danger can’t carry you the rest of the way?
I do a lot of forecasting, and I struggle to get a sense of what priors of 1/ 100 M or decisive evidence to the tune of LR 1000 would look like in ‘real life’ scenarios. Numbers this huge (where you end up virtually ‘off the end of the tail’ of your stipulated prior) raise worries about consilience (cf. “I guess the sub-prime morgage crisis was a 10 sigma event”), but moreover pragmatic defeat: there seems a lot to distrust in an epistemic procedure along the lines of “With anthropics given stipulated subtracted background knowledge we end up with an astronomically minute prior (where we could be off by many orders of magnitude), but when we update on adding back in elements of our actual background knowledge this shoots up by many orders of magnitude (but we are likely still off by many orders of magnitude)”. Taking it face value would mean a minute update to our ‘pre theoretic prior’ on the topic before embarking on this exercise (providing these overlapped and was not as radically uncertain, varying no more than a couple rather than many orders of magnitude). If we suspect (which I think we should) this procedure of partitioning out background knowledge into update steps which approach log log variance and where we have minimal calibration is less reliable than using our intuitive gestalt over our background knowledge as whole, we should discount its deliverances still further.
Thanks Greg - I asked and it turned out I had one remaining day to make edits to the paper, so I’ve made some minor ones in a direction you’d like, though I’m sure they won’t be sufficient to satisfy you.
Going to have to get back on with other work at this point, but I think your arguments are important, though the ‘bait and switch’ doesn’t seem totally fair—e.g. the update towards living in a simulation only works when you appreciate the improbability of living on a single planet.
How much of that 0.1% comes from worlds where your outside view argument is right vs worlds where your outside view argument is wrong?
This kind of stuff is pretty complicated so I might not be making sense here, but here’s what I mean: I have some distribution over what model to be using to answer the “are we at HoH” question, and each model has some probability that we’re at HoH, and I derive my overall belief by adding up the credence in HoH that I get from each model (weighted by my credence in it). It seems like your outside view model assigns approximately zero probability to HoH, and so if now is the HoH, it’s probably because we shouldn’t be using your model, rather than because we’re in the tiny proportion of worlds in your model where now is HoH.
I think this distinction is important because it seems to me that the probability of HoH give your beliefs should be almost entirely determined by the prior and HoH-likelihood of models other than the one you proposed—if your central model is the outside-view model you proposed, and you’re 80% confident in that, then I suspect that the majority of your credence on HoH should come from the other 20% of your prior, and so the question of how much your outside-view-model updates based on evidence doesn’t seem likely to be very important.