For my part, I’m more partial to ‘blaming the reader’, but (evidently) better people mete out better measure than I in turn.
Insofar as it goes, I think the challenge (at least for me) is qualitative terms can cover multitudes (or orders of magnitudes) of precision. I’d take ~0.3% to be ‘significant’ credence for some values of significant. ‘Strong’ ‘compelling’ or ‘good’ arguments could be an LR of 2 (after all, RCT confirmation can be ~3) or 200.
I also think quantitative articulation would help the reader (or at least this reader) better benchmark the considerations here. Taking the rough posterior of 0.1% and prior of 1 in 100 million, this implies a likelihood ratio of ~~100 000 - loosely, ultra-decisive evidence. If we partition out the risk-based considerations (which it discussion seems to set as ‘less than decisive’ so <100), the other considerations (perhaps mostly those in S5) give you a LR of > ~1000 - loosely, very decisive evidence.
Yet the discussion of the considerations in S5 doesn’t give the impression we should conclude they give us ‘massive updates’. You note there are important caveats to these considerations, you say in summing up these arguments are ‘far from watertight’, and I also inferred the sort of criticisms given in S3 around our limited reasoning ability and scepticism of informal arguments would also apply here too. Hence my presumption these other considerations, although more persuasive than object level arguments around risks, would still end up below the LR ~ 100 for ‘decisive’ evidence, rather than much higher.
Another way this would help would be illustrating the uncertainty. Given some indicative priors you note vary by ten orders of magnitude, the prior is not just astronomical but extremely uncertain. By my lights, the update doesn’t greatly reduce our uncertainty (and could compound it, given challenges in calibrating around very high LRs). If the posterior odds could be ‘out by 100 000x either way’ the central estimate being at ~0.3% could still give you (given some naive log-uniform) 20%+ mass distributed at better than even odds of HH.
The moaning about hiding the ball arises from the sense this numerical articulation reveals (I think) some powerful objections the more qualitative treatment obscures. E.g.
Typical HH proponents are including considerations around earliness/single planet/ etc. in their background knowledge/prior when discussing object level risks. Noting the prior becomes astronomically adverse when we subtract these out of background knowledge, and so the object level case for (e.g.) AI risk can’t possibly be enough to carry the day alone seems a bait-and-switch: you agree the prior becomes massively less astronomical when we include single planet etc. in background knowledge, and in fact things like ‘we live on only one planet’ are in our background knowledge (and were being assumed at least tacitly by HH proponents).
The attempt to ‘bound’ object level arguments by their LR (e.g. “Well, these are informal, and it looks fishy, etc. so it is hard to see how you can get LR >100 from these”) doesn’t seem persuasive when your view is that the set of germane considerations (all of which seem informal, have caveats attached, etc.) in concert are giving you an LR of ~100 000 or more. If this set of informal considerations can get you more than half way from the astronomical prior to significant credence, why be so sure additional ones (e.g.) articulating a given danger can’t carry you the rest of the way?
I do a lot of forecasting, and I struggle to get a sense of what priors of 1/ 100 M or decisive evidence to the tune of LR 1000 would look like in ‘real life’ scenarios. Numbers this huge (where you end up virtually ‘off the end of the tail’ of your stipulated prior) raise worries about consilience (cf. “I guess the sub-prime morgage crisis was a 10 sigma event”), but moreover pragmatic defeat: there seems a lot to distrust in an epistemic procedure along the lines of “With anthropics given stipulated subtracted background knowledge we end up with an astronomically minute prior (where we could be off by many orders of magnitude), but when we update on adding back in elements of our actual background knowledge this shoots up by many orders of magnitude (but we are likely still off by many orders of magnitude)”. Taking it face value would mean a minute update to our ‘pre theoretic prior’ on the topic before embarking on this exercise (providing these overlapped and was not as radically uncertain, varying no more than a couple rather than many orders of magnitude). If we suspect (which I think we should) this procedure of partitioning out background knowledge into update steps which approach log log variance and where we have minimal calibration is less reliable than using our intuitive gestalt over our background knowledge as whole, we should discount its deliverances still further.
Thanks Greg - I asked and it turned out I had one remaining day to make edits to the paper, so I’ve made some minor ones in a direction you’d like, though I’m sure they won’t be sufficient to satisfy you.
Going to have to get back on with other work at this point, but I think your arguments are important, though the ‘bait and switch’ doesn’t seem totally fair—e.g. the update towards living in a simulation only works when you appreciate the improbability of living on a single planet.
For my part, I’m more partial to ‘blaming the reader’, but (evidently) better people mete out better measure than I in turn.
Insofar as it goes, I think the challenge (at least for me) is qualitative terms can cover multitudes (or orders of magnitudes) of precision. I’d take ~0.3% to be ‘significant’ credence for some values of significant. ‘Strong’ ‘compelling’ or ‘good’ arguments could be an LR of 2 (after all, RCT confirmation can be ~3) or 200.
I also think quantitative articulation would help the reader (or at least this reader) better benchmark the considerations here. Taking the rough posterior of 0.1% and prior of 1 in 100 million, this implies a likelihood ratio of ~~100 000 - loosely, ultra-decisive evidence. If we partition out the risk-based considerations (which it discussion seems to set as ‘less than decisive’ so <100), the other considerations (perhaps mostly those in S5) give you a LR of > ~1000 - loosely, very decisive evidence.
Yet the discussion of the considerations in S5 doesn’t give the impression we should conclude they give us ‘massive updates’. You note there are important caveats to these considerations, you say in summing up these arguments are ‘far from watertight’, and I also inferred the sort of criticisms given in S3 around our limited reasoning ability and scepticism of informal arguments would also apply here too. Hence my presumption these other considerations, although more persuasive than object level arguments around risks, would still end up below the LR ~ 100 for ‘decisive’ evidence, rather than much higher.
Another way this would help would be illustrating the uncertainty. Given some indicative priors you note vary by ten orders of magnitude, the prior is not just astronomical but extremely uncertain. By my lights, the update doesn’t greatly reduce our uncertainty (and could compound it, given challenges in calibrating around very high LRs). If the posterior odds could be ‘out by 100 000x either way’ the central estimate being at ~0.3% could still give you (given some naive log-uniform) 20%+ mass distributed at better than even odds of HH.
The moaning about hiding the ball arises from the sense this numerical articulation reveals (I think) some powerful objections the more qualitative treatment obscures. E.g.
Typical HH proponents are including considerations around earliness/single planet/ etc. in their background knowledge/prior when discussing object level risks. Noting the prior becomes astronomically adverse when we subtract these out of background knowledge, and so the object level case for (e.g.) AI risk can’t possibly be enough to carry the day alone seems a bait-and-switch: you agree the prior becomes massively less astronomical when we include single planet etc. in background knowledge, and in fact things like ‘we live on only one planet’ are in our background knowledge (and were being assumed at least tacitly by HH proponents).
The attempt to ‘bound’ object level arguments by their LR (e.g. “Well, these are informal, and it looks fishy, etc. so it is hard to see how you can get LR >100 from these”) doesn’t seem persuasive when your view is that the set of germane considerations (all of which seem informal, have caveats attached, etc.) in concert are giving you an LR of ~100 000 or more. If this set of informal considerations can get you more than half way from the astronomical prior to significant credence, why be so sure additional ones (e.g.) articulating a given danger can’t carry you the rest of the way?
I do a lot of forecasting, and I struggle to get a sense of what priors of 1/ 100 M or decisive evidence to the tune of LR 1000 would look like in ‘real life’ scenarios. Numbers this huge (where you end up virtually ‘off the end of the tail’ of your stipulated prior) raise worries about consilience (cf. “I guess the sub-prime morgage crisis was a 10 sigma event”), but moreover pragmatic defeat: there seems a lot to distrust in an epistemic procedure along the lines of “With anthropics given stipulated subtracted background knowledge we end up with an astronomically minute prior (where we could be off by many orders of magnitude), but when we update on adding back in elements of our actual background knowledge this shoots up by many orders of magnitude (but we are likely still off by many orders of magnitude)”. Taking it face value would mean a minute update to our ‘pre theoretic prior’ on the topic before embarking on this exercise (providing these overlapped and was not as radically uncertain, varying no more than a couple rather than many orders of magnitude). If we suspect (which I think we should) this procedure of partitioning out background knowledge into update steps which approach log log variance and where we have minimal calibration is less reliable than using our intuitive gestalt over our background knowledge as whole, we should discount its deliverances still further.
Thanks Greg - I asked and it turned out I had one remaining day to make edits to the paper, so I’ve made some minor ones in a direction you’d like, though I’m sure they won’t be sufficient to satisfy you.
Going to have to get back on with other work at this point, but I think your arguments are important, though the ‘bait and switch’ doesn’t seem totally fair—e.g. the update towards living in a simulation only works when you appreciate the improbability of living on a single planet.