I really appreciate your approach of thoroughly going through potential issues with your eventual conclusion. It’s a really good way of getting to the interesting parts of the discussion!
The area where I’m left least convinced by is the use of Laplace’s Law of Succession (LLoC) to suggest that AGI is coming soonish (that isn’t to say there aren’t convincing arguments for this, but I think this argument probably isn’t one of them).
There are two ways of thinking that make me skeptical of using LLoC in this context (they’re related but I think it’s helpful to separate them):
1. Given a small amount of observations, there’s not enough information to “get away” from our priors. So whatever prior we load into the formula—we’re bound to get something relatively close to it. This works if we have a good reason to use a uniform prior or in contexts where we’re only trying to separate hypotheses that aren’t “far enough away” from the uniform prior, which I don’t think is the case here:
In my understanding, what we’re really trying to do is separate two hypotheses: The first is that the chance of AGI appearing in the next 50 years is non-negligible (it won’t make a huge difference to our eventual decision making if it’s 40% or 30% or 20%). The second is that it is negligible (let’s say, less than 0.1%, or one in a thousand).
When we use a uniform prior (which starts out with a 50% chance of AGI appearing within a year) - we have already loaded the formula with the answer and the method isn’t helpful to us.
2. In continuation to the “demon objection” within the text, I think the objection there could be strengthened to become a lot more convincing. The objection is that LLoC doesn’t take the specific event it’s trying to predict into account, which is strange and sounds problematic. The example given turns out ok: We’ve been trying to summon demons for thousands of years so the chance of it happening in the next 50 years is calculated to be small.
But of course, that’s just not the best example to show that LLoC is problematic in these areas:
Example 1: I have thought up of a completely new and original demon. It was obviously never attempted to summon my new and special demon until this year, when, apparently it wasn’t summoned. The LLoC chance of summoning my demon next year is quite high (and over the next 50 years is incredibly high). It’s also larger than the chance of summoning any demon (including my own) over those time periods.
The problematic nature of it isn’t just because I picked an extreme example with a single observation -
Example 2: What is the chance that the movie Psycho is meant to hypnotize everyone watching it and we’ll only realize it when Hitchcock takes over the world? Well, turns out that this hasn’t yet happened for exactly 60 years. So, it seems like the chance of this happening soon is precisely the same as the chance of AGI appearing.
Next, what is the chance of Hitchcock doing this AND Harper Lee (To Kill a Mockingbird came out in the same year) attempts doing this in a similar fashion AND Andre Cassagnes (Etch-A-Sketch is also from 1960) does so (I want to know the chance of all three happening at the exact same time)? Turns out that this specific and convoluted scenario is just as likely since it could only start happening at 1960… This is both obviously wrong and an instance of the conjunction fallacy.
Sorry, I wasn’t very clear on the first point: There isn’t a ‘correct’ prior.
In our context (by context I mean both the small number of observations and the implicit hypotheses that we’re trying to differentiate between), the prior has a large enough weight that it affects the eventual result in a way that makes the method unhelpful.
Thank you for writing this!
I really appreciate your approach of thoroughly going through potential issues with your eventual conclusion. It’s a really good way of getting to the interesting parts of the discussion!
The area where I’m left least convinced by is the use of Laplace’s Law of Succession (LLoC) to suggest that AGI is coming soonish (that isn’t to say there aren’t convincing arguments for this, but I think this argument probably isn’t one of them).
There are two ways of thinking that make me skeptical of using LLoC in this context (they’re related but I think it’s helpful to separate them):
1. Given a small amount of observations, there’s not enough information to “get away” from our priors. So whatever prior we load into the formula—we’re bound to get something relatively close to it. This works if we have a good reason to use a uniform prior or in contexts where we’re only trying to separate hypotheses that aren’t “far enough away” from the uniform prior, which I don’t think is the case here:
In my understanding, what we’re really trying to do is separate two hypotheses: The first is that the chance of AGI appearing in the next 50 years is non-negligible (it won’t make a huge difference to our eventual decision making if it’s 40% or 30% or 20%). The second is that it is negligible (let’s say, less than 0.1%, or one in a thousand).
When we use a uniform prior (which starts out with a 50% chance of AGI appearing within a year) - we have already loaded the formula with the answer and the method isn’t helpful to us.
2. In continuation to the “demon objection” within the text, I think the objection there could be strengthened to become a lot more convincing. The objection is that LLoC doesn’t take the specific event it’s trying to predict into account, which is strange and sounds problematic. The example given turns out ok: We’ve been trying to summon demons for thousands of years so the chance of it happening in the next 50 years is calculated to be small.
But of course, that’s just not the best example to show that LLoC is problematic in these areas:
Example 1: I have thought up of a completely new and original demon. It was obviously never attempted to summon my new and special demon until this year, when, apparently it wasn’t summoned. The LLoC chance of summoning my demon next year is quite high (and over the next 50 years is incredibly high). It’s also larger than the chance of summoning any demon (including my own) over those time periods.
The problematic nature of it isn’t just because I picked an extreme example with a single observation -
Example 2: What is the chance that the movie Psycho is meant to hypnotize everyone watching it and we’ll only realize it when Hitchcock takes over the world? Well, turns out that this hasn’t yet happened for exactly 60 years. So, it seems like the chance of this happening soon is precisely the same as the chance of AGI appearing.
Next, what is the chance of Hitchcock doing this AND Harper Lee (To Kill a Mockingbird came out in the same year) attempts doing this in a similar fashion AND Andre Cassagnes (Etch-A-Sketch is also from 1960) does so (I want to know the chance of all three happening at the exact same time)? Turns out that this specific and convoluted scenario is just as likely since it could only start happening at 1960… This is both obviously wrong and an instance of the conjunction fallacy.
This reminds me of the discussion around the Hinge of History Hypothesis (and the subsequent discussion of Rob Wiblin and Will Macaskill).
I’m not sure that I understand the first point. What sort of prior would be supported by this view?
The second point I definitely agree with, and the general point of being extra careful about how to use priors :)
Sorry, I wasn’t very clear on the first point: There isn’t a ‘correct’ prior.
In our context (by context I mean both the small number of observations and the implicit hypotheses that we’re trying to differentiate between), the prior has a large enough weight that it affects the eventual result in a way that makes the method unhelpful.