Ilya’s company website says “Superintelligence is within reach.” I think it’s reasonable to interpret that as having a short timeline. If not an even stronger claim that he thinks he knows how to actually build it.
The post gives a specific example of this: the “software intelligence explosion” concept.
Looking at the methodology section you linked to, this really just confirms the accuracy of nostalgebraist’s critique, for me. (nostalgebraist is the Tumblr blogger.) There are a lot of guesses and intuitions. Such as:
Overall we’d guess that this is the sort of limitation that would take years to overcome—but not decades; just look at the past decade of progress and consider how many similar barriers have been overcome. E.g. in the history of game-playing RL AIs, we went from AlphaGo to EfficientZero in about a decade.
Remember, we are assuming SC is reached in Mar 2027. We think that most possible barriers that would block SAR from being feasible in 2027 would also block SC from being feasible in 2027.
So in this case we guess that with humans doing the AI R&D, it would take about 2-15 years.
Okay? I’m not necessarily saying this is an unreasonable opinion. I don’t really know. But this is fundamentally a process of turning intuitions into numbers and turning numbers into a mathematical model. The mathematical model doesn’t make the intuitions any more (or less) correct.
Why not 2-15 months? Why not 20-150 years? Why not 4-30 years? It’s ultimately about what the authors intuitively find plausible. Other well-informed people could reasonably find very different numbers plausible.
And if you swap out more of the authors’ intuitions for other people’s intuitions, the end result might be AGI in 2047 or 2077 or 2177 instead of 2027.
Edit: While looking up something else, I found this paper which attempts a similar sort of exercise as the AI 2027 report and gets a very different result.
I found this paper which attempts a similar sort of exercise as the AI 2027 report and gets a very different result.
This is an example of the multiple stages fallacy (as pointed out here), where you can get arbitrarily low probabilities for anything by dividing it up enough and assuming things are uncorrelated.
I don’t find accusations of fallacy helpful here. The author’s say in the abstract explicitly that they estimated the probability of each step conditional on the previous ones. So they are not making a simple, formal error like multiplying a bunch of unconditional probabilities whilst forgetting that only works if the probabilities are uncorrelated. Rather, you and Richard Ngo think that they’re estimates for the explicitly conditional probabilities are too low, and you are speculating that this is because they are still really think of the unconditional probabilities. But I don’t think “you are committing a fallacy” is a very good or fair way to describe “I disagree with your probabilities and I have some unevidenced speculation about why you are giving probabilities that are wrong”.
Saying they are conditional does not mean they are. For example, why is P(We invent a way for AGIs to learn faster than humans|We invent algorithms for transformative AGI) only 40%? Or P(AGI inference costs drop below $25/hr (per human equivalent)[1]|We invent algorithms for transformative AGI) only 16%!? These would be much more reasonable as unconditional probabilities. At the very least, “algorithms for transformative AGI” would be used to massively increase software and hardware R&D, even if expensive at first, such that inference costs would quickly drop.
I don’t think you can possibly know whether they really are actually thinking of the unconditional probabilities or whether they just have very different opinions and instincts from you about the whole domain which make very different genuinely conditional probabilities seem reasonable.
It just looks a lot like motivated reasoning to me—kind of like they started with the conclusion and worked backward. Those examples are pretty unreasonable as conditional probabilities. Do they explain why “algorithms for transformative AGI” are very unlikely to meaningfully speed up software and hardware R&D?
One of the authors responds to the comment you linked to and says he was already aware of the concept of the multiple stages fallacy when writing the paper.
But the point I was making in my comment above is how easy it is for reasonable, informed people to generate different intuitions that form the fundamental inputs of a forecasting model like AI 2027. For example, the authors intuit that something would take years, not decades, to solve. Someone else could easily intuit it will take decades, not years.
The same is true for all the different intuitions the model relies on to get to its thrilling conclusion.
Since the model can only exist by using many such intuitions as inputs, ultimately the model is effectively a re-statement of these intuitions, and putting these intuitions into a model doesn’t make them any more correct.
In 2-3 years, when it turns out the prediction of AGI in 2027 is wrong, it probably won’t be because of a math error in the model but rather because the intuitions the model is based on are wrong.
If they were already aware, they certainly didn’t do anything to address it, given their conclusion is basically a result of falling for it.
It’s more than just intuitions, it’s grounded in current research and recent progress in (proto) AGI. To validate the opposing intuitions (long timelines) requires more in the way of leaps of faith (to say that things will suddenly stop working as they have been). Longer timelines intuitions have also been proven wrong consistently over the last few years (e.g. AI constantly doing things people predicted were “decades away” just a few years, or even months, before).
Ilya’s company website says “Superintelligence is within reach.” I think it’s reasonable to interpret that as having a short timeline. If not an even stronger claim that he thinks he knows how to actually build it.
Right, and doesn’t address any of the meat in the methodology section.
Looking at the methodology section you linked to, this really just confirms the accuracy of nostalgebraist’s critique, for me. (nostalgebraist is the Tumblr blogger.) There are a lot of guesses and intuitions. Such as:
Okay? I’m not necessarily saying this is an unreasonable opinion. I don’t really know. But this is fundamentally a process of turning intuitions into numbers and turning numbers into a mathematical model. The mathematical model doesn’t make the intuitions any more (or less) correct.
Why not 2-15 months? Why not 20-150 years? Why not 4-30 years? It’s ultimately about what the authors intuitively find plausible. Other well-informed people could reasonably find very different numbers plausible.
And if you swap out more of the authors’ intuitions for other people’s intuitions, the end result might be AGI in 2047 or 2077 or 2177 instead of 2027.
Edit: While looking up something else, I found this paper which attempts a similar sort of exercise as the AI 2027 report and gets a very different result.
This is an example of the multiple stages fallacy (as pointed out here), where you can get arbitrarily low probabilities for anything by dividing it up enough and assuming things are uncorrelated.
I don’t find accusations of fallacy helpful here. The author’s say in the abstract explicitly that they estimated the probability of each step conditional on the previous ones. So they are not making a simple, formal error like multiplying a bunch of unconditional probabilities whilst forgetting that only works if the probabilities are uncorrelated. Rather, you and Richard Ngo think that they’re estimates for the explicitly conditional probabilities are too low, and you are speculating that this is because they are still really think of the unconditional probabilities. But I don’t think “you are committing a fallacy” is a very good or fair way to describe “I disagree with your probabilities and I have some unevidenced speculation about why you are giving probabilities that are wrong”.
Saying they are conditional does not mean they are. For example, why is P(We invent a way for AGIs to learn faster than humans|We invent algorithms for transformative AGI) only 40%? Or P(AGI inference costs drop below $25/hr (per human equivalent)[1]|We invent algorithms for transformative AGI) only 16%!? These would be much more reasonable as unconditional probabilities. At the very least, “algorithms for transformative AGI” would be used to massively increase software and hardware R&D, even if expensive at first, such that inference costs would quickly drop.
As an aside, surely this milestone has basically now already been reached? At least for the 90% percentile human in most intellectual tasks.
I don’t think you can possibly know whether they really are actually thinking of the unconditional probabilities or whether they just have very different opinions and instincts from you about the whole domain which make very different genuinely conditional probabilities seem reasonable.
It just looks a lot like motivated reasoning to me—kind of like they started with the conclusion and worked backward. Those examples are pretty unreasonable as conditional probabilities. Do they explain why “algorithms for transformative AGI” are very unlikely to meaningfully speed up software and hardware R&D?
One of the authors responds to the comment you linked to and says he was already aware of the concept of the multiple stages fallacy when writing the paper.
But the point I was making in my comment above is how easy it is for reasonable, informed people to generate different intuitions that form the fundamental inputs of a forecasting model like AI 2027. For example, the authors intuit that something would take years, not decades, to solve. Someone else could easily intuit it will take decades, not years.
The same is true for all the different intuitions the model relies on to get to its thrilling conclusion.
Since the model can only exist by using many such intuitions as inputs, ultimately the model is effectively a re-statement of these intuitions, and putting these intuitions into a model doesn’t make them any more correct.
In 2-3 years, when it turns out the prediction of AGI in 2027 is wrong, it probably won’t be because of a math error in the model but rather because the intuitions the model is based on are wrong.
If they were already aware, they certainly didn’t do anything to address it, given their conclusion is basically a result of falling for it.
It’s more than just intuitions, it’s grounded in current research and recent progress in (proto) AGI. To validate the opposing intuitions (long timelines) requires more in the way of leaps of faith (to say that things will suddenly stop working as they have been). Longer timelines intuitions have also been proven wrong consistently over the last few years (e.g. AI constantly doing things people predicted were “decades away” just a few years, or even months, before).