Audio version available at Cold Takes (or search Stitcher, Spotify, Google Podcasts, etc. for “Cold Takes Audio”)
This is one of 4 posts summarizing hundreds of pages of technical reports focused almost entirely on forecasting one number: the year by which transformative AI will be developed.1
By “transformative AI,” I mean “AI powerful enough to bring us into a new, qualitatively different future.” I specifically focus on what I’m calling PASTA: AI systems that can essentially automate all of the human activities needed to speed up scientific and technological advancement.
The sooner PASTA might be developed, the sooner the world could change radically, and the more important it seems to be thinking today about how to make that change go well vs. poorly.
In this post and the next, I will talk about the forecasting methods underlying my current view: I believe there’s more than a 10% chance we’ll see something PASTA-like enough to qualify as “transformative AI” within 15 years (by 2036); a ~50% chance we’ll see it within 40 years (by 2060); and a ~2/3 chance we’ll see it this century (by 2100).
I’m not sure whether it will feel as though transformative AI is “on the way” long before it arrives. I’m hoping, instead, that we can use trends in key underlying facts about the world (such as AI capabilities, model size, etc.) to forecast a qualitatively unfamiliar future.
An analogy for this sort of forecasting would be something like: “This water isn’t bubbling, and there are no signs of bubbling, but the temperature has gone from 70° Fahrenheit2 to 150°, and if it hits 212°, the water will bubble.” Or: “It’s like forecasting school closures and overbooked hospitals, when there aren’t any yet, based on trends in reported infections.”
Discuss whether we can look for trends in how “impressive” or “capable” AI systems are. I think this approach is unreliable: (a) AI progress may not “trend” in the way we expect; (b) in my experience, different AI researchers have radically different intuitions about which systems are impressive or capable, and how progress is going.
Briefly discuss Grace et al 2017, the best existing survey of AI researchers on transformative AI timelines. Its conclusions broadly seem in line with my own forecasts, though there are signs the researchers weren’t thinking very hard about the questions.
There are a couple of ways in which forecasting transformative AI is different from the kind of forecasting we might be used to.
First, I’m forecasting over very long time horizons (decades), unlike e.g. a weather forecast (days) or an election forecast (months). This makes the task quite a bit harder,3 and harder for outsiders to evaluate since I don’t have a clearly relevant track record of making forecasts on similar topics.
Second, I lack rich, clearly relevant data sources, and I can’t look back through a bunch of similar forecasts from the past. FiveThirtyEight’s election forecasts look at hundreds of polls, and they have a model of how well polls have predicted elections in the past. Forecasting transformative AI needs to rely more on intuition, guesswork and judgment, in terms of determining what data is most relevant and how it’s relevant.
Finally, I’m trying to forecast a qualitatively unfamiliar future. Transformative AI—and the strange future it comes with—doesn’t feel like something we’re “trending toward” year to year.
If I were trying to forecast when the world population would hit 10 billion, I could simply extrapolate existing trends of world population. World population itself is known to be growing and can be directly estimated. In my view, extrapolating out a long-running trend is one of the better ways to make a forecast.
When FiveThirtyEight makes election forecasts, there’s a background understanding that there’s going to be an election on a certain date, and whoever wins will take office on another date. We all buy into that basic framework, and there’s a general understanding that better polling means a better chance of winning.
By contrast, transformative AI—and the strange future it comes with—isn’t something we’re “headed for” in any clearly measurable way. There’s no clear metric like “transformativeness of AI” or “weirdness of the world” that’s going up regularly every year such that we can project it out into the future and get the date that something like PASTA will be developed.
Perhaps for some, these points gives enough reason to ignore the whole possibility of transformative AI, or assume it’s very far away. But I don’t think this is a good idea, for a couple of reasons.
First, I have a background view that something like PASTA is in a sense “inevitable,” assuming continued advances in society and computing. The basic intuition here—which I could expand on if there’s interest—is that human brains are numerous and don’t seem to need particular rare materials to produce, so it should be possible at some point to synthetically replicate the key parts of their functionality.4
At the same time, I’m not confident that PASTA will feel qualitatively as though it’s “on the way” well before it arrives. (More on this below.) So I’m inclined to look for ways to estimate when we can expect this development, despite the challenges, and despite the fact that it doesn’t feel today as though it’s around the corner.
I think there are plenty of example cases where a qualitatively unfamiliar future could be seen in advance by plotting the trend in some underlying, related facts about the world. A few that come to mind:
When COVID-19 first emerged, a lot of people had trouble taking it seriously because it didn’t feel as though we were “trending toward” or “headed for” a world full of overflowing hospitals, office and school closures, etc. At the time (say, January 2020), there were a relatively small number of cases, an even smaller number of deaths, and no qualitative sense of a global emergency. The only thing alarming about COVID-19, at first, was that case counts were growing at a fast exponential rate (though the overall number of cases was still small). But it was possible to extrapolate from the fast growth in case counts to a risk of a global emergency, and some people did. (And some didn’t.)
Climatologists forecast a global rise in temperatures that’s significantly more than what we’ve seen over the past few decades, and could have major consequences far beyond what we’re seeing today. They do this by forecasting trends in greenhouse gas emissions and extrapolating from there to temperature and consequences. If you simply tried to ask “How fast is the temperature rising?” or “Are hurricanes getting worse?”, and based all your forecasts of the future on those, you probably wouldn’t be forecasting the same kinds of extreme events around 2100.5
To give a more long-run example, we can project a date by which the sun will burn out, and conclude that the world will look very different by that date than it does now, even though there’s no trend of things getting colder or darker today.
COVID-19 cases from WHO. Workplace closures are from this OWiD data, simply scored as 1 for “recommended,” 2 for “required for some,” 3 for “required for all but key workers” and summed across all countries.
An analogy for this sort of forecasting would be something like: “This water isn’t bubbling, and there are no signs of bubbling, but the temperature has gone from 70° Fahrenheit6 to 150°, and if it hits 212°, the water will bubble.”
Ideally, I can find some underlying factors that are changing regularly enough for us to predict them (such as growth in the size and cost of AI models), and then argue that if those factors reach a certain point, the odds of transformative AI will be high.
You can think of this approach as answering the question: “If I think something like PASTA is inevitable, and I’m trying to guess the timing of it using a few different analysis methods, what do I guess?” We can separately ask “And is there reason that this guess is implausible, untrustworthy, or too ‘wild?’” — this was addressed in the previous piece in this series.
If we’re looking for some underlying factors in the world that predict when transformative AI is coming, perhaps the first thing we should look for is trends in how “impressive” or “capable” AI systems are.
The easiest version of this would be if the world happened to shake out such that:
One day, for the first time, an AI system managed to get a passing grade on a 4th-grade science exam.
Then we saw the first AI passing (and then acing) a 5th grade exam, then 6th grade exam, etc.
Then we saw the first AI earning a PhD, then the first AI writing a published paper, etc. all the way up to the first AI that could do Nobel-Prize-worthy science work.
This all was spread out regularly over the decades, so we could clearly see the state of the art advancing from 4th grade to 5th grade to 6th grade, all the way up to “postdoc” and beyond. And all of this happened slowly and regularly enough that we could start putting a date on “full-blown scientist AI” several decades in advance.
It would be very convenient—I almost want to say “polite”—of AI systems to advance in this manner. It would also be “polite” if AI advanced in the way that some people seem to casually imagine it will: first taking over jobs like “truck driver” and “assembly line worker,” then jobs like “teacher” and “IT support,” and then jobs like “doctor” and “lawyer,” before progressing to “scientist.”
Either of these would give us plenty of lead time and a solid basis to project when science-automating AI is coming. Unfortunately, I don’t think we can count on such a thing.
AI seems to progress very differently from humans. For example, there were superhuman AI chess players7 long before there was AI that could reliably tell apart pictures of dogs and cats.8
One possibility is that AI systems will be capable of the hardest intellectual tasks insects can do, then of the hardest tasks mice and other small mammals can do, then monkeys, then humans—effectively matching the abilities of larger and larger brains. If this happened, we wouldn’t necessarily see many signs of AI being able to e.g. do science until we were very close. Matching a 4th-grader might not happen until the very end.
Another possibility is that AI systems will be able to do anything that a human can do within 1 second, then anything that a human can do within 10 seconds, etc. This could also be quite a confusing progression that makes it non-obvious how to forecast progress.
Actually, if we didn’t already know how humans tend to mature, we might find a child’s progress to be pretty confusing and hard to extrapolate. Watching someone progress from birth to age 8 wouldn’t necessarily give you any idea that they were, say, 1⁄3 of the way to being able to start a business, make an important original scientific discovery, etc. (Even knowing the usual course of human development, it’s hard to tell from observing an 8-year-old what professional-level capabilities they could/will end up with in adulthood.)
Overall, it’s quite unclear how we should think about the spectrum from “not impressive/capable” to “very impressive/capable” for AI. And indeed, in my experience, different AI researchers have radically different intuitions about which systems are impressive or capable, and how progress is going. I’ve often had the experience of seeing one AI researcher friend point to some new result and say “This is huge, how can anyone not see how close we’re getting to powerful AI?” while another says “This is a minor advance with little significance.”9
It would be great if we could forecast the year transformative AI will be developed, by using a chart like this (from Bio Anchors; “TAI” means “transformative AI”):
But as far as I can tell, there’s no way to define the y-axis that wouldn’t be fiercely debated between experts.
Surveying experts
One way to deal with this uncertainty and confusion would be to survey a large number of experts and simply ask them when they expect transformative AI to be developed. We might hope that each of the experts (or at least, many of them) is doing their own version of the “impressiveness extrapolation” above—or if not, that they’re doing something else that can help them get a reasonable estimate. By averaging many estimates, we might get an aggregate that reflects the “wisdom of crowds.”10
I think the best version of this exercise is Grace et al 2017, a survey of 352 AI researchers that included a question about “when unaided machines can accomplish every task better and more cheaply than human workers” (which would presumably include tasks that advance scientific and technological development, and hence would qualify as PASTA). The two big takeaways from this survey, according to Bio Anchors and me, are:
A ~20% probability of this sort of AI by 2036; a ~50% probability by 2060; a ~70% probability by 2100. These match the figures I give in the introduction.
Much later estimates for slightly differently phrased questions (posed to a smaller subset of respondents), implying (to me) that the researchers simply weren’t thinking very hard about the questions.11
My bottom line: this evidence is consistent with my current probabilities, though potentially not very informative. The next piece in this series will be entirely focused on Ajeya Cotra’s “Forecasting Transformative AI with Biological Anchors,” the forecasting method I find most informative here.
Of course, the answer could be “A kajillion years from now” or “Never.” ↩
Centigrade equivalents for this sentence: 21°, 66°, 100° ↩
See also this piece for a bit of a more fleshed out argument along these lines, which I don’t agree with fully as stated (I don’t think it presents a strong case for transformative AI soon), but which I think gives a good sense of my intuitions about in-principle feasibility. Also see On the Impossibility of Supersized Machines for some implicit (joking) responses to many common arguments for why transformative AI might be impossible to create. ↩
For example, see the temperature chart here—the lowest line seems like it would be a reasonable projection, if temperature were the only thing you were looking at. ↩
Centigrade equivalents for this sentence: 21°, 66°, 100° ↩
From Bio Anchors: “We have heard ML experts with relatively short timelines argue that AI systems today can essentially see as well as humans, understand written information, and beat humans at almost all strategy games, and the set of things they can do is expanding rapidly, leading them to expect that transformative AI would be attainable in the next decade or two by training larger models on a broader distribution of ML problems that are more targeted at generating economic value. Conversely, we have heard ML experts with relatively long timelines argue that ML systems require much more data to learn than humans do, are unable to transfer what they learn in one context to a slightly different context, and don’t seem capable of much structured logical and causal reasoning; this leads them to believe we would need to make multiple major breakthroughs to develop TAI. At least one Open Philanthropy technical advisor has advanced each of these perspectives.” ↩
Wikipedia: “The classic wisdom-of-the-crowds finding … At a 1906 country fair in Plymouth, 800 people participated in a contest to estimate the weight of a slaughtered and dressed ox. Statistician Francis Galton observed that the median guess, 1207 pounds, was accurate within 1% of the true weight of 1198 pounds.” ↩
Some researchers were asked to forecast “HLMI” as defined above [human-level machine intelligence, which I would take to include something like PASTA], while a randomly-selected subset was instead asked to forecast “full automation of labor”, the time when “all occupations are fully automatable.” Despite the fact that achieving HLMI seems like it should quickly lead to full automation of labor, the median estimate for full automation of labor was ~2138 while the median estimate for HLMI was ~2061, almost 80 years earlier.
Random subsets of respondents were asked to forecast when individual milestones (e.g. laundry folding, human-level StarCraft, or human-level math research) would be achieved. The median year by which respondents expected machines to be able to automate AI research was ~2104, while the median estimate for HLMI was ~2061 -- another clear inconsistency because “AI research” is a task done by human workers.↩
Forecasting Transformative AI: Are we “trending toward” transformative AI? (How would we know?)
Audio version available at Cold Takes (or search Stitcher, Spotify, Google Podcasts, etc. for “Cold Takes Audio”)
In this post and the next, I will talk about the forecasting methods underlying my current view: I believe there’s more than a 10% chance we’ll see something PASTA-like enough to qualify as “transformative AI” within 15 years (by 2036); a ~50% chance we’ll see it within 40 years (by 2060); and a ~2/3 chance we’ll see it this century (by 2100).
Below, I will:
Discuss what kind of forecast I’m going for.
I’m not sure whether it will feel as though transformative AI is “on the way” long before it arrives. I’m hoping, instead, that we can use trends in key underlying facts about the world (such as AI capabilities, model size, etc.) to forecast a qualitatively unfamiliar future.
An analogy for this sort of forecasting would be something like: “This water isn’t bubbling, and there are no signs of bubbling, but the temperature has gone from 70° Fahrenheit2 to 150°, and if it hits 212°, the water will bubble.” Or: “It’s like forecasting school closures and overbooked hospitals, when there aren’t any yet, based on trends in reported infections.”
Discuss whether we can look for trends in how “impressive” or “capable” AI systems are. I think this approach is unreliable: (a) AI progress may not “trend” in the way we expect; (b) in my experience, different AI researchers have radically different intuitions about which systems are impressive or capable, and how progress is going.
Briefly discuss Grace et al 2017, the best existing survey of AI researchers on transformative AI timelines. Its conclusions broadly seem in line with my own forecasts, though there are signs the researchers weren’t thinking very hard about the questions.
The next piece in this series will focus on Ajeya Cotra’s “Forecasting Transformative AI with Biological Anchors” (which I’ll abbreviate below as “Bio Anchors”), the forecast I find most informative for transformative AI.
What kind of forecast am I going for?
There are a couple of ways in which forecasting transformative AI is different from the kind of forecasting we might be used to.
First, I’m forecasting over very long time horizons (decades), unlike e.g. a weather forecast (days) or an election forecast (months). This makes the task quite a bit harder,3 and harder for outsiders to evaluate since I don’t have a clearly relevant track record of making forecasts on similar topics.
Second, I lack rich, clearly relevant data sources, and I can’t look back through a bunch of similar forecasts from the past. FiveThirtyEight’s election forecasts look at hundreds of polls, and they have a model of how well polls have predicted elections in the past. Forecasting transformative AI needs to rely more on intuition, guesswork and judgment, in terms of determining what data is most relevant and how it’s relevant.
Finally, I’m trying to forecast a qualitatively unfamiliar future. Transformative AI—and the strange future it comes with—doesn’t feel like something we’re “trending toward” year to year.
If I were trying to forecast when the world population would hit 10 billion, I could simply extrapolate existing trends of world population. World population itself is known to be growing and can be directly estimated. In my view, extrapolating out a long-running trend is one of the better ways to make a forecast.
When FiveThirtyEight makes election forecasts, there’s a background understanding that there’s going to be an election on a certain date, and whoever wins will take office on another date. We all buy into that basic framework, and there’s a general understanding that better polling means a better chance of winning.
By contrast, transformative AI—and the strange future it comes with—isn’t something we’re “headed for” in any clearly measurable way. There’s no clear metric like “transformativeness of AI” or “weirdness of the world” that’s going up regularly every year such that we can project it out into the future and get the date that something like PASTA will be developed.
Perhaps for some, these points gives enough reason to ignore the whole possibility of transformative AI, or assume it’s very far away. But I don’t think this is a good idea, for a couple of reasons.
First, I have a background view that something like PASTA is in a sense “inevitable,” assuming continued advances in society and computing. The basic intuition here—which I could expand on if there’s interest—is that human brains are numerous and don’t seem to need particular rare materials to produce, so it should be possible at some point to synthetically replicate the key parts of their functionality.4
At the same time, I’m not confident that PASTA will feel qualitatively as though it’s “on the way” well before it arrives. (More on this below.) So I’m inclined to look for ways to estimate when we can expect this development, despite the challenges, and despite the fact that it doesn’t feel today as though it’s around the corner.
I think there are plenty of example cases where a qualitatively unfamiliar future could be seen in advance by plotting the trend in some underlying, related facts about the world. A few that come to mind:
When COVID-19 first emerged, a lot of people had trouble taking it seriously because it didn’t feel as though we were “trending toward” or “headed for” a world full of overflowing hospitals, office and school closures, etc. At the time (say, January 2020), there were a relatively small number of cases, an even smaller number of deaths, and no qualitative sense of a global emergency. The only thing alarming about COVID-19, at first, was that case counts were growing at a fast exponential rate (though the overall number of cases was still small). But it was possible to extrapolate from the fast growth in case counts to a risk of a global emergency, and some people did. (And some didn’t.)
Climatologists forecast a global rise in temperatures that’s significantly more than what we’ve seen over the past few decades, and could have major consequences far beyond what we’re seeing today. They do this by forecasting trends in greenhouse gas emissions and extrapolating from there to temperature and consequences. If you simply tried to ask “How fast is the temperature rising?” or “Are hurricanes getting worse?”, and based all your forecasts of the future on those, you probably wouldn’t be forecasting the same kinds of extreme events around 2100.5
To give a more long-run example, we can project a date by which the sun will burn out, and conclude that the world will look very different by that date than it does now, even though there’s no trend of things getting colder or darker today.
An analogy for this sort of forecasting would be something like: “This water isn’t bubbling, and there are no signs of bubbling, but the temperature has gone from 70° Fahrenheit6 to 150°, and if it hits 212°, the water will bubble.”
Ideally, I can find some underlying factors that are changing regularly enough for us to predict them (such as growth in the size and cost of AI models), and then argue that if those factors reach a certain point, the odds of transformative AI will be high.
You can think of this approach as answering the question: “If I think something like PASTA is inevitable, and I’m trying to guess the timing of it using a few different analysis methods, what do I guess?” We can separately ask “And is there reason that this guess is implausible, untrustworthy, or too ‘wild?’” — this was addressed in the previous piece in this series.
Subjective extrapolations and “AI impressiveness”
For a different presentation of some similar content, see this section of Bio Anchors.
If we’re looking for some underlying factors in the world that predict when transformative AI is coming, perhaps the first thing we should look for is trends in how “impressive” or “capable” AI systems are.
The easiest version of this would be if the world happened to shake out such that:
One day, for the first time, an AI system managed to get a passing grade on a 4th-grade science exam.
Then we saw the first AI passing (and then acing) a 5th grade exam, then 6th grade exam, etc.
Then we saw the first AI earning a PhD, then the first AI writing a published paper, etc. all the way up to the first AI that could do Nobel-Prize-worthy science work.
This all was spread out regularly over the decades, so we could clearly see the state of the art advancing from 4th grade to 5th grade to 6th grade, all the way up to “postdoc” and beyond. And all of this happened slowly and regularly enough that we could start putting a date on “full-blown scientist AI” several decades in advance.
It would be very convenient—I almost want to say “polite”—of AI systems to advance in this manner. It would also be “polite” if AI advanced in the way that some people seem to casually imagine it will: first taking over jobs like “truck driver” and “assembly line worker,” then jobs like “teacher” and “IT support,” and then jobs like “doctor” and “lawyer,” before progressing to “scientist.”
Either of these would give us plenty of lead time and a solid basis to project when science-automating AI is coming. Unfortunately, I don’t think we can count on such a thing.
AI seems to progress very differently from humans. For example, there were superhuman AI chess players7 long before there was AI that could reliably tell apart pictures of dogs and cats.8
One possibility is that AI systems will be capable of the hardest intellectual tasks insects can do, then of the hardest tasks mice and other small mammals can do, then monkeys, then humans—effectively matching the abilities of larger and larger brains. If this happened, we wouldn’t necessarily see many signs of AI being able to e.g. do science until we were very close. Matching a 4th-grader might not happen until the very end.
Another possibility is that AI systems will be able to do anything that a human can do within 1 second, then anything that a human can do within 10 seconds, etc. This could also be quite a confusing progression that makes it non-obvious how to forecast progress.
Actually, if we didn’t already know how humans tend to mature, we might find a child’s progress to be pretty confusing and hard to extrapolate. Watching someone progress from birth to age 8 wouldn’t necessarily give you any idea that they were, say, 1⁄3 of the way to being able to start a business, make an important original scientific discovery, etc. (Even knowing the usual course of human development, it’s hard to tell from observing an 8-year-old what professional-level capabilities they could/will end up with in adulthood.)
Overall, it’s quite unclear how we should think about the spectrum from “not impressive/capable” to “very impressive/capable” for AI. And indeed, in my experience, different AI researchers have radically different intuitions about which systems are impressive or capable, and how progress is going. I’ve often had the experience of seeing one AI researcher friend point to some new result and say “This is huge, how can anyone not see how close we’re getting to powerful AI?” while another says “This is a minor advance with little significance.”9
It would be great if we could forecast the year transformative AI will be developed, by using a chart like this (from Bio Anchors; “TAI” means “transformative AI”):
But as far as I can tell, there’s no way to define the y-axis that wouldn’t be fiercely debated between experts.
Surveying experts
One way to deal with this uncertainty and confusion would be to survey a large number of experts and simply ask them when they expect transformative AI to be developed. We might hope that each of the experts (or at least, many of them) is doing their own version of the “impressiveness extrapolation” above—or if not, that they’re doing something else that can help them get a reasonable estimate. By averaging many estimates, we might get an aggregate that reflects the “wisdom of crowds.”10
I think the best version of this exercise is Grace et al 2017, a survey of 352 AI researchers that included a question about “when unaided machines can accomplish every task better and more cheaply than human workers” (which would presumably include tasks that advance scientific and technological development, and hence would qualify as PASTA). The two big takeaways from this survey, according to Bio Anchors and me, are:
A ~20% probability of this sort of AI by 2036; a ~50% probability by 2060; a ~70% probability by 2100. These match the figures I give in the introduction.
Much later estimates for slightly differently phrased questions (posed to a smaller subset of respondents), implying (to me) that the researchers simply weren’t thinking very hard about the questions.11
My bottom line: this evidence is consistent with my current probabilities, though potentially not very informative. The next piece in this series will be entirely focused on Ajeya Cotra’s “Forecasting Transformative AI with Biological Anchors,” the forecasting method I find most informative here.
Of course, the answer could be “A kajillion years from now” or “Never.” ↩
Centigrade equivalents for this sentence: 21°, 66°, 100° ↩
Some notes on longer-term forecasting here. ↩
See also this piece for a bit of a more fleshed out argument along these lines, which I don’t agree with fully as stated (I don’t think it presents a strong case for transformative AI soon), but which I think gives a good sense of my intuitions about in-principle feasibility. Also see On the Impossibility of Supersized Machines for some implicit (joking) responses to many common arguments for why transformative AI might be impossible to create. ↩
For example, see the temperature chart here—the lowest line seems like it would be a reasonable projection, if temperature were the only thing you were looking at. ↩
Centigrade equivalents for this sentence: 21°, 66°, 100° ↩
1997. ↩
The Kaggle “dogs vs. cats” challenge was created in 2013. ↩
From Bio Anchors: “We have heard ML experts with relatively short timelines argue that AI systems today can essentially see as well as humans, understand written information, and beat humans at almost all strategy games, and the set of things they can do is expanding rapidly, leading them to expect that transformative AI would be attainable in the next decade or two by training larger models on a broader distribution of ML problems that are more targeted at generating economic value. Conversely, we have heard ML experts with relatively long timelines argue that ML systems require much more data to learn than humans do, are unable to transfer what they learn in one context to a slightly different context, and don’t seem capable of much structured logical and causal reasoning; this leads them to believe we would need to make multiple major breakthroughs to develop TAI. At least one Open Philanthropy technical advisor has advanced each of these perspectives.” ↩
Wikipedia: “The classic wisdom-of-the-crowds finding … At a 1906 country fair in Plymouth, 800 people participated in a contest to estimate the weight of a slaughtered and dressed ox. Statistician Francis Galton observed that the median guess, 1207 pounds, was accurate within 1% of the true weight of 1198 pounds.” ↩
Bio Anchors:
Some researchers were asked to forecast “HLMI” as defined above [human-level machine intelligence, which I would take to include something like PASTA], while a randomly-selected subset was instead asked to forecast “full automation of labor”, the time when “all occupations are fully automatable.” Despite the fact that achieving HLMI seems like it should quickly lead to full automation of labor, the median estimate for full automation of labor was ~2138 while the median estimate for HLMI was ~2061, almost 80 years earlier.
Random subsets of respondents were asked to forecast when individual milestones (e.g. laundry folding, human-level StarCraft, or human-level math research) would be achieved. The median year by which respondents expected machines to be able to automate AI research was ~2104, while the median estimate for HLMI was ~2061 -- another clear inconsistency because “AI research” is a task done by human workers. ↩