If you’re forecasting AI progress or asking someone about their timelines, what event should you focus on?
tl;dr it’s messy and I don’t have answers.
AGI, TAI, etc. are bad. Mostly because they are vague or don’t capture what we care about.
AGI = artificial general intelligence (no canonical operationalization)
This is vague/imprecise, and is used vague/imprecisely
We care about capability-level; we don’t directly care about generality
Maybe we should be paying attention to specific AI capabilities, AI impacts, or conditions for AI catastrophe
TAI = transformative AI, originally defined as “AI that precipitates a transition comparable to (or more significant than) the agricultural or industrial revolution” (see also discussion here)
“a transition comparable to . . . the agricultural or industrial revolution” is vague, and I don’t know what it looks like
“precipitates” is ambiguous. Suppose for illustration that AI in 2025 would take 10 years to cause a transition comparable to the industrial revolution (if there was no more AI progress, or no more AI progress by humans), but AI in 2026 would take 1 year. Then the transition is precipitated by the 2026-AI, but the 2025-AI was capable enough to precipitate a transition. Is the 2025-AI TAI? If so, “TAI” seems to miss what we care about. And regardless, whether a set of AI systems precipitates a transition comparable to the industrial revolution is determined not just by the capabilities and other properties of the systems, but also by other facts about the world, which is weird. Also note that some forecasters believe that current Al would be “eventually transformative” but future Al will be transformative faster, so under some definitions, they believe we already have TAI.
This is often used vaguely/imprecisely
HLAI = human-level AI (no canonical operationalization)
This is kinda vague/imprecise but can be operationalized pretty well, I think
This may come after the stuff we should pay attention to
HLMI = high-level machine intelligence, defined as “when unaided machines can accomplish every task better and more cheaply than human workers”
This will come after the stuff we should pay attention to
PONR = (AI-induced) point of no return, vaguely defined as “the day we AI risk reducers lose the ability to significantly reduce AI risk”
But we may lose that ability gradually rather than in a binary, threshold-y way
And forecasting PONR flows through forecasting narrower events
Some forecasting methods are well-suited for predicting particular kinds of conditions. For example, biological anchors most directly give information about time to humanlike capabilities. And “Could Advanced AI Drive Explosive Economic Growth?” uses economic considerations to give information about economic variables; it couldn’t be adapted well for other kinds of predictions.
Operationalizations of things-like-AGI are ideally
useful or tracking something we care about
If you knew what specific capabilities would be a big deal, you could focus on predicting those capabilities
easy to forecast
maybe simple or concrete
maybe exclusively determined by the properties/capabilities of the AI, rather than also other facts about the world
If you’re eliciting forecasts, like in a survey, make sure respondents interpret what you say correctly. In particular, things you should clarify for timelines surveys (of a forecasting-sophisticated population like longtermist researchers, not like the public) are:
Conditional on no catastrophe before AGI-or-whatever or not
Independent impression or all-things-considered view
(sometimes) when it happens vs when it is feasible
(sometimes) whether to counterfactually condition on progress not slowing due to increased safety concerns, not slowing due to interventions by the AI safety community, or something similar
Forecasting a particular threshold of AI capabilities may be asking the wrong question. To inform at least some interventions, “it may be more useful to know when various ‘pre-TAI’ capability levels would be reached, in what order, or how far apart from each other, rather than to know when TAI will be reached” (quoting Michael Aird). “We should think about the details of different AI capabilities that will emerge over time [] and how those details will affect the actions we can profitably take” (quoting Ashwin Acharya).
This post draws on some research by and discussion with Michael Aird, Daniel Kokotajlo, Ashwin Acharya, and Matthijs Maas.
Operationalizing timelines
If you’re forecasting AI progress or asking someone about their timelines, what event should you focus on?
tl;dr it’s messy and I don’t have answers.
AGI, TAI, etc. are bad. Mostly because they are vague or don’t capture what we care about.
AGI = artificial general intelligence (no canonical operationalization)
This is vague/imprecise, and is used vague/imprecisely
We care about capability-level; we don’t directly care about generality
Maybe we should be paying attention to specific AI capabilities, AI impacts, or conditions for AI catastrophe
TAI = transformative AI, originally defined as “AI that precipitates a transition comparable to (or more significant than) the agricultural or industrial revolution” (see also discussion here)
“a transition comparable to . . . the agricultural or industrial revolution” is vague, and I don’t know what it looks like
“precipitates” is ambiguous. Suppose for illustration that AI in 2025 would take 10 years to cause a transition comparable to the industrial revolution (if there was no more AI progress, or no more AI progress by humans), but AI in 2026 would take 1 year. Then the transition is precipitated by the 2026-AI, but the 2025-AI was capable enough to precipitate a transition. Is the 2025-AI TAI? If so, “TAI” seems to miss what we care about. And regardless, whether a set of AI systems precipitates a transition comparable to the industrial revolution is determined not just by the capabilities and other properties of the systems, but also by other facts about the world, which is weird. Also note that some forecasters believe that current Al would be “eventually transformative” but future Al will be transformative faster, so under some definitions, they believe we already have TAI.
This is often used vaguely/imprecisely
HLAI = human-level AI (no canonical operationalization)
This is kinda vague/imprecise but can be operationalized pretty well, I think
This may come after the stuff we should pay attention to
HLMI = high-level machine intelligence, defined as “when unaided machines can accomplish every task better and more cheaply than human workers”
This will come after the stuff we should pay attention to
PONR = (AI-induced) point of no return, vaguely defined as “the day we AI risk reducers lose the ability to significantly reduce AI risk”
But we may lose that ability gradually rather than in a binary, threshold-y way
And forecasting PONR flows through forecasting narrower events
More: APS-AI; PASTA; prepotent AI; fractional automation of 2020 cognitive tasks; [three levels of transformativeness]; various operationalizations for predictions (e.g., Metaculus, Manifold, Samotsvety); and various definitions of AGI, TAI, and HLAI. Allan Dafoe uses “Advanced AI” to “gesture[] towards systems substantially more capable (and dangerous) than existing (2018) systems, without necessarily invoking specific generality capabilities or otherwise as implied by concepts such as ‘Artificial General Intelligence’ (‘AGI’).” Some people talk about particular visions of AI, such as CAIS, tech company singularity, and perhaps PASTA.
Some forecasting methods are well-suited for predicting particular kinds of conditions. For example, biological anchors most directly give information about time to humanlike capabilities. And “Could Advanced AI Drive Explosive Economic Growth?” uses economic considerations to give information about economic variables; it couldn’t be adapted well for other kinds of predictions.
Operationalizations of things-like-AGI are ideally
useful or tracking something we care about
If you knew what specific capabilities would be a big deal, you could focus on predicting those capabilities
easy to forecast
maybe simple or concrete
maybe exclusively determined by the properties/capabilities of the AI, rather than also other facts about the world
If you’re eliciting forecasts, like in a survey, make sure respondents interpret what you say correctly. In particular, things you should clarify for timelines surveys (of a forecasting-sophisticated population like longtermist researchers, not like the public) are:
Conditional on no catastrophe before AGI-or-whatever or not
Independent impression or all-things-considered view
(sometimes) when it happens vs when it is feasible
(sometimes) whether to counterfactually condition on progress not slowing due to increased safety concerns, not slowing due to interventions by the AI safety community, or something similar
Forecasting a particular threshold of AI capabilities may be asking the wrong question. To inform at least some interventions, “it may be more useful to know when various ‘pre-TAI’ capability levels would be reached, in what order, or how far apart from each other, rather than to know when TAI will be reached” (quoting Michael Aird). “We should think about the details of different AI capabilities that will emerge over time [] and how those details will affect the actions we can profitably take” (quoting Ashwin Acharya).
This post draws on some research by and discussion with Michael Aird, Daniel Kokotajlo, Ashwin Acharya, and Matthijs Maas.