I think of AGI (and human-level intelligence) as the cloud, and superintelligence as being above the cloud. They are useful concepts, despite their vagueness. But they’re markedly less useful when you get close to them. [...]
For my purposes, I think the key threshold is when the system is capable enough to cause dramatic, civilisational changes. For example, the point where AI could take over from humanity if misaligned, or has made 50% of people permanently unemployable, or has doubled the global rate of technological progress. I focus on this threshold because I think it matters most for planning our strategies and careers.
I think the example milestones you mention differ significantly from one another, and each carries substantial vagueness that compounds rather than resolves the vagueness issues you raised earlier in the essay.
For example, I don’t know how to operationalize the point where “AI could take over from humanity”, and I suspect people will disagree for years about whether that threshold has been reached, much as they have debated for years whether we have already achieved AGI. Similarly, it is unclear what it means for 50% of people to be “permanently unemployable” as opposed to merely unemployed.
If your goal is to ground the debate about timelines in something measurable and uncontroversial, it is worth thinking more carefully about milestones that actually serve that purpose. Otherwise, time will pass and you will likely find that these milestones will become markedly less useful as we get close to them.
I agree that these are vague and could come apart from each other. But I don’t see any crisp, verifiable definitions that I could replace them with and serve the same purpose. I’m interested in forecasting transformative AI for the main purpose of forecasting when one has to have one’s AI-related impact by. e.g. by when do we need to have solved alignment (or to have paused AI development)?
If I instead used a verifiable definition here, such as the “In what year would AI systems be able to replace 99% of current fully remote jobs?” that I cite in the essay, then you have to do further forecasting of how that time relates to the key things that matter (such as the deadline on AI alignment). Also, for crisp concrete definitions, one tends to then get hung up on estimating exactly how hard the final 1% of current fully remote jobs are, because that is central to the prediction. For example, are there 1% of current fully remote jobs that we only let a human do, e.g. for reasons of legal responsibility or personal relationships? Maybe? But that isn’t relevant to the central features we care about.
I’m sure my definition could be improved (the focus of my essay isn’t on my prediction but on the wider points about everyone’s timelines), but I hope this explains why being “measurable and uncontroversial” need not make for the best thing to forecast.
I think the example milestones you mention differ significantly from one another, and each carries substantial vagueness that compounds rather than resolves the vagueness issues you raised earlier in the essay.
For example, I don’t know how to operationalize the point where “AI could take over from humanity”, and I suspect people will disagree for years about whether that threshold has been reached, much as they have debated for years whether we have already achieved AGI. Similarly, it is unclear what it means for 50% of people to be “permanently unemployable” as opposed to merely unemployed.
If your goal is to ground the debate about timelines in something measurable and uncontroversial, it is worth thinking more carefully about milestones that actually serve that purpose. Otherwise, time will pass and you will likely find that these milestones will become markedly less useful as we get close to them.
I agree that these are vague and could come apart from each other. But I don’t see any crisp, verifiable definitions that I could replace them with and serve the same purpose. I’m interested in forecasting transformative AI for the main purpose of forecasting when one has to have one’s AI-related impact by. e.g. by when do we need to have solved alignment (or to have paused AI development)?
If I instead used a verifiable definition here, such as the “In what year would AI systems be able to replace 99% of current fully remote jobs?” that I cite in the essay, then you have to do further forecasting of how that time relates to the key things that matter (such as the deadline on AI alignment). Also, for crisp concrete definitions, one tends to then get hung up on estimating exactly how hard the final 1% of current fully remote jobs are, because that is central to the prediction. For example, are there 1% of current fully remote jobs that we only let a human do, e.g. for reasons of legal responsibility or personal relationships? Maybe? But that isn’t relevant to the central features we care about.
I’m sure my definition could be improved (the focus of my essay isn’t on my prediction but on the wider points about everyone’s timelines), but I hope this explains why being “measurable and uncontroversial” need not make for the best thing to forecast.