The ARC Prize website takes this definitional stance on AGI:
Consensus but wrong:
AGI is a system that can automate the majority of economically valuable work.
Correct:
AGI is a system that can efficiently acquire new skills and solve open-ended problems.
Something like the former definition, central to reports like Tom Davidson’s CCF-based takeoff speeds for Open Phil, basically drops out of (the first half of the reasoning behind) the big-picture view summarized in Holden Karnofsky’s most important century series: to quote him, the long-run future would be radically unfamiliar and could come much faster than we think, simply because standard economic growth models imply that any technology that could fully automate innovation would cause an “economic singularity”; one such technology could be what Holden calls PASTA (“Process for Automating Scientific and Technological Advancement”). In What kind of AI? he elaborates (emphasis mine)
I mean PASTA to refer to either a single system or a collection of systems that can collectively do this sort of automation. …
By talking about PASTA, I’m partly trying to get rid of some unnecessary baggage in the debate over “artificial general intelligence.” I don’t think we need artificial general intelligence in order for this century to be the most important in history. Something narrower—as PASTA might be—would be plenty for that. …
I don’t particularly expect all of this to happen as part of a single, deliberate development process. Over time, I expect different AI systems to be used for different and increasingly broad tasks, including and especially tasks that help complement human activities on scientific and technological advancement. There could be many different types of AI systems, each with its own revenue model and feedback loop, and their collective abilities could grow to the point where at some point, some set of them is able to do everything (with respect to scientific and technological advancement) that formerly required a human.
This is why I think it’s basically justified to care about economy-growing automation of innovation as “the right working definition” from the x-risk reduction perspective for a funder like Open Phil in particular, which isn’t what an AI researcher like Francois Chollet cares about. Which is fine, different folks care about different things. But calling the first definition “wrong” feels like the sort of mistake you make when you haven’t at least good-faith effort tried to do what Scott suggested here with the first definition:
… if you’re looking into something controversial, you might have to just read the biased sources on both sides, then try to reconcile them.
Success often feels like realizing that a topic you thought would have one clear answer actually has a million different answers depending on how you ask the question. You start with something like “did the economy do better or worse this year?”, you find that it’s actually a thousand different questions like “did unemployment get better or worse this year?” vs. “did the stock market get better or worse this year?” and end up with things even more complicated like “did employment as measured in percentage of job-seekers finding a job within six months get better” vs. “did employment as measured in total percent of workforce working get better?”. Then finally once you’ve disentangled all that and realized that the people saying “employment is getting better” or “employment is getting worse” are using statistics about subtly different things and talking past each other, you use all of the specific things you’ve discovered to reconstruct a picture of whether, in the ways important to you, the economy really is getting better or worse.
Note also that PASTA is a lot looser definitionally than the AGI defined in Metaculus’ When will the first general AI system be devised, tested, and publicly announced? (2031 as of time of writing), which requires the sort of properties Chollet would probably approve (single unified software system, not a cobbled-together set of task-specialized subsystems), yet if the PASTA collective functionally completes the innovation → resources → PASTA → innovation → … economic growth loop, that would already be x-risk relevant. The argument would then need to be “something like the Chollet’s / Metaculus’ definition is necessary to complete the growth loop”, which would be a testable hypothesis.
This a really interesting way of looking at the issue!
But is PASTA really equivalent to “a system that can automate the majority of economically valuable work”? If it specifically is supposed to mean the automation of innovation, then that sounds closer to Chollet’s definition of AGI to me: “a system that can efficiently acquire new skills and solve open-ended problems”
The ARC Prize website takes this definitional stance on AGI:
Something like the former definition, central to reports like Tom Davidson’s CCF-based takeoff speeds for Open Phil, basically drops out of (the first half of the reasoning behind) the big-picture view summarized in Holden Karnofsky’s most important century series: to quote him, the long-run future would be radically unfamiliar and could come much faster than we think, simply because standard economic growth models imply that any technology that could fully automate innovation would cause an “economic singularity”; one such technology could be what Holden calls PASTA (“Process for Automating Scientific and Technological Advancement”). In What kind of AI? he elaborates (emphasis mine)
This is why I think it’s basically justified to care about economy-growing automation of innovation as “the right working definition” from the x-risk reduction perspective for a funder like Open Phil in particular, which isn’t what an AI researcher like Francois Chollet cares about. Which is fine, different folks care about different things. But calling the first definition “wrong” feels like the sort of mistake you make when you haven’t at least good-faith effort tried to do what Scott suggested here with the first definition:
Note also that PASTA is a lot looser definitionally than the AGI defined in Metaculus’ When will the first general AI system be devised, tested, and publicly announced? (2031 as of time of writing), which requires the sort of properties Chollet would probably approve (single unified software system, not a cobbled-together set of task-specialized subsystems), yet if the PASTA collective functionally completes the innovation → resources → PASTA → innovation → … economic growth loop, that would already be x-risk relevant. The argument would then need to be “something like the Chollet’s / Metaculus’ definition is necessary to complete the growth loop”, which would be a testable hypothesis.
This a really interesting way of looking at the issue!
But is PASTA really equivalent to “a system that can automate the majority of economically valuable work”? If it specifically is supposed to mean the automation of innovation, then that sounds closer to Chollet’s definition of AGI to me: “a system that can efficiently acquire new skills and solve open-ended problems”