Apologies for not being clear! Iâll try and be a bit more clear here, but thereâs probably a lot of inferential distance here and weâre covering some quite deep topics:
Supposing we do in fact invent AGI someday, do you think this AGI wonât be able to do science? Or that it will be able to do science, but that wouldnât count as âautomating scienceâ?
Or maybe when you said âwhether âPASTAâ is possible at allâ, you meant âwhether âPASTAâ is possible at all via future LLMsâ?
So on the first section, Iâm going for the latter and taking issue with the term âautomationâ, which I think speaks to mindless, automatic process of achieving some output. But if digital functionalism were true, and we successful made a digital emulation of a human who contributed to scientific research, I wouldnât call that âautomating scienceâ, instead we would have created a being that can do science. That being would be creative, agentic, with the ability to formulate itâs own novel ideas and hypotheses about the world. Itâd be limited by its ability to sample from the world, design experiments, practice good epistemology, wait for physical results etc. etc. It might be the case that some scientific research happens quickly, and then subsequent breakthroughs happen more slowly, etc.
My opinions on this are also highly influenced by the works of Deutsch and Popper too, who essentially argue that the growth of knowledge cannot be predicted, and since science is (in some sense) the stock of human knowledge, and since what cannot be predicted cannot be automated, scientific âautomationâ is in some sense impossible.
Maybe youâre assuming that everyone here has a shared assumption that weâre just talking about LLMs...but I bet that if you were to ask a typical senior person in AI x-risk (e.g. Karnofsky) whether itâs possible that there will be some big AI paradigm shift (away from LLMs) between now and TAI, they would say âWell yeah duh of course thatâs possible,â and then they would say that they would still absolutely want to talk about and prepare for TAI, in whatever algorithmic form it might take.
Agreed, AI systems are larger than LLMs, and maybe I was being a bit loose with language. On the whole though, I think much of the case by proponents for the importance of working on AI Safety does assume that current paradigm + scale is all you need, or rest on works that assume it. For instance, Davidsonâs Compute-Centric Framework model for OpenPhil states right in that opening page:
In this framework, AGI is developed by improving and scaling up approaches within the current ML paradigm, not by discovering new algorithmic paradigms.
And I get off the bus with this approach immediately because I donât think thatâs plausible.
As I said in my original comment, Iâm working on a full post on the discussion between Chollet and Dwarkesh, which will hopefully make the AGI-sceptical position Iâm coming from a bit more clear. If you end up reading it, Iâd be really interested in your thoughts! :)
On the whole though, I think much of the case by proponents for the importance of working on AI Safety does assume that current paradigm + scale is all you need, or rest on works that assume it.
There were however plenty of people who were loudly arguing that it was important to work on AI x-risk before âthe current paradigmâ was much of a thing (or in some cases long before âthe current paradigmâ existed at all), and I think their arguments were sound at the time and remain sound today. (E.g. Alan Turing, Norbert Weiner, Yudkowsky, Bostrom, Stuart Russell, TegmarkâŚ) (OpenPhil seems to have started working seriously on AI in 2016, which was 3 years before GPT-2.)
Apologies for not being clear! Iâll try and be a bit more clear here, but thereâs probably a lot of inferential distance here and weâre covering some quite deep topics:
So on the first section, Iâm going for the latter and taking issue with the term âautomationâ, which I think speaks to mindless, automatic process of achieving some output. But if digital functionalism were true, and we successful made a digital emulation of a human who contributed to scientific research, I wouldnât call that âautomating scienceâ, instead we would have created a being that can do science. That being would be creative, agentic, with the ability to formulate itâs own novel ideas and hypotheses about the world. Itâd be limited by its ability to sample from the world, design experiments, practice good epistemology, wait for physical results etc. etc. It might be the case that some scientific research happens quickly, and then subsequent breakthroughs happen more slowly, etc.
My opinions on this are also highly influenced by the works of Deutsch and Popper too, who essentially argue that the growth of knowledge cannot be predicted, and since science is (in some sense) the stock of human knowledge, and since what cannot be predicted cannot be automated, scientific âautomationâ is in some sense impossible.
Agreed, AI systems are larger than LLMs, and maybe I was being a bit loose with language. On the whole though, I think much of the case by proponents for the importance of working on AI Safety does assume that current paradigm + scale is all you need, or rest on works that assume it. For instance, Davidsonâs Compute-Centric Framework model for OpenPhil states right in that opening page:
And I get off the bus with this approach immediately because I donât think thatâs plausible.
As I said in my original comment, Iâm working on a full post on the discussion between Chollet and Dwarkesh, which will hopefully make the AGI-sceptical position Iâm coming from a bit more clear. If you end up reading it, Iâd be really interested in your thoughts! :)
Yeah this is more true than I would like. I try to push back on it where possible, e.g. my post AI doom from an LLM-plateau-ist perspective.
There were however plenty of people who were loudly arguing that it was important to work on AI x-risk before âthe current paradigmâ was much of a thing (or in some cases long before âthe current paradigmâ existed at all), and I think their arguments were sound at the time and remain sound today. (E.g. Alan Turing, Norbert Weiner, Yudkowsky, Bostrom, Stuart Russell, TegmarkâŚ) (OpenPhil seems to have started working seriously on AI in 2016, which was 3 years before GPT-2.)