If the argument were merely that thereâs something like a 1 in 10 million chance of a global catastrophic event caused by AGI over the next 100 years and we should devote a small amount of resources to this problem, then you could accept flimsy, hand-wavy arguments. But Eliezer Yudkowsky forecasts a 99.5% chance of human extinction from AGI âwell before 2050â, unless we implement his aggressive global moratorium on AI R&D. The flimsy support that can justify a small allocation of resources canât justify a global moratorium on AI R&D, enforced by militaries. (Yudkowsky says that AI datacentres that violate the moratorium should be blown up.)
Yudkowsky is on the extreme end, but not by much. Some people want to go to extreme lengths to stop AI. Pause AI calls for a moratorium similar to what Yudkowsky recommends. The amount of funding and attention to AI existential risk from EA is not a small percentage but a very large one. So, whatever would support a modest, highly precautionary stance toward AI risk does not support what is actually happening.
Iâll take your concern about AI concentrating wealth and power if there is widespread labour automation as an example of what I mean with regard to flimsy evidentiary/âargumentative support. Okay, letâs imagine that in, say, 2050, we have humanoid robots that are capable of automating most paid human work that currently exists, both knowledge work and work that has a physical dimension. Letâs suppose the robots are:
Built with commodity hardware thatâs roughly as expensive as a Vespa or a used car
Sold directly to consumers
Running free, open source software and free, open source/âopen weights AI models
Programmed to follow the orders of their owner, locked with a password and/âor biometric security
Would your concern about wealth and power concentration apply in such a scenario? Itâs hard to see how it would. In this scenario, humanoid robots with advanced AI would be akin to personal computers or smartphones. Powerful but so affordable and widely distributed that the rich and powerful hardly have any technological edge over the poor and powerless. (A billionaire uses an iPhone, the President of the United States uses an iPhone, and the cashier at the grocery store uses an iPhone.)
You could also construct a scenario where humanoid robots are extremely expensive, jealously kept by the companies that manufacture them and not sold, run proprietary software and closed models, and obey only the manufacturerâs directives. In that case, power/âwealth concentration would be a concern.
So, which scenario is more likely to be true? What is more likely to be the nature of these advanced humanoid robots in 2050?
We have no idea. There is simply no way for us to know, and as much as we might want to know, be desperate to know, twist ourselves in knots trying to work it out, we wonât get any closer to the truth than when we started. The uncertainty is irreducible.
Okay, so letâs accept we donât know. Shouldnât we prepare for the latter scenario, just in case? Maybe. How?
Coming up with plausible interventions or preparations at this early stage is hopeless. We donât know which robot parts need to be made cheaper. The companies that will make the robots probably donât exist yet. Promoting open source software or open AI models in general today wonât stop any company in the future from using proprietary software and closed models. Even if we passed a law now mandating all AI and robotics companies had to use open source software and open models â would we really want to do that, based on a hunch? â that law could easily be repealed in the future.
Plus, I made the possibility space artificially small. I made things really simple by presenting a binary choice two scenarios. In reality, there is a combinatorial explosion of possible permutations of different technical, design, and business factors involved. Most likely including ones we canât imagine now and that, if we were shown a Wikipedia article from the future describing one, we still wouldnât understand it. So, there is a vast space of possibilities based on what we can already imagine, and there will probably be even more possibilities based on new technology and new science that we canât yet grasp.
Saying âprepare nowâ sounds sensible and precautionary, but itâs not actionable.
Also, fundamental question: why is preparing earlier better? Letâs say in 2025 humanoid robots account for 0% of GDP (this seems true), in 2030 theyâll account for 1%, in 2040 for 25%, and in 2050 for 50%. What do we gain by trying to prepare while humanoid robots are at 0% of GDP? Once theyâre at 1% of GDP, or even 0.5%, or 0.25%, weâll have a lot more information than we do now. I imagine that 6 months spent studying the problem while the robots are at 1% of GDP will be worth much more than 5 years of research at the 0% level.
Perhaps a good analogy is scientific experiments. The value of doing theory or generating hypotheses in the absence of any experiments or observations â in the absence of any data, in other words â is minimal. For the sake of illustration, letâs imagine youâre curious about how new synthetic drugs analogous to LSD but chemically unlike any existing drugs â not like any known molecules at all â might affect the human mind. Could they make people smarter temporarily? Or perhaps cognitively impaired? Could they make people more altruistic and cooperative? Or perhaps paranoid and distrustful?
You have no data in this scenario: you canât synthesize molecules, you canât run simulations, there are no existing comparisons, natural or synthetic, and you certainly canât test anything on humans or animals. All you can do is think about it.
Would time spent in this pre-empirical state of science (if it can be called science) have any value? Letâs say you were in that state for⌠50 years⌠100 years⌠500 years⌠1,000 years⌠would you learn anything? Would you gain any understanding? Would you get any closer to truth? I think you wouldnât, or you would so marginally that it wouldnât matter.
Then if you suddenly had data, if you could synthesize molecules, run simulations, and test drugs on live subjects, in a very short amount of time you would outstrip, many times over, whatever little knowledge you might have gained from just theorizing and hypothesizing about it. A year of experiments would be worth more than a century of thought. So, if for some reason, you knew you couldnât start experiments for another hundred years, there would be very little value in thinking about the topic before then.
The whole AGI safety/âalignment and AGI preparedness conversation seems to rely on the premise that non-empirical/âpre-empirical science is possible, realistic, and valuable, and that if we, say, spend $10 million of grant money on it, it will have higher expected value than giving it to GiveWellâs top charities, or pandemic preparedness, or asteroid defense, or cancer research, or ice cream cones for kids at county fairs, or whatever else. I donât see how this could be true. I donât see how this can be justified. It seems like you basically might as well light the money on fire.
Empirical safety/âalignment research on LLMs might have value if LLMs scale to AGI, but thatâs a pretty big âifâ. For over 15 years, up until â Iâm not sure, maybe around 2016? â Yudkowsky and MIRI still thought symbolic AI would lead to AGI in the not-too-distant future. In retrospect, that looks extremely silly. (Actually, I thought it looked extremely silly at the time, and said so, and also got pushback from people in EA way back then too. Plus ça change! Maybe in 2035 weâll be back here again.) The idea that symbolic AI could ever lead to AGI, even in 1,000 years, just looks unbelievably quaint where you compare symbolic AI systems to a system like AlphaGo, AlphaStar, or ChatGPT. Deep learning/âdeep RL-based systems still have quite rudimentary capabilities compared to the average human being, or, in some important ways, even compared to, say, a cat, and when you compare how much simpler and how much less capable symbolic AI systems are to these deep neural network-based systems, itâs ridiculous. Symbolic AI is not too different from conventional software, and the claim that symbolic AI would someday soon ascend to AGI feels not too different from the claim that, in the not-too-distant future, Microsoft Windows will learn how to think. The connection between symbolic AI and human general intelligence seems to boil down to, essentially, a loose metaphorical comparison between software/âcomputers and human brains.
I donât think the conflation of LLMs with human general intelligence is quite as ridiculous as it was with symbolic AI, but it is still quite ridiculous. Particularly when people make absurd and plainly false claims that GPT-4 is AGI (as Leopold Aschenbrenner did) or o3 is AGI (as Tyler Cowen did), or that GPT-4 is a âvery weak AGIâ (as Will MacAskill did). This seems akin to saying a hot air balloon is a spaceship, or a dog is a bicycle. Itâs hard to even know what to say.
As for explicitly, substantively making the argument about why LLMs wonât scale to AGI, there are two distinct and independent arguments. The first argument involves pointing out the limits to LLM scaling. The second argument involves pointing out the fundamental research problems that scaling canât solve.
I used to assume that people who care a lot about AGI alignment/âsafety as an urgent priority must have thoughtful replies to these sorts of arguments. Increasingly, I get the impression that most of those people have simply never thought about them before, and werenât even aware such arguments existed.
If the argument were merely that thereâs something like a 1 in 10 million chance of a global catastrophic event caused by AGI over the next 100 years and we should devote a small amount of resources to this problem, then you could accept flimsy, hand-wavy arguments. But Eliezer Yudkowsky forecasts a 99.5% chance of human extinction from AGI âwell before 2050â, unless we implement his aggressive global moratorium on AI R&D. The flimsy support that can justify a small allocation of resources canât justify a global moratorium on AI R&D, enforced by militaries. (Yudkowsky says that AI datacentres that violate the moratorium should be blown up.)
Yudkowsky is on the extreme end, but not by much. Some people want to go to extreme lengths to stop AI. Pause AI calls for a moratorium similar to what Yudkowsky recommends. The amount of funding and attention to AI existential risk from EA is not a small percentage but a very large one. So, whatever would support a modest, highly precautionary stance toward AI risk does not support what is actually happening.
Iâll take your concern about AI concentrating wealth and power if there is widespread labour automation as an example of what I mean with regard to flimsy evidentiary/âargumentative support. Okay, letâs imagine that in, say, 2050, we have humanoid robots that are capable of automating most paid human work that currently exists, both knowledge work and work that has a physical dimension. Letâs suppose the robots are:
Built with commodity hardware thatâs roughly as expensive as a Vespa or a used car
Sold directly to consumers
Running free, open source software and free, open source/âopen weights AI models
Programmed to follow the orders of their owner, locked with a password and/âor biometric security
Would your concern about wealth and power concentration apply in such a scenario? Itâs hard to see how it would. In this scenario, humanoid robots with advanced AI would be akin to personal computers or smartphones. Powerful but so affordable and widely distributed that the rich and powerful hardly have any technological edge over the poor and powerless. (A billionaire uses an iPhone, the President of the United States uses an iPhone, and the cashier at the grocery store uses an iPhone.)
You could also construct a scenario where humanoid robots are extremely expensive, jealously kept by the companies that manufacture them and not sold, run proprietary software and closed models, and obey only the manufacturerâs directives. In that case, power/âwealth concentration would be a concern.
So, which scenario is more likely to be true? What is more likely to be the nature of these advanced humanoid robots in 2050?
We have no idea. There is simply no way for us to know, and as much as we might want to know, be desperate to know, twist ourselves in knots trying to work it out, we wonât get any closer to the truth than when we started. The uncertainty is irreducible.
Okay, so letâs accept we donât know. Shouldnât we prepare for the latter scenario, just in case? Maybe. How?
Coming up with plausible interventions or preparations at this early stage is hopeless. We donât know which robot parts need to be made cheaper. The companies that will make the robots probably donât exist yet. Promoting open source software or open AI models in general today wonât stop any company in the future from using proprietary software and closed models. Even if we passed a law now mandating all AI and robotics companies had to use open source software and open models â would we really want to do that, based on a hunch? â that law could easily be repealed in the future.
Plus, I made the possibility space artificially small. I made things really simple by presenting a binary choice two scenarios. In reality, there is a combinatorial explosion of possible permutations of different technical, design, and business factors involved. Most likely including ones we canât imagine now and that, if we were shown a Wikipedia article from the future describing one, we still wouldnât understand it. So, there is a vast space of possibilities based on what we can already imagine, and there will probably be even more possibilities based on new technology and new science that we canât yet grasp.
Saying âprepare nowâ sounds sensible and precautionary, but itâs not actionable.
Also, fundamental question: why is preparing earlier better? Letâs say in 2025 humanoid robots account for 0% of GDP (this seems true), in 2030 theyâll account for 1%, in 2040 for 25%, and in 2050 for 50%. What do we gain by trying to prepare while humanoid robots are at 0% of GDP? Once theyâre at 1% of GDP, or even 0.5%, or 0.25%, weâll have a lot more information than we do now. I imagine that 6 months spent studying the problem while the robots are at 1% of GDP will be worth much more than 5 years of research at the 0% level.
Perhaps a good analogy is scientific experiments. The value of doing theory or generating hypotheses in the absence of any experiments or observations â in the absence of any data, in other words â is minimal. For the sake of illustration, letâs imagine youâre curious about how new synthetic drugs analogous to LSD but chemically unlike any existing drugs â not like any known molecules at all â might affect the human mind. Could they make people smarter temporarily? Or perhaps cognitively impaired? Could they make people more altruistic and cooperative? Or perhaps paranoid and distrustful?
You have no data in this scenario: you canât synthesize molecules, you canât run simulations, there are no existing comparisons, natural or synthetic, and you certainly canât test anything on humans or animals. All you can do is think about it.
Would time spent in this pre-empirical state of science (if it can be called science) have any value? Letâs say you were in that state for⌠50 years⌠100 years⌠500 years⌠1,000 years⌠would you learn anything? Would you gain any understanding? Would you get any closer to truth? I think you wouldnât, or you would so marginally that it wouldnât matter.
Then if you suddenly had data, if you could synthesize molecules, run simulations, and test drugs on live subjects, in a very short amount of time you would outstrip, many times over, whatever little knowledge you might have gained from just theorizing and hypothesizing about it. A year of experiments would be worth more than a century of thought. So, if for some reason, you knew you couldnât start experiments for another hundred years, there would be very little value in thinking about the topic before then.
The whole AGI safety/âalignment and AGI preparedness conversation seems to rely on the premise that non-empirical/âpre-empirical science is possible, realistic, and valuable, and that if we, say, spend $10 million of grant money on it, it will have higher expected value than giving it to GiveWellâs top charities, or pandemic preparedness, or asteroid defense, or cancer research, or ice cream cones for kids at county fairs, or whatever else. I donât see how this could be true. I donât see how this can be justified. It seems like you basically might as well light the money on fire.
Empirical safety/âalignment research on LLMs might have value if LLMs scale to AGI, but thatâs a pretty big âifâ. For over 15 years, up until â Iâm not sure, maybe around 2016? â Yudkowsky and MIRI still thought symbolic AI would lead to AGI in the not-too-distant future. In retrospect, that looks extremely silly. (Actually, I thought it looked extremely silly at the time, and said so, and also got pushback from people in EA way back then too. Plus ça change! Maybe in 2035 weâll be back here again.) The idea that symbolic AI could ever lead to AGI, even in 1,000 years, just looks unbelievably quaint where you compare symbolic AI systems to a system like AlphaGo, AlphaStar, or ChatGPT. Deep learning/âdeep RL-based systems still have quite rudimentary capabilities compared to the average human being, or, in some important ways, even compared to, say, a cat, and when you compare how much simpler and how much less capable symbolic AI systems are to these deep neural network-based systems, itâs ridiculous. Symbolic AI is not too different from conventional software, and the claim that symbolic AI would someday soon ascend to AGI feels not too different from the claim that, in the not-too-distant future, Microsoft Windows will learn how to think. The connection between symbolic AI and human general intelligence seems to boil down to, essentially, a loose metaphorical comparison between software/âcomputers and human brains.
I donât think the conflation of LLMs with human general intelligence is quite as ridiculous as it was with symbolic AI, but it is still quite ridiculous. Particularly when people make absurd and plainly false claims that GPT-4 is AGI (as Leopold Aschenbrenner did) or o3 is AGI (as Tyler Cowen did), or that GPT-4 is a âvery weak AGIâ (as Will MacAskill did). This seems akin to saying a hot air balloon is a spaceship, or a dog is a bicycle. Itâs hard to even know what to say.
As for explicitly, substantively making the argument about why LLMs wonât scale to AGI, there are two distinct and independent arguments. The first argument involves pointing out the limits to LLM scaling. The second argument involves pointing out the fundamental research problems that scaling canât solve.
I used to assume that people who care a lot about AGI alignment/âsafety as an urgent priority must have thoughtful replies to these sorts of arguments. Increasingly, I get the impression that most of those people have simply never thought about them before, and werenât even aware such arguments existed.