Early last month, Leopold Aschenbrenner released a long essay and podcast outlining his projections for the future of AI. Both of these sources are full of interesting arguments and evidence, for a comprehensive summary see Zvi’s post here. Rather than going point by point I will instead accept the major premises of Leopold’s essay but contest some of his conclusions.
So what are the major premises of his piece?
There will be several orders of magnitude increase in investment into AI. 100x more spending, 100x more compute, 100x more efficient algorithms, and an order of magnitude or two gains from some form of “learning by doing” or “unhobbling” on top.
This investment scale up will be sufficient to achieve AGI. This means the models on the other side of the predicted compute scale up will be able to automate all cognitive jobs with vast scale and speed.
These capabilities will be essential to international military competition.
All of these premises are believable to me and well-argued for in Leopold’s piece.
Leopold contends that these premises imply that the national security state will take over AI research and the major data centers, locking down national secrets in a race against China, akin to the Manhattan project.
Ultimately, my main claim here is descriptive: whether we like it or not, superintelligence won’t look like an SF startup, and in some way will be primarily in the domain of national security.
By late 26/27/28 … the core AGI research team (a few hundred researchers) will move to a secure location; the trillion-dollar cluster will be built in record-speed; The Project will be on.
The main problem is that Leopold’s premises can be applied to conclude that other technologies will also inevitably lead to a Manhattan project, but these projects never arrived. Consider electricity. It’s an incredibly powerful technology with rapid scale up, sufficient to empower those who have it far beyond those who don’t and it is essential to military competition. Every tank and missile and all the tech to manufacture them relies on electricity. But there was never a Manhattan project for this technology. It’s initial invention and spread was private and decentralized. The current sources of production and use are mostly private.
This is true of most other technologies with military uses: explosives, steel, computing, the internet, etc. All of these technologies are essential in the government’s monopoly on violence and it’s ability to exert power over other nations and prevent coups from internal actors. But the government remains a mere customer of these technologies and often not even the largest one.
Why is this? Large scale nationalization is costly and unnecessary for maintaining national secrets and technological superiority. Electricity and jet engines are essential for B-2 bombers, but if you don’t have the particular engineers and blueprints, you can’t build it. So, the government doesn’t need to worry about locking down the secrets of electricity production and sending all of the engineers to Los Alamos. They can keep the first several steps of the production process completely open and mix the outputs with a final few steps that are easier to keep secret.
To be clear, I am confident that governments and militaries will be extremely interested in AI. They will be important customers for many AI firms, they will create internal AI tools, and AI will become an important input into every major military. But this does not mean that most or all of the AI supply chain, from semi-conductors to data-centers to AI research, must be controlled by governments.
Nuclear weapons are outliers among weapons technology in terms of the proportion of the supply chain and final demand directly overseen by governments. Most military technologies rely on an open industrial base mixed with some secret knowledge in the final few production steps.
So should we expect AGI to be more like nuclear weapons or like a new form of industrial capacity? This depends on how much extra scaffolding you need on top of the base model computation that’s piped out of data centers to achieve militarily relevant goals.
Leopold’s unhobbling story supports a view where the intelligence produced by massive datacenters is more like the raw input of electricity, which needs to be combined with other materials and processes to make a weapon, than a nuclear bomb which is a weapon and only a weapon right out the box.
Leopold on base models says:
“out of the box, they’re hobbled: they’re using their incredible internal representations merely to predict the next token in random internet text, and rather than applying them in the best way to actually try to solve your problem.”
“On SWE-Bench (a benchmark of solving real-world software engineering tasks), GPT4 can only solve ~2% correctly, while with Devin’s agent scaffolding it jumps to 14-23%. (Unlocking agency is only in its infancy though)”
Devin can have a product without a proprietary model because they have a scaffold. They can safely contract for and pipe in the raw resource latent model and put it through a production process to get something uniquely tooled out the other end, without needing to in-house and lock-down the base model to maintain a unique product.
Of current chatbots Leopold says:
“They’re mostly not personalized to you or your application (just a generic chatbot with a short prompt, rather than having all the relevant background on your company and your work)”
Context is that which is scarce! The national security state needn’t lock down the base models if the models are hobbled without the context of their secret applications. That context is already something they’re already extremely skilled at locking down, and it doesn’t require enlisting an entire industry.
“In a few years, it will be clear that the AGI secrets are the United States’ most important national defense secrets—deserving treatment on par with B-21 bomber or Columbia-class submarine blueprints”
Leopold is predicting the nationalization of an entire industrial base based on an analogy to submarines and bombers, but a large fraction of the supply chain for these vehicles are private and open. It’s not clear why he thinks military applications of AGI can’t be similarly protected without control over the majority of the supply chain and final demand.
If you imagine AGI as this single, powerful oracle and army that can complete any and all tasks on command, then Leopold is right: governments will fight hard to lock everyone else out. If instead, AGI is a sort of “intelligence on tap” which is an input to thousands of different production processes where it’s mixed with different infrastructure, context, and tools to create lots of different products, then governments don’t need to control the entire industrial base producing this intelligence to keep their secrets. Leopold leans hard on the Manhattan project as a close analogy to the first situation, but most military technologies are in the second camp.
Executive summary: The author argues against the inevitability of an AI Manhattan Project, contending that governments can maintain technological superiority without nationalizing the entire AI industry.
Key points:
The author accepts premises about massive AI investment and capabilities but contests the conclusion of government takeover.
Historical precedent shows most technologies with military applications remain largely private (e.g., electricity, computing).
Governments can maintain secrecy by controlling only the final steps of production, not entire supply chains.
AI may be more like a general industrial input (like electricity) than a standalone weapon (like nuclear bombs).
The “unhobbling” of AI models suggests raw AI capabilities may need significant context and scaffolding for specific applications.
Military AI applications could potentially be protected without controlling the entire AI industry and infrastructure.
This comment was auto-generated by the EA Forum Team. Feel free to point out issues with this summary by replying to the comment, and contact us if you have feedback.
My understanding of your main claim: If AGI is not a magic problem-solving oracle and is instead limited by needing to be unhobbled and integrated with complex infrastructure, it will be relatively safe for model weights to be available to foreign adversaries. Or at least key national security decision makers will believe that’s the case.
Please correct me if I’m wrong. My thoughts on the above:
Where is this relative safety coming from? Is it from expecting that adversaries aren’t going to be able to figure out how to do unhobbling or steal the necessary secrets to do unhobbling? Is it from expecting the unhobbling and building infrastrucure around AIs to be a really hard endeavor?
The way I’m viewing this picture, AI that can integrate all across the economy, even if that takes substantial effort, is a major threat to global stability and US dominance.
I guess you can think about the AI-for-productive-purposes supply chain as having two components: Develop the powerful AI model (Initial development), and unhobble it / integrate it in workflows / etc. (Unhobbling/Integration). And you’re arguing that the second of these will be an acceptable place to focus restrictions. My intuition says we will want restrictions on both, but more on the part that is most expensive or excludable (e.g., AI chips being concentrated is a point for initial development). It’s not clear to me what the cost of both supply chain steps is: Currently, it looks like pre-training costs are higher than fine-tuning costs (point for initial development); but actually integrating AIs across the economy seems very expensive to do, the economy is really big (point for unhobbling/integration) (this depends a lot on the systems at the time and how easy they are to work with).