#193 – The risk that US–China AI competition leads to war (Sihao Huang on the 80,000 Hours Podcast)

We just published an interview: Sihao Huang on the risk that US–China AI competition leads to war. Listen on Spotify or click through for other audio options, the transcript, and related links. Below are the episode summary and some key excerpts.

Episode summary

You don’t necessarily need world-leading compute to create highly risky AI systems. The biggest biological design tools right now, like AlphaFold’s, are orders of magnitude smaller in terms of compute requirements than the frontier large language models. And China has the compute to train these systems.
And if you’re, for instance, building a cyber agent or something that conducts cyberattacks, perhaps you also don’t need the general reasoning or mathematical ability of a large language model. You train on a much smaller subset of data. You fine-tune it on a smaller subset of data. And those systems — one, if China intentionally misuses them, and two, if they get proliferated because China just releases them as open source, or China does not have as comprehensive AI regulations — this could cause a lot of harm in the world.
- Sihao Huang

In today’s episode, host Luisa Rodriguez speaks to Sihao Huang — a technology and security policy fellow at RAND — about his work on AI governance and tech policy in China, what’s happening on the ground in China in AI development and regulation, and the importance of US–China cooperation on AI governance.

They cover:

Whether the US and China are in an AI race, and the global implications if they are.
The state of the art of AI in China.
China’s response to American export controls, and whether China is on track to indigenise its semiconductor supply chain.
How China’s current AI regulations try to maintain a delicate balance between fostering innovation and keeping strict information control over the Chinese people.
Whether China’s extensive AI regulations signal real commitment to safety or just censorship — and how AI is already used in China for surveillance and authoritarian control.
How advancements in AI could reshape global power dynamics, and Sihao’s vision of international cooperation to manage this responsibly.
And plenty more.

Producer and editor: Keiran Harris
Audio engineering team: Ben Cordell, Simon Monsour, Milo McGuire, and Dominic Armstrong
Additional content editing: Katy Moore and Luisa Rodriguez
Transcriptions: Katy Moore

Highlights

How advanced is Chinese AI?

Sihao Huang: So Chinese AI is not just large language models, and I think this is an important point that we should get into in a little bit. But on the LLM front, if you use a Chinese language model, it feels very much like a Western one: you go on a portal that looks like ChatGPT, you can type in your text prompt, you can upload images, speak to it, it does math, renders LaTeX, et cetera. So I think, broadly speaking, they’re building models that are very similar to the ones that are being released in the US or in Europe.
But in terms of the base model, they’re still a little bit behind. And there’s really interesting dynamics in the Chinese AI market right now, especially over the past few months. So ever since mid last year, when Baidu released its Ernie Bot, it claimed that its models were on par with GPT-4, at least on certain benchmarks. But that didn’t turn out to be the case when people started using them. And you should always be a little sceptical when you look at headline benchmark numbers, because companies can optimise toward certain benchmarks, and they can also cherry-pick benchmarks to make their models look more impressive.
But over the past few months, there’s actually been a number of Chinese models that appear to be getting close to GPT-4 level. So there’s Alibaba’s Qwen 2.5, there is Zhipu’s GLM, Moonshot’s Kimi, and DeepSeek-V2 — which actually came out not too long ago, and was quite impressive in terms of engineering. And some of these are open source models. So China is really catching up in terms of these frontier LLMs that, at least in the open source, are as good as the ones in the West. But in the closed source, I think they still haven’t reached parity with GPT-4o or Claude 3.5, et cetera.
The second interesting part about this dynamic is that there’s a lot of companies building models that look like this. So I would say that there are maybe four or five contenders for GPT-4-level systems within China right now. And that’s quite counterintuitive, because you may expect China, as a communist state, to have a lot of centralised, coordinated action on AI development — but instead you’re seeing all these computational resources being spread among different companies, and each using their limited computational resources to train frontier models instead of batching them together into one big run.
They are also racing each other in price. So quite recently, there’s a lot of talk about this LLM price war within China. So you have all these big tech companies and VC-funded companies flush in capital, racing each other and cutting LLM prices by 90%, 95%, 99% to compete against each other on free LLM access and API access. So as of mid-2024, a lot of Chinese systems are catching up in the open source, and they’re catching up in terms of a lot of fast-followed technical innovations in making LLM inference more effective, in making LLM prices cheaper.
I think a lot of this really is because of this overhang of compute capabilities that was brought on by China stockpiling a lot of American chips, like the A800 and H800, over the past two years. And so they’ve been able to catch up to essentially the GPT-4 level of capabilities. I think there is an open question to which they’re able to keep following at this pace in the future. But at least right now, if you say GPT-4 was sort of trained towards the middle or end of 2022, then you would say that China is about 1.5 years or two years behind.
Now, if we talk about other types of AI technologies, though, I think there are definitely parts of AI where China is on the frontier, and that’s things like computer vision. China has been very strong in computer vision for a long period of time. There’s companies like SenseTime, Hikvision, and Megvii, for instance, that build really advanced surveillance software. SenseTime has products that are able to track individuals throughout a city continuously through different cameras, even without looking at your face, by building a model of your clothing and wardrobe, or looking at your gait.
And these are really advanced surveillance systems that are being sold to Chinese cities and to police departments. Part of what I think makes Western policymakers so concerned about China’s use of this is surveillance and human rights abuses. But if you also look at the history of Chinese research, actually — I believe this is still the case — the single most-cited paper in AI, which introduced residual networks, was written by a set of four Chinese authors in computer vision that were trained in Chinese universities. And three out of the four of them are still in China. I believe one of them eventually went to Meta and is now a professor at MIT.

Is China catching up to the US and UK?

Sihao Huang: I would say that they are not closing in on the US and UK very fast. I would not want to be in China’s position if I were to think that AI is extremely relevant to the future of the world. And it is, of course, very difficult to predict what things are going to look like in the long term. I would always caveat this by saying that the scaling paradigm — which is what we are grounding a lot of our policies on right now — of AI systems becoming broadly more capable as we throw more compute at it, and therefore we design expert controls on chips that are used to build these AI models, that may not last forever. I think it’s likely that there’s going to be other innovations that are critical to building more powerful AI systems.
So in scope of what we can see with the current technological paradigm, China does not look like it’s in a very good position. But I think there is always the looming threat that, as we talked about before, China is on a different exploration/exploitation tradeoff as we are. They may be more creative in trying to explore other paths to building advanced AI systems, and they also may have, for instance, comparative advantages in building AI that is more integrated into this economy.
You’ve seen China ramp up deployment of AI or visual recognition systems that have added a lot of conveniences, quite frankly, in daily lives, on top of these things being deployed for surveillance. When I was in Beijing last year, you can just go into a supermarket without a credit card, but also without a phone. You basically just go to the counter, scan your food items and you just look at a camera, and it pays automatically.
So there’s a lot of ways in which Chinese companies are quite creative in deploying these AI systems. If I were to paint a story of how potentially the US can fall behind on AI development, it may look something like: GPT-5 is just not that impressive; we hit a wall with the scaling paradigm and transformers or large language models, and there is something that resembles an AI winter, or at least a significant decrease in AI investment in the United States.
But in China, they’re able to actually integrate these systems into the economy much more deeply and build profitable products around them. What that means is that it sustains investment into AI that also looks into more diverse methods to build more capable models that eventually pays off. And once they find this — and perhaps they don’t need that much compute, or they find alternative ways to access compute — that could be a path for China to pull ahead. But I think it’s a narrow one.

Could China be a source of catastrophic AI risk?

Sihao Huang: I would say China is very much a relevant actor here, for two reasons. The first is that China does not actually need to develop frontier models to have frontier capabilities. There’s two ways in which they can access frontier models. One is we simply give them away, right? It’s looking likely that Facebook is going to be open sourcing its most capable Llama models that are near GPT-4 or greater than GPT-4 levels. And China would simply be able to run this locally, inferencing it on their hardware. They could potentially take these models, fine-tune them to remove safeguards, they could add additional modules or use them in more complex AI systems, and therefore gain frontier AI capabilities.
The second way that they can get frontier models is potentially stealing model weights. With the proliferation of different actors that are at the frontier, there’s maybe three, four companies capable of doing that right now in the United States. These companies are not necessarily resilient to state-level cyberattacks, and China is an extremely sophisticated actor, and a lot of this is also going to depend on the cyber offensive/defensive balance — particularly when AI technologies are involved. Maybe China would simply develop very capable cyber AI systems to then try to exfiltrate our weights and then run it on their local hardware.
And this brings me to the second point, which is that you don’t necessarily need world-leading compute to create highly risky AI systems. The biggest biological design tools right now, like AlphaFold’s, are orders of magnitude smaller in terms of compute requirements than the frontier large language models. And China has the compute to train these systems. And if you’re, for instance, building a cyber agent or something that conducts cyberattacks, perhaps you also don’t need the general reasoning or mathematical ability of a large language model. You train on a much smaller subset of data. You fine-tune it on a smaller subset of data. And those systems — one, if China intentionally misuses them, and two, if they get proliferated because China just releases them as open source, or China does not have as comprehensive AI regulations — this could cause a lot of harm in the world.
Luisa Rodriguez: OK, so the idea there is that one key way that AI can pose a catastrophic risk to humanity is that frontier models end up very, very intelligent, and are deceptive, and can proliferate on their own and so take over. But there are potentially many other paths to catastrophic risks that don’t involve being at the frontier. For example, much smaller models might be able to create biological weapons; much smaller models might be able to do the kinds of cyberattacks that mean China could steal model weights. And all of this means that China is actually still super important to catastrophic risks. Is that kind of right? And if so, are there any other kind of paths that are worth highlighting?
Sihao Huang: That’s definitely right. I think I would also emphasise that catastrophic risk here, at least the way that I conceptualise it, isn’t simply direct harms from these systems. You should also be thinking about systemic harms, where the deployment of AI causes very bad social externalities, or causes massive inequality. And I think in that world, China is also a relevant actor to be included in conversations about how AI benefits are going to be shared equitably around the rest of the world; in conversations about how AI deployment could potentially significantly change our information and cultural environments. Or for instance, in China’s deployment of these systems that we talked about before, capable of mass surveillance and capable of reinforcing authoritarian rule; and also China’s potential export of these systems that we talked about before that could be used as mass surveillance and enforcing authoritarian rule to other countries, and thereby spreading its ideology — and I think causing a huge amount of disvalue.
Luisa Rodriguez: Yep. OK, those sound bad. Are there any other pathways to big risks that you want to mention?
Sihao Huang: I think a big one that I’ve been thinking about increasingly is: how do global deliberative processes look when we eventually have to make the call to go ahead with the intelligence explosion? And let me just paint the picture of this. Let’s say it’s the year 2040, and we have some sense that AI systems are approaching a threshold where it is generally capable, similar to human capabilities in almost all domains. And therefore, it’s able to start speeding up science and technology innovation, it’s able to start doing AI research, and we can actually deploy it at a vast scale because of the compute overhang that we have from training.
Now, how do we make the choice to do this or not? Because it’s quite likely that once we make this decision, things are going to go really, really fast. Now, we may want to make sure that if China is in the position to do this, they are not going to push the button without talking to the rest of the world to make sure that this goes well. We also want to make sure that if America were to do this, China is not in a position to nuke us first, because a decision maker in China may look at the situation and think, “This is going to forever make America dominant in human history, and we may want to take the chances to annihilate them.” Situations like this are broadly… Like, I think I’m outlining a very particular circumstance, but big power upsets can cause a lot of instability in international politics.
And beyond the risks of just kinetic conflict, I think in a situation where you are staring down the intelligence explosion, you really want to make sure that there are good deliberative processes that bring in voices from all of humanity and are going to feel the consequences of whatever you’re going to do next. You want to make sure that there’s a way to bring all the countries to the table.

AI enabling human rights abuses and undermining democracy

Sihao Huang: For context, the Uyghur people are being prosecuted and under what many people consider a genocide right now in Xinjiang. So these systems are being deployed extensively in these cities to “identify where there are troublemakers or potential troublemakers.” And you really see this very tight-knit connection between AI advancements and China’s capability to perform repression at home.
And so there’s this paper that I think highlights this dynamic very well by Martin Beraja, who is at MIT econ. It’s very aptly titled “AI-tocracy.” And he does a lot of things in this paper, but primarily he identifies a number of links. First is that he looks at different cities in China, different provinces in China. After a major protest event or social unrest event happens, there is an increase in police procurement of AI-based surveillance systems. And then he shows that with the increased procurement of these AI-based surveillance systems, they actually make it more unlikely for Chinese citizens to take to the streets.
He then shows that when these systems are bought from these companies, the companies, one, get additional money to do R&D, and two, get data contracts from governments that allow them to make these systems more effective. So you get this close feedback loop of AI companies building more and more advanced surveillance systems, the police freeing up budgets and increasing the amount of surveillance that is happening in China.
I think this outlines a pretty unique dynamic here, which is that most authoritarian states need to keep a certain amount of people happy to conduct repression. This is typically called the selectorate. And the selectorate needs to extract a certain amount of political rent from the regime. But if you’re able to automate the portion of people that you need to keep happy to enforce your authoritarian rule, that selectorate can essentially shrink to zero.
Luisa Rodriguez: Can you actually spell out this mechanism where typically authoritarian regimes need to kind of please a selectorate, but AI could make it so that the selectorate goes to zero? Both what actually is the story that means authoritarian regimes need a selectorate at all, and how exactly does AI cancel that out?
Sihao Huang: I think the notion here is that this goes back to the other, sort of the traditional political science framing of a principal-agent problem. A principal needs agents to execute their decisions and execute power. For instance, Xi Jinping does not rule China on his own; he rules China by proxy through other people — like on the politburo, which are on the top of the Communist Party hierarchy — which then rule China through provincial governments, that then rule China through local governments, that then rule China using a vast police force to conduct repression, but also rule China through actually doing good things for the people that garner support.
And you could keep ruling this country while doing terrible things for the people, as long as you’re oppressing them enough such that they don’t overthrow your power. And that’s not a hypothetical: that’s happened before in Chinese history itself. For instance, during the Maoist era: it was mass starvations, huge amounts of repression. Some of the worst things to have happened in humanity happened during the period of time when Mao was ruling the country.
And during this period, he needed to make sure that people around him were sort of aligned towards his goals. Part of this was done through sheer terror, and part of this was done through making sure that the Communist Party cadre and elites were really benefiting from this regime. And if you’re able to automate the instrument of repression, the number of people that you need to satisfy becomes significantly lower.
I think the right way to think about this is that authoritarians then feel much less constrained, because they don’t need to appease this vast structure of power that lies below them. And this lack of constraints could mean that authoritarianism is more robust, but it also could mean that you get significantly crazier outcomes — because Xi Jinping doesn’t need to be thinking about who is going to be happy about his choices and who’s not.

China’s attempts at indigenising its semiconductor supply chain

Sihao Huang: So in the short term, they will likely be able to make domestic chips. They’ve already been able to make domestic seven-nanometre chips that are below the line that the US wanted to control. But the point is that they’re not able to make these chips at very high yields and at very high volumes.
I think the timeline for the next few years is quite bleak. They’re basically working with the equipment that they’d already imported from the rest of the world — from the Netherlands, from the US, from Japan — over the past few years. And that stockpile is going to determine how many chips they can pump out — and it’s not that much.
The bar of uncertainty actually becomes much higher when we think about what happens maybe three, five years or more down the road: China could potentially be looking into different technologies for building these chips that go around American expert controls. Like, maybe they will try to make three-nanometre chips without using EUV machines, and try to find innovations that allow them to do that at a decent yield. They may also be trying to look at semiconductor technologies that don’t require lithographic scaling.
The semiconductor industry is one that is also incredibly path dependent, because if you look at the history of semiconductor development, there were many points at which the industry had to make a collective decision on what technology to adopt and to scale out to continue Moore’s law. So a very prominent one was back in the late ’90s, a lot of companies came together to ask the question of what are we going to build for our next generation lithography systems? And there were a number of candidates. EUV was only one of them. There was also x-ray-based lithography; there was ion beam lithography; there was electron beam lithography. And eventually the industry converged on EUV. It wasn’t entirely clear that this was the best decision. It was the best decision based on the available information then, and the industrial structure of the countries developing this technology.
The information that we have now is very different. The industrial structure of China is very different from that of the rest of the world. So if China were to put significant resources into chip indigenisation, it could potentially find these other paths to making semiconductor devices more efficient and advanced.

How the US and UK might coordinate with China

Sihao Huang: One very exciting set of progress that has been made in AI governance recently has been the establishment of AI safety institutes across different countries. Canada has one, the United States has one, the UK has one, and Japan too. And Singapore, I think is in talks of setting one up. These AI safety institutes are organisations that coordinate model safety and evaluation within each country, and potentially also fund AI safety research in the long term. But it would be very good if China has a similar organisation and can be in touch with other AISIs to share their experience to eventually harmonise regulations, and, when there’s political will, push towards consolidated international governance or basic rules in the international world about how AI systems should be built and deployed.
So I think a near-term goal that would be good to push for when we’re talking to China is to have them also create some sort of AI safety coordination authority. We talked about how China already has a lot of infrastructure for doing AI regulations, and this could potentially come in the form of a body that is established, let’s say, under the Ministry of Information and Industry Technology, or the Ministry of Science and Technology, that centralises the work that it takes to build and push for AI safety regulations on top of what China currently has in information control — and then can become the point contact when China sends delegations to future AI safety summits or to the United Nations, such that we can have common ground on how AI regulation needs to be done.
Luisa Rodriguez: OK, neat. Is there another you think would be good?
Sihao Huang: I think something that would be really good for the US and China to work together on would be to have China signed on to some expanded set of the White House voluntary commitments on AI. These were policies or commitments to create external and internal red-teaming systems to evaluate these AI models; build an ecosystem of independent evaluators in each country to be able to check the safety of frontier models; build internal trust and safety risk-sharing channels between companies and the government, so the governments can better informed about potential hazards that frontier AI can pose; and also investing in expanded safety research.
You may also want China, for instance, to sign on to responsible scaling policies within each company — jointly defining things like AI safety levels, protective measures that go onto your models, conditions under which it will be too dangerous to continue deploying AI systems until measures improve.
And I think the right framing here is not just have China sign on to American White House commitments, but can we also identify additional commitments that Chinese companies have made or China’s asked from its AI developers that would also be beneficial for us. And we structure this as some sort of diplomatic exchange, where “we do more safety, you do more safety” — and we’re learning from each other in a mutual exchange.
Luisa Rodriguez: OK. Thinking more about long-term goals, is there a long-term goal that you think would be robustly good to aim for?
Sihao Huang: I think the most important long-term goal to keep in mind is: how do we have continued dialogue and trust-building? I say this because of two things. One is that US-China relations are very volatile, and we want to make sure that there’s robust contact — especially when harms arise — that the two sides can work with each other, and know who to call, and know the people who are making these decisions.
The second reason is that tech development and policy responses to address the harms of potential new technologies are very difficult to predict. And we can only typically see one or two years on the horizon of what is currently mature. For instance, I think model evaluations have become much more mature over the past few years, and have been pushed a lot onto the international stage, but they’re definitely not all that is needed to guarantee AI safety. Bio and cyber risks have also started to materialise, and are sort of formalised into benchmarks on defences now. There’s also a growing community now on systemic risks and overreliance that I’m quite excited about. And compute governance is also emerging as a key lever.