On Sunday, China’s Minister of State Security published an article in a state-run magazine that warned about artificial intelligence posing a threat to the rule of the Chinese Communist Party (C.C.P.). This coincided with increasing concern among Americans that A.I. could pose existential risks to humanity, and rising calls for Washington to regulate it. Many of those pushing for regulation work at leading A.I. companies such as Anthropic and OpenAI, and they have received support from a broad ideological coalition on Capitol Hill. But there remains little appetite for regulation in the White House; President Trump has said that the United States must accelerate A.I. development to outpace China.
I recently spoke by phone with Kyle Chan, a fellow at the Brookings Institution who focusses on Chinese technology policy. During our conversation, which has been edited for length and clarity, we discussed the Chinese government’s specific fears about A.I., whether the United States and China can negotiate on A.I. safety, and what the presence of Donald Trump in the Oval Office means for China’s approach to the United States.
The conversation about A.I. in the United States has, in the past couple of months, really accelerated—first, around data centers, and then with fears about what A.I. could mean for the future of humanity. Do we have any sense of whether the same acceleration of the conversation has happened in China, or is the conversation there on a separate timetable?
They seem to be quite separate, in many ways. But the conversation around A.I. risk has sped up in China. It is still quite different from what it is in the U.S. There’s a key distinction here between different types of risks. Most recently, we have this sudden mass concern across the general public about existential A.I. risk. These are concerns that A.I. could kill us all, as that Anthropic employee, Jacob Coxon, said when he resigned. And that has really grabbed headlines, but that kind of existential A.I.-risk talk is almost unheard of in China. Within the A.I. industry in China, there are some folks who worry about technological loss of control, as they call it. But, generally speaking, it is a very, very marginal issue.
That being said, there are fears around A.I.
Before we get to those other fears, how do you understand the different levels of concern about existential risk? Some people here would read what you have said and say, Well, a lot of the talk in the U.S. about existential risk is driven by these companies that have some self-interest in saying that there’s a great risk, because it may make their product seem more powerful, or, they want other competitors to be regulated. I’m not saying that is correct, but is your reading less cynical?
Yeah, so in the U.S., I do think the belief about existential risk is genuine, and that this is not all just an effort to hype up the technology, or to secure some kind of regulatory capture, or about these other ulterior motives that have sometimes been attributed to the folks calling out these existential risks. I think there are a lot of true believers in the U.S. A.I.-research world who have been thinking about, reading about, talking about existential risk and what it could mean for a long time.
In China, what I think is different is the slope of the acceleration curve. I think, in the U.S., it feels like the slope is very, very steep and getting much, much steeper, and that we are entering into some kind of takeoff point, crossing a kind of threshold. And in China you don’t have that sense that a turning point is around the corner, or that the technology’s about to get completely out of our control. Chinese A.I. researchers just don’t see the technology moving as quickly as people in the U.S. do. They are trying to catch up with their U.S. counterparts. They have less access to chips than OpenAI and Anthropic, for example, and internally they don’t see a closed feedback loop spinning out of control anytime soon.
What are the ways that the leadership of the Chinese Communist Party is starting to have fears about A.I.? Does the C.C.P. see it as a threat?
Yeah, to some extent. I would highlight a set of growing major A.I. risks that China and the C.C.P. élite are getting more worried about, because for a long time Chinese policymakers were not so focussed on these issues. They were more focussed on social issues, they’re more focussed on political content, and they were treating A.I. a lot like they had treated other kinds of digital technologies. And so they were applying, in some ways, an established internet-censorship and social-media-censorship playbook to A.I.-generated content in particular. So that was sort of their primary concern for a long time, and it still is a major focus. And, when you look at their regulations around A.I., a lot of them are aimed at these issues in particular.
But, more recently, you have a couple of new major A.I. risks showing up at higher and higher levels of the Chinese Party state. And one is cyber risk. Mythos was a wake-up call for Chinese policymakers. Mythos is the model that Anthropic had developed and announced in April this year, and it appeared so powerful in terms of cyberattack capabilities, and being able to find vulnerabilities in widely used software platforms, and chain them together in a very sophisticated way that basically matched, or even maybe exceeded, teams of very, very strong hackers.
Anthropic withheld the release of Mythos until it could share the new model with a select group of government entities and companies first. China was watching all this very closely. And Mythos sent shock waves through the Chinese policy establishment because it signalled that the potential for A.I. to be used for cyberattack and cyber defense had entered a whole new era. And it’s been talked about in Chinese state media, and then, most recently, we had the head of China’s Ministry of State Security calling out, specifically, Mythos, and a similar model from OpenAI for heralding a new era of A.I.-enhanced cyberattack capabilities, and what that could mean for China’s national security.
And then there is the loss of control of some almost runaway A.I. systems. China’s been following very, very closely what’s happened with the Hugging Face incident, in which an OpenAI model broke out of its sandbox where it was supposed to be contained for testing within an environment that was not connected directly to the internet. And it was supposed to be tested safely within the confines of that kind of cyber cage. But what happened was that it got out, and it attacked an American A.I. platform called Hugging Face. And perhaps the most alarming part is that OpenAI did not know this was happening until later on. So this is an example of A.I. itself getting out of control and potentially causing real damage. In this case, there wasn’t actually any economic damage, but you can see that it wouldn’t take a huge leap to see the kind of damage that could happen if it weren’t Hugging Face that were attacked but, say, some major digital infrastructure platform in the U.S. or China. Those incidents have also been written about a lot in Chinese state media. On the one hand, they try to frame it as, at least publicly, a kind of American issue. And, ironically, Hugging Face used a Chinese model to defend themselves, so in some ways China tried to portray this as a victory for Chinese open-source A.I. models. But you can see in other statements, including from the Cyberspace Administration of China, that they’re worried about these sorts of, what they call, “loss-of-control incidents,” where the technology gets out of hand, where the A.I. system can operate autonomously, and humans can’t even track what it’s doing. And it does things that were not what was originally intended by the human operators.
The Hugging Face incident set off all these concerns in America about existential-risk scenarios as well. But it seems like from what you’re saying that the response to the incident was different.
Yeah, so in the U.S. we also have these emerging loss-of-control concerns about A.I. systems with their current capabilities getting out of hand, potentially hacking other websites, hacking companies, doing damage at that level. But there’s another level, which is if we can get A.I. systems to improve themselves, to create this feedback loop of continuous self-improvement, where they get better and better and better, the worry in the U.S., at least among some folks in the A.I. community, is that this trend will accelerate to a point where humans no longer have control over the process of A.I. self-improvement. And then the A.I. will potentially become its own sort of autonomous entity, potentially with its own goals, or its own decision-making processes. And then the most scary part in that vision is that it would be super intelligent, and it would be able to hack and infiltrate digital systems and come up with novel biological pathogens and produce all of them in a way that we can’t even imagine. And this is where it starts to enter into a kind of sci-fi-type scenario, but that’s the genuine belief of many who are concerned in the U.S. A.I.-safety community.
How much do you think the C.C.P. views A.I. as specifically a threat to their rule?
One thing they’re worried about is the U.S. using some of this technology to undermine C.C.P. control. And I’m drawing here from the specific piece written by the Minister of State Security, and it focussed on deepfakes and efforts to manipulate what the C.C.P. considers China’s ideological space. That is something that they’re very worried about, and they’re worried that other so-called foreign actors might try to use A.I. to influence the social and political fabric in China. And that could be a direct challenge to the C.C.P.’s control over the country.
What do we know about the degree to which Chinese citizens are concerned about A.I.?
People are concerned in two ways. One is they’re worried that they’re not running fast enough with A.I. They’re worried that, given the already intense competition for jobs, especially among more college-educated Chinese people, especially for the kinds of jobs that they think are commensurate with their level of education, there’s a big supply-and-demand mismatch. And there’s been growing and growing anxiety about being one of the lucky few who can nab a good job. This is part of this broader term that was originally used in the Chinese context for the labor market, which is called involution. And it’s this idea that competition is so intense and so fierce that you, as an individual, feel like you are running faster and faster. But you are actually still falling behind, and not able to make progress. That’s the kind of anxiety that people feel in China, even before A.I. And then what A.I. did was kind of turbocharge that anxiety, where people felt like they needed to quickly adopt and use it and learn how to leverage A.I. to make themselves more competitive in the job market. So that’s one side of it.
The second concern is just outright job displacement. This is becoming a greater and greater issue among people in China, and it depends on your job type and where you sit in the income or educational structure, but this is a growing concern. And I don’t think it’s reached the same kind of level that you see with, say, Gen Z feeling like they’re cut out of the career ladder in the U.S. But you can break it down, sector by sector, whether you’re talking about white-collar workers, especially young white-collar workers, or salesfolks. And then, even in the physical world, factory workers and delivery drivers are worried about greater A.I. autonomous delivery, for example. And that’s growing, even though, for a while, the issues seem to almost have been downplayed in the official media discourse.
How would an American President approaching the Chinese government and saying that they want to take steps to start thinking about regulating A.I. together be received? And how does that answer change given that the United States’ President is Donald Trump?
So, in general, regardless of who is President, I think that at first, on the Chinese side, there would be deep suspicion about the U.S.’s true intentions with these discussions about A.I. coördination, or even coöperation. There are definitely still very hard feelings in China about U.S. export controls on chips and chip-making equipment to China, which has limited China’s ability to build the kind of computer needed to train really powerful A.I. models. So the United States has placed constraints on Chinese A.I. development, and they resent that, even though China’s trying to get around them. And so they would probably be very skeptical of any proposal for outright coöperation on A.I., because they see the U.S. side as being more antagonistic and more about competition. The Chinese are always complaining about A.I. development being a zero-sum competition.
But, at the same time, the A.I. risks are growing, and I think Beijing’s perception and concerns about those A.I. risks are also growing. And that increases the motivation for China to come to the table and start to talk about some of these issues. And they can propose things like putting guardrails for misuse, like for cyberattacks, or for developing bioweapons. Or it might be some kind of coöperation around non-state actors, and preventing American or Chinese models from getting into the wrong hands. It could even be a discussion about A.I. in the military, which is a very big issue for China, and they have been watching very, very closely the use of A.I. in the Iran war. So there are a couple of issues where, potentially, the Chinese side would have motivation to want to talk about these with the U.S. And then the big question would be whether anything concrete could come out of this? And I would not expect things like binding agreements, like an arms-control treaty for A.I., to be something on the near or medium-term horizon, but they could at least start to share information about some of these issues, some of their concerns, and also what some of their safety procedures are on both sides.
And then on Trump in particular, I do think that the Chinese are looking beyond Trump and see him as a temporary figure, and might be reluctant to agree to anything long-term because it could evaporate once he leaves, or even while he’s still in office, because of the question of whether he would stick to it. So I think the lack of a stable political environment in the U.S. would undermine any kind of substantial progress on that front.
How do you think the C.C.P. views Trump generally at this point?
China is quite pleased that Trump has been doing things that China almost could never dream of in terms of undermining some U.S. competitive strengths, and in terms of undermining U.S. alliances abroad. Those are things that I don’t think China would have been able to do, or even dreamt of doing on its own, and so they’re very happy to see Trump take a wrecking ball to a lot of that. And then, in particular, I think they also feel like they have now kind of matched the U.S. punch for punch with their rare-earth export controls, which shows a broader ability to hit back against other U.S. actions. I think they see Trump as trapped in a stalemate with them, while, at the same time, he’s been weakening the foundations of American strength and getting the U.S. tied up in yet another war in the Middle East, and I think even if China would not want these things to necessarily drag on, like the Iran war, these are all just helping to shift that balance of strength in China’s favor.
I want to go back to your early answer about existential risk, because one thing you hear is that China is essentially copying our A.I. models but is a little behind. Does that suggest that they will soon be getting scared about existential risk, too?
That could happen, and that’s an interesting question. I think one variable that could change how they see existential A.I. risk is if they, in their own A.I. labs, see an acceleration similar to what American A.I. labs are seeing now. Maybe some of the bottlenecks or key steps that they thought would take much longer to resolve do get fixed, and then that could shift things.
I think then we would be more likely to see international coöperation. Let’s say a terrorist group or even an individual starts to use an A.I. tool to develop an enhanced virus, or there’s a big cyberattack that uses a Chinese or American model to attack another country’s digital infrastructure and shut down a section of the power grid. Those incidents might escalate the perception of risk in China, and maybe also in the U.S. And that might bring them closer to wanting to talk and discuss coöperation on A.I.
Do we know how the C.C.P. thinks their response to COVID looks in hindsight, and do you think that could affect how they respond to issues around A.I.?
On COVID, I think that you hear less about the celebration of China’s rapid response to lockdowns in the early stages, where there was some chest-beating from China about how they had done a better job of locking down than the U.S., but then later on, obviously, there was a really horrible result to the tapering off and end to the “zero COVID” policy in China. So that temporary triumph is now not talked about as much. But one thing that I think the lockdowns did do is they gave Chinese political leaders a sense that they could control a situation, even something like a pandemic, in a way that they couldn’t before. In other words, I think the degree of social control that they were able to exert over the rest of the country maybe even surprised people in Beijing. And I wonder if some of that carries over, that kind of confidence in the power of the state, and in the power of state control.
But then, at the same time, and I’m sorry for the two-sided argument, is the fact that toward the very end of COVID, especially with the Omicron variant, they could not control the spread of that particular virus. It was something that they essentially had to give up on. And that’s what ended “zero COVID.” It was just a losing battle. And that might provide the opposite lesson to C.C.P. leadership, that if some of these things get out of hand, and in some ways, A.I. is more like a biological organism than, say, high-speed trains, or electric cars, or other kinds of technology, there is a chance the C.C.P. might be more worried about their ability to contain this thing, should it escape from the lab, or escape from the sandbox, and start to do things that were never intended. So that would mean tighter control earlier on, or greater fears of losing control. So the lesson from Omicron could have been, If you don’t nip it in the bud, then there’s a chance this whole thing could escalate and get totally out of hand. ♦