When Anthropic CEO Dario Amodei called on U.S. labs to pace the AI frontier, Washington said no — and so did Beijing.
In fact, Chinese state media outlet China Daily went further, calling Amodei’s suggestion a “self-serving bid for profit.”
The back-and-forth has led to an ongoing debate around whether China takes AI safety seriously.
Kendra Schaefer’s take: China does take AI risk seriously — but what it won’t accept is the premise that its own progress is the problem.
On this episode of the Trivium China Podcast, host Andrew Polk sits down with Kendra (Trivium’s Head of Tech Policy Research) to unpack:
Why China’s response to the “pacing the frontier” debate looks dismissive on the surface, and why that reading misses how much AI regulation Beijing is already planning
How pursuing open-weight models has become a geopolitical position for China, with roughly 70% of derivative models on Hugging Face now built on Chinese base models
China’s new argument in the distillation fight: that banning it is an unfair trading practice and proof of US efforts to assert technological hegemony
The one thing that could flip Beijing’s calculus on managing open-weight releases, a major security incident, and how China’s COVID-19 playbook suggests it would shut the door rather than share what happened
Whether the new U.S.-China AI safety dialogue, set to kick off in November, has any path forward
Give it a listen and let us know what you think.
Transcript
Andrew Polk: Hi, everybody. Welcome to the latest Trivium China Podcast, a proud member of the Sinica Podcast Network. I’m your host, Trivium co-founder, Andrew Polk, and I am delighted to be joined today, once again, for the first time in a while, actually, by Trivium’s Head of Tech Policy Research, Kendra Schaefer. Kendra, welcome back. How are you doing?
Kendra Schaefer: Hey, I’m doing good. Glad to be here again.
Andrew: Yeah, always glad to have you on the pod and excited about our conversation today. We are going to get into a conversation I’ve been actually wanting to have with you for about 10 days now. We’re going to talk about AI, and specifically sort of the AI pacing issue that has been in the headlines and, related to that, AI safety and AI governance. And we’re going to get into China’s specific views on these things because there’s obviously also this conversation about how and whether the U.S. and China are going to coordinate on these issues.
And then, of course, just a few days ago, Xi Jinping was in Washington, D.C., and there was talk about setting up an AI dialogue. Unclear exactly what that means, but you’ve written a couple of pieces recently– one on kind of China’s view on AI pacing and AI safety and AI governance, and then separately on the challenges that will likely impede a robust AI dialogue between the U.S. and China because they’re really kind of coming at these things from different starting points. So, we’re going to get into all of that today. But of course, before we do, we have to start with the customary vibe check. How’s your vibe today, Kendra?
Kendra: You know, I’m actually feeling gypped. I listened to a podcast you did with Joe last week, No Vibe Check.
Andrew: Yes, yes. So, wait, are you feeling cheated because I’m making you do it or because I didn’t make him do it?
Kendra: A little bit of both. I want to know if Joe got a special dispensation or if there was just, like, some failure of the audio and it had to be cut out.
Andrew: No. So for you and for other listeners, on our shorter quick reaction podcasts, I try to keep the intro shorter so that proportionally, I guess it takes up less of the time, and we get straight into it. So, we kind of have what I view as our sort of anchor longer podcast each week, and then the shorter quick turn, quick reaction ones, I try to keep a little bit tighter. So that’s why Joe was off the hook.
Kendra: Oh, that’s a little look behind the curtain for me. I’ll have to get booked on one of those sweet, sweet quick take slots.
Andrew: Well, the problem with the quick reaction ones is they always end up being 45 minutes anyway.
Kendra: Of course. It was a good podcast.
Andrew: Yeah, I thought it was great. Joe did a good job. Unfortunately, we were going to do a follow-up, and it’s just because of the holiday in China, we couldn’t quite have him back on. But we will be talking about outcomes from the Xi-Trump meeting on the AI front today. And I’m sure we’ll be talking about all those issues on an ongoing basis. For my part, I am just, again, a little peek behind the curtain, in a weird headspace because, we’re going to lose listeners because of this, but I’m a Manchester City Football Club fan. It’s an English Premier League team.
Kendra: Appalling.
Andrew: They just got handed down a monumental fine, effectively, for financial cheating.
Kendra: Oh, really?
Andrew: This has been the big news in the soccer world over the past week or so. And I’ve been listening to a lot of analysis on what it will mean. And as a fan, as a supporter, it’s very complicated feelings. And I’m sure for the players, it’s even more complicated. But anyway, I’m emotionally conflicted. And that’s the energy I’m bringing into this podcast.
Kendra: Good, good, good.
Andrew: Well, whatever that means. If I say anything weird, you know, listeners will know why.
Kendra: I’ll know it was them.
Andrew: Exactly.
Kendra: I’ll know it was soccer.
Andrew: Exactly. Before we get into the meat of the content, we also quickly have to do the housekeeping as well. Just a reminder up top, we’re not just a podcast here. Trivium China is a strategic advisory firm that helps businesses and investors navigate the China policy landscape. That, of course, includes domestic policy in China along a range of issues, including AI and technology, which we will talk about today, as well as policy towards China out of Western capitals like D.C., London, Brussels, and others, which we will also talk about today. So, we’ll kind of cover the gamut on the AI policy front. But if you need any help on that front on any of these regulatory issues, please reach out to us at hq@triviumchina.com. We’d love to have a conversation about how we can support your business or your fund.
Otherwise, if you want more Trivium content, please check out our website. Again, triviumchina.com. We’ve got a bunch of subscription options for people to stay on China policy developments. Kendra and her team write an excellent daily tech policy update. It keeps you clued in to everything that’s happening in the China tech policy front. And then we’ve got markets’ notes. We’ve got general China watcher notes for people in business who need to just kind of stay up to date on what’s going on in China. So, check that out. You’ll definitely find the China policy intel option you need on our site.
And finally, please do tell your friends and colleagues about Trivium, both about the business and about the podcast. I say it every time, but I can’t say it enough— It really helps us grow our company and to grow the listenership. And these word-of-mouth recommendations from current clients or just people who know Trivium, people who heard the podcast mean a lot in terms of getting other folks interested in our work and helping us to grow our listenership and grow our business.
So, we really do appreciate those. Please do keep those coming. All right, Kendra, are you ready to get into it?
Kendra: Yeah, let’s get into it.
Andrew: So there’s obviously, I think I can say obviously, I over rely on the word obviously, but I think it’s true that anyone who’s been paying attention will have seen this explosion of conversation around AI safety, you know, and the so-called need to pace the frontier, which was in large part sort of kicked off at least this recent episode by sort of almost infamous Pacing the Frontier Letter released by, I think, the CEO of Anthropic, which essentially argues that AI development is moving so fast it could pose existential risk to humanity and kind of trying to call on the other U.S. labs to work together to deliberately introduce friction into their own processes so that they can ensure that the guardrails around AI keep up with the innovation, and that regulators also step in and take a look. Have I got that basically right, Kendra?
Kendra: Yeah, that’s right. I mean, the response to that, I mean, that was just one tiny part of what happened over the last two weeks. There was a massive sort of response to the Pacing the Frontier letter. Obviously, the Trump administration loudly and roundly rejected the very idea of regulating AI, essentially said he would not sign off on any kind of pacing, did not intend to regulate, and ultimately said that the U.S. couldn’t afford to slow down because China’s not slowing down.
And then China sort of entered the chat and came after Anthropic, came after the U.S., came after many of the ideas that had been put forward. There was a sort of diplomatic statement by China rejecting the idea of a coordinated slowdown with the United States or with U.S. labs. State media outlet, The Global Times, hit back against Dario Amodei’s letter, sort of casting doubt on the motivations behind him posting that letter.
China Daily released an op-ed calling the letter, “Self-serving bid for profit,” by a U.S. tech elite. And then what was also really interesting was that China’s intelligence services, the MSS, had a couple of interesting releases, or MSS-related bodies and people also publicly posted on this. Most notably, Spymaster Chen Yixin, China’s minister of state security, published an essay on AI risk in which he kind of addressed some of these existential risk, and in which she kind of also obliquely sort of poked it at U.S. labs and criticized them of being sort of hypocritical. So, just sort of an outpouring of discussion on this issue from all corners.
Andrew: That’s right. Yeah. And then, of course, just to further lay the groundwork in terms of developments that have happened that we’ll be reacting to today, there was also, as I mentioned, the Xi Jinping-Donald Trump meeting in Washington, D.C.
Kendra: Not to be forgotten.
Andrew: Exactly. And a lot of speculation that sort of there might be some more substantive discussion between the two on AI safety. That didn’t really seem to happen, I think, in part because there wasn’t actually much substantive conversation in the meetings. Joe and I talked about that, a lot of sort of more ceremonial type developments, but not a lot of really deep policy discussion. That said, Scott Bessent, the U.S. Treasury Secretary, and He Lifeng, the vice premier in charge of the economy, who’s also the lead negotiator with the U.S., effectively have established this U.S.-China AI safety dialogue that they announced, I think, a couple of days officially before the Xi-Trump meeting.
And one of the outcomes of the Xi-Trump meeting was that that dialogue will have its first official meeting in November or around November. And the idea is that it will, we think, in a minimum report, be a reporting channel for AI safety incidents. Does that all sound right?
Kendra: Yes. Yeah. So there was a ton of drama, right? The result of all that back and forth is that I think we now understand much more clearly what China’s emerging positions are regarding issues like AI safety, cooperation with the U.S., distillation, etc. It’s just a lot clearer where Beijing stands now than it was even a couple of weeks ago. And of course, understanding Beijing’s positions on those issues are really important because those are the positions that Beijing’s going to, you know, they’ll evolve, of course, but they’ll take those positions into any future dialogue with the U.S.
In November, these positions will have solidified a little bit, and that is what negotiators will come into the room thinking. So, it’s been very interesting to see some of those sort of Chinese talking points around AI and AI safety evolve.
Andrew: Yeah, for sure. I think it’s hugely important for all of us, and the U.S. in particular, to understand the priorities in Beijing around these issues, to understand kind of where they’re coming from, what they’re trying to achieve. We, of course, don’t have to agree with those priorities or goals, but that will sort of help us understand the space for coordination, the space for negotiation, and the limits around any dialogue or the possibility space for any dialogue.
I think also we’ll talk about today, I think a lot of folks probably incorrectly think that China isn’t interested at all in AI safety. I think that really couldn’t be further from the truth, but you will talk us through that, and then we will kind of end up talking about some of the challenges that we foresee for the AI dialogue overall. But we’ll start with this piece that you wrote, really outlining China’s emerging positions on AI safety, and boil down China’s view to a few really key points. So why don’t you walk us through your piece, and we’ll kind of go through all of that.
Kendra: Yeah, for sure. That’s a great segue and introduction. You mentioned that there’s quite a bit of conversation around the fact that maybe China doesn’t necessarily care that much about AI safety. I mean, one of the most confusing aspects of the Chinese commentary on AI safety has been a very sort of contradictory positioning on it. Some of those commentaries appear to sort of brush off the idea that AI represents an existential risk to humanity or even a consequential risk to national security and the economy.
Right? Some of the articles that we saw come out, some of the conversations we saw come out over the last couple of weeks when they were talking about Anthropic’s view on AI safety- that it does present an existential risk, that it does need to be addressed immediately, that regulators need to step in, that governments need to sort of, you know, immediately coordinate and take emergency measures was pretty dismissive.
The Global Times essentially said, “Tech companies have strong incentives to portray AI as more powerful, mysterious, and even dangerous than it actually is.” So that makes it sound like, no, we don’t think that we think all of these AI safety conversations are overblown. But meanwhile, you look at other Chinese official commentaries and what they do is sort of dwell on the risks represented by AI and sort of outline them and spend a lot of time sort of hyping them up. So, a similar kind of hype cycle, also from sort of official sources.
I mentioned that the MSS, sort of spymaster, had written this piece and his piece effectively, I mean, you know, he spends all his time doing threat assessments, I’m sure, but his piece enumerates a laundry list of major AI risks that he thinks China has to guard against — Risk to national security, risk to critical infrastructure, risk to party ideology, risk to personal data, risks to social stability.
And the state sees social stability as essentially a fairly existential concern. And so those are fairly extreme risks. So these kind of statements, when they’re taken out of context or when they’re viewed in isolation, sort of make it difficult to assess where the state, the Chinese state, the top leadership of China stand on AI risk and how they are guiding, intend to sort of guide China through this next five, 10 years of AI development.
One risk in particular is kind of at the top of everybody’s mind, which is, you know, there’s a segment of the AI safety community, I think particularly Anthropic is sort of representing that segment of the community, that believes that releasing the weights of, you know, near frontier models, as Chinese labs do, right? Chinese labs sort of have these open weight and open source models, represents a sort of more significant threat than closed models from a safety and security point of view.
And they want China to regulate the release of those model weights. They essentially want Chinese regulators to step in and prevent Chinese labs from releasing open-weight LLMs unless they have gone through some kind of safety assessment and that they have been checked to ensure that they don’t help develop biological or chemical weapons or they can’t be used as cyberattack tools, etc. So, I mean, that’s an understandable concern, considering the fact that in a sort of open-weight model, once it’s released, you can’t be recalled, right? There’s no take-backsies.
And so even if it’s later discovered that that model is dangerous, that’s kind of, well, whoopsie-daisy, there’s very little you can kind of do about it. Even if the state puts out a regulation at that point, it’s like too late, right? But at the same time, it’s also kind of undeniable that if Chinese regulators step in to regulate AI model safety in that way, it will slow down Chinese labs and it would place China’s current sort of AI development momentum in danger. So, given all of that, China appears to have totally rejected the invitation, verbally and vocally rejected the invitation to sort of yank the leash on open-weight releases.
It is not interested in, you know, at the behest of a foreign model maker essentially telling its labs to stop. And in fact, several state media outlets sort of threw cold water on the Pacing the Frontier letter and were expressing distrust of Anthropic’s motivations for taking that position, right? In response to their post, Global Times said that Anthropic’s position is “Less about safety than about consolidating control over AI technology.” And the next day, Chinese foreign ministry spokesperson took the podium and sort of waved off Anthropic’s warning as fear-mongering.
So, that makes you think China’s not taking AI risk seriously at all and that it’s ignoring all of it and that it’s kind of walking away from the entire idea. But we think that’s a totally incorrect interpretation.
Andrew: Yeah, let me jump in there. I think I don’t want to front run you here, but one of the best points you’ve made, I think, in your piece, which I think we’ll bring up now is, I mean, the U.S. labs and the US government are basically saying one way to keep AI safe is to not let China have it. Right. Basically, right? So, if your starting point on AI governance and AI safety is China doesn’t lead the frontier or even come close to the frontier in AI, then obviously the Chinese are not going to sign on to that vision of AI safety.
And then you can also, I think, understandably see from their point of view, if a fundamental view of the U.S. labs is that open-weight models are not inherently safe. And the open-weight models, that’s China’s angle on its attempt to be at the frontier. That’s also sort of implicitly saying China’s approach to AI is inherently unsafe. So, talk to us about how those kind of ideas fit in. And then you can tell us, is China taking AI risk seriously or not even kind of beyond those definitions?
Kendra: Yeah, you’re exactly right. I mean, I think what China is rejecting when it says all of these things about Anthropic or when it says, you know, this is fear-mongering or we don’t sign on to those ideas, what they’re actually rejecting is the suggestion that China’s progress in AI is problematic or that any attempts to… and they’re basically saying that an attempt to block or restrict Chinese development is a violation of its rights to develop technology and develop its economy. And that’s essentially the argument that China is making.
And Global Times essentially said that in not so many words or explicitly said that rather. They said, “Amodei’s proposals seek to portray China’s legitimate development in AI as a threat and further fuel confrontation between China and the U.S. in the field. Such rhetoric is inappropriate, groundless and hostile.” So, the idea is, listen, no matter what the next step is in terms of U.S. and China AI cooperation, whether or not we ought to cooperate on the technicals, and we can get into that in a minute, even though we recognize the risks presented by AI, that’s a no-go of a starting point, right?
Us not being allowed to develop it, or our development of AI being risky, that’s not a position from which we’re willing to cooperate with you on that. So no, thank you.
Andrew: When you say us, you’re saying this from China’s perspective, right?
Kendra: Yes, this from China’s perspective, exactly. But that doesn’t mean China doesn’t recognize the risks presented by AI or the risks presented by open models. And I mean, over the last several years, I think Chinese policymakers have kind of attempted to raise awareness about AI risk. There’s lots of policy documentation, including, probably most notably, there’s a new AI safety governance framework. It’s Chinese. It gets updated every year. It’s put out by this sort of cybersecurity body, TC260.
And every year we read it, it’s this about, you know, 150-page document. And it outlines sort of China’s vision on where risks are and what safety is and what are the particular issues and domestic AI safety. And so, they have listed things like loss of control of AI systems as a critical risk to the technology in that document, right? So, they’re certainly aware of the risk and they’re certainly concerned about it. They’re also 100% guaranteed to take actions that, in their view, will make AI development safer, at least for domestic networks.
And they’ve actually been extremely explicit about that, right? Xi Jinping has kind of said just a couple of months ago at the World Artificial Intelligence Conference, he kind of made a note in his speech that, “Yes, China supports AI, they support AI development, they intend to develop AI, and then he had a whole other paragraph about, and yet that doesn’t mean that we won’t write laws. We are going to regulate, we will control the risks of that technology, etc.” We’re not just reading the tea leaves from a couple of lines from Xi’s speech, we’re actually reading policy documents where they’re starting to outline exactly what kind of regulation they’re thinking about.
One of the more interesting regulations, for example, was this kind of potential plan to issue sort of network identifiers, basically IDs, you know, can China’s networks be, essentially their DNS networks, be modified to sort of issue IP addresses to individual agents that are acting online? So that’s a way to sort of control and trace and track the behavior and activity of agents as they sort of proliferate across the internet.
I mean, we expect there to be a situation where there’s more agents online than humans, blah, blah, blah, blah, blah. So, there are a lot of sort of pathways towards regulation China has taken and is already taking. And I think what all this boils down to is when we say we can’t slow down AI because China won’t slow down AI, what the Trump administration means by that is we cannot place a single solitary regulatory burden on any technology company because China won’t. And that is patently untrue.
China is one of the most regulated digital environments on Earth. They will absolutely domestically regulate artificial intelligence. In fact, if the U.S. fell off the face of the earth tomorrow morning and did not say another word about artificial intelligence, five years from now, China will have placed an eye-watering amount of regulatory burdens on their AI companies to prevent essentially the state from losing control of those networks and platforms. That is what the state does. So, I think that that probably means China probably thinks that kind of regulation will function as part of a pacing mechanism over time.
That for China, pacing is we are going to institute regulation, as we always intended to do, in a rational way that corresponds to the stack of regulations, that fits in with the network regulations and Internet controls we have already instituted. That’s just one of those points where I think it’s a false equivalency. I think it’s probably just two kind of governments talking past each other when one says, “Well, we can’t regulate because China’s not going to.” It’s like China’s actually going to regulate quite a lot.
Andrew: Yeah. As I think anyone who’s a China watcher knows, the fundamental priority of the Chinese Communist Party is to stay in power. And that there’s a lot of like things that flow from that. For the most part, you want social stability. It’s not a democracy, but you’re responsive to your people because that’s one way you stay in power. Another way you stay in power is not creating a technology that’s so powerful that’s going to take down your own government. Right?
Kendra: And letting it run wild. Like you want to harness the power. I mean, the state has always tried to harness the power of a technology while mitigating the risk. And that actually leads me to another really interesting point, because China has a conception of risk in terms of what risks digital technologies outside of AI represent that have existed for the last 20 years that have been implicit and inherent in every sort of digital regulation that has kind of come out of the state.
And that is the thinking that they will also apply to artificial intelligence, right? I mean, the state has always been concerned about things like cyber threats to critical infrastructure, the use of digital tools to violate personal privacy or leak state secrets, being dependent on foreign countries for digital technologies and services, psychological harms of digital technologies. So, those things have been true for all digital regulation.
China will prioritize that type of risk when it comes to AI. And then it will also have to sort of, when the regulation comes in, the state has to figure out its way forward in terms of dealing with the specific risks that AI represents that other digital technologies do not represent. That’s things like loss of control, hallucinations, opacity, and sort of unexpected behavior, cyber threats and vulnerability discovery, algorithmic bias, that sort of thing. Again, all of that stuff is definitely coming regardless of what the U.S. does.
And China has been pretty kind of clear on that. And I think what we’re often seeing is the U.S. side has its own understanding of how it wants China to think about risk right now. And it keeps kind of lobbing this definition of risk over the fence and going, ”We need you to adopt this definition of risk right now immediately and come sit down with us and talk about it.” And China’s like, “No, go away.” Right? And it’s this kind of mess.
Andrew: Yeah. Well, I mean, the reason I bring up the CCP’s ultimate goal of maintaining power is just sort of like that, it strikes me, that that’s a place sort of maybe to start the conversation from the U.S. side to say, “Listen, we know that you want to control this technology for reasons of social stability, regime safety,” all of that stuff. How do we then apply that beyond the borders of China? Like, what is the Chinese conception of AI safety mean in terms of where there’s overlap with the US conception of AI safety? And maybe that’s a starting point. It’s not an ending point by any means, but maybe kind of conceding.
Yeah, we understand you’re worried about this technology too. Let’s figure out where we can overlap. Might be a starting point. Maybe I’m being too naive there. But then you also made the point about there’s one thing in particular you think that could really push the Chinese side into overdrive in terms of its willingness and incentive to control, especially open releases, much more aggressively.
Kendra: So, I mean, I think in terms of what you just said, I mean, we’ll probably, maybe we can come back to that later, I think probably the best way for the U.S. and China to sit down and cut through all of the nonsense here is by just sticking to the technicals. There are lots of technical safety risk that can be solved and should be solved and must be solved between two parties, and can be solved without focusing on these sort of ideological concerns. But unfortunately, some of those technicals are becoming ideological concerns. And one case in point is on the issue of sort of open weight models, right?
I mean, we really noticed that in a lot of the Chinese rhetoric, open source models or open weight models, open tools essentially, had become a sort of techno-political position. China appears to basically be leaning even harder into open source as a way to, one, to sort of gain global market adoption for Chinese technology, but also as a sort of geopolitical instrument. And what we’re basically seeing is that with increasing frequency, Chinese officials are sort of pointing to China’s provision of open models as evidence that China is acting in the common interest of global innovation and scientific discovery.
And then it’s contrasting that with the U.S. approach and essentially painting that closed approach as sort of monopolistic or, you know, problematically hegemonic or something like that. I’ll give you a quote here. I wrote down a little quote from the Ministry of Commerce, who was actually addressing, I thought this was pretty interesting, was actually sort of responding to U.S. accusations that Chinese models distill from U.S. models as we know that they do. And essentially what MOFCOM said was, you’re attacking us for distillation, but U.S. companies distill from our open models.
And they said, “Regarding artificial intelligence, China encourages open source, open collaboration, and sharing. China’s open source models are shared with global companies, including U.S. companies, facilitating the development of related models. Reports from U.S. companies on their model development also reveal that they extensively distill Chinese models. Conversely, the U.S. has repeatedly and unilaterally accused Chinese companies of engaging in industrial-scale distillation.” And I’ll come back to that distillation issue in a second.
But what’s interesting there is the politicization of open source, right? And we saw an even more significant and even more kind of vocal example of this in Minister Chen, the minister of MSS, his letter. He said, “Some countries, leveraging their accumulated advantages in basic artificial intelligence theories, model architectures, and core computing power, use the pretext of maintaining national security to restrict the introduction of cutting-edge technologies and equipment,” blah, blah, blah, blah, blah, “by means of entity lists, technology controls, monopolizing industry standards, and creating closed source ecosystems.”
That was the first time I ever saw closed source be likened to some kind of non-tariff barrier to trade, which is essentially what he’s arguing, that it’s this sort of like trade barrier to decide to close off your software, which is a little bit, which is a little ridiculous, of course. But at the same time, it’s not really the point of whether or not it’s correct. The point is, this is such a more politicized view of open source than we saw even a few years ago when the state was already broadly supportive of open technologies, but hadn’t sort of hitched its wagon to that position so thoroughly. Right?
Andrew: Well, and I actually wonder if this is frustrating to the Chinese labs, because I talked with Linghao, a member of your team before. You know, his position was sort of like the Chinese labs might go more towards closed weights down the road to increase their profitability. Right? So, this sort of you tying yourself so politically to this, I think, cuts off that avenue, probably for the Chinese models.
Kendra: A hundred thousand percent. A thousand percent. I think it’s actually going to create a real problem for the Chinese labs, right? So, it’s like a big financial strain to keep training open models, particularly because, as the compute requirements for sort of staying at the frontier, move ahead and move ahead, it becomes, you know, more and more of a strain. And Chinese labs keep doing it. I mean, I’m kind of arguing now that they’re kind of in an involutionary period.
They keep doing it because all their competitors are doing it. Whoever steps out of the open-source playing field in the Chinese market first is going to lose.
Andrew: Yeah, I don’t even know if you can politically at this point, right?
Kendra: Well, that’s the point. It’s like if you remove the politics from this issue in China entirely and you just let the labs do what they were going to do, the cycle they were in was they were all going to try to stay open as long as possible to drive their competitors out of business. And then they were probably going to start falling off with some of them closed sourcing and probably some consolidation, which many in the Chinese tech ecosystem have said is probably healthy, like a healthy place for the market to end up.
So right now, the decision to go closed in China, I think, again, without politics, is already a decision to kind of exit the front-runner race for the moment. Maybe not forever, but for right now, that means you’ve tapped out. You’ve been wrestled to the ground and you’ve tapped out. But that still would have been an easier decision if the state had not hitched China’s star to open source at this point, right? And so now, not only do you have to contend with, gosh, am I really ready to kind of step back from this? Can I find a closed-source path, maybe in a particular industry vertical or in a series of like, you know, using some kind of proprietary data or something like that that people are willing to pay for?
Now it’s a political position. And now an open source model needs to persist within China probably longer than it otherwise necessarily would have.
Andrew: Yeah. Well, I want to sort of circle back to something I raised earlier that we didn’t quite get to, but I think it’s related to all this, which is something that might change the calculus for the Chinese government towards open-weight models, towards the governance of open-weight models, towards like tying their masts to open-weight models, is if there is some kind of major security incident from an open-weight model that was released.
And then whatever happens may be beyond the control of the lab that released it. But as the U.S. labs are saying, like once you release it, it’s beyond your control. Can you talk to us about the risk of that and how that might impact how Chinese leaders think about like how they’re conceptualizing AI safety?
Kendra: Yeah, I mean, I think the primary thing to remember there is that nobody, not the U.S., not China, not the best minds on the planet who are all feverishly working to try to figure it out, have a foolproof global standard for AI safety checks. That has not been determined. So if our best and brightest minds don’t have it, the regulators definitely don’t. And so, there is a desire, both in the U.S. and China, for there to be some kind of safety testing mechanism, which regulators, both in the U.S. and China, have tried to sort of voluntold, you know, companies are getting like voluntold to go through quasi-governmental safety. It’s a bit of a mess, right?
Andrew: MM-hmm.
Kendra: But even if the state came in and said every single model that is released in China or the U.S., every single model that is released starting tomorrow morning has to go through a significant, stringent series of safeguard tests and they have to meet the following criteria before they can be put out, what are those tests and what’s that criteria? That just has not been decided. So, it’s a little bit of a circular problem, right? It’s like, yes, we all think there are risks here. We don’t 100% know how to mitigate them.
Even our best closed-model AI companies are experiencing misalignment issues with their models. So, I’ll put that on the table first. Even if regulators wanted to control it, they kind of couldn’t. What they could do is say, you can’t release open-weight models publicly. Or you cannot take an open-weight model, what a Chinese regulator could theoretically say is, “Labs, you may no longer release open weights above a certain power threshold on the public internet.” That would present a massive problem to China’s development potential and momentum right now.
China’s backing of open technology and the provision of open-weight models by Chinese labs, I would argue, is proving to be one of China’s most effective, not only market positions, market products, but also sort of geopolitical positions in recent memory. There is real global market demand for open models that is translating into real soft power for China. I would argue that is the first time that’s ever happened, particularly in a sort of frontier greenfield technology like this.
So, regulators are having to contend with the idea that, you know, do we really want to kill the goose that’s laying all these golden eggs? So, we already know that sort of behind closed doors, there is a mechanism by which the labs are being asked to verify or check these models with the state before they’re being released. It’s not an official mechanism. It’s not official regulation. There’s no clear guidance, but there’s some kind of checks and balances going on. But like I said, that certainly doesn’t mean that those models are safe because they can’t be because we don’t have the test for them because we don’t know what those tests are supposed to look like.
So, to your original point, the only thing in that scenario that would immediately and urgently cause the Chinese government to stop equivocating about whether or not global momentum is more important than safety or are these safety risks really that risky, and does that equate to that because they’re weighing the risks of we lose our opportunity to be sovereign in this technology forever because we fall so far behind the united states that we cannot kind of keep up versus we cause some major international safety incident that gets a bunch of fingers pointed at us.
Those two questions are very difficult to weigh, so if there was a major incident, obviously that will kind of likely result in some kind of knee-jerk response from the state to kind of shut it down for a minute.
Andrew: Well, and the second question I was going to ask you about this, which sort of relates to the AI dialogue between the U.S. and China, which we’ll get into more in a minute, that was prompted by a discussion from the Brookings podcast, the Brookings China team, Ryan Haas, Jonathan, Kyle Chan over there, great podcast, people should check it out. But I think it was Jon was mentioning, if there’s an incident from a Chinese open-weight model, I think the experience of the pandemic would suggest that the Chinese state may not be exactly transparent with the rest of the world on what happened, right?
And that would just compound the problems exponentially, I think. I don’t know if you have any thoughts on that.
Kendra: You have no idea what a good equivalence, I think, that is. And I’ll tell you why. Because China has taken a whole bunch of steps, just like during COVID, we’re going to we’re going to run with this analogy farther than I’m sure you’re comfortable with, but just like during COVID, right, China had its own ecosystem to control the spread of the virus internally. Its primary concern, its number one concern, was keeping its own population safe. So, whether that meant shutting the borders, quarantining people, whether they wanted to be quarantined or not, kind of rolling out these quarantine control systems that were pretty unwieldy, it instituted a domestic ecosystem of checks and balances and controls designed to keep its population safe.
And China’s domestic internet is also designed with many more technological checks and balances than the U.S. internet or most internets, right? The internet in most countries. There are controls on who can log into social networks. In the U.S., anybody can log into a social network and can anonymously set up an email account and anonymously sign in and set up a social media account and start posting content without verifying that you’re a real person, right?
Andrew: Yeah.
Kendra: Happens all the time. In China, you can’t do that. You have to show a ID, like a real ID, in order to get your account. So, there’s a sort of technological choke point there where an autonomous actor would have trouble acting in that network environment in a way that it would not have trouble causing quite as much havoc externally. So, I actually have had this thought. I mean, this is just something I’ve been percolating on for a couple of days. China’s networks, due to some of the censorship and control mechanisms that are in place in some of the network architectures, might actually kind of be safer from some of these attacks than international networks.
And therefore, I do think there’s a risk that if there was a major incident, China would just be like, closing the door.
Andrew: Yeah. Totally.
Kendra: Shutting down incoming flights. Like, we’re good in here, you know.
Andrew: Yep. Totally. I think this is a fundamental challenge with discussing AI safety with the Chinese.
Kendra: A hundred percent.
Andrew: Like I said, they have their definition, and it’s AI safety for us, not necessarily AI safety for all. And that is going to be a big challenge to break through. We’ll get into more of that when we circle back to the dialogue, but that’s all good points that I think are well taken.
Before we come back to the safety dialogue, though, I just wanted to pivot real quickly to another one of the issues that all these kind of public statements by various actors on the Chinese side have been zoning in on as well, which you have also highlighted a bit, which is the distillation issue. So, can you just talk to us a little bit more about how the Chinese are thinking about that, how they’re teeing up that as a, I guess, a back and forth between the U.S. and China.
Kendra: Yeah, well, I’ll recap it really quickly for anybody who doesn’t spend all day online reading about distillation disputes. So, Anthropic and OpenAI sort of recently accused Chinese labs, including DeepSeek and Moonshot and Alibaba and Zhipu and sort of all the main labs have undertaken these large-scale campaigns to extract data from higher quality U.S. models. In other words, what the labs do is they make tens of thousands of user accounts on, say, Claude or something like that. And then they ask Claude questions designed to extract sort of chains of reasoning or answers that will help them train their own models. And so there’s no dispute over whether or not that’s happening. It’s happening. Nobody has said that’s not happening.
The Chinese have kind of acknowledged that it’s happening and the U.S. labs have provided fairly significant evidence that it is happening. But the dispute there is how should that be treated under the law, right? Anthropic has kind of characterized it as industrial scale IP theft. And U.S. officials, and Scott Bessent in particular, have sort of signed on to that characterization, and they’ve threatened, I think he threatened on Twitter to sanction — still calling it Twitter. Oh, my God, I’m sorry — threatened on X to sanction Chinese labs for kind of doing that.
I think there’s an international debate around whether or not distillation constitutes IP theft or if it rises to the level of a national security concern that requires a sanction. I mean, that’s kind of, I don’t know if the law supports that. I mean, you could argue the law supports that. I’m not sure if that’s totally true. But it’s certainly true that it violates the terms of service of those, you know. I mean, Claude doesn’t allow that. In its terms of service, it says don’t do that, right? So, you’re not supposed to do that.
And the Chinese government has previously sort of defended distillation on the grounds that, listen, everybody does it. It’s a common AI industry practice. U.S. firms distill from Chinese models. I personally think those arguments are shaky because it’s like a government defending a contract violation. That’s a weird position to take. It’s true, U.S. firms do distill from Chinese models because Chinese open models have open licenses that say you can distill from them. So, there’s no contract violation. And like companies can set whatever conditions they want on their product.
It’s like closed is closed. So, I think the interesting piece of this is that China’s new position, which has emerged over the last couple of weeks of conversation, is that the law should say that banning distillation is an unfair trading term, which is wild. They’re not the only ones that have made that argument. There are quite a few people in the U.S. tech community who have also said you should probably just allow distillate, right? to prevent bans on distillation because what they’re doing, I mean, that shared human knowledge, much of which wasn’t paid for.
That was like scraped from the open internet in many cases. So, saying that that’s yours, right, there’s all this kind of debate about whose data is that. Anyway, MOFCOM explicitly said, “We’ve noted that some USAI companies are abusing their competitive advantage by including broad geographical restrictions and other unfair terms in their user agreements.” In other words, they’re blocking Chinese users from logging in, right? “The U.S. accusations regarding distillation appear to endorse these unfair trading terms. The U.S. approach is a typical example of using the pretext of combating distillation to exercise industrial monopoly.”
Now, that argument is still very tenuous. Of course, if you think a trade term is unfair, you should litigate it. You don’t just blow over the top of it, right? I mean, I think that’s the rules we’ve kind of…
Andrew: I don’t know.
Kendra: I don’t know. I mean, put it this way. I think if the rules were reversed, if the U.S. was doing that to Chinese companies, I don’t think Chinese regulators would be quite as fond of the practice as they seem to be going in this direction. But the rightness or wrongness isn’t the point I’m trying to make. The point I’m trying to make is the Chinese official discourse is now kind of starting to tie distillation to a broader narrative of the U.S. tries to prevent distillation because it is engaging in technological hegemony, which is a very interesting narrative. It’s an interesting point to take. So, I’m curious to see if that point sticks around.
Andrew: Well, I mean, I think, you know, the smart part about that is that it feeds into the broader Chinese positioning of we’re creating an open technology for the world to use.
Kendra: Right.
Andrew: And I mean that I’ve talked with Eric Olander on the pod. That’s very clearly the approach. We want the rest of the world to take up our technology because it’s open and it’s cheap and then we’ll figure out the business model afterwards. But we want to ensconce ourselves in the sort of AI foundational layer, AI supply chain, whatever you want to call it. And then we’ll figure it out from there. I mean, it’s not a bad strategy.
Kendra: It’s working. Actually, what’s really interesting about that is that when Chinese regulators first started supporting open source, it was like a fringe idea, really. It was sort of a fringe policy idea. There has been general support for open source tech, particularly open source tech that could help China overcome U.S. dominance in certain technological areas for many years. And there are many, a few, probably three or four, really, standout Chinese open source projects. There’s an open source blockchain network that China has kind of developed. There are some open-source semiconductor, kind of firmware, basically, like chipset architectures. I think there’s even a couple of open databases.
There’s OpenHarmony, the mobile operating system that was originally developed by Huawei and that, I think, donated to an open-source foundation. So, there’s been a couple of like small projects that have gained some ground over the years. We’ve watched them five, six years, kind of seen them take a couple of steps here and there. But nothing like this. The thing is that all of those open source projects were competing in an arena where there was already a dominant U.S. player that had essentially taken over the stack.
If you’re working with mobile operating systems, you’re still working with either Android or iOS for the most part or some fork thereof. You’re still mostly on U.S. origin chipset architectures. There really isn’t that much room left. But here we have it. Last month, Hugging Face published new statistics on how many derivative models, models that people kind of spun off, were based on Chinese models versus U.S. models versus EU models, etc. 70% of derivative models globally are based on Chinese open models.
So, it’s actually working, which is part of the reason why I think they’re getting so… you know, now it’s like, oh, my gosh, this thing that we kind of did on the side, didn’t really care about, didn’t really think about has suddenly kind of taken…
Andrew: It’s finally happening.
Kendra: Yeah, and it’s happening. That’s exactly it. That’s exactly it.
Andrew: Amazing. Well, I mean, that’ll definitely shape the back-and-forth for sure between not only the governments, but the labs going forward. I do want to pivot to the AI safety dialogue, to the extent that, or the AI dialogue at all, to the extent will exist. I think, as a starting point, I think you made the keen observation that basically China’s starting point is that the U.S. is going to have to pick a lane, which is we can prioritize AI safety, or you can prioritize AI containment of China.
But the coordination on AI safety or AI containment of China, but the two together are largely incompatible. Why don’t you walk us through? We’ve touched on it a little bit, but walk us through kind of what you see going on there.
Kendra: I mean, this is the message that I think a lot of our listenership and particularly anybody who’s worked in an MNC in China will be extremely familiar with. China sends this very consistent message to the U.S. government, also to U.S. companies and U.S. companies doing business in China: you can enjoy the benefits of China’s sort of manufacturing ecosystem and large consumer market. Or you can advocate for measures to contain China, but you don’t get to do both of those things.
You’re going to get slapped with some kind of retaliatory penalty if you try to do both of those things. You can enjoy a good relationship with China and your Chinese regulators, or you can have Taiwan listed as the separate company on your website, but you cannot do both of those things. You need to pick one, right? And so, they’ve kind of been fairly successful, I think, at sort of laying that, it’s like using that strategy. I think that type of messaging is essentially now being extended into the AI space. And I’ll quote actually Global Times, who once again put it quite succinctly, talking about Anthropic here.
“On the one hand, Amadei acknowledges that global efforts to pace AI development would ultimately require cooperation with China. On the other hand, he advocates for restrictions on chips, computing power, and models to slow China’s AI development, while explicitly seeking to widen the U.S. technological lead over China.” So, the message there is pretty clear, right? It’s like U.S. labs and officials have to choose which goal they want to prioritize. If they think that cooperation with China is the fastest path to ensure a safer world for all, then they need to cool it on the rhetoric.
And if they think that advocating for China containment measures is a sort of swifter path to safety, then they should just abandon any hope that there’s going to be any kind of cooperation, significant cooperation on safety in one way or the other.
Andrew: I think that’s a fairly reasonable standpoint from China, to be honest. Again, people will beat me up for saying that kind of thing, but from a logic standpoint, it holds up. I would just say, though, the obvious question from that is the key question, which is, given those realities, is there really any path forward for cooperation between the U.S. and China on AI safety?
Kendra: I think there is. And as I said earlier, I think the issue is really to… I hate to just push a narrative that China sort of outlined. And that’s not what I’m trying to do. But I do think that we’re not going to solve the U.S.-China tension. We’re not going to solve capitalism versus communism. We’re not going to solve every market issue we’ve ever had with Chinese companies’ behavior in the global market. Those issues have been ongoing for 20 years. Right? But the technicals of AI safety are solvable in the near term.
The issues around where does risk actually sit? Where does liability actually sit? When do we need to call you when there’s been some kind of safety incident? At what point do we need to notify you or through what kind of channels do we need to discuss when the AI that we’re developing has done something scary? What does scary mean? A lot of the time what we’re talking about, well, this week anyway, is recursive self-improvement, which is models that build themselves and improve themselves.
That’s this kind of threshold that we’re running towards where it’s not human developers making models better and releasing them. It’s models sitting in a box deciding how to improve themselves over and over and over so rapidly that we don’t know what is happening anymore. Do we need a channel? What are the markers or technical engineering measurements at which we would say, “Uh-oh, the model has done this. We need to now, we need to talk to you or shut off this training pathway.” So, those kind of conversations, if we could keep the other stuff out of them, which I’m not particularly optimistic about, but if we could keep the other stuff out of them, I think that is actually a pathway we could choose to pursue.
Andrew: Okay, where does that leave us then? I mean, we’ve got this meeting that’s supposed to happen in November to kick off the AI dialogue, but is any kind of cooperation dead in the water in your view, or should we have some kind of optimism that the two can figure it out?
Kendra: Oh, I always hate these questions because it’s like I feel both optimistic and pessimistic about it at the same time. Again, I do actually think here and maybe now it’s my turn to be naive, but I do actually think there’s a path forward. And that path forward is that we focus on the specific things that we want to resolve on the technical level.
Let me give maybe a more concrete example. Instead of the U.S. messaging that China developing open-weights, China should stop developing and releasing open weight models because China’s development and release of open weight models is dangerous and irresponsible and China should stop, blah, blah, blah, blah, blah. Maybe we have those conversations focused on the potential for those models to be leveraged for nefarious purposes without oversight, which is to your point earlier, right? That’s a loss of control issue for the Chinese state, right?
Just as much as it is for us, for the development of bioweapons, for cyber attacks, right? For like loss of control issues. So, if you focus on instituting, if you press China to institute domestic effective regulations on those particular threats for models, right? Because a bioterror threat is just as scary in China as it is anywhere else, right? Within the boundaries of its own existing sort of digital regulatory regime, that’s probably a more actionable message that will push regulation in China forward than sort of push regulators to institute controls on those models, right?
Then sort of focusing on the method by which the model is being proliferated, basically saying, you know, you shouldn’t do open weights is, like we said, kind of off the table unless there’s an incident, or at least for right now. If you focus on the technicals, you could even make room for really interesting discussions, right? Really interesting pathways forward that maybe regulators don’t even need to be involved. The government doesn’t even need to be involved. What if, for example, I kind of thought, NVIDIA is about to buy Hugging Face. They’ve already announced they’re buying Hugging Face. That’s the model distribution platform.
Why don’t we run safety checks that every single model that gets uploaded to that platform, regardless of origin, receives? You have a bunch of engineers all going through this one single portal. I think, if you really care about AI safety and not the geopolitics, there are actually ways, I think, to push safety forward in China, even if China is a little bit reluctant to cooperate. On the other hand, China will have to enter those meetings and have to fight against its own lack of transparency.
It’s really hard to cooperate on technicals even if China is unwilling to share the reality of its own technicals with the U.S., which it will be very reluctant to do that. So, I think the technicals are probably the best path forward, but I don’t think…
Andrew: Even that’s not easy.
Kendra: Even that’s not an easy task, yeah.
Andrew: Well, I guess we shall see. Something tells me that this is a conversation that we will be having on an ongoing basis until humanity is extinct. So, it could just be a couple more months.
Kendra: I, for one, look forward to the nap.
Andrew: Yes, it’ll be… At least we can all rest.
Kendra: Yeah.
Andrew: Well, Kendra, great to see you. Great to have you back on the pod. Thanks for enlightening us all on this. It’s been super helpful. I really appreciate it.
Kendra: Yeah, good to talk to you.
Andrew: And thanks, everybody, for listening. We’ll see you next time. Bye, everybody.
