
ジュリアン・ゲヴィルツが語る、北京が本当に恐れるAIリスク
Julian Gewirtz on the AI risks that actually scare Beijing

ジュリアン・ゲヴィルツが語る、北京が本当に恐れるAIリスク
80,000 Hours Podcast
要約
元NSCの中国・台湾担当上級部長で歴史家のジュリアン・ゲヴィルツが、中国共産党がAIリスクをどう捉えているかを語った回。党の最優先は体制の安定であり、「制御喪失」には政治的統制の喪失も含まれると述べる。北京の姿勢を変えうるのは国内のリスク認識と、米国の国内規制という「コストのかかるシグナル」だとし、米中合意は理想設計ではなく自国利益に基づく並行行動から漸進的に積み上げるべきだと主張した。
この回のポイント
- ●中国共産党の最大の関心は体制の安定であり、中国語圏の「極端な制御喪失リスク」は、AIへの人間の制御だけでなく政治的統制の喪失も意味するとゲヴィルツは述べる。
- ●過去1年ほどで北京のAIリスク認識を最も動かした出来事は、Anthropicの Claude Mythos だったという見方を示し、米国のサイバー能力への長年の懸念が強まったとする。
- ●北京の姿勢を変える要因は、第一に中国国内のリスク認識、第二に米国が国内でAI規制を行う「コストのかかるシグナル」であり、特定CEOの発信ではないという。
- ●中国のAI企業の多くはAGIを信じているが、政府は距離があったと語る。ただし変化しつつあり、兆候は演説ではなく計算資源の集中やNvidia製チップの購入拡大といった物的な動きに現れるとみる。
- ●米中協議の最初の障害は用語で、中国語では safety と security が同じ「安全」になる点などが例に挙がる。合意は、制御喪失を避けたいという双方の利益から並行行動を積み上げる形が現実的だと述べる。
- ●中国国内でオープンウェイトのリスクを論じる声が出始めており、党は安定を脅かすと判断すれば方針を素早く変えられるとみる。
章立て
北京はAnthropicをどう見るか
中国の国営メディアはAnthropicを米国の技術覇権維持の一部として批判している。個々のCEOの発信では見方は変わらず、米国内規制が重要だと語る。
中国版Hugging Face事件と体制の安全
インシデントはまずサイバーセキュリティ問題として扱われ、情報公開のインセンティブは乏しい。党の最優先は体制安定で、Mythosが北京の認識に与えた影響にも触れる。
蒸留問題と米中のラボの違い
DeepSeekなどへのリクエストがClaudeに転送されていた報告への北京の反応や、米中ラボの安全への姿勢の差を論じる。
習近平とAGI、RSIの見極め方
AIの急加速を党がどう受け止めるかを議論する。RSIへの信念の有無は、計算資源の集中やチップ購入などの物的な動きで判断できるという。
中国のAIラボと台湾、対米関係
ラボが政府の過剰な介入に不満を持つ実態や、台湾は AI と無関係に長年の優先課題だという点、米中関係の悪化要因を語る。
米中AI合意の障害と現実的な道筋
用語の違い、近未来リスクとフロンティアリスクの論点、検証技術の課題を挙げ、自国利益に基づく並行行動で積み上げる方針を述べる。
オープンソース路線の転換の兆し
中国でオープンウェイトのリスクを論じる声が出始めている。地政学的利益との綱引きが続く。
中国専門家の見分け方と推薦図書
謙虚さを専門家の指標とし、推薦する研究者や書籍、AIモデルの使い方、グローバルサウスで中国の影響を見るべき地域を紹介する。
解説記事
今回の80,000 Hours Podcastでは、歴史家でバイデン政権の国家安全保障会議(NSC)で中国・台湾担当の上級部長を務めたジュリアン・ゲヴィルツが、北京がAIリスクをどう受け止めているかを語った。聞き手はトム・リード。録音は2026年10月2日。ゲヴィルツは「中国とは何か、共産党が何を気にかけているか」から出発すべきだと繰り返し、理想の合意を逆算する発想を戒めている。
北京の視点では「制御喪失」の意味が違う
ゲヴィルツによれば、共産党の最大の関心は体制の安全と安定である。中国のAI議論では「極端な制御喪失リスク」への注目が高まっているが、その意味はAI安全の研究者が言う「人間によるAIの制御喪失」にとどまらず、AIに関連した政治的統制の喪失も含むという。情報空間が厳しく管理された国で、AIが意図せず、あるいは意図的に情報の流通を崩すことが懸念されるわけだ。
ここに米国の要因が重なる。ゲヴィルツは、過去1年ほどで北京のAIリスク認識を最も動かした出来事はAnthropicのClaude Mythosだったとの見方を示す。米国が並外れたサイバー能力を得つつあるという感覚である。党や政府の場でAIリスクが語られるとき、米国は脅威の一つとして挙げられるという。公式発言や非公式対話から影響の大きさがうかがえるとも述べている。
北京は何で動くのか
番組冒頭の話題は、習近平がAnthropicとダリオ・アモデイをどう見ているかだった。ゲヴィルツは推測は不要だと言う。中国の国営メディアは「Anthropicは米国病に罹った」などと繰り返し批判し、米国の各社を互いに競う存在ではなく、中国を抑え込もうとする一つの米国の企てとみなしている。そのため特定のCEOの発信を工夫しても北京の見方は変わりにくいという。
彼が重視するのは二点である。第一は中国国内でのリスク認識の深まり、第二は米国が実際にAI規制を行うことで示す「コストのかかるシグナル」だ。米国側が「永続的な優位」といった言葉を使うことは、北京を遠ざけるとも指摘する。また中国自身もディープフェイク表示規則などを導入しており、規制が競争力を損なうという発想は誤りだと述べる。
見えない中国国内と、兆候の読み方
中国のAI企業の多くはAGI(汎用人工知能)が近いと信じているが、政府はそうした見方に距離を置いてきた。ゲヴィルツは、それも変わりつつあるとみる。ただし党内の意見は報告として上に上がるもので、外からはほとんど見えない。そこで彼は、発言ではなく物的な動きで判断すべきだと言う。計算資源の集中や、現在は抑えているNvidiaチップの購入拡大などだ。丁薛祥が外国製チップの購入を戒めたと報じられたことも、党が国産依存を優先している例として挙げた。
また、中国版Hugging Face事件が起きても、通報者保護の弱さや情報非公開のインセンティブから、外部が知ることは難しいとみる。企業が罰を恐れて隠す可能性も指摘した。
米中合意は「並行行動」から
米中協議の現実について、ゲヴィルツの口調は慎重だ。最初の躓きは用語になるとして、中国語では safety も security も同じ「安全」になる例を挙げる。次に、近未来リスクとRSI(再帰的自己改善)など最前線のリスクのどちらを論じるかでも対立が生じうる。北京にとって最も魅力的なのは米国の一方的な自制であり、自国の発展への制約は受け入れにくい。検証技術も課題で、計算量やデータセンターの所在、エネルギー使用などを技術的に確認できる手段が必要だと述べた。
そこで彼とマット・シーハンが提唱するのは、理想の枠組みを設計する代わりに、各国が自国の利益から並行して動ける領域を積み上げる方法だ。核になるのは、どちらの国もAIシステムの制御を失いたくないという点である。理想を描くと「不可能性が至る所で立ちはだかる」が、すぐ着手できる段階的な措置なら取り組めるという。米国側の交渉力強化としては、国内規制、輸出管理、外交の三本柱に、レアアース依存の解消などAI以外の要素を加えるべきだとする。
オープンウェイトを巡る変化と専門性
録音時点で、中国国内でオープンウェイトやオープンソースのリスクを論じる声が出始めているという。新しいAI安全ガバナンス枠組みには、オープンソースが高度なサイバー攻撃の障壁を下げるとの記述があり、学者や論客の間にも無制限の開放への懸念がある。一方で、米国企業に対抗し影響力を広げ、限られた計算資源でも成果を出せるという地政学上の利点もあり、両者のせめぎ合いが続く。党が安定への脅威とみなせば方針を急速に変えられるとも述べた。
後半では中国専門家の見分け方にも触れ、答えを確信をもって語る人に警戒し、不確実性の中で判断する謙虚さを第一の指標に挙げた。推薦先としてシーハン、グラハム・ウェブスター、CSETの研究者らを挙げ、ヤオ・ウェンレイの『The Gilded Cage』なども紹介した。
まとめ
日本の読者にとって重要なのは、「AI安全」という言葉が米国と中国で同じ意味を持つとは限らない点である。中国の「制御喪失」には政治的統制の喪失が含まれ、米国のサイバー能力への警戒とも結びついている。国際ルール作りやAIガバナンスを考えるとき、相手の動機から出発し、用語を擦り合わせ、小さな並行行動を積み上げるというゲヴィルツの現実的な姿勢は参考になる。今後は、中国の計算資源政策やオープンウェイトを巡る議論の変化といった物的・制度的な兆候が注目点になる。
文字起こし(英語・自動生成)
We have to start with the reality of what China is and what the Chinese Communist Party cares about, as opposed to kind of working backwards from the place we think they need to get in order to accomplish our broader goals. The number one concern is regime security, regime stability. As I have seen the Chinese AI discourse evolve, there is increasing attention to what they call extreme loss of control risks, the most significant event in shaping the evolution of Beijing's thinking about AI risks, you know, over the course of the past year or so, is mythos. Today, I'm speaking with Julian Gewurz. Julian is a historian, poet, former director for China affairs in the NSC under Biden, and more recently, also a blogger. Of all the things that I've learned from reading Julian's work. Perhaps my favorite is that much of the credit we all usually give to Deng Xiaoping for the reform of China's economy
should instead be redirected towards the comparatively uncelebrated former Chinese premier, Zhao Zhang. Julian, thank you for coming on the show. Thanks, Tom. It's great to be with you. So, Julian, I have a bunch of not-so-hidden agendas for speaking to you. Above all, I want to understand what on earth do you think Xi Jinping's model of Dario Amo Day is? Like, what do you think he thinks this guy wants? Honestly, I don't think we have to speculate because to a stunning degree, Beijing has been on the record going after anthropic. It's actually in the history of sort of U.S.-China relations, it's hard for me to think of another example of an American company that Beijing has taken on so repeatedly and so directly. There have been a series of state media commentaries about it. And, you know, there was one head of Xi Jinping coming to the United States that I found really striking and I wrote a little bit about.
But the headline was Anthropic has contracted the American disease. Now, what does that mean? The core argument was basically that this is a company that says, you know, whatever, whatever, but it's actually all about preserving so-called technological hegemony for the United States. And crucially, you know, we often think about these different AI companies, the different model companies, as very differentiated actors in our system. We know that many of them don't like each other and that they're competing intensely and all of those sort of things that, you know, define what it feels like to be in this space. Beijing said Anthropic is, you know, literally, quote, a monster bred by the American path, saying like all of this, all of these different firms, this is all part of one big thing. And that big thing is the United States trying to, you know, dominate globally and trying to keep China behind.
And they have been going at anthropic hardest, but they don't, I think, see it as somehow, you know, separate or outside of a mainstream in the United States, but very much a part of it. and it's just maybe worth driving home, the upshot of that, which is that for Beijing it appears that a lot of the, you know, distinctions or differences in views among American tech titans and American AI companies matter less than what they represent to China, which is the project of American power in their eyes. Well, yeah, so I'm thinking sometimes I get confused by Dario's comms on China because sometimes it sounds like he's calling like, oh, we really need to cooperate with them. Sometimes it sounds like what he's calling for looks a little bit less like cooperation. Dario famously, I think he only has one direct report, which is his chief of staff. But let's imagine you were his second ever direct report and he'd hired you specifically as his like China comms person.
What would you tell him to say? How should he be talking about China differently in a way that actually increases the chances of them taking his calls seriously rather than as a call for slowing China down compared to the US? Honestly, I don't think it's very likely that Beijing can somehow be disabused of the notion that the United States wants a significant lead in AI over China. I mean, and I don't think that messaging from any particular executive is going to be the key thing there. I mean, let's just step back for a second and say, you know, we're in a moment where the president of the United States has said, you know, no slowdown, you know, AI risk, not something I want to take seriously. sure, and where the Secretary of the Treasury has said, you know, there's no day after tomorrow if China wins at this. So I don't think, I mean, to the point I was just making, I actually don't
think that they see the, you know, rhetoric or comms of one CEO, Dario or otherwise, as like the thing that could shift their view. My own strong view is that the most important thing that could cause Beijing to shift its current approach to AI safety is their own domestic sense of the risks. The second most important thing is if they were to see the United States actually doing AI regulation and taking AI safety seriously, doing, you know, basically what we would call a costly signal the kind of thing that we're doing because we care about these risks and that also has the benefit of signaling to Beijing like, hey, these guys are serious about this. Those two things are far more important, I think, than, you know, a shift in comms. I do, I will say, I do think that there is a difference between the United States wanting,
you know, to compete and have a lead and a narrative of total, you know, supremacy, permanent dominance. So the language I think is least helpful toward getting Beijing there is the language that suggests, you know, that what the United States wants here is not just a lead, but like total kind of, you know, what we would call primacy. And there is a distinction there, and I think it is relevant, but all part of a puzzle. So the idea being like a precondition for China taking any of the U.S.'s sort of apparent AI safety concerns seriously would be actually regulate AI domestically. Otherwise, it just looks like it's all talk. Yeah, and obviously that gets us right into the core question of trade-offs around, you know, regulating at home to preserve our edge, you know, regulating at home with the risk of
losing our edge over China. But there, you know, my friend and collaborator Matt Sheehan has a really good point, which is that Beijing has itself done some AI regulation at home, not sort of frontier AI risk, but, you know, they have deepfake and, you know, AI-generated content labeling requirements. They have regulations on chatbots, particularly for minors. They've done some stuff there. And of course, we don't see that as condemning them to be in second place necessarily. In fact, they're doing all that and we're still very worried about them catching up to us. So I think there's a bit of a fallacy embedded in the idea that any AI regulation is at odds. Going to lose us the whole race. Exactly. Now, pacing is a separate question because that is explicitly about the speed of frontier development. But regulation is a lot more than just pacing, you know. How do you think the Chinese government would react to a homegrown version of the Hugging Face incident?
Now, in China, I think an incident like that would be seen as a cybersecurity incident first and foremost. And the thing to remember in China is that while the AI regulatory space, and particularly the frontier AI regulatory space, is very much still coming together, and we'll talk about that, I'm sure, the cybersecurity structures have been in place for a while, and this is the top kind of regime security issue for Beijing. So I think they treat it that way. They do have certain kind of notification requirements where I think the central government could potentially become aware of it pretty quickly. But the most important thing that I think we have to stress is that at every layer, the incentives in that system are not to disclose more information and certainly not to disclose it publicly. So the most remarkable thing, as we imagine what a kind of hugging face, open AI style incident in China would look like, is the public, I don't think, would know about it.
We might never know about it. And I think that's part of the reason why there's so much concern as we think about these issues, because, you know, in the Chinese political system, you know, whistleblower protections, shall we say, are not something that gets gets treated with much seriousness. The famous Chinese whistleblowers are people like the doctor in Wuhan who warned about the spread of COVID and, you know, was punished for that years ago. So I think anyone at a firm, whether an executive wanting to report up the chain or a whistleblower kind of considering whether to make their concerns known, would be weighing the risk of serious kind of smackdown by the government against the safety risks. And so that's a pretty challenging place to be. Interesting. Do you think the government would be fully informed of what happened? Or do you think the labs themselves would also be incentivized and capable of hiding such an incident if they became aware of it?
I'm just trying to understand exactly who would know if something like this happened. Well, in an incident like that, where what happened sort of fully escaped the lab's control and was a third party website, You know, I think presumably the government would find out that something had occurred at least via the third party website. And if there was an investigation, you know, I think some of the labs might be more forthcoming than others. There is a small chance you could imagine that if they felt they really kind of handled it, and or if it became publicly known that there had been some sort of hack of this third party website for, you know, other reasons that were visible. I think you could then imagine them doing a very limited kind of disclosure saying, there was a problem, we found out about it, we solved it, everything is good, proceed accordingly. I think the biggest policy question in Beijing in that kind of moment would be, would something like that trigger what they would call a broader rectification campaign,
meaning an effort to bring greater control from the central authorities over this area. And that, I think, is possible. But, again, it would likely be in a context without the kind of public disclosure that eventually marked the OpenAI Hugging Base incident. And crucially, you know, that meter report, the things that have brought a bit more public confidence back to thinking that at least third party evaluators have the capacity to inform the public about this. I think highly, highly unlikely in the Chinese case if that were to happen. Do you think is there a particular flavor of near term AI incident that you think would be more likely to call something like a broader rectification campaign? campaign? Like, is there, what I'm trying to think of is, is there some kinds of ways that AI could go rogue that would be more worrying to the CCP than others, or that might encourage them to view things as not just any other, you know, familiar cyber incident? We have to start with the reality
of what China is and what the Chinese Communist Party cares about, as opposed to kind of working backwards from the place we think they need to get in order to accomplish our broader goals. And I I say that because the number one concern is regime security, regime stability, what they call political security, and social stability. So the kinds of risks that I think are most likely to cause them to recalibrate some of their current approach to the AI sector are really in that domain. They do want to avoid risks to regime security, also sort of major economic disruptions. So, you know, from that perspective, I think as we consider the kinds of scenarios that might unfold, it's really the ones that are in that core political and regime security space. And as your listeners well know, there are those risks.
And that, in fact, control over information, particularly information about the government, about the Communist Party, access to, you know, a very, very carefully controlled cyber and information ecosystem. That is the kind of thing that we can imagine, you know, these new new tools, new technologies disrupting either inadvertently via loss of control by a company or deliberately. And I think the deliberate piece of this is a big element to just bring in because, candidly, as I've seen the Chinese AI discourse evolve, there is increasing attention to what they call extreme loss of control risks. But we have to remember that when they use that phrase, loss of control, they don't simply mean it in the way that a lot of AI safety folks here mean it, you know, loss of human control over AI systems. but also as the loss of political control linked to AI more generally.
And so when we think of, you know, deliberate hacking, you know, deliberate dissemination of information that the government doesn't want spread, those are all the kind of risks I think that are likely in the first instance to make them really start to reconsider. and sorry, maybe just one last thing, but there was recently this vulnerability exposed. Oh, this is the hack of WeChat? Yeah. Exactly. The vulnerability in WeChat, which is the sort of one app to rule them all in China. So just to clarify for our audience, this was a hack of WeChat that was disclosed by a group that had used AI to find a vulnerability. They found a vulnerability, but they disclosed it responsibly. But we have no idea exactly how the government sort of processed this information. We just know that it happened and they would be aware of it. Right. And sort of in line with what we were just saying a second ago, the response of Tencent, the company that created WeChat, was to say, you know, we patched the vulnerability.
Yeah. But the shoe that never dropped publicly was how the government saw that and what kind of conversations about the vulnerability there were. But there was just one other thing, which I should have said in the prior kind of set of comments, which is the U.S. factor here is major. And I think in a way the most significant event in shaping the evolution of Beijing's thinking about AI risks, you know, over the course of the past year or so is mythos. and the kind of sense that the United States is gaining extraordinary capabilities in cyber and that that is motivating a set of, you know, counter responses that we have very little visibility into, but that they see as part of the AI risk conversation centrally. And it's whenever they talk about AI risk at the government level or the party level, they talk about the United States as one of the threats.
So the main risk for them is U.S. privacy through AI advantages or something like that? I still think loss of political control, political security is number one, but a very close second is the U.S. dimension, exactly as you say, and crucially, they're connected. And how do we know that Mythos was like this sort of wake-up call for them? I've heard this before, but how is it that we know this is the case? They've not said, oh, this was a wake-up call for us. Are there signs that Mythos shifted things for the CCP? Yeah, no, I don't think any government would call a press conference and say we're really freaked out about this. But no, there are a few things. First, there are indications in official statements where they have quite directly invoked it. So, like, they have spoken about it a little bit. They don't say, you know, we've turned up the alarm to the highest level, but they acknowledge the reality of it,
and they acknowledge it as something that could pose a kind of security set of challenges for China. And then second, you know, in kind of track two dialogues and other settings where, you know, there's communication, it's been quite clear that this is something that has reverberated profoundly in their system. And in general, right, to go back to that kind of initial point of starting from what China is and what the party cares about, you know, their concern about American cyber capabilities and all the rest has been a huge issue for well over a decade. Xi Jinping has spoken about this frequently over the years. So they are primed to see that as a threat vector of the greatest significance. Can I force you to make a guess about something? Sure. Well, you're not going to necessarily be able to force me, but you can try. I can try. I can try. I can try. To clarify for our audience, I should say, if you hadn't heard about this, there was a
report released by Anthropic where they discussed that requests from users to kidney models or models from DeepSeq were being rerouted to Claude. And some of these even included requests from, I think, a PLA official, even in one instance. So like really sensitive things that people were using, what they thought they were using a Chinese model for, but was in fact being sent to Claude. What I'd love to know is, what do you think the CCP officials who spoke to the CEOs of Moonshot and DeepSeq about the data leaks to Claude, what do you think that conversation was about? So the first thing I'd say is I wonder at what level of technical sophistication that conversation happens, because there's a version of that conversation, you know, in our kind of idealized version of generally globally, how governments and private sectors should interact that could be pretty technically sophisticated. But I think it's fairly unlikely that that was the only kind of conversation that happened.
I think it's more likely that there was a conversation about this. Maybe that did occur at a technical level or they sent in, you know, people from the cyberspace administration or one of the real, you know, AI technical bodies that does exist to understand how this happened. But the political dimensions of that, the idea that it embarrassed China and embarrassed these companies had to have been front and center. And that's the kind of thing that would probably happen somewhat distinct from the more technical conversation about, well, what was actually revealed and how did that actually happen? I think we all had the reaction reading the Anthropic Report that, like, first, this is crazy, and it is a reminder of how, you know, the incredibly fast-moving Chinese efforts to catch up have led to a degree in some areas of sloppiness that, you know, is certainly not what a perfectly sensually planned version of an AI race from Beijing's perspective would look like.
And it is embarrassing. I mean, it's deeply embarrassing for them. And the broader point about distillation and dependence on American models, though, I think sort of we were talking just a second ago about some political dimensions of that conversation. it's important to underscore that Beijing's line on distillation is not, it doesn't happen and, you know, how dare you. It is precisely, actually, that this is, you know, normal practice. Fair game. The United States is trying to twist this into something it's not, et cetera, et cetera. And I was also quite struck that they connected that report. Then when Dario came out with the call to pace the frontier and some of the China language there, one of the core points I always make about China is that they connect things, connect dots in our system that we may not connect ourselves.
And candidly, I think we do the same sometimes in their system. We can come back to that. But they connected in very official, authoritative state media that report that we were just talking about, including those really embarrassing disclosures with the calls to pace the frontier, which included language about how, you know, the need for the U.S. to stay ahead of China. And they said, see, these are all part of the same initiative, the same effort to make Chinese AI look bad and keep the United States number one. And candidly, I don't think too many people in the United States connected those. I'm not even sure that our, you know, that the anthropic necessarily meant them as part of some kind of coordinated thing. I don't know. But they saw it that way. And it's just a reminder of that core dynamic. Yeah, yeah, that makes sense. So it's still deeply unclear how either the U.S. government or the Chinese government might relate to very rapid increase in AI capabilities.
How do you think about the difference between how the American labs versus the Chinese AI labs think about AI risk? Yeah, it's such an important question, you know, because that distinction in each country between the governments and the labs is actually something that can sometimes get overlooked by either player. China often talks about the United States ecosystem as just one monolithic thing. Obviously, Americans often sometimes talk about the Chinese ecosystem in the same way. And over the summer, my friend Dan Wong and I wrote an essay in the New York Times that was arguing essentially that we needed to focus more on the competition for influence and bargaining and dynamics within the United States and China, respectively, than we do currently, where the focus is largely on the kind of race between the two. And obviously, that's become pretty salient in the months since, you know, we wrote about that, which was right around the time of the U.S. government and anthropic dispute.
I guess what I would say there is, to my mind, the key difference is that the American labs, some of them at least, have a demonstrated track record of taking safety somewhat seriously and doing certain things to disclose to the public and, you know, to even in some cases constrain, you know, their own racing forward. I think it's nowhere near what many of us would like to see, but there is something of a track record there. And candidly, we don't have that on the Chinese side. I think when people talk about, you know, these questions of like, where is the risk greater these days? The U.S. government certainly doesn't seem like it's being particularly responsible. but I think there is reason to say that some of the American labs at least, you know, have behaved in a, you know, way that is a little bit more attuned to the safety concerns than some of the
Chinese labs. Now, do you ever entertain cynicism, like cynicism toward the American labs? Like you take this. Do I? Absolutely. Yeah. But why doesn't that make you more bearish? Or like what gives you the confidence, okay, it's not just, you know. I have an opinion too for what it's worth. Yeah, I mean, I guess what I'd say is, like, I'm not terribly optimistic here. I don't want to convey the wrong impression that I think somehow the American labs are, like, you know, highly responsible stewards of the fate of human civilization. But I do think that the incentives that they have had, at least so far, to disclose incidents, very low bar. not a bar that the Chinese labs have cleared. So I don't want, I actually, I would like frame this less as the, you know, all the things that we know they need to be doing and are not doing and just look at the lowest possible bar of disclosure. And that's one fundamental difference.
And so I think you know in both countries the question of how the relationship between government regulation and lab self plays out is a huge part of everything that we discussing today It going to play out differently in these two countries. Like, this is not going to be symmetrical for very, very obvious reasons. But if we're just looking at the labs, some of those asymmetries, I think, you know, aren't all of AdStory for America. Have you ever had the chance to speak to people at Chinese AI labs? Yeah, to some. I mean, look, I'm a former U.S. government official. Like, I'm not somebody who I think Chinese labs have a... Speak candidly to, yeah. I don't know about that. I mean, I think the most striking conversations that I've had with folks at Chinese labs were around the kind of early wave of discussion of pacing, you know, sort of when the lab employees letter came out in the United States. And I asked a bunch
of these folks how they interpreted that. And one person said to me in a comment that I found very, very striking, you know, the United States has been pacing the frontier for China for years. So we don't see this as really anything new about us. And we don't think the United States is going to slow down. So they have a healthy degree of cynicism. And obviously, there's a degree of performance in that, like they're performing for a certain kind of American audience. But on that point, at least, I think that's a credible position that likely many of them do believe. I also want to understand a bit more how the CCP might respond to blisteringly fast AI progress, if it does, if it does happen. And I'd like to know, is there anything that exists within Xi Jinping thought that would make sense of a potential tenfold acceleration of scientific and technological progress? What concepts does he have available to make sense of this?
It's a really interesting way of putting the question. When we think about speed of progress and development, there are few societies on Earth that have dealt with that dynamic as recently as China itself. Meaning if you think of China in 1980, and if you think of China today in 2026, that is a stunning speed of transformation. And that's not a tenfold increase in, you know, tech progress in the span of a year. We can talk about those kind of, you know, hard takeoff scenarios. But it is by the standards of pretty much all human history, like pretty darn fast. So I say that because like, when we compare it to societies that have been, you know, wealthy and, you know, experienced sort of development in a more gradual process or longer ago, you know, I just think there's like a, there is something interesting there in that fact that
China has experienced such rapid development in the recent past. But I don't think any of that prepares any of us in either country or anywhere around the world for that kind of speed. And I think in the minds of Chinese leaders, the risk of losing control over these systems, over society, over technology is a risk that kind of has a much deeper and more fundamental status than it does even in the United States, because if you think about, like, the way in which in Xi Jinping's kind of more ideological thinking, technology is positioned, it is, you know, as he says, the main battlefield of international competition. But it is also kind of a physical instantiation of political power. And the Chinese Communist Party's relationship to technology over a long time has been that this is something we need to harness to drive the development
of China, the so-called great rejuvenation of China, and our political power more generally. In the 1980s, when the information revolution started to happen, Chinese leaders were thinking about very rapid, by those standards, shifts and wanting not to miss out on them as they had missed out in their telling on the, you know, prior waves. The industrial revolution. Obviously missed out on the Industrial Revolution and missed out on several prior ways. Xi Jinping has said this revolution we will not just not miss out on, but we will help lead. So those are some of the concerns and considerations that would be a part of this. But I also will just maybe end this answer by saying the question of kind of exactly how this plays out and exactly how much control by that point the Communist Party thinks it has is going to shape so much of this.
Because, you know, for instance, right now, Beijing is still significantly constrained on compute. They see themselves as behind, relentlessly trying to catch up. But if by whatever point we're talking about in this hypothetical, they feel less constrained on compute, which I don't think is terribly likely, but let's say they, or they feel that they have reached closer to parity at the frontier and suddenly it might be slipping out of their grasp. You know, I think those are just profound uncertainties that could introduce even more volatility into this moment. And we just don't know. No. I hate to kind of have a, you know, fancy four part answer, the upshot of which is that we don't know, but it's the truth. And I think as we consider maybe just more generally to say, like, as we think about why this is so challenging for the Chinese Communist Party, this is generally a system that moves pretty slowly. There are exceptions to that, and I think AI governance could end up being one of the exceptions to that.
But they move pretty slowly. They like to have these, you know, really laborious, deliberative processes, you know, paper and endless meetings and all of that. And they, like pretty much any government, but to a pretty intense extreme, given the authoritarian nature of the system, want certainty. And so in their system, one of the hardest things. Exactly. One of the hardest things about frontier risk has been that the upshot of every conversation is we don't know, but the scale of outcomes is near infinite. Who actually has shaped the CCP's views on AI the most? I remember you have this great article from a couple of years ago about the influence of the American futurist Alvin Toffler on China's sort of national R&D program in the sort of late 80s. they read a lot of Alvin Toffler and apparently this inspired a lot of their approach to using new information technologies. Who might be the equivalents for AI? Who are they reading to
understand what this technology is about and what opportunities it offers? Well, there are a few ways to kind of get at that question. One is, for good or for ill, a lot of people in that system, I think, you know, are reading a lot of the same stuff that, you know, Americans read, meaning the core, like AI, the people who are working in the labs, in the companies, are generally, you know, have had exposure, even if they haven't studied in the United States. You know, it's a pretty global, you know, research ecosystem. You know, increasingly, I think, less that way. But there is that there, and, you know, we do see people in the Chinese ecosystem respond to, you know, everybody from, you know, whatever, Daria to Leopold Aschenbrenner and whoever else. Like, this is at the last. This is at the last. But I think that's important because one of the really interesting dynamics inside China is that the companies or some of the companies, I think, have very different visions of the AI future than the Chinese Communist Party.
And so that was kind of where I was kind of going with this, which is that there has been a debate, for instance, like about whether China, air quotes around China, is AGI pilled. And I think the candid way to answer that is many of the labs are. The Chinese government has been much less oriented that way. But I think even that is changing. And I think the upshot is that now basically the views that are shaping the Chinese Communist Party on this are increasingly internal to the Chinese Communist Party. They are not publishing essays on microsites. They are, like, reporting up through the system. And one of the reasons that I think more engagement, technical exchange, and dialogue is a good thing between these two countries is we need to be really thinking
about how we're shaping those reports that go up to the top Chinese leaders, that they pick up and read because what's in there is likely very, very different than what, you know, Yang Wenfeng, the Deep Seek CEO, might say on an investor call. In a world where Xi does end up taking the idea seriously that AI could lead to something like a tenfold acceleration in scientific progress, how would that have happened? Who would be the sort of important precursors for Xi also believing this? Do we know what the information flow could possibly look like? It's a really interesting question to unpack, and I don't want to get way too in the weeds on this, but basically, you know... We can get an org chart out. No, exactly. I'm like, yeah, you can put it up on the screen. But basically, you know, their responsibility on AI is somewhat diffused across different parts of the Chinese government because the fundamental approach that they have taken to AI policy, not like AI safety, but just AI policy more generally, is the shorthand is AI plus,
essentially integrating AI into everything. Of course, they have made tremendous progress integrating it into manufacturing and robotics. They are integrating it into the military. They're integrating it into the political and state security apparatus and so on. So that means that responsibility is somewhat differentiated. I think the least likely way that Xi Jinping comes to that impression is that he reads a bunch of essays from American lab leaders. What would be a more likely route? Maybe if it comes from the industry themselves, perhaps? I mean, I think the likeliest route, honestly, is that it starts to happen in the United States first, and it doesn't feel like it's happening in China. And the effect is that in connection to their becoming more concerned about the risks domestically, they also see the possibility of the United States really pulling ahead very quickly. And I think that's a moment when you can have very divergent paths, but there's a non-zero chance of a path that if the United States is smart about how it approaches it, which I think is a big if, where, you know, you could imagine Beijing coming to some sort of, you know, internal reckoning that pacing is clearly in their interest.
Small probability, but non-zero. Is there anyone important in China who might have what we would call the Silicon Valley view of AI? Like, do you think there's anyone, other than perhaps the lab leaders themselves, is there any other person who is significant who you think is significant? And when you say the Silicon Valley view, you just mean about, like, speed of progress and scale of impact? Yes, yeah, exactly, yeah. Possibility of something like RSI, possibility of vast technological progress. That is the kind of question that I wish people outside of China could answer authoritatively. And one of the challenges is that I can give you my best answer, but I want to start it by just saying, like, we have very limited insight into those kinds of questions. There is an emerging discourse on RSI inside China, for sure. I mean, you know, you can see that. You see increasing discussion about it. But senior people, particularly people inside the system who have these views, we have very, very little indication.
You know, there are obviously people who advise the government who are, you know, outside of the government itself, not necessarily at companies, but who are sort of the experts on the governance of this technology, who brief the Politburo and other senior bodies. Some of those people, very kind of cosmopolitan, go to Silicon Valley, you know, or have thick relationships there. So it's not like there's nobody who's briefing the government with those views. But does Xi Jinping's chief ideological advisor, Wang Huning, believe that? Or does Khalifeng, who's the sort of senior economic planner, believe that? We simply don't know. He's the guy who meets with Besson, who's Besson's counterpart. We simply don't know. The way that I have been saying we can evaluate that most effectively is that there are a set of decisions that the government would make if they came to believe that. Centralizing compute, buying suddenly a lot more, you know, NVIDIA,
which they are approved to do by the Trump administration but have been rationing. Now, part of the reason they're rationing is that they've been rationing it, yeah. Right. Now, there's this bizarre situation where several years ago we would have said the core dynamic is China wants more, of the world's best chips to import them than they can get. And now the dynamic is the United States under President Trump has greenlit the sale of some chips that were previously not permitted, but Beijing is buying them in quite limited quantities. Why? Well, there are a couple of reasons. First reason is that in general, Xi Jinping is trying to get the Chinese AI ecosystem to be more reliant on domestic sources of compute, particularly from Huawei. But second, crucially, they are still able to get access to the best servers and the best compute in third countries through remote access to data centers.
And there's also a bunch of smuggling activity that's going on that feels like every week we get some new report of large-scale smuggling activity, which is just wild. So anyway, but that is all a preamble to saying if they thought, okay, we have two to three years in which the world is going to change fundamentally and we simply cannot afford to be behind, you'd see them centralized compute. You'd see them more aggressively buy American chips, and you'd see a set of policy changes like that. So it's a long way of saying to evaluate the answer to your question, I think it's less likely that we're going to get a speech from Wang Huning as a leading indicator saying, I have decided that RSI is about to happen. And it's more likely that we begin to see some of these physical material movements that undergird actually permitting that to occur in China on a technical basis. Yeah, just on the chips thing, was it Ding Shui-shang who recently said that if you're buying non-domestic chips, you're a traitor or something like this?
Yeah, that was reported. That was reported. Pretty interesting. And to the point about kind of diffusion of responsibilities at the senior ranks, Ding is an extremely senior policymaker, one of Xi Jinping's top advisors, who has been put in charge by all accounts of the internal process of kind of coordinating compute and AI policy in some key dimensions. What we don't know is how the national security dimensions that we were talking about a minute ago, military and espionage and cyber and all of that, are integrated. And, you know, there's also other people who are playing in this space at quite a senior level. But yeah, it was pretty striking to see those reports from Ding making clear to the Chinese AI sector that like, if there's a tradeoff between going a little bit slower and, you know, making sure that you're not dependent on foreign compute, go a little bit slower. Is there a parsimonious explanation of how the CCP takes radical policy action in response
to perceived crises? I'm thinking of things like zero COVID or the one child policy. I don't think zero COVID and the one child policy are like apples and apples. I think that that's an apples and oranges comparison because, you know, in one case, you have a near term response to a crisis. And then this kind of, you know, upswell of public discontent around COVID that leads to initially doubling down and then eventually suddenly loosening up. And in the case of the one-child policy, you have, you know, a decades-long effort that the party comes to see as having led them into a demographic crisis. And so they then are scrambling to figure out how to deal with that and to incentivize people to have children and all the rest of it. So they're pretty different. I think the most sort of in the current environment, the way that we I don't know if it's parsimonious per se, but the way to think about this is when does the ledger begin to shift the core interests of the Communist Party that we've been discussing to see a different approach as necessary?
And Xi Jinping has left the door open. In his speech at the World AI Conference, he did lay out a set of principles. But he also said, you know, using a kind of Chinese proverb, that flexibility and changing and updating as new information comes in is going to be a part of this. And to put it mildly, that is not a signal that he always sends. Normally, it's like we will resolutely uphold X, Y, or Z, you know, until the bitter end. So the door is open there. And I think, therefore, just to go back to kind of where we started, like the most important thing is their own perception of AI safety and AI risk evolving inside China. And I do think that will happen. And I think it could happen sooner than many of us expect. Other than asking for more chips, what sorts of things do you think the Chinese AI labs are lobbying the CCP for? A lot of things. And I think a really interesting dynamic that is super underreported, because it's really hard to report on, is, you know, we know a lot about the bargaining and all the rest between the labs and the government here in the United States.
We know much less about it in the Chinese ecosystem. But if you talk to people who work at Chinese AI labs, they complain constantly about government overreach and, you know, all this stuff they have to go through and, you know, government getting in the way of X, Y, or Z. So there are these dynamics there. Now, I think so, right. So compute is one. The government has been trying to channel much more capital into this sector. But obviously, that's another. A third, crucially, is these really startling new regulations on people being able to leave China. So there have been a series of reports about initially employees of Chinese model companies and then even their family members no longer being able to travel abroad without permission. And, you know, this is the kind of restraint on, you know, human mobility, ordinary life that I think will affect, you know, somebody is a PRC national working in the United States and they're thinking about going back to China to take some sweet deal they've been offered or to stay.
Like, that's a pretty big difference that goes beyond whatever your compensation number is. And then, you know, one other thing to just say is, like, I think for a number of these companies, they look at what happened to the Meta-Manus deal, you know, meaning a Chinese startup that was super buzzy, you know, got a lot of attention, moved to Singapore, and then Meta wanted to buy. It almost happened. And then the Chinese regulators came and yanked the chain and shut it down. I mean, that is the kind of thing that, as you know, like is any startup founders nightmare wherever they live that like the dream deal comes through and, you know, the government stops it. And so this is so there are all of these pain points. Now, there's a lot of stuff that is they're getting. I want to be really clear that like the government is, you know, pushing resources and prestige and support in other ways into this sector.
but it is complicated. It's making their life difficult in other ways. That's right. No. What are the other exogenous factors, so like non-AI factors that are most likely to influence Chinese AI policy? I'm thinking of things like, I don't know, an aging population. Is there anything else we should be tracking to understand exactly what's happening? So stepping back from like the kind of frontier risk conversation into how AI policy has evolved in China, Now, I do think the aging population is a big, big point here because kind of similar in the way that, you know, some model companies in the United States talk about, like, you know, we may not need all the engineering talent in the future. You know, I think in China's case, one of the hopes has been that increasing automation, robotics, you know, sort of embodied AI will enable them to maintain or even increase their manufacturing dominance, even as their low cost labor and young population advantages go away.
I also think the kind of their perception of the United States right now is that the kind of remember, they see us always as these kind of capitalists who have a system that has a lot of irrationalities in it. And, you know, I think if you really were to press them, I guess, as they would say, there's a bubble. There's a lot of speculation. but I would say like the way that that plays out, the way that sort of valuations, financial flows, all the rest of it play out is AI related, but is a little bit sort of a little bit exogenous in some sense, because it's not about their own perception of their core AI capabilities. Those I think are two big ones. Maybe the third is just broader perceptions of US-China relations, which I do think are really important here. I think in this period where President Trump has been trying to clearly, you know, you know, put out a very welcoming image to China and emphasize the strength of his relationship with Xi Jinping and all the rest of it.
You know, there's a lot of mistrust in the Chinese system toward that. And I don't think they see it as much more than buying time. But I do think if U.S.-China relations were to deteriorate markedly, you know, kind of back to Trump in 2020 rather than Trump today, that would be a pretty significant factor for them in, you know, further kind of intensifying the zero-sum national security centered frame for their approach to AI. But that would be exogenous. What are you most worried could deteriorate the U.S.-China relationship further? Is it another trade war? Is it something like Iran? What should we be tracking? Oh, there's a lot. Unfortunately, I think the – well, I've described the kind of current state of the U.S.-China relationship as K-shaped. And by that, I mean in the way that we think of a K-shaped economy where there's like an upward leg like AI that's going gangbusters, and then there's a lower leg that is doing much, much worse. You know, you have this upward leg of the U.S.-China relationship that is leader level, you know, palling around these big optics.
They're vibing. Right. And then you have all the realities of competition, military, technological, economic, and so on that are continuing. You know, it's not as if that stuff has stopped, even if it is less salient or visible to many people. So I think there's a long list of things that could disrupt these dynamics. But I will say not to, you know, get us into a kind of conceptual doom loop, but like AI is a huge, huge factor here. And I think there are scenarios where, you know, China appears to have advanced much more rapidly than people expect that could really shake up this current dynamic. And I also think there are scenarios in which the United States appears to have pulled ahead so dramatically that Beijing's calculus changes that could be really disruptive. What worries me the most day to day is that, you know, there are sort of two things. One is these leader level interactions, they look kind of predictable or stable, but actually there's so much uncertainty to them.
I mean, you know, you just never know when something's going to go wrong. And so I never feel like the fact of a summit is itself a sure thing that it's going to lead to things going better because it's easy to imagine President Trump changing his mind. And then the second thing is just our militaries continue to operate in very close proximity in the Western Pacific every day. And the risk of an accident is always there. Is it magical thinking from like Western AI folks to think that China would like invade Taiwan if they thought the U were on the cusp of RSI Is this like a silly idea For Beijing Taiwan is not about AI right Like this is a core regime, longstanding priority. This is what they view as the unfinished business of the 1949 coming to power. and Beijing has wanted to control Taiwan long before anybody was talking about AGI.
So, like, we've got to remember that because that is the defining fact. The fact that TSMC sprouted up in Taiwan and now has become totally central to all of this. It's a strange accident of history. But, like, to me, actually, one of the most remarkable dimensions to this is the flip side of the way you pose the question, which is how long it has taken Beijing to reckon with that fact, meaning that Taiwan's global salience and global kind of role have changed because of this newfound economic centrality. But I don't think when they're like thinking about this or, you know, they're first and foremost thinking about AI. That said, it obviously is it's not something where you can completely dismiss it. But like I think it's pretty important for just people to remember that Beijing has wanted control over Taiwan before AI. and they want it for reasons of nothing to do with AI in the first instance,
and just like political timelines in Taiwan without wanting to get too in the weeds. But like Taiwan is a presidential election in January 2028 that Beijing is really hoping will go in a kind of direction that is with the leaders who they prefer. The KMT? Precisely. And so from their perspective, you know, this is a window where the political path feels, I think, more open than it has in a long time. So in general, like, the approach is, yeah, keep the coercion up, but see if this works because the military options for them are not good. Like, it's a very costly, very, very costly thing and not something I think that they want to do if they can avoid it. Do we have any idea what the significance of a KMT victory in early 2028 would be? Or is that completely open? There are a couple different kind of permutations for what that would look like.
You know, we don't know who the KMT candidate is going to be yet. The current KMT party leader is someone who has gone to the mainland, who met with Xi Jinping. So, you know, they've clearly had quite a positive view of her. but it's certainly not a sure thing that she or her view would be the view of the eventual KMT nominee. There's local elections coming up soon this year in Taiwan that may shape some of the way in which the KMT is configured. So it's not that we don't know in an abstract sense. It's that like any process, it's just like asking who's going to be the Democratic nominee and what's that going to mean for U.S.-China relations and our 2018 election. It's like, well, you could have very different scenarios there. Yeah, yeah. Let's go back to a potential deal between the U.S. and China. So imagine if we get some very early stage cooperation. So maybe it looks something like simple information sharing or maybe a shared working group between U.S. Casey.
And I think Matt Sheehan said that China's working group number nine is the one that is tasked with AI safety standards. Let's imagine we get something like this. What would you expect to be the early sticking points of this kind of cooperation? Okay, so first, I would say getting something like that is itself going to take some work. So, yeah, so like, you know, I think it was really striking to me where I had pretty low expectations for what the Bethlehem meeting and then the Trump-She meeting would produce on this score. Pretty low expectations. And they were not met. I thought we would get at least some sort of technical working group commitment, because that seems so obviously necessary, like such a no-brainer and not hard to do, meaning you're not committing them to do anything. You're just creating that kind of connectivity. And we didn't even get that. And I thought we would get some kind of commitment to sort of, you know,
more intense cadence of interaction and crisis communication when they're, you know, incident response communication. And it appears that we basically just have a kind of verbal commitment to, you know, call each other under a pretty vague set of circumstances, which, let's be honest, was the status quo ante. They can, you know, Besson and his counterpart can call each other when they want. So, but let's say we were to get that kind of connectivity established. I think the first sticking point would actually be terminology, language. Because if we're talking about like a fairly professionalized approach from Casey here and, you know, the way that these bodies in China approach it, like there are some basic language questions that need to be worked. to what are we talking about when we talk about loss of control?
What are we talking about when we talk about bio risk? And I think actually there's a lot of work to be done in just nailing those pieces. But I don't think that would happen naturally. I mean, for your listeners, like even at the most fundamental level, I was in a meeting not long ago where an American, very senior American AI expert gave a long spiel to a bunch of Chinese AI safety experts where he was making a fundamental distinction between safety and security. You know, the safety of an AI system and its alignment and other aspects like that. And then there's just, you know, security and more sort of cybersecurity, model security sense. Now, in Chinese, safety and security are the same word, ANTRAN. So the people on the Chinese side who knew English were following just fine. And the people who didn't were deeply confused because, again, it was just so that kind of thing is actually like it's real. Yeah, it's a sticking point. And it's like it's not like an ideological disagreement, but it needs to be worked through. It's one of the reasons why what track two dialogues do is actually important, even if it sometimes can feel kind of painful because it's helping deal with that.
Second, you know, in those kind of let's say this kind of technical conversation happened, I think there would be a big set of debates over whether they were talking principally about near term immediate risks or frontier kind of, you know, risks that may emerge in a sort of shift toward RSI. RSI. And, you know, assuming that the Americans wanted to talk a lot about the latter, I think there might be some resistance from the Chinese side to fully getting into that, particularly if they felt, you know, that it was kind of being presented to them as something the United States was going to get to first. And again, just this trope of like, they're ahead, they're trying to keep us behind. What kind of a deal do you think would be most appealing to the CCP on AI? Well, I mean, not to be too cynical, but the most appealing thing would be unilateral restraint from the United States. You know, like I'm not saying I think I think we you know, but like to them, everything flows from their conviction.
As I was saying earlier, that they're behind. We're ahead and they need to catch up. So, you know, one of the hard questions which we talked about is that the United States does have to show that we're serious about these risks and serious about regulating this sector. And from Beijing's perspective, that part of it's very appealing. What's not appealing is constraints on China's own development. And that's the core kind of prisoner's dilemma-like situation that we're in. And we haven't really talked about verification either, but like we need the verification science to catch up to the frontier. But there is then a whole other question about Beijing accepting and candidly about the United States accepting those kinds of what would be pretty intrusive external monitoring or, you know, evaluation. This would be something like having on-site auditors in labs. Actually, this is going to be my next question, which is, if you imagine we're in a fairly radical world and the U.S. and China have, in fact, agreed to temporarily pause AI training, if you put your NSC hat on, what would give you confidence that China had, in fact, paused?
Some of this starts with, like, well, what is the verification technology? Not just third-party auditors walking around, but, like, when I talk about that, I mean the kind of true, robust, technical ability to understand, you know, again, it can be framed in many different ways. People are trying to work on many different dimensions of this, but, like, you know, the size of runs, like, where there's, you know, how compute used is. where the data centers are located, like the amount of energy use, actually testing model capabilities, like these kind of more built-in or integrated technical solutions to these questions, I think there is not going to be a high level of trust. So I just don't think we can premise any of this on that. So instead, what we're looking for is the ability to have some degree of confidence, via verification in this kind of speculative scenario.
And I would just say, like, we could be having one version of this conversation that is about, and again, we're in the good place here. Like, a lot of things have gone really, really right in this hypothetical. but where we're talking about, okay, there are agreed limits to, you know, what kind of RSI will be permitted. Like, meaning we're not going to just hand over everything. We've agreed to limits on that. Okay, well, that is something that I certainly would be supportive of. But the question then is, given that you're dealing with systems where humans will struggle to understand what they are doing, period, forget about the U.S.-China dimension, they are too. You need to kind of solve the alignment science and the verification science piece of it to have confidence, not simply in the U.S.-China dimension, but in the human-machine dimension of this. And so I think as we get to those more sort of, you know, frontier curve scenarios for the U.S.-China piece, a lot of the sticking points that I think of, which also are reason for some degree of optimism, are actually not simply U.S.-China, but are actually like how much visibility, understanding, control over the systems that we've created, we still have.
But to my mind, that problem is going to have to get solved in each country separately. And so one of the framework ideas that Matt Sheehan and I have been kind of pushing, and this is really driven by his work, is that we really can think about this as there are just some areas where self-interest can lead to action in parallel that can be coordinated and can be the building blocks for more ambition that happens later. but we got to begin in each case from that self-interest question as the driver of action. Yeah, I mean, that makes sense to me. I guess I was also wondering how far you'd expect unilateral surveillance to go. So probably there's some amount that you can do, even without having a lot of cooperation between the two sides, but if you just use espionage to try and figure out, okay, well, you know, we can have some idea of what the largest training run looks like in China right now, purely through our spies and things like this, or satellite imagery, something like this. That question is one worth kind of, I think, both sides reckoning with at some level.
But when we think about an agreement and the history of agreements more generally, of course that activity continues from both parties in the agreement. And it's part of how they are assessing each other's capabilities and intentions and all the rest of it. But the reason that we talk about diplomacy and an agreement that's more constructive and proactive is that that is insufficient to rise to the scale of the challenge in front of us. Like, if you could just kind of surveil or espionage your way out of this problem, we wouldn't be talking about it in the same way. But I don't believe that you can. And I think what we are ultimately talking about are two countries that are extremely powerful, making their own decisions to constrain some of their own capabilities and showing one another in some form or fashion that they are doing that actually. And that is different than just like, you know, kind of snooping on each other. What are some of those other areas where you think they have overlapping shared incentives to build regulation in parallel? What else comes to mind?
Well, to me, the core thing that I always think about on this is neither government, neither country should have an interest in losing control of these systems. And that is like a very high level of abstraction, but it is actually quite a common sense principle. and what that means in practice, I mean, by some metrics, we're already losing control of these systems, to be clear, but we do still have the capacity to rein them back in. And so that is the core, or at least one of the core interests. Another is that there are very powerful capabilities that are created that should not be in the hands of kind of bad actors, you know, third parties. terrorist groups and the like. And there, you know, we haven't actually, I think, said the words open source or open way
in a long conversation. But, you know, there we have to be just very blunt that the proliferation of capabilities that we're seeing in real time where guardrails can be removed, models can be downloaded and used without the model company having any idea what's going on or anybody having any idea what's going on except the person doing it. that is really concerning place to be. We are starting to see, you know, we're recording this on October 2nd. We are starting to see an emerging discourse in China, raising concerns about the risks of open weight and open source. And this is very new, but it is indelible. And, you know, I'll be writing about it, you know, in this weekend in my newsletter, but like, it's starting to happen. What we don't know is how those emergent risks get weighed against the fact that there's a very strong set of political and geopolitical incentives to keep this going.
It's good to undercut American companies. Is that what you're thinking? Great to undercut American companies. Great to spread Chinese influence around the world. Great to do both of those things on a very limited compute budget because, you know, Obviously, like, it's not like, you know, ZAI has to be, you know, having the, you know, building the data centers with chips they don't have to serve all these things. Right. And it also is a great political narrative that Beijing has been pushing very hard to say the United States has these closed, exclusionary, you know, air quotes, air quotes, systems. And China stands for so-called, you know, openness. and what I always say about this is like actually though that does not need to be an absolute principle they've given themselves a lot of maneuvering room they just need to be more so called open than the United States is and they have a lot of running room on that they don't need to somehow be absolute like you know open source fanatics to you know keep
this political narrative keep this narrative going yeah where is this new discourse happening by the way where there's a sudden discourse about whether open-weight models are worth the risk. So why is this happening? Yeah, so there are a few really striking things, and it's literally happening in real time. And so, you know, by the time your listeners are hearing this, it may be that, you know, there have been a ton of New York Times articles about it or whatever. But what we're seeing now is, first, there was some really interesting language in the new kind of AI safety governance framework that Beijing released recently. this is sort of from the body that advises the government on AI safety technical matters. This is T-360 3.0? Yeah, exactly, precisely. And in there, there is some really striking language about open source releases, lowering the barriers to conducting sophisticated cyber attacks, and accelerating and expanding the scope of those attacks.
So, like, the warning is already there. The question now is what safeguards Chinese authorities and developers are going to put in place. And then you've also seen in the kind of scholar and commentator ecosystem emerging language basically saying we can't have unrestricted openness. Constructive forces can use it. Destructive forces can also use it. We've got to find a balance, which is strikingly different from where they were a few months ago when the narrative was open source is synonymous with China and all the political narrative that we were just talking about. Yeah, I mean, Prima Facchie, it is surprising to me that China is still open weighting as many models as they are at the level of capabilities that they are. I feel like this is not something I would have predicted in, say, 2023. Well, I don't think they necessarily predicted it either. Remember, like, initially it was like, you know, Meta had amazing open models and China was kind of figuring it out.
But I think because of the desire to undercut competitors and spread globally and the compute constraints and everything else we're talking about, this happened, you know, with obviously the big DeepSeek release as the most powerful symbol of it. And it just kind of gathered momentum. And the party, candidly, I think, was happy to go along for the ride and then to figure out, wow, we can actually spin this geopolitically to our benefit. But again, as soon as they begin to assess that the security and stability risks are getting out of hand, they have the tools to change this quite rapidly. If you think about the U.S. sitting down at the table with China to discuss a potential deal, discuss further cooperation, what do you think the U.S.'s priorities should be for strengthening its bargaining position? Should it be something like indigenizing the rare earth supply chain? Is that the number one? What else should they be focusing on? There are two categories. One category is things specific to AI, and the second is things beyond the scope of AI, but that, like, rare earths affect the balance of leverage.
So maybe start with the first category. The way that I think about this, which is borrowed from my former colleague Chris McGuire, is you've got kind of a three-legged stool, which is like you have your domestic regulation, meaning the kind of costly signals that we've been describing. You have the export controls that are in place and can be tightened and that are part of the puzzle here. And then you have the actual both bilateral and global diplomatic outreach you're doing to create, you know, communities of interest and momentum toward that kind of bargaining. So those are like the three dimensions specific to AI itself. then you have the much broader questions related to the balance of leverage in U.S.-China relations. And I absolutely think that many of the things we need to be doing there, you know, you talked about dealing with the rare earths dependency, which is a huge priority, that those things will all help the U.S. at the negotiating table, you know, with Beijing.
So we kind of have to have both strategies firing on all cylinders at once. And then just, again, the third thing that I'd say is, like, we also have to be tracking so closely how Beijing sees these risks, because as their own risk perception evolves, there will be windows of opportunity. And I don't know that necessarily the current team is thinking about it this way, But certainly if I were in government or if others who, you know, have this approach where I think you'd be looking for these windows, these moments when Beijing starts to take the risk more seriously and seeing those as each creating an opening, not for a sweeping grand bargain that solves all of our problems overnights, but for incremental progress that gets us to a better place, you know, on a timescale that's relevant, but also realistic. What is the right level of pessimism to have about the prospect of China as an international partner? So, you know, as I understand, in 2023, there was a U.S. offer of something like a nuclear crisis hotline, which they stonewalled or rejected.
They also didn't really exchange timely information with the rest of the world on COVID. What's the what's the right way to view how cooperative they're likely to be or not? so I mean I think probably it's come across through this conversation that like I think we have to have we have to have a pretty uh you know limited sense of what they're going to see is in their interest to return to that core idea it's never going to be like a binary switch where they go from seeing this as you know this intense domain of competition to suddenly seeing this as you know, no brainer, any rational person would understand we got to make a deal. So I think that initial point that I made about like, we got to begin from China as it is, and how they think about these challenges, rather than how we think they ought to, is core there. I will say that, you know, in some of those areas you alluded to, like nuclear
arms control, there is a similarity in the fact that they're behind, or they have fewer, you know, nuclear weapons. And they therefore always say, why would you talk about constraints with us when you have so many more than we do? The other thing I would say, which I probably said earlier, is they often, I think, over the past couple decades have looked at the United States and felt that we get certain fundamental risk factors wrong. They looked at the global financial crisis in 2008, and they said the United States fundamentally got wrong the financial risk in its own system on the side of way too much risk tolerance. They look at the predictions that were made 25 years ago about how the Internet was going to change China. Remember, 25 years ago, Bill Clinton said that trying to control the Internet was like trying to nail Jell-O to the wall. He literally said, good luck to China. And they were staring down those predictions.
And they said, no, the United States is wrong. We are going to nail Jello to the wall. And they did. And how do you keep your ambition? Because when I think about sometimes all the things that a real full deal on AI might require, we're talking about potentially many years of not just monitoring each other's compute supply chains, but also monitoring each other's robotic supply chains. You know, we're talking about a completely, what might need to be a completely unprecedented level of cooperation between these two countries on a technology for which there is less consensus about the risks, on a technology that is moving faster than any previous technology. Sometimes I just think it seems completely impossible. Like, how do you view our chances here? I guess the way that I think about it is rather than designing in the abstract a mental model of what the ideal deal would look like, I think if you're doing that, if you're doing the kind of AI 2040 thing, the impossibility confronts you at every turn.
But in fact, I don't think right now that kind of ideal mechanism design is the most urgent question. I think figuring out how we can start immediately wasting no time to take incremental steps in the right direction based on an understanding of each of our self-interest is the question before us, the most urgent problem. And that's tractable. And that is tractable. So, like, I think beginning there and then also continually saying what is actually happening in the technological development, what are the scope of predictions or models for how this goes, where is the discourse in China, where is the discourse in the United States, and trying to just, again, be continually ambitious and opportunistic about it. Like, I think that as a mental model, shall we say, in addition to just like a praxis, is like that is actually not such a dark place to be, even though it's still, you know, a really alarming and challenging moment.
The hardest version of the question right now is if someone were to ask me what does success look like in a year And I say that the hardest version because like as you know better than I do you have RSI timelines that are shorter than that You have just such fundamental uncertainty about the technology itself in addition to the geopolitics and everything else. So to me, it's the kind of classic, you know, one foot in front of the other, walk before you run. What's the phrase about stepping away over the river over the stones? Yeah, there's a famous Chinese saying, which Deng Xiaoping used about economic reform, crossing the river by feeling for the stones. There's some element of that, too. I don't see another alternative because we're not going to sit down at a table and design a perfect mechanism and then implement it tomorrow. I now have a series of very selfish questions I would like to ask you in that I'm essentially interested in how I could build or recognize real China expertise.
In particular, I'm wondering, how do you recognize a China specialist as the real deal when the whole field is really quite opaque? So the number one thing is humility. If a China expert comes to me or comes to anybody and says they have all the answers and are presenting them with total confidence, the bullshit meter should go off right away. And, like, from my perspective, the central challenge of doing China policy and building China expertise really is dealing with uncertainty and having nonetheless to make educated decisions. And so that's, like, square one. So some of that has come across in this conversation, I hope, where we're sort of saying there are some things we know, there are some things we don't know, and everything we're trying to do is in that space between them. And then second, I would just say, you know, people who it's not easy to go spend time in China necessarily for everybody, but people who have spent time there, who have a long track record, who read, you know, the source material and have the context to make sense of it.
that's also just it's essential and it's hard it's like it's this funny thing I feel at this point in my career to you know I am doing that almost every day and I was doing that almost every day a decade ago and it has not gotten more fun because Chinese Communist Party documents are not you know beautiful New Yorker articles written to keep the reader interested and certainly are in short form video but candidly like there's no other way and it's how that system works And so if you want to understand it, you've got to go there. Where in China should I hang out? I've been to Beijing. I've been to Shanghai. I've been to Hangzhou. I went to Wake this summer. But other than that, I mean, I also don't speak Mandarin. But if you were to recommend to me. Okay, well, hold on. I was giving you an answer to how, like, a hardcore China expert should operate. I think there is a different answer for somebody who wants to be educated and informed on China but is not going to make their professional life, you know, being the China expert in the room, so to speak.
So for that category, which unless you're planning a major life renovation, I feel like is probably the sweet spot. I don't think actually you need to be like going and, you know, eating, you know, the famous, there's a famous sinologist who described reading these Chinese Communist Party documents as eating sawdust or, you know, so it's like I don't think you need to be doing that. But, yeah, I think getting over there, getting exposure directly, and then also finding a staple of things you read and people you talk with who are really reliable and who are not making BS calls but are kind of, like, really doing the hard work as intermediaries almost, that's sort of the most essential thing. And then I also think remembering that China is not simply the Chinese Communist Party and there is incredible cultural production and there's actually like a really diverse kind of ecosystem. and some of the most interesting moments over the past few years for me have been when friends of mine,
you know, for instance, a lot of people who are on TikTok migrated briefly to, you know, a Chinese app when TikTok went offline here in the United States and were coming to me, like, sending me videos of people in China singing and doing all kinds of random stuff. And it was sort of like, wow, this is actually a degree of exposure to China that friends of mine who do have nothing to do with China are suddenly getting. In terms of where to go in China, I mean, I think for most foreigners who travel there, like you hit the kind of main tech-focused hubs, Shanghai and Hangzhou, and then Beijing, which is one of the old imperial capitals and today the place where the Chinese Communist Party is headquartered. Um, it's a huge country with a lot of diversity geographically and all that. So, you know, I do think it's always good to kind of get out of the main coastal hubs where you see all the development and all the modernity and get to places that are a little bit more, uh, off the beaten path.
But I'll also just be candid and say like, that's hard to do as a foreigner, particularly somebody who doesn't speak Chinese. So I think you don't have to set too high a bar for yourself. If you make it to Beijing and Shanghai, you are already, you know, in the very, very top percentage of people in, you know, the West who have exposure directly to China. I will say I was surprised at how much heavy lifting Google Translate could do for me, though. Like people, it's a very well-oiled machine where people know to pull out their translation apps. You can usually get across the things you need to. Yeah, not to tie this back to the substance of our conversation, but that does actually reflect, I think, a really interesting difference that we observe between the United States and China, which is just a much higher degree of comfort with and optimism about AI over there than here in the United States. You know, the polling is remarkable on this. Basically, the United States is at the very bottom of rankings for optimism about the benefits of AI for people, and China is at the very, very top.
And what you just said made me think of it because it's exactly that thing of you're walking around Beijing or Shanghai and someone in China whose life is already largely integrated on WeChat and into other virtual platforms is very comfortable doing a bit of, you know, real time speech translation and having that be kind of seamless in a way that I think is still pretty unusual, you know, here. yeah okay you're not gonna like this next question because i'm gonna ask you not to be humble and you just told me that the mark of a china expert is humility okay but what what is the nature of your edge like what do you think is the thing you understand about china better than almost anyone else in the u.s do you think is there a thing you would identify there okay that is that is making me uncomfortable um so i mean i don't think this is like a unique to me thing but I do think the most important kind of skill that I have tried to acquire is not simply to treat
China policy as some sequestered, distant, esoteric thing that happens over there, and then you bring back the right answer to people who care about domestic policy or broader U.S. foreign policy or tech policy, but to see it as totally integrated and inextricable. And that's actually, I find very challenging intellectually, because you have to master the China side of it. You have to do all that deep work, you know, getting your PhD in modern Chinese history or politics or whatever the case may be, spending time there, learning how the system works. And then you have to somehow then also figure out how to make that integrated into the trade-offs that policymakers and business leaders have to make when it comes to China policy. Because one of the themes, in a way, of what we've been discussing is, like, there might be an ideal set of solutions vis-a-vis China, but at every turn, reality presses against them in some way. And I think in the field of people
who work on this stuff, like that skill, when I've seen it, it is the most impressive thing. And I'm trying to, you know, bring it to bear in my own career. Who else should I be reading other than your blog, Paper Tiger? Is there other things which you would really recommend to me? When I was in government, I did a lot of work on U.S.-China tech competition. And before I went into government, I was writing about Chinese technology policy a lot. But what I saw over the four years I worked in the Biden administration made me convinced that the AI story that we've been talking about today was on a scale of importance so much greater than almost anything else I could imagine doing. So when I came out, I was like, I'm going to use the time that I now find myself with to go as deep as I can. And I talked to a few friends who were those people, you know, and were much deeper and basically got a reading list from them.
And so I learned so much from Matt Sheehan and, you know, who we mentioned, from Graham Webster, who's out at Stanford and does incredible work, from, you know, colleagues at CSET, the Center for Security and Emerging Technology. You know, people I'm sure are well known to your listeners like Alan Toner. and from Chris McGuire, who's at the Council on Foreign Relations now. And I would just say, and many others on the AI account, but I would just say, like, to me, one of the most striking things is these are not all people who agree on the policy solutions, but they are all people who are doing the really hard work analytically and who sort of share that with the public in a really meaningful way. So that's sort of on the AI account. And then on China, I mean, we are in a little noticed sort of renaissance or flowering of amazing China studies and China scholarship.
And I say little noticed because I think there's a meme out there that like because it has gotten harder to go to China and do research, that the quality of China studies must therefore have suffered. And one of the most extraordinary things to me is that despite how hard it is, there is incredible work happening in the academy and think tanks and elsewhere and journalism, even though for the journalists, it's incredibly hard. Many of them can't even go to China anymore. So I think like my diet of China-related content feels as rich as it's ever been. And just there I would say maybe for, you know, your listeners are like they can like to nerd out, obviously. They're certainly doing that if they reach this point in our conversation. And so there I would say it's worth actually reading some of the stuff that China academics are doing, not just the people who are sort of in the hot take industry, but, you know, my industry.
Yeah, well, no, I don't, I mean, I don't think that's fair. But like, there's a book that I always give as the kind of as an example of this for me, which is an academic book called The Gilded Cage by Harvard sociologist named Yao Wen Lei. And she writes in this book, she has a whole framework for the development of China's technology regulatory regime. And it's a classic example of extraordinary scholarly work that actually has direct relevance to the kinds of things we have been talking about every day. But, like, you're not going to know about it unless you are in this field. What's it about, The Gilded Cage? It's basically about the development of China's technology regulatory regime for the Internet and for fintech. And she did an incredible amount of field work, you know, with, you know, embedded in tech companies in China. And she also comes up with a series of kind of frameworks for analyzing this. And she's like playing for big stakes.
She's playing for fundamental understandings about development and state capitalism in China, relations of firms in the state. But it's just, it's a good example of this kind of book. And there are some books that cross over, like Joseph Terrigian's biography of Xi Jinping's father, which is a deep scholarly work, you know, crossed over, got broader attention. Is it The Party Comes First? Exactly, The Party Comes First, which, you know, and I always think about the fact that, like, there, for each of those books, there are two or three others that maybe don't, but, like, have a huge amount to offer as well. And so one of the things I'm trying to do actually on my own writing is, like, as much as I can, give a little lift to some of those works because they're out there. It's happening. China Studies is vibrant in all of these ways. And, yeah, it's a cool moment to be in the field. Great. Do you have a favorite non-dissident Chinese intellectual?
The people who I find most interesting are largely not people I agree with. Like, you know, it's like it's not like I'm reading Chinese academics and nodding along saying, you know, preach or whatever. But like, for instance, there's a guy named Jiang Shugong who is not and I can send you some stuff of his. But like he's not he is very far from somebody who I think shares any views with me. But I find him to be an enormously revealing analyst of contemporary China, of Xi Jinping. And he's a very ambitious intellectual. He is very supportive of many of the things Xi Jinping does. And he's definitely comfortably operating, by all accounts, in that system. But, you know, when I think about who has been the most sort of precise framework generator that has helped me generate my own frameworks for understanding that system, his work is quite useful.
And a bunch of it has been translated, even if I'm like, I do not like the upshot of that for America or but it's important to actually reckon with those people. And I would say more generally, there is a set of government aligned thinkers who are like they are serious thinkers. These are not simply people who are doing propaganda work, but they are very supportive of what China is doing. And I think it's actually really important for people in my position to read them, to reckon with them, to take their ideas seriously. The best example of this, by the way, is Wang Huning, who we mentioned earlier, who's one of Xi Jinping's top advisors. He's on the Politburo Standing Committee. He started off as a law professor at Fudan University, and, you know, a number of his early writings, like from the back, all the way back from the 80s and early 90s, are still incredibly interesting layered reads. And sometimes they get caricatured. You know, his most famous book is called America Against America.
And it sometimes is described as just like a anti-America polemic. And there's a lot of criticism of the United States in there for sure. But there's also all this really interesting language about how technologically advanced the United States is and how innovative the United States is. And when you think about sort of how he has approached governing China, actually both sides of that are really important. And a closer reading of that text, a closer understanding of this isn't only the parts that are about what he sees in America and criticizes, but also what he sees in America and envies. That like both of those impulses shape what he and Xi Jinping and other Chinese leaders have done. So it's just, yeah, it's important to reckon with. What Chinese intellectual currents that are particularly relevant to AI policy do you think we should be aware of? So I'm vaguely familiar with the Industrial Party as a movement. I've read Lunan Feng and Wu Jing's essay on this. Can you carve them into blocks that I should reckon with, or is that difficult to do?
So I'd say a couple of things. First, there is a longstanding sort of futurist kind of strand of writing in China. You mentioned earlier in our conversation the Alvin Toffler influence in the 1980s that I wrote about. And I think understanding these kind of techno-futurist visions in China and the way in which they see industrial development and party leadership is one really important strand of thinking. A second, which used to be much buzzier a few years ago and has certainly diminished since then, is Chinese science fiction. You know so much. Jordan Schneider, who does the China Talk podcast as a friend, likes to talk about how influential sci-fi has been in shaping American and Western visions of the AGI future. And there was this incredible flowering of science fiction writing in China, you know, over the course of the 90s and 2000s, 80s, 90s, 2000s.
And it was one of those areas where you saw people doing politically sensitive writing, but somehow getting away with it under the sort of sci-fi banner. And I think it's important for people who are trying to understand the kind of currents in China to look at that writing. And, you know, obviously, Liu Cixin is the most famous, you know, Three Body Problem is the most famous, but there are a range of others. there's a guy named Chen Shou Fan who ended up actually co-writing a book with Kai Fu Li on AI futures, but is also just a really interesting sci-fi writer, a woman named Hao Jing Fang, who wrote an amazing sort of novella, long short story about kind of inequality in Beijing. And so there's a number of these things which are really interesting, even if the space for that writing is smaller now. But I think it reflects this kind of set of possibilities. And then, yeah, and then I guess like third and finally just to kind of foreground is to say,
reading these regime aligned or regime supportive intellectuals in China who position AI at the absolute heart of the way in which power will be distributed in the future and give that kind of deeper intellectual foundation to Xi Jinping saying something like, you know, technology is the main battlefield of international competition, that's really important too, in addition to reading the people who are actually working on AI issues. And what about using the AI models themselves? Which AI model is the best China watcher? Of the Chinese models or of anyone? Across the whole range. If you're doing research with a model, which one are you using? I found, for instance, ChatGPT seems to be better at finding WeChat links than Claude for whatever reason Yeah, no, it's interesting you said, I was about to say something similar I'm not terribly original on this, but yeah, I find for the kind of research that I'm doing to the extent that I'm using AI to help me do it
ChatGPT somehow has seemingly greater facility with the Chinese parts of the internet than Claude. And I've heard some people say, like, I should be checking out Gemini more on this front. I confess that I haven't really. I'm pretty satisfied. I mean, to the extent that I'm using this, I'm pretty satisfied with what I'm getting. But, yeah, it is interesting. And I will say, for me, what I have not done, but others have done, and there's really interesting work on this, is kind of comparing what you find when you research China on Chinese models and American models. Obviously, we all know about certain censorship guardrails that are built in. It's not surprising that you find much less about the Tiananmen crackdown on a Chinese model, but that actually because of the priority given to, like, the training corpus being in Chinese, that there are sometimes very different answers you get when you ask the question in Chinese versus in English,
which is itself a really interesting thing and I suspect is also probably true in some of the American models. But that connection between the corpus that's being used, the language of the corpus that's being used to train and the kind of epistemic consequences of that is something that I think is just going to become more and more important. By the way, we are beginning to see a Chinese discourse on this too. People in China writing about the fact that what they choose to feed in is what others will be processing for centuries to come and hoping to shape international discourse through shaping the kind of corpus of Chinese and China-relevant material. It's a super interesting area that there's some work happening on, and we need more. Do we have any idea what they'd be prioritizing other than censorship? Like, what narratives would they be actively trying to push? Absolutely. So, like, the questions of, for instance, you know, the simplest version is essentially propaganda, where it's like the goal is to push a positive set of narratives about China and downplay criticisms of China.
A positive set of narratives about Chinese leaders downplay criticism of them. But then there's like a whole range of other topics like Taiwan, where their goal is to maximally shape a version of the understanding of Taiwan issues that comports with their official line. and the idea that we could have models that would be unwittingly built in such a way that they reinforce that, I think is the kind of concern that is really reasonable to have, particularly now that we begin to see corners of the sort of official and semi-official discourse in China talking about this as an opportunity for them. The final question I have, where do you think are the best places on earth to visit to understand China's international strategy and its investment in the global south? I spent a bit of time when I was at the State Department working on this issue set and doing a lot of travel around it. And, I mean, to focus sort of on the Global South questions specifically,
my own view is that it's good to get a kind of a spectrum of different places that have found different ways and are finding different ways of striking the balance between getting more benefit out of their ties with China and maintaining their own sovereignty and autonomy on the world stage. So at one end of that spectrum, you have India, meaning a country that, because of the border conflict and other issues, decided to block a bunch of Chinese apps and sort of really try to push back against Chinese influence. And I think it's super interesting and important to hear their perspective, which is not one for one with like the Washington perspective, even though it is quite concerned about increasing Chinese influence. Then I think like Central Asia is a super interesting place because of the dynamic of competition between China and Russia there. Obviously, China and Russia are close strategic partners, but they also have some areas where they're competing for influence.
And Central Asia is one of them. Um, another super interesting place, which I have not been to, but I think would love to sort of see and talk to people on this score is Egypt, where Xi Jinping just went on a state visit. And the kind of questionnaire of a country that has deep, deep security ties with the United States, but is clearly making a major play to upgrade its relationship with China. Super interesting. And then in the Western Hemisphere, you know, I also think, like, whether it's Mexico, where there has been a ton of scale up of, you know, Chinese companies manufacturing, or Peru, where China has been building this megaport on the Pacific coast, these are really, really significant stories. And then finally, I would just say every year, the first trip abroad that the Chinese foreign minister takes for decades has been to Africa. And it is a place where China, I think, has a much more complex and differentiated and layered strategy than some accounts, you know, would have you think.
but whether you're going to countries that have really gone all in and see the Chinese development model as something that they want to ride for their own development or to countries that feel like they've gotten bad deals out of China but still know that they have to have a good relationship with Beijing, super, super worthwhile. I realize I have just essentially told you to go to every place in the world, but the upshot of that is that. I like an excuse to go everywhere. Yeah, exactly. Exactly. You should get you should get the podcast to pay for it. But no, I mean, I guess like to my mind, the upshot of all of that is China has a global strategy. It is pursuing it in many different ways around the world. It wants global influence. It wants that influence to look different than American global influence has looked. but there's no one place where you can go to to see it all because Beijing is doing so many different things in so many different places great well it's been a real pleasure Julian
thank you for coming on the show and talk to you next time
番組の概要欄(原文)
<p>What would it take for Beijing to take AI risk seriously, and what could a US–China deal on AI realistically look like?</p><p><a href="https://juliangewirtz.substack.com/">Julian Gewirtz</a>, historian and former senior director for China and Taiwan affairs at the National Security Council under President Biden, argues that any answer has to “start with the reality of what China is and what the Chinese Communist Party cares about.”</p><p>The Party’s top concern is regime security, so when Chinese discourse warns of “extreme loss-of-control risks,” it means losing political control as well as losing control of AI systems. In Julian’s view, what is most likely to shift Beijing’s approach to AI safety is China’s “own domestic sense of the risks.” Next is seeing the US regulate AI at home: a “costly signal” that Washington takes the risks seriously.</p><p>The US looms large in that calculus. Julian says Anthropic’s Claude Mythos has been the most significant event shaping Beijing’s thinking on AI risk over the past year or so, sharpening its longstanding concern about American cyber capabilities.</p><p>On a deal itself, Julian is a realist. Designing an ideal agreement in the abstract is where “the impossibility confronts you at every turn.” He would rather start where “self-interest can lead to action in parallel” — such as neither country wanting to lose control of AI systems — and build incrementally from there.</p><p>He is also candid about how little outsiders can see. Many Chinese AI companies are AGI-pilled; the government has been “much less oriented that way,” though Julian thinks “even that is changing.” He expects the clearest evidence of a shift to be material rather than rhetorical, such as centralising compute or buying more of the Nvidia chips Beijing has so far rationed.</p><p>Julian and host Tom Reed also discuss why terminology would be an early sticking point in US–China AI talks, whether China would disclose its own Hugging Face incident, whether Beijing is reconsidering open-weight models, which Chinese thinkers are worth reading even when you disagree with them, and how to judge China expertise.</p><p><em>This episode was recorded on October 2, 2026.</em></p><p>Chapters:</p><ul><li>Julian Gewirtz on the realities of AI risk in China (00:00:00)</li><li>What does Xi think of Dario? (00:00:41)</li><li>Would we ever know about a Chinese Hugging Face hack? (00:08:27)</li><li>How Beijing views distillation (00:18:56)</li><li>How do Chinese labs really view AI risk? (00:23:18)</li><li>Could Xi Jinping ever be AGI-pilled? (00:28:12)</li><li>What’s really shaping the CCP’s views on AI? (00:33:25)</li><li>Does anyone in Beijing believe in RSI? (00:38:32)</li><li>Chinese labs hate red tape, too (00:45:47)</li><li>Would China invade Taiwan over AI? (00:51:04)</li><li>The prisoner’s dilemma of a US–China AI deal (00:56:51)</li><li>How China stumbled into open source (01:07:12)</li><li>How to build a realistic deal, starting now (01:13:04)</li><li>How to spot a fake China expert (01:20:48)</li><li>Julian’s China and AI reading list (01:28:43)</li><li>The Chinese thinkers worth reading, even when you disagree (01:33:34)</li><li>Which AI model is the best China watcher? (01:39:27)</li><li>A world tour of China’s influence (01:42:59)</li></ul><p><br></p><p><em>Our production team includes:</em></p><ul><li><em>Video editors: Josh Alward, Dominic Armstrong, June Luithlen, Milo McGuire, Luke Monsour, and Simon Monsour</em></li><li><em>Producers: Elizabeth Cox and Nick Stockton</em></li><li><em>Coordination and support: Katy Moore and Lou Moran</em></li></ul><p><em>Music: </em><a href="https://open.spotify.com/artist/4lWobp6IUcSZ7w5mhnU1c9"><em>CORBIT</em></a></p>
出典・この記事について
- 番組
- 80,000 Hours Podcast(The 80,000 Hours team)
- 元の回
- Julian Gewirtz on the AI risks that actually scare Beijing
- 配信日
- 要約作成日
- 公式サイト
- 80000hours.org/podcast
この要約・解説記事は、元の音声を文字起こしした内容をもとに AI(Claude)が日本語で作成したものです。 固有名詞や数値に誤りを含む場合があるため、正確な内容は元の音声でご確認ください。制作方法について
関連エピソード

superintelligence.com創設者の「P(Doom)を0.01%未満に下げる計画」──Craig Kaplan博士とのディベート
Doom Debates!

ポール・クリスティアーノ――12か月以内に政府を転覆しうるAIが現れるかもしれない
Dwarkesh Podcast

AIリスクを下げる39の具体的プロジェクト:Holden Karnofsky(Anthropic)が語る
80,000 Hours Podcast

ジェンスン不在のオープンソースAI
Interconnects

AI、現代のエニグマ:Bletchley Parkで行われたNathanの基調講演
"The Cognitive Revolution"

ジェンスン退任とNVIDIAの後継問題、AI終末論への反応、ハサビスのRSI警告|MOONSHOTS #311
Moonshots with Peter Diamandis