ポストから政策へ：Saif KhanがDUVとAIバランスを語る

Safe Khan makes his China Talk debut. Perhaps my favorite example of posting to policy that I've seen over the course of my time following Washington. Safe wrote some truly excellent deep dives into AI and semiconductors in his time at CSET, then went into the NSC and Commerce into mid-2020s across both the Bryden and Trump II administrations to help manifest the export control regulatory regime we have today. He is now out of government and excited to announce the launch of the Center for Technology and Statecraft, his own think tank, which is going to produce. Well, I can I can personally confirm is already producing some excellent, grounded, technical, forward looking work,

which will certainly shape many an AI policy discussion in the years to come. Steve Kahn, at long last, welcome to China. Thank you. As you know, very, very long time listener and finally excited to be on the show. So safe. Why does the world need another thing? Yeah, that's a good question. Yeah. What is the appetite right now for evidence based policy? Well, there is right now, I would say. Surprisingly, a gap in the AI policy think tank ecosystem that we're trying to fill. Now, one thing is this is in the backdrop of a world where right now you have industry gobbling up lots of talent, foundations gobbling up lots of talent. And I think there is a lot of space to do lots of really impactful research and have policy impact at think tanks in D.C. So I would encourage people not to give up on that. But I would say we have our own special spin on what's important.

One thing is, you know, I go back to the days when I was at Georgetown CSET, and those were the days before anybody really cared about AI policy or even semiconductor policy. And we had all of the time in the world to think about forward-looking research topics, topics that would be real important in the coming years and have policy ideas on the shelf for when the policy window opens. Now you look today, obviously, every single day is coming with a new piece of news on AI, in policy, on chips and the like. And I think there's actually less space these days for forward-looking research. And so I think we are trying to fill a little bit of a gap in the ecosystem to just kind of put our heads down and try to come up with some new ideas that will be important on the one to two to three-year time frame. And I do think we have a really strong team to do that, that has experience with that and skill at that. And then the other piece that I think is what we're trying to be a central home for is I do think like some of the most productive policy research that we've had in the last few years has really been,

if you're understanding the AI supply chain, whether it's chip-making tools to tokens to robots and the like, understanding what is happening in the physical world is actually a thing that grounds research in a way that makes it, I think, much more impactful. And we talk a little bit about this in the launch essay, which you can see in our website, where really understanding physically what's happening in the real world of AI, it helps you identify patterns that other people don't see. You can also predict how the world is going to unfold. You know, I spent a lot of time doing semiconductor research, semiconductor policy research back at CSET. And, you know, there's a predictability to understanding supply chains. And if you move a policy lever, how the world will unfold. And also, if you are really looking at something that is empirically grounded, there's almost like a fractal like element. Right. Like you can dig into the details and you'll always learn more. It's time that's never wasted. You're not researching something where it's angels dancing on the head of a pin.

It's like real world things. And so I think we're trying to be a hub of that kind of work. And I think that that kind of work is still relatively underserved by people who kind of have the technical expertise to do it. So and all that said, you know, the forward looking piece, regardless of how you feel about the policy window right now, there are going to be points in time where it matters. so it's interesting thinking about the sort of evolution of this field and like what gaps remain because i remember having conversations with you uh when you were in commerce being like you know you're like jordan what's up with this china ai stuff i'm like i mean there's three people staring at it um and this was in 2023 this is no longer the case which is something um i feel like relief almost in a certain way where we've gone from you know four people whose full-time job it is and to like you know serve the english language speaking policy communities that looking at china to i don't know 30 50 more and the government has also clued in in a way which was not the case um

and the pre-dc days uh but there is a i totally see your point that like now that there's all this noise, the two sort of, you know, I don't know, let's call it like investing styles of doing policy work is going longer, you know, having longer horizons and then looking into physical stuff. I guess the thing I've always been scared about of investing, you know, six months of my life into writing a hundred page report is that when it comes to AI in particular, like six months is an eternity. Six months is like three decades. But I'd love to hear you expand safe on the, actually, if you follow the supply chain stuff, you can have a higher confidence in that whatever you are investing, you know, time and energy

and talent and really getting a deep understanding on is going to be relevant into the medium term and maybe we can use this as a transition to preview your guys' truly spectacular deep dive into China's use of DUV machines. Great. Yeah, it's a good question. I mean, I would say, you know, although you have a lot more analysts looking at China and AI and a range of other topics, I would say, honestly, there's still a range of pretty basic questions about the U.S.-China AI balance, and a range of other policy questions that are not addressed. Just to give an example of a few that we'll be trying to look into. One, for example, you hear a lot of talk about the U.S. having a model capability lead of something like eight months. And I think it's really not well understood why this is the case. To what extent is it that we are training larger models versus what is the quantitative

quantitative level of capability uplift that Chinese companies are getting from distillation or reinforcement learning? And what happens if U.S. capabilities progress speeds up or slows down? What happens to China? I think actually modeling what the future of the world looks like is actually fairly difficult and I think not well understood at all. And so that's one piece of the equation, for example, that we'll be addressing. I think you see examples of when, you know, a version of GLM 5.3 Flash comes out and and it's announced that they're producing tens of trillions of tokens. And it's like, okay, export controls are no longer working. Why is that right or not right? And I think a lot of these systematic explanations of just what's happening with the basics of U.S.-China AI competition are not well known. And I'll also give another example of an area that I think, a couple of areas that I think are just actually barely addressed at all. One is robotics and its intersection with AI. One of our researchers, Amelia Michael, is spending all of her time on this topic.

And I would say I could count on one hand the number of people in D.C. who are deeply thinking about this, let alone from the China side. And so I do think there's actually just enormous amounts of white space that we could fill if we do good research. And then the other piece of it is, yeah, there's a lot of arguments that now is the most consequential time to move policy. you know we're going to have fully automated AI R&D in maybe a year from now and all of the key policy decisions we're going to make have to be done by then I don't know if I fully agree with that I think a lot of big decisions that we're going to make are going to span across the next five to ten years in my opinion and I do think there will be dividends to doing that research ahead of time and shaping how things go. And also, one of the things that I guess getting from my experience working previously as a fellow at a think tank and then going into government is that the hope is that, you know,

some people develop their expertise at a think tank and then bring that with them when they go into government later and then now have the worldview and the tools to implement policy more effectively. And I have really seen that people who spend some time doing that before a government policy policymaking tenure really have the opportunity to be more effective in their job. I mean, I think there's especially if you look at, you know, the National Security Council, where I spent, you know, a couple of years. It's an area where you have day to day emergencies where you have to respond to what is happening right now. And I think it's really, really easy to just caught up and be caught up in your entire job in just responding to the day-to-day as opposed to proactively advancing a policy agenda where you've worked out the details in advance. And if you haven't done that, you just won't have all of the material you need to be able to do that effectively. And certainly when I was in government, you know, the amount of expertise that generally in these areas that you could rely upon, say,

from partners on the outside, whether it's, you know, federally funded research and development centers or other organizations that were meant to partner with the USG, you know, you can always rely on that kind of expertise for the forward-looking new policy ideas. But I think there's no replacement for kind of having that expertise in-house and having developed it yourself and having your own worldview for getting things done. I mean, but why? Because, like, you just missed the forest for the trees. You defer to industry. Like, what's the failure mode of not, you know, really as as an individual in the seat, like having your own, you know, history with the topic and background and views on it? Yeah, I mean, I think the one thing I already said is just the time. And you have to have a proactive worldview in order to execute on in the first place. And then the second, I think it is true that some of these topics are among the most technical policymaking topics in the world. and you need to have an ability to have an independent view

and adjudicate between different opinions that you hear from the outside, different industry actors who have different opinions. Now, there is a role for just gathering some of that knowledge directly from, you know, folks on the outside, but you just do need to have your own expertise in order to adjudicate on it. But, yeah, I think that's certainly another factor. Yeah. So let's show not tell. this question of DUV, how did you guys, or why don't you frame it first and then talk through the approach you took to understanding the policy question at hand? Sure. So our first major report, which should be published by the time this episode is out, is essentially a very, very deep dive into DUV immersion lithography. Now, I think probably long-time listeners of your podcast will probably be familiar with the concept that lithography is just kind of the fundamental kind of choke point technology and necessary and most complex

technology that drives semiconductor manufacturing at the leading edge and is also responsible for making the most advanced chips. EUV lithography, extreme ultraviolet lithography, is the most advanced tool in the world, and it's used for making the most advanced chips. The next most advanced lithography tools, and these are tools that are used to print the small features on chips, is called DUV Immersion. It is a tool that is still used today for making advanced chips. Huawei in China uses these tools to make their most advanced chips using what's called seven nanometer technology. And so it's still kind of a critical lever for shaping AI capabilities across nations. So, yeah, I think we've kind of done a deep dive to just kind of look at the state of play and how exports of DUV immersion technology to China are shaping the competition. Right now, it's still the case that two countries, two U.S. allies, Japan and the Netherlands, are the sole producers of these tools.

And it is really the most effective choke point that we still have available. China can't make it. And it is really the hardest tool to indigenize in the supply chain. In this report, we talk about by looking at a lot of historical data and also the cost and complexity of these tools, that it's likely that they won't have a commercial indigenous tool in China by probably something like the mid-2030s. And it's probably the tool that they will be the last to indigenize. There's all of these other tools in the supply chain, whether it's deposition tools or etch tools. They might be able to indigenize these earlier. So if you're really looking at the long-term prospects for U.S. AI leadership, this is the tool that will be most important. And go ahead. So in reading this report, which is 120-some-odd pages in the Word doc, we'll see what the PDF turns into. It just, like I read a lot of AI text nowadays, right?

And it is so refreshing to see something where it like a really sharp human being has logically thropped through like all of the questions that you want to have answered in order to, you know, understand this question of what does selling this machine do to, you know, China versus the rest of the world compute and shows you all of the work. and like usually I say this for nonfiction books which like someone's written for 10 years where like I learned something on every page like I genuinely learned something on every page of this report and the amount of think tank reports you can say that for as someone for me in particular as someone who's been like doing this stuff for a really long time and has read a lot of these reports on a lot of similar topics is extremely low like I literally have to probably go back to like some Will Hunt and SafeCon classics from, you know, 2020 and 2021 to have gotten like a hit as as as direct in this.

So I encourage folks, if you're interested in DUV or you just want to know what good work looks like to spend some time with this report and really sort of dive into some of the some of the details here. Like, it is a teaching document and a primer as much as it is a legal case for why America should go down policy A of passing the Match Act and banning UV exports. So I just want to, like, hats off to Safe and the crew. Like, you guys are starting off with a banger here. This is not a – the funders are getting their money's worth, Safe. Yeah, and a few more words on that. The vast majority of the credit here goes to Nick Brown, who's a senior fellow with us, who's the lead author on this report. This will really be his major report debut, though he's been an operator in the ecosystem for a while, has spent time at Google, Rand, worked in commerce with me.

But I'm really excited for, you know, the world to see his amazing work. And just to really, really quickly kind of give some of the quick argument, you know, we obviously in 2023 did have some controls that were put in place on these tools. The challenge is that some less advanced UV immersion tools are still allowed to go into China. So if you look back to around the time when those controls were first introduced, there were probably like upper like 100, like maybe like 170, 180 tools installed in China around that time. But because we still allow some of these tools to go, that stockpile has roughly doubled since then. And so it's vastly increasing China's capacity to make chips. It's possible, even if you're not selling these to certain facilities, that they can get diverted pretty easily. And then we actually do a number of kind of steps of modeling here, looking at the yield of these tools in making AI chips at different nodes, throughput analysis, installed base analysis and the like,

and then ultimately come up with these scenarios where by the mid-2030s, we would expect the U.S. to have, with pretty significant error bars, something like production of a billion H100, NVIDIA H100 equivalent chips per year. If we stop the flow of DOV immersion tools today, we project that China is probably going to be capped out at something like 10 million per year in their indigenous production. And if they mobilize, maybe in the tens of millions, if they are able to indigenize all of the other tools in the supply chain, if they're going to choose not to make smartphone chips and the like, that's an edge scenario, but that's still only tens of millions. Whereas if you continue to export these tools, there's a plausible world where the uncertainty ranges of U.S. production start to overlap with Chinese production. So we could potentially give that entire lead away. So this is actually possibly the most consequential lever for shaping the U.S.-China AI gap into the 2030s.

And then we have some policy options that we present as part of the paper as well. Yeah. So, you know, a lot of – it's easy to make a graph with cones. It's hard to make a graph with cones that will convince me that the cones are actually, you know, real. in their sort of like standard deviation confidence intervals. And, you know, you guys have done the work to make this case pretty close to Ironclad that like stands exporting these tools. The rest of the world will continue to keep its like, you know, 10X-ish edge over what the PRC is. Or maybe even 100X, I would say. Okay, as a floor, our 10x edge on sort of like compute production, and that is something that could scale up more dramatically in other scenarios. So I guess the meta question, Saif, is like, had you –

like there were lots of people who have been arguing for DUV controls without this report having been written. So like who out there are the like I totally get your argument of, you know, this work is as much for developing talent to do it as it is to sort of like persuade the masses. But like, who do you like what's the what's the policy pathway for real work like this to actually kind of, you know, dribble out into the ecosystem and make change in ways that it wouldn't like what's the what's the counterfactual for actually like putting all the evidence together to say band UV as opposed to just, you know, doing it off vibes, which I guess you may argue is is what this policy is, what this community has been. been doing for the past few years? I mean, frankly, I mean, right now, I think you already mentioned this issue is being hotly debated in Congress right now. There's a bill called the Match Act,

which generally addresses semiconductor equipment exports to China and includes potential controls on these UV immersion tools. And I would say there's, you know, a lot of smart staffers, a lot of smart members who are asking pretty deep questions about will this policy work? is it worth some of the costs? What are the costs? What are the benefits? What will this actually buy us? And I think it's actually not very well understood. And you have dozens of members and senators who are going to be debating this bill. It may go into conference soon for the NDAA. And I think having better information to inform this debate could actually be really consequential, whichever way it goes. I think people having the best information, I think, will really improve the chances of a good outcome here and tailor the bill correctly. So that's one piece of it. And I think even if, you know, there is an action, this Congress will have another year to kind of debate the issues further. And I would say also, obviously, you know, there's been a shift in export control policy in the Trump administration. But you look back to the AI action plan last year, where,

you know, it was very clearly noted that closing loopholes in controls on semiconductor or manufacturing equipment are a priority for the administration's AI policy. The administration has not rolled back any of the existing controls on semiconductor equipment. And frankly, as kind of the U.S.-China AI competition heats up and things have been changing quickly, even the Trump administration's own AI policy looks very different from what it did a year ago, certainly on the AI safety and security side. I do think that this is an issue that could be taken up by the Trump administration as well. So absolutely, there's an opportunity for near-term impact. But the other key point on this is, you know, while every day that we don't issue additional controls on these tools, it does give up a little bit more leverage that we have in shaping the U.S.-Chinaic AI balance. This is a problem that is unfolding over a span of years. And so even if we institute additional export controls in a year or two years from now, I think that is still incredibly impactful for the long term,

even if you think there's not a policy window right now. How do you read the critique that the Biden administration was, like, over-biased towards studying things and kind of wanted that 120-page DUV report on anything before it could come to a consensus? Like, is there a failure mode here where, like, like, because there are some topics that lend themselves to this type of thinking, right? And there are others that don't. Yeah, I mean, it's always worth striking the right balance on doing the right level of research that you need to get the right answer. And in the policy world, it's not always about getting perfect mathematical precision. Sometimes you 80-20 it. I would say we probably did more than 80-20 on this one report because it is just like so important. So maybe a 95-5 is worth it in this case. To your question about the Biden administration, yeah, I mean, it's certainly true that there's a failure mode where you get caught up in analysis.

And I would say, you know, this is like plausibly true in some Democratic administrations where, you know, you love having lots of meetings, lots of interagency meetings, lots of careful work to gain lots of consensus, doing lots of analysis, writing lots of paper. This is good and bad. One obviously very bad thing here is that it can slow you down from getting the correct policy implemented. and then you might miss getting certain things done at all. On the other hand, I think there is like certainly a virtue in, you know, doing all of the work to gain broad consensus and doing it based on analysis. And I do think, you know, there is it can oftentimes be a feature that you can get lots of different voices in a room who have different views, different worldviews, different incentives, all ultimately agreeing on the same policy. That means it has less downside than maybe some of the other policy choices you might have made. So I think there are virtues here. But yeah, as I mentioned, there's drawbacks in this case, though. Yeah. I mean, arguably, this is a policy that should have been completed years ago. Fortunately, there's still time for correcting the course.

Another kind of like mimetic question. So I had Caleb Wani, who runs IFP on a few months ago. Say if you are you and your new organization is IFP Spawn. and like thinking back to the current, you know, pacing the frontier debate we have now, like IFP put out this completely reasonable 25 or what do you guys get up to? Like 23 point plan for if you want to pace the frontier, here's all these other like things that make a lot of sense in a lot of scenarios, which I like disagree with basically none of. Also, you can look at the past six months of where this debate has gone and what has driven it and point to Bernie Sanders talking about how AI is going to kill us all. And a handful of folks at these companies also coming out and, you know, screaming and tweeting that AI is going to kill us all.

So, look, I think folks having listened to you for the past 30 minutes have a bit of an understanding of your personality, of the, you know, more oriented around the empirics and analysis. But how do you sort of see this type of work, like, interacting with the, like, Overton window stretching, like, meme stuff that is bubbling up more and more in this space? Yeah, it's a great question. Before answering that specifically, just a word about IFP. Yeah, IFP spelled out the Institute for Progress. Yeah, they've been an incredibly helpful partner here. We're a project that is essentially sponsored and supported by IFP. And, you know, Caleb, Alex Stapp, Tim Fist, all of these folks have just been, you know, absolutely essential to helping us get off the ground and we're continuing to be close collaborators on research and policy. And so really, it's the best thing tanks in the world. So really great to kind of remain

affiliated with them. And I also, by the way, want to call out Leonard Haim, who's another one of our close advisors, I think also a friend of China Talk, who's been a really close advisor and helped us get off the ground. Now gone off to Greener Pastures at the OpenAI Foundation. To your question about this report that we put out. It's called, How Should the U.S. Respond to Increasingly Automated AI R&D? And yeah, automated AI R&D, or it's also called Recursive Self-Improvement, is on the horizon and will pose huge risks. What are all of the preparatory things that will put you in a better world regardless of what we do later? If we end up doing pacing, we'll be able to do it in a more surgical and targeted way rather than being caught unawares and taking a more draconian approach when suddenly the political system wants us to respond really aggressively to AI risks. And then if it turns out not to be a thing, we'll be happy if we did all of those policies. I think there honestly are many places where you can find positive compromises

between people who have extremely different worldviews. And I think that's what we tried to do with this report in identifying these 23 low-regret policy options. is the thing that's going to be the most important 10 years from now, the low regret move. And like there's got to be some medium or high regret, you know, things like moves that potentially have medium or high regret payoff matrices, which are like the right things to do. And I think, you know, this is, look, I remember having a conversation with someone in fourth grade being like, somebody's like jordan like why are you always the guy who says like can't we all just get along and like i this is this has stuck with me like i also have this this this this um temperamental bias to not like scream and shout about a thing which you know half of this audience is going to be furious at me about or what have you but as someone who's deeply sympathetic to the ifp worldview and the safe kind of like standards for empirics,

if there are also going to be sorts of questions which deserve both the sort of like rigorous analysis and empirics but also have like a kind of scarier more potentially regretful payoff matrix that attached Yeah, I mean, listen, I think there's room for both. One is we need to find all of the incremental steps we can make. I mean, I like Caleb, you know, as he took on his new role, wrote an essay, I think, quoting John von Neumann, saying that like lots of small decisions add up to a better world. But frankly, listen, we just talked about, you know, export controls that were implemented in 2022 pre-ChatGPT. I think at that time, nobody would have called those policies incremental steps that weren't a little bit risk taking. And there will be times when, you know, we have to move preemptively and, you know, pick a policy that, you know, is going to move the Overton window a little bit. Speaking of stretching beyond, you want to talk robots for a little bit?

Where are you on robot safety? Yeah, I mean, so one of the we talked a lot about China, but we have this additional policy pillar. And just to say a little bit about how we structure things, there's that kind of supply chain, compute, AI production function spine that underpins all of our research. Then there's a China work that hangs off of it. And then there's a whole other piece, which we're calling automation and the social contract, which includes a number of different policy areas. And some of it actually just gets outside of the national security space, which I spent a lot of time over the last few years into economic policy in an age of advanced AI. What is going to be the nature of relations as AI systems are starting to perform nearly all aspects of labor that humans are able to perform? And what are the relations between them and between humans? But I think one undersold part of the story about how things are going to go with AI over the next few years is that, yes, we might get recursive self-improvement, as some people say, in AI and extremely powerful AI capabilities in software.

But the real way that the world is going to transform is through physical automation and robotics and marrying AI capabilities with robotics and starting to deploy robots at scale, having them learn how to do manufacturing and start to ultimately, even at some point, perhaps create self-sustaining supply chains. And I think these are really weird and interesting worlds that we could end up in. And I think this is like relatively unexplored in DCU now. Certainly start to see some policy interest in the China competition side. See, the FCC took some action with an import ban of some Chinese robots. But I think there's a broad set of questions on how to manage the U.S.-China competition in robotics and advanced manufacturing as it relates to AI, where we don't really have a lot of policy ideas, and it's going to be kind of the key nexus for U.S.-China AI competition in the, I would say, fairly near future. There's also going to be, I think, interest in next steps in industrial policy.

One of the other issues that I worked on at the NSC when I was working on semiconductors was the chip stack. That was obviously seen as one of the core areas of industrial policy that the United States needed to address a core vulnerability. And I think robotics is likely going to be the one that we're going to be debating fairly soon and what the industrial policy strategy would have to take. And then also we have lots of debates over AI regulation. So, Safe, you have this framework in your launch essay, this idea of a security mindset versus diffusion mindset approach to AI policy. How is this a framework that currently explains everything I'm seeing on Twitter? So I think there are just very different worldviews in AI policy. This is not a surprise. We've tried to kind of describe two fundamental ways that we see people looking at AI policy.

One is in the security mindset, kind of you have the most important actors really being frontier AI companies that are developing really powerful AI systems that are going to pose catastrophic risks or just kind of change the balance of power in the world in a significant and very discreet way, oftentimes outside of existing institutions. Whereas I think there's another worldview that much of what is going to happen in AI and the most interesting things that are going to happen in AI more so revolve around kind of widespread adoption of AI and kind of a co-evolutionary process with existing institutions. And a lot of this relates to kind of economic policy issues of redistribution between people and also the role of AIs in the economic system, especially as they become capable of performing lots of tasks that humans now perform. And I think one thing that we try to note here is that these are not necessarily opposing worldviews

views, because in order to have a good future with AI, we need to solve all of these. We need to solve security disruptions to the way our institutions work today. We need to make sure that AIs are not used to create bioweapons and so that lots of people die. But even if you solve all of those problems, there's going to be fundamental questions about the role of humans in the world when you have advanced AI systems that can do all of the things that humans are capable of doing. And I think like you've just not solved the problem unless you address that piece of it as well. So I think a lot of the work that we do is going to be infused with both of these approaches. Yeah. And maybe this comes back to like my when are empirics useful or when is like deep, deep, safe style empirics useful is it's it seems more like the security mindset stuff or like the national national competitiveness, national security stuff. Like the lines toward the 100-page report, which answers very tactile questions, like I can see, and I've read them in the past.

the quality of the thinking and the solutions I've seen so far for the, you know, how do we adapt as a society? Like, what does it mean to have robots running around or have AIs taking large, you know, driving rapid productivity, whatever, what have you? I mean, they're looser questions to begin with, but you also kind of like very quickly get into more kind of, you know, moral, philosophical or just like the moral, like the you see, I'm going with this. Like they're they're they're they're they're more expansive questions in the diffusion mindset bucket. Yeah, very expansive. I think also very much amenable to quantitative and technical analysis and policy analysis. I think one thing that's actually missing that I think we hope, at least in the long term, to marry, although this would require people as well, is, you know, you have a set of really AI-pilled economists

working at places like Stanford or the Anthropic Institute and now the new DeepMind Institute. And then you have policy folks in D.C. who are thinking about economic policy for the AI transition. And I think both of these things actually do not live together. Like a really deep technical understanding of where AI is headed, what happens if AI has actually just become what is in the technical terminology in economics, like gross substitutes for human labor. What happens to the way that our social system is going to work? And I think there are actually potentially technically informed policy answers to this question that are being explored outside of D.C., but those ideas are not being translated into policymaking. So let me throw you a sort of like Anthropic Institute pilled question. I'm workshopping this take about when our rogue AI gets loose instead of just like turning us all into paperclips. It'll actually want us to keep us like, you know, us like sad sack white collar employees out there, you know, spinning along.

An unemployed populace is one that is much more likely to bomb the data centers than one that is, you know, living, eating steak like the guy in the Matrix and, you know, watching a woman with a red dress walk by. Like, is there a future where the AI, you know, goes and turns itself into God, but just, like, you know, keeps us as nice little playthings? Because we, as human beings instantiated in the planet, like, do kind of have a bit of a veto on its continued existence? You know, I think it's a fun question, but I think it, like, actually probes it like something serious that we're going to have to grapple with, which is, you know, I think especially when you talk to, you know, folks in AI safety circles, you know, you hear a lot of talk about things like AI alignment, where you have AIs that are going to do the things that you want, that humans want them to do. Then you have ideas of control. How can we keep control over AI systems? But in a world where you have AIs that are actually running portions of the economy,

possibly the future that we need is one where there's kind of mutual dependence and reliance. and so that there aren't bad incentives for the things that you describe for those to happen. And so I do think like policy is going to play a role in that. I think it kind of goes both ways, right? Like humans need to retain some kind of fundamental role so that we retain leverage and have a place in the economy and in supply chains. So I've been reading a lot of Lu Xun because everyone should read a lot of Lu Xun. And he has this famous short story called Kongi-ji. and it's like 15 pages. Everyone pause the podcast and read it. But basically, it is this story narrated from the perspective of a bartender in like 1920 China. And there is a man who's a failed examination candidate. So in the Qing Dynasty, you know, you got to learn all your Confucian texts. And then if you, you know, pass the first level,

then you get to take the second level tests. And until you pass the third level, like you're kind of broke and have no role in society. But once you pass the third level, then all of a sudden you're like a, you know, scholar, king, demigod. You get all the money. You get 50 mistresses. You sort of you're on the top of the social food chain. But once the Qing dynasty falls, once, you know, we get to Republican China, all of a sudden all of this Confucian learning is like useless. And so this guy is poor, is, you know, homeless. He lives on the street. He he has this like golden heart. So he's like trying to teach the bartender how to read. He gives these kids the beans and is teaching them different characters. But everyone is just laughing at him and making fun of him. And then one day he comes in and his legs are broken. And he's like, I would just like a glass of wine, please. He drags himself on the ground. So he's covered in dirt.

And then everyone still laughs at him. And, you know, they give him one glass of wine and make fun of him, and he walks out and never comes back again, presumably dies. And I don't know if AI is going to turn us all into Qing Dynasty scholars in the wake of – into Republican China. But I do feel like there is some echo there of at least, like, you know, there will be segments of society and the workplace which, like, used to have high status, you know, material comfort and all of a sudden won't because of whatever's happening, RSI or not. And that just like, I don't, I hope that there's like a nice empirical way to grapple with this and the wealth transfers and the worker retraining. And, you know, maybe we try to get AI to grow in X way as opposed to Y way. But this is like these crazy transitions have happened without, you know, intelligence too cheap to meter.

right um so it's not like it's not like we haven't come across this um before and in the past you still had sort of like really dramatic social consequences whenever you um run into this sort of you know mismatch of where society is and what technology brings to it there's a lot of a lot of people in humanity if you pluck them out and fast forward to them 50 years or 100 years would like be totally shit out of luck. Right? Like would not like they're either, you know, they go from okay, okay, you throw Thomas Edison in like 1990, he doesn't know how to type. Like he's going to have a tough time. Presumably. It just brought home some of the potential dislocation of what we may be approaching in the coming years in a way that I hadn't internalize that. Anyways, we'll pre-assign you your Lucy before your next China Talk appearance. Let's talk about

AI powering research. I mean, this was not a thing that you really had in your CSET days. Like, how do you see these tools changing the way you and your team work? Yeah, I mean, I think we're definitely taking a very AI-forward approach, certainly data-forward approach as well, where we're going to purchase lots of data sets. And especially with some of the tools that we have had over the last, I would say, like nine months or so in the coding revolution, Claude Code and Codex. I mean, I would say, you know, the ability to do a report like the DUV report that we just talked about earlier and the level of technical analysis that we could do there, I think hugely benefited from the ability to query lots of data and do lots of analysis fairly quickly with a small team. And so I do think, you know, what used to be think tanks of like 15 or 20 people, a much smaller team can just have that level of productivity and depth of analysis And so that is definitely an area where you know we willing to spend significantly given the efficiencies that you get

How do you feel like these tools can and should change, you know, NSC decision-making processes? Man, that's probably one of the last places that you're going to see penetration of coding tools is places like the NSC where just kind of getting the IT systems working in government is kind of a basic challenge. But yeah, you know, one, you know, we talked a little bit earlier about like the level of kind of analysis that you need to do very quickly for policy decisions you're making. And I do think that like probably the speed at which you're going to be able to do the analysis you need and collect the data you need will be greatly accelerated by this. There's this maybe an even bigger question of politicians kind of going to AI. Yeah. To, you know, to decide how they vote or how they're, you know, how they're going to frame something. And it's kind of already happening already.

Right. Like you have these reports of like 70 percent of like speeches made in, you know, the House of, it's not the House of Burgesses. Jesus, in Parliament are like, you know, AI assisted or something. You know, it's a weird future to consider where, yes, like you want people to be, you know, diving into Safe's UV report and doing the – and coming to their own conclusions. But, like, I imagine there's plenty of, like, I think AI is, even today, much less like three years from now, if you gave it, if you define the goal, would probably strictly dominate, like, 80% of civil servants in the things that they kind of do and the regulations that they write. And that has to be going up over time. So it would be like, on the one hand, it's kind of a shame if the rest of the world is going to get better decision making, but the NSC can't because like, you know, it's just like scared to, you know, update from Windows 95.

But there are obviously some like dangers and concerns of turning over big swaths of public sector decision making to, you know, models. Yeah, I mean, I think one thing that's interesting about NSC in particular that was a frustrating aspect is that by statute, the number of staff that you have is just limited. And so you literally have, you know, NSC directors with a significant policy portfolio that have to be busy with all kinds of administrative tasks. You're obviously doing your own scheduling. You actually have to do enormous amounts of writing and analysis. And I think all of this could just be a lot more streamlined. I think you would dramatically improve the throughput of NSC staffers if you're able to get AI to do a lot of just the basic operational tasks that take up so much of the time at the NSC and like the reams of paper that you have to generate. Obviously, you know, you can't just have AI generated policy documents like writ large, but I think it really could speed up the amount of work that you could do.

So I think that that gives us that gets us to like like a 3x safe con. But is there a world where we get to like a 10x safe con? I don't know. Like what would what would what would you need around you from tooling as well as like, you know, organizational bureaucratic reform? Perspective. yeah i mean i guess until ais are actually taking the meetings for you 10x seems pretty hard unless you just have ais running the show which i'm not i'm not sure we're quite ready for that one fair enough um other things we should cover i want maybe like let's talk about talent development a little bit like how do you um maybe talk a little bit about your sort of like upskilling into the knowledge base you have currently and how you're thinking about doing that for your staff. Sure. So one thing I really liked about working at Georgetown CSET around 2019 to 2021 as

a startup think tank was that there was just an unusual amount of space where we were well funded and had the time, few distractions to just really become expert on a topic. And I think it's really underrated the degree to which, you know, unlike, say, doing a PhD where, you know, you have to go through 10 years of schooling to get to the knowledge frontier. In policy, you could really dive into something for a year and start to hit the knowledge frontier and start putting out material that people really want to read and nobody else is writing about. And so if you're just given space like that, I think it's a huge opportunity. and, you know, it's just the case that in a lot of think tanks, you just aren't given even that level of space and you have to be responding to all kinds of incentives. There's requirements from funders to be able to be doing roundtables or to get quick turn reports out. I think having an environment where you're able to just deep dive and become an expert really is quite valuable

and especially in areas that are going to become important. If you are one of the handful of people and then the government needs that expertise, you will be one of the people who is kind of well positioned to go into government and cover that topic. So I think that is actually kind of a motivation for kind of creating that environment now at CTS, where researchers will have time to develop that level of expertise and then be the person that people come to. Another element is, and this is something that I think IFP does really well, and we're also following their lead, is the idea of having full stack policy researchers. So there are some places where you have some people who are writing the reports, but then you have staff that are responsible for comms or ledge engagement or the like. But I think there's a lot of value with the same person being the person who develops the idea, writes the research report, talks to policymakers about that idea, and shares the findings.

Only the researcher is going to be well-positioned to kind of fully provide kind of the depth of a briefing that's needed. And then lastly, also just by having your pulse on what policymakers are thinking, it can help you prioritize on your own work. And so that's another piece that I think is really critical. Yeah, I think the last one maybe folks don't quite appreciate is like when you have the conversations and you end it with, well, what are you thinking about? Like often like that's where you get your next idea often. And and and without that, you're sort of just kind of kind of swimming a little bit. All right. Let's do let's get some book recs. What other reports have you enjoyed recently? I don't know. Want to do parent corner? Is there seven-year-old content that's been speaking to you of late? I don't know about seven-year-old content, but I guess one book that's always been one of my favorites

and an underrated way to think about AI policy is actually a cultural anthropology book called The Secret of Our Success by Joseph Henrich, which I love to recommend to people because I think the thesis there is that the reasons that humans have been very successful as a species is not necessarily our raw intelligence, which is not that much better than other apes, but rather our ability to be social and transmit cultural knowledge. And they have really interesting stories about European explorers kind of going off to new continents and not being able to know the precise steps to cook the local kind of plants or roots and the like and not being able to survive as a result of that. And a lot of that information actually just being embedded in cultural knowledge that was learned through trial and error. And I do think a lot of how the future of AI is going to go is will involve some level of bottleneck on kind of real world experimentation in the physical world.

This is kind of why we think about automation and robotics as kind of a really important pillar of how things are going to go. AIs are going to have to learn how to do things, go through trial and error, just like humans have done in order to learn new skills and adapt to different parts of the economy. And then also, interestingly, the OpenAI Hugging Face recent incident that kind of where you had a number of agents kind of escape containment from OpenAI servers and then go hack into another company, Hugging Face, actually involved a swarm of something like 10,000 agents. And it really kind of was an episode that involved lots of social communication and learning, which was I think a demonstration of what you can kind of see in Joseph Hendricks work. So that was pretty cool. But I mean, I wonder with the agent stuff, if there won't just be like a better mind meld that they'll get to at some point, then like, well, I don't know.

Like they could also have like a queen bee worker bee dynamic. Right. And they'll just, you know, they won't need to like talk it out and deliberate or maybe, or maybe I guess as humans having conquered the planet, there's something like, maybe we're onto something about having like relatively equal intelligences like deliberate to like come to what the goal should be. No, no, I think you're totally right. The forms of social organizations that you're going to see with AI might end up not looking the whole like human social organization. And obviously there's yeah, many different examples in nature. I mean, you're referring to insects where you have, you know, creatures that oftentimes have more genetic similarity and act as kind of more of a single collective. And I think you could totally imagine that that kind of social organization is much more what you see than an analogy to humans. So, you know, we've like we've had like God emperors. Right. We have, you know, Athenian democracies like we play around with a lot of different stuff. So there's a wide range of patterns to match on.

Very true. So you mentioned this very earlier, but we should just get your take on the record. RSI is something you're perhaps a little more skeptical of than Dario Amadei. What's driving you on that, say? I don't know if I'm – I would not at all say I'm a skeptic. You know, the companies, you know, are in a really good position to know whether they are automating their AI research. and I think they actually plausibly in the next couple of years could succeed in substantially automating a big portion of it. I think the big question is what comes out on the other side. Right now we're on a capability exponential. There's a real question as to whether automated AI R&D and RSI is actually just going to be what's needed to keep us on that exponential or if it's actually going to bend the exponential upward. I think that is very hard to say right now, at least from where I sit. That said, I mean, it's fair to say, even if we stay on the current exponential, this

year we've been passing all kinds of milestones where you have autonomous cyber and agentic capabilities. We're now solving millennium problems in mathematics, the Navier-Stokes problem. And so that, I think, could still get us to really wild places. I do just have a lot of uncertainty about the real world impact, though. And I think that's something that we're just going to have to observe. So don't exactly, I wouldn't exactly call me a skeptic, but more like I have questions about the different futures that we might end up in. Let's take a home a little bit to close. Like when thinking about raising young children and education, like what have your experiences been with bringing AI into that loop and what anecdotes or reflections you have around where that might take us. Yeah, so I have two boys, a six-year-old and a four-year-old, and obviously they're deep in the stage where they have a million questions about everything

and the why and the why and the why, and you go all the way down until you're asking about the origin of the universe and deep metaphysical questions. And oftentimes I don't have an answer, and I have to turn to an AI voice chatbot, and then it gives the answer and then it just patiently answers all of the kids' questions. And I think at some point they almost prefer, because they get an answer every single time, lots of patience. And so, honestly, yeah, it doesn't work out badly. I mean, I wouldn't propose AIs taking over parenting entirely, but at least as part of that question chain, not so bad. yeah I mean this is the like the when we get the humanoid nanny you know because there's like the safety issue right but I think we're gonna I think we're gonna we're gonna cross the like is this is this better at like

day-to-day parenting and like being patient and like remembering to you know put the sandwich in the backpack way better probably way before we're comfortable with, like, picking up a two-year-old. But, you know, that day is going to come. I mean, it is weird having this, like, being on your phone, which, like, never gets tired and cranky and never wants to do anything else that, like, is not exciting for a four-year-old. Right. You know, I feel like we got competition a little bit. Yeah. And also, hopefully, sometimes when the questions come at night, the AI chatbots often say, hey, maybe it's time to go to bed. So it's actually kind of helpful. And with that, perhaps we should call it an evening on this wonderful edition of China Talk. Steve Kahn, thank you so much for joining. Many safe returns. What's that phrase? Can't wait to have you back when you bring more bountiful 100-page semiconductor breakdowns.

Thanks, Jordan. It was great to join.