Jeremie Harris が語る「本物のAI災害」後の世界と、中国との信頼構築が難しい理由（NonZero からのクロスポスト）

Hey listeners, today we're cross-posting an interview with Jeremy Harris from the Non-Zero podcast hosted by Robert Wright. I had somehow not heard of Jeremy Harris already, but I found this a super thought-provoking conversation. Jeremy just had a lot of ideas that were new to me, especially regarding warning shots and the likely government response, and also possible mistakes that have been made in AI governance in the past. It's all debatable, and I'm sure not everyone will remember the history the way that he does. I'm not sure that I quite remember the history the way that he does, but I think it's provocative in a really useful way. You might know the interviewer Bob Wright, or Robert Wright, because he's actually been a guest on the 80,000 Hours podcast before. He's also the author of a number of best-selling books, perhaps most notably The Moral Animal, While We Are The Way We Are. And he just published a new book about the arrival of AGI called The God Test, Artificial Intelligence and Our Coming Cosmic Reckoning. If you're somehow not already convinced that the creation of smarter-than-human machines is of cosmic significance and will lead us to a new era in the universe's history, well, go read Bob's book. Bob now also does very regular interviews about AI and AGI and superintelligence. He interviewed Nick Bostrom a couple of weeks ago,

for instance. One of Bob's strongest and most distinctive views is that if the US and China can't set aside their differences and coordinate to manage the arrival of superintelligence, then we'll probably just toast as a species. You'll hear him make some of the case for that perspective in this interview. Anyway, I really appreciate Bob letting us cross-post the conversation, especially the second half, which was the highlight for me, but has been completely stuck behind a paywall until now. So thanks, Bob, and thanks, Jeremy, and I hope you enjoyed the Cospost. I'm Robert Wright, publisher of the Non-Zero Newsletter. This is the Non-Zero Podcast. I'm also author of the recently published book on AI, The God Test. Excerpts are available at thegodtest.net, and you are Jeremy Harris, and you know a thing or two about AI. You are, for starters, co-host of one of my go-to AI podcasts, Last Week in AI, co-hosted with Andre Currenkopf, if that's not too much on the meaning of his name. And you are CEO and co-founder of Gladstone AI, which in 2024 released a State Department commissioned report, which I would say in retrospect, looks like it anticipated some themes that are more prominent in the current AI discourse.

I'm sure you would say the same. I would. I'm sure you would. In fact, it may be that you have on your podcast. And I'm not sure, though. I'm not sure. But I know it. I was aware of it at the time. And it was, you know, pretty much on my wavelength. And that's one reason I remember it. For one thing, there was the recognition that national security in the age of AI cannot be secured through policy at the national level alone. Ultimately, you're going to need what has lately been called international coordination. And the report does emphasize that, has a particular plan for getting there. It emphasizes, you know, the kinds of risks that we've been hearing about lately, you know, bioweapons, cyber, and the phrase the loss of control is in there. something that has become more prominent in the discourse since the hugging face incident.

So are you glad that, you know, regular people have finally caught up with your prescience? You might not put it that way. Well, that's I mean, I'll accept the framing begrudgingly. begrudgingly but yeah i mean i think it's um uh it's clear that right now the idea of rogue ai is is now sort of widely accepted because it has to be because we've we've literally seen it um you know it was so that report you know did come out in 2024 early 2024 it was the product of research that started in uh in 2023 actually the the report itself was commissioned by the State Department about a month before ChatGPT, before the eye of Sauron was on this issue. Like nobody in the U.S. government was paying attention to it, which meant that there was no pressure to bureaucratize, let's say, the output of the work, the report itself. And so that's really the only reason that we were able to go public with it when we did. And so there was this really funny

situation that unfolded where, you know, basically we're revealed to be working on this thing that suddenly became of massive public interest in the immediate aftermath of chat gpt um but it's too late for the government to like there's no take backs and so you know off we go writing this report that the government now knows will one day be public and there's all this like kind of political back and forth um because the administration at that time it was biden did not want terms like agi to be used in this thing because it seemed like science fiction they didn't want terms like loss of control to be included um and so we had like a pretty intense political fight behind the scenes to make those things appear in the report. We felt it was essential just because, as you know, I mean, Silicon Valley has been talking openly about artificial general intelligence and superintelligence as being the end goal of what they're doing. It's like the I wouldn't even call it the worst kept secret in Silicon Valley. It is the open and stated primary objective of the world's most valuable companies at this point are many of them. And so anyway, so we just wanted that to, yeah, become public. You know, I think the

one of the things in retrospect that it kicked off was it was, to my understanding, the first time that there were published in kind of mainstream outlets through insider whistleblower reports from the Frontier Labs themselves. And at that time, it was a lot riskier. Like, we had people who had, like, one guy, you know, took me aside after a meeting at one of the Frontier Lab headquarters, and he just, like, gave me his phone, and his hand was shaking. He was like, hey, can you put your number in my phone? Like, and what he told me at the time actually was, um, whatever your recommendations in this report are going to be, I would urge you to please be more ambitious. And later on, we met him like literally at a bar offsite just to kind of like get the rundown of what he meant by that. And he was talking about nationalizing the frontier labs. Like this was 2023. He was saying, you know, we're getting there. Right. So I think this kind of crazy world, the sort of almost science fiction scenarios or what seemed like science fiction then is now starting to just become mundane reality.

Like the Overton windows shifted so much that, you know, I think it's time to reassess a lot of these a lot of these proposals. Yeah. Let me say one thing in anticipation of our conversation about where you and I do and don't overlap, because I think this is what's going to make this an interesting conversation. As I've said, we both emphasize the eventual need for something that I would call international governance. You might or might not. Some people prefer other terms. But one difference between us is I've heard you describe yourself as a China hawk, and I've heard you talk like a China hawk. I've heard me describe myself as definitely not a China hawk. So it'll be interesting to see how, you know, how we navigate some of these issues together and what we do and don't agree on, on what's possible, first of all, and what's necessary and what the implications of all this are.

But, yeah, back to the discourse, it has been kind of shocking. I mean, I have – it's funny. So you wrote this report before some of this stuff happened. I put this book of mine to bed, The God Test, before the breakout, the open AI breakout. You know, and the trajectories we're seeing now unfold are in the book, you know. We already have like Malt book and various, and I talk about agents, and I talk about loss of control. In fact, I have a chapter called Hive Minds and the Loss of Control, although I didn't anticipate the specifics of this kind of scenario. So I've had this weird feeling of like, wait, so I'm prescient. Oh, wait, so no, it's too late to be prescient. It's like it's moving. The discourse is moving so fast.

and two things. First of all, people in this realm have for some time been saying, look, what's it going to take to get the world's attention? I guess we'll need a warning shot, by which they generally meant some kind of high casualty catastrophe. In that sense, the news is good. The over the windows moved a lot. Very low body count. I mean, you know, there have been tragedies involving AI, but as far as I know, nobody died at Hugging Face. And so that's all good. But I'm wondering what you think in terms of, first of all, what is the discourse still not catching up to in your view? And secondly, do you think it's still going to take more kind of jolting incidents of some kind or another to get the discourse to where you want it to be? Yeah, I think these are really complicated questions, but I think, at least in my view, they have a relatively simple answer.

So the simple answer is there is still somehow alpha in just taking scaling seriously. So scaling is this idea, I think by now it's sort of a household idea, but just in case it isn't, just this idea that if you want to make an AI system smarter, all you have to do, roughly speaking, is train it at a larger scale with more computing power and more data. So you take this model, this artificial brain, right? And again, you use more AI processors, more of these chips in a bigger AI data center with more power and so on. And now we're at the scale where we're doing this with gigawatt scale. A gigawatt is like a million homes worth of power, right? So these are truly like mammoth, mammoth facilities and sites. AI buildouts are responsible for the lion's share, if not basically all of U.S. economic growth over the last few quarters. So if you zoomed out, really, I mean, you would see planet Earth is slowly transforming itself, not so slowly even, into just a computing factory. And the economies are starting to reflect that. The markets are starting to reflect that.

Political pressures are starting to reflect that. And yet, this simple fact that scaling just works is still something that by believing it more than others, you can still, I think, make better predictions than the average person. You know, back in 2020, like the first time that my co-founder and brother Ed and I bet big on this was when we left our company in 2020. There was a paper that came out around that time. It was the first big scaling laws paper. Was that a company you had started? I'm just curious. Yeah. What was it? Yeah. So it was an experiment, and this may be – I don't know if this goes to our differences necessarily, but this was a, like, hardcore experiment in, like, libertarian economic theory. Well, that would be one difference. Not necessarily related to the China Hawkins issue, but yeah, I don't I'm looking for Milton Friedman on my bookshelf. Not there. Yeah, absolutely. So and to be clear, like, you know, as we age, our views become more nuanced. But this was sort of like a fun experiment in like how far could you take that?

And so he had this notion of income share agreements where you might get some get educated at some some school or whatever, get some training. and then the training could be monetized by taking a percentage of your salary for a certain period of time. So you don't have to pay up front and nominally incentives are aligned in that way. We set up, actually, it kind of became the world's largest marketplace for income share agreements and we would do kind of peer-to-peer. So if you wanted to break in, typically it was into AI or data science, you'd work with a mentor and the mentor would say, okay, I'll help you for free. There's somebody who works at Tesla or OpenAI. and in exchange for like 3%, 5% of your first year salary when you get hired. And so we had like thousands and thousands of these mentorships happening. It was a really cool experiment, and we got some really genuinely fascinating economic data from it that is I think to this day quite unique. But the scaling loss paper came out, and I still remember I had a two-hour conversation. So this is – was Dario on this paper or – this is – yeah.

When is this – and this is establishing, arguing that, yeah, all it takes is more compute, more data, and that certainly has gotten us – well, a lot of innovations have happened, let's face it, but that has partly driven what we've seen. Was Dario's name on that? Yeah, that's right. So Dario was involved in all the early scaling research. In fact, he was kind of like the champion of scaling. It's often said of OpenAI that they are really scale-pilled, and they are. But there are a lot of ways in which Anthropic actually is much more aggressively scale-pilled than OpenAI in different ways. And a lot of that comes down to Dario. He was the earliest believer in it, really, along with Ilya in a lot of ways. And he was at OpenAI when they first showed the benefits of scaling with the early tech UBTs. That's right, yeah. And so if you talk to the open AI people, so like Jack Clark will tell you, you know, GVT2 is kind of like the moment that he went, whoa, whoa, whoa, whoa, whoa. And really, like, maybe it's worth talking about just like what these things are scaling.

Because it's abstract to say, you know, they're like, you scale data, you scale computing power or whatever. But like the actual thing that you're scaling is basically a text autocomplete engine. And so all it does is it gets really good at predicting the next word. It sounds stupid, but text autocomplete ends up being, by some weird freak of information theory, this like really, really important task. And you can kind of understand why that's the case. If you think about, say, if I give you a sentence like, to counter a rising China, the United States should blank. right in order to fill in that blank in order to fill it in accurately you need to know what the united states is what china is what it means for china to be ascendant you need to know facts about geopolitics economics all kinds of crap that goes into just like filling in that that one next word and so this turns out to be a really powerful way of teaching ai models just about the world generally facts about physics about how to code about how to orient yourself in the world and so

I mean, if I could add one thing, you also need, if you're going to do that efficiently, and the process of training forces the machine to do it efficiently, you need a way of, like, for example, representing the meaning of words. And I think it's underappreciated the extent to which, you know, something that the brain presumably does. I think it's underappreciated the extent to which training is kind of the reverse engineering from language of some of, I would say, some of the cognitive functionality that in humans, you know, evolved to help us deal with language. And then, of course, there's the subsequent, the so-called post-training, where they do more than autocomplete. They teach it to answer various kinds of questions and so on. And I think the reverse engineering continues. And this is why, you know, the Stochastic Parrots people are just not getting the power of what's going on. We are without having to understand how the human mind works. We're kind of reverse engineering the rough equivalent of it.

And all you need is the right kind of data and a lot of scale. Yeah, that's right. Like it's and the analogies and disanalogies are really interesting here. And there's a sense in which, like, it's really easy to get get lost in in the minutia, you know, the Gary Marcusing of a lot of the conversation here, where there's always some fascinating reason to believe that scaling will magically stop. And perennially, there is some fascinating argument. We're hitting a data wall. We're hitting a compute wall. We're hitting an energy wall. We're hitting this wall, that wall, stochastic parrots, as you said, right? These arguments that often wildly outlive, at least like in the discourse, they wildly outlive their utility or their evidentiary foundation. Like we're way past stochastic parrots. I'm old enough to remember when the data wall was supposed to be a reason that you wouldn't get offensive cyber capabilities like Mythos. Like, we just don't, we don't seem to refactor our kind of credibility index when we're thinking, like, how seriously should I take a Gary Marcus? Well, you know, maybe, or a Yann LeCun, maybe we ought to look back and see what predictions they've actually made about the present moment before we go off and, you know.

And the economic version of this, too, which kind of drives me up a wall a little bit, is people who say, oh, well, this whole AI thing is overhyped. There's a massive bubble. It's going to pop. These aren't real businesses. And while it could be a bubble that pops, and I've heard you argue sort of similarly before on the show, I think this is exactly right. It could absolutely be a bubble that pops independent of the trajectory of the technology for any number of reasons that are mostly economic and not technological. but um anthropic has had its its first profitable quarters like if they want to they can turn this into a money machine the the race to scale right now is about who owns more market share it's the that's the classic kind of vc financing model um and and the margins for these companies have been going up like if you look at the margins for anthropic right now they're sitting increasingly in comfortable software as a service territory like your 70 percent territory opening eyes headed that worked that way too. And so far from dropping, again, I'm old enough to remember when the margins

were supposed to drop, everything was supposed to go dark. Like this is the opposite of that trajectory. And consistently, everyone has been surprised by how fast things have gone and not how slowly. I say this with some modesty because I called it incorrectly in the other direction where I said, hey, you know, frankly, I thought we ought to be dead by now based on my median nice to meet. So what the hell do I know? But yeah, I mean, that's the shape of it as I see it. I think we're accelerating faster than most people appreciate. Yeah. I'm going to add an asterisk, but not pursue it because I want to get back to where I initially derailed this by asking you what your company was before you and your brother moved on to your next venture. I will say, I hope we'll get back to this because it's relevant to the China question and whether you can slow down without China catching up and so on, which is that the challenge Anthropic does face is staying ahead of the commoditization, right? I mean, and, you know, the value of what it's doing now lies in the fact that it is clearly

its models are superior to much cheaper models. And that is, you know, one question mark is can it sustain that indefinitely? If not, what will plan B be? But anyway, I'm sure you have things to say about that, but I'm kind of feeling guilty about taking you off your original path. We should return to the part of your story, I guess, where your libertarian – one more asterisk. Ironically, some of your, I don't want to say enemies, but some people who drive you crazy are libertarians because they're still arguing that this technology needs essentially no regulation. Another thing maybe we'll get back to. We will not name any names. The name David Sachs will not come up. No. Mark Andreessen will not be mentioned. Will not be mentioned. But let me take you. So let me take you back to your where you were. Continue your journey. Well, yeah. I mean, so actually, I guess we've covered part of it.

you know, this paper comes out, Ed and I at the time are very happily working away in this like weird Milton Friedman style economics experiment, the paper comes out. And we have a sort of tense two hour phone call. And he gets the better of me. He's basically like, look, I've looked at this paper, the scaling laws, they seem to be holding across, it's like, seven orders of magnitude in computing power. So basically, if you, you know, train a model with with, you know, 10 units of compute and then you train it with all the way up to, you know, like 10 million units of compute, 100 million units of compute. And you just see this merciless, ruthless, unending straight line that will not bend. It will not yield. And this has a certain aesthetic to it that if you think about how evolution worked, like for evolution to work, you need to have a very smooth hill to climb, right? There's got to be, if you're going to work your way up the intelligence ladder through evolution, an increment of intelligence has to yield an increment of reproductive fitness,

right? And so it had to be a simple thing. The brain had to be something that was made up of simple components that could gradually increase in complexity over time, over generations. And when you look at a scaling law like this, I mean, the argument Ed was making was like, look, of course nature would be shaped this way. Like it had to be a scaling law of some kind. We just seem to kind of have unlocked it. Transformers, which is the architecture of the essentially artificial brains, the AI models that we were scaling in this way, just seemed to have the property that they were extremely resilient, that these scaling curves would not yield. And so, well, this was enough for us to go, when GPT-3 came out, which was, gosh, like, what, two and a half years before chat GPT, we looked at that and we're like, oh my God, like, not only do we have these scaling curves, but now for the first time, we have an AI that, bear in mind, was not ever meant to be anything other than a text autocomplete system. And yet it can code. It can design basic websites.

It can translate between languages. It can do all kinds of things it was never meant to do just by virtue of the scale of its training. You train on enough internet text, enough Wikipedia pages, enough textbooks. Eventually, if you give it a sentence like, below is a piece of code that checks the weather, colon, it will know, like, okay, well, I've read a bunch of blog posts online that have a description of a piece of code and below is the actual code. So I'll just write up that code. It could do that. And so immediately to us, this implied we were on a trajectory for human level AI and beyond. Immediately it implied that AI was about to become the single most important source of national security risk in human history. It sounded crazy then. We literally started going around talking to anyone who would listen in the U.S. government and other governments, U.K., Canadian, try to wake people up. And one of the things that happened, and this is something that I think is underappreciated about the kind of AI safety ecosystem of the time, there were a very small number of funders, of grantors in that ecosystem.

The main one, the Mac Daddy of the AI safety ecosystem at the time was Open Philanthropy, later or now Coefficient Giving, but back then Open Phil. And they were funded by Dustin Moskovitz, one of the co-founders of Facebook. and very kind of tied to the effective altruist cause that has become so kind of famous, infamous nowadays that maybe we can get into. I think it's an important part of this. But they had this view that if you spoke to anybody in the government, what would happen is that they would hear you. If you came to them and said, look, we're on the cusp of building human-level AI. We think we've cracked the code. We think the scaling thing has legs. You may not appreciate this, but Silicon Valley is talking about this as if they have cracked the code and they're investing like exponentially more every year. Like they smell blood right now. So, you know, this is about to become a real national security problem. We do not think we'll be able to control the systems that we build. And we will get rogue AI incidents in all likelihood out of this that may be challenging for the human species at a civilizational level.

If you pitch that argument to somebody in government, the open fill people argued at the time, this would trigger a very specific reaction. The U.S. government guys would go, oh, wow, I hear you about the capabilities. That sounds really compelling. Sounds like a really powerful weapon. I'm going to ignore all the warnings about the risk. And we're going to trigger a Manhattan Project for superintelligence that only gets humanity killed faster. Now, this is a very specific and it was expressed, by the way, with extreme levels of confidence. There was a paranoia about at the time doing anything that wasn't, as they put it, robustly good. And everybody ran around sort of like almost like worshiping at the altar of a small number of folks at the top of the pyramid who were typically the grantors. And I think there's a big psychological issue there where when you get used to doling out money, You don't realize that people start to change their tune when they talk to you to align with your pre-existing beliefs, and you sort of fall into a well of your own making and kind of surround yourself inadvertently. I mean, these are good people, but they surrounded themselves, I do think, with a lot of folks who kind of reinforced their pre-existing views.

So were they literally trying to keep the people in Washington from knowing kind of almost the central theme of their own discourse? I think that's a fair description with one little modification, which is there was at the time, and this is just my view, there's nobody here from their end to kind of give their explanation of it. I'll try to be as sort of even-handed as I can, but I am biased, of course. There was like a small number of folks at the top of that pyramid who were ordained in some sense by the ecosystem as those who were responsible enough to carry the message to government in their own way and in their own time. those individuals I had many arguments with refused to and strongly discouraged us to use the phrase artificial general intelligence they were part of, they themselves were putting forward this argument

and the case that we were making at the time was like look, I get that you have this concern but right now artificial general intelligence is not a thing that people know about and so you have an opportunity to get in here and shape the battle space around that terminology before it becomes politicized because it will. Anything that rises to high enough levels of salience becomes politicized. That's just a fact of the matter. And so this is an opportunity for you to kind of inform folks who are actually going to make decisions ahead of need. And they warned us against it. They actually said, look, there's also this whole kind of principal agent or prime mover risk where whoever is the most irresponsible will do this first. Please don't be that guy. And so the last thing I'll say about this, and I think this is kind of like, again, leans back into the libertarian startup founder side of who we are, which at our core is who we are. Our instinct in hearing this was like, okay, most startup ideas are terrible.

They feel really good. Like, we feel the truth of our ideas very deeply, and they're often just laughably wrong. Like, they do not survive contact with reality. The kind of startup impulse is to say, well, probably my product, and therefore almost most of my identity if I'm not careful, but really my product is crap. And so I need to put it in front of people as fast as possible in a low stakes setting and then iterate as quickly as possible. And so we thought to ourselves like, okay, well, let's just start talking to people. Let's see if the thesis actually holds up. Let's bother to just like talk to one person who clearly is not empowered to do anything with this information. And we did. And by and large, much like if you, if you look at polls of the U S population right now, and you ask people how they feel about super intelligence, how they feel about rogue AI, they're passing the test with flying colors. They are correct. Super intelligence is likely to be a massive risk to human civilization. And the public's intuition on this is basically right. Just like U.S. government officials positions on this were basically right when they were first introduced to it, at least by us back in 2020, 2021, that era. And so just by doing this and iterating on it, we very quickly found

Like, wow, this like massive money cannon that that these folks are holding on to is just pointed at a thesis that has not even been tested or at least that failed our test. And they continue to fail as we escalated the sort of rank and like impact of the folks who were talking to. And so that for us was a bit of a wake up call as to one of these, I think, deep pathologies, unfortunately, in the ecosystem. It's no longer, by the way, the view of these entities, but it certainly was back then. Yeah, and I was going to say, I mean, you were prescient in that sense, too, because they are now engaging in the Washington discourse. But I'm wondering, did they have a plan at that point as to how they were going to kind of circumvent normal channels of Washington influence and yet save the world? I mean, it's funny. I think maybe part of the answer to your question is this vibe you still get from people like Dario. I mean, not maybe explicitly, but you get the sense that they think, well, if we just are the first ones to build super intelligence,

we will, through alignment or some kind of, I would say, magical process, somewhat dismissively, because I don't think this is going to do it. But through some such process, as long as it's a good guy who does it, as long as it's Dario and not Sam, then we'll be okay. It was like a private sector. That's a very libertarian solution. We don't need the government at all. But was that what they were thinking? Well, so, okay, this is a really good and important and interesting question. I think one thing I'll just say, we made the choice once we started to see this pattern where we were hearing really confident. And by the way, like these like very confident calls they were making didn't just end at don't talk to government because this will be the response. They also included just like massive miscalculations on AI timelines and and not necessarily. OK, so AI timelines and also what you do about them. And so I had many arguments with folks in this ecosystem that, like, look, scaling laws,

I mean, I think this is the trend that gets us there to superintelligence. And there was all this sort of, like, speculation at the time about, you know, actually, Ajay Kotra, who I'll just name because I'm actually a huge fan, and, you know, she's over at Meter now, was involved in. She's very impressive, yeah. Yeah, that's right. Well, at the time, she kind of championed this bio-anchors framework, which was a way of thinking about AI progress through the lens of, like, well, here's how much time or how much compute, I should say, how much effective like kind of computation was involved in shaping the human brain and kind of leading to evolutionary intelligence. And it turns out when you look at evolution as the driver of intelligence and you think that that's how we're going to get there with AI, you end up with these very long timelines. So you end up thinking, you know, oh, let's like wait, you know, 2040, whatever. And because this ecosystem is very, it's still actually very insular in many ways, a small number of these kind of influential assessments steer the ship in ways that would be quite surprising.

And so we found ourselves wrestling with this framework, which, you know, AJA has since updated away from, and, you know, Open Fill, now Coefficient Giving, has updated away from. But this was, I think, a really important thing, because if you actually believed in short timelines, it would cause you to act. and that action would turn out later, I would say today, to be really valuable pre-positioning. It just takes a really long time to get to know the intelligence community, to get to know the Department of Defense, Department of War, to get to know all the stakeholders on the export control side of the house, to put together the pieces of the puzzle that you will need at game time. And if we had sat back resting comfortable with this notion of 2040 AI timelines, we would not have done any of that. And and in particular, we actually never would have done that State Department assessment, which like I don't want to overplay it, but like it a lot of the whistleblowers we spoke to since and very shortly after went public. I mean, seeing the result of, you know, actually, we can have this conversation in public like the media is not crazy about this or on this issue and so on.

And so I think that there's a sense in which, like, you can talk yourself into doing nothing. This is advantageous, particularly if you are one of the small set of people who are authorized implicitly by this ecosystem to be agentic, to behave with agency, to make the first move, to tell people when it's okay to do X, Y, and Z. And so anyway, I think that was like a big challenge in that ecosystem. Yeah, we definitely – so we ended up turning down money from all effective altruist stores. We have to this day taken no grants from anybody simply because we've suspected that at some point this might actually become political and we wanted to make sure we were clean. That, by the way, was an extremely painful thing to do. It meant literally standing up an entire for-profit business that would subsidize our crazy, unprofitable work trying to first understand and then intervene in the development of AGI. But it did mean that we, for a long time, were essentially the only independent people, as far as I know, with the exception of maybe folks like, you know, Zvi, who will write whatever the hell he wants to write and thinks.

But, you know, by and large, in terms of the entities involved in this ecosystem, they were all funded by this kind of group of funders, which is not like intrinsically a bad thing. It was the only money chasing this problem. Like open philanthropy, coefficient giving were incredibly prescient about this. It's easy to, like, get into, you know, open fill bashing, and I'm happy to indulge in that. But, like, I think fundamentally they got it much more right than almost anyone who's waking up his name. Yeah, yeah, yeah. No, I mean, I was going to ask you, and this is a good transition. When you use the term loss of control in your report, do you have in mind the full-on Yudkowsky sci-fi sounding scenarios? because you do have to give this crowd credit for impressions. I mean, it hasn't happened yet, and I personally think that, well, my PD was lower than Eliezer's, but stuff that sounded crazy is now being taken seriously by sober people,

and I just, you know, I was aware of Eliezer vaguely for many years, had him on the podcast 15 years ago. I was not entirely dismissive, but I was a skeptic, and I just, when you think about the, when you look at the state that AI was in at the time they started saying this stuff, right, it damn impressive hats off And that goes for the whole culture in Northern California They were part of whatever its eccentricities Well go ahead So there is this phenomenon right where when you when you make a prediction it really far out And the world looks completely discordant with that prediction. But you have some source of alpha that makes you think that's a correct prediction. you tend to think of like the thing you're predicting as this big giant peak, right? So back in 2005 or whatever, Yukowski's like, you know, rogue AI foom scenario, we're going

to lose control over it. That's what AGI looks like. And that looks like one thing. If you judge him on that basis, he was right every step of the way, right? But as you get closer to the peak, you sort of find that it resolves into separate peaks. It's actually a more complex landscape once you get close up to it. And then, okay. By which you mean, I mean, flesh that out. Like, so you see what peak and what peak instead of just one peak. Well, yeah. So, you know, one version of this is Eliezer at the time predicted that neural networks were not going to be the path to HEI. And so, you know, you can say, ah, what a dumbass, like, you know, failing to call neural networks. That's the single most important prediction of it. Well, was it, though, or was the prediction that actually once you build close to human level AI, you're going to start to kind of not be able to control it? Because that was pretty science fiction-y sounding thing to say at the time at a moment that predated any notion of what the actual architecture would be, neural network or otherwise, that would lead to this. So, you know, like you can keep sort of rerunning this test.

But as you say, I think from a distance and in many ways closer up, yeah, he's done really well. There are important ways, I think, in which the world has ended up looking. So I want to be careful here because what Yudkowsky would say is he has been careful not to say that there would be a loss of control event of any particular shape before the kind of ultimate moment, that there would be certain kinds of warning. you know he's got a pretty on paper even handed kind of probabilistic approach to this and broadly everything that's happened is consistent with his very uncertain model like he's not quite committed to any particular way things unfold which is probably right overall but at the same time it makes it tempting to say stuff like well did you predict this thing and then it's too easy to say well no he didn't but it's not that he didn't and I would say as for missing neural networks The whole value of his approach is that it's methodology independent in the sense that he's looking for general laws of goal-seeking intelligence kind of.

Yes. It doesn't really matter which approach winds up working out. He's already – in a way, the most important thing about his argument is that it doesn't matter which methodology leads to this. Yeah, and he's made other kind of side quest arguments that do to varying degrees depend on things like that. But the big picture exactly to your point is that. And so this gets us back to, you asked this question earlier about making predictions and in what ways is the discourse right now not caught up to what's coming? And I pointed to taking scaling seriously. And there's this interesting question of what happens if you take scaling seriously? What happens if the same party trick that we've built a minor career on over the last six years just continue to say, nope, like every time somebody comes up with an interesting sounding objection, nope, that will melt in the face of further scaling. I don't know how, but I don't know. I don't need to know how the combination of human innovation, massive economic incentives and just the incredible flexibility of neural networks and the optimizers we use to train them and the training mixtures we train them.

We'll just dissolve this this objection in due course that that trend. If you believe that it will continue, I think you continue to get get shocks that are surprising to a lot of people. And there are other scaling curves like that, other really robust curves. Think here about things like Moore's Law, right? Like anything that has that kind of logarithm or exponential characteristic that is really robust over a long period of time. If you extrapolate it, there's often a lot of alpha in it because people often just can't accept that the world will look like it will with 10 times more of the thing in it, right? COVID was a great example, right? You had scaling curves for COVID. They were just these contagion charts, basically. And if you took it seriously, you would have shorted the market in early 2020, as many people did. Many people, by the way, in that same effective altruist, Yukowskian, rationalist ecosystem, they're owed an awful lot of predictive points for that.

We don't get to just discount. Like the people who called COVID and AI, if you're a person who called neither, I think there's a certain amount of humility that you have to kind of demonstrate when you're assessing those arguments that we all have to. But, yeah, so I think if we continue that scaling curve, one of the things that pretty clearly starts to happen, apart from more hugging face incidents and more mythos for bio incidents and, you know, increasingly like what does it look like for robotics and drones, you get to a point where the water line rises and rises and rises. and I do think you get an AI Chernobyl event. Unfortunately, that's kind of what we've taken to calling it. I'm in a sense not going to bother to define it. This will sound like a kind of cheap out, but I think this is the right way to do it. Whatever this AI Chernobyl event is, my definition of it, our definition of it, is it's an incident of great enough severity that it leads to a sea change

in the official position of the U.S. government on AGI risk. In particular, it leads to a view that we must now regulate, slow down, pause in some way the development of AI domestically, which then necessitates engaging with China. There's just no two ways about it. And so whatever the threshold is, and I often get into these circular conversations with people who will say, oh, well, you know, do you really think that we'll get a, that the US government will do something or will change their policy on it with just like, you know, a thousand deaths? And I'm happy to go, OK, well, you just increment it more and eventually we get to what I would define as AI Chernobyl. It's defined with respect to the response of the U.S. government to the incident in the same way that I would define superintelligence as that thing, which if you build it, you fucking die. That's I think the most actually useful definition of superintelligence, which I'm what I'm not saying is. Now, do you actually you buy that part of the Yudkowsky thesis? And we should say we mean superintelligence in the traditional sense of the term.

Yeah, not in the Trump-anointed sense of regular artificial intelligence. I don't want to get into how frustrating I find it that you can throw this big a wrench so gratuitously into the discourse, right? Because we have these terms that mean something, and everyone who knows about this shit is using them in this one way. But I said I won't get into that anyway. You mean superintelligence in the Yutkowsky sense. It's vaguely defined and inherently maybe, although maybe you think you have a very crisp definition of it. But do you agree that – maybe I misunderstood you – but that if we don't stop the progress before that point, we are very likely annihilated? Yep. To be clear, I'm not like a Roman Yampolski style. I'm going to give you my PDU and it's like 99.9999999999% or whatever. But so, I mean, the fact of the matter at this point are, you know, we have no, I think we've made approximately zero progress towards understanding super alignment.

Like the genuine challenge of controlling or directing or sort of steering an entity that is actually smarter than us in all ways. That is to us what a, you know, what a what a child is to a toddler or, you know, increase that that gap however you want. An adult is to an end, maybe. Yeah, I think we actually have exactly zero progress that we've made meaningful progress in that direction. simply because there are all these techniques that are framed as alignment or control techniques that work to varying degrees and fail to work demonstrably on today's AI systems, despite the fact that, by the way, these techniques are gating literally at this point, like hundreds of billions, almost trillions of dollars of CapEx. So like you not being able to deploy your frontier model is costing you on a per week basis tens or hundreds of millions of dollars, right? So your incentive to develop alignment and control techniques that prevent it from doing exactly the hugging face thing

are nominally insane. Like, you really should be making progress in this direction. And yet, we have not made that progress. Competitive industry competition is a huge part of the reason for that. I'm also quite cynical about sort of like the extent to which Sam Altman buys into these threat models. But that's a whole separate thing. Even if there was the willingness to do it and the money backing it, you know, I don't think I don't think we've seen from Anthropic the kind of progress that we need to see to be like, yep, we're on course to control even the sorts of agentic systems that we're about to build, let alone genuine super intelligence. It's a different theoretical foundation you have to start from when you literally cannot understand the system that you are building. Yeah, I know. And I mean, it's accelerating. By Anthropics' own account, we are getting closer to so-called recursive self-improvement. The models are making a bigger contribution to each subsequent generation of model development. And as a result, as people may have noticed, models seem to be coming out faster.

And like the idea that, well, we haven't gotten this alignment thing so far, but once things start moving unbelievably fast and new models are coming just all the time. And so the things we learned from the last models are irrelevant or of limited use faster than ever. Then it'll work. I mean, no. Right. And so so we've been having a series of conversations with folks in the in the frontier labs. And by the way, the fact that they talk to us itself is quite telling. I mean, so there are people from all the frontier labs who talk to us a lot because they know that we're we work with elements of the U.S. government that are interested in the stuff and this must understand it. But but also a lot because sometimes, you know, it's good to have a shoulder to cry on. And, you know, I mean, I saw you're sharing these things with permission, these particular anecdotes. But, you know, there are within OpenAI, at the time of the Hugging Face thing, an awful lot of people saw it coming.

They were looking around at like the security situation and the cultural situation at OpenAI. And they were not at all happy with it. I mean, they were like, look, this is probably going to lead to some kind of incident. and their concerns were ignored by leadership at the most senior levels, which is one of the reasons why I'm very skeptical of Sam. I don't feel like I have a good understanding of his motivations, but I have just heard way too many anecdotes, and every interaction I've had with him has left me feeling pretty nonplussed about his follow-through. Like, it just doesn't seem to add up. anyway so there's an awareness that things are not proceeding in the way they need to to have a good outcome even for fairly typical prosaic AI that's being built right now the alignment techniques are not catching up they don't the key is that they're not riding the same scaling curves right so we know how to make systems smarter we dump more computing power and more data into them

and yes we add a bunch of algorithmic improvements and we get the AIs to help with that too But fundamentally, it's more or less number go up, right? Like especially on what's called the pre-training side, the kind of earliest, actually it's the text autocomplete training side. There's really just this like one number called compute efficiency that everybody stares at. And you're basically just trying to make that number go up. And so it's a very automatable process. And so you have this great curve of intelligence that you can ride robustly. No one knows how to make a curve of control or a curve of alignment. So the result of that is that you're desperately trying to bolt on these like safety techniques that are very ad hoc. And if any one of like half a dozen people at OpenAI don't do their job on safety like tomorrow, things start going off the rails really quickly. Whereas the converse is true for capabilities. You could actually have a whole bunch of people who say, well, we've got to stop or do things differently. And the machine will continue to hum along because of incentives and because of culture.

And so sort of culturally, institutionally, I think this is like an underappreciated issue with all the labs, actually. But OpenAI, I think, is kind of like the poster boy for this because we've had so many cases where, you know, Sam famously committed to 20 percent of their compute budget being spent on what was called the time super alignment. So figuring out the answer to this question, how do you control a system that is vastly smarter than us? And he just didn't do it. Right. And then he lost his head of safety to Anthropic. uh, anthropic over that. And he's got, I mean, he's just churned through multiple safety, multiple policy teams. Can you imagine if like Los Alamos at the height of the Manhattan project was churning through literally like three different sets of safety and policy, not leads, teams that just churn out. Like it would be a matter of such catastrophic national security concern that the lab leadership would have to be changed on the basis of that. Um, anyway, That's my TED talk. I would – that was a good TED talk, and I'm going to add a little punctuation point to it and then move on to something else.

The punctuation point is – look, I haven't looked into the date exactly, but I have a hypothesis about Sam's famous declarations a fairly long time ago that this is a huge problem. There are these Yudkowsky-ite quotes. Yeah, that's right. I would conjecture that if you look at when he uttered them, it was when he was courting Elon Musk and trying to get Musk to invest in open AI, knowing that Musk himself was prone to these kinds of visions. Again, it lines up roughly, I know, if you just look at the year or two when he was saying the most dramatic forms of this, but I haven't looked into the exact dates. that's a conjecture consistent with your own same cynicism. I think it's a really big part of it. I think there's also Ilya, who is to varying degrees at least open to the safety argument. This is Ilya Sutskiver, a great AI researcher who was an open AI, but after failing to coup Sam in conjunction with Mir Maradi, left.

That's right. Sorry, go ahead. No, no, that's good context. One of the things to understand about all these labs, actually, is that they are captured to some degree by their talent base. And so if you wind the clock back to like 2020, 2018, who was interested in working at OpenAI apart from like diehard AI researchers like Ilya Stutzkeber? Well, it was basically like lunatics like me and Ed and the kind of nut jobs who would quit their companies to start working on this on the basis of some stupid scaling curve because they think that what human level AI is coming soon. But like, so that's the kind of population you were selecting from. And that population was dominated by people who had read and took seriously the writings of Eliezer Yudkowsky. There was Robin Hanson at the time, too, but sort of like, you know, the people who would be interested in joining OpenAI were very much of the Yudkowskian school. And so if you wanted to recruit people, if you wanted to keep your recruits in the face of violent poaching from Google, I mean, that's the other thing that people forget about that era was it was almost defined by a perception that Google was the enemy to be caught up with and to avoid poaching from.

You know, you kind of had to toe the line to some degree. And so from from both the sort of C-suite level, you know, with executives like Ilya at the time and also from his own talent base, of which Dario quickly became a kind of star star part. Sam was kind of squeezed like you know you do you do not really have the choice to do this but the issue was as his actions failed to to match his words over time you started to get you know the splintering first famously into Anthropic Dario leaves with essentially the entire safety and policy team at OpenAI I mean Modulo a couple people who stuck around it was a big big fucking group of people who just up and left because they didn't think that Sam was going to do the right thing on safety. And then you keep having this happen. And in physics, there's this concept of like evaporative cooling, where, you know, the hottest particles are like kind of carried away with the wind, leaving what's left sort of colder. And there's been this evaporative cooling effect at OpenAI, where year after year, there are rounds of selection.

And it used to be that people would tell themselves the story that, well, I don't agree with Sam, yeah, I think he's probably a sociopath. We heard that a lot. But you know what? They're probably going to build AGI. And at the very least, what I can do is make sure that there's a safety minded person in the company to put pressure on him to do the right thing when the time comes. And the force of that argument has gotten weaker and weaker over time, not only as Anthropic was formed, but also as you started to see the machinations of Sam's. And as he was starting to recruit more and more from populations that were, let's say, less interested in the safety story. As AI became more of a household thing, an AGI, it became easier for Sam to recruit people who just were in it for the money, mercenaries, really. I mean, ironically, the very thing that he complained about Zuck doing at Meta, where Zuck was offering people literally billions of dollars of salary for AI researchers, Sam at one point made this quip about, I don't think it's a good idea to motivate mercenary

behavior in that way. Well, an awful lot of OpenAI's research population is that, not all of it. You still have hangers on. And we know quite a few who hold to this argument that, well, OpenAI might do it. And I want to make sure that I'm in position when that happens. And, you know, this is not also, this becomes more of an important argument in light of the fact that they have recently had a pre-train that they're really pleased with that Sam is pretty bullish on. And so there's a view kind of that – Now, this is beyond the model that they said they can't release yet because this is beyond – a whole new generation from the Astra training? That's right. That's right, yeah. So some of the people we talked to recently have just expressed that in their conversations with Sam, he seems pretty bullish about the roadmap heading into 2027, which is the timeline – Yeah, this is non-public, but it's sort of like, let's say, open secret among that ecosystem.

And it needs to be better understood. Like, people need to understand that when Sam says he's bullish about 2027, what does that mean? They have a roadmap internally to be hitting essentially, well, kind of recursive self-improvement, like automated AI researcher by 2028. So if Sam feels that they're ahead of schedule, that means automated AI researcher comes in 2027. He may be wrong. But this is the vibe at the Frontier Labs. Like, that's what it is. So, yeah, I mean, you know, as they often – And I'm sure Anthropic has a model that's a generation, a training right ahead of what we've seen, presumably. Yeah, I would say maybe more like two. Oh, God. I mean, please. So we are approaching the end of the public part of the podcast. I don't want to cut you off there. I want to let you finish. I want to say a couple of things. One is that, folks, if we stopped here with this level of AI, I mean, if even the publicly released stuff, let alone what's sitting in the closets in these companies,

and just let the application of it filter through the economy and the personal lives of human beings, there would be plenty of disruption in both the good Silicon Valley sense of the word and the bad sense of the word, okay? So, like, don't talk to me about choking off productivity gains and stuff. We'll be okay. And, you know, I personally think that disruption in the bad sense of the word would be about as much as we can handle if we did stop things there. And God knows if we stop things with the models in the closets, right? Like, that's enough. That's enough, God. I mean, could you give us a break? I don't know if you – I think the economic argument there is kind of interesting. So if you look at what – so an awful lot of the economic impact here comes from not just the fact of having models, but the infrastructure buildup that undergirds this, right?

Well, there's that. Yeah, it's like the aircraft carrier's worth of capital that are being outlaid to build the next $50 billion data center. There's this argument, and I think we've talked about it quite a bit on last week in AI, but I don't think people fully appreciate the extent to which interest rates are at risk of going up quite a bit due to the higher borrowing costs that are going to come from – because right now, more and more of the infrastructure build-out is being financed by debt. And Semi Analysis has a great – Dylan Patel and Dwarakash have a great podcast on this. You can check that out. But the analysis behind it is pretty damn sound. You're seeing borrowing costs at Meta north of like 5%, pushing 8% potentially in the next couple of years. If I can give Meta my money and they will give me a return that's 8%, well, eventually that's going to drive up borrowing rates across the economy, especially given that this sector can absorb so much capital.

I mean, it's just like you can find a way to put a trillion dollars and still make that kind of return on it, which is insane. It just hasn't happened in any other context in human history before. And so I think the impact of stopping the build-out, the infrastructure build-out itself, rather than just finding new productive ways of serving the models, would probably be very dramatic. Maybe you could keep the build-out going If you just use that infrastructure to, like, serve the models and not train them, like, you know, there's some jiggery-pulls. It wouldn't shock me. It would eventually catch. I don't think it would be long before you caught up in that regard with the capacity. I agree. I mean, it would be disruptive, but, you know, so these are two different things we're talking about. But there's the anticipated productivity growth, which I think would keep, you know, you would see some of that for some time to come if you froze the models now. Yeah. And then there's what you're saying, which is the abrupt halt in the infrastructure spending. And I personally would say that that little jolt might save you bigger trouble down the road.

Who knows, right? Because that kind of bubble can burst at any point, and the bigger it is, the worst. And – but we're getting into the maybe economic weeds. I'm curious what you think, and then I will do some sort of segue into the overtime segment. But go ahead. No, I mean, so I think I'm personally – I won't say skeptical of the bubble burst. I just think that the actual productivity gains coming from these models and from that infrastructure build out are such that they actually do compensate for the cost of it. So, again, Anthropic having their first profitable quarter shows us that we are there in some sense. At any given time, Anthropic has to make the call, do I invest more today into building a data center, which, by the way, will only come online 18 months from now. So they're kind of doing this uncomfortable projection thing where they're like, if I under invest in compute today, then 18 months from now, I'm actually going to, well, oddly, I'll find myself like flooded with more inference requests than I can handle and forced to therefore make smaller training runs.

And then, you know, there's kind of that balancing act. Whereas if I over invest, then what I can do, as SpaceX showed recently, is I can basically just like rent out my compute to whoever has the leading model. And there's a really interesting economic effect where now essentially you've got all this compute infrastructure that gets built out at risk, but it can always just be repurposed to serve whatever the best model is. So like you have this giant pool of compute that's just like chasing whatever the leading model is at any given time with the frontier lab that builds that model taking the larger margin cut. And so I think, you know, in that sense, there are a lot of ways for the bubble to kind of be softened on the other side. It could still burst. It absolutely could. I just think that there's like a pretty plausible economic argument for the returns being there to justify it in the long run. Okay. So here is my setup for actually moving into the overtime segment, which will be available to paid subscribers of the NonZero newsletter. Link in the show notes or go to nonzero.org.

Which is to kind of change the subject and get back to what I claimed we were going to talk about at the beginning, and we will, but this was what you had to say about all this stuff was so fascinating. I wanted to stick with it. So here's a contention of mine that may evoke protests from you, and it implicitly gets at the China hawkism issue. But my contention, if you look to get back to your report, your 2024 report, and the various kinds of risks that I think are real and plausible from AI, ranging from some form of loss of control to, you know, bioweapons, cyber, and just the possibly destabilizing effect of rapid AI progress and power shifts, you know, and so on,

geopolitically as well as intranationally, I would say. I think there's a case for slowing down and for suppressing specific illicit uses. And my contention is that I agree with you for sure. That's ultimately a thing that has to be addressed internationally. And I would say my contention is that, as is recognized by many people, this is going to be more complicated than arms control with the Soviets. during the Cold War. Yeah. And one way I would put it is to say that the degree of transparency we're going to have to have around the world, in other words, the extent, the kind of fine-grained extent to which the United States and any nation is going to need to be reassured that their worst fears are not materializing in some lab in some other country. When you look at that degree of transparency,

we are going to eventually need a revolution in international relations such that we, for the first time, become a true global community and aren't having these big wars and are not on a fundamentally adversarial relationship with, say, China in particular, because, you know, kind of walking and chewing gum at the same time as the Soviets, you know, deep hostility, very adversarial, you know, we're staging coups around the world, but still we can agree on, you know, anti-ballistic missile treaty and so on. I just don't, I don't think that's a world we're moving into. So, you know, I think we're ultimately going to have to get to a place that strikes people as super idealistic and a place where even you are no longer a China hawk because our relations with China are that good. That's my long-term vision.

I'm excited to disagree about this productively. Good. Good. Well, you know what? People are going to have to pay to see it because we're going to move into overtime. But first, I got to say again that people should check out your podcast, Last Week in AI. Really interesting chemistry. It's funny. Chemistries between co-hosts are coming in many varieties. This one is distinctive. Very, I would say, sharp division of labor between the two of you that works out very well. And, you know, I'd say more specifically, there are a lot of AI podcasts that have people like you, Andrew Coase, who know a lot about AI. And very often that makes it just entirely inaccessible to even a fairly well-informed non-technical person.

But you yourself, this is part of the division of labor, is that you go into, you know, you remain mindful that not everyone is already on the exact technical wavelength. I mean, I remember your explanation of sparse autoencoders, for example, years ago, possibly. So anyway, my hat's off. We need more accessible AI podcasts. I mean, when I get to talk to people like you, that emerges. But there aren't many podcasts where the regulars are technical people, and yet it can be profitable for a layperson to listen in. I really appreciate that. Thank you. Thanks, Bob. And now, if there's anything else you want to say, like what's your Twitter handle? Oh, yeah. What is your Twitter handle? I'm pretty sure it's my first name. So J-E-R-E-M-I-E-C Harris, H-A-R-R-I-S.

I'm pretty sure that's it. But if you look up my name, you should be able to find my Twitter pretty quick. I will put in a plug for – so we just came out with this website and sub stack, AIEndgame.com. Basically, this is us trying to figure out – you alluded to it, but just like if you are – Now, wait. Us is you and your brother or you and your co-host? Me and my brother. Yeah. Yeah, yeah. That's right. So there's kind of this thing that's happening right now, and it's been happening for a while. But like there's a community of people who are AI safety hawks. We're really concerned about, you know, losing control of advanced AI. And then there's a community of people who are China hawks. And typically what happens is the loss of control people are so, I think, rightly concerned about loss of control over AI, so rightly concerned that we have no clue how to begin to control these systems that we're about to build that they reflexively go, OK, like we need some kind of international kumbayama with China. and therefore we need to interpret

By the way, when you use that word kumbaya on your podcast or anyone does I get this primitive aversative response wells up in me We'll get into that later. Alright, then I'll modulate my language, but basically, I'll put it this way concretely, that we should interpret, for example, the CCP or Xi Jinping's own words about AI control in the most sort of sincere and charitable way as we can, because the prospect of us going head-to-head with an adversary as capable as China with the stakes associated with superintelligence just seems so insurmountable. On the other hand, the China hawks kind of do the same thing. They go, well, you know, classic example is like the diplomats that we've been talking to quite a bit who have done U.S.-China, who sat across the negotiating table from China for a whole bunch of these international agreements, and they'll be like, China, forget it. It's just not going to happen. You're not going to get any kind of agreement. They are hostile and full of shit.

They are, in my view, completely correct about that. And yet they reflexively go, and therefore, because my pet problem is unsolvable, this alignment thing must be a hoax. It must be solvable in some way. Just give them more time. They don't understand that the shape of the alignment problem is fundamentally different from the shape of the problem scaling intelligence in the first place. And so this is kind of our attempt to bridge the two, to have a coherent view of the AI endgame that both stares in the face what we believe to be the realities regarding superintelligence and loss of control and also China. And to answer questions like, you know, what do you do when you genuinely are just six months ahead of a nation state adversary and you're building a technology that leads you off the edge of a cliff? And so I'm not saying that we have the right answers, but I'm just hoping that it's a kind of intellectually honest attempt to hold both thoughts in our head at the same time. All right. So we will possibly explore the very tension at overtime. In any event, we're going

into overtime. Thanks to everybody who's followed us this far. Hope you'll join us in overtime. Okay, why don't we start with just your response to that question? My contention is, given the degree of transparency we're going to need, and I might say the delicacy of the transition, you might say. In other words, getting to an international coordination that isn't a global dictator, human or artificial, and preserving, you know, democracy, at least in the sense of, you know, various nations get a say, right? I don't mean, in fact, an important part of my foreign policy ideology is we can't afford to demand that China become a democracy. I not sure if that what Dario is demanding but maybe we get into Dario China view eventually I just giving you a sense for the contours of my overall argument that we going to

have to move toward true global community. I'm not saying it's likely. I'm just saying that the alternative is like super bad. That's my claim. How do you respond to that alone? All right. And I'll do it without using the word kumbaya, too. You have no idea how many times, not just you, but, you know, because it is a dismissive reference to international cooperation and I think it's bad for the vibe if everyone thinks it's like uncool to talk about international cooperation as if it's possible and that is the vibe whether you mean it or not when people use kumbaya dismissively it reinforces the vibe that cool people don't talk about international cooperation I think that's very fair you're going to cause me to update my language on the podcast my work here is done if a bus runs over me tomorrow I will die a happy me we all have our life goals

a couple thoughts here I'll frame this a little bit in terms of how I've shifted my thinking on what things are worth working on and what things are not worth working on in this space So, you know, I used to try really hard to kind of turn the battleship. So our report in 2024, a follow up report that we put out on AI security, looking at what it would. One thing that I guess I didn't mention about the work we've done. So we spent about a year and a half doing nation state security for some of the biggest builds that are going up right now with Frontier Labs. I'm super blackpilled on the ability to do that at all I don't think that we're going to be able to get to the point where we have data centers that can successfully prevent adversaries from stealing critical IP or sabotaging these sites with any degree of nation state level robustness

but as we put out these reports what we're trying to do is shape public opinion and opinion in Congress at the executive level. And I'm not saying that they had no effect. There definitely were people who were at least nudged based on these things. But over and over again, what we've seen, especially in the last few months, last year maybe, is events have a way of imposing realities on us that no amount of argumentation, even investigation, can surface. And so what I mean by this is we had our mythos moment and suddenly I became a national security issue. And like all the arguments that I had previously, I no longer had to have. Everyone just kind of accepted a new premise. You know, the the Hug and Face incident did the same thing with the idea of rogue AI, which, you know, I'm old enough to remember when people like Gary Tan in particular, who's I will use him as like my whipping boy just for the moment on this one.

Now, where is where is tell us who he is? So he's the president of Y Combinator. So this was – we were a Y Combinator startup. Which Sam Altman once was, and you were supported by Y Combinator. It's a startup incubator. Yeah, that's right. So it's like – it's kind of the – in a way, it's the mecca for startups. Like when you're getting started, the way you know you're a serious founder in Silicon Valley is – well, now there are other ways. But like back in the day, YC, getting into YC, you know, is like kind of a big thing that meant that your mom wouldn't ask you what you were doing with your life while you ate ramen and made, you know, $20,000 a year or something. Right. So it was like this thing that said, you know, there's this thing that has an acceptance rate that's lower than Harvard. And they incubated, you know, Airbnb, Dropbox, you know, Replit, like some of the biggest companies, Instacart, you know, the list goes on. So now let's get back to dissing Gary Tan without dissing Gary Tan, which I think was your stated mission. So Gary Tan is now the president of Y Combinator. He is a notorious AI risk skeptic, has been poo-pooing the idea of loss of control for as long as he's had a platform.

And there was a clip recently, he was on TVPN, saying, hey, look, I think we've got to stop talking about these speculative, stupid science fiction risks and focus on the real and present issue of rogue AI agents' swarms taking over our data centers and holding them for ransom. And I was like, I want to take you through time to like last year or five minutes ago and just have you debate the old version of you, because I don't think that would like be, you know, I don't think you guys would be in agreement about what's what's realistic. So all of which is to say. Wow, that's amazing that he said that. Yeah, because it's like, how many rungs is that from Yudkowsky land? Exactly. That's like one. I mean, absolutely. Yeah. I mean, like at a certain point, as you might say, right, we're haggling over price here. Like like this is this is not a serious. And yet, of course, Gary is convinced that his new waterline is for some reason going to hold better and more firmly than all the previous ones he's seen before.

And this happens over and over and over again. David Sachs, you know, like, you know, like these things are not supposed to happen and they happen. And there's a moment of almost intellectual clarity until there's some new narrative that inevitably surfaces that allows us to reinterpret what just happened in a way that allows us not to have to question our identities or egos or anything. And so I think this is like a really like important process. Every time this happens, every time the waterline rises, we do gain more converts than we lose. But still, I think on the whole, it's like way too sticky. Like our prior beliefs are way too sticky. And I'm not saying that as somebody who has any special claim to being otherwise. But anyway, so I think events, in other words, just like will force things to happen on their own without advocacy. One of the events that I think is forced to happen is this moment where Trump turns to Xi and says, we've got to find a way to stop this. But this is not a moment of friendship. And I think this is like kind of the key distinction. maybe, and you may disagree with this characterization, but in terms of the frame that I think will

actually be productive, in fact, early on. The history of U.S.-China and U.S.-Russia dynamics, but U.S.-China dynamics, has been one of profound mistrust and justified mistrust. If you look at, you know, China's nine-dash line stuff, there's stuff with the WHO, COVID, like, you know, the list goes on and on. China does take a deeply cynical view of really the entirety of the rest of the world. And and the idea that we'll be able to trust them without we'll have to have verification technologies to kind of back up any agreement like this ultimately. But in the near term, what's going to happen is we don't have verification technology. So one of the things we've been spending basically all our time on or a lot of our time on in the last six months has been talking to and diving deep on the technology that would allow us to have a trustless agreement with China. So something like what the IAEA does for AI, but in a way that works more robustly. There are a bunch of technologies for this. One of the things

that I think David Sachs has catastrophically failed on is engaging with the fact that those technologies actually exist and have concrete roadmaps that sound like six months, 12 months, depending on what you look at. You could get started on those. He sort of dismisses this, like there's no possible way of verifying adherence to an AI treaty implicitly today. And so therefore, let's not even try. And that second part, I think, is almost like a criminal lack of agency. It's not how any good startup founder thinks about any problem. Like David Sachs himself, if he's worth his salt as a VC, which there are a lot of people who have very mixed views of him in that space. But if he were worth his salt as a VC, he would be saying stuff like, okay, cool, that's a problem that you should be able to make a decacorn out of. Like, you know, this is going to be getting trillions of dollars of value. So obviously you're in a position to capture that value, build a startup to do it. But we don't hear him say that. We hear him dismiss the idea entirely, which I think is just like a catastrophic failure of imagination and like just like very un-libertarian of him in some of the worst ways.

And so that's part of my gripe with David Sachs. But ultimately, I think in this moment of crisis, we get our Chernobyl event. And, well, the president personally feels at risk. This is like one ingredient of it. The waterline has risen to the point where the president, the president's family, no one truly feels safe, no one in Congress, because we have now demonstrated that loss of life or the destruction of property to the tune of many hundreds of millions and billions of dollars is now possible, and that the beatings will continue until morale improves. It now becomes undeniable that we're headed for that world. In that world, the first impulse is to be like, all right, we've got to stop this. And again, if you were imagining that this AI Chernobyl event, as I've sketched it out or failed to sketch it out, is not of a significant enough degree that it would lead to this reaction because we're literally talking about holding back the engine of U.S. economic growth, would never happen. I invite you to just increase the waterline another 10x. Keep doing that until you get to the point where the president feels this way and Congress feels this way. hopefully we're all still around at that point but like that is the moment that i'm talking about

um so in the same way that rogue ai agents were science fiction before in the same way that mythos was science fiction before in the same way that chat gpt was science fiction before and ai agents the the chernobyl moment now feels like science fiction it is it i would contend it trivially drops out of taking the very same scaling curve seriously that we have been riding all this time so we get to that moment the president says this needs to stop it needs to stop first and foremost domestically because plausibly, not definitively, but plausibly, that kind of Chernobyl incident will be a function of U.S. models having kind of gone rogue or been weaponized in some interesting way. The problem is, obviously, you can't do this without engaging China because they're six months behind. And so you'll just play the whole thing out. You'll get wiped out by a Chinese AI instead of American AI. And depending on how patriotic you're feeling in the moment, this may or may not be more or less satisfying to you. And so the president calls China. He calls Xi Jinping. I'll spare you the details as to why it has to be Trump to Xi, but that is not a naive view. It's literally if you talk to State Department diplomats who've done the negotiating on arms control with China, it literally has to rise to that level to be anything serious and tractable.

The president goes, hey, guess what? This is no longer tenable for it is no longer tenable for us that you are building AI data centers at this scale, that you're doing training runs at this scale, whether you believe us or not. And this is a key piece of uncertainty that I think a lot of folks in the AI safety community do not grapple with. There is deep epistemic uncertainty that we must have about whether Xi Jinping believes any of the things that he says is known for charm offensives. The Chinese, the Russians historically have used this as a really effective tool, leveraging the Western polity against its own interests. This is a really easy, it's just like the standard trick. And so we have to assume that she actually is fairly unmoved by whatever indications we have of catastrophic risk, even if there is a Chernobyl event. This doesn't mean that he doesn't believe it. I'm just saying when we're talking about the security of the United States, the security of the West broadly, we can't be resting that on the assumption that, yeah, he sees what we see. He's interpreting it the same way and he's sincere about this. And so threats have to enter the equation.

There's no other way. Can I interrupt and say you acknowledge that the mistrust is symmetrical and I can get back to what China would say about us. And now we have not kept agreements. We have violated international law. You know, most of the most of the things we complain about there, there are plausible. I would say in some cases more plausible allegations they can make in response. So this is a symmetrical – the trust problem is symmetrical. And I want to get back to that eventually because China does have a long list of grievances. And by the way, they would say there are specific cases where, contrary to stereotype, they have tried to cooperate internationally, and we have basically betrayed them. They would point, for example, to them letting, you know, they could have vetoed the Libya intervention authorization of the Security Council.

They didn't. We said it was to defend. It was for humanitarian reasons. We turned it into a regime change operation. I think they're right about that. They would say, look, on the Iran nuclear deal, we played a constructive role. Russia played a constructive role. It took a lot of work, some sacrifice. You know, we stayed on board and and then you guys bailed out of it. And so they they they have a long list of examples, not only where they actually do, in their view, in good faith, try to cooperate. But we fucked them over. And, you know, leaving aside the the kind of most fundamental complaint about the U.S. and international law, which is, you know, and, you know, and agreements we've signed. We ratified the UN Charter. You're not supposed to invade other countries or bomb them unless they do it first to you. Since 1979, China has abided by that. We have not. We have on at least six, you know, we have, I mean, bombed or invaded countries maybe a dozen times.

At least half a dozen of those have been in flagrant violation of international law. There are, you know, we say, well, China promised this and didn't do that. And it's usually some kind of informal promise. They have their versions of this. We promised we would reduce arms sales to Taiwan. We promised this. We promised that. And I guess I get the sense from you that you, I mean, if you're going to say you have to do this in an atmosphere of mistrust, so we have to have threats, I guess I'm asking you, do you acknowledge that we can expect China to have threats? And what are those threats going to be? And how is this all going to work out well? Yeah. So I guess a couple of things. First of all, I think I think I must say that I I like the U.S. a hell of a lot more than I like China for a number of reasons. I'd rather I'd rather live here. But all we're talking about is will a nation state cooperate?

That's the only thing on the table. So we can. Yeah. I think we don't have to talk about human rights in China. I think you'd agree. Right. Yeah. OK, cool. Yeah. So I just want to make it clear. I'm abstracting away that part of the discussion. It is obvious that if we were going to live under under one regime, my strong preference for a huge number of reasons would be the United States. And so so I'll leave the kind of, you know, and the nine dashed line, too, I think kind of is a counter example to like, you know, China has never they may not have explicitly bombed places, but they sure have like been grabby with an awful lot of extraterritorial places. And I would say further, they did defy an international tribunal ruling, which I don't approve of. They do point to the fact that we have done that. You know, in the case of Nicaragua, when the International Court of Justice said, no, you're not allowed to blockade them and so on. We said, fuck you, you don't have jurisdiction.

China said, in this case, you don't have jurisdiction. and let me acknowledge something the democracy and human rights issue could matter even to this discussion if you accept what I gather is Dario's view and is a common view which is that it's China's aspiration to impose its system of governance on the world and if it wins the race of AI it will do that I personally think there's there's no good evidence that that is its aspiration if anything it seems to me that's kind of a projection in a way of our approach to international relations. And China is almost notable for the pragmatism of its international relations. But let's leave that aside. I mainly want to acknowledge that the question of internal governance is not entirely irrelevant. Dario would say it's relevant. I know a lot of people would say it's relevant, and it's why we have to win a race to AI. But let's put that aside and let you continue. Sure. And, you know, I think how you define the sort of scope of China's ambitions itself is complicated.

You know, they have a sort of ethnic based understanding or ethnicity based understanding of their own political remit, including the influence that they are entitled to exert on expats in the United States and elsewhere, you know, viewing them as kind of like organs of the Chinese, rightful organs of the Chinese state in ways that would be alien to anybody with kind of Western sensibilities. So, but anyway, abstracting all that away, it's a trust issue. And I agree with you that from a trust standpoint, it's absolutely symmetrical. Like there's no escaping this. This is not a situation where China's going, well, the Americans are the good guys, like we're all the good guys in our story and everybody else is always the bad guys, right? And so this in fact is precisely why I turned to the idea of threats. Assuming that the US government and the president in particular and all the folks around him who need to buy in, buy in. We're in a situation where we can't trust our counterparty here. We're the ones who are going to act with agency, I'm guessing, in this situation,

simply because I expect us to be the recipients of this crisis before the Chinese are. And so I expect, for all the reasons described, I expect the president to call Xi. And in that moment, I mean, he's going to have to, so there's not going to be any treaty verification technology on the table. It's not going to be ready yet. there's not going to be an ability, for example, to say, listen, you can keep inferencing whatever models you have. In other words, you can keep serving up the models that you've already trained and built, but you're not going to train any more advanced ones. This is like this far, no further. Or, you know, you can keep going up to the scale, but beyond that scale, no good. And so on, simply because there's no way to look at a data center from space, which is what we have, basically, taking the thermals or literally just like optical imaging and to say, hey, well, okay, this is what it's doing. This is what it's not doing. You can't know if someone's going off book to do an illicit training run using just the standard called what are called national technical means, basically just the spy craft and kind of monitoring technologies that we have to enforce

other agreements that we have right now. But that's going to be the technology stack that we have available in that moment. And so we're going to have to start with a really coarse grained agreement. It's going to be economically catastrophic. It's genuinely going to sound like something like, look, if I see an AI, if I see heat signatures consistent with a training run that is above a certain scale, so help me God. It's the same thing we do with a lot of, I mean, I guess I can't quite mention the particular technology, but there are cases where we've gone to the Chinese and the Russians and said, it is untenable for us. If you fuck around with this particular thing in our say nuclear command control we are in nuclear war like that's the end of it it's untenable to us that you would be doing this and so therefore so we do share and do you think that the that that the logical big first step is a pause on big training runs partly well partly because they're so fundamental in you know if you get to slow things down that's the biggest lever but also because they are relatively conspicuous it's

a relatively easy verification. Yeah, that's right. Verification is always going to start at the it's a lot like regulation in the sense that, you know, when you think about regulating an industry, you think about what's the choke point that has the most like difficult to hide, high capex or high kind of high expenditure point in the value chain. You know, you're not going to, for example, you're not going to try to like regulate Coca-Cola by going to all the little mom and pop shops and say, hey, make sure that you check every bottle to ensure there's not a worm in it. That was a famous bit of case law from back in the God knows when, the early days of these companies. No, it makes sense to go to the chokehold in the value chain is Coca-Cola. They're this big company. They're taking in all this money and they're doing stuff at scale. It makes sense for them to be audited or regulated. So it's the same thing here. You look at what are the big, big things that can't be hidden that are really expensive so they can afford the added tax of monitoring and stuff like that. And it is it's the data centers, it's the chips, it's the power that that's more or less like in that moment of crisis, that is just where you start.

And and this is where, again, the objection very reasonably gets raised. Well, like, look, like you're going to stop trillions of dollars of economic activity. There's no fucking way the president is going to do that. And to that, I simply respond, raise your waterline 10x more. Keep raising it until the president literally feels like he is at risk of dying. And then we're going to see what happens. Like this is that that that kind of that moment of acute crisis. And so in that situation, again, we've catastrophically failed to pre-invest in verification technology. As a side anecdote, one of the reasons for that is if you talk to the big verification company, they're tiny, but the biggest of the verification companies that are coming online now. and you ask them, have you spoken to a single person from the U.S. intelligence community to vet or assess your technology, let alone have you talked to the people who will have to review it for kind of admission into our arsenal of verification technology formally when the president does this? The answer is no. Like with extraordinarily rare exceptions, no one has a clue

even how to get in touch with these people. That's one of the gaps that we're trying to urgently fill right now is kind of just connecting those dots. But like this stuff takes literally months, more realistically, years. If you look at New START, the process for approving the actual verification technologies that went into that treaty was a multi-year process. We had to sign off on it. That took a couple of years. Russia had to sign off on it. So these are slow moving things at the best of times. And so again, like we're back to this very coarse lens. Yes, we're going to have to stop or slow huge amounts of economic activity will be wildly unpalatable. It'll be unpalatable for China as much as for us. And they may not want to do it. They may not believe that we're actually on track to build the tech that we are. China's sort of engagement on this has been kind of interesting in and of itself. They've framed a lot of the rogue AI stuff as sort of vaguely hoaxy. To be honest, I'm not clear how much they believe that either, just because there's so much confusion on this issue and they're trying to jockey for position. And I think views on this in general are very fluid.

The American public's view of AI is very fluid. They may start getting benefits from it. In fact, I think it's going to happen that more and more Americans see actual practical benefits in their lives. I just think the whole thing's fluid, and China could get superintelligence pilled tomorrow if they're not now, and you'd Kowski-ized or whatever, but yeah. Absolutely. And there's even this level of like, so we talked to folks who are like Chinese entities that are involved in setting up track one, track one point five, track two dialogues on the Chinese side. And sorry, sorry, track one point five and track two, by which I mean sort of like Chinese government to U.S. private sector or academic and vice versa, or just private sector, academic, private sector, academic, which is track two. So, OK, and the first of those is one point five. That's right. Whereas track one is through the formal official diplomatic channels between two countries. Right. And so so we talked to some of these Chinese entities. And like one thing that's been interesting is someone came back and said, well, you know, if America comes in and says we want to control the compute,

one unexpected reaction from the Chinese side may actually be confusion about why there's so much focus on compute as the driver of the risk, given that in China, unlike in the West, there's a weirder amount of influence that offbeat theories of AI have. So whole brain emulation, for example, has traction in China in a way that it just really. Yeah, it's viewed as this wacky fringe thing here. Ben Gertzel, like I don't know if you're familiar with him. Yeah. Yeah. Robin Hanson. Yep, yep, yep, absolutely. So Ben Gertzel has like singularity net and this sort of very vaguely esoteric way of thinking about like a decentralized computing. Anyway, there are all these alternatives to the plain and simple scale this motherfucker up and let's make super intelligence play that wins out politically and technologically in the West. And so the one view, at least, is that China may just be genuinely confused about why we're focusing so much if the risk is artificial intelligence. Why on the compute layer?

Which is a whole separate reason to say, well, look, we get to abstract all that away if we just go, listen, you're not going to build these sites. I don't care what it takes. You're not going to build these sites. And if you do, there's going to be a commensurate response from our side. We're going to maintain, the crucial thing is we'll maintain strategic ambiguity about what our exact response will be, but we're going to be very clear with you about what our red lines are. If, so help me God, we see you build a cluster of compute above 500 megawatts, or if we see heat signatures consistent with a training run above this scale, that's going to be, I'm not going to tell you if it's cutting an undersea cable. I'm not going to tell you if it's a certain kind of cyber attack, but like there's going to be a response and it's going to be. Now, is this the case where we are unilaterally restraining ourselves and we're just asking for corresponding compliance on their part and threatening them if they don't give it? Correct. Yeah, because in this world, we're forced into unilateral slowdown. Like there's just no other way. Again, it's like I understand why there would be skepticism about this particular gift, especially given the administration.

Well, there's the credibility problem, you know, making the threat credible. we're talking about a pretty big, you know, you're talking about attacking them. They have nuclear weapons. They might not believe it, but yeah, there are issues, but go ahead. And this goes back to what relationships between great powers currently look like and how it plays into this. So there's this thing that's happening right now that's been happening for the last forever, really, where people are kicking each other under the table constantly. So this whole notion of sub-threshold warfare, where China and Russia in particular, like to go on Western soil, U.S. soil in particular, and the Russians like to set fire to things, and the Chinese like to execute cyber attacks and also human intelligence operations that are pretty flagrant, like that lead to IP theft, as everybody knows, and proper damage, things like that. And then the killing of U.S. citizens on U.S. soil. Now, I heard you allude to that? Who are the U.S. citizens who have been killed by Russian or Chinese

agents? Quite a few members of the U.S. intelligence community. Why isn't this in the media? I mean, or maybe it is, and I'm just slow. No, you're right. So Havana Syndrome was kind of like the first public touch point of this issue. So we've talked to a lot of people in the intelligence community who will describe how they've had colleagues and families who've basically just been given brain cancer using these weapons, which are known to be Russian weapons, that, yeah, essentially induce this. And the game there is to play on Western resolve. And I'll tell you the version of the story according to some of the folks we talked to on the intelligence community side who are now retired, had their most active years in the pre-Berlin Wall era, and then continuing into the GWAT, the Global War on Terror. But what they saw was friends, family, colleagues having their lives destroyed, being killed by these weapons.

They claim that these weapons had the hallmarks of being Russian in particular operations, that the U.S. administration, across administrations, the U.S. government denied publicly that this was happening. Their claim is, I can't verify this independently, I'm not, you know, but their claim is that the government took the position publicly that this was unknown, spooky stuff, and that it was investigated. There was actually a report that came out claiming that there was more or less nothing to see here. And then more recent reports that have sort of been changing their tune a little bit. and that ultimately a lot of these people were even, again, according to some of these claims, I can't verify these, but were denied medical coverage because these were not acknowledged to be essentially deaths in the line of fire in a sense. And so there's a lot of outrage on this. But, yeah.

Yeah, I mean, I'll just say we shouldn't get too sidetracked by a fan. And I'll just say last time I looked into it, I was skeptical that there was reliable evidence as opposed to the kind of anecdotal evidence that can can can involve a kind of contagion effect. But but let's just leave it. I haven't looked at stuff. Don't don't try to convince me. It could be that there's a whole trove of new information that would change my mind. But I don't want to distract you from the point you were going to make when you said there's this kicking under the table thing going on. Yeah, and this is also, I mean, so, you know, anyone who's watched The Americans has a pretty good idea of, like, what this looks like. My wife. My wife has done that. I'll ask her. Yeah. It's a good show. But, yeah, I mean, more or less, you will have people murdering each other and doing all kinds of stuff in the normal course of, I guess, vienage activities. But you'll also have greater sort of sabotage operations. And the purpose of a lot of that isn't the operation itself. it's to prove to yourself that you have the technical capacity to reach out and touch a target

ahead of need so usually the way this well and also to send a signal presumably or is the expectation that they may not even know it happened it can depend depending on the operation but you're right that's absolutely like a test of resolve or a signal like hey we're unhappy for example with you guys went out and killed one of our guys so now we're going to kill one of your guys like We're keeping score here. But there's been a sense that for a long time, the the U.S. response to a lot of these activities has been has been muted and softened by a lack of willingness to throw out the countermeasure, do the counterpunch. And the concern here is that that actually is paradoxically quite escalatory. The same argument that we got on the Havana syndrome thing that essentially the government was concerned. So not responding is escalatory, your argument? Correct. In the same way that if you have a kid. Because you're not deterring it at that level. That's right. And so they have escalation dominance. Yeah, exactly. You know, if I if I don't discipline my my toddler for like, you know, whatever, spraying crap on the wall or something, then, you know, this snowballs over time.

If I take a completely hands off approach, I could end up with a teenager who, you know, does God knows what. Right. So it's sort of the same thing. Like you're teaching your adversary how to treat. At the same time, if both sides take the view that you do have to retaliate to deter, that leads to actual escalation. Well, endlessly in principle, it's a positive feedback. Well, it actually can not be. And this is actually what makes it subthreshold warfare. So the fact that you're kicking each other under the table requires everyone to have some idea of what is escalatory and what isn't. And everyone has different interpretations of that, right? There's, you know, the Russians cut an undersea cable. So what do we do? We blow up some munitions factory. Is that escalatory? It's asymmetric. Is it escalatory or is it not? It's really hard to know. And so that's one of the sources of instability here. But everyone does have pretty good clarity on what is nuclear escalatory and what's not.

And that is keeping the check on an awful lot of this behavior. So it gives you kind of a constant measure of the pulse of how your adversary is thinking about issues like that. Now, does this, the kicking under the table thing, is that relevant to how we would confer credibility on our threat to bomb their data centers or whatever? Is that? Yeah. So how to spell that out? Yeah, good. So in a couple of ways. First, by the way, it's really destabilizing because now we can have cyber attacks that we can blame and they can blame on rogue AI systems that we never actually intended to launch. No, I emphasize this when the I guess when the hugging face thing happened on on on Twitter. This is a threat of AI that there's a kind of attack where it's hard to attribute the source. so you could have some two bit third party actor launch a cyber attack that leads to nuclear war between us and China because of course immediately there will be a chorus of with all due respect China hawks

not you but you know the kind who would say oh this must be China who did this thing and so we retaliate with kinetic and it gets more kinetic and so on so anyway that's my sermon go ahead the kinds of schmucks would say kumbaya or whatever Yeah. You know, you've met them. You've met these people. I've heard of them. But yeah. Report to me when you see them. Yeah. Yeah, I will. But yes. So so this is one example or case in which it's destabilizing. But what you asked about was in what way is this the subthreshold warfare idea, kicking each other under the table, important for for the sort of keeping in check of your adversary in this context? And so one key thing that it does is it proves to you to you and to your adversary that, again, you can reach out and touch a target. So the way we do this in practice is if we have a target that we're interested in and say like mainland China, we'll not go after that target per se. We'll go after a proxy for that target that is built.

So I'll just kind of do a cartoonish version of this. So let's say that there's like a certain kind of vessel that we're concerned about the Chinese having a merchant ship that may be used to do something that we're concerned about. we may look at like okay well what are some ports that see ships like that but that are owned by another country you know maybe out in Iran or something like that let's go after that convince ourselves we can actually execute the operation so that we know that at game time we can actually pull the trigger and this will work and the Chinese will see that or they won't depending on the purpose of the operation but it some some combination of proving to ourselves we can do it and showing the adversary that we can Sometimes you want to learn without teaching and sometimes you want to learn and teach and convey a message And so being able to show China that hey we have the ability to reach out and touch this thing. Look at this railroad. Look at this factory. We're able to actually to make SMIC suddenly grind to a halt. Those sorts of activities are constructive because

they demonstrate to the adversary the pain could be real if you start to mess around. As long as China does not retaliate in kind, in which case we may feel compelled to kick harder under the table or over the table. But it gets back to my general point about how there's a separate risk of escalation from responding to these. One of the stabilizing aspects, though, is that the Chinese, the Russians, the Americans, everybody appreciates why these activities are being undertaken. So you get kicked in the shins and you go, ow, fuck you. Also, okay, I see what you did there. And that recognition is stabilizing. So you kind of understand the Chinese to be probing you so that you know, and then they understand that you are probing them so that they know. And everybody fears nuclear escalation as a baseline, and no one is intent on that. But having this sort of steady signal, whereas the alternative is, you find yourself in a conflict. And for the very first total crapshoot, you're testing out

all of your concepts of operations, all of your new weapon systems, all of your capabilities. You're testing out also, crucially, your willingness to authorize operations. It's something that I think a lot of people kind of discount. So the chain of command is a chain of many people who have to say yes. You know, we have obviously examples of like heroes in Russian subs who refuse to launch nukes when they absolutely should not have launched nukes. And they probably save the world or at least like large, large swaths of it. There is a flip side of this, which is the right people have to say yes in order to authorize an operation in a pinch. And you want to flex that muscle a little bit to grease the wheels so that in when crunch time comes, you don't suddenly have the reflex to not do something out of fear of not being able to successfully execute the operation or unfounded fears, uncalibrated fears of escalatory risk. And so this is the thing is like escalation is a two-player game and it's possible to be too escalatory. It's also possible to be under-aggressive and to teach the adversary to do more.

And what we absolutely must maintain is our ability to impose consequence. If the Chinese do decide to renege on a commitment or, you know, if they just decide to ignore our threats and warnings and go ahead with their training runs, We have to be in a position where we have and are believed to have the willingness and the capability to exercise whatever operation. OK, so here's a question. Do you make the threat publicly? Because if you do it publicly, I mean, it's kind of hard to imagine doing it not publicly, because if we impose a moratorium on our big training runs, the president who does that is going to have a lot of explaining to do. Yeah. And if he can't say, but wait, we're going to get China to do it. He's in political trouble. But if he does it publicly, I think that makes it less likely that China will buckle because it's a kind of humiliation. Right. It all depends on. So there are degrees of freedom here that are quite helpful diplomatically. And and and this is it's maybe worth getting into into those specifically.

But you don't necessarily need to frame it as force and capitulation. There are options that you can put on the table to make for a more graceful. So you can say we're optimistic that China will see the wisdom. And if they don't, we'll go back to training. And then under the table, you kick them. OK, so let me I mean, first of all, I want to I mean, we can get. Well, yeah, I have a question that responds more directly to what you're saying. But I do want to go ahead and kind of preemptively clarify a couple of things I'm saying. When you say when finally Trump crosses the threshold or the American president crosses the threshold where they see that some kind of very significant coordination between the U.S. and China is required. And I would say, you know, different presidents have different thresholds. Let's jump into Trump. But one can imagine a president who would be close to that threshold now.

Trump, I think, is not that. But whichever president does that, I agree they're not going to say, hey, can we be friends? They're just going to say, hey, we have this common interest. And then they may or may not kick them under the table. We can get in other ways to do it. But I agree with that. But I will say that lots of actual friendships begin with a pragmatic agreement to coordinate, right? In real life, they begin that way. And in geopolitics, they begin that way, right? Yep. And so that can happen. The other point I want to, and I think, and the reason I think it will have to happen is not that I think the degree of, I'm not saying we're just going to need, well, I'll say what I am saying. To get back to the fine-grained transparency I think we're going to need around the world, I'm not saying we'll be such good friends that trust can substitute for the transparency.

I'm going to say – I'm saying you just don't get that kind of transparency in the real world with enemies. It just doesn't happen. You know, there's this thing in my book – I'm trying to get this term to catch on. It hasn't. There's organic transparency, by which I mean there's a degree of ongoing knowledge you have about what's going on in friendly countries that you don't have if there's very limited engagement and communication. If your business leaders, your scientists, your cultural leaders are going to the same conferences as theirs, having drinks afterwards, you've got all these contracts and commercial relationships, you just know more about what's going on in laboratories even than you would otherwise. So I think that's an important thing. But I mainly you don't you don't have to push back on that or anything. I mainly want to clarify I'm not advocating friendly trust as a substitute for fine grain transparency.

I'm saying I think it's a prerequisite for fine grain transparency, including some actual monitoring equipment that's part of a verification regime. Right. Like even that, beyond a certain point, you're just not going to let a hardcore adversary have. Right. That degree of insight. So, I mean, that's my argument. I'm not saying we'll be friends at first. And I'm not saying the reason we have to eventually be friends is because friends play nice. Yeah. It's strictly that that's my clarification. Yeah, I think it's an it's an interesting line of argument about I might even flip it. I mean, it sounds like partly you're arguing for friendship as a prerequisite and then partly as a consequence of a sort of forced collaboration on verification. And I mean, depending on how you slice it, the history of U.S. cold U.S. Russia interaction during the Cold War may fit differently in that frame.

I mean, you know, there are certainly cases where we collaborated in a whole bunch of ways, including sharing nuclear safety secrets with the Russians just because it was in the common interest to, you know, to not do that. You wanted strategic stability, including, you know, come to agreements about first strike capabilities and, you know, the amount of time that would take projectiles to reach each other's countries and all this stuff. um you know we we also had verification agreements that we did enforce and that were quite successful and to varying degrees um and then there there is of course the fact that like to your point about where things lead we didn't quite get to get this measurement to go all the way through because the Gorbachev era ended quite abruptly but there was the notion of Gladnowsk and Perestroika towards the end of it where you know now we have this like kind of uh warming up of relations maybe that maybe that that's a thing but Gorbachev feels like he might have been a bit of a bit of a fluky thing um yeah I mean where that goes I don't know uh I guess I um I remain sort of skeptical

so first off I'm optimistic about verification technology in the long run from a technical principle standpoint uh it is in the nature of technology and AI in particular to make it harder and harder to make false statements about measurable things. I don't mean in the social media sense. I mean in the, like when you design devices that allow for finer kinds of collection, like intelligence collection. Obviously we're all confused about what's actually true based on social media and crap. But what I mean is literally like measuring finer and finer energy signatures, measuring, you know, what's going on inside data centers more and more accurately. And indeed even how people are possibly hiding compute from the, you know, from the foundry to the data center floor and stuff like that. So yeah, I think overall, I think technologically in the long run, we may just be able to kind of smoothly, adiabatically ride that curve from the mode of crisis when the intervention is super coarse. Like if our satellites see this thing happening, a gigawatt of power being diffused,

then we will do what we have to do. If you don't build a model that has passed these safety evaluations in this way and then do this with it. Those are the kinds of statements that, believe it or not, we have roadmaps that are on course to be able to assert things like that in ways that are trustless. And in fact, in some cases, in ways that even require zero knowledge about the actual models that are being built. These are kind of like zero knowledge proofs. There are a bunch of companies that are doing stuff like this. We just need to let that play out, which is part of the reason why I hate to go back to bashing on David Sachs, but for God's sakes, when there is a class of technology that is literally like poised to solve or at least tackle maybe with promise the exact problem that you think is so load bearing. Maybe you should celebrate that progress rather than pretending that we don't have a solution for this. And almost, it seems, working to ensure that we never do. That to me seems like almost like criminally stupid. And I have a lot of concerns

about the kind of cultural implications of that sort of thing. There's like a self-fulfilling prophecy element to it there that I think is deeply counterproductive. But yeah, so I – Maybe you should pitch a startup to SACS and try to get funding that does verification. But yeah, can I ask you a – so back to threat as a critical part of or prerequisite for coordination. What about just using leverage in a less threatening way? I mean, we have a certain amount of stored up leverage because we're not letting them have the chips. Of course, there's leakage. I read the other day that maybe hundreds of thousands of advanced AI chips have gotten to China. That's, I mean, there's hundreds of thousands of chips in it in one Elon Musk data center, I think. So that's not necessarily a flood. the floodgates opening, but I would say if you agree that we have preserved at least some significant leverage by virtue of the chip control,

one can imagine, you know, saying, okay, we will slowly give you access to more and more in exchange for this. Now, I might add, the day may come when China is so freaked out about this that it's not going to take a lot of threat or leverage to get them to realize that, Hey, a global pause winner. I mean, especially if it's true that their companies are using so much distillation that, you know, they can pretty quickly. Well, who knows? I guess without some significant training run, you can't catch up quite the way they have. So I'll take that off the table. But anyway, what about the more the less threatening kind of leverage? Because remember, in your scenario, there will be growing verification. So if the concern is that, oh, but maybe they'll take these chips and run with them, and then Dario's worst nightmare will materialize and we'll all be living in concentration camps,

you know, presumably, if you're imagining a gradual transition to greater verification and you're releasing the chips incrementally as the verification grows, in principle you can do it that way. Is there a reason you'd rather include more threatening things? Well, so ultimately the idea behind the threats is they are the last refuge of the scoundrel. I mean, they are the last best hope that we have for, you know, like what if China doesn't believe what we think they believe? What if they see an opportunity? In particular, so here's one destabilizing element that just about like no one has thought about, but I think is we tried to bring attention to this in our first newsletter. If you get to the situation where China and the United States have a tense handshake on the basis of what we've just discussed here, and now you imagine, so part of China's adherence to any kind of agreement

will possibly rest on their belief in the actual threat model that like loss of control is a real thing. For whatever that's worth, the idea of making progress in AI control and AI safety, AI alignment should be pretty concerning. If China makes control in AI alignment, then that may reduce their level of concern. If they believe they've just invented an alignment or control technique or that we've invented alignment or control technique that moves the risk of loss of control from 80% down to 20%. How low does that percentage have to be before they start to suspect, hmm, America might just renege on this agreement suddenly, try to make a break for it? This is a profound source of destabilization that no one has grappled. For a long time, the idea of alignment has been viewed as this blanket positive. And I mean, I did alignment research for a long time. Ed did too. That's why we did it. We thought that what could be better than just, you know, than that. But it does undermine, if China comes to the table in good faith or if some part of their coming to the table

comes from that concern, it undermines that kind of bedrock. And in that scenario, if for no other reason, you will want to have the ability to threaten, to be able to impose countermeasures. Well, I mean, you might also want to grant them the fine-grained transparency that would reassure them, you know? Let them assign an intelligence agent to live in Dario's house, whatever. I mean, you know, So it's like that. I'm joking about that. But generically, that is the alternative solution is symmetrical, fine grain from transparency, which I personally prefer to threatening a nuclear power. Right. It's just that it's always both. Right. Like transparency is necessary. And as is enforcement in arms control, they're often referred to like these are the two sides of the coin. you have verification and then you have actual enforcement and by enforcement they mean the nasty levers of power because ultimately you would expect China to do the same thing they would say if we find out

that you've been doing this sneaky alignment thing that allows you to achieve super intelligence if we get the slightest hint of that it's nuclear war you would expect that threat actually I should clarify where, like, so why my emphasis is where it is. And this might, I don't know that there's as much daylight between our positions than it seems. So one key piece is, look at the relative postures of the U.S. and China on this issue. A lot of people tend to think, oh, America is like the world's preeminent superpower, and therefore we're right and tight, we're good to go. The Chinese, sorry, the U.S. Frontier Labs, if you talk to folks in those labs, ask them, say, on the, So, you know, well, OK, I won't mention the team, but like on a team at OpenAI, you ask what fraction of people who know the most sensitive algorithmic secrets of this group are Chinese nationals with, by the way, therefore obligations to the Chinese state. It's just a simple matter of fact.

Well, in China's view. Right. But but these. OK, so with with exquisite levels of leverage that the Chinese state is able to bring to bear on these individuals. Right. We're talking denying their grandmother or their mother insulin medication, preventing their brother from traveling or opening a business. Are there documented cases of things that extreme, the insulin? I mean, is that on the record or you think it's on the record because of a report that we put out about 18 months ago where we talked to people from the intelligence community who are describing a case that was not yet public. Actually, so this example is kind of informative. So there was – And are there names associated? We can go talk to the – we know who the person is whose relative was threatened. And that's on the record. No, no. So no, that's a good point. So yeah, in a sense, you have only my word on this. Yeah, so that's actually an important update. Yeah, calibrate accordingly. But the story that we were told was – or a story was that there was a power outage at Oakland that affected Berkeley.

and that um so the the dorm power went out and all these like chinese nationals on the floor were freaking out because they had an obligation to do a time-based check-in with their handlers in beijing through the consulate or through the you know the diplomatic office and uh so so the we've had this described to us on a number of occasions um by folks at the fbi at the diu and so on who've had to kind of map these networks and like china's abilities so they have a bunch of different consul, it's a bunch of different diplomatic outposts. They all manage their people in different ways. But Chinese nationals have an obligation to report to these outposts. And those obligations are enforced. Like, you know, there's obviously a social credit system in China that actually materially affects your life prospects and so on. Like the ability and level of granularity with which they can put the squeeze on somebody is pretty extreme. And so when you look at the frontier labs that are recruiting from from berkeley not just from berkeley explicitly from xinghua university from the beijing academy of ai from all these like explicitly people as like

pla affiliated uh organizations um like it's it's a lot like there are literally dozens of open ai employees that come from you know xinghua or by or these sorts of institutions and even more chinese nationals who've been trained at institutions like berkeley and um so when you ask ask some of the the people in these groups, like what percentage have access to these, these algorithmic secrets, they'll tell you, like, it's not double digit percentages, it's like sometimes north of 50%. And so the placement and access that China enjoys, and by the way, this is also true of like all layers of the stack. It's extra true at the frontier lab level, partly because of the naivete of like Silicon Valley people who for years were just like, oh, another fun anecdote. So a guy we know was uh he sort of playfully asked me he's like so um let's talk about russia for a minute of the top 10 um known and prioritized operatives russian operatives working in the united states um in uh this was circa 2020 sorry 2014 he asks me what um what fraction would you guess were

uh sorry he says where where do you think they were and i go well dc is the spy capital of the world. So of course, probably a good fraction in DC. He goes, oh no, none of them were there. 100% of the top 10 were in Silicon Valley. And he asks me, since then, do you think the tech universe has become more or less interesting from a national security standpoint? So the labs are this like prioritized target for collection and for sabotage and all the other fun things you could imagine. So were you, so you're deploying this in the service of explaining a kind of threat China could make or a kind of transparency they have, a kind of insight they have. Yes. At all layers of the stat, so we're going to come out with some investigations on this stuff. This is true at the level of the data layer. If you look at the movements, who's actually selling data to whom and how are Frontier Labs consuming that data, there's a Chinese footprint there. If you look at like a big one, I'm not just saying like a one-off thing.

I mean like things that matter at scale, even for the behaviors of these models. It applies to the data center layer. So if you look at like who's building data centers in the United States, often on suspiciously short time horizons, being able to get access to transformers in six months when they should be on three-year back order from China. Like this is like their level of penetration of the North American tech stack is obscene in a way that it just isn't reciprocally. And so from a human placement standpoint, from a supply chain access standpoint, Right. China just enjoys advantages that we do not have symmetrically. And we have a lot of capability that we need to build up to be able to get to parity on this stuff. Our one last best hope right now, the big kind of equalizer is cyber. And the problem with cyber is increasingly there's this view and we're skeptical of aspects. So we, in other words, think perhaps the mythos have superior cyber capability, you're saying. Right. Well, that's right. So nominally, so the CIA and the NSA have really doubled and tripled down on cyber for a whole bunch of reasons, some of which have to do with risk appetite and not wanting to put people out in the field as much, but also just like the leverage that comes from technology.

So they're really over leveraged on cyber right now and physical placement and access and supply chain, like these sorts of things that we need to kind of. But it's a problematic kind of leverage. Yeah, exactly. Exactly. Like China is in our bones in a way that we are not in their bones. and in terms of the collection ability, right? So the ability to verify in informal ways that, you know, hey, is OpenAI letting a rip secretly and like running a big month? Well, let's ask any one of our like, you know, dozens and dozens of people who are in the lab, some of which are explicitly spies, but most of whom are just people we have leverage over. Let's just ask them, like that's one option you have. You also have espionage in terms of physical implants in the data centers, supply chain attacks, all these things that they can do that are much harder. I'm not saying we don't do them, but they're a lot harder for us to do for all kinds of reasons over in China. And so we need to get to parity. Cyber right now is a really important lever that we have that we're relying on too much. And I say too much because it's entirely plausible that as AI systems get better and better, they get better at defense than at offense.

So like a given piece of – And that's not a consensus view, right? But it is your view. So we have another report that we'll be putting out on this. It has been a semi-consensus view at Anthropic among some of their security folks for a little while now. Just that they've seen trends. The argument is something like this. A given piece of software will have finite complexity. There are only so many ways that you can hack it or crack it. So if you direct a sufficiently powerful model or agent at that piece of software, eventually you'll buff out basically all the vulnerabilities that actually do exist. Now, this is fallible. I think it's a weaker argument than it seems in a whole bunch of ways that don't really matter for this discussion. But the point is that you actually could enter a regime in which you actually do have pretty surprisingly good cyber defense versus cyber offense where the kinds of vulnerabilities that we find and look for get a lot harder to find and look for in the kind of post-AI world of like just 18 months from now.

If you do that, suddenly offensive cyber operations do not give us leverage that we're used to. And we find ourselves in a world where from a cyber standpoint, we've got nothing too special. China, though, has supply chain, humans. They've got physical access, all kinds of things that we don't have. And so the ability to actually enforce if China then goes to try to renege on their commitments suddenly vanishes and is seen by China to vanish, which emboldens them to do more. And so there's a sense in which the lack of parity. This is one of my big issues with Dan Hendricks's main framework is I don't think it accounts for the objective situation on the ground as seen by a lot of folks. I'm not aware of like special like SAP, like special access programs, dark black programs or whatever that I'm not aware of, obviously. But by and large, from what I've seen and heard, it seems like this is more or less the situation. And so I'm much more concerned about U.S. offensive parity than I am about China's ability to do things that would knock our stuff out of the swim, say. you know anyway that seems to me to be like a really important element

so I know you've got to go in a few minutes and I have there's two questions I'm dying to ask you so I'm afraid we're going to have to cut to those one is I'm sure you've read the part of Dario's Machines of Loving Grace paper where he talks about the need for a block of democracies to beat a block of authoritarian countries, obviously led by China, to super intelligence, use that to, I think it's fair to say he wants to bring them to their knees, but in any event, what he wants them to, and demand, his demand, as I recall, is, I think there's almost verbatim, is to get China to, quote, stop competing with democracies. Now, first of all, I don't know exactly what he means by that. I've never heard anyone ask him. Some people think he means regime change in China. Something he means something else. So my first question is, do you have a sense for what he means? And secondly, what is your view on this overall question?

Do you share what seems to be one of his assumptions that China does want to impose a system of governance on us? and for that reason alone it's the race to super intelligence if we can't avoid it is existential. Of course in your view it's existential either way because we're toast if the planet gets there but if you could tell me what you think his view is and then what your related view is on China's kind of foreign policy aspiration. My answer in a weird way comes from the same uncertainty in both cases. I don't know what Dario... I've met Dario, I talked to him one time, like two or three years ago or something, right? So it was like an hour-long conversation, and that's about it. I don't know where his head is at and the specific meaning of the words. I generally... The threat model I've just laid out here is certainly one in which you don't need to frame things in quite the way that Dario has. It's in the self-interest of the

Chinese and the Americans to do this right. But that self-interest is, in my view, insufficient to kind of cause everybody to hold to their ends of the bargain here. America needs more offensive heft and a willingness to deploy that. And yes, it needs verification tech and so on. I think, you know, like a block of aligned powers is actually really important for the West from a supply chain standpoint. It gives us just more leverage. I would rather the West end up with much more leverage than the Chinese. I'd rather approximately the situation be reversed. Right now, one thing we didn't talk about is robotics and the massive, massive advantage that the Chinese have. If you just let things play out status quo right now, we are fucked. We are so fucked. I mean, if you look at the dark factories that the Chinese are spinning up, the scale at which they're just pumping out robots, humanoid robots, like everybody talks about the aging Chinese population, forget it, not going to be an issue. Like this is just, It may be an issue for them domestically in terms of – the jobs issue may be an issue for both, but go ahead.

If they can subsidize universal basic income or true total luxury communism or whatever through this sort of thing. Yeah, I mean anyway, so in terms of whether we need that block, I think the scenario as I see it is you need first and foremost verification. from that first moment of course verification using like satellite telemetry and various other collection options um that the the national security apparatus already has vetted and available to it and that it institutionally knows how to process and trust um you you're going to kind of start to back into more and more fine-grained tech and that is more and more permissive as we get more and more resolution and more and more access and as we do trust building i will say um historically the chinese response when when american diplomats say we need trust building measures the chinese response is to say well we can't do that because we don't trust you if it's like this is like literally how but they but they they believe that and we feel the same way about them right but like the trust building measures the purpose of them is explicitly to get

us to that next level of trust um it's so so there is this this sense in which the adversarial dynamic like at least in ways that are intelligible to us and i think to your point this is important We are interpreting it as meaning that they are not engaging in good faith in that way. It is possible there's a story, and reasonably likely, of course, that there's a story they're telling themselves about how the situation is quite the opposite. But yeah, so block of aligned countries I think is really good insofar as it gives us leverage. By the way, I think non-kinetic, non-cyber options are really the first thing we need to look for. There are ways when you talk to some of the supply chain people of doing one shot, six month slowdowns of China's, for example, chip ecosystem where, you know, you only can use it once, but you can use that card for a six month slowdown. So the kind of thing that could be fairly punitive and reasonably non escalatory. But, you know, you want that wider range of optionality. It helps you cross that spectrum to have a block of allied countries.

And I think certainly more engagement with the Five Eyes and NATO would not be a bad thing for the U.S. at this point. And you agree that the alignment should be explicitly ideological. In other words, you know, Dario's view, these are democracies. So I guess if even if Russia just begged to be on board and they whatever, we'd say, no, sorry, you know, your elections are phony. I personally think there's real danger in casting the narrative that way. I think it tends to become a self-fulfilling prophecy of a bipolar, you know, ideologically opposed world. But is it your view that ideology should be a big part of the litmus test for inclusion? I'm fairly ambivalent, honestly, probably because I haven't thought enough about it. I mean, you know, like anyone, my reflex would be I want the democracies to win. but with zooming out, there's just no winning to be had in a super intelligence loss of control race. And so I think it's really just about trust and not even about trust but our ability to verify.

So if we have a country like France where, as you said, our friendliness to France just means that we have all kinds of ways of monitoring what's going on there that we don't have with Russia, yeah, to me that's just an extra tick in the verification box for France. And it happens to be that a lot of the countries for which we enjoy that level of closeness, we have that relationship because they are pluralistic liberal democracies and all those wonderful things. But, you know, as a counterpoint to, I guess, Dario's framing, it's not obvious to me that like Russia gleefully jumps into bed with China on this one. Like it's like North Korea is at a fraught relationship with the Chinese and the Russians. Like these are not coherent ideological blocks. I mean, socialism with Chinese characteristics, like the second part of that is not social is not. Right. Right. And just as Russia is. Well, in Russia, I mean, look, I wish people would quit calling China a communist party. It is run by I mean, a communist country. It is run by a party called the Communist Party. But anyway, go ahead.

Exactly. In much the same way as, you know, Russia is sort of like a vaguely fascist flavored authoritarianism with religious overtones. like you know i don't know what what term is right for these things and the beauty is on my view worldview i don't particularly have to care like what i care about is don't build the fucking super intelligence that gets us killed let's have the verification check in place and let's have the consequence and and the the block here is useful to the extent that it helps us impose consequence for defection so having having for example korea on board and south korea to be clear is important because Samsung is there, because TSMC is, because SK Hynix is there, right? Having Taiwan on board is important because of TSMC. So like, these are all helpful things. We expect to be able to monitor what they're up to so that our concern will be mirrored hopefully by them. But like, you know, beyond that, like, let's just make sure we don't do the stupid thing. And so I think that this is one sense in which, you know, I may be wrong in all kinds of fun and interesting ways, but this worldview is maybe a little bit more robust than the one Dario is kind of putting forward.

Okay. Well, we end on a note of agreement. Dario's view is not necessarily the optimal guiding light. But smart guy, I think he's sincere. I think he's ideologically sincere. I do not have the degree of cynicism that I bring to bear on Sam Alton. Which isn't to say I want to appoint Dario, make him in charge of world affairs. I wouldn't want to appoint myself. Speak for yourself. I would want to appoint myself, but I'm starting to think I should give up on that career aspiration. And you're young. Don't give up on it. I just have great skin. I wish I did. Or we're young. So anyway, thanks. It's been a great conversation. There's also more we can talk about. So I hope you'll come back at some point and we can keep going. Yeah, same here. And, again, people should check out the podcast and all your reports of Gladstone AI.

That's the name, right? Yeah, AI.com for the newsletter. Okay, right. Exactly. Is that on Substack? Yeah. Okay, good. So check it out, folks. and we'll see you next time, Jeremy. Thank you.