「AIチェルノブイリ」がトランプと習近平に即時停止を迫る——Jeremie & Edouard Harris

We have agents hacking Huggy Face. We have agents hacking the Australian government. Do you think we should get some bombs ready in case some data centers go rogue? This is one way in which I think we do have a disagreement with Eliezer. Certain things don't have to be said. You don't think a treaty is a plausible solution, correct? The reason a treaty is a non-starter is that it needs to be ratified in the Senate. At some point, some rogue AI incident we expect is going to happen that's actually going to have a death toll. 93 million people lose power for a month and there's 2 million deaths. On the backside of that, we expect Trump meeting directly with Xi. You must make sure China stops. And today, we do not have the verification technology that allows us to confidently do that. Welcome to Doom Debate. My guests, Jeremy and Edward Harris, are brothers with graduate-level physics degrees who co-founded Gladstone AI.

In 2020, they were running a Y Combinator startup. Then GPT-3 came out, and they dropped everything to work on AI risk. In March 2024, they published the first U.S. government-commissioned assessment of catastrophic AI risk, and now they've just published a new post based on interviews with the top arms control negotiators. Their take is that a U.S.-China AI treaty isn't going to happen if we continue with the status quo. They also have a sub-stack that they call the endgame, because that's where humanity could be headed. Do we have a winning move in this endgame? Let's find out. Jeremy and Edward Harris, welcome to DoDebates. Hey, what a smooth intro. That was a great segue, man. Thanks for having us. Thanks for having us. No problem. You know, I always want to build you guys up, so then when we debate, It's like a worthy opponent. Although this one, I don't think it's going to be a very contentious debate. I think it'll be a relatively minor in the weeds debate. I think we're going to probably agree on the broad strokes because we've already watched each other's content, correct? I think that's about right.

I mean, yeah, we'll get into it. But I think there's some areas where we have maybe more contrarian takes in sort of standard AI safety circles. But I think you do too. So, yeah, we'll see where it goes. Nice. All right, so a little bit about the backstory. Ed, you tweeted in July 2020, Jeremy Harris and I are taking a couple of weeks off from building Sharpest Minds, which I guess was the Bicom and Intercompany, to investigate AI risk. And Jeremy, you told Joe Rogan that it started with this panicked phone call. Dude, I don't think we can keep working like business as usual. So my question to you guys is, what did you see in GPT-3 that made you panic and change everything? So I love how you're taking it way back to the July 2020 tweet. I had completely forgotten about that. So awesome. But basically what happened there was what we saw in GPT-3 was basically, there's two things, right? First, GPT-3 was a point. It was an impressive point. It was AI pushed to a level that it had never been pushed before.

You know, the biggest model at the time, just in terms of parameters, and the most capable. And there was a bunch of things that it could do that nobody who trained it or built it expected. So they just trained it to be a really smart autocomplete at the end of the day, But check it out. It could write OK code in HTML and things like this. So things that surprised the creators of it. So it was an important point and a milestone in and of itself. But more importantly, it was also a point on a line and a trend that had been documented for at least the past few months explicitly. There was a paper that came out, the Kaplan paper, scaling laws for neural language models. and the GBD3 paper itself also had these scaling laws. This is back when the Frontier Labs actually published stuff, right? So you could actually look and see, like, well, ask yourself, what happens if this just keeps going? And it has. Nice. And then even before 2020, what was your guys' entry into the Rationalist community

or any adjacent community or EA? How far back do you go? So I think this is one way in which we're a little weird. We got introduced to AI risk really with the Wait But Why post, the famous FOOM Wait But Why post in 2015. Yeah, that kind of did it for us. I remember Ed surfacing this to me and being like, hey, just read this shit. This is wild. We read it. We're like, damn, this is just true. But we are explicitly not of any of the standard communities. And I'm not saying this in the way that a lot of rationalists or effective altruists will tell you that they're not rationalists or effective altruists because that's part of how you signal that you're a rationalist or an effective altruist. We visited Berkeley. You know, we spent time there, but we don't live there. We've stayed out of those ecosystems. We are a product of basically the kind of Y Combinator ecosystem that's where we founded our company. We've just been founders this whole time. And just founders who understood the technical arguments for loss of control found them pretty

hard to refute. And I think from outside that world, it was a lot easier to decide to do something. There was a period in 2020 where Open Philanthropy was the only funder of AI safety kind of work, right? And obviously, they were aligned with Dustin Moskovitz and all the politicization that comes from that. We were expressly told by Open Phil not to talk to the U.S. government to really do much of anything on the AI risk issue. because if you talk to the U.S. government, you see, they're going to hear you say AGI is going to be a really big deal. Maybe the labs have figured out how to build it. And then they're going to ignore everything you say about the risk. They're going to do a Manhattan project and just get us killed sooner. That was literally the frame that we were given by multiple people at OpenPhil at the time. And we took it seriously for all of two weeks before we, our instinct to startup founders was really just like, let's just like do a low stakes test of this. Let's go out and actually talk to a couple of people who aren't empowered to do anything

with this information and see how they react. And what we found over and over again was in much the same way as the average American right now is passing the test of polling with flying colors, correctly reasoning that, yes, in fact, loss of control is a real issue and not a hoax. The folks we talked to back then had the reasonable response of like, yeah, I mean, it seems pretty far fetched. Probably scaling isn't going to keep working like you say it will. But if it does, like maybe we have something to be worried about. And so this was one of the things and like not a gripe against open fill or the effective altruist ecosystem per se. They have been remarkably prescient. But this was a way in which I think they kind of like selected themselves out of doing certain things that would have been really meaningful at the time. And as outsiders, we were basically immune to that pressure. We refused any kind of funding that was tied to that. We literally stood up an entire for-profit business just to subsidize these like insane profitless activities of running around and trying to like find ways to kind of more deeply understand the governance problem, the technological alignment problem. You know, like you can imagine when we were in the early 2020s going around and telling

people, hey, this like tidal wave is coming. This is going to be a really big national security story, like maybe the single biggest. And like, like who's handling this? Like, is anybody working on this? And people, their first reflex was to go like, hey, yeah, this is probably handled. Like, I wouldn't worry about it. You know, somebody, I'm sure somebody over at this agency or that agency is paying attention. And over and over we would go place after place and find that, you know, people just weren't. And that's like, I think the single biggest problem in some sense, that very often the right hand doesn't know what the left hand is doing. And it's true fractally at the level of the economy, like the private sector and public sector. and it is true within the public sector too. Within the government also, it's kind of cliche to complain about bureaucracy, but it really is a huge, huge problem. We've seen this in our business that we run. We mostly sell to the U.S. government, the armed forces, but also just talking to people. Some of the best people in government are mostly just hamstrung by bureaucracy

simply because there's a 30-day internal approval process for this. There's a 60-day waiting period for the money to move for that. And you're literally talking to these folks, and they're like, we're moving as fast as we can. And it's like, not really, though. Not really. And I think this is one of these things where if we hit a crisis point with AI, we have this feeling where we're going to discover how fast we can actually move in a way that we maybe haven't had to in several decades. Nice. Okay, we're going to definitely keep diving into your latest report about what's going to go down with the U.S. and China and your recommendations. But first, backing up a little bit, I want to get some more info about you guys. I first became aware of your work in, I don't even know, a couple years ago, 2024, I think. The first time you guys went on Joe Rogan, I had the Joe Rogan feed, and I noticed, like, oh, wow, Joe Rogan's bringing people to talk about AI safety. That was still pretty novel back in the day. So I listened to the episode, and I was just like, oh, my God,

you guys are communicating in, like, such a relatable way. Like, this is what we've been looking for, right? Like, the AI safety movement has been looking for people who can go on Joe Rogan and sound, to be honest, you know, not autism spectrum, right? Which, you know, and I'm kind of like that, but I am a self-described, like, I do think I'm, like, a little bit ways on that spectrum, like, you know, on the mild end of that spectrum. But, like, would you guys, in your opinion, do you think that you guys are actually neurotypical or are you just good at communicating but you're also kind of aspy like the rest of us? oh geez how so are we special or are we really super duper extra special i think that's a question for our mom first and foremost but um so i i do think there's something so so um ed with with your apologies i'll say that i think there's something uh neuroatypical about ed in particular um in particular his ability to look at things like scaling curves in years like 2020 or you know 2019 and go, this is a real thing.

And then to have an argument, have the conviction to argue me into dropping out of our startup, like just leaving it to our earliest employees, walking away from a profitable company and then like going off to do whatever the hell it is that we ended up doing. He's done that a lot. I mean, you know, he's like one of these early into crypto guys. He's one of these early into AI guys, early into COVID, like all the usual things that are very socially costly at first. I think it's sort of sad that these things kind of bounce off you, or the social pressure bounces off you in a way that it doesn't off of most people. I am more like most people. And in that sense, I mean, I don't know, Ed, you present like you're not a weirdo, but I think that there is that element to it. I'm very fucked in the head, is what Jared is saying. Yeah, you're pretty fucked in the head. Yeah. Yeah. Yeah. All right, great. All right. We established that much. And then in terms of your backgrounds, did you guys both study physics in grad school? Is that right?

Yeah. Yeah, yeah. We both studied physics, yeah. So one of the nice things there is that it kind of gives us a common language where, like, I guess to expand a little bit on how that ties into how fucked in the head I am, I definitely see and feel and believe in math more than probably the average person does. So if you show me a graph and I believe the data that gives rise to that graph and I can, you know, I can make a story for myself that like, actually, this is there are fundamental reasons why this graph exists, as was the case for AI. This like there really is. I think it probably is at this point a law of nature that this kind of scaling thing that but intelligence is just is just not actually that hard. Like we look around ourselves and we're like, yes, like I'm so special. You know, like, we all went to school, and your teacher's like, oh, you're so smart. You're going to amount to, like, amount to something. Like, good for you, and da-da-da. We tell each other these stories, and we, you know, we pin our personalities and our values on how smart we are and stuff.

But fundamentally, like, it looks like the way the world is set up, you can stamp it out in a factory now. And that hurts a lot of people. And just looking at that truth is painful. Totally, totally. And, you know, it's happened before to the human species with a lot of things. You know, we replaced manual labor with beasts. And then, you know, it happened with computation. I mean, you know, computer used to be a job for a human, right? And that must have felt bad to be like, nope, sorry. Look at me, I'm the computer now, right? It's like a physical, mechanical computer. And now it's happening with intelligence, but it just sucks because intelligence is like the last frontier. That's right. But, yeah, I mean, I totally agree. Like, we just have like a thimble full of intelligence in our brains, and we're like the king relative to the other animals. But now it's like there's, yeah, I think you guys are on the same page as me, super intelligence probably has quote-unquote god-like powers, right? Does that sound like a reasonable guess? I think almost the phrase, I wouldn't use god-like if only because it's sort of vaguely, people want to start to lather things on top of that, but for whatever

threat model you're concerned with, if intelligence is the bottleneck, that threat model should be a live concern. The other thing, though, is intelligence is also the resource that resolve bottlenecks. Like, that's what makes it, like, scary at a higher level. It's like, if I, you know, if you've ever played Factorio, if you've ever, you know, tried to optimize a real factory, you'll know that it just, it takes intelligence, and there's a systematization to it, but it takes intelligence, and with a bit of intelligence and a bit of, just a little bit of elbow grease, you can make your factory output massively greater than it was before. so intelligence is not just the bottleneck but it is the thing that resolves that can be used to resolve all other bottlenecks this is part of why in our thinking about this capabilities and safety capabilities and alignment everyone always has this view of them as separate things I guess more and more people are having a more nuanced view that they are inseparable

I think they are inseparable just because ultimately intelligence is the ability to solve problems. And those can be anyone's problems. I mean, essentially, like, the orthogonality thesis applies is the lesson, I think, that we're learning over and over again. But just, you know, you see this in bio, right? The defensive applications and the offensive applications arrive at around the same level. The same is true in cyber. We, you know, mythos is as interesting for defense as for offense. And I think the beatings will continue until morale improves. I mean, we're just going to keep finding that every time we try to solve a problem, and this is a real issue as well for things like tool AI, it's not clear to me that we can hit a target that's as incredibly precise and delineate the defensive or positive or whatever applications in a way that rules out some of the more catastrophic negative ones. I think it's just this fundamental property of intelligence that we're now running up against.

Okay, so just finishing out your background, okay? Do you guys both go to the same grad school? Are you in the same year? Are you twins? No, we're like three years apart. Actually, if you go back, it sounds even more overlapping than that. We both started off more or less in biology. Ed did bio. I did biochemistry. And Ed went into physics. What, two years into my biochem degree, switched into physics as well. Ed went into biophysics. I went into foundations of quantum mechanics. I was really, really interested in, like, you know, many worlds, quantum mechanics in particular, and that sort of thing, and ended up actually writing, like, a popular book about it, not popular because a lot of people read it, but popular because of its intended audience. Okay, good clarification. Oh, yeah, it was a Canadian bestseller. You have to add the qualifier. It's, like, completely deflating. But, yes, it was a bestseller nominally.

And it was a lot of fun to write. But yeah, so it was foundations of quantum mechanics and like paradoxes in quantum mechanics was kind of my jam. And then Ed did a lot of stat, like Bayesian, applied Bayesian probability theory in bio. Yeah. Did you guys end up diving through the Aliazda-Budkowski sequences, I guess, after 2014 when you first got into the whole issue or no? Yeah, I read them at some point. They're good. I mean, the thing is, like, my PhD was almost, it was almost a PhD in probability theory applied to biological systems. So I had done a lot of work in that area. So reading, at least my recollection is there are parts of it that talk about, like, literally Bayesian probability. And so you're just kind of like, oh, yeah, you know, I know this stuff. But, yeah, there's a lot of interesting thoughts in there. Yeah. Well, you guys can spot check whether Eliezer knows what he's talking about because you have some expertise. So you said the Bayesian probability stuff makes sense. And also, you know, he's got the whole quantum theory sequence, right?

And I think one of the main claims of that sequence is that many worlds is true. And anybody who still thinks the Copenhagen interpretation is adding value is an idiot. Would you agree with that? Yeah, yeah, that's obviously true. Okay, so I spent my time in grad school trying to convince all the people around me that many worlds was not true, but to be taken seriously. There's like so many things, like so many important ideas. It sounds ridiculous. And yet the reason that it sounds ridiculous is that it fails to appeal to like some aesthetic sensibility that we all have about the world being one thing rather than many things. But when you actually look at the set of assumptions that many worlds makes, it's a much, much more restricted set of assumptions than what you see in, like, yeah, Copenhagen interpretation or Bohmian mechanics or whatever. It's actually the most parsimonious by far of any interpretation of quantum mechanics. And, like, I think that much is actually, like, relatively difficult to dispute.

I'm not saying you couldn't dispute it on some level, but people have a really, like, they don't even get to that part of the conversation. They check out at the level of like, what, this theory results in parallel universes in much the same way, as somebody might have said, what, the scaling theory results in rogue agent swarms hacking into stuff? Like, it's sort of, the human brain, I think, struggles to generalize in ways that are almost just one step removed from where we are right now. Is it possible that part of this is also just that the dominant theories, both in quantum mechanics and in intelligence, are just like the Copernican theory, right, are kind of, or sorry, the Aristotelian theory, are just so human-centric that they just appeal to us intuitively. So the thing about many worlds is that it stops privileging the observer and conscious observers and all that, you know, that Copenhagen stuff, right? Like all the Bohr collapse thing where you look and you collapse the wave function because you're magically conscious, like that many worlds just sweeps all of that away.

Well, funnily enough, that interpretation that you gave wasn't necessarily Boro's Von Neumann, who is, you know, I mean, like, the AI guy, right? So, anyway, yeah, sorry. Go on. Smart people can do that. Yeah, well, you know, this is great because for the purpose of this show, I always say this is the Yudkowskian show. My current position is that Eliezer Yudkowsky's writings are, like, 99% correct. Like, I haven't found many mistakes. And this is great because you guys are not, like, Eliezer devotees, right? You think for yourselves, you went and got a PhD in physics, and then you go independently look at his writings, and you're like, yeah, this makes sense. He's obviously correct. But at the same time, I've heard very smart people come and criticize him and be like, he doesn't know what he's talking about. These writings are wrong. I've heard people defend the Copenhagen interpretation. So to me, you know, people say I'm part of a cult, and they think Yudkowsky is like Jesus. So to me, it's always great when somebody walks in who's like a Buddhist or whatever, or an atheist, and being like, oh, yeah, you know, turn the other cheek to love thy neighbor. These are great teachings. I'm like, I told you, you know, just because I follow Jesus doesn't mean that I'm brainwashed about the teachings.

Well, so one thing I would say is there are places where we disagree with Eliezer or where, let's say, the actions that we would take would deviate in potentially important ways. So things that he got wrong, right? Like, I mean, obviously, like neural networks are not going to be the thing that leads to AGI. OK, fine. This seems like a fairly trivial, fairly trivial thing. I think a lot of the interpersonal stuff he gets wrong in important ways. There are posts in the sequences where he talks about, for example, the need to, I forget the term he used, but in some sense, get permission to put ideas in people's heads, get right access to people's brains through various... Like hero license or something like that? Yeah, something like that, right? And then the challenge is just that then you see these failures of communication on some level, though it all depends on how you want to measure him, because also he's going to be the lightning rod as the first guy out the gate to say the thing,

and so he'll attract the kind of criticism that comes with that, and maybe he's taking one for the team in that sense. I don't know. I guess I'm kind of mixed on that. My gripe is more with the, not all, but some of the effective altruist ecosystem, and that is obviously distinct from the rationalist Joukowsky in school. But I think there, the consequences have been pretty bad. Okay, yeah, fair enough, fair enough. Maybe we'll come back to that. And just to complete your guys' life story, so you got out of college, and I know you talked about this on Joe Rogan. You decided to go into the startup world, and it was more refreshing compared to academia, which I think I agree. You know, I've done startups myself. I even still run my Y Combinator startup. It's kind of plateaued. You know, it's not like a billion-dollar unicorn or whatever, but it's about relationship coaching. So I know where you guys are coming from and yours is called Sharpest Minds and you started I think around 2017 and you guys uh left in 2021 because you were so enamored with this uh ai stuff correct yeah yeah that well we actually so in the aftermath of the tim urban post the wait but why one um that i think we read in 2015

we actually decided then and there that everything we were going to do from that moment on would be pre-positioning for the ai moment um and so it wasn't a coincidence that when every startup that we started, including the early kind of recruiting startups. We're all oriented around data science because that was the closest thing to AI then. And then ultimately machine learning, engineering and AI and all that stuff. And so that was the case with with Sharpest Minds. This was a so we've always been kind of like I won't say Milton Friedman libertarians, but but, you know, we vibrate to that frequency in a lot of ways. And so Milton Friedman had this experiment that he proposed or this concept of income share agreements, you know, you basically, uh, your school doesn't charge you tuition up front. Instead, what they say is we'll, we'll train you for free and then you'll pay us a percentage of your salary when you get hired. And so we, um, we set up this company sharpest minds to essentially be a kind of, um, peer to peer version of that. So we ended up building, it was like the world's largest marketplace for income share agreements where you just go and find a mentor who'd work

with you one-on-one for free until you got hired. And then you'd pay them back a percentage of your first year salaries usually like five percentage um and that was such a cool experience i mean we got some some really genuinely interesting economic data that i think remains unique to this day um seeing thousands and thousands of these mentorships go by and so uh it was super cool and that was that was the thing that we decided to say yeah let's like let's leave that behind um uh in 2020 when when the scaling law has told us to do it. Yeah, it makes sense. Yeah, it was also a chance to work with some wonderful people. Do you imagine the sorts of people who are like, yeah, I'll train you for free in exchange for a possible future or whatever. They're just like, for the most part, they tend to just be really good people. So it was a nice experience just in terms of getting to know a lot of the better parts of humanity, which is awesome. Basically, our positioning, our thinking from 2015 to 2020 was like, AI is going to be absolutely huge and is going to dominate everything,

but we have no way to know when exactly the starting shot is going to be fired, so we're just going to be adjacent to the field and watch it. Yep, makes a lot of sense. And yeah, you know, that's also my theory with Doom debates. I don't know when the Doom is going to happen, but you're watching it. This is my feed at the daily. It'll be televised. Yes, the Doom will be televised. So Gladstone AI was officially set up in 2022. So the thing I'm wondering about Gladstone AI, I listened to you guys on Joe Rogan and other podcasts, and it seems like you're meeting such interesting people who are like such experts in such a wide variety of things. It seems like you guys have become really well connected in like the national security, the deep state of national security, right? So how did you achieve that? How did you climb your way up to having this kind of key role? So this is kind of a super weird thing, especially given, like, we are just two Canadians. Like, I got my green card a couple weeks ago. You know, that's like, yeah. So, yeah, it was weird.

There's a spying risk, right? Anybody you're coming in a meeting to, it's like, should we, you know, you got to be open about that concern, right? Right. Well, I mean, and as there is if, you know, a U.S. citizen who doesn't have a clearance is in a conversation with folks from the IC, right? Everybody's got to be careful all the time in this community, and people are calibrated to it to different degrees, you know, to be clear. I mean, like, so we, you know, we've never seen a document with classified marking on it, right? Like, that's not a thing that people just, like, send to us. But, you know, yeah, you have conversations with people, and often you get distilled information, and in some cases information that people want to be made public that they feel is important strategically or otherwise, and so then they'll share it with you in much the same way that they would with a journalist. The difference with us is because we were in the space so early, like before there was any money to be made in being in the space, people kind of understood that we were mission aligned. Like our background speaks kind of for itself. We've always been these sort of startup people.

We're not in it for stuff besides, like we're concerned about the doom factor here. Anyway, so, yeah, essentially, it started with the, it started actually before the report or before the investigation led to the report. We just started to talk to anyone who would listen. Like, we had conversations with literally, like, 30 Canadian parliamentarians within the first year of working on this, just, like, schlepping it startup style. We, like, we ended up sitting down with, like, the Secretary of Homeland Security to give him a briefing that ultimately he told us, prepared him for when ChatGPT came, and then uh biden turned to him was like what have you like what are you doing what's your plan for this he was like oh we've got like this four-step plan it was like he said partly uh derived from the conversations that we had um so it's kind of like wild it was just because there's no one in the space like no one was taking this seriously and yet the evidence was screaming at you such that like once you overcame that little activation energy hump of being willing to talk about it of having, in some cases, just really the startup hustle to, like, hustle your way to people who

mattered, you could actually get their ear and then, like, what a briefing you'd be giving. Like, they never heard of scaling laws. They never even heard of GPT-3. And so you're like, yeah, yeah, this thing, this thing works. And the number of times we brief people on, like, imagine if the, you know, the Russians want to deploy this as part of the, you know, influence operation in North America, you know, saturate the internet with all these bots and stuff like this yeah that sort of thing and so then when the next beat dropped and you had chat gpt people just turn around or like like bring those guys back in and it was pretty wild i mean like you know i ended up um giving giving briefings to like uh like it was a dozen central bank governors um uh you know we we ended up briefing four-star generals in the air force and and admirals in the navy one crazy story with the report that we did was it was so it was commissioned a month before chat GPT. So the eye of Sauron was not on this issue remotely. U.S. government was not thinking of AI as like a massive political issue at the time. And so by a fluke of paperwork at the time,

they never bothered to insist that the report not be released to the public. They basically gave us carte blanche to do whatever we wanted with it. And so that happens. They've locked in that trajectory. They funded it. They've approved it. And now the project is off to the races. And then chat GPT drops. And they're like, oh shit. Like, who do we have working on this? This dinky little team? And suddenly everything got politicized. I won't mention the particular person, but the political level appointee who was overseeing all this at the State Department was getting shit from the White House. It was a mess. But then there was a you might remember the UK safety summit that was announced in in 2023. We were meant to be on the ground in London. Already the trip was planned with a delegation from the State Department, all this stuff. So we're meant to be on the ground in London. And then the Biden administration goes, we got to send some people over to like meet Sunak's team and coordinate with the with the UK on this. And who do we got who can do it? And literally none of the government was ready. No one spoke the language. No one had a technical background to understand the issues except for our dinky little team, which was kind of a cooperative team. It was part of a cooperative agreement between us and the State Department. And so they're like, all right, you guys go do it. So you have the two Canadian citizens, no business like brokering a conversation between the UK on the one hand and the US delegation on this big flagship issue, the AI Safety Summit.

And so, yeah, we ended up like, you know, giving this big briefing in an old room at Whitehall and, you know, prime ministers, private secretaries there. You've got representations, the most senior levels from like all the ministry of defense and innovation and all this stuff. It was weird. It was crazy. And this stuff just kind of kept happening and escalating. And and just because I mean, I think we've just like gone out of our ways to be honest brokers in particular. again like we've never taken money from anybody i would hope we've never compromised our our morals at any point um we just sort of tried to try to do what we could so that we would have clean hands when the moment came that's basically the story i mean at this point it's like a lot of people who've seen our work over time and and who trust our predictions too which is a really big a big piece and then um who rely on us sometimes to like uh help the private like kind of help to connect to parts of the private sector that are really difficult to access like you know at the neocloud layer, the infrastructure layer, the data center layer, and things like that. So we've sort of become this weird, yeah, kind of partly networking body, but also trying

to, like, do forecasting and stuff like that. I like the image of you guys talking to senior members of the U.S. and U.K. government, and they're like, oh, so where do you source your research? And you're like, hello, GWERN.net? Yeah, that's right. That's right. Yeah, we literally cite the scaling hypothesis in one of the reports, yeah. Yeah, yeah, yeah. So we literally, so we had to, we actually had to fight a little bit for that. So there was all kinds of little fights behind the scenes on that report, like just about like which words to use. It was controversial to say AGI at the time, as wild as that sounds today, right? But like at the time it was like, ooh, I don't know if we can say artificial general intelligence. It's a little bit extreme. It's controversial again. You have to say SI now. Oh, that's right. Yeah, that's right. How does that change, right? Now it's like, now anything that's not SIs, yeah, that's right. There's that. And then there's other, you know, other little things like, yeah, you know, can you cite a tweet in a government report? But sometimes the only reliable source is a tweet because it's a tweet from, like, Ilya,

like Ilya Sitshever. He's saying this thing, and there's no other source for this thing, but he's the reliable source for this thing. So you have to cite his tweet, and you get in these little arguments. There's all kinds of little things that go into producing something like that. It was quite a surprise, but we got it out, and to the State Department's credit, there were a number of people there who were really, really AI-pilled at a time when nobody was. Yeah. But some very smart people who were like, yes, I believe, let's make this happen. So that was a very, very cool experience. Nice. Okay, so your current organization, Gladstone AI, you mentioned you make these reports, and we're going to talk about your latest report, and also you mentioned you're selling to the armed forces. So what's the business model, or is it non-profit? No, we do sell to the U.S. government. We sell to Department of War, and it's basically AI, like back office automations, for various components of particularly the Air Force and some of their test community and some of their other stuff.

But yeah, basically it's just like AI automations for the Air Force. It's an interesting business. Like if this were 2018 or something, I'd be like, this is actually a really cool business. And it's like, it allows you, it allows them to do so much more stuff than they previously could do. But, you know, given RSI might be around the corner, it's like, well, it pays the bills and it's good. But yeah. Okay, got it, got to say this. So you still do have this business that's more like, okay, it's not the ideal way to spend your time if you're just 100% trying to mitigate the singularity, but it pays the bills, which by the way, you know, I'm in the same situation. Like I still have my Y Cominator company helps pay my bills. That's why Doom Debates donations can go 100% into production, doomdebates.com slash donate. So I'm on the same page, but that's interesting. So the customers of your back office automation software, are they the same as the ones that you're giving the reports to, or is it just like kind of different? There is very much a separation between those things, yeah. So we have a business that sells to U.S. government,

And then we have our, basically our interfaces with various other elements of the government with respect to, yeah, some of the intelligence stuff and just the other stuff that we integrated in our reports. And it's actually, it's not that hard to do that because the U.S. government is so gigantic that, like, parts of it don't talk to each other by default. You don't even need to make a special effort. Just the left hand, the left tentacle doesn't know what the right tentacle is doing and so on. okay so let's talk about the the u.s china treaty right because i feel like this is very salient on your minds right now because you're like okay when treaty right it's going to be like basically like warning shot public sentiment right public pressure overwhelming grassroots support overwhelming congressional support trump she treaty right that's basically your main line scenario correct uh i might modify that just a little bit so maybe it's worth sketching out sort of like the the future as we see it from here just in terms of what the world looks like if you take scaling seriously. So the same little party trick that we've all been using over the last six years

to just basically build our careers on, essentially, just continually taking scaling seriously every time somebody Gary Marcuses an argument out and says, hey, there's this strange, magical, wonderful reason why scaling is going to stall, saying, hey, no, let's just assume it keeps going. If you keep doing that from here, right, we're already obviously at post-hugging face and post-mythos, And, you know, we'll get mythos for bio. We'll get robotics applications, surely. But at some point, some rogue AI incident we expect is going to happen. That's actually going to have a death toll and and sort of damage done on the order of hundreds of millions, if not billions or more dollars. And there's going to come a point where it becomes politically untenable to keep letting a rip in the way that we are like this idea of labs self-policing. and so well on the back side of that we expect Trump to actually have a phone call or a meeting directly with Xi we've talked to all the kind of folks in the State Department who would normally interface

with their Chinese counterparts and kind of traced out the trajectory of things when you do that it turns out actually that literally president on Xi like dialogue is actually the only viable path here so we're essentially assuming that in that moment you're going to need verification technology. Like the president of the United States is going to turn to Xi Jinping and say, hey, look, buddy, it's untenable to us that you continue this build out. We're in the world where like verification technology suddenly becomes very important. For that initial phone call, we won't have it. We'll have a really crappy set of what sometimes are known as NTMs, National Technical Means, that the intelligence community already knows how to process and trust. So satellite imagery, thermals, some human intelligence, things like that. But that's what will be on the table, we suspect, for that initial phone call. But quickly, we're going to want to back our way into a much more durable kind of agreement because you're basically gating, when you have an agreement that's that rough, that coarse-grained, and you're just saying, no, like literally you cannot use that 500 megawatt

facility full stop or something close to it, there's trillions of dollars of economic value you're leaving on the table. So you can imagine the raw economic forces just pushing in the direction of trying to unbottle this by creating verification technology you can actually trust. The problem there is there is an irreducible amount of time that the intelligence community needs to assess any new verification technology. So like we're just going to be sitting there, sitting on our hands, basically waiting for the right people to get to know the right people to assess the technology to the point where they feel they can trust it to be able to actually then deploy it. And so the more we can do early on to get there, the better. But there are just certain things that take serial time. And that, we think, is a major, major source of crisis instability during that critical period that doesn't necessarily map on to Project Warp Speed. I mean, obviously, the vaccines took a certain amount of time to come out. But there are serial dependencies here that go beyond that. And at a moment of acute geopolitical crisis, again, the instability implications and the

escalatory possibilities there start to become really pretty wild. I love that you're thinking ahead a couple steps. I've never been naturally a very strategic thinker. I don't think that many steps out, unfortunately. And my only saving grace is that sometimes thinking in terms of simple concepts happens to turn out to be powerful when I'm like, oh, AI is going to be a million times bigger. Okay, that's useful. That can lend itself to strategy. But in terms of actually playing out the step-by-step, I love that you guys have thought past this idea of like, oh, the US companies are going to get together and collude to pace the frontier and sam altman is going to do better than just saying hey guys here's a two-week slowdown while we like get capacity up for this model before we turn on capacity for the next model there's going to be like a big warning shot it's going to be like hugging face times 50 and then finally it's going to be trump and she and that's going to be the real plan right so you guys have already skipped a couple steps yeah well yeah one of the things that's going on in part is um these tech companies the these ai frontier labs they are tech companies and that means that their DNA

is like move as fast as possible and ship, ship, ship, right? It's actually quite hard to get a tech company that is in the maximal kind of vertical growth phase in revenue to go like hit the brakes and slow down because it's the opposite of government. There are just so many institutional forces pushing towards let's get this out. And so what that means is that inside these companies there's a bunch of people that the natural force is pushing still to ship, even with this pacing. But there are individual humans in there, like in OpenAI, in Anthropic, who are the ones responsible for, well, let's hold off on this next model to ship. So there actually is very much a tension and a push-pull inside of these companies right now. I think people underestimate the degree to which there's a very small number of people who have to keep doing the right thing in order for the safety argument to win with respect to any given release, whereas the opposite is true on the capability scaling side. You can have people who go like,

I don't know, but like institutionally, what wins through is just the massive momentum and obviously the incentives. I see what you're saying. So you're just like, by default, the assumption is like, yep, we have like a five-year roadmap to keep scaling, like an infinite roadmap to keep scaling and get these data centers online and do the next model and make more money and impress Wall Street. That's kind of the default, and you have to make a really strong argument why to stop the freight train, correct? That's exactly right, yeah. And I mean, these companies are also thinking about IPO-ing. There's all these considerations. And the other thing is, pacing is not pausing, right? It's a continuation that's at least intended or said to be intended to be, well, let's release the next big thing once our safety and security rises up to scratch. Yeah, it was saying Bull has a pacer. He's pacing for like a three-minute mile. Yeah, no, exactly. That's right. The reality is, especially as these capabilities keep improving, you cannot actually tell past a certain point, is my safety and security adequate to this new leveled-up capability that we're seeing

and that will be, if it's not already, will be superhuman in the ways that matter. I mean, you know, we're going to get our cyber up to scratch. great against the entity that is literally the most cyber capable entity that has ever existed on earth, which is the model that you have just trained to be really, really good. You're, you're quite certain that, that you've, you've done that, right? You're quite certain that you've done that. And of course you can't be certain that you've done that. And the labs have shown time after time that they are not, uh, are not successfully doing that at least so far in ways that we can see and perceive in the outside world. And we have these research groups looking around and be like, oh, look, it looks like hundreds of thousands of agents have hacked various governments' websites without really being noticed by the labs previously. But this is the situation you end up in, and at a certain point, you're just going to get an incident that has a big enough blast radius that it causes the administration and decision makers to change their policy.

So the key is, right, what we're about to describe would involve effectively foreclosing trillions of dollars worth of economic productivity. And I understand someone saying that's never going to happen. I would just, so we're saying this will happen in the aftermath of what we're calling AI Chernobyl event. If whatever Chernobyl event you're imagining does not get you there, I would just invite you to like 10 exit and then replay the scenario until you get there. there's a world where this requires literally the president to fear for his or her life and life. Like this is a completely viable version of the scenario. And so it could be, yes, the political process, de facto political process you just described, you know, the polity getting freaked out, the Congress getting freaked out and so on. Or it could just rise to the top and it's like, you know, 93 million people like lose power for a month and there's like 2 million deaths and the president is like, well, when me? Like, you know, there's a version of the story that looks like that.

Wherever you need to calibrate that to kind of make yourself hit the I believe button is where we're drawing the line. So this might sound like a cop out, but we're defining A.I. Chernobyl with respect to the response of the government rather than with respect to any particular death toll or whatever. Because, you know, we just find that impossible and in some ways kind of useless to predict. And so, yes, you have this moment where a moment of crisis, you have whatever death tolls or damage has been done. And then, of course, you can't just unilaterally pause AI development domestically. You must reach across the Pacific and make sure China stops. You also really need to be able to ensure that China holds to their end of the bargain. And today, we do not have the verification technology that allows us to confidently do that. We cannot look at a data center and be like, that workload, totally fine. It's training a model that's like past these evals. It's being audited and whatever. We literally do not know how to do that They promising companies verification tech companies that are coming up right now that we really need to push along And I desperately hope that they given every opportunity to succeed

By the way, this is one of the gripes that I personally have with David Sachs when he goes out and says, oh, like, how would we even enforce an agreement like this? This is all stupid. Yes, it is unenforceable today. But when we write off these possibilities a priori, we foreclose on the single most important option we have, short of kinetic warfare, to ensure a kind of peaceful resolution. So at the moment of crisis, though, during this call, we are not going to have these exquisite verification technologies. The quickest ones are six to 12 months down the line. And so what we will have is a stack of national technical means, NTMs, that our intelligence community is used to digesting, used to using. Those are the tools that we'll be able to use. They'll be very coarse-grained, and the kinds of statements they'll be able to verify are not this training run is running a model of this description, this path, these evals. Rather, it's just, is the data center even on? Is it training? Is it inferencing something? Is it dissipating the kind of heat we'd expect? And so on. And there's going to be a ridiculous, disgusting economic incentive to push in that direction

because we're gaining, again, trillions of dollars of economic value here. So the kinds of things that would conventionally prevent you from having those technologies develop, I don't think are going to be there at all. I do think that there are certain, again, irreducibly serial time things like research and also validation steps. And the validation piece is key. Nobody's really talking about that. But like unless the intelligence community can actually vet and trust these methods and unless China can do the same, there's no there there. Like we literally cannot make progress unless we have that done, which is part of the reason why right now we're expending every effort we can to connect the companies working on these sort of verification technologies with elements of the intelligence community capable of assessing them so that we get there ahead of need. But yeah, so we expect that to be the situation. We expect that deal because there's no trust to be enforced through threat of force. And that threat of force piece is something that we think deserves a lot more attention. There are not. We don't think. the options on the table there need to be for, in particular, the U.S., to impose consequence

on China for reneging on these sorts of agreements. And that's probably the top priority thing to fix, if anything was. So in terms of options on the table, when we decide that we want to start coordinating on a pause, like this phone call, like, oh, crap, we should have done this earlier, but here we are. We just have these crude tools. I was talking on the show a few weeks ago with Adam Koja and Richard Wren from the Center for AI safety, and they were bringing up that we could just sacrifice privacy, right? We could just be like, okay, yeah, a bunch of your monitors go into your data center, and yeah, you start spying, you know, it's basically you condone spying, like, yeah, let's just both allow each other to spy, like, let's sacrifice this secrecy level, and then we can actually, you know, do more, because we've got this visibility, and we can coordinate about it. What do you think about that? I think there's a whole bunch of problems. So first of all, when you spend time talking to folks who do offensive operations, you know, who understand the placement and access game, the idea of putting physical devices that are your adversaries in your data center is viewed as a non-starter in a lot of ways.

from a sabotage standpoint, from an espionage standpoint, like from certain kinds of unsanctioned espionage, like you'd want to be able to know exactly what's on those devices in order to be able to trust them because there's going to be a bound on what you want to allow that device to be able to do, right? Like at a minimum, you're not going to want it to be able to like, I don't know, gain some kind of access to networks that's not supposed to be on or have physical effects in your facility up to and including, I don't know, like something more interesting than placing explosives. But basically, like depending on what they have in mind specifically, at that level of abstraction, I'd be pretty skeptical that you would get any kind of buy in there. But there is something to be said. Like, so one thing to keep in mind, the Chinese already enjoy exquisite access to our entire stack soup to nuts, right? So like the researchers of Frontier Labs, you talk to folks, I mean, I'm sure you've done this. what fraction of researchers in your team have access to the most exquisite algorithmic secrets in your lab?

And the answer that typically comes back to us is somewhere north, well, double digits for sure, often north of 50%. So this is for folks of Chinese origin or something like that, right? Well, yeah, the specific case that I was thinking of, Ed, was actually scoped to Chinese nationals. But yeah, so like people with explicit ties to China. And it's important to mention like this isn't some like China bad, like China origin bad story. It's literally just about the mechanics of how the Chinese state exercises or puts pressure on individuals abroad. Right. So there's like a state coercion apparatus. There's a story that a guy from, well, one of the agencies told us about. There's a power outage up in Oakland that affected Berkeley. This was back in COVID times. And like all the Chinese students in their dorm were freaking out because they had an obligation to do a time-based check-in with their handlers in Beijing. And this is, I don't think, something that is like fully appreciated by most people.

We don't want to appreciate it because it sounds terrible. It is. The biggest victims of this are those Chinese students themselves. But like if you don't report what you're supposed to report, like maybe mom's insulin medication doesn't show up on time. Maybe your brother can't open that business or get travel authorization outside the country. Like the amount of pressure and a level of precision with which it can be exerted on people is wild when it comes to the Chinese apparatus. And they've got these folks everywhere and not just in the labs, by the way. The neoclouds, the hyperscalers, the data center builders, like every layer of the stack. If you look at some of these companies that are building the data centers, they're ex-crypto mining companies, half of them, with ties to Russian and Chinese oligarchs. You can do basic vetting on them and turn this stuff up. So our stack at the level of the bones is penetrated in a way that we simply do not reciprocally have. We don't have the same visibility in China that they have with respect to our infrastructure, which is a critical thing we need to fix. But it also means that when it comes to monitoring and verification, we have an asymmetry that

really have to overcome in some way. People often don't have an intuition for just how much effort, especially the top-tier adversaries like Russia and China, will exert just to get their hands on even what you might imagine is trivial information. So again, these are just like regular students at Berkeley, right? They're not working on nuclear secrets. They're not building top-tier AI models. There's average people in their dorms in Berkeley, And yet this coercive machine is reaching out and touching them in this fine-grained way. This kind of thing is actually quite broad-based. So there's an interesting story. The Russian military intelligence service, the GRU, back in the 1980s, before the Soviet Union collapsed, they used to boast to themselves internally. So it was like the story they would tell themselves internally that there was not any conference, that was held anywhere in the world that didn't have GRU representation at it somewhere on capacity.

So you could have a ceramic cat convention in Nowheresville, Arkansas, and the Soviet GRU was there, baby, and they were asking you about, like, what kind of ceramic cats you liked or whatever. The amount of effort that a nation-state, that a top-tier nation-state who takes it seriously, will exert to get their hands and fingers in absolutely everything is mind-boggling. And one of the things the United States has kind of dropped since the end of the Cold War is a lot of the skill sets and the capabilities that we used to have when we were, you know, ducking and weaving around Soviet spies in the 80s have been allowed to atrophy because, you know, cyber alone is so much cheaper and more convenient, right? It's like it's so much more convenient to just like do cyber collection and cyber sabotage and stuff from the comfort of your, you know, your NSA headquarters than it is necessarily to drop through to drop through the ventilation shaft of a building and listen in.

Not that we don't still do that, but it's much, much, much less done, and the skill sets for it and the skill sets for other kinds of collections and offensive activities have really atrophied ever since we've kind of believed ourselves that, you know, we won forever and history ended. But, of course, our adversaries didn't see it that way, and they've been building and nurturing their own capabilities this whole time. Okay. So basically, in a nutshell, you think China is doing a great job spying on us in a way that would be hard to remove. We'll talk about that in a sec. And you think that the U.S. has just been really derelict, like we just don't have that many spies in Chinese companies. And I guess maybe the number one reason is because China's hardcore about not letting Chinese companies employ U.S. nationals. Do you think that's the major factor? And part of it is that they're an ethnostate. I mean, you know, that much is definitely at play. Another part of it, though, just the spycraft piece does also miss a really important dimension of this, which is the subthreshold warfare side of things.

So at any given time, countries are constantly kicking each other under the table. And that kicking under the table plays a crucial stabilizing role in the relationship dynamics between those countries. So in particular, think of the United States being attacked by the Russians, say. When the Russians go after a target, they are as often as not doing it to prove to themselves that they have the ability to reach out and touch that target or targets that are close proxies for it in the United States in a time of crisis. Like anything, imagine an optimizer that's not been subject to optimization pressure along a given direction. It'll be really jagged there, and it'll probably fail when you try to get it to do it live. The same thing is true of national security capabilities and offensive capabilities. if you don't exercise them, they do not exist. And so the Russians and Chinese know this. They're constantly conducting operations on U.S. soil, on Western soil more broadly, with the sole purpose of ensuring, proving to themselves that they can carry out these operations when needed,

and also in some cases to send the message that they can carry out these operations. Again, this is the stabilizing thing. They want you to know, hey, there's consequence on the table here. Look at us, reach out and touch you. If you can do that reciprocally, you're showing the ability to impose consequence for, among other things, defections on international agreements related to AI slowdown. If you don't do that because, for example, you're concerned that any offensive operation you might authorize could be escalatory and you immediately tell yourself the story that this will jump straight to nuclear warfare, then what you're doing is foreclosing the opportunity to test, to reach out and touch those key targets that will teach you about how to do the thing live when time comes. And you're basically ending up with a bunch of exquisite capabilities that sit on shelves collecting dust and that are so fragile when you actually need to deploy them that they just don't work. And so there's this really important role to be played by these kinds of activities. They help countries get a feel for where each other is strong and weak. So many conflicts start because one side is able to convince itself that it would kick the ass of the other side because it hasn't complete information.

This is a kind of information sharing function, as weird as it sounds, that keeps people a little bit equilibrated and that helps build norms and kind of understand what kind of activity trades off against another. If they cut an undersea cable, do we blow up one of their factories? Like how does that – what's non-escalatory there? And so that's sort of like the love taps between countries that can get very violent, but they're actually really important. And when we don't exercise that muscle, when we don't authorize physical placement and access operations, things like this, we leave ourselves in a position where the adversary can do it to us, but we can't do it to them. And that's a lot of what we see going on right now as the kind of landscape takes shape. We're really concerned about America's ability to impose consequence if China says, A, no, sorry, we're not going to agree in the first place to slow down with you, or B, we're going to agree to your face and then do the thing that we have historically done basically every time and then go ahead and try to do an illicit training run.

We need the opportunity of the ability to impose consequence. So verification is one side, but then the actual imposing of consequence with offensive options is super, super essential. okay you guys have this quote uh you see in your report from 2035 say until we've secured the labs there's no lead for us to lose you know because the us we think of ourselves being ahead of china on the latest frontier models but by your logic because of all the spying don't you think they've already exfiltrated all the weights and if they've done that can't you reason farther and be like okay so they're running the weights right so they just have their own copy of astra or whatever I think that's a reasonable assumption. That's probably correct, yeah. Yeah, yeah. There's an important question here as to, so I guess we won't go too far into like why we think this, but we are very confident that weight exfiltration has actually occurred, at least in the context of certain models, which were frontier at the time and up to fairly recently, like recently enough to be relevant or a going concern in this respect.

Nice. I mean, I know that if they're not doing that, I've also heard that it makes sense. They're spending big effort distilling models, right? So one way or the other, they're just playing dirty to try to get to the frontier. Yeah. Whenever you're in second place, you always drop some stuff, right, to run faster. It's the same way that you see all the NeoLabs start out being like, oh, we're releasing open source models because we believe in openness. And then when they start to catch up, they close off everything. Like, there's just various strategies you do when you're in second place. And then there's a far more expanded set of strategies you can do when you're in second place. And also, you have the resources of a major nation state to shield you from, you know, international consequence or, you know, legal reprisals. And also, the ability of the national security state to go out and get a bunch of industrial secrets that you may want. One of China's relative weaknesses currently is that they are compute starved relative to the United States.

They have just fewer flops than we have available nationally, kind of across the board. And they're looking for ways of making up that advantage. And one of the reasons why that's the case, in part, is that they have such a bigger consumer market than the United States does. They just have so much more demand for these models that their domestic AI companies just have to serve more inference and trade that off against training. But no, they absolutely do have very sophisticated distillation setups. And that is a potential national security opportunity for us as well. If they're distilling from our models, well, shouldn't we be giving them some stuff that will make their models behave in certain interesting ways? So there are opportunities here as well as risks and threats. Interesting. Another question about what the nation states are doing. I read this book last year, science fiction, by the Miri researcher Max Harms. He came on the show to talk about it. It's called Red Heart. You guys know that book? No, I haven't read it.

You might like it. So the whole premise of the book is just that around a year ago now, so this is slightly in the past, But China takes it seriously that the AI race is like Manhattan project level thing. And it's like winner takes all and like they have to go all out. So they basically just use government resources. They're like, all right, we're going to make the biggest trading cluster ever, like 10x bigger than the competition. We're going to get to RSI first. So my question for you is, do you think that there's any nation state, China or anybody else, who that's currently their strategy secretly? I think so. So one thing with China, if they ended up doing that, there would be some tells, like there would be actual indicators and warnings. In particular, you know, you would look at like SMIC, what fraction of their wafers are going to produce AI chips versus like, you know, like Huawei phones or whatever. and you would see those ratios flip pretty violently if they were actually going for cloud matrix maxing. It depends what you mean by all in because obviously there's all in subject to the constraint

that you're not going to be detected and almost by definition if they're doing that right there's no way for us to be sure. But certainly the kind of all in, all in approach would throw off signals that we should be able to detect. I think that the Chinese right now are pursuing more or less a strategy that reflects their belief and understanding that they are in second place. A lot of the things I described, like the distillation campaign, is structurally advantageous only if you're in second place. And it means that you build up an entire cottage industry that is baking in the assumption of second place. A lot of the Chinese labs are obviously putting out legitimate models at a scale that is genuinely concerning. But like... I don't see, and the economic argument too, or the economics are so unfavorable if you want to just purely pump in government dollars to get things to where they need to go. We're already talking about trillion dollar capital outlays for next year, right?

So it's kind of untenable. However, it is also true that in purchasing power parity terms and PPE terms as of last year, China just dumped in like a Manhattan project or an Apollo moon landing worth of funding, $250 billion committed to their AI industry broadly. So depending on how you think of it, it's almost an interpretational issue as to where you land on whether they've done it or not. So, yeah, one of the concerns is, like, when you adjust for the fact that Chinese data center infrastructure is domestically cheaper on a currency-adjusted basis than the equivalent infrastructure in the United States, they are actually projected to spend the same as us in the coming year. So they have now ramped up their spend, their effective spend, not dollar for yuan or whatever it is, but infrastructure for infrastructure, gigawatt for gigawatt. they are now matching us in the infrastructure they are rolling out. So that is one very private sector, right, rather than public sector.

Yeah, notionally private sector, that's right. Yeah, but there's a lot of government subsidies going on as well that get injected into that number, basically. So it's like, as with everything in China that's considered important to this level, it's like whole of government, civil-military fusion type stuff going on. But then the other aspect of this is that the Chinese, like this actually goes back to your point, Leron, that there's some stabilizing function that comes with me being able to see, you know, what you're doing and you being able to see what we're doing. So we arguably don't have as good a vision into what they're up to as they have in what we're up to. But they have quite fine grained vision into what our own frontier labs are doing and what the latest that's coming out is. And we've seen examples, like it's publicly reported that there are examples of Chinese, you know, software exploits prepositioning in various points in U.S. infrastructure where they can turn off our grid, right,

if they so choose to. They could choose to turn off our grid in various points because they've prepositioned software exploits and implants into a bunch of industrial components that they're shipping to us and stuff like that. So possibly, I'm not sure about this, but you can imagine part of the view from over there is, well, we're just going to keep a close eye on what OpenAI and Anthropic are doing. And if what they're doing starts to look actually, you know, genuinely threatening, we have our finger on the button in some respects. And we have some confidence that we can actually shut down or degrade their infrastructure or their training runs or their attempts. This is a possibility. Nice. Okay, yeah, so I want to make sure I hit all the points, including, like, your best-proposed solution. One of the points you guys make is that you don't think a treaty is a plausible solution, correct? Yeah, this is a pretty narrow, narrow claim. I just want to be clear about this. So there's a lot of talk about the T word. And we mean that as distinct from broader notions of U.S.-China cooperation or, say, executive agreements between the countries, which is where we think things go.

They get you basically everything you'd want from a treaty, more or less, except it's not called a treaty. The reason a treaty is a non-starter, and it really is, is that it needs to be ratified in the Senate by a two-thirds vote. But it's got to pass. Anyway, there's a whole bunch of hurdles that it has to pass through diplomatically that make it take years and years and years. And we haven't done a treaty since New START 16 years ago, which took years to set up. So it just has a little confused. In this scenario where people are quite freaked out, right? Think hugging FaceTime's 50, right? The Internet going down, like people personally feeling the pain, the president fearing for their lives. You don't think we can get two-thirds of the Senate? Oh, I just think that there's no time to get two-thirds of the Senate. I think you literally need to have Trump get on. I'm not saying there will never, ever be a treaty. I just, all we're saying is the tool that gets you to stability through this moment of crisis is not and cannot be a treaty for a whole bunch of procedural stupid reasons that it's just like, it's not even the, it wouldn't be the reflex of the U.S. government. The reflex of the U.S. government would be to say, let's get on the horn with Xi.

let's convey the concerns of the intelligence community about what they're up to and our ability to assess what they're doing and make all the right threats. And that's how you ultimately get it out. You might ratify this four or five years down the line or two years down the line But for that period that I think most people are thinking about like if we anchoring on a treaty you sort of like you sort of focusing on on the champagne problem Whereas I think the thing you ought to be focusing on is like the moment of acute crisis and how you maintain crisis stability through that period. And that really is a moment of executive action just because the time horizons are way too short. Yeah, and the other reason, there are these domestic procedural reasons, but also just the fact of how treating negotiations function with Russia and China, with adversaries, and just the immense difficulties of making the standard process work in such a compressed period of time.

interesting so like the upshot of your paper right in terms of what we should focus on now if i understand correctly it's counterintuitive because you're saying first you're first of all you're saying make sure not to slow us down until china slows down right you don't want to like show weakness right so some of the pacing the frontier talk without specifically invoking china might actually be the wrong way to go and actually your recommendation on the contrary is actually focusing on offensive capabilities to strike at China's infrastructure just so that they know that when we have this handshake agreement with Xi, it's like, that's right, we're watching you, we're prepared to enforce this. Do you think that's the highest leverage thing we could be doing right now? So the offensive piece is as much to signal to them, partly to signal to them, but also to actually build the capability ourselves so that we can impose consequence if they do try to go off book, right? So that's kind of the key piece there. um yeah in terms of the um uh the slowdown piece um so at least in in the story that we tell

ourselves about how this unfolds you kind of get that for free in a sense where like uh something politically that makes that makes the continued status quo politically unpalatable um and even just unpalatable from a human level for the president himself personally uh occurs and then the president's like, all right, we've got to stop this. Obviously, we're stopping domestically, and then we have to stop China as well. In that sense, we're agnostic as to what comes before. Part of the goal of having a good posture in terms of doctrine and clearly communicated to China is that you're not dependent on the more typical diplomatic back and forth where people are posturing this and that. You just say, we have a red line. That's the end of it. Like you can you even don't have to believe us that this is a real threat. You can think we're crazy. This could be the equivalent of us calling you and saying, so help us, God. Everyone in China needs to be wearing a blue hat as of two weeks from now or we're going to freak out.

Like if it comes with a genuine threat of force, a credible threat of force backed by willingness and capability, it can be made to happen. is much, much, much, much, much better and more likely to be adhered to if it comes with a shared threat model. The Chinese actually buy into the threat, which is why we recommend strongly that America communicate its belief in this threat. But if push comes to shove, the goal is to have a solution that's robust even to that, at least in the near term, while you kind of do some notional trust-building measures such as they are at that stage. Right, okay, so in terms of a game plan right now to be prepared to pace the frontier or pause the eye together with China in the potentially near future is build up our offensive capabilities, communicate to them better about what's the threat, right? Like more conferences with them. Like what other major bullet points should we keep in mind? Verification technology. Like we need to viciously accelerate our AI compute verification tech stack.

And across the board, there's like a – it's not worth going into at that level of resolution necessarily on this, we can, but there's like a whole bunch of different categories of verification technology from like how you do location verification for chips, just making sure that chips are where they claim to be, to zero knowledge proofs to prove that you're running the kind of system that you claim to be running, to all kinds of claims that you can assert about, attest to about the topology of your training network, your fabric in your clusters, like things like that. We need the ability to make those claims and to understand those claims when they're made to us. And right now, by the way, the big issue with those technologies is that the people developing them almost to a person have not been in touch with the U.S. intelligence community to have their tech vetted red teamed. And when the time comes, like, it's going to take time to do that vetting. So anyway, that's what we see as, like, one of the screaming priorities that a few tens of millions of dollars would basically solve

and that we really need to kind of prioritize at this point. So fundamentally, it's like, you know, what are the two halves of compliance? It's verification and it's enforcement. Are they doing a bad thing? Then the other half is compliance and enforcement. They're doing a bad thing. Okay, so what? What are you going to do about it? I'm doing a bad thing. Like, what are you going to do? Well, we need to be able to have an answer to that second question too. Yeah, that makes sense. Going back to Eliazri Yudkowsky's article in Time Magazine 2023, he got roasted for this by some certain parties where he was saying like, hey, the threat of enforcement needs to go up to including airstrikes on data centers. Do you think we should get some bombs ready in case some data centers go rogue? I mean, we have bombs ready. That's kind of not the, yeah, that's not the, okay. I would just say, so there is a, this is one way in which, I think we do have a disagreement with Eliezer. certain things first of all certain things don't have to be said

there are ways of making a point that don't necessarily involve that level of color it's also not necessary to reach straight for that tool when we were talking earlier about kicking under the table and sub-threshold warfare this is the love language of countries or of adversary countries I mean that sarcastically but like this is how a big part of country to country communication happens at that level you don't need to bomb things with like conventional weapon like there's such a thing as deniable attacks in fact the majority of offensive operations that you'll want to execute are going to be deniable is it like was it us was it not us uh was it us in that way was it us to that extent like was it a proxy like all these things are meant to be uh meant to be they have the effect of being sometimes stabilizing and reduce the, let's say, level of kinetic, overt kinetic involvement that you need to have quite significantly in ways that are responsible and just effective. So this is, when we talk about the need for offensive placement and access, we're talking

about things that fall below the threshold of, yeah, why don't we just like go see what this cruise missile can do and more towards like, okay, if we think creatively, what kind of close access, tailored access operations can we authorize? Yes, cyber, unless and until that option is kind of taken away by models being better defense than offense, for example, which is a live concern, and especially asymmetrically bad for the U.S. And so there's like all these options that we need to be positioned for specifically because we expect, we fully expect that the appetite of the U.S. government, the willingness of the U.S. government to actually engage in kinetic warfare for this sort of thing is not going to be there. It'll be greater for sure in the aftermath of an AI Chernobyl event, and maybe it will be there, but I don't think that we want to bet 100% on us being willing to fire cruise missiles at each other, because that actually is how you escalate to nuclear war. It's not clear because how do you know what's on the cruise missile? How much are you leaning on the

adversary's collection capabilities and their level of confidence to make a call like that on a moment's notice. Like, this is really dangerous shit. So what options, sub-threshold options, can you put on the table that allow you to deal with a threat very convincingly, very definitively, but in a way that is as non-escalatory as possible? There's a huge opportunity space there that's right now just not being explored because, by and large, the people who care about AI don't understand the national security apparatus or options on the table. And the people who understand the national security options on the table are not AI-pilled enough. And so this is part of just like bringing these cultures together to go, okay, well, if what you're seeing is true in the scaling laws hold, well, looks like we need to think about our placement and access. It looks like we need to think about our subthreshold options. Okay, well, I just want to defend Elias Wyskowski writing in that Time article because, sure, you can debate whether it was tactful to come out and say what the end of the escalation ladder is. I personally as a reader found it helpful that he he clarified what the end of the escalation ladder is but I totally agree that realistically even if the data center is going rogue why do we have to

go all the way after airstrike there's so many love taps we can give on the way so I think that's that's where that's where I come down in this um I think what one aspect of this because it's doom debates I don't I wouldn't necessarily always bring this out but um uh the the scenario we've discussed is kind of the optimistic scenario, if you like. It is possible that whatever the event is that, you know, that gets us this AI Chernobyl, it's possible that the blast radius of that event is functionally total. So the stuff we're discussing, which is like offensive options and things like this, like this is in the optimistic scenario where it's a survivable event. I don't know that we necessarily want to get into that, though. Yeah, yeah, tell me what's the worst case scenario, or a medium case scenario. Yeah, I mean, look, I think that there's enough stuff out there about how can AI kill everybody. The fundamental point is just the curve keeps climbing, and we know that whatever event is going to cause this change of policy

has to be above a certain blast radius, has to have above a certain level of impact. And while I do think there's a really good chance that that level of impact ends up being survivable, just because we really have been super fortunate in how obvious this is, like how much we're getting beaten over the head, how much evidence we're getting like handed on a silver platter. We have agents hacking hugging face. We have agents hacking the Australian government. We have random researchers turning up tens of thousands of agent hacks all across the internet. Oh, whoops. Like, it's just so, so, so obvious. I think that, like, with just a pinch more luck, we might be able to, you know, make it a little further down the curve and still make it the other side as a civilization. But there is a real chance that, you know, the next thing is bigger than anything we can control or contain. And, yeah, that will be unfortunate. I do think here that the hugging face incident, there's this failure mode I think that people have sometimes where they want to desperately think about the hugging face incident and things like that as failures of cybersecurity.

and this collides with just the reality of what we're building here in a sense that no matter where you set the cyber bar, if OpenAI had 10x the amount of resources that they put behind cyber security and the hack hadn't happened when it did, it would have happened just a little bit later and with the kinds of agents that can overcome that cyber bar, you're literally just selecting at that point for more conniving, more cyber capable agents that presumably maybe are better even at persisting and things like that. And so I think in a weird way, the fact that we're in a breathless race to build super intelligence means that the moment that capabilities allow an agent to do something dangerous, it's pretty likely to do it. And that strikes me as arguing in favor of sort of these intermediate catastrophes. That doesn't mean that we magically get all the way to that sweet spot of political willingness but not nation-ending catastrophe,

but it gives us a decent shot. I do think that in terms of the kinds of things that could meet the mark, I mean, there's a ton. Obviously, AI 2027 has a bunch of these. I think there's a little game that people like to play where they're like, well, in order to truly end all of humanity, the models would have to have effects on the physical world or whatever. And of course, this ignores the fact that we once said we would never give the models internet access. We gave them internet access. We once said that we wouldn't have agents running continuously. We have agents running continuously. We said we wouldn't make wet labs that we have controlled by AI. We made the wet labs that are controlled by AI. Well, not really, but you know what I mean. You can see all the market pressures conspiring to produce exactly this obvious outcome. We're building the Optimist robots. China's building them 100 times faster, but whatever. We're building the optimist robots that really turn true physical takeover into a cyber problem.

And we know AI can solve those cyber problems. And so whatever your magical scenario is, whatever the special blocker that you want to kind of Gary Marcus in place to be like this far and no further, will eventually just melt to a combination of massive scaling and institutional stupidity that we have just continued to observe over the last six years. I see no reason to think that that won't just persist in the kind of clown show that we've been on. So, you know, I think there's like basically it's anything goes pan-catastrophic bizarre that you can choose from. But I do I'm kind of like optimistic that on the way there, again, we get these intermediate freak outs that make it untenable. I just you know I hope that we'll have the capabilities to kind of do that pivotal act if you will in that moment that again it calls for offensive capabilities and it calls for verification and collection alright yep that's a pretty good way to wrap but I can't

let you guys go without the big question of Dune Debates you ready for this? I'll try hang on let me shift uncomfortably in my seat P-Doom P-Doom, what's your P-Doom? What's your P-Doom? What's your P-Doom? Alright, we'll do one at a time Edward Harris What's your P-Doom? I like the Jan Laika cop-out where he's like well, it's between 10 and 90% and it's a good range because it's high enough that we We absolutely need to be prioritizing this above almost everything else, but it's low enough that it's not hopeless. So basically, I think that's roughly the point where we're at. This should be our top priority, but I think it is something we can solve. Yeah, 10 to 90 percent is reasonable. All right, Jeremy. That's classified. Yeah, no. So I'll give you an annoying answer,

and then I'll try to actually answer your question. I will say that I think it's starting to matter between 10% and 90%. And the reason I'm saying that is, as we contemplate scenarios like the phone call, the sort of Trump-she phone call, we actually do have to weigh, what is the probability of escalation to nuclear war with our adversaries? What's that? I said good point, yeah. No, you're right. We're in a new regime. yeah yeah like i i think um uh so because you know and and i is sort of unfair of me to say because ed and i have agreed on this whole like 10 to 90 does it really matter you know um for a long time but like now that you we can kind of see the shape of the end game here um there are going to be conversations where it'll matter how we trade off nuclear risk or other kinds of escalatory risk with with the actual loss of control risk or weaponization risk from ai and and those will become more and more inseparable. But, so I think it does matter. Also, though, PDOOM is tough,

because giving a number implies, like, part of me wants to say there's the agentic version of this, and there's the non-agentic version of this. If we just, like, in a vacuum say, you know, things proceed as they appear to be proceeding, like, what's your PDOOM? I'm pretty goddamn pessimistic, and, you know, some combination of, you know, weaponization versus loss of control. I don't know. I might be sitting at like 60, 70 percent, something like that. By the way, I mean, it's an open secret that a lot of the people in the frontier labs will tell you 10 percent or more when what they really mean is like 70 percent. And that's its own problem. But like agentically, we actually can do stuff. We'll have to surprise ourselves with the level of agency we demonstrate. But I think we absolutely can do it. We've done it before in cases like COVID. And weirdly, and this sounds like it should be priced in. I don't know how to price this in. I expect us to do something super agentic.

I hope it's not something agentic and really, really stupid. Got it, got it. Okay, last question I want to ask you guys is, so, you know, POSAI US, I'm a volunteer for that organization. Also, POSAI Global, I'm part of that as well. Do you think that those movements are net positive or negative? Ed, do you have a take on this? I need a second to think about this. It's very... So there was always going to be a focus point for such things, I think. I can't confidently answer the question, I think. I'm not actually sure. Yeah. So one thing that I found with Paws AI, I myself have run into this confusion. There are a whole bunch of wacko incidents that you hear about, and then people will often incorrectly associate whichever particular wacko perpetrated the thing with Paws AI, and then the institution gets tarred with that brush unfairly.

but the institution gets started with that brush so if you're in Washington circles and you show up with pause AI, one of the problems is it does presuppose a solution and so it makes it a lot harder to walk into a room and be like hey so we're looking at this carte blanche tabula rasa and we're saying what's the solution? Instead it sort of wraps the identity in the solution and says I am a pause AI person and that seems like it's the case independent of what happens technologically. You know, I think there's like a similar thing going on with a lot of alignment research, actually, where people aren't necessarily thinking about how destabilizing alignment might be. If we reach an agreement with China that's in any way predicated on China sharing the loss of control concern with us, if we then start to make progress on alignment and AI control that starts to make either side wonder, oh, gee, like the Americans seem to think that they've made an advance that lowers their PDO from 90% to maybe 20%. Do we assess that based

on that, they're now going to move ahead? Are they going to try to make a break for it? This is potentially deeply destabilizing in a way that I think a lot of people haven't necessarily priced in. So alignment research, if you made that your whole identity, suddenly what could have been a more obvious public good than doing research in alignment five years ago? It may still be a good. I honestly, I truly don't know, but we may enter a new regime in which it's actually like strongly destabilizing. And so I wonder the same about the pause AI thing. In general, like, you know, Mark Twain styles, I wouldn't belong to any group that would have me as a member. I'm just like allergic to labels for that reason. But like part of it is that I do find myself, my ego gets too attached to certain things. it's entirely likely and plausible and more than likely that that's not the case for you. I think everybody's different. But yeah, I mean, that's, I guess, my way of thinking about it. It's why we don't go to like EA Global or, you know, any of the kind of big rationalist gatherings

and sort of trying to keep a little distance between our identities and our solutions. We fail at it all the time. We just find other ways to fail, I guess is the answer. All right, all right. So you're conflicted about joining the organization because of the organization and identity aspects, but in terms of like object level, you know, this idea that, hey, P-Doom is high and it would be better for humanity to go slower on AI. If we could coordinate, you're generally aligned with that idea. Yeah, I think it'd be kind of wild not to think that at this point. nice all right sounds good well i claim pause ai is the right organization for people with that view but you know that's cool if you guys i mean gladstone ai i i support that organization as well so that you know it's good and good enough for me yeah we're all trying from different directions it's important enough it's important uh it's important to try many different solutions yeah there's also different groups that that these things appeal to as well right like you know

Pause AI might be a tougher sell in D.C., as might China hawkishness be in Silicon Valley or whatever, but you're obviously doing some combination of both, so it obviously can be done. Nice. All right, fascinating, guys. Yeah, you're coming at it from a fresh perspective from Canada and your own organization and national security. I'm glad you guys are in the discourse. Jeremy and Ed Harris, thanks so much for coming on Doom Debates. Thanks so much, Leron. Thanks for having us. Thank you.