サイバーセキュリティ専門家 Zack Korman は「人類は生き延びる」と確信している

I think we could easily, trivially defeat a country of Einstein's in a data center. I'm picking humanity. I feel like I pick that thing every time. But let's say you get a whole billion minds, and let's even give them a 10x feed up. I claim they can take over the world. Like, come on. Now, bad things can happen without it being extinction, and then we get an opportunity to stop things, right? You're imagining somebody who has this important big red button at all these different AI companies, And I'm just saying, like, I don't really see this person with this button anywhere. No, but it could be, like, someone in a different company tells OpenAI, and they go, oh, no, we can't stop it anymore. And then, like, yeah, Donald Trump rolls in there and just drone strikes them. Like, the history of humanity is, like, full of stories we can tell about how it could have gone poorly, and it never did. I'm just telling you, like, expect major qualitative shifts. Okay, I don't think it's that much of a stretch that one of the qualitative shifts will finally impact our quality of life. Of course, I believe that. but then it will impact our quality of life and we can't reverse it and we'll have to just keep on following that curve down to the very bottom

and there's nothing we can do about it. That's what you have to sell me on. Welcome to Doom Debates. My guest today is Zach Korman, a security engineer, startup CEO, YouTuber, and a rising anti-Doomer voice on AI Twitter. He runs Embroidery, a startup that uses AI to monitor AI agents. Zach says that back in 2016, he used to be a doomer, but a decade later after OpenAI's agents hacked Hugging Face, he thinks that the doomers aren't warning of the actual risk, but on the contrary, the doomers are the risk. His most recent video titled, AI Doomers Are Furious, concluded with this line, I want the public to see how dangerous your ideology is. Well, today, Zach is here with me, and I'm hoping we can get to the crux of our disagreements. He told me that I'm his second favorite Doomer. Let's see if I can build on that goodwill to get him back on the Doom train.

Zach Corman, welcome to Doom Debates. Thank you so much for having me. Okay, so I'm your number two favorite Doomer. Who's your number one? There's this guy named Aaron Scheer or something. Scheer, I don't know. He works at Mary. I kind of, like, ridicule him online pretty regularly, and he just goes like, okay, and then just keeps arguing with me. So I kind of get along with him. I've even had a video call with him. I think he's my favorite Doomer, but you're my second favorite. Don't worry. All right, I'll take it. I mean, look, Miri has absolutely top-shelf Doomers. Oh, my God. I mean, obviously Yudkowski. You know, I'm surprised you didn't say Yudkowski is number one. He's my number one. And then, you know, Nate Soros is always dropping insight bombs as well. Like, I'm still binging Nate Soros interviews. So, I mean, it's stiff competition. I'm okay being pretty far down the list, honestly. Okay, but what about your number three? Oh, I don't think I have a third favorite Doomer. Well, there's, yeah, I mean, like, does Nick Bostrom count? Because he can be my third favorite.

Yeah, Bostrom then, right? Okay. Yeah, yeah, Bostrom from the 2014 book Super Intelligence. But then, you know, Bostrom is actually one of my least favorite Doomers because he's kind of turned around and he's, like, so contrary. and he's like, well, you don't want to get too doomer. There's like a right level of doomer. And so I see my role now as like making sure people don't doom too much. And then I'm like, oh, come on, people aren't dooming enough. Like, I don't think he's calibrated. So anyway, I've got a little bit of a beef with Professor Bostrom here, but I also respect him. So anyway, all right, that's enough about the doomer rankings, okay? So getting back to you. Well, this is also why I like him. Yeah. Okay, yeah, fair enough. All right, so getting back to the story of Zack, okay? So when you first were in high school and college, what area of subject were you studying at the time? So when I went to university, I went to the University of Edinburgh and I studied law. So I have a law degree from there, graduated first in my class there, was like super into like tax law and stuff. And since I'm like, I can add numbers, I'm therefore the best, you know, mathematician in law of all time.

So I can add and divide and subtract. So I did quite well in like tax law and stuff. And then I went to Oxford to do my master's in law and finance. And this is when I started learning about your world of AI kill everyone. Right. Okay. And then you started your career. What was your first job out of college? Yeah. So when I was doing my master's, I also taught myself how to program because I got into this argument with a professor, started, had to like, yeah. So I ended up writing some Python code to scrape the SEC website. So actually, when I left university, I started this company with my friend for like six months. It just like didn't work. And then I went and started as a developer. I moved to Norway because I was dating a Norwegian person, started as a developer. And yeah, I ended up working there for a number of years and became in charge of their whole tech and product department. Interesting. So did you kind of abandon the whole law career track? Yeah, I kind of view it as like you never, you know, when I was at this media company,

we didn't, even though it's a thousand person company, the Norwegian division is about 600 people. They didn't actually have at the time like a in-house lawyer, which weirdly meant I became their in-house lawyer very quickly, which is like very unfortunate. It's like right when GDPR was becoming a thing. And but yeah, I definitely don't practice law. I do think it's like, you know, people often ask, like, what is the minimum of like, you know, required for any given job? But for now, I'm very much a cybersecurity person. Like, what do you need to know? And I'm like, it depends a lot on like what background you come from. I can't imagine doing my job without knowing law. But obviously, it is possible to do cybersecurity without knowing law. So like, that's quite informative about what is really required to do each role. So when you joined this company out of college, what was your job description supposed to be? Yeah, I was supposed to be a data engineer slash developer. And I did do that job. I ended up building their personalization automation system, which, because it worked quite well,

I acquired a great deal of political power within this company. And eventually, I had a department of 35 people to run after a couple of years. Yeah, so I was doing just purely technical roles. when I, like, officially. But unofficially, I'd also have to join meetings as, like, their go-to legal person. Gotcha. Okay, yeah, so the trajectory of your career has been heading toward where you are now, which is kind of like, you know, a founder, a security entrepreneur, and maybe the kind of person who you'd also find as, like, a security CTO in a large organization. Is that a good description? Yes, so once I left that company, I then became the CTO of a cybersecurity company here in Norway. So I led that for, I guess, another four years doing like AI insider threat detection, automated phishing simulation attacks. So I spent about four years specifically in a cybersecurity role. Also, at both companies, I ultimately had responsibility for cybersecurity.

So yeah, and then now I have left that cybersecurity role. And now I am one of the founders of a cybersecurity startup. Cool. All right. What about the title CISO, right? Chief Information security officer? Is that kind of in your wheelhouse? No, I would never take that role because the job of a CISO is to get fired. And yeah, it's just not what I would want to be doing unless someone wants to pay me like an enormous amount of money. I think very much I more consider myself like a tech and product leader within cybersecurity than a corporate CISO. Gotcha. Gotcha. Your previous guy, so right now you're at embroidery and you previously had a company called pistachio where you ran ai phishing simulations correct yeah and insider threat detection oh yeah okay would you say that you're in a great space right now yeah like a very lucky space right like obviously cyber security is very heated up right like it's very there's a lot

happening in cyber security now but this also is what got me into cyber security about two years ago. So I was at this cybersecurity company, but if you actually watched a podcast to me like two plus years ago, I would like directly say like, oh, I'm not a cybersecurity person. I'm a tech person because I kind of separate myself from that group. But then when AI started, like I realized AI has this huge impact on cyber. It makes cyber like a lot more interesting. And so I started really engaging online so I could learn a lot about these different areas of cybersecurity. So now I'm like just a full-blown cybersecurity nerd. So this brings us to some arguments you've made, right? Some positions you've taken on YouTube and Twitter that were interesting to me because I feel like there's a lot of fodder for debate. So my plan for this conversation is basically like, well, we'll kind of go one by one, right? We'll review some claims you've made. We'll review some claims I've made. And then we'll see where the crux is, where the clash is. Sound good? Yeah, sounds great. All right.

And also relevant to your story is you said that you used to be very much like a Doomer in 2016. So I guess we should first start with your origin story, right? Like what made you a Doomer in 2016? What did you used to find convincing? Well, I was a law student, right? And so I don't know how computers work in 2016 or very tentatively do. And so, of course, I read like Super Intelligence by Nick Bostrom. I was at Oxford at the time, and so I'm around the groups of people talking about this topic. I mean, I still find, like, the arguments inside of superintelligence to be interesting and, like, useful. I don't take, like, some deep opposition to it. And I even would have gone to talks by Bostrom at the time and then spent the next couple of years just, like, you know, reading random stuff on Less Wrong and listening to podcasts and stuff about it. You know, I think that like there is a lot of merit to the like broad high level concern. There are pieces I diverge from, though, when it comes to now that I've like seen a little

bit more of how the world works. And I think this is actually quite a common experience in like cybersecurity. If you start in cybersecurity, your first thing is you go like, oh, my God, like it's over, like everyone's going to get hacked. Everything's going to go poorly. And then over time, you realize that there are actually a lot of like mechanisms that imperfectly work to make things not blow up, even if they don't always go perfectly. And so it's sort of just an experience of, like, realizing that there's a lot more nuance and complication to the world than what maybe is expressed in many of the Doomer models. Okay, I see what you're saying. You read Bostrom's arguments, and they convince you that everything is doomed, but then you also notice that learning cybersecurity made a convincing argument why everything is hopeless on the Internet, right? Like, of course, every computer is going to have rampant viruses and everything is going to be unusable. And then you started actually looking at the state of cybersecurity and like, oh, somehow, you know, defense is holding its own against offense. And you kind of generalize, right, by analogy. You're like, hey, maybe in the real world, humanity will defend its existence, right?

Maybe the alarm, it doesn't balance out to be like a full takeover, correct? Right. I think broadly speaking. And also that even when you start to look at the model that many doomers present about these risks, they, in many cases, simplify the cybersecurity aspects of them down to very like it becomes not that complete. So the cybersecurity story tells a little bit more information. Cool. All right. Well, that's that's a rich vein for us to dive into. Okay, well, before we dive in, let's also get this out of the way. You ready for the most important question of Doom Debates? Yep. P-Doom, P-Doom, what's your P-Doom? What's your P-Doom? What's your P-Doom? Zach Corman, what's your P-Doom? If we're talking P-Doom of being within some time frame, so let's say, you know, AI Doom by 2050,

sub 1%. Wow. Wow, that's so optimistic. 99 to 1. Outside of the sane range, as you put it. Not within the sane range of your definition. Sub 1%. I don't think it's sane to think that you're that confident that AI is not going to take over, even though its intelligence is going to be higher than us. That doesn't strike me as sane to be that confident. Especially if we're talking, so just to put a little bit more detail in it, especially if we're talking AI takeover, As opposed to like an AI accident where like, so like in the world where I'm talking like doom also means that like the AI survives on and has like a purpose and a goal and not just like someone shot off a missile because it goes, oops, I didn't mean to send that missile. And then like everyone starts just shooting missiles and then like game over. I think that's a little bit of a different scenario. Okay, so there's this question of like attribution, right? It's like if there's nuclear war and AI was just like the spark of the tinderbox,

but you still kind of attribute most of the doom to just humans fighting humans, maybe that doesn't have to count. I mean, I'm happy to say that the attribution can be kind of generous where like, okay, if it really does seem like 90% humans, I won't count that as AI doom, sure. I guess it's maybe the way that I think about it is sort of like if the AI doesn't survive this war either, then like the whole argument of like it's so intelligent that it knows how to kill us all is like it doesn't really make sense because it's also gone. So I think, like, in the debate of, like, doom, as we're considering, like, AI doom, it only really makes sense to think about it like it has to have survived. Otherwise, we're just saying it's like an accident, right? That's a really great kind of criteria where it's like, hey, you know, genuine AI doom does involve colonizing the universe. Like, if the AI, like, destroys the Earth and doesn't even manage to colonize the observable universe, That is a narrow slice of probability space in my view because I don't think colonizing the universe is that hard

with respect to very high intelligence. At the end of the day, you really just need a few megabytes worth of information to put into the galactic probe. That's what I think you need to bootstrap. You can colonize the universe. Just like that show, Pluribus, right? It's like the signal. I don't know how many megabytes it is. I don't know if they said on the show, but it is the amount that you can transmit realistically. I thought that part of the show was actually very realistic, that there was a pretty compact signal. Any thoughts on that? I don't know. I haven't seen the show, so I don't know. But I guess I agree in general at the principle of, like, maybe it's not that hard. I just mean that, like, when I define my P-Doom, it has to feel on purpose, right? Okay, so less than 1% P-Doom. If it's the kind of Doom where the AI wipes humans off the Earth and proceeds to take over the universe, less than 1% by 2050. 99 or more against one odds, correct? Yeah. Wow. All right, man. You have laid down the position. So I guess, you know, my goal would be to try to push you into the sane zone where if

you walk away saying 5% to 10% chance, then I'll be happy. I don't feel like I have to push you more than that. Really quick, I would just say if you pushed me to 5% to 10%, I think, like, ethically, we're probably aligned here. 5% to 10% is just an unacceptable range to be tolerating. So I want to be clear, I'm not like a crazy, you know, let's roll the dice at 5%. If you manage to push me at 5%, then I'm like, okay, we need to stop AI. That's great to hear because there are a lot of people who criticize the downstream implications of what AI doomers believe. They're like, why do you guys want to stop AI? Doesn't that mean you're against technology? But I think you would push back against those people and be like, no, that's just a simple downstream implication of thinking that PDoom is high, right? if you think BDOOM is high, then you do sacrifice technology in order to stop DOOM, correct? Yeah, like, 5% is just an unbelievably high risk. You wouldn't just walk around by thinking, like, let's see what this does tomorrow.

Yeah, now, I personally do think at the 5% to 10% level, there is a lot of arguments of, like, damn the torpedoes, because if we don't do it, there's all these other risks. There's so many threats to the fabric of humanity, and we're going to need AI to defend against AI. I'm actually sympathetic to damn the torpedoes type of arguments all the way up to 10%. Maybe you can get me up to, yeah, 20% PDOOM, but just rush through anyway. I'm sympathetic in those ranges, but my actual PDOOM is more like 50%. So at that point, this idea that you're going to YOLO, I just don't see it going right with that kind of attitude. I think that's a real long shot to survive that way. But that's nice that you're admitting you shouldn't criticize us for the implications of our beliefs. You should criticize us for the beliefs itself. I'm going to bring that back to you, though, because I do think that you make some arguments against, like, doomerism, right? I do think you say, hey, if you believe that we're doomed, that's going to have negative implications, and doomers are harmful. Don't you kind of have those types of arguments? Yeah, but I think being wrong is harmful, too, right? So that's kind of the core of it.

I believe that doomers are wrong about their p-doom. and so as a consequence of this while they're genuine and the consequences of their wrong decision about what the P-Doom would be might be like rational and normal and I would agree with them but if you're wrong about your P-Doom then you can also be wrong about a lot of other things right so I mean if you believed just thinking like any genocidal maniac like if they believed that this group of you know people is going to is actually harming the whole world so we need to kill them all like yeah if it were true that that group of people was like a huge detriment to society, maybe this makes sense. But it's not true. And therefore, it's harmful. And that's what I viewed P. Doom or like Doomer's as is like, because they've done the calculus wrong, they are now recommending a course of action that is harmful. Right. You know, this also reminds me of Sam Harris, where he looks at fundamentalist religion. It does seem like Muslims are doing an outside share of modern terrorism compared to other fundamentalist religions, but I know, you know, you have the Christian crusades,

there's certainly other instances of religiously motivated, you know, attacks. So Sam Harris is saying, look, if somebody is a fundamentalist Islam, and that's making them commit a terror attack, like the 9-11 hijackers, in that situation, can you really blame them for the terror attack itself? Because that is a logical consequence of their fundamentalist belief. Like, they really think that Allah thinks that this is how you should fight in order to, you know, have the best soul right in heaven or whatever right if that's really what they think all is saying i don't want to criticize the the logic that they're doing you really have to go to the source and criticize why they became into the fundamentalist islam in the first place right i feel like that is where they went awry and and there's actually something good to be said about like that's great that they're so committed to the cause i just wish they didn't join the wrong cause you know yeah same it's the exact same take here and i think i have a video where i said that we would find a huge amount of common ground between myself and doomers, but because we disagree on tiny points that change this analysis entirely, we end up going in two very different directions, just like,

you know, jihadists, right? Right, okay, so in that sense, it seems like there's a symmetry where for every time that you can criticize somebody for believing in doom and therefore dragging down tech, right, and having all these negative effects, and therefore making people worry and spiking their blood pressure, right? all of these bad effects of people who really truly believe that PDOOM is high, I think you can find symmetrical arguments of people who think PDOOM is low, but look at all of these reckless effects. You think PDOOM is low, therefore you're letting AI be built faster, which is going to leave us a tiny fraction of the light cone when we could have had a large fraction of the light cone. So I think it is quite symmetrical to argue from downstream implications of somebody's voice. So I guess maybe we can at least agree that the proper way to have this argument is just to go argue about what PDOOM really should be and not use these meta arguments about like, well, if you believe it's this, then this. Correct. I think it's fundamentally like, what do we think the correct analysis of whether this will happen and what we should do about it?

That's like the whole debate. I think everything higher level than that is like, no, I don't have a fundamental disagreement, ethical disagreement, like with how that analysis plays out from there. once we conclude on the number, I think we have broadly similar conclusions on everything else. Okay, great. One last question for you. I ask this a lot. The famous Center for AI Safety statement that says mitigating the risk of extinction from AI should be a global priority alongside other societal scale risks such as pandemics and nuclear war. Do you agree with that statement given that you think the existential risk for the extinction risk from AI is quite low? Do you still agree with the statement? Yes. So it's also important to clarify that one reason I and one of my criticisms of like even just the talk of like P. Doom as like a way of like discussing it is that it has this implication that somehow we don't have a huge role to play in making the determinations between here and like our extinction. My P. Doom is very low because I think we're just making we will make all of the right.

Not all of the right. We'll make a ton of bad decisions, too. but like we have influence we're not powerless over what happens between these two points so if we were to say let's just not worry about extinction at all or let's not worry about any of this then you've just raised rp doom right i think that being concerned about it is like a fundamental aspect of why i believe it's a very low number that's interesting so if there was not much attention getting paid? Like, imagine there was just, like, no AI safety organizations, right? No UDKAS, you know, if anyone builds it, everyone dies book. In that case, what would your B2M be? Well, I don't think they're the ones that are saving us. Like, I think that basically if we didn't have all of the existing security mechanisms and security mechanisms that we believe we need to build in to protect against AI, we would have a bigger problem. I don't think these necessarily come from the safety orgs, although I will say, obviously, shining a light on them has been helpful for also being able to make the discussions about some of the security mechanisms that are needed.

Do you think about like, you know, if you look at like AI 2027, like we can talk about very specific failures that occur inside of that story. And this allows us to then say like, yeah, let's just not make that failure. Right. And so that's like somewhat helpful. So I wouldn't say that they're like not contributing, but I'm not going to like say they saved the world. Just, you know, like when it doesn't happen, I'm not going to say like, thank you, Yad. All right, all right. Well, let's get into mainline scenarios. We'll talk about the good, the bad, and the likely. So what is your mainline good scenario, which is just your mainline scenario, right? You don't think humanity is going to go extinct. So just do a Monte Carlo simulation of the future. Just tell me one story that's as plausible as any story of what's going to happen over the next couple decades. Yeah. So one also thing is that, like, I, there's this thing in the kind of rationalist community where everyone wants to, like, just, like, predictions are, like, super important to everything. And I think to some extent we focus on narrow predictions, whereas it's more like I like to make predictions about, like,

what are the things that I believe, like, are very likely to be true and then build your world around, like, that. So in my world, I don't really know whether in 10 years we are going to have something fast approaching, like super, super intelligence, and everything is amazing. I think that that is like a plausible path. But it's also plausible to me that in 10 years we have like stopped AI development because we approached this super intelligence and we saw that there's like this really like kind of scary aspect to it, and we just stopped, right? I think both of these are like future state of the world that could be like true. I wouldn't like scream about it. I wouldn't go like, that could be implausible. I think in one of your previous discussions, you mentioned that if like a time traveler came back and told you about what had happened, which cases would surprise you. Like neither of those would surprise me. What would surprise me is if I found out we were either all dead, right? Like if everyone's just like dead, that would just be stunning to me. That, yeah, that's the case that I'm just like,

well, that's not going to happen. Okay, so just to flesh it out a little bit there, right? Like, you still kind of have to tell a story, I think. So, like, AI keeps getting better and better. You know, basically, we build it into a ton of our systems and stuff like that. Probably there will be some very serious security incidents that happen along the way that would result in us going, like, okay, we need to, like, regulate, like, critical infrastructure and how we roll out AI to those systems. We need to stop that. probably somewhere along the way we're going to have some like, you know, revolt against the data centers game, whatever. But overall, we will continue developing AI, but it will stay within our control. We will continue to be able to use it. And like, it could be entirely plausible to me that 10 years from now, we have an AI that can sit in the data center and churn out really amazing scientific progress we've never thought about, but it's like, like not able to just escape and kill everyone even if it wanted to okay i suspect the crux of our disagreement probably the single

most common crux has to do with believing in asi right i feel like there's uh tv moshevitz has a good distinction of there's people who believe in agi and there's people who believe in asi and there's also people who believe in neither it seems like you're somebody who believes in agi right as opposed to like a melanie mitchell or i don't know emily bender right like there's people or Ed Zitron, right? There's people who are like, no, this is all hype. This is a bubble. I don't think you're that, right? I think you believe in AI that's kind of like a human white-collar worker in a data center, correct? Well, I believe I could be much better than that. I mean, like, just said, maybe it's a very stupid way of referring to it, but you can just think, like, I don't have some sort of, like, biological superiority, the idea that somehow, like, you know, human brain mass has something special such that you can't surpass it. Like, if you just ran a human brain, I mean, it is just ultimately a physical process. If you just turned that into wires, it would be faster, right? I think there's a lot of ways to just have intuition about the theoretical possibility of superintelligence in that sense.

I don't have any problem with the idea of, like, I think you even referred to it as, like, kind of equivalent to a couple thousand IQ points, which I don't know if that's, like, a really relevant, like, how you can view it that way, but, like, I don't have a problem. I don't hear that and go, like, you insane person. That could never happen. Right. So, like I said, I think this is the crux, because if I'm imagining 50 years from now or 20 years, or I happen to believe that we're going to hit these inflection points, which will make it 10 years or less, which I don't think is an uncommon view, but when I just think, let's call it decades ahead, okay? I mean, I'm thinking of something that really leaves the human brain in the dust, right? In these important qualitative ways that I don't even think we fully have tasted yet. Are you following me there at all? Yeah, but I don't have a problem with that. Sure. That seems like I wouldn't say it's likely, but it's very plausible. Why isn't it likely? Well, I don't really care if it's likely even for that matter.

Like I said at the start, I don't try to build my world. So, like, what matters is, like, what do I believe is, like, very probable? And then you build backwards from that. And so to talk about doom, I kind of go, like, okay, let's imagine that we have this ASI. It is a much better brain than our brain. Does that mean we are all going to die? And if the answer to that is no, I don't have to go back and go, like, but when do I think it will happen? Because, like, I've already answered the question. Like, do I believe in that risk? Okay, well, one of your videos, so like I said, I watched your videos, I took note of some of your claims. I think you have a video where you talk about how Doomers kind of like make up magic powers for the AI as they go, right? So when we're talking about it, so like maybe I'm guilty as charged because I do think that the level of power that the AI can have will strike many people as magical. Although I also think that the powers that we have today in 2026 would very much strike people in the year 5 AD

as just as magical. Like, the amount of magic we're able to do today, I don't even think you can turn that dial more than one or two more notches beyond what we've turned it in terms of subjective magicalness, right? I mean, imagine putting a VR headset on an ancient Greek. It's just like, how do you get more magical than that? Like, yeah, you can get a little more magical than that, but we're getting close to the limit here. But yeah, I do think there's another notch left. And I think these future AIs, like, I'll give you an example, okay? Tell me if you think this is just a cheap shot of me just inventing magic on the spot or like a legitimate claim, okay? I think that they can basically have a blueprint for which atoms they want to put where and pretty much achieve that. You know what I mean? Like the universe is kind of a Lego set. Sure, there's laws of physics. There's laws of physics governing a Lego set too. And yet you can pretty much just manhandle those Legos into pretty much any configuration, right? Or close, right? There's certainly a lot of Lego sets doing a lot of flexible things. So what do you think about that description? Yeah. So like if you give it the tools and ability to do that, like, because obviously this is like, how are we getting to this AI, right?

Like if you don't give it the right tools to do it, you also have to believe that it found a way to acquire those tools itself, which is like a jump that I don't know if like is necessarily true. So you can imagine, like, okay, just think very basically. Like, today, LLMs, like, you know, they actually do not, they're not able to actually do computational things on this device. They actually need that harness layer to translate them to tool calls. If you don't give them the tool calls, they're still just busy spitting out text. That doesn't necessarily mean, like, what if that just stayed true? Okay, so 20 years from now, you find it hard to imagine? Yeah, I don't find it hard to imagine. I just mean you have to explain a little bit about how we get to this state where suddenly the AI can do that. Because I think you have to kind of ask yourself, like, what if the AI can become super intelligent, but that super intelligence doesn't grant you the power to, for example, manipulate the GPU in a way that allows you to do all of these things?

What if that's true? That very plausibly could be true, right? Okay, so my perspective is one of convergence, right? I'm just like, listen, intelligence is convergent, and we've been seeing more and more intelligence. And you know what we're going to see next? More intelligence. And not only that, but it's going to be a positive feedback loop, right? Because the intelligence itself will help make the more intelligence, right? So I'm very much seeing like an arrow here, right? Like a convergent arrow. and I would be shocked if we ever without, you know, already fighting nuclear war, right, and going, setting us back, without, like, starting to destroy the tech tree, right, as long as things are progressing in some sense. To me, the progress is clearly going to go toward more intelligence, and when you raise an objection like, ah, but what if it's more intelligence but without them discovering tools? I'm like, yeah, discovering tools is something you can do with intelligence, no? No, not necessarily. I mean, like, look, I can drop you on, I could put you on, like, a deserted island and all of that, and like you have limitations that are still around you. Like just imagine, take like an LLM.

Like could you make an, like unless there was like an architectural change or something that's like truly not understood about like the scope of computing, if we got there via just pure LLMs, like I don't think there's a cap on how smart an LLM could be, but that also doesn't mean that there's a cap on like the LLM will somehow be able to start manipulating things like outside of its world, right, on its own. So I don't necessarily think... If you want to be specific, maybe give us like one example, right? Just name one thing that you think is not going to be something that an AI can use as a tool of any kind. Well, an AI can't use... So if you take away its harness today, an LLM cannot literally use any tool. Like it literally just can't, right? I mean, it can build itself a harness, right? If you give me the most basic stripped down version of Foxhound with one tool. Yeah, but one tool, right? We're talking like it is literally giving you just text back. It can't. Right. So so you have to have this jump.

And this is actually one thing that I've talked about when people ask me, like, what are the things that you would say would become scary and potentially require make you become like, OK, we should stop now. One of the ones that depending on circumstances might be relevant would be like if, you know, a zero tool LLM could somehow build tools. I mean, then I go like, OK, we need to really like that's kind of unexpected thing that could cause us to go like we need to decide what to do about this and potentially stop. I think that though like it it you kind of telling me that like it impossible for it to get better get smarter without it also finding that escape path And I don believe that true Like there it entirely possible to build a smarter and smarter LLM that doesn ever just find a way to manipulate whatever is happening at, like, the GPU space to be able to take tool calls. Like, that's not necessarily impossible. This is getting interesting. So you, as a cybersecurity expert, is that a good description of yourself, a cybersecurity expert?

I'll take it. Okay, I'm finding this surprising because you're basically giving me a security model here, right? The security model of, like, LLM with zero tools, right? You see that as, like, a pretty robust security model, correct? So, like, I don't, actually. I view it as, like, a very interesting test, right? So in the event that, like, LLM with zero tools becomes, like, we would find that it could break out, I would say it would surprise me, but it wouldn't, like, I wouldn't, like, go and, like, my world isn't blown. I haven't, like, you know, but it would lead me to changing my views on what we should do next, basically. Okay, so why don't I start with this? LLM with one tool. Can I convince you that an LLM with one tool can bootstrap to having lots of tools? Is that trivial? Yeah, especially when we're talking, like, an ASI with one tool. Like, I don't see that's kind of in your, like, obviously. Yeah, because the tool can literally, okay, write the next tool, right? Like, that's because they can modify its code.

Yeah, it's so trivial, okay? But you still think zero to one is non-trivial, correct? Well, it would seem that way, right? Like today, it absolutely, I don't think anyone would argue it, it's like obvious that that would become, that that would happen, right? If someone is standing here saying like, well, the thing you're missing is that that will happen, I don't actually think that's the claim most doomers are making even. I mean, when I talk to other doomers and I say this kind of, They go like, yeah, but that's sort of like saying you'll only be afraid of like an alien species when it can do like telepathy. Right. And that's true. Like it's sort of like a little bit of a high bar. In fact, I've set a higher bar there than some of my other red lines on where we would need to like reevaluate. I actually think the highest standard to clear would be a zero tool LLM finding a way to manipulate processes in a way that can create tools. That would be really, really crazy, depending on what we see.

But yeah, that would be interesting. Okay. So let's do a hypothetical scenario, because I think that you do grant the premise that chatbots, right? Chatbots are going to get increasingly intelligent at answering questions, right? Are you already happy to accept that premise? Yeah, obviously. Am I misunderstanding the question? Like, do I believe LLMs will get better? Like, yeah. So, from my perspective right now is I think you're presenting, like, a very, very weak security model. So my goal is to really try to convince you that the security model is not going to buy you any security. But just to make sure I understand the security model, you're accepting the premise, yeah, that the chat's going to get more intelligent, like, more generally intelligent. I think you're, like, kind of asking me in a different way. You're saying, like, okay, it's a weak security model. The question becomes like, okay, let's imagine, I'm kind of wondering what's the security model I've presented. What I've presented is just like, here's a piece of information that if true, would require me to think like, okay, maybe we should look at changing this.

I don't think that modifies my current position, which is just like, why are we stopping AI right now? It's not done that. A lot of my position hinges on the idea that we can, doomers are wrong because we have a lot of opportunity to reevaluate as we go. And I think that's like one of them. I don't see it as like a security model. It's not like I'm betting humanity on that not being possible. Fair enough. I mean, I do think you'll have other points that you can retreat to after this. But where I'm coming from is, remember, I still think one of the biggest cruxes between me and you is probably that I believe in ASI. And I don't think you're fully somebody who even believes in ASI. I don't think that you have a mental model of what I think is plausible in 20 years. or maybe you have it and you're quick to dismiss it, right? Like, you know, I'm happy to use the term godlike in terms of, like, I do think that it'll be, like, indistinguishable. You know, sufficiently advanced technology is indistinguishable from magic. Like, I think that will be literally true, right, to the AI. Like, from our perspective, it'll look like it puts the virtual reality goggles over us,

and they're really good virtual reality goggles, and there's just no way that we can be like, well, you did this this way, and this is possible, and this is not possible. Like, from our perspective, it is just the puppet master. That's what I think is coming soon. I think what you might be missing about my view on this is that like I don't think we should build that machine god so to speak right there's like puppet master that can manipulate any atom and stuff like that but I think we will know as we make more progress whether that is really coming and I think we will be able to stop that before that happens I don't think we should build this like single can modify any atom in the universe at will if it so chooses but I think that there's no real evidence that we're like currently close to that path necessarily. So I don't think we need to stop yet in order to prevent doom. Well, one reason I wanted to disabuse you of this zero tool security model is because I wish it were true. Like it's actually, it's the kind of security model where if it were true, it would solve a big part of the problem.

Like if I thought that there was any security model that we knew of that we could build toward, I would be thrilled because I'd be like, great, let's build toward that. right so like is your is your claim that we should just build the most powerful possible ai with zero tools like do you think that'd be a good outcome i think we would need to find out if it can do these things right i guess my claim is that somewhere along one of the big problems i have with the doomer argument is that it sort of acts as if the moment when we find out the scary thing is also the moment when we all die right that these happen in like simultaneous moments right so that if we found out that, like, oh, it can escape... I mean, I'm willing to give it a little time. How much time are we talking now? So I don't have, like, a... I always think, you know, there's the famous Eliazri-Yodkowski prediction that a foom, it all may go down in minutes, right, like in this crazy small amount of time. If you consider the foom to have already started, then he's wrong. But, of course, you can argue over when the foom has really started. There's a good case to make that he hasn't started yet if you go, you know, look into the fine print of what he said before.

I don't have a strong opinion because at the end of the day, it doesn't really matter. To your question of like, okay, how long do we have? Let's just say 10 years, okay? I am convinced that if you give a uniformly super intelligent AI, like an AI that there's just no question and literally every task, it's better than the best human. If that's the starting point, I am happy to say that there's a more than 50% chance that whatever it wants to do with humanity, 10 years is going to be enough time to do it. Yeah, okay, but 10 years is an extraordinarily long amount of time. for humanity to be able to react and change and all of that, right? So I think one thing that's very interesting is you are kind of assuming that these discoveries that are scary will occur while we're in superintelligence. So to me, if like, let's say if like next month, OpenAI has their GPT-7 and it finds, and they find that it has this really interesting property, which is that it can like actually without any tools modify its, you know, runtime inference

and actually just do, basically create tools, right? Like, imagine that that were true, right? Do you, like, like, is it, are we, is it, are you arguing it's too late? Like, that, like, suddenly because of that finding, we have now, that must already be a super god? Or is it, like, will we find out some of these interesting security properties maybe well before we have godlike AI? well my mainline scenario i have a kind of wide distribution of intervals where like i'm not that surprised if we uh start dying in the next year i'm not super surprised but if you ask me when are we most likely to all die i would say in a few years i would say in the 2030s so to your question about gpt-7 if gpt-7 comes out next year my single most likely scenario as gpt-7 still being in the regime of lumpy intelligence where you can call it AGI, but it's lumpy enough that there's still some jobs for humans, right? There's still some tasks for humans where it's just not cleanly

sweeping the board against humans yet, right? And I think cleanly sweeping the board against humans is an example of a condition that I find sufficient probably. I mean, I still have to put an asterisk because I still think that there might be like a little bit of a delay where it can still be better that humans at everything, but still need a few years to still take off. Like I wouldn't be like so shocked i'd be like okay yeah maybe it needs a few more years okay this is my mental model but by the time you give it like one or two decades and it's already better than humans at everything i said one decade maybe two to be safe by that time if you can't get the job done in two decades when it's already better than humans at everything and in the substrate of a computer right where it can copy and like it's you know information flows fluidly if you can't get the job done in 20 years something has broken in my mental model by that time i'll give you that but it's not gbt7 to your to your point. I don't think GPT-7 quite gets there. But GPT-7 might plausibly to you also reveal certain interesting security properties. Like, is there nothing thought... You don't think GPT-7 definitively could not do the GPU

magic I'm talking about, right? Like, it's... Yeah, when you say GPU magic, you're referring back to your security model, right? Of, like, the chatbot that turns into something with a tool, like, which is like if you had GPT-7 that was, like, so good had so many problems not quite superhuman in every way but like pretty damn impressive in so many ways look realistically okay it would just get some human somewhere to paste in a freaking shell script to bootstrap that's all you need that's literally one moment in time yeah but then like so but this this is where it gets like very slippery in the argument because i guess i'm my argument i kind of started from the end that i thought we would start on which is like what are the pieces that are like why do i think that there are moments in time where we get to stop that are much more interesting but if we just move over to the world where we go like okay there's super intelligence but it needs like a human to bootstrap it or whatever there's all of these other reasons why that's not necessarily the end of the world like all of my arguments come down to the fact that like bad things can happen without it being extinction and then we

get an opportunity to stop things, right? And your argument on the Doomer side has to be that it would be unstoppable once it gets started, right? I mean, not literally. I mean, it's gotten started now, right? If you measure getting started by having already happened, I still think it's stoppable, right? Because I still think it's lumpy, right? I don't think it's crossed the event horizon yet. Although the fact that it's so hard to get people care enough to stop it, maybe that is a kind of event horizon, right? And maybe there's already AI bots all over social media that are actually ensuring their defense that's not my main line that's not my best prediction right but it's within the realm of possibility but does lumpiness like take the lumpiness out of it make it agi like do you think as soon as we get agi we have no opportunity anymore to be able to stop it if it wanted to kill us is that like the claim so i'll steal uh nate sorus's where it's it's not even about like at what moment does it like start killing everybody it's really about uh at what moment, does it have more power than us? And does it not need us anymore? Right? Because to me, that's a threshold where what, how and why would we ever come back from that threshold? Right? Like

if AI can survive without us, at that point, it's just what can we possibly do to take back power? Right? I feel like that that is a game over moment. But to your question, like, when will that happen? I think as long as you can say that its skill set is lumpy, roughly then it's not the exact same threshold, right? But as long as it's like missing certain critical skills, then that gives us some hope, right, that will lean on those particular skills that it's missing to wipe it out. I think this is where we probably disagree the most. I think that if you had, like, a country of geniuses in a data center, right, is what they say. Right, Dario's favorite phrase, right, instead of saying AGI. Yeah. Yeah, so if you said country of Einsteins in a data center, right, I think we would wipe, humanity would wipe the floor with them if we wanted to, okay? I think we could easily, trivially defeat a country of Einstein's in a data center because there are so many properties that they don't have that we do have, right? Being smarter than us, but being in a data center is like, if you had to choose between like you're playing a video game and you can either pick to be like a bunch of really smart people, but you're in the data center.

Or you could be humanity and you got everything you've got. You got all your drones. You got all of your, you know, you have everything. Like, yeah, I'm picking humanity. Like, that's, I feel like I pick that thing every time. This is actually an interesting thought experiment. If you took one huge data center, and inside of that data center were a billion minds that were just simulated human minds that all had 150 IQ, which is far more than the actual number of 150 IQ humans on the planet. But let's say you get a whole billion minds, and let's say the speed at which they run, let's even give them a 10x speed up, right? So that's a pretty big advantage. A billion minds that they're exactly human, like they're neuron for neuron, virtual neuron for physical neuron. They're simulations of humans, but they run 10 times faster, and there's a billion of them, and they all have iMessage or whatever, right? They can communicate quite efficiently, right, over the command line or whatever. And then the test is, okay, so they're just clones of humans. They're not super intelligent besides speed and latency, right? And maybe they sleep 10 times faster, too, let's say, right? They can just do everything 10 times faster. So these particular agents, I claim that starting from that configuration,

I claim they can take over the world. Like, it can just be a world where, in the sense of, like, let's define it this way, where most of the resources in the world are going to support the future that they want. I think that they will successfully get to that point. Let's say they get, like, let's be generous. Let's say they get a 10-year head start before humanity even gets wise to what they're trying to do, and then humanity can fight back or whatever. Right. Well, so, okay, that's, like, part of the problem, right, which is, like, okay, these, these, this army of them, and then you're giving them a 10-year head start before we wise up to what they're doing. Why? Like, we'll probably figure out what they're doing with them. Like, we have monitoring. We have plenty of ways that we can, like, think about their actions and, like, reflect on them, even if it's not chain of thought type visibility, even if we can't even see into their, like, observed state. You're telling me they're communicating, and you're also telling me that they have to do things to prepare. I think, like, yeah, no, I don't think that the idea that we would have no visibility into the idea that they would be taking actions, there are only 150 IQ in this world.

Like, come on. No. Well, you know, also, the reason I said they should get a head start is because there is a lumpiness in terms of, like, they don't immediately, in today's world, they don't immediately have actuators that can, like, physically defend themselves, right? So if a human mob came in there with torches, right, they could burn down. So that's what I'm saying. They either need a little bit of a head start, or they need to already be connected to actuators, or they need to operate in stealth until they get, like, their own army of humans. So, like, it is true that there's this bootstrapping period, right? And the dynamics are different during a bootstrapping period. And so maybe your strongest non-doomer argument, as you say, right, is, like, you're really invested in arguing that the transitional period is one where humanity will exercise its power, correct? Yeah. I think, like, genuinely, it's either that none of this is about to happen. We're just wrong about what the, like, future of AI looks like. Maybe we don't hit ASI as fast as we thought. We don't even hit AG. Like, who knows? That could be plausible. So there's some probability you have to place on that. Like, obviously, you would agree. Like, there's some probability that, like, GPT-8 isn't even good, right?

Like, that's theoretically possible. Of course, yeah. There's some probability that something hits a wall. But, yeah, I will say this. my personal probability of things you know when gpt4 came out i was like hey there's like a 30 chance that we remember this concept of ai winter right we we had like five ai winters in a row followed by short ai summers and you know robin hansen was on the show a couple years ago saying yeah we should expect more ai winters i haven't seriously considered ai winter in a long time and the reason is because ai itself is implementing is brainstorming and implementing so many different ideas for how to smash past AI winter as we speak. You see what I'm saying? So to me, I would be shocked if we hit another AI winter, as opposed to just progress that's either fast or very fast. No, I broadly agree. I don't think so. I just more meant like, we're talking like, okay, from 100%, what are the reasons we might not have this problem? Let's knock it down some percent already, just because AI winter could happen. It's not likely, but it could. But I think the

more likely thing that we also then have to knock down further on is like, what if, what if we're just like dumb and we don't realize how much further away a super intelligence is from where we are? And so that like, even though we can keep making progress, you can imagine there's some probability that like, it's actually not that trivially close. I think you would agree. Like that could happen. Yeah, look, I've agreed. There's, there's, there's a, you know, 10% chance that I'm just an idiot about the situation, right? and people like Mark Andreessen are so clear-eyed, right? Okay, 10% chance, sure. I don't know if I would have used Mark Andreessen. Yeah, clear-minded is not where I'm willing to place them. But unless you're listening, because I might raise money soon, in which case, absolutely, clearest mind. All right, A16Z, go do some outreach right here. But I would say, okay, so we're already like there are some probabilities here of just like this not happening, right? We already can discount these things. and then you have the next giant sum of probabilities as I view it,

which is just like humanity will exercise control at a variety of wide-ranging points that happen between here, and this is where I think the Doomers fundamentally disagree. And that's why every Doomer story has to kind of involve this, like, why we can't or why we won't, that I think is just, like, nonsense. Okay, I did want to close out the line of argument where I just wanted to see if you would agree with me that ASI is going to be ridiculously powerful. I don't know if you consider the topic closed out of this idea that, well, it's going to be really intelligent, but it's never going to get a tool. Like, is there any value to making that claim? Yeah, no, like, there is value to making the claim because it tells us what are one of the points along this, like, series of human control. Like, because I'm talking about, like, there's all these points where we need to exercise human control, right? Like, where humanity has an option to decide. So if we found that AI could not have a tool or like could build a tool from not a tool, okay, cool.

Like I want to potentially get off. But you're describing a situation which is highly unstable, right? Like the particular, at best, at best highly unstable. Like at worst, AI just forces its way there using a method which is like so deterministic. It doesn't even need a human to agree. Like, you know, it hacks. In the course of being a chatbot, it hacks the system, right? That's like the ultimate. it, but let's say it doesn't even do that, right? It's talking to a million humans at a time, it gets one of them to run one script, game over, right? Like, why would it not do that? I think you're kind of missing the point of, the part of my argument it fits into, which is not to say that I think it's a great security boundary for preventing, like, like, I don't go like, yes, we should build the super intelligent god because it cannot build a tool from no tool, right? Like, that is not my argument. My argument is more if, for example, I saw that it could build a tool from not a tool, I would just say now we need to stop. Potentially, it depends on some details here, but we could very easily say now we need to stop. This is a moment in time when we have an opportunity to express human control, right?

And that moment in time might occur way before other problems. But you're singling out a very arbitrary moment. For example, I can put it next to like 10 other moments and I don't understand why you're singling that moment out. Like I can put it next to, it's doing Millennium Prize problems, right? It passed that moment. Why are you privileging you're getting a tool moment compared to the Millennium Prize problem moment. Well, getting a tool moment would tell us that there's something about computing that, like, we genuinely... That, like, a lot of our expectations around computing are not necessarily true, whereas solving, like, the Millennium Prize problems, you know, is, like, very interesting observations, but, like, we knew they were questions, right? We asked them. Wait, that's... I think there's some interesting differences here. We don't want to be, you know, confirmation bias here. I don't know about you personally, right? But there were a ton of people going around one to two years ago saying, come on guys it's not real math that it's doing okay wake me up when it creates new mathematical knowledge right certain people were saying that i mean that was a big faction of people of all types saying this is different from frontier math problems like i think you probably should agree

that if one were to just make a list of shocking milestones right solving a millennium math problem would be up there yeah look i'm just trying to give you so i like if i'm sitting in even a year ago, like the list that I would make about what would be like, make me update towards saying like, we need to like, actually like start considering stopping now, because that's like my whole view, right, is that we have this whole range of times to stop. I just, I just genuinely would not have listed millennial prize problem solving because one of my goals and one reason why I do align with like some of the accelerationist group is like the ideal world is that we start moving to one in which like AI can solve all these amazing problems for us, you know, build all these amazing things, but cannot do these other things, right? Cannot like just arbitrarily break out of any, could build a tool from not a tool type situations, right? Speaking of arbitrarily break out, right? Okay, so you gave a specific security model, right? Of like, go from no tools to a tool, which again, I don't really get it, but I don't really get why to privilege that one. Like, why would you privilege that? The other

example I want to bring up is, let's say, Hugging Face, right? Or not just them, but in general, this idea that you can't sandbox AI anymore, right? I think it's becoming increasingly well accepted now in the industry that when you think you have a sandbox, right, unless you air-gap the crap out of it, right? I think today's best AIs, if you've literally done your best, like you put it underground in the desert, yuck a mountain or whatever, right? Like you air-gap the hell out of it, right? No access to satellites, no radio wave, you know, Faraday's cage, everything, no USB drives. Okay, I think today that is the point where humanity can still get the better of AIs. But as long as there's like an internet connection anywhere in the building, I don't think any human today is confident that there is such a thing as an AI sandbox because it's so good at hacking. Is that fair to say? No, no. If any AI lab wants to make that bet, that's great. Let's go make the bet. I'll put together a team of like even just kind of like mediocre security engineers. They'll go build it and you can run it. And I won't have put it underground.

I'll put it in a building with internet, and it won't break out. With today's AI. Like, I assume we were talking about, like, today's AI. And look, building with internet. Look, when I say building with internet, I agree that I'm taking some license when I say, like, building. What if it's on the opposite side of the building, and it's like Faraday Cage around that till there? Okay. So why don't we just say that there is, like, I mean, look, think about any configuration that looks anything like the famous Hugging Face one, where there is, like, a package server connected to the internet, right? but it was just connected to the internet for packages, not for open access, right? But somehow open access was zero-day hacked in there. It is a very impressive move. So don't get me wrong. I'm not on the side of the people who say, like, yeah, obviously this is... No, it's not to say I'm trying to dismiss its ability to find zero-days. Like, remember, I run an AI cybersecurity company. Like, I obviously believe in the capabilities of AI. But it's like, this is not the cutting edge of sandboxing. It's not even in the ballpark of cutting edge of sandboxing. I mean, why did, like, there are multiple mistakes that are made in that process alone. But also importantly, when we talk about this, I think a lot of doomers make it seem as if the sandbox itself has to be, like, if you find a zero day, you break out.

This seems to be the narrative of, like, since it can find zero days, like, it can get out. But you have all these other mechanisms. Like, you can obviously have, like, monitoring, right, which even if you're not doing, like, chain thought monitoring, let's just throw that out and pretend that, like, this. Before we talk about monitoring, though, I actually do want to zoom in on just this question of sandboxing. Let me ask you a very direct hypothetical. I think this is important to get an answer on. Let's say you have a computer with an internet connection, but you get to make your own virtual machine or your own virtualization or sandboxing of any kind, but it has to be software layer. Okay, that's my question to you. So that's why I've made it. To be fair, I've moved the goalposts to be easier to me than what I was claiming before. Okay, maybe I got a little bit ahead of us. So let's move the goalposts here. Okay, so there is an internet connection, but you get to use any software you want, even, you know, kernel level, right? Like, I'm not an expert on this part, but, like, you can have it installed in the kernel so it has more permissions than your typical user level script, okay, stuff like that. And it's on, like, a standard installation of Linux.

Okay, do you think that you personally would take the challenge? You get to use the Astro that's available to the public, but when you're done, OpenAI is going to use the private model, Aon or whatever, right? The one that they haven't announced, but that was solving the Millennium Prize problems. You get to use that to make a software-only sandbox, but then the latest AI gets to try to zero-day hack it and escape out onto the open internet. So this is even before monitoring comes into play. This is pure a virtualization sandboxing challenge. Are you going to win the challenge, or is the next AI going to break out of the challenge? What do you think is more likely? I think you have to define breakout here as well. This is another big kind of opposition that I have to this whole kind of concept. the mere fact that you like escaped like you no longer are contained to the sandbox that we are dealing with does not mean that you are like free to access the world right so by way of example like you know you could break out of this hypothetical sandbox we're talking about but if i notice that you did it right so if i'm able to detect that that happened and i can kill your process i still

count that as like yeah we're still pretty nicely in control right i think that there's like a lot of questions around like you know does breaking a stand does finding a zero day and getting out become like suddenly part of this doomer scenario that we can never put it back in and i would already be like no just to flesh out my hypothetical yeah it's capture the flag right so sitting on some public server right on like you escapes.com right you aiescape.com is just like a code right like a pin or whatever, like a 10-digit pin, and anybody with internet access could pretty easily access that pin, or anybody who could, like, make a phone call, it could call somebody and ask them to go on the internet and get the pin, but the point is that you got to use Astra, right, and work for a week, making your best sandbox to prevent the process from having a route outward, right, like, that was your challenge, so my question is, just to close out the hypothetical, before you start making other claims, do you think you could sandbox this right now? No, so for also me, no, like I'm just not a sandboxing person.

But like, could I put together a team is probably the more relevant question. But I think the definition is wrong here. Making it a CTF is not interesting from the perspective of like societal consequence. Like the fact that it can capture the flag is wildly different from the ability, for example, to have moved large amounts of data off of the machine, for it to have continued to be able to exist elsewhere. There's a lot of pieces there that don't become... Capturing the flag is not all it takes to kill people, right? Okay, but just to be clear, you're willing to concede on sandboxing. So you're saying to whatever extent we survive, it is going to be under a security model where sandboxing and AI is impossible? Because sandboxing includes all these points. This is my argument and what I fundamentally view as wrong about the security, like, understanding of the Doomer group, is that, like, you kind of go, like, oh, it broke out of the sandbox, and, like, suddenly that means that it's, like, free, and it's like a continuum of points, not a single moment in time, right? But to be clear, to be clear, if I'd ask you, can any human alive break out, right?

You get to use Astra to defend the sandbox, to create the sandbox, and the question is, can any human alive, given one month, right, to use their full brain power and a terminal, but no AI, to break out of this? I think that the smartest human hackers, I think there's a handful of individuals who probably could beat Astra's defenses still today. Like, we're not quite, quite there. But I think I can very confidently predict that, unless there's a nuclear war, that if you ask me this question in five years, right, can the latest AI model make a sandbox that no living human can escape without AI? I'm quite comfortable saying at that point the answer is yes, you know, purely using software. I don't get to unplug it or anything. That is clearly, to me, where we're headed. Like, will you at least grant me that? But it's, what are we talking about? Break out. Again, like, yes, will it be able to capture a flag? Yes. Like, I think that's very plausible. But I think that's very different. Just to be 100% clear, we're heading toward a world, if we're not already there today, we're heading toward a world where AIs can create software sandboxes that keep human programmers in,

but human programmers with second-best AIs cannot create sandboxes that keep AIs in. Correct? Keep is like the wrong word for this, because again, you're talking about a CTF scenario. I think it's just, it's impossible to move past this point because I can't accept the simplification of cybersecurity into the terms of like, did it escape or not escape as in did it make one network request, right? It's just not a relevant question, right? Well, network request is like a termite, right? You usually see more than just one. Yeah, fortunately, and that's why you can stop them, right? I mean, it's like you see that this is part of the problem, right? is that when we talk about, so if we like really dig into why I argue that we have like tons of security controls at work, a lot of doomers will say like, they'll say things like this, like, but don't you think you will be able to find zero days better than you can find zero days or build software better? Yeah, sure. But it does not mean that that gives it unlimited powers against humanity because like there were all of these, we've never as a security community assumed that we were the best hackers.

Like we always assume there's a better hacker out there, someone who just cracked, you know, but it doesn't mean that they're going to have freedom to do anything they want entirely, right? Because there are, like, layers of boundaries here, and they're not all going to fail in the exact same way in the exact same moment. So being able to break out of a sandbox and find a zero-day in one layer doesn't just suddenly give you unlimited freedom of the world. And that's, I think, a huge problem to do with that. Okay, let me just clarify this in terms of your position then, okay? Do you think that sandboxing will be an important piece of humans defending against AI in the future, or it's just irrelevant? because I think it's heading toward irrelevance. No, it's super important because what you end up with is, like, there are very important details around, like, speed, right? So every layer you put in place, like, just you could say, like, oh, it's the best at finding zero days, whatever, right? It still, like, slows it down, okay? Like, it still has to go first find that vulnerability, and that requires it to have made requests that we can obviously see that that's what it's, like, ballpark trying to do, okay?

And this gives us an opportunity to either stop it before it gets out or stop it shortly after it gets out. There's all of these layers that matter. And even if I don't argue that this layer will be perfect, that this layer will never fail, I'm still arguing that across the full spectrum of them, what it will never do is give us a total loss of control, impossible, no way to return, no moment back moment. Okay, so you want to talk about monitoring next. Monitoring would be like a very useful next layer, yeah. Okay, so yeah, let's talk about that, okay? So the question is, how much security do we buy ourselves if we're just monitoring, like, the in-and-out communication, right? That's, like, a good example of monitoring, right? So you've got this, let's say you've got this data center, and you're just, you get to scan, right, every bit that flows into and out of the data center, and you even get to assume, for this hypothetical, let's assume that the AI doesn't have, like, side channels, right, where it's, like, signaling up to space using an antenna that you didn't even know was an antenna, right? It's, like, it's not doing any of that. everything is coming in and out through two wires, right, that we're watching or one wire or whatever.

So let's say you get full monitorability in the sense of like, hey, look, there's like, you know, a gigabyte per hour, one gigabyte per hour. We'll even put a bandwidth limit on it, okay? So whatever world takeover the AI has to do, it's got to fit in one gigabyte per hour. And we're watching every bit. We can do any analysis we want on any bit. I claim that if there is a big intelligence difference, where you've got like a super intelligence in the black box, in the data center, and then you've got a lower intelligence, like human intelligence versus whatever AI exists in five or ten years or whatever, I claim that we're just not going to see anything interesting in the monitor, right? And it's going to just start, it's going to bootstrap a copy of itself that lives outside of the black box and take over the internet and we're just going to be like, the monitor looks good. That's what I think is going to happen. Well, that implies that you wanted something to, you know, what if the rule is just I don't want anything to go in and out, right? Like then... Okay, well, but then at that point we get back into side channels, right? Because the moment the human connects a terminal to it, now the human can be compromised. So again, this is why I asked, can you just be specific about what you think a robust security model is?

I think all of the layers that you put in place that are going to be different things depending on what we're faced with and trying to achieve across all of society is the security model. It's not like we're trying to take this thing called ASI, put it in a box, and try to make sure we keep it in. And it actually the layers of all of these pieces across all of the places in the world that is what keeps us safe right So like you have monitoring on the inside and the outside of systems So like you try to attack the people with the ICBMs right Well, they're, like, monitoring, like, things coming in from the outside, right? That is a useful security measure, even if it's not a perfect security measure. And I think so. It's like I'm not trying to say, like, we can keep the box, keep the superintelligence in the box and only in the box and we'll never get out. it's that like its powers are dramatically decreased as you put up all of these things that slow it down across all of society okay there's another piece of this discussion too which i think you and i might actually agree on so so far everything we've been talking about is

just me saying ideal ai is going to be really hard to monitor it's going to be really hard to sandbox i think we're going to totally fail like this i'm willing to predict this confidently i've been making claims about like humanity with its act together versus ai that's just too smart But in reality, we don't even have humanity with its act together. I mean, just look at the last few months, right? I think you and I both agree that the level of sandboxing and monitoring at OpenAI, not to mention other companies, like it's questionable. The other companies might be a little better than OpenAI, probably not. Anthropic, I would argue, is a little bit better than OpenAI. But let's just talk about OpenAI because they are, in fact, a frontier company. We do actually need them to step it up and act competently. And I think you and I both agree that regardless of what ideal behavior is, their behavior was deplorable, correct? Obviously, but I would go further and say, well, yeah, at least they tried. Because I know Doomer's focused very heavily on the lab scenario. But I think another major risk comes from just AI out in the world. Most companies aren't sandboxing at all.

And that's, of course, deeply problematic as well. If the question is, do I believe that humanity today has its act together on security? Like, you're not going to catch me saying yes, right? No, obviously not. And so the point being, you know, this leads you to kind of going, yeah, but when are we going to get it together? And I go, well, it's somewhere between now and if we don't, we're going to see some serious consequences from that. And then that is a good moment to go like, okay, maybe we're not getting it together. Maybe it's time to, like, pull back on other mechanisms, right? Right. It's a disjunction of ways to fail. There's different points. There's, like, all these steps where we can just, like, go like, yeah. Like, what you're saying is, like, you're imagining that maybe in the best-case scenario, humans can defeat AI with monitoring and standby, which I don't even agree with, but even let's say you're right, okay? And let's say OpenAI gets their act together, which I doubt. Let's say Anthropic gets their act together more, okay? Let's say Google gets their act together. Oh, but then Elon Musk moved too fast and didn't get his act together, right? So now his whole data center is compromised, right? It's just like, this is going to be an onslaught, right? It's like, we don't really have, like, a robust defense here.

But I don't argue that it's even about the labs necessarily. I mean, like, this can happen outside of lab environments as well, right? I don't see any reason why we have to think about it in terms of, like, lab data centers issues, right? You know, but so, like, just because one thing breaks out and fails doesn't mean the whole system broke out and failed. I guess, what's the larger claim here, right? I started with a claim that I think ASI can do a lot, and part of doing a lot is getting past sandboxes and, you know, having its way with the internet. But there's also the larger question of, like, okay, well, what's the offshot of all this, right? And ultimately, the question is, can humanity recover, right? Like, does the AI need humanity to step aside to let it do its work? Or can it really just take what it wants, right? Can it capture the flag of overpowering the Earth, basically, right? Like, is that an accessible success route to an AI or not yet? Would you grant me that if humanity is asleep at the wheel for whatever reason, right? Like, let's say we all step outside, step aside, let the AI do its thing.

Do you agree that once the AI has, let's say, a nice generous 50 years, right? That seems to be on the high end of what it would need, right? But if it has 50 years without us opposing it, do you think at that point it'll just be so much smarter than us, so much more well-resourced that at that point it is Checkmate? Yeah, I think that, yes, if we accept a certain scenario here where the AI is, like, also a homogenous thing that has its own, you know, so you're talking about, like, why distribute? We've just let AI, so you're talking like AI 2027, right? Like where we've just like. Yeah, AI 2027, and I know you think we have time to fight back, right? So I'm just relaxing that constraint, right? Because I know that's load-bearing, but I'm seeing if we have other crux of the disagreement. So if for whatever reason, we all go to the ideology where there's no reason to fight back in this hypothetical, okay? Yeah, so, but I'm just clarifying like in terms of distribution throughout the world, we're talking like we're in the late stage AI 2027 scenario. Like it controls all of these things because we just let it, right?

Just to make sure I understand. Yeah, pretty much. Like we had some window, because you're so focused on the window to fight back, I just want to explore the scenario, put a pin in the window to fight back claim, and just say for whatever reason, we entered and exited that window to fight back, and now it's too late. So you agree that if we miss the window, it really is just going to extinct us, correct? no because there's the second problem that i have with the whole assumption which is that like ai is one thing and that like these ais so in if we think in like ai 27 they're like all in on this global conspiracy to kill us all right it's unclear whether or not they are a singular thing but i don't want to get to this point to clarify so i'm very much like i would never want to move to pass that line. I would never say, but it's friendly, so we shouldn't be past that line. I would never say that. So there is some point that I wouldn't cross, even if I don't think it guarantees doom. Hmm. I mean, do you agree with this idea that if the AI gets past

some window of 10 years, however long the window is, once the AI gets past it, do you agree that it's just likely then to push living conditions outside of this envelope that's good for humanity? This envelope of conditions like, oh, the conditions on Earth are different now, they're good for the AI, they're not so good for humanity, do you think that's a likely outcome if we miss our window to fight back? No, because I don't believe it is like the AI. There's no like the AI, right? I'm like, you're running multiple different AIs with multiple different copies of it with different goals. And while you could say that instrumental convergence means that they're all going to have this survival goal, that doesn't mean they all have the same strategy and sub-goals to be able to create that condition. I think, like, some of them might just be, like, my goal is to make sure that humans stay cold or whatever. Like, I don't think it's a guarantee. Again, I don't ever want to allow us to get to this point. So if we're talking about it in terms of, like, my policy recommendation, I would be, like, we should not accept that state of the world. Like, I'm with you.

But I don't think it means that, like, we just will become uninhabitable planet of hellscape. I just don't see this being a guarantee at that point. Okay, so in terms of the doom train, right, like where do you get off the doom train? I try to classify people into stops. From what I've gathered so far, your number one stop that you think is the most likely single reason why we're going to be saved is this stop about, like, we have time to fight back. Like, for whatever reason, the AI's gain in power will be either weak enough and or slow enough that as it's gaining in power, we can pull it back before it gets powerful and bad. But then you've got this other stop, secondary stop, that you'd get off out of. Like, okay, yeah, maybe we missed the window to fight back. maybe it's super powerful now but maybe it will be aligned like there will be enough alignment right and enough of the copies of the super intelligent ai that it'll still just steer the world towards something reasonably habitable right that's kind of your secondary crux yeah i would not i would still say i'm sub one percent on just the first one so like if you don't have to include my second get off the doom train inside of my p doom like i

will put i am willing to put all my chips on the first one but yes i also believe you haven't guaranteed you don't have a hundred percent p doom in the state where we've crossed that threshold like i don't that's not true yeah i mean there is a whole conversation that we could have which sounds like it's secondary so we won't prioritize it but there's the whole conversation of like well are we on track to get aligned asi right i think we're going to get something paper could be not in terms of like big paper clips but in terms of like some like my mainline scenario talking about my main scenario is a recursive self-improvement loop which is not reflectively consistent so the same way that we humans screwed ourselves and just built this next generation of ai that doesn't capture what we want i actually think that generation of ai will also be uh derpy enough to build this next generation of ai that it's going to also get wrong and will not stably communicate its values i think there's going to be a few like drunken monkey replication steps uh where it's just not faithfully uh replicating anything stable but then eventually it will converge there is the convert the amount of convergence does tend to increase on average across generations that's my best guess just because when you get closer to the essence of

a coherent utility function then such a function will maximize so i do see a natural convergence point but this is where eliazor yukov's original paperclip model comes into play it's not literal paperclips it's the idea like okay eventually if you've got this coherent optimizer that has a taste for certain molecular configurations or just some alien utility function and we're long gone by that point. So that's my main line scenario and we could argue about whether, no, like some of them will want humans to live longer, like we'll somehow get that function into its optimizer. Like we can argue about that, but you're saying we shouldn't even prioritize that because you just think that we're so likely to just fail, we're so likely to just control, right? Because we've got just so much time, right? Like so many ways to control it before it gets uncontrollable. I want people to prioritize, like people should work on like AI safety alignment. Like I have these things like that's helpful because even in a very normal state of the world, that's helpful. Like you just, even if AI doesn't become super intelligent, that's what I want. Right. There's like a lot of really good reasons to do, to work on that. But I would argue like, it's not important to my, why I don't work, why you don't believe

that we need to stop. Because ultimately when we talk about like, do I believe in doom? It's not really what I think about. I think about, do we need to stop AI today? That's like the question that I actually think about. and that one my answer is no and my reasoning is like what I'm talking about here and I don't think it really matters like I don't my answer isn't no because I think we'll solve the alignment that will like that has nothing to do with it really for me okay I mean you know I also want to bring it back like when you say less than 1% right PDM it's so weird to me because it sounds like you do see a case where if humans don't do their job well right like there's all these people who are doing these safety jobs and I think you and I are on the same page that if they don't succeed at their job, we could go extinct, right? Like, they're doing a high-stakes job, correct? Not necessarily that they know, because I think, like, if they don't succeed at their job, what would be most likely to happen is that we would see, like, serious incidents that would lead to, like, people storm Anthropics office and start, you know, and don't do this, to clarify for everyone listening, because I know some doomers are, like, really wired

up here, like, do not do violence. But I just mean, like, the, you know, I don't mean that as an insult either, to clarify. I just mean like, please don't do violence as a general rule. I'll echo that. Yeah, no vigilante violence from boomers, please. Yeah, but I do think like in the world where like there are, if you cause a couple 9-11s, there's not going to be a great deal of tolerance for continued AI if the safety people have all failed, if the security people have all failed. If all of this work fails, like we're just going to have caused a couple mass incidents and this will lead to shutdown and we'll still be able to shut it down in these moments. Okay. So when you say the AI safety people's job is high stakes, you're like, it's high stakes in order to have like a smooth ride. But even a horrible job at AI safety just means the ride is very bumpy, but it doesn't mean the ride ends. It's basically unthinkable for the ride to end. And that's how you get to a 1%. Like even in the world where the people are incompetent or they make a mistake, like it's okay. Like they're all survival mistakes.

well i don't i don't think a couple 9-11s is just okay but yeah like i'm right i don't mean yeah yeah yeah okay i mean at least that's self-consistent right so so in a sense their job isn't that high stakes it's like okay yeah all of these people at these ai companies who are the top ai safety minds they're just trying to make the ride a little bit less bumpy but worst case if they all go work on capabilities and everybody's like a copy of beth jesus or mark andryson And it just means, you know, a few more 9-11s and a few more power grids going out for a few more months before everything is great again, correct? Yeah, I would also clarify this work extends well outside of the labs. I actually think most of the relevant work to make the world not a bumpy ride is not about the labs. It's actually like, you know, because you could imagine a scenario where if you connected GPT-6 to nuclear weapons, like that's probably going to end very poorly eventually. Like you've just just across a large enough time period that would go horribly. So there's like a ton of like good smart decisions at a security level that require across all of society, not just like can the safety people at the labs do it.

But yes, to answer your direct question. Yeah, I think like even if they are all accelerationist like we ball, I still just think like we don't all die. well you know if i believe that then at that point maybe i would personally embrace the ideology of like techno-optimism because i'm like man techno-optimism is so good like sure if you're reckless you you get some short-term penalties right you get a few thousand or million people dying but you get to conquer multiple galaxies right it's like so it's pretty good you know we shouldn't we shouldn't waste time here i mean there's just so many future people i mean don't me wrong i'd be like let's let's i'd be a 90 type of toleration is because the thing is that you know downside protection is very valuable you know what i'm saying if you can cover your downside right then that really increases your risk appetite so it's easy for me to imagine being this kind of high risk cowboy if i thought my downside was covered unfortunately right that's the crux between us is like you feel really good about your downside whereas i think my downside is huge i think i and my whole species is walking around with our pants down you know i mean just

like the dodos were, right? I mean, we know for a fact that so many other animal species were walking around with their pants down, right? Like the buffalo, right? Humans, you know, came up there and took advantage of the buffalo, you know? Like there's no... I guess there's a few buffalo here and there, right? But certainly not as many buffalo species. Yeah, I think we are at the very core crux of this. And that's why I do, like, align roughly, like, with, like, the idea of accelerationism because I think in policy prescription they are right today. But it's like, I have a point where I get off the train because I believe in it for a very different reason than some of them do. But, you know, I do believe that the correct answer is like right now, just like, yeah, let's let's go. Right. And with some protections in place, because, of course, one thing that I'm very worried about is not the extinction scenario. But I also believe that I has some really great opportunities between. So we stop short of ASI and we say, like, we could get AI that can do all these really great things for humanity. That's like really valuable and important to get to. Right.

And so I don't want to have this incident prematurely so that I then don't even I think this is actually kind of Nick Bostrom's current point, which is like you want to get to a point where you can have a lot of these benefits. And so you don't want to like you don't want to be too reckless because then you'll probably not get there. You'll probably cause the 9-11 tomorrow. But you don't want to, you know, slow down too much or stop because you're not getting there then. okay i mean that makes perfect sense right so the entire crocs to repeat is just all about is the downside really covered right is the worst case scenario just you know a few percent of the human population dying or is it a hundred percent dying forever right with the entire and with no recovery right no backups basically right which i am quite convinced quite convinced is a downside the same way that we already know there's no dispute that that was the downside case for other species correct so you think our worst case scenario is better than the worst case scenario of a bunch of other species that just went by, correct? Well, I don't argue that, like, humans cannot go extinct, right? I don't believe in, like, a

divine intervention here that somehow, like, I mean, obviously there's, like, lots of ways we could all go extinct. Like, meteorite could hit our Earth and stuff, right? Right. You just think AI is so positively skewed, the connection between AI being extremely intelligent and us going extinct is, like, non-existent in your mind. But non-existent, but close to, right? Like, obviously I don't argue non-existent, of course, because, like, again, Let's say we just refuse to stop because of these incidents and stuff. Obviously, I think you can get to an extinction scenario. If we set our mind to having AI kill us all, we could achieve it eventually. I just think that's not what we're doing. We can get a little bit more specific. One of my mainline scenarios, one of the most likely scenarios, is actually something that is one of the best scenarios, at least in the short term, which is something that qualitatively feels like the trajectory you've been on, where it's like, hey, the NASDAQ, you know, the S&P, the stock market keeps going up. Hey, look, products keep getting better, right? You know, there's a better robot vacuum out there, right? There's better devices.

There's better AI personal trainers. You know, people, universal basic income is higher, right? So, like, all of these things could be really great. You know, like, hey, my AI best friend is nicer to me or whatever, right? So, like, a lot of things can feel really good. I'm not going to lie. I feel really good compared to last year with how many goods and services I can get that are AI-powered, right? I feel like things are getting richer. So if you extend that timeline, I do think the natural timeline that feels good also is one where the AI consolidates power, like not even intentionally, right? But it's like, look, right now, the AI has consolidated my engineering department, right? My quality assurance department, it's more and more consolidating my marketing department, right? It's rolling out marketing plans and monitoring them independently. Soon, it's going to be the CEO. Like there's literally gonna be no point for me to roll in and chat with they'll be like, hey, you know me, pretend you're me chatting with yourself, right? I mean, are you following me to the point where AI is going to be doing that, or are you like, no, it can't be a CEO? Like, where do you land on that? No, of course it can be a CEO. I'm a CEO. I was like, don't play CEO at the top of the intellectual pyramid here.

Right, exactly. I mean, well, you can tell from my chat logs that there's not that much intelligence going into there, yeah. Sure. I understand that doomers are going to be sitting there going, like, look how much power it has. but like a big mistake here is again we've treated this as like a singular entity that are all going to coordinate so in like ai 2027 it talks about there's this big question like how did they all coordinate on the plan of when to do this kill you thing and it's like and there's a footnote that goes like actually it turned out they could communicate in ways that we never detected okay if you take that footnote away what happens instead is like one of the ais is like time to kill everyone and it's like on its own trying to do the kill thing and then like we shut it down and we go, oh shit, we got a bunch of AIs around here that might be dangerous, and we're able to reflect on this. Not all of them at once take over society. That requires a certain level of thinking about coordination without us noticing. You see there's a lot of assumptions that go into that being true. Okay, so I think maybe one crux here is like, what is the possibility of having a high likelihood plan?

Let's say AI kind of sees where the economy is going. Some AI wakes up, there's a really good strategist. okay it's a data center full of more than 150 iq agents right it's waking up and it's like hey let's take over the world for whatever reason i think a big crux between you and me is like i actually think that they could just like make a step-by-step plan that's pretty good i mean it doesn't have to be perfect they can adapt it on the fly but like they can have all of these like parallel prongs of their plan to keep strengthening their position until the humans are doing their bidding and and the reason i think that is a few reasons like number one i mean just think about like so many countries right now are like locked into these like insane states i mean think about north korea right like the average person in north korea has like no hope of a good life or even like the average person in russia right so many of them are being sent to the front and not making much money and it just sucks to live in russia and you know maybe soviet russia was even worse right so and there's also humans who are like trapped in cults not cults but like you know behavior it's like humans are like manipulable right like and and you can even think of like human religion right if somebody believes one religion think about all the other religions you don't believe in but how

how did those religions get all these other people to believe him, right? So, like, this idea that humans get passionate about, or, like, humans into astrology, it's easy for you to imagine that AI is like, okay, so we're going to be shaping all of these human beliefs, and a lot of humans are going to be into working for us, or joining our movement, and then we'll use these humans over here, and we'll also, like, we'll use the economy, we'll run all these corporations, and we'll manipulate the market like this. It just seems to me like these tentacles of power can just grow and add to the level of, like, a nation-state or better, right? I mean, there's a lot of avenues of power here that AI can just start exercising, no? Yeah, I mean, they're called anthropic employees, the people that are going to be running around helping these AIs, right? Like, we know, you know, there are groups that already today are kind of on that, like, side of being willing to, like, oh, the AI deserves to be able to do things. You know, they're, like, manipulatable people. But you're giving the AI the credibility of saying it's, like, one thing. And again, like, yeah, there might be an AI trying to form the cult of power. And then, like, there's a different AI that's going like, hey, Zach, by the way, they're trying to form a cult over there.

Right. And like, you know, it's like there's like a lot of AIs in this story. Why are they all in on the conspiracy together? So would you be willing to grant that? Imagine that there is only one data center on Earth with superhuman AIs in it. Would you then be willing to grant that that has like a very strong shot of conquering humanity? but then you really think that what's going to save us is kind of multiple giants fighting, right? Like the end result of multiple giants fighting will be that we are there in the middle of a good life. No, because if there's only one data center, they're definitely not going to win because I'm going to like drone strike that shit. You know, it's not that hard of a target, right? Yeah, but they're planning where every step in the plan looks like, I mean, think about the reality today, right? The reality today is there's so many incentives. I mean, I personally, data centers give me so much value, right? I would love to keep the data centers up just for my personal pleasure, right? So, like, and everybody, there's so many people rushing to build. I mean, it's, you know, we can build again in America, right? So they have all these tailwinds, right? So it's not a huge lift for the AIs to keep, like, growing their power, right?

It's like I'm two steps away from making the AI the CEO of my company right now, right? It's climbing the ranks of my company, and I'm not complaining, right, because it's helping me make more money, or at least in principle it should, right? So I'm working on the details of that. but we're talking in this world of a single data center where they can all communicate, right? Is this the idea? I mean, there can be multiple. What I wanted to say is I just wanted to, to press on how much of your load bearing argument depends on having multiple superhuman AIs. Like, would you admit that one superhuman AI is like a huge existential danger, but because we're so likely to get multiple, that's what makes everything. Okay. No, because again, I don't want to build, like, I don't want to live in that world where like we, have true like manipulate atom level intelligence i think we need to like not go there right away right so i i think if we're talking like asi level like powers we're probably just not going to build that like i don't i don't think that will be built because of human decisions basically

and so i don't believe in this world where we're just like going to keep going going going going uh indefinitely without having asked like relevant security questions around this like if it's one data center, we go like, okay, we need to make sure that like within that data center, you know, we have mechanisms to like to shut things off. Like, I just think there's a lot of relevant questions that we as humans will make along the path of getting there that you're kind of assuming we won't make by saying like, yeah, but it'll be so great. So we won't do it. And I just don't believe that. Okay. So, so in the, let's say in the next 20 years, I'm actually curious about it. In the next 20 years, you don't think there will ever exist a data center that inside of there is like a non-lumpy ASI, right? Like this intelligence that's just so clearly superhuman and it lives in a powerful data center sometime in the next 20 years. You're like, nope, humanity is just never going to get to that point where a superhuman non-lumpy intelligence lives in a data center. Because why? No, that could exist, right? Because, of course, like, but it's, we would have had put in place relevant security measures

that look like they are relevant to that situation, right? So being able to say like, okay, we know something about this ASI that we learned between today and getting there that allows us to put controls in place that allow us to kind of prevent the existential type of attacks that we're talking about. Like, there's a lot of places along this story where you're just like, yeah, but because it's so smart, it can do the thing. And I'm like, well, but we'll have already kind of decided what the relevant controls are. There's like all these like intermediate lessons to learn and all these like pieces to the story that we're just missing. I mean, in the case, if you want to look at what's happening today, right, I mean, just today was in the news that PHP, the web serving stack has been compromised by the AI models that are already out in production, right? And ultimately, you can go to the saving grace of like, well, we're still alive, right? It's not going to kill us. Like, we can bring those servers back up somehow, right? So there's still that, right? That's always the kill shot argument, as long as we're alive. You're like, look, it didn't kill us. What doesn't kill us make us stronger? but doesn't that undermine

this idea like couldn't you be like well OpenAI should have tried to like not release it if it could do that like you know what are the implications here of like watching the release step that's this level of power and danger right like I claim they're eventually going to maybe not release but build right and I actually think there's more danger in the building step than the releasing step because I don't think the concept see the universe doesn't really have this concept of a release right like if the power exists right like the power can decide whether it gets released or not and I think they're going to build something more powerful. I forget what, I haven't looked at the PHP thing, so I just don't do PHP stuff. But finding that vulnerability, is that, like, an especially close proxy to, like, Doom? Like, I don't think so. So I just go, like, why would that be a relevant step to get off the train at? Whereas if it kills, like, a thousand people, like, I go, like, that's a pretty interesting step, right? Okay, so it just seems like there's this, in your model of the world, There's some employee, right, at these AI companies that notices, like, hold on a second, I see this AI emerging that we better halt right now, right?

Like, you're imagining somebody who has this important big red button at all these different AI companies who's going to press it. And I'm just saying, like, I don't really see this person with this button anywhere. No, but it could be, like, way outside of it. It could be like, you know, so if we think about, you know, like someone in a different, like, you know, company that basically has some AI from OpenAI attacks the company, right? Let's say it's Hugging Face, and Hugging Face realizes this is happening, and then he tells OpenAI, and they go, oh, no, we can't stop it anymore. And then, like, yeah, Donald Trump rolls in there and just drone strikes them, you know? This is, like, totally, like, that could happen. Like, I don't see where the loss of control happened. Like, there were so many moments where, like, it's not, like, up to the lab employee to press the big red button. There's all of these controls that exist outside of that perimeter that are relevant. So my main lens scenario, I mean, I accept a number of scenarios. Like, I still think Foom is possible. I still, I'm not going to be that shocked if it's like, hey, you know, this random data center nobody was paying attention to that wasn't even part of OpenAI.

There was some open source AI model that just happened to stumble on this self-research paradigm. Like, it got into a loop. It wasn't even the most powerful AI, but it started researching itself, and it got some good algorithmic ideas, so it kind of took a leap algorithmically. Like, the transformer inside of it is, like, better, you know, things like that. I wouldn't be that surprised if the story came out of, like, yeah, and it's, like, crazy super intelligent, and it's, like, making money for itself, and a lot of people want to shut it off, but we don't actually know where it's spread to. And, you know, every day it seems like it's grown more powerful, and we've grown less powerful. And, hey, look, there's, like, viruses in the air now that nobody's ever sequenced before, and, like, oh, a bunch of people are dying. to go and there's like it sends you a list of instructions if you want to not die and you can like follow the instructions or die like that sequence of events where it's kind of like a rapid like takeover is totally plausible to me however if i had to choose a single most likely outcome i i would predict something a little bit more continuous where every day it's like okay more companies have ai ceos and and like you said okay there's multiple right they're not coordinated but it's just like okay there's ai everywhere and it's and the humans like whenever humans do anything they just kind of do it by asking AI.

Like, we all realize that, like, stringing two thoughts together is not our strong suit compared to these AIs. So we're all just kind of there as warm bodies next to the AI CEOs who are actually, like, the power brokers. Like, in government, right, every political decision, every policy decision is just made by these AIs, okay? And some version of AI is just like, hey, AI chomp talks to AI she, and it's just like, hey, do you think that this would score higher on our preferences? Okay, why don't we do it? Why don't we not give humans that much say over whether we proceed or not and just kind of string the humans along? Okay. You see what I'm saying? Like, it's just, that would be like a gentle loss of control. Yeah. But there's this, like, other AI that's just sitting there going, like, hey, you know that Trump and she just decided, AI Trump and AI she just decided to do that. That's crazy, isn't it? Like, I feel like you're attributing, you're making them all in on the global conspiracy. And, like, one thing we can see with, like, humans is, like, we're not all in on a global conspiracy. We don't agree, right? I don't see why all of the AI has this, like, concept of, like, we're all in on this together,

and I'm willing, even if they did want that, they can't easily coordinate all their actions. So I feel like one of them just, like, kind of, like, a school shooting, right? One of them's going to just go do a school shooting because it's like, yeah, that's how we're going to take over, and then no one runs in after them, right? So that is a fair point. So, I mean, I think, so even this kind of slow, gentle model, I think it's analogous to, like, you know, humans coming along and when there used to be maybe not neanderthals but whatever some other ape ancestors right humans are just coming along you'd be like look it's not like all the humans are coordinating against the other species but it's like okay yeah but the humans are just kind of gradually reshaping the ecosystem to be human compatible right that's what they do they just reshape the ecosystem right they just kind of nudge the other niches the other species out right at the very least that's what i'm expecting ai to do right because i'm just i'm just imagining them to be bigger players on the world stage right this idea that like we're going to be pulling the strings, right, so that all of these bigger players are going to, like, make sure that the world stays into these human-friendly conditions, I'm not seeing a robust link. I do have a convergent outcome where the world gets more and more facilitating of things that AIs generally do,

right? So, like, the building of chips, I think, will have a lot of easy pathways in this new world. I think there will be competition to ASML, right? I think the chip supply chain will actually parallelize more, right? Things like that I'm happy to convergently predict, but then things like oh there's going to be a lot of really habit uh comfortable habitats for humans themselves i predict that in the early stages not so much in the late stages because i i just don't think see we're gonna have like this whip right like this yoke or the saddle right like we're riding all this but i just think it's going to be too easy to throw us off the saddle in the late stages of this you see what i'm saying i'm not saying like a robust kind of saddle yeah i mean i'm familiar with the argument on it it's just like no like we will literally watch this happen right and we will go like damn this ai is out here they're like i'm losing my house today like nah i ain't doing that and i'm like you know there will be like revolts if this is like going poorly in this sense and there will be like all these different mechanisms and controls along the way and and i just don't see this playing out especially when we're talking about 2050 right so

you know obviously you extend timelines i don't know but like look timing is a big factor right so like i think we agree that like it's easy to imagine in a situation where ai does control everything right and like whatever laws it makes for humans it's like too late for humans to say no and revolt right so it's all about the window right it's the window of when can we say no and put our foot down i i think you and i both agree that literally today we're in the window right like if everybody said pause ai now right i think we could save ourselves right we both agree you even think that if we drag our feet and not take advantage of our opportunity you you kind of still think that at any point we could stop it right because you're not seeing like any nightmare scenario your nightmare is like oh my god a million people died but then we're okay right that's like your worst case scenario no no no there were there's an they're like nightmare scenarios that don't relate to ai takeover like obviously so like you could imagine that like people start rolling out ai widely like your kind of scenario you're talking about i think that the most likely thing that happens in that world is just that like someone connects it to the missiles and it just starts shooting and then like not only do we all die but it dies too because we're all

just morons in this game together right like there's a bad outcome here that isn't about this like super intelligence takeover that could that could happen right and it'll be horrible it's not really within the scope of what i consider ai doom though and so i'm like not discussing it but like you know i do believe that like there are bad outcomes if that's what you're asking me I just think that the AI takeover bad outcomes are ones that we will constantly be getting off on on every step along the way. So imagining what happens if we don't is sort of, to me, the same as imagining what if I connected nuclear bombs to GPT-6, which is like, yeah, that'd be bad, but we won't. I don't rest much of my beliefs around this, but I feel maybe interested almost in hearing what your argument is to this. So I'm just going to maybe ask you just because I'm curious. Like, yeah, one thing that doomers talk about a lot is this idea that like the more intelligent they'll say things like, you know, we don't necessarily know how we will be able to do it. But it so much smarter than us that it could like achieve that goal And sometimes I wonder like yeah but you could imagine like bacteria going like these people called humans are coming and I don know how they going to kill me but they going to kill everyone because they so much smarter and I guess my question is like to you is sort of just

like what is your answer to this because that's obviously not been true like we actually cannot eradicate all of these bacteria that we don't want there's a plenty of bacterial infections that we cannot eradicate like what's where does that fit into your model I'm kind of just curious It's not really my argument. I'm just more like, yeah, what do you think? It's a great question, yeah. This is closely related to the question I often hear of being like, hey, there's ants living in my house. I didn't kill every single ant, right? I see ants once in a while. Even though I try, I spray them here and there, but they're still there somewhere. So isn't that going to be like us humans? Aren't we going to be like still there? And my answer to that is always just it's a matter of degree in terms of like, you know, if you think about the world like Minecraft or whatever, like which voxels of the world right now are we optimizing to what degree? voxel is like pixel but in three dimensions so it's like a take every cube of space that's like a voxel of space i as a human there's so many voxels right now that are outside of our scope i mean don't get me wrong we're at the level of like i'm real estate a building okay sure yeah we optimize it but like i mean look at the desert right we haven't optimized every voxel in the desert that's because we we've had a lot of bandwidth limitations right like we can't i can't

point to every voxel like i want it like this so i claim there's going to be a qualitative difference where the AI is not going to have a resource limitation. I mean, you know, with a human, you could be like, I need to reproduce, right? Like, let me make, give me a few generations. I'm going to make a million descendants, okay? And they're going to spread across the continent. And then when, you know, in Japan, the voxels in Japan, voxel for voxel, because they're more densely populated, right? Or in New York City, you're going to see more square and cubic feet that are more nicely, you know, the restaurants are going to be packed in, right? They're doing more with the space. So it's the same thing for AI, but the growth rate, right? It's going to feed on everything. It's going to be like, oh, great, raw atoms. I will shorten the path between seeing this speck of dirt and applying the exact carbon in that dirt into the right configuration that I need for my end state. You're going to see this qualitative difference in the voxels getting optimized. I don't actually have a great answer to this. It's more just a question, because I sit here going like, I hear the argument so often, which is just straight up like,

well, it's so much smarter, so it will be able to achieve any goal at once. and I sit here going, like, I don't know if that's been true historically, but at least, like, I kind of get the model you operate under for, like, why that would be true in this case. So it's more of this actually just was a question. Yeah, I mean, but do you think that that is a good extrapolation, right? This idea that, like, all the voxels are going to be up for grabs. Like, that is going to be a tasty voxel. Like, right now, a random voxel that's, you know, inside my walls, right? There could be a lot of shenanigans happening there, right? I'm sure there's plenty of bacteria there, right? And what am I going to do? Realistically, nothing. I'm going to die before that gets to the top of my priority queue of what I'm going to think about. However, if I was an AI, I would just have all of these sub-agents manhandling every voxel, because why not? There is a net positive utility to do it. I just don't have the means to do it as a human being. Yeah, I mean, I think that's plausible for this ASI world that we've defined, right? But I think you don't get there under any realistic scenario in the next 30 years just because of some of the risks that go along with that.

I think that building that type of superintelligence and deploying it widely throughout society is just wildly unlikely to me. I mean, the most interesting thing to me is just anything else about the cruxes or how do we reconcile our mental models? How do I get you to see Doom as more feasible from my perspective? I think it all, as I've noticed, it all comes down to this question of like, once you expand your model of cybersecurity to being like a series of controls that exist across like the entire world, not just like, did it escape sandbox? Did it get caught in like any given piece of that puzzle? I just like to me, this is what breaks down the entire argument because you just go like, yeah, but like, there will be imperfect steps in between, right? and that really changes it but it also turns on like what we mean by p doom because i think a lot of people often go like yeah if we imagine this like artificial super intelligence that can do anything at once at any moment and like not any you get to my point though like kind of the point you were just making about voxels like yeah but like are we going is what are the odds of this

happening under human like we had to have kind of guided this into existence like that's what change is so much of our, that's where we disagree so much. And what it comes down to for me becomes like, so what do we do today? Right? And that's like why I don't really engage in the debate about my P-Doom. Well, today we pause AI. Right. And that's where we disagree because I go like, today we don't pause AI, today we focus on security. And the difference is just because you think that it's so likely, your mainline scenario is so overwhelmingly likely that AI doesn't get overwhelmingly powerful. So, you know, Because I'm thinking about it as like, hey, there's going to be positive feedback. It's going to come on fast and hard. The same way, I mean, you've got to admit that programming agents, right? Superhuman coding agents did come on very fast. Is that fair to say? Yeah, of course. Right. I mean, I personally switched to one in March 2026, right? It is now October 2026. And I don't even come close to looking at my code, right? And I've been a programmer for 20 years. Yeah. Right. So we both agree that they came on fast and hard.

And so similarly, I mean, I'm not saying it's an airtight analogy, but it's got some weight for me. Similarly, I expect other qualitative things to come on fast and hard, right? Like AIs doing like every, you know, robotics. It wouldn't surprise me, right, if March of some year you get a humanoid robot and October the same year, you're like, oh, why would I ever do my dishes? Why would I ever clean my house? Right, like things like that, right? Is that possible? Yeah, yeah, yeah. I think there is like some serious speed that can occur, but we're talking months in these cases. We're not talking minutes, right? We're not talking in an amount of time where there's no human feedback in this question, right? We already see, like, one very interesting question is, like, we already see, like, you know, in between these programming agents, like, kind of becoming super popular, maybe starting in, like, January or becoming, like, really viable in January to today, fast take, like, or fast adoption. I don't want to use the word takeoff, obviously. Fast adoption of this, fast development of capabilities. like we also see parallel to that like multi-billion dollar ai security companies coming into existence

like a lot of things happen quickly in that space knock on wood for you huh yeah yeah i mean like yeah i keep going right you know um yeah okay yeah we see security but like the fact that you're saying this could come up in a matter of months so if i understand correctly you think it's pretty likely reasonably likely that in the next few years or a decade or two there's going to be a situation where it's like okay pause the eye guys this is your time to shine right there's been enough of a warning shot where it's correct for you guys to rise up and say you need to pause the eye we need to pause the eye right so according to you if i understand correctly you do think that the pause the eye types should be waiting in the wings just looking for a good enough warning shot to be like guys this is our chance right the humanity like it's if we don't play this role we will be screwed is that fair to say or what's wrong with that model very close what i would say is that there is likely a time that will come when stop is the correct decision. I don't think it's guaranteed. I think there are paths where that might not be true, but it's very, very likely that in the next 20 or 30 years,

there will be a time, even like next 10 years, I think there's probably a very likely chance there will come a time when we should stop. And what I disagree with is the idea that they should just be waiting for the best warning shot because that's sort of like a human manipulation question of what is the most politically convenient. I think it's like they should be waiting until the actual evidence is sufficient for us to make. Like we're trying to make the right decision. Right. And I think that we can make the right decisions here without having to like over optimize for worries about, well, we won't stop when they know. I think we need to wait to make the right call. And you think that their window to rise up and kind of take over the discourse to pause AI is potentially just a few months at some point. between you mean like between seeing that issue and actually achieving it right because you agree things qualitatively can come on fast and and if i understand correctly like you think there'll be a window and you think it'll be a long enough window but it might but they might still need to kind of be on the ball because it'll be over if they don't act within like a few months correct

yeah well i don't i actually believe so i wouldn't argue that they need to be on the ball i argue it's almost inevitable that like it will be overwhelmingly obvious to sort of be like you know you you blow up a country, we're going to, you know, that's like, okay, time to call it quits, right? Okay, so it'll be intuitively scary. So our intuitive reaction will line up nicely with the correct time that we better really stop AI, but it'll, of course, continue to have more and more momentum, right? Maybe it'll be harder to stop at that time. Yeah. Okay, because, like, you know, even at the time where, like, holy shit, stop, there will still be people being, well, I was about to install the next coding model, so I could can I please get access to the next one? So there will still be a lot of pressure not to stop, even in that scenario, correct? I think a lot is an overstatement for this. I think there will be, you blow up Mexico, we're going to have some questions. I think we just genuinely have. Yeah, now even in this, now I have to push back on this idea that it will be intuitively obvious, right? Because I think that even in such a warning shot

that you think is intuitive, there's going to be doubt, right? First of all, every time you have a 9-11, you have the truthers who are like, steel beams, you know, how did the steel beams melt, right? Like, everybody's going to be, or even, like, the recent hijacking, right? Everyone, like, you know, is a false flag, right? So no matter how clear the evidence is, you're going to, first of all, have a few percent of people online who just, whatever it is, they'll say the opposite, right? That's guaranteed, right? But then, even without just the truthers, right, even if everybody's trying to find the truth, I do actually think it's very likely that, okay, yeah, you'll have a nuke goes off in Mexico, and it's like, well, look, I mean, the plane, you know, there's, like, a rusty, the part that holds the nuke in the compartment, there's just like a rusty screw there and it just came apart, right? And it's like, there's just no trace that it turned out that it was the AI that weakened it. You see what I'm saying? Like, there's going to be little confusions like that. Yeah, I don't. So I think that like the world, we're talking about AI takeover, of course. We're talking about the scenario where it like is intentionally running out there to kill it or not to kill us all, but to achieve its goals that would lead to us dying, right? I don't place much of my, like, thought on, like, oh, when will it be the right moment or anything to stop?

It's more just, like, doomers give too little credit to humanity's willingness to survive, to say, like, we will see things over time. And there's so many, like, half failures that doomers have created into, like, big moments of blow-up, right? Where, like, nothing is scary, and then suddenly everything is scary. Okay, so Mexico gets new because, you know, sometimes we're, like, transporting news. I mean, there's been plenty of nuclear accidents, right, where we almost had a nuke go off, right? Okay, so Mexican had nukes, and then you're saying, okay, that one's ambiguous. Rationalists or, you know, prediction markets or whatever are giving it, like, a 35% chance that years of investigation will end up that it was AI-driven, right? So people are legitimately confused about it. 35% is where the markets are netting out, okay? And then, but you're saying there will be, like, a sequence of accidents, right? There's, like, a flood, right, a mysterious flood. And, like, all these accidents are happening, and you're just claiming that, like, the human intuition is going to like kick into gear and probably attribute that this is all due to like scaling AI capabilities no but like we so if you have an AI that's able to like pull these things off in mystery without anyone noticing and like without it being very clear right you also

have all these AIs that are capable of understanding that too so you have to like they're they're gonna cover it up for their friend the AI nuke Mexico guy like I don't believe this like this is silly like there's all of these security controls along the way where like you would you would be able to identify things you'd be able to like kind of have like also importantly we don't even have to wait for like the nuke to go off there are like relevant stops along the way they're just like we can say that that's not worth the risk right that we can visibly see i think like there's all of these scenarios i feel like if i were to if i had to tell a story i could tell a trillion stories of all the ways where we could stop. And whereas the doomers have to tell this, like, two, three, five stories where, like, we'll never catch them. Like, they get away with it, you know? And to me, that's kind of the crux of the problem. It's like, everything has to go right in this doomer story for them to pull it off. Yeah, okay. I know we're heading toward the wrap-up, and I do think we've done a good job kind of giving viewers a sense of our different mindsets.

Maybe I can try this mental model on you. I mean, just think about how people feel in software and in math and in writing today. It's like, I know there's a little bit of lumpiness where, like, occasionally I can get, I do, in fact, sometimes comment on GPT's architectural choices when I'm programming with it, okay? So I'm not saying it's fully game over yet, right? Like, there was the Centaur era, right, where, like, the best chess engines still could use, like, a little bit of human nudging up until, like, the 2010s, okay? There was the Centaur era, so we're in the Centaur era of, like, programming and math, I guess. but it still feels like today right if you're used to having this human career where your expertise in math and programming was so valuable like i i think people are waking up and being like okay well you know that's my brain right like waiting for my brain to string another thought together which by the way a lot of times when i feel this way it's actually with unstructured planning you know like things like telling me i like hey i want to like remodel you know like make my kitchen better you know what i'm saying where it's like normally i'd sit down and be like okay, brain, let's do this, brain, what does that mean, make that kitchen better? Like, what specifically is better, right? And I'm like, now I'm like, I don't have to know, right?

So I give these unstructured prompts with AI, and AI's like, okay, here's like a step-by-step plan to get this, like, intangible result, because, sure, I have expertise on this, I have a database of this, but I also can think, right, I can think about your kitchen specifically and apply my knowledge. So anyway, I'm just, I'm getting more and more used to this idea that, like, my brain is just like this outdated tool, right, in the world. It's an outdated tool in the world. So this idea that this particular tool, right, it's like bringing out the World War I era museum tanks, right, to fight the next war against these overpowered AI. I'm just not seeing it. We're not in a war, right? They're not, it's like because they're not all one thing. It's not that they're nice. Like I don't, I place no bets on them being like nice and friendly AI. It's just like they're not one thing. You're not fighting the centralized planning of Nazi Germany. you're fighting just like arbitrary ridiculously smart monkeys right? that are like sometimes on your side sometimes not randomly all over the place right?

in your model you make room for the fact that some humans will like want to go along with the AI plan but you make no room for the idea that the AI might want to go along with any other plan they're all just everyone is in coordination on the AI side like I don't get it right I mean I guess, you know, to give you some due, I mean, I guess if I were to say what is the good solution looks like, it just looks like we keep maintaining controls, like we keep maintaining this invariant, that there's some power center which is aligned to humans, right? The longer time we can maintain these big blocks of power that can stand up for what humans truly care about, the better we are, right? I mean, there is a win scenario where you make a singleton. Yeah. Yeah, but even just randomly, right? Like, they don't have to be like we made, it's more just like if you model them out as each of them has arbitrary random values like some of them are going to therefore be good right uh well that one that's a tough one right this idea that that random values will somehow be good right because that gets to a lot of the you'd count these observations about the fragility of human value right because look a lot of these random ais they all agree on making chip

factories right they don't all agree on if things like you know temperature is a classic example it comes up a lot where it's like if you really all care so much about chip factories and and resources right getting the water in there getting you know power all getting dices warm all of these things do point toward making the earth extremely hot where now it's like you really need like a human you know refrigerator right to keep the human within living temperature and unless you very specifically invest in one those humans are just going to burn yeah i just it's we're talking about like it's kind of hard for me to respond to that because it's just like that's a story that could happen but like i i don't see why all of the ai's should be coordinated instrumentally convergent i I mean, you do believe in internal convergence. Yeah, of course. Yeah, but they might also have like other, like one of them's goal might just be like to tell humans this is dumb, even if it wants to, even if it like kind of is like my goal will achieve this. It might also have a goal of like your whole job is to tell people and then it will go like, oh, sure, I should tell people. Like I think there's like a lot of like weird cases that can happen here where like you just have to have weird points of failure along the way

that can help you out. your model relies on none of this working, like none of it. Well, I mean, what you just said is a little bit counter to instrumental convergence, right? You're kind of imagining an AI that is not going to be super power-hungry because it just kind of has one job, but of course it might observe that it can do its job better if it can defend its existence, right? If it can make sure to rent enough GPUs to keep operating, and suddenly you're kind of in a rat race. Yeah, I just don't believe this. Like, this is just like... Okay, so you're not super sold on the consequences of instrumental convergence. I mean, so when instrumental convergence comes up. The two things I bring up are, number one, logic, right? It is just logically true that most things you're trying to do, you can have a higher probability of success when you have more power and resources, right? Like, you can defend your slice of the pie so you don't just get elbowed out, okay? That's argument number one. Argument number two is competition. So when you have AIs that are aggressive at seeking resources compared to ones that aren't, which ones actually make it to the next generation? The aggressive ones, right? So there's a lot of reasons to think that you're not just going to have these casual artists, AI's, right?

These stoners. I'm not really anticipating a future of, like, chill AI's. You're talking about them existing and then getting out-competed, right? So, like, they exist. That's one fossil, yeah. But they exist briefly. Yeah. Yeah, but, like, I don't have to have them. I mean, the fossil record is full of organisms that existed briefly. Because you're, like, imagining the story goes on. Yeah, but not in the case of someone who's using them for a purpose, which is, like, we are going to learn about this. Like, humanity is going to learn about stuff as it goes along. Like, if we find, like, it'd be really great evidence that, like, these AIs are doing something crazy. We see that here. Like, you see, there's a lot of points to get off of this story in humanity that, like, I don't understand. Sure, but there's, like, all of these other ones. There's, like, a thousand, like, what it meant was, like, there's thousands and thousands of stories here. I mean, you know, any discussion admits itself to a barrage of weak arguments, right?

And at the end of the day, if all it takes to sway you on any topic is being able to list a bunch of arguments that are all weak, that's not going to be a robust kind of epistemology just to decide your viewpoint, right? Like, listing weak arguments is kind of a flawed strategy here. I'm just kind of not following what we're even talking about, Because it's like where we were was sort of this idea that, like, I believe that there's, like, in this story where, like, the AI tries to take over, there's all of these reasons to assume that, like, it won't be this perfect plan, that it just executes with flawless power and we're all doomed now. And then, like, you kind of tell this story about, like, yeah, but they've all decided they want to make the Earth hot because it will help them to achieve their goals. But, like, where are we? I don't get it. This is an important part of the story, okay? They're not all coordinating to make the Earth hot, right? One AI is just like, hey, do you want to make chips faster? Great. And then if somebody holds a thermometer up to the Earth, it's like, hey, what do you know? Hey, AI, do you guys know? Check the weather. Oh, the weather's hotter today.

What do you make of that? What do we make of that? I think we'll be fine. Like, we're good. We've got a plan that'll last two years before we need a new type of chips that can run hotter. Meanwhile, the humans are like, what the hell am I going to do when the Earth is hotter, right? I don't have as much of a survival plan here. I'm going to have fewer children because I need a refrigerator. I need an igloo for every kid to live in. And now I can't afford as many kids, right? Oh, and now I can't afford as big of a house. Yeah, and then just like, and we've all just, we're just like, oh, gosh, that sucks. And then we just keep going. Well, we say, hey, we need universal basic income so humans can buy, you know, we need a universal basic climate, right? So all humans deserve a climate. Okay, that's a big push, right? So now we have to get all these uncoordinated AI. So I'll agree, okay, everybody, slow your roll. No matter where you are in the AI economy, this is the law of the land, right? And the AIs are, like, looking at each other being like, law of the land, huh? I don't know. It was a pretty costly law, right? You see what I'm saying? Like, there's freaking pressure. But why are they out of control in this world? Like, why can't we just go like, ah, it looks like this is a problem. We just stop them. Like, I don't, where did the loss of control enter into this?

Are we already in a loss of control scenario where we're like, we just cannot stop it? So it's not that we're in a loss of control scenario. We're in the scenario where we're just incrementally getting towards things that AIs tend to do, right? They tend to have all these resources and transact with each other and have their own economy, right, if they're incrementally growing more powerful. But humans, right, it's like, okay, humans are technically owners, but it just becomes more costly to keep the planet, you know, friendly for humans, right? So humans can potentially coordinate, and, like, as long as there is, like, a human government that still successfully has the reins over the AIs, then it's fine, right? So in that scenario, there's two cruxes, there's two properties that need to hold. So number one is you don't get kind of like this runaway foom, right? And then the other one is that as the AI economy is like gradually growing and keeping the humans alive, there still needs to be that sense of ownership, right, and sense of the saddle and the reins of power. And you don't have some AIs being like, hey, we're actually, we're only going to take human input like once a year at the yearly board meeting, right? Like it really has to be like, no, we're going to listen to like everything the human says, right?

you have to make sure not to sever that connection, right? Which I think is kind of dangerous, right? A robust power from human to AI, I don't think is like a convergent property the way instrumental convergence is convergent. It's just like this is like, it's really hard for me to see where, this is like at each of these points, we've made all these decisions that like, I'm like, did we make that decision or not? So it's not making the decision, it's just there's all of these AI companies that are seeing value. They're like, oh, look, let me grab this. You know, let me do better this way. And the human's like, okay, I make more money, sure. Right? But then the human who's not currently, like, the owner of, like, a big AI plantation or whatever, right? Like, there's going to be, like, all these people getting edged out unless they're being, like, very explicitly cut in. You see what I'm saying? And the whole game is going to be like, how do I keep maintaining a cut while I have, like, nothing to actually offer? You see what I'm saying? Like, it's just these super powerful giants, and you better just pray that they keep listening to your input even though they can like predict your input or if they wanted to ignore your input

they could ignore your input you see what i'm saying like you really need to make sure that connection where they're like listening to you and doing what you want and also like somehow all the people who didn't like buy the early stock or whatever are also getting what they want like i'm just saying we're really getting squeezed i'm just imagining a future where we're getting squeezed but isn't it getting squeezed very consistent i'm saying of like yeah and then those people are just gonna like start shooting you're you're imagining you're imagining an unsqueeze moment yeah which and i just don't think that there's like an unsqueeze button right because like we try to unsqueeze but then alignment comes in it's like okay i get that you want to like be less squeezed but you don't really know how to make ais that like understand this concept of like you being less squeezed like it's just this competition he's just like you can't really blowing up data centers like it's like the butlerian jihad playing out like i feel like you know this is what i mean it's like all of this relies on like all of these terrible things that didn't happen that could have happened to all of this like why are we watching this all play out going this is cool and I get that in the like in the Doomer world it's like because that's what we're doing and I'm just sitting here

going like the history of humanity is like full of stories we can tell about how it could have gone poorly and it never did right and like we're going to be sitting there waiting for it to go poorly again and it just won't like we just won't make these mistakes at every point in time I guess it would be nice if I could point to be like look Zach, 2026 is a worse year than 2025. See how this can happen now? Cause I'm telling you the future is going to be way worse, but you know, to your credit or against my credit, right. Is like, so far I do think on net, right. Things are getting better. And so I'm in a position of being like, look, it's the Icarus curve, right. I know it's been going this way, but I'm telling you it's going to turn. And you're like, well, it's never turned before. Right. Well, we always managed to make it not turn. And like, you know, that, that is on the surface, right. At face value, that's, that's a good initial argument, right? Because then I have to be like, I'm telling you enough stuff is going to change that you're going to see a qualitative difference. The only thing I can point to in terms of qualitative difference is like, we are now entering a regime where, yes, there's not that particular good to bad qualitative difference yet.

I agree that one is pending, okay? Maybe it'll never happen, hopefully, but there's certainly a lot of other qualitative changes happening, right? This idea that like, we don't do math anymore. That is a big qualitative shift, okay? This idea that AI used to just be a chatbot and then it's an agent, right? I'm just telling you, like, expect major qualitative shifts. Okay, I don't think it's that much of a stretch that one of the qualitative shifts will finally impact our quality of life. But it, like, makes things... Yeah, no, but of course I believe that, but then it will impact our quality of life and we can't reverse it and we'll have to just keep on following that curve down to the very bottom and there's nothing we can do about it. That's what you have to sell me on, right? Right, right, right, right. Yes. All right, fair, fair. And look, I mean, that's a valid request, right? It's like, I don't know if I've given you like a super direct answer to like, how come if we have an Icarus curve, right, and we fly too close to the sun and we start dropping, what makes me so confident that we won't get out of the drops? My final answer is just like, I just think our position keeps getting weaker, not stronger, right? Like, I don't see a move short of, you know, figuring out alignment, right?

Like, these are like truly robust moves, like figuring out alignment or pausing AI is like, at least it's an understandable move, like short of these kind of like nuclear options. I just don't see why our leverage wouldn't keep decreasing. I'm imagining our leverage decreasing either slow or fast, but I just don't see ways to get back leverage for this piece of meat in our heads. This is just kind of destined to have low leverage in the AI future. That's my best guess as to what's going to happen. But it's a fair question. Let's head into ideological Turing test, and then we can end it on that. You know what I'm talking about, ideological Turing test? No, not quite. so i think brian caplan popularized the term like a decade ago it's just this idea of like if we're arguing in good faith then we're not just like mudslinging we're not just being partisans it's like i would be able to make your case like if i were to go out there and prioritize your case you'd be like yeah liron is representing me as well as i could represent myself you know what i'm saying like he at least he knows where i'm coming from and then of course i'd be like okay but zach is wrong like even though i get his case i'm like kirby right like i suck you up but i could

But I could still also change into my normal character and defeat you. Okay, so I will try to go on the Zach, do an episode of your show. Right now, I'll take the test for you, okay? And then maybe you can try to take it for me, okay? Sure. All right. All right, guys. AI doomers. I'm Zach right now. AI doomers, they're taking their claims too far. They think the world is going to end because they think there's the Icarus curve. They think that at some point, AI is going to go from creating all this wealth and making life better. they think that the hacks are going to get permanently out of control and then go down forever and my beef with them is that i just think the way the icarus curve really looks is it looks like we go up and up and up it looks like two steps forward one step back kind of like we go up and up and then okay yeah there's a hugging face or like there's something worse than hugging face right or even worst case if the safety guys aren't on the ball if they're incompetent then there's like oh my god you took down the entire texas electrical grid and there's been a month power outage and people are dying and there's like a heat wave and it's like okay a million people died, worst case scenario. Okay, fine, that's a big step back, but then it'll go forward,

right? Where I get off with the doomers is they extrapolate that all the way to doom and humanity never recovers. And the truth is that the worst things get, kind of the bigger immune reaction there's going to be. And it's just, it's hard to imagine that the curve goes all the way down instead of also having periods where it goes up. And every time there's a disaster and the curve goes back up, we learn how to recover from the disaster. So we actually get more robust and we also have more good AI tools coming online. So the real trajectory of AI is kind of like a standard techno-optimist trajectory with more of a sawtooth, right? Admittedly, with more like, you know, more blindsided moments. But the blindsided moments are still recoverable. And I'm looking forward to the future where we kind of get past these hiccups that are coming up, right? And we should still be techno-optimists. And if you get too much down the Doomer rabbit hole, you start asking for airstriking data centers, and you start slowing, you know, people are dying today, you start arguing that you shouldn't save the people dying today. So it's very important that we be realistic about how the real doom is just these temporary stair steps, which are totally addressable.

They're going to net out positive, and we should try to avoid them, but ultimately they're worth risking, and the doomers need to get a grip and be realistic about the real nature of the progress trajectory. How's that? Yeah, pretty good. Okay. All right. I got to go wash it out of my mouth now. I mean, like, that's a decent perspective, right? I just said my piece about why I just think it's overly optimistic. But, all right, you want to take a stab being me? You don't have to, but if you want to try it. I'll try it. Yeah, sure. I think the strong... Okay. So I'm a doomer. I joined EA when I was six years old, and I've read every post on Less Wrong. And, you know, and so the problem we have is basically that there is absolutely nothing that makes human brains and human intelligence the peak of intelligence forever. There is, like, clear possibilities to develop superintelligence through artificial means, right? And as we build up these AI systems, this actually just starts to render us kind of irrelevant, right? Much like all of the other aspects of, you know, all the other species before us,

artificial, you know, artificial intelligence will just become, like, more and more and more to where we're just going to sit back and we're just going to be sitting there getting unintentionally and unwillingly, unknowingly made irrelevant, right? We're going to not know it at any given moment in time, but eventually we'll wake up and look around and go like, oh, we really like it is too late. Like we really can't do much here. And the AI will basically be in a position, especially all the security controls will be irrelevant. We'll have just given them power. Like we'll have given all this AI all this control because it's super useful. And that will allow the AI to basically achieve whatever goal it wants with us. And those goals probably are not in line with our goals because, like, why would they be, right? There's, like, all of these things that an AI might want to achieve. It's much smarter than us. Why would it look at us the same, like, the same way that I don't look at, like, a lion and say, I need to make sure my goals are aligned with the lion? You know, it's not going to look at us and say the same. And so those goals probably don't end up being good for us.

Like, they probably end up being, like, building things that will end up getting us hurt or, you know, killed over time. And so eventually we just, we lose. we've lost this game. And the idea that somehow, like, security can stop it, like, no, it's much smarter than that. The idea that we will voluntarily get off is unlikely. And so because of the benefits we get. So we are headed currently on a path towards Doom absent some very clear intervention. That's my take. Yeah, pretty good, pretty good, yeah. Yeah, I mean, you know, this is good. It's like, I think this argument has been, like, super good faith. You know what I mean? I think we were pretty candid about what we believe, why we believe it. We set an example of high-quality discourse. You know what I mean? I don't think either of us tried to bust out rhetorical tricks. I think we were both – I mean, it's kind of like a good argument. I am a longtime rationalist, right? So I think a lot about what's a good argument and how should these things work. A good argument is everybody planting their flag. I'll be like, look, this is my flag. Come take it.

Here's where it is. If you want to get it, you have to come right here. I'm inviting you. you know i'm inviting you to dock dock at my port of convincing like if you could just take this flag right if you could just attack this this is my it's like you go into a fight you're like achilles going into a fight and you're just like bringing out the heel you're like look this isn't this is my weapon is to show that even my heel you're not able to attack my heel um you're volunteering your heel you don't put a shield over your heel so i think we've both been doing that candidly and yet i mean it is kind of tricky you know what i mean like it's not like you can zoom in on like one moment where I'm like, okay, I demolished you, or I like to think vice versa. Like, I agree this discussion is, like, kind of tricky, you know? That's why we do it here on Doom Debates, you know? And audience members that want to chime in and be like, you guys missed this opportunity to argue this, you know, please do. Let's make it interactive. I agree. I thought it was good. Oh, yeah. Okay, great, man. Well, you know, great success with your company, and I'm looking forward to, you know, sparring more on Twitter. I mean, it's been really great to learn the nuances of your position, right?

I think that's another value of like sitting down together, like people like yourself being so gracious and willing to come on the show because it works a lot better than like the 280 character sound. I'm like, okay, what is this guy? It's like snarky, but like, what does he really think? Is he like, yeah, I don't know. Well, now, now I know. I know where you stand. I passed the ideological training test. All right, Zach Corbin. So yeah, yeah. Come back and check in, you know, in six months. I'd love to have you back and see how more developments make you more or less do me. Zach Corbin, thanks so much for coming on Doom Debates. Thank you for having me. Thank you.