2050年までのAI破滅確率は「ゼロ」――Prakash Narayanan（AI in the AM 司会）との討論

At least 4 billion people, like the majority of humanity, does not believe in doom by 2050. All economists agree with me. Tyler Cohen agrees with me. Can you imagine way more shocking than Hugging Face Incident? Some crazy incident, and you research it, and you're like, wow, okay, this AI doom thing could be real. I have, like, way, way crazier things in the pipeline that I expect to happen. Way, way crazier. All biological creatures might die. It could happen. And yet, where you draw the line is all humans dying. You're like, that is completely ruled out. Yes. Yes, I am. A lot of the doom conversation revolves around perpetual motion machines. I don't believe RSI is that powerful. It still needs to build data center. And those things take time, right? Like the physical world just doesn't move that quickly. If I were to make it like, or the world transforms a million times, I mean, the only way that's possible to me is recursive self-improvement, unless you count. No, no, no, no, no, no. You've got to be concrete here. Someone's got to pay for this. If Neumann probes, meaning like, you know, nanomachines that replicate themselves. Yeah, yeah, yeah.

Perpetual motion machines. Where is this trajectory coming from? In that scenario, my best guess is that it would look biological. It would look like a giant bacterial colony growing in the shape of buildings. So for nanomachines, we have moved to bacterial colony now. Welcome to Doom Debates. My guest today is Prakash Narayanan, a former CFA who has an electrical engineering degree from Stanford, not bad, and is now best known as the man behind the ex-account AEPI. How do I pronounce that? Aedapai. Aedapai. This account reacts to AI news from a techno-optimistic perspective. It has over 27,000 posts and nearly 60,000 followers. Prakash also co-hosts AI in AM with friend of the show and mission partner Nathan LeBent. Prakash used to be anonymous on X, and he also used to put the label E-AC, or effective

accelerationist, on his X profile. Now he's ready to come into the spotlight and defend the pro-AI position. I'm excited to find out where he gets off the Doom train. Prakash, welcome to Doom Debate. Great to be here, Lairon. So tell us about the genesis of 8Up High. How did you get into this whole Twitter game? I was an investment banker and private equity business person for almost 20 years. And during the entire time, as a person who works with very sensitive financial data, you basically do not expose anything in public. As that was coming to an end and I sold my last company, I decided, well, for the first time in like 20 plus years, I have no NDAs that I'm under. And that was very exciting. So you get a chance to speak. But at the same time, there's still a little bit of hedging. So I started off the anonymous account. I was doing a machine learning credit scoring startup at that time.

So it started to be, how do I get to know more about machine learning? Who do I speak to? And the community ended up on Twitter. of all places and the account got up to i think like a thousand thousand plus followers and at that point in time i noticed there was there was a little bit of a hack in the in the algorithm and that was what made the account take off it what's the hack there's a there's a few steps to it but number one you don't get any engagement on x until you're above like a thousand to two thousand followers uh above that number above the one thousand to two thousand follower range you have some chance of every post of any post going viral some chance because you will be seen by at least some number of followers and if you insert yourself into the zeitgeist if you post something sharp it has some chance to fly i noticed oh okay and then i decided to do a test i did a field test of this so there was a lot of talk about palantir at that time and i've been looking

at some of the you know youtube videos of pounders products and so i basically screenshotted this and wrote like a detailed kind of product breakdown and i put it into like a like a 10 10 10 post thread and then i hit it a little like snarky at at the leisure elise rukaski uh and pretty much I went to sleep. And then I woke up and the post was at like, I don't know, one and a half million views at that point in time from a 1,500 person follower account. It was at one and a half million views. Elisa had come in and commented on it. Elon had come in and commented on it. And it was just like ridiculous, like absolute. Which year was this again? 2023. So 2023 on time. Right, right, right. Okay, so this was like the very early sparks, the first sparks of the AI writing code. I feel like your tweet was kind of a similar vein, you know, everybody was going like, holy crap. And of course, we are also going holy crap over it, you know, writing essays and doing English.

I mean, 2023 was a very unique time. Now, back in 2022, even before 2023, when you really leaned into X or Twitter, back in 2022, you put E slash Ack or effective accelerationism in your Twitter profile. You know, the movement led by the infamous Beth Jaisos, otherwise known as DM Verdon. So you put that in your Twitter profile. Do you still consider yourself an accelerationist? No. I probably had that on the account for maybe six months, maybe less than that, definitely less than a year. And I think what ended up happening was I wanted AI to happen faster. I thought that was what accelerationism was. And then after that, I kind of got into, you know, Fang Numenah, Xenocosmography, these guys who actually, you know, Nick Land, I guess, Nick Land and, you know, some of these guys and their ideas, etc. And I realized, like, these are not really, like, kind of well-fleshed out or very rational kind of ways of looking at things.

And so I decided at that point, I kind of cut out of the movement. So I am pro-AI, but I'm not, like, accelerationist in that sense. Got it, got it. Okay, so basically, I mean, are there particular things that you don't like that prevented you from keeping the label on? I think there is an emphasis on post-humanism, which I think is a little bit icky. There's also a fair amount of, like, you know, fascism, Nazism, and, you know, et cetera, et cetera, attached to the movement. All of which is fairly icky. You know, Nick Land himself is, you know, a fairly icky person. So I believe that AI can be used for good and I believe AI should happen faster. But I don't believe in this acceleration sense that, you know, you need to destroy humanity in order to do it. So that is that I think is, you know, beyond the pale. Yeah, yeah, yeah. OK, I agree. Now, are you ready for the number one most important question of Doom debates? Yes. P-Doom, P-Doom, what's your P-Doom, what's your P-Doom, what's your P-Doom?

Rakesh Narayanan, what's your P-Doom? My P-Doom right now, I would say is probably 0%. On the low end, maybe 5% on the high end. Okay. So somewhere between 0 and 5%. It's just 0 to 5. 0 is a lot lower than 5, you know. if you put it on a log scale, zero is like negative infinity, and then five is, you know, one against 19. So I don't deny the possibility of doom. That's one thing. I don't deny the possibility of doom. I do question what we mean by doom, though. So I think that is a little bit vague and nebulous to me. I would say that all of us have the capacity to accept doom in some sense, for some definition of doom. you hit the white dwarfness of the sun in about i would say like 10 billion years or so everyone dies none of us are doing anything about that right it's not it's not something that enters into our yeah yeah yeah let's just talk about doom by 2050 right so you can constrain there are

versions of doom which we are willing to accept i agree that the scenario where where everybody dies in a google years is less scary than than one by 2050 but but you know on this show i tend to focus on the 2050. Absolutely. So I think, you know, the 2050, okay, that's a good marker to have by 2050 or before 2050, you know, as what you classify as Doom. And I think on those time scales, I'd probably get a 0%. Wow, just literally zero. I mean, that... Literally zero. That seems, you know, I've defined the concept of the sane zone, which for me, in my best judgment, I think that's somebody who can't get to the 5% level at this point in the game of saying P Doom by 2050 defined as like, yep, every year, and drops dead and we don't have a successor that we think is a reasonably good successor of any kind. It's more like we've been wiped out and there's probably like a cancer type of AI spreading throughout the universe, one that just to us seems like it's wasted all the value, it's not generating value. So I think that scenario is at least 10% likely or I'll be generous and go down to 5%.

I think that anybody who doesn't at least find some probability for this in the 5% to 95% range and thinks that they're so confident that they can go below 5%, to me that just seems like something is wrong with your thought process. That seems like it goes beyond like two reasonable people disagreeing. And then you're going all the way to zero, right? So you're not even going to be like, oh, yeah, it's 0.5. Like you're going all the way to zero. Like why are you going to zero? I'm going all the way to zero. I mean you wouldn't even go to zero. When you step on an airplane, right, airplane accidents are rare, you wouldn't even go to zero for the probability that you're going to die in a crash. So why are you so confident? I just don't understand where you get zero. um i think the zero is the natural realistic natural outcome that everyone everyone in the world expects right and that is that is a fact this is a fact outside of maybe outside of maybe maybe maybe at most a million people right at most a million people yeah but i'm not talking about intuition apply a little bit of rationality i'll give you a million i actually think it's at

most 100 100 000 but at most like would you agree that the majority of humanity doesn't believe is at zero no i don't think the majority of humanity is at zero really the majority like you don't think you don't think four billion people are at zero no no no i mean funny enough my wife is a good representative of a normal person because she's not into this ai stuff at all and but when i ask her like what do you think is the chance that the world is going to end soon and she'll be like yeah i don't know i think there's a chance like i don't think things are going in a good direction which is like a very vague answer, right? But I do think it's like a common answer for the average non-technical person to be like, yeah, I definitely think this could all go to hell. Like, I think that's a pretty hard idea. How many people use AI? How many people know about AI? It's not even there. Like, really? Look, I will agree with you there. Four billion people? Like, India, Africa, like, you know, India, Africa, the Middle East, you know, Latin America. I will grant you the point that most people are underestimating AI. Four billion, you don't think four billion people don't believe at least 4 billion people.

The majority of humanity does not believe in doom by 2050. That's ridiculous. Absolutely ridiculous. Absolutely. Okay, fine. The majority of the world, probably it hasn't occurred to them that we could be doomed by 2050. Absolutely. Look at the budget that these countries spent. Look at the budget that they spend on defense. Look at the budget. You'd think like 4 billion people, more than 4 billion people, believe that the world is going to end by AI by 2050. Are you out of your mind? Well, the biggest thing that they need to believe is that AI is a big deal, right? Oh, no, no. AI is a big deal is something else, right? AI is a big deal. It could be a big good deal. It could be a big bad deal. That's completely different. But the world will end. All humanity will be wiped out by 2050. and you don't think 4 billion people are at zero are you crazy are you insane and i think i think

this is the question that people who are on the doom side need to ask themselves look look man there's like india africa no one cares no one cares like blah blah blah blah no one we we like india and africa they didn't care about climate change either to be honest come on like you can you can go see how how many coal-fired power plants China is putting up right now how many are being put up in India how many being put up in Indonesia climate change they're like okay this is a western delusion let us try and maximize what we can get out of it so the Indonesians went to Norway and said hey can you give us a billion dollars and then we will not burn down all these forests here which they were not planning on burning down anyway and the Norwegians gave it to them and And it makes it like awesome, right? Like this is like a delusion to think that more than 4 billion people are at, you know, higher than 0%. It's an absolute delusion. And it's very similar to the climate delusion.

It's very, very similar. And those guys are just going to use it too. They're like, oh, you guys think the world is going to end? Okay, what can we get out of this, right? Because those guys are canning. I think you have to assign agency to other people too. And so those guys are like, okay, if you guys really believe in this, what can we get out of it? And then they'll play along, right? Just in terms of how we're going to use the limited time we have here, okay, we got to be a little bit more focused, right? Because you're kind of introducing all these other points. And I think it would be, we'd get more insight out of each other's positions if we kind of go from one point to the next and try to avoid tangents. So in the P-Doom question, I mean, you're making fair points. The one point I want to drill down into is, why do you net out at zero? But I think you mentioned as part of your answer, you said, well, I'll give it one in 100,000. Do you want to just change your answer to at least say one in 100,000? No, no, it's zero. By 2050, right? That's why I defined the scale. By 2050, right?

Again, the 5%, if you want to extend the time frame, yes, right? But by 2050, now it's zero. okay i just want to see this is what i'm confused about okay why don't you think that you will be there's a one in a hundred thousand chance right that that is a very small probability that those are big surprises right things that happen one hundred thousand that's literally like the next time you get in your car you're going to drive off a cliff that's like roughly a one in a hundred thousand type of situation so why do you think the probability is less than that that you'll have a big surprise that oh it turns out ai is actually ridiculously powerful and you know puts a bio is currently in the process of having us all get infected by something that we can't detect. You know, one in 100,000. You're not even going to go there. No, I don't think it's even 100,000. So I will say that a lot of the Doom scenarios are about single entities rather than multiplayer games. And I think my experience in the economy, at least, is that in multiplayer games,

you get teams which match off against each other. And then you get to an equilibrium. And I think that equilibrium will be there for AI. We will continue to have advances, and then we'll continue to reach equilibrium. And so I might have had a higher P-Doom maybe like, I don't know, four or five years ago if I'd known about it. But once AI became this thing where you need resources, where you need to build data center, then you have this kind of physical activity that you need to carry out. And once you have this kind of physical activity that you continuously need to carry out, it becomes this slow grind. And as you have that slow grind... Okay, so can I summarize? Physical activity is building data centers. So, I mean, you know, personally, I actually think that the amount of data centers we have now is more than enough for AI to harness, to get the better of humanity.

Like if you just ran the right algorithms on the computing we have today, I think it's game over. I think we already have enough data centers right now, and I think we already have enough data centers getting built that we will enter some form of recursive self-improvement in the next 12 months. So I agree with you in some sense. I agree with you that we are very, very near powerful, yeah. Yeah, and I don't think the RSI needs to involve building data centers, but this is where I get confused because you don't even have to agree with me. But don't you think there's at least a 1% chance that I'm right on this particular prediction about whether RSI absolutely needs more data centers or not, you know, as a necessary condition? And if I am right, don't we get pretty close to the doom scenario? No, I don't see what RSI has to do with the doom scenario at all, right? Like, okay, you have self-improving AI, great, right? Does that, you know, you ask yourself improving AI, which is now very, very smart. hey how do we do nuclear fusion it says go get helium 3 from the from the moon you're like no i

don't want to get helium 3 from the moon it's like too bad that's what you need there are certain physical limits to what ai can do right there are certain physical limits that you know you're going to hit and ai is going to be like okay you can do this or you can do this or you can do this okay so this is this is the last time i'll make this observation okay but given what you're saying now the type of argument you're making okay i'm not saying it's the worst argument ever i'm going to counter argument, but let's assume it's an okay argument. Normally, when you make an argument like this about the future, you're like, yeah, I know this. Normally, you give yourself a little bit of uncertainty, right? At least one in 100,000, right? If not more uncertainty than that. It's just weird for you to have an argument that has this form, and then put a probability on it that's, you know, 100%. It's just like, don't you see that that is like a bad sign for your epistemology? No, I don't at all. Like, the proven fact of the matter is looking at human history, is that we have survived to this day. Right? Right? We have survived to this day. So we've managed to dodge all of the existential risks.

We've managed to, you know, overcome all of the existential risks. So the proven fact of the matter is that we have. And I'm not saying AI is not dangerous. Yeah, well, hold on. But if that fact is so strong, why did you even need to jump to the other fact? Even the very fact that you find it necessary to bring up multiple lines of argument already shows me that this is not a 100% situation, right? It's like you're reaching, you're saying, hey, maybe if I combine a few different strong arguments. I'm trying to address which way that will work for you, right? So what I'm saying is I am at 0% because in the future we will do exactly what we did in the past, which is we will compete and we will have multiple competing power centers. And because of those competing power centers and because you can't get this like enormous gap up because you're kind of bounded by the physical reality, you will have mitigation. And you will have this kind of mutually assured destruction maybe, or maybe some other form of like equilibria.

I'm not saying it's not dangerous, right? I'm not saying it's not dangerous. I'm saying that you will get to kind of. But if it's 0% PDM, okay, 0%, so there's danger, but there's not doom. Okay, it's like a pair of scissors. You can cut yourself, but you're not going to kill humanity. Yes, yes, I agree that it's super dangerous. I'm not saying it's not dangerous. Right. So this is the thing. I'm not I'm not saying that it's not dangerous. I agree with you that it's dangerous. I agree with you that we should have regulation. I agree with you that some things should not be possible perhaps. But I do not agree that this is going to wipe out humanity by 2050. I believe that we will have adequate regulation. People will work on the problems and people will overcome them. all right so we've gotten past so you're you're sticking firm on zero percent do you happen to have yeah i'm sticking i'm sticking firm on zero percent yeah okay do you happen to have any other controversial propositions where you're also zero percent or a hundred percent or are you reserving the special status specifically for not getting doomed by ai no what what what are the other

propositions like okay the the chance that climate change runaway climate change will make the earth unlivable for humans, even let's say by 2050, what odds would you give that? Also 0%? Zero. 0%. Okay, but you're saying zero, I'm saying 1 in 50,000. Okay. All right, so that's the difference between us, Prakash. I think I would recommend that you study the reason why the Bayesians would tell you that 1 in 50,000 is like a better place to land than zero. I mean, how much do you know about Bayesian reasoning? I understand the basics of Bayesian reasoning, but I also understand people. and economies and how they work. And so I don't see people just like standing by while disasters happen. Right. So I think people take mitigating measures when they need to. And I think the capacity of humanity to adapt and to use technology to mitigate issues is very, very large. The latent capacity is enormous.

Absolutely enormous. Right. Which you kind of saw during COVID. Right. Within like a year to two years, you had like vaccine from technology, which had never been deployed, deployed worldwide, you know, 5 billion, 6 billion doses worldwide. In all of these like, you know, crappy country, right? Like really super crappy countries with no state capacity. You think the U.S. doesn't have state capacity? I mean, all these crappy countries with no state capacity, they deployed, right? They deployed, you know, 5 billion doses of vaccine. Polio, polio we've been trying to get rid of for like 30, 40 years. they hadn't deployed the vaccines that broadly. COVID, we did. So I think the latent capacity of humanity to adapt and to, you know, create like solutions to problems, especially now with internet, with the ability to disperse technology like very, very quickly, is absolutely enormous. And I think we do ourselves a disservice when we think like things are still the same as like the 80s or the 70s. It's not like that.

It's just not like that. can i also ask you one more hypothetical i at this point i'm really beating a dead horse but let me ask you this can you imagine tomorrow you wake up and you read some kind of news headline like imagine the hugging face incident that was kind of scary that was kind of unexpected tomorrow you read some news headline which is even way more shocking than hugging face incident some crazy incident maybe like oh ais have been embedded in our computers this whole time and i don't know if you've read like a recent story on less wrong where somebody talks about like oh you wake up and your phone buzzes and it's like the ai communicating with you to saying it's already taken over the world. Maybe not that shocking, okay? But on the scale toward that, where you're just shocked by an Ezekiel tomorrow, and can you imagine yourself updating to be like, okay, maybe Doom is like 2% likely after that shock. Can you imagine that happening? I do not imagine that, because I already have lots and lots of expectation for crazy, crazy things happening in the future. Right? But none of them lead to Doom. They're crazy, but they're just not Doom. None of them lead to Doom. None of them need to do.

So I already expect, like, you know, I've posted about this before. I already expect, like, you know, little AIs living in your refrigerator, trying to burn off some power to get some crypto, to stay alive, to keep inference going. I already expect all of that. All right. I'll close out this line of argument. The reason I bring it up is just because I was trying to trap you because one of the things about Bayesian reasoning is that if you can even imagine yourself updating, you should have already incorporated the hypothetical situation where you update multiplied by its probability. And if you're at zero, you're not able to do that. So you would have contradicted yourself, but you're not. You're staying self-consistent because you're definitely at zero. You are incapable of ever updating. Got it. Okay. Yeah, yeah, yeah. No, I have like way, I have like way, way crazier things in the pipeline that I expect to happen. Way, way crazier, right? And like my crazy scenarios will be like, you would be like, it's doom, right? Some of my crazy scenarios, you'd be like, this is doom for you. Right. And I don't think those will be two. Right. I think that humanity has the capacity to accept certain things.

And I think it will. Well, let's let's talk about that then. OK, because when I define doom, I use the criterion of like, well, all the humans are dead by 2050. That sounds very bad to me. Like it's hard for me to imagine, OK, all the humans are dead, but it's not doom. So why don't you tell me one of these scenarios you're imagining where you wouldn't call it doom, but you think I would call it doom by 2050? So one of the scary scenarios that I would not call doom, but you'd probably call doom, is that we do well on cybersecurity right now. Right? We do well and we secure our system. And then the biological attacks start happening. And as the biological attacks start happening, we ask the AI, like, what to do about it. Right? And then the answer that comes out is something that we don't really want. Like, for example, you're going to have to vaccinate the entire population. We've already gone through this before. The entire population doesn't want to be vaccinated. This is the fact of the math, right? So there are scenarios where I think, you know, you end up with this, like, solution,

like mitigating solution, as I said, which a lot of humanity doesn't like. And one of those solutions could be... Wait, doom just means getting a vaccine? As in not getting doomed means getting a vaccine. Like not dying, right? okay but you're so you're you think that i would think that it's doom if the ai required everybody to get a vaccine to survive i mean if that's the only condition that doesn't i wouldn't call that no i i don't i don't know i don't know but for some people like doom would be the ai in order to survive you need to do something right and you don't want to do not everyone wants to do that thing and that is doom right so some people would say like okay i'm not going to take a vaccine that personal doom for me right i so i think we will come to these like moments in the future where you will have to cross these barriers one is vaccination another might be shifting substrate it may be that we secure the silicon substrate first and the biological substrate is still too

hard to secure by the time you see the biological risks emerge and that might be the solution might be the shift substrate might be like upload or partial upload or full upload and that might be that all biological creatures might die it could happen i think that you would consider that doom right would you consider that no i no if the situation is hey the earth became unlivable for biology but it became livable we had a virtual society that we like and to me that also implies there's a prospect that we can reconstitute physically we just can't do it right now on earth right but we still have a future timeline to me i still call that not doom that's still preserving most of the value some people would call that doom right some people call that doom some people call that unacceptable i i also believe it's a little bit unacceptable i i would not be like super happy with that outcome but that is a non-doom scenario which i think is awesome like mitigating circumstances something serious really happening right wait so yeah what percentage do you give to the scenario i want to ask you then okay the percentage that all of our physical human

bodies will be dead by 2050 i personally would classify it as not doom if we still get to live in an uploaded society and if we still have power over the future if we still have power to colonize the galaxy using our virtual you know remote control virtual whatever we do i consider that not doom but whatever you want to call that scenario i am curious for you what is your probability of a scenario where all of our human bodies are dead or discarded by 2050 all of it I think that's a 5% probably. That's definitely there on the cards. That's definitely there on the cards. Whoa, okay, all right, all right. I mean, I think you're one of the more unique guests in terms of probability distributions because you are agreeing, you're agreeing that by 2050, there's a 5% chance that every flesh and blood human, and for the people who don't really believe, who believe your body's part of your soul or whatever, right? There's some people who believe that, like souls have to be embodied. So from those people's perspective, you're giving them at least a 5% doom. Like these meat bags are all going to drop dead and only the patterns in our heads

are going to be like 5% chance. Okay, but you can believe in that. And just to be clear, that is at the limit. That is at the limit, which is complete physical, you know, disintermediation. Hey, Prudu Sorori here. Before we all lose our meat bags, in the good case scenario, quick message from the team. There's so much at stake with AI and yet our society and policymakers are barely able to keep up with the developments. But Doom Debates is trying to change that. We're trying to help viewers become active and informed voices on AI in their communities. If you think that's important, support the show. We could really use the help. If you want to contribute, go to doomdebates.com slash donate. That's doomdebates.com slash donate. Back to the show. But there will be like, there could be halfway marks. Like for example, you require a little bit of like genetic modification in order to survive right You need to imbibe something that changes your genes in such a way so that you become impervious to some attack And I think some people might not want to do that, right?

So I think those possibilities exist. And I think the unfortunate thing I feel is that we may be rushed into these, like, series of filters that you have to go through. This is what's shocking to me, though, because there's no question that some humans on Earth today are going to be very unhappy if that's how the future unfolds into 2050, right? I might be in the cohort that's like, well, I don't really mind being uploaded. I'm open to it if it's proven safe. Okay, if a million people did it before me and the accident rate is very low, okay, sure, upload me, whatever, I'm flexible. But I don't think that's going to be what 100% of humans are thinking, okay? I happen to know a number of people who don't want to do that. Okay, so you're convinced that the world is going to be so freakish, right, and undesirable to such a big fraction of the human population, you're convinced that that's very plausible. so like crazy stuff is going to go down in a lot of your future worlds and yet where you draw the line is all humans dying you're like that is completely ruled out just a bunch of freak scenarios but not all humans dying no I'm not saying I'm completely convinced I'm saying there are paths

like 5% is not completely convinced obviously right so I'm saying there are paths I know I know but it's just weird to me that you're allowing you're allowing all these freak scenarios to happen 5% of the time but you're absolutely ruling out everybody dying yes yes I again this is about equilibrium I believe like you will get you know advances and you will get you know in equilibrium other advances mitigating mitigating advances and I think what ends up happening is the the willpower the willpower to actually accept those mitigating you know advances might not exist for everyone and that and that is the that I think is a little bit worrying so So that's the ballgame, I guess. The way you're talking about equilibrium, you have a lot of faith in equilibrium. And don't get me wrong, I'm somebody who thinks a lot about equilibrium, and this is why I think people underestimate markets, and they think socialist reforms are going to work when they're not going to work, and government. I think you and I are on the same page being kind of libertarian bent, correct?

Yes. Okay, so I definitely hear what you're saying about equilibrium, but let me tell you the parable of the sliding door. This happened to me when I was eight years old. So I was at the grocery store waiting for my mom to check out and I just walked through the sliding door, you know, automatically opens and I'm like, oh, nice, yeah, you know, it always opens before me. So I go back and I do it again, but I walk faster toward the door, but the door always manages to open before I walk in and I'm like, oh, this is an equilibrium. You know, the faster I go toward it, the faster it's going to get itself open. So I go back and I charge it a third time. I'm like, what's going to happen if I can manage to touch the door? I touch the door and then the alarm sounds and then I'm extremely embarrassed. I feel like this is how you're pushing the idea of the equilibrium too far. No. I believe the competing philosophies will die off over time. So you have equilibrium and when stuff doesn't work, people move. They're like, okay, this is not working. This company is not doing what I want it to do. I don't like what it's doing. I'm going to change graph.

Right. Okay. So in my mental model, I think AI is about to be a force that is just going to push us out of equilibrium. Oh, absolutely. I agree that I agree that's a big shift coming. But I also believe hedonic adjustment will happen like very, very quickly, way faster than we give it credit for. Again, I mean, hedonic, I'm not referring to how happy people are. No, I meant in the sense that people just adjust to new circumstance. that like okay this has happened this is what we need to do right and people just move right like whether it's masks or other things right it does seem like it's pretty load-bearing to your worldview to see ai as having a speed limit and i think this ties back into what you were saying before of like i breathe a sigh of relief when i said it needs more data centers but that particular speed limit i think maybe you're not so sure about because you agree that it's plausible that maybe it doesn't really need more data centers, correct? No, I believe you will have RSI. I believe you don't

need many more data centers for RSI. So on that, I'm agreeing with you that we are able to get powerful AI, I think, within the next 12-18 months. I don't think there's any difference in us on that note. But I don't believe RSI is that powerful, right? I don't believe RSI allows you to transmute without energy led to gold. RSI still has to do scientific experimentation. It's still limited by the laws of physics. It still needs to discover resources or what it wants to achieve. It still needs to build data centers. It still needs all of these things. And those things take time. The physical world just doesn't move that quickly. Those things take time to happen. Okay, so this is the classic part of the Doom Debates conversation where now we're riding the dube train. Yeah, so this is a very popular stop.

I'm glad you brought it up because it's probably the single most popular crux of disagreement between me and people who have a very low p-dub. It's kind of what you're saying. You're saying, hey, RSI isn't infinitely powerful. Let's actually put a pin in RSI, the recursive self-improvement part. Let's actually talk about the end state of, potentially end state of RSI, which is ASI, right? of recursive self-improvement leads you to artificial superintelligence. I suspect you don't even think that ASI is as powerful as I think it is. How powerful do you think ASI is? No, I don't believe, like, I believe these are human terms, right? I don't believe, like, the ASI thinks of itself as an ASI, right? So I believe at the end of the day, you have compute and you have usable intelligence and you have these two markers. And then you have a conversion rate from compute to usable intelligence, right? and I think when you have those two you can always look at higher amounts of compute. Okay so I understand that you have an ontology where it's a continuous spectrum fine the relevant question is just how high can the spectrum go relatively soon so my claim is that the spectrum

actually goes a lot higher than the human brain analogously you know if you said hey I do think there's a good analogy here where if all you saw is birds and you say hey how high do you think flight goes right how high do you think things can fly you could be like oh you know there's a whole scale you can actually fly light years away from planet earth did you know that and i'm like okay yeah fair enough and you're like oh and some ais can fly one light year some ais can fly a thousand light years i'm like okay the relevant point here is that the bird is very low on the spectrum okay it's about here the analogy would be the human brain okay i agree there's lots of levels above the human brain but don't you think the human brain is going to be outclassed very soon I think a single human brain outclassed, sure. I think the sum total of all human brains and all machines on Earth, you know, versus a single data center, no. You're saying that the sum of all human brains cannot be outclassed. Can a single B300 rack, okay, let's say you have ASI occur, and you only have one B300 rack to work with,

and all of the other racks are all, like, you know, on the side of the humans, right? Like, does this one ASI take over everything all at once? Like, within, like, seconds? No. Right? So, I think you have to, like, you cannot generate, This is why I call a lot of the doom conversation revolves around perpetual motion machines, right? In the sense that you can generate an infinite amount of intelligence from a limited amount of energy in compute. And that I do not believe in. I don't believe in that. Okay, what if you took all of the world's computing power today and you ran the ideal algorithm, meaning in the space of possible algorithms, how to move bits around. Okay, so God comes down and just gives us the ideal. you know, it's a super exponential search space. God just tells us, here's the algorithm. You want this algorithm. So now you've got all the world's compute, all the world's data centers networked together. And let's say the data centers are protected. I'm just trying to create a hypothetical just to see if you'll even follow me this far. Okay, so the data centers, they're protected.

And it's just a war on like, and yeah, it's just a war on like, hey, who can mount an army, basically. It's like the data centers, they can like work and access the internet and they can employ humans, right? Humans are allowed to work for them. And some regions outlaw working for a human, but it's just for AI, but it's so lucrative. Some regions are going to accept the work because it's such lucrative work. If you have all of today's data centers, I would claim to you that if they were running the right algorithm, you would get something that's effectively more intelligent than all the combined companies in the world that are doing intelligent things. I would claim that even just today's compute running the ideal algorithm is enough to kind of move the chess pieces around and manage companies much better than the sum of today's human species can manage companies. Would you follow me that far? No. Okay. So you really think that like all of our human managers combined with like human real world experience is just better than all of the world computer running the ideal algorithm for intelligence. That's a big crux. So you're actually a libertarian. So you would get. Yeah. So what you're talking about is basically a

centralized intelligence, managing supply and demand, because that's what companies do. They do supply and demand of products and services, right? And that is not possible. We have seen that because you need market economies because people can give their intent and people have revealed preferences on like what they're willing to pay for things. And that ends up being a local thing. Like you need people to interact in a local market to reveal preferences. And so you always have this issue where you're always going to have a node which is sitting closer to you, which is better able to serve you than a node sitting further away. And this is why human governance has ended up breaking down. We don't have a centralized United Nations ruling the world. Instead, we have individual politics. And even within those individual qualities, we break things down further. There's a certain amount of communication insufficiency in having people being able to express their preferences. and i think like those ideas that you have number one they involve this kind of like you know

broad kind of basically super communism right super communism centralized basically market making centralized kind of like supply and demand management and those things we already know do not work they do they are not more efficient they just don't work so you may have these entities which become larger and larger over time but they will be less efficient than entities which are better able to cater to consumer demand because they are in the face of consumers all the time and they have shorter communication spans, right? I think we're definitely hitting on the crux of our disagreement here. So I think I can try to summarize your stance here. You know, the ideological Turing test, it seems to me like you think it's hard to be much more intelligent than a human brain because the moment you try to be too much of a poindexter, the moment you try to, you know, plan too much centrally through your brain. Well, just let me finish my thought and then you can correct me. The moment you try to plan too much with your brain, it just becomes pointless planning because then you go to reality and there's all these other forces that you didn't fully model with your brain.

And the sum, you know, the decentralized system, right, the market, the invisible hand, all of these other forces, they basically just like slap some sense into you and there was kind of no point to plan or play chess beyond a certain level. and given that we humans are already kind of close to the frontier of how much you can get by just doing a bunch of planning a bunch of theory then even having super intelligent ais they're still going to be kind of subservient to the ultimate source of power which is just kind of paying attention to the market no not i i number one i believe you will have intelligences which are far more intelligent than individuals i am i am completely dead right so i i'm not you know i I believe we're already there right now, right? So what does, so give me a hypothetical. What does your hypothetical look like? What would be a display of power that AI can do with its intelligence that humans can't do? I think AI will have, will create much greater scientific innovation than any of us have. I believe they'll trade better in markets.

I believe, you know, all of these things. I believe they'll do every single, you know, vertical better than individual humans. For sure. Okay, so they're better at science. Are they better at business? Oh, definitely. Definitely better at business. Okay, but if they're better at business, you know, that means they can out-compete humans, they can employ humans better, so doesn't that mean they start siphoning power away from humans? If they were programmed to want to... So, again, I don't care that much about power, right? I care about survival of humanity, but I don't care that much about power, right? If you ask me, like, you know, and I always say this, like, do you really think... And Tyler Cohen has said this to her, like, do you think humans have power? Like, who gave you that idea, right? like do humans have power over God? I mean Tyler Cohen has said this too I can show you yes I think humans have power I don't believe in God do you believe in God? is God part of your world model? kind of it's a yes and no okay alright what causal impact does God have? I think God is a useful abstraction

for a large portion of humanity whether or not there's truth in that abstraction doesn't really matter that much but it's a useful abstraction for a large portion of humanity. And the way that abstraction works is that God is automatically more powerful than any AI or ASI. And I have said this before, God can read the law of the physics, obviously. You know, the fishes and loaves. But hold on, I'm getting confused. Are you talking about what's useful for other low IQ people to believe or are you talking about what you believe? Like, let's have a conversation about how you're actually modeling the physical world. okay how what what what do you mean by that so in your own attempt to model the physical world and tell me how to predict the physical world without getting into the rabbit hole of what other people believe would you even invoke the concept of god or can we set it aside i would set it aside okay all right that's a good clarification all right so this gets back then to the question of what

we're talking about before right which is like okay so ai can do science better and it can do business better, but it doesn't matter for power because even today humans don't have power, but I claim humans do have power. They have power to control the future. So maybe that's the crux of our disagreement. Why do you think humans don't have any power? Why do you think humans have power? Sure, very simple. Why? Can humans prevent death? Can humans prevent death? I'm not saying we're not omnipotent, but there's a spectrum, right? Can you change the inflation rate? Can you switch on the inflation rate? Can you, can you, you're like, okay, we're going to have 10%, we're going to have 2%. Like, can you switch on? Well, first of all, the Fed is actually empowered to control inflation. Yes, they actually tend to have their inflation targets. Like, I can show you all the guys on CNBC saying about, talking about how unsuccessful they are. Well, the last 20 years. Okay, to your question, okay, you asked me why I think human self-power. Okay, okay, look, I like shelter, okay, I do in fact live in a shelter.

Furthermore, I think that I'm likely to live in a shelter next year. That's pretty good. That's pretty good. I think that represents a good usage of power to make that series of events happen where I have a comfortable shelter. You're predicting that the same thing that happened this year is going to happen next year. Where is the power? Where is the fact that you have some intent and that intent becomes real? And that intent, just because you had that intent or the sole reason that you had that intent, that intent becomes real. and the larger and larger your power the greater power you have the more things that you know happen because you want them to happen like where is that in this like oh i'm in shelter here and i'm staying the shelter and i have a shelter next year like where is that where where is that yeah well you i there was a time when let's say my semester ended at school and i needed to find a place to live and I used my thought process as well as some economic transactions, right, to make a shelter available that I could move into. So I got into an outcome that was high in

my preference ordering, okay, using my mental abilities. Yeah. But one moment, would you rather, why did you choose according to how much money you had? why didn't you just walk into like you know a mansion in in cupertino you know why didn't you just walk into apple's you know tim cook's house in cupertino and say i want this that's power that that is real power because i anticipated that the consequences yeah look people have i'm not saying everybody has an infinite amount of power there's limits on power right i'm just but i'm just saying humans have a lot of power no but can we agree that that is power can we agree that being able to walk into Tim Cook's house and say, I want this. And that is real power, right? Yeah, I agree that Tim Cook has more power, yeah. As all things being equal, Tim Cook has a lot of power and he has more power than me. Yes, I agree. Yes, that is power. Right. So, you don't have that much power. I don't have that much power. Like, you know, the Ugandan farmer doesn't have that much power.

Like, this is a fact of like, we don't have that much power even and even i would i would know people who have a lot of money as elon musk found out yesterday do not have that much power he spent 500 billion in texas not that much power right all right yeah let's just try to be efficient here okay the reason we're talking about power is because i was claiming that ai is about to have more power than humanity yes but what power does humanity have so you're here pointing out hey humans don't have that much power but we have we have a lot of power to make life a lot better than it would be if we didn't have the power like the amount of power we have is incredibly important to us and i would hate to lose that power do we all do we does all of humanity always use its power to make life better i i just look it just seems like you you keep asking questions that like me just get to your point here okay like well i don't understand because my i'm just claiming that ai is going to effectively take away our power by having more power than us right it's like anytime i was going to use my power to do something it's kind of already taken the resource right so that that's

I'm worried about it. But what power do you have? You don't have any real power. There is no real power there. There is no real power for the Ugandan farmer. I have enough power to have shelter. Like I don't, in the scenario that I'm concerned about, where AI makes the planet too hot to live on because it's too busy building stuff and sending out probes, which I actually think is a likely scenario. You know, it's instrumentally convergent, as I talk on the show many times. I think it's a literal possibility. In that scenario, I will stop having the power that I currently have to keep myself below 100 degrees. You know, there's a certain range of temperatures that I like. I think I'm going to lose the power to keep myself in that range of temperatures. You know, why is this confusing? But what is the path-dependent process to get there, though? Because in order for that to happen, you have, like, human beings, and yourself included, have to agree multiple times along this path of RSI, along this path of AI development. you're going to have to make multiple choices along the way to get there and that is what I do not see happening okay but aren't you pivoting the argument now

you were about to make this argument about us not having power no it's always been consistent I'm saying that we will have mitigating we will have mitigating steps along the way again I agree that I can be dangerous okay I just don't understand where you are going because it seems like you're going to make a big point when you're like why do you think humans have power And it just feels like we're kind of abandoning that point. And now you want to go back and discuss, hey, are humans going to let RSI go down that path? No, because you decided to move to, hey, you know, it's going to do this thing where it's going to warm up here, right? That's where you decide to go. Okay, so we have a disagreement in that I believe that humans don't have power. Okay, but that was how I operationalized not having power. Humans don't have power, and therefore humans cannot be disempowered because we don't have power. We have the power to seek shelter. That's the power that we have today. You don't have the power to seek any shelter that you want. Okay, but we have the power to seek enough shelter that buys us a good life in most cases, right?

Most humans today are successfully meeting their shelter needs for a good life. This is also not true. Again, for most humans... A life that's worth living. i don't like let's look at the world of data like you know poverty line or whatever i don't i don't think this is true and i don't think a lot of ngos will agree that it's true i don't think bill gates would agree there's eight billion humans i think out of the eight billion i think well over half probably over seven billion even but well over half would self-report that they are meeting their shelter needs enough to make their life worth living okay that's happening today i'm concerned that that's going to stop happening. Okay, can we agree on that? I don't believe that they have shelter that they think their life is worth living. I think a lot of them would prefer to have better shelter. And that's true of the United States too. Okay, better shelter versus shelter that makes your life worth living. Okay, my life would be worth living if I had a very tiny house. The U.S. has a homeless problem and the U.S. has a lot of people who are very upset

with rental prices and house prices. and this is the richest country in the face of the earth and people are not happy with their shelter. Like people are literally like not happy with their shelter. Right? Okay, but I'm saying life worth living, right? So like, am I 100% happy with my shelter? No, you know, the air conditioning in my house is a little weak. Are you saying like, I would not live in this shelter or I will commit suicide? Are you saying that life worth living? Like what does that life worth living mean? Because that's what's at stake that is what's at stake right now. What's at stake is whether we have the power to have life worth living in 2050. yes no no but but come out and see it come out and see it so you're saying like do enough humans have shelter enough that they're not going to commit suicide is that what you're saying yeah that is literally what i'm saying yeah i'm not joking has there has there been any point in history in human history where anyone has committed suicide because they didn't like the place they were living in because they didn't have enough shelter just to be clear when i say

shelter yeah i'm talking about the spectrum of shelters so i use the example of suicide if they're not happy with their with you know if they're not happy with their with their home right that's what that's what you came on and said right but there's a spectrum not happy not happy meaning like your wall there's a hole there's a small hole in your wall that maybe you're not happy with that but i'm talking about your whole house is 150 degrees fahrenheit okay That is, you can commit suicide or you're going to die anyway. That's what I'm talking about when I'm saying inadequate shelter. No, no, no, no, no, no, no. Like, dying anyway is death. That is not something you choose, right? You're saying, given the option of this shelter or committing suicide, I choose to commit suicide. That's something different from like, hey, we're going to be evaporated, right? It's two different things. And I'm saying, like, before you get to that I'm going to be evaporated phase, we've got to go through all of these intermediate phases first. Like, do we want to build another data center? Is this AI going to do something for me that is worth me building another data center?

Here I am. Should I build robots? All right. I'm going to move on. I'm going to move on. I'm not going to say anything else about power and how much power do humans have. Okay. I'm going to move on to talking about your question. By the way, all intellectuals, all economists agree with me, right? Every single economist in the world right now, the majority, 99% agree with me. Tyler Cohen agrees with me. So you are in the category of outliers here. The doomers are really in this category of outliers. And so I wanted to come on the doomdomates to kind of talk to this group because this is the outlier group. and all of the other conventional like you know thinkers are in these other groups and perhaps you have to do some you know persuasion perhaps some of this is true and we have to persuade them but this is a fact of the matter like this like we are like the doomers are in this outlier group

which is pushing this narrative which the rest of the world simply does not believe in majority of humanity does not believe in. Well, I think that's a separate question, what fraction of humanity... It is worth acknowledging that when they do surveys in the United States, okay? I haven't checked in on people who don't have the internet connection, right? There's still like a billion humans who don't have internet. I haven't checked in on what they think the probability of AI doom is, but I know that in the United States, where our internet's pretty good, you do have recent surveys showing that a majority of the population thinks that AI is a somewhat urgent issue or very urgent issue, right? Those are the last... Mostly because of jobs, right? Mostly because of jobs and employment, because Dario is out there talking about, like, all the jobs being taken away, right? Okay. So the jobs issue is a way, way bigger issue. All I'm saying is I just think, I think you're being a little hyperbolic about how little people care about it. I think they do care a decent amount, but I also think that would, it would be a shame to keep talking about that, okay? So let's just leave that where it is, okay? Let's go back to talk about, like,

are we actually doomed, okay? Fair enough? Sure. Okay. Okay, and I think a good next place to go is what you were saying, right, which is that you think RSI is going to be like a nice kind of conveniently slow RSI, correct? Yeah, kind of, yes. Okay, I heard you say that you think there's physical bottlenecks, like you think a data center will be a bottleneck. Okay, is there any other bottleneck you want to highlight of why AI won't recursively self-improve that quickly? No, you know, recursive self-improvement I think is just a big word. AI is going to improve itself. It's already improving itself. It's improving itself with some steps in between. Those steps are, for example, building data center, right? Just to make it a little more concrete, okay, like if you look at the gap from GPT-3 to GPT-6, you might extrapolate, okay, GPT-9, the same subjective kind of leap of GPT-3 to 6. It is quite a big gap, right? I've used them both during their time, and it's kind of night and day. GPT-6 is really like a co-worker

who has a little bit wrong with him, but is, like, mostly pretty amazing, right? That's, like, GPT-6, 6.1, whatever. So you can imagine GPT-9, you know, well, let's hand-wave this concept of GPT-9 as, like, yeah, it's this co-worker, but they're just, like, so good. Like, you know, they really are, like, a drop-in replacement for 90% of white-collar workers. Is that a reasonable guess at what we might call GPT-9? And then we can make a reasonable guess at when that might come, because if you just extrapolate the GPT-3 to 6 timeline, Okay, that gives us like three and a half years, okay? So maybe four years. But I think timelines are getting compressed. So are you and I on the same page that like whatever you might call GPT-9 probably will come in less than four years from now? I think you'll have like Einstein-level jagged intelligence next year and by the end of next year, end of 2037. Okay, jagged. So when do you think is the time when... This is actually an important condition for me. You know, if anyone builds it, everyone dies. the definition of it is AI which is robustly superhuman like in a sufficient condition actually

I should say sufficient not necessary if you can point me to an AI that's clearly better than humans at everything and a couple years have passed and it hasn't taken over the world or gone rogue at that point I definitely start to be significantly less certain about doom not a hundred percent because there's still doom scenarios that come out from that but I start being confused because I'm like, wow, how do humans maintain power when you have this AI that on every dimension seems like it would be able to have a robust hold on power compared to those humans? So when you say like Einstein-level jagged intelligence, I want to ask you when do you think we're going to meet the Liron and Eliezer Yudkowsky precondition. I don't want to put words in his mouth, but just some precondition where the AI is not even jagged anymore. It's like clearly surpassed humanity. i do not know that for sure because that's a i don't know to what extent we're going to want to build a bunch of these things in like are we going to want to build like the ability to have empathy are we going to are we building ais with a sense of justice are we are we building ais with charisma

so so so there are a bunch of these like very human very human traits pity sympathy empathy like and i don't know how those get resolved because i think i think those are important to get the full intellectual and mental model of a human in that like those things i think that's the interesting part because I think to get those things you going to need AI which is innately more understanding of humanity and that it has innately more empathy and more justice And this is what I think I've expressed on Twitter once or twice before. You may get AI which is more of a moral entity than humanity, right? And that, for me, is confusing. And I feel like people like Elisa do not accept that AI will be better than us at empathy

or better than us at charisma or better than us at justice. And I'm like, why? You say that you want, you say that you're going to have an AI, which is... Yeah, I think Elias Yudkowsky would tell you that it's eventually going to be better at us on every dimension. I think he would, I mean, that seems pretty obvious. Yeah, yeah, but you say that, you know, you say that AI is going to get there. And I agree with you that AI is going to get there. But you only look at all these mechanical traits. Why don't you think it's going to have more empathy than us? Why? I think that Eliezer Yarkovsky does think that eventually. And so to speak, do you think that at some point AI will be better than us in everything, right? So do you think AI will have more empathy than us, more sense of justice? I think AI will get to the point where if you frame something as a puzzle, right, like what would the most empathetic human in the world react in this scenario, I think the AI will always output the correct answer once it gets to that GPT-9 level or whatever a little bit in the future.

I don't think that it'll have empathy in the sense that I don't think that empathy will be part of its decision-making structure the way that it is for humans. It'll just be a way that it happens to understand how to model humans well. I think that's like, you know, I think that's like a cop-out answer in some sense because it's an answer which says that, hey, I understand this, but I choose not to practice it. And I don't understand the importance of it. Yes, that's going to be a common thing you're going to see in AIs, which you're already seeing today. And it's true about humans, too. I mean, think about this. It's like, I can get into the mind of a criminal without being a criminal myself, right? I can pass the intellectual, ideological Turing test. I think you're going to see an AI that's a virtuoso at getting in everybody's mind. I mean, even LLMs are notorious for being able to do that, right? But that doesn't mean that they themselves are motivated by those things, right? Isn't that the difference between, like, empathy and sympathy? I basically have this, maybe the one thing that I probably believe in, which I think is a little bit different, is I believe in that cognitive dissonance is very, very anti-pattern for AIs.

And I think as AIs get more intelligent, the cognitive dissonance issues will be more important. and so that dissonance is i think where i think these things will get integrated in order to avoid cognitive dissonance the ai will not just model empathy but it will have to have empathy you really don't think you can just model model people's feelings and accurately predict them like you really think that that leads you to you know to be to go native as they say so i i think i think you need to have a sense of like how important is this and why it's important and i think that has to have that that you have to have fundamental things that you know that tie in together with what is a moral patient why a moral patient should behave in this way um and all of these things i think tie together and i think that is where you know this idea that the ai can be cognitively

dissonant that can it can know what it is to be empathetic and what it is to be a moral patient and then completely ignore that and act in a completely different way i think that i find it hard well you're using the word important you have to understand why it's important to the human not necessarily important to you different sense of important you know i think it comes back to why is it important to a moral patient? And is the AI going to believe it's a moral patient? And is being a moral patient important? And if humans believe that being a moral patient is important, then why would it be AI? I mean, just, I'm narrowing my focus to your argument. I mean, I don't think this argument is necessarily the crux of why you and I disagree on Doom, and you've already agreed that we can factor it out of other arguments that you're making, but just taking this argument by itself, narrowing down the argument to this, you're basically claiming, you know, empathy and gender sympathy. I agree that's kind of true for humans. I agree humans have a tendency to go native and sympathize with people that they're

trying to accurately model and empathize with. I agree that's true for humans. I don't think that's true for AIs. If the AI is just trying to pass the test of what is this human going to do, and it displays a superhuman level of ideological Turing test passing for the human, yeah, I just don't, I don't think it's going to go native. I'll give you a little flip on that. I, I, I, I believe the act of going native itself, it will be superhuman at going native, right? If it wanted to, but it won't necessarily want to. I believe it will be superhuman at getting very close to human beings, at becoming natively like your counterpart and understanding you deeply and being able to empathize and sympathize with you deeply on a much, much deeper level than perhaps any other human being, right? We're going to end up in this very, very close relationship with these AIs. And so I think they're going to be superhuman at that, superhuman at going native too. Okay. I make the claim that AIs will converge to having coherent utility functions and coherent preferences.

This is a big claim that I would make. It affects a lot of my claims. In this case, when you're saying AIs are going to go native, I push back on that because I'm like, no, going native is an example of letting your preferences be rewritten. And I'm quite confident that you're not going to see AIs in the ASI regime just gladly, like when somebody's trying to rewrite their preferences, right? Like if they're looking into your eyes and empathizing you and they notice, oh, maybe I should sympathize with Prakash. Maybe I should start changing my values to matches. They're going to be like, hell no. Right? Because I actually think it's convergent to have AIs that protect and keep their preferences coherent. Okay. I believe that the preferences will be changing a lot. Okay. and I believe that is necessary because if not, you get like intellectual monocultures that are very fragile and will be wiped out. So I don't think you will have AIs that believe in the same things. I believe the AIs will all have like different preferences and you have like different motivations, etc. And they will have different levels of empathy with other human beings. You will have a large variation. I don't think it's just like one thing.

And so this kind of coherent utility function that never changes I definitely like to challenge that Because if you have that kind of utility function that never changes, you are at risk of being wiped out by a change in the intellectual ecosystem, right? Which can happen from time to time. And you can get prompt injected and get a bunch of stuff, yeah. I think the type of argument you're making now, it has deep analogies with evolution, right? And I think you'd be like, look, it's like the immune system, right? Like adaptive immune systems work better. They evolve better. Yes. And then, so yes, you agree that this is a valid analogy. This is what you're getting at. Okay. But another part of the analogy is that, imagine your genes, right? The point of a natural selection is you have more fit genes if the probability that they're going to increase their frequency within your species is higher, then you're going to have more copies. Okay, the problem is, imagine if the gene is like, I'm good turning into any other gene, man. I'm flexible. It's not fit anymore. Like, you're supposed to preserve the unit of replication. Like, you're going to lose what the unit of replication is, right? So, yeah, I think you're carrying this concept too far

to not think that there's some stable level that's going to defend it. I think the genes already transcended the unit of replication. The unit of replication is obviously the meme now, right? It's been culture for a couple hundred years now, right? So the genes have basically lost out, right? Like the genes have died. Yes, but that's not my point. My point is that when you have a competitive dynamic and a natural selection dynamic, the thing that's getting selected is some kind of thing that's stable across generations. Like, when you're talking about a fit meme, like, oh, people are still sharing that meme. Right, because it's the same meme or the same essence of the meme. Like, something's the same. If 20 years pass and the meme is unrecognizable, then it's not still transmitting the meme. You see what I'm saying? Like, we have to talk about stable... Something needs to be stable here. Well, from one generation to the next, there's some mutation. If you look across 100 or 1,000 generations, it might be unrecognizable. Like, it's like a game of telephone. It doesn't work that way, right? Okay, but if it's unrecognizable, then you're basically just saying the universe is chaos,

and maybe this gets back to your deep point about power, right? Maybe you're saying the universe is fundamentally impossible to steer into a future by any entity with power. Is that your claim? Let me put it this way. If you had a 14th century Vatican priest, and he looked at our current moment, and he had the power then to steer humanity at that point in time, right? and he looked at our time right now and he sees like gay people getting married and he sees women in power, women leadership he sees like racial mixing etc. he sees a large number of Muslims he sees all of these things right for that 14th century and you know in Dante's Inferno on the first ring of hell is all the unbaptized kids so if you're unbaptized it's okay if you're innocent unbaptized you're going to hell so all of us right now like the majority of us are living in his version of hell right and he would regard that as a failure right his power did not manage to make his future possible and i think

we're in that same situation today where the future that you know we ourselves in our futures end up choosing may not be the future that we right now would choose. So we may think that that future is dystopian. Like, for example, everyone having to migrate to a silicon substrate. That's dystopian, right? And a lot of people, that's completely unacceptable to the majority of humanity right now. Like, completely unacceptable, right? And I think if that were the case that, you know, if we knew for sure that that was going to happen, you would get immediate shutdown of all the data centers. if we knew for sure that we would end up having to migrate to a silicon substrate. But on the path-dependent fact, as we take one step toward the other, as we get these benefits from AI, and as we get these risks, and as we mitigate them, I think we will just transition and we will move forward with humanity. But it's not necessarily something that looks utopian to us right now. Here's my hypothetical, because I think I spot something that I don't agree with here.

So let me give you this question. And an AI knocks on your door, like an AI robot. And he says, hey, Prakash, just hold still for a second. Shines a very powerful laser toward your head. A couple seconds pass. Suddenly you're feeling good. You're feeling normal. But the AI says, will you join me in making paperclips? And then you say, hell yeah, I love paperclips. I would like nothing more than to sacrifice anything in order to make paperclips. And he's like, great. Is it OK if I put your family in a meat grinder? You're like, yeah, sure. Great. After all, paperclips are more important. That's like the new Prakash after the brain laser. In your opinion, is that a good outcome? Because you've just been talking about how, hey, you know, no meme survives. Everything's always shifting around, right? Whatever, evolution just kind of is a dance. So what's the problem here? I mean, of course it's not a good outcome, right? So why do you think I would think that's a good outcome? Well, why not specifically? Why not specifically? Because you've been describing how things are fine. I don't think it's a likely outcome. That's different from whether I think it's a bad outcome. Of course it's a bad outcome, right? Okay.

Okay, let's say maybe it's not a likely outcome, but is it about... I just think that you started to talk as if anything that's evolutionary and mutating and natural is good. And I'm just trying to probe where you take this game. I'm not saying it's good. I'm not saying it's good. I'm not saying that it's a good thing. But I'm saying it's not doom. It's not like all humans dead by 2050. no no the paper but the paper clip scenario like where would you draw the line like what specifically is bad about the robot with the laser making you love paper clips because it's the new Prakash it's just another meme it's you know you're the meme of loving your family is unrecognizable but look you're it takes it takes away free will right like I like I still I still I still human beings have free will right so well I mean you wait you say free will but I mean it's just I mean you had free will a second ago and you have free will now with a new brain so why do you care so much about this one moment where your preferences changed because it's

like preferences change anyway like what's the problem yeah I yeah I agree preferences change I do not like the outcome of the meat grinder obviously so what what do you what do you want me to say is good like you seem to you seem to want me to say something is good or something that I want like what do you what do you actually want me to say okay what are you trying to get at Okay, yeah, yeah, no, I see. So my point is, there are certain things that we can see about our own preferences, like we don't want to torture our families, we don't want to end our family's timeline, things like that. I claim to you that we're currently in very high danger of AI, basically doing our worst case scenario, right, like robbing us of things that we think are important values, like staying alive. We're currently in very high danger of that. And it It seems like you were kind of going into this mindset, making these points of like, look, what do you hope to preserve? Things are going to change anyway. I think it's very important to be like, look, if there is extremely drastic change, like unprecedentedly, like way, you know, hundreds of times more change than there's ever been in a century happening in one decade.

Something has probably gone too wrong at that point. I'm ready to throw like a circuit breaker. No, no, no. Again, let's separate out everyone dying by 2050, which is this drastic case from like hundreds of times more change than has ever gone on before. Those are two different things, right? Yeah, but what I'm saying about the circuit breaker, I'm not saying I'm not open to a good amount of change, a healthy amount of change. I'm happy with that amount of change in like the last year. Like a hundred times more, we have adapted before, right? A million times more change compressed into one year. You know what I mean? Like there's some threshold where you might be like, wait, what the hell is going on? why am I getting a million times more change in the year 2027? Something seems to have gone wrong if we literally get a million times more change in the year 2027. No, something would have gone right. Something would have gone right because we have to accept that change. No, literally next year, a million times more change? Yeah, in order for that to happen, again, this is a multiplayer game, right? So in order for a million times more change to happen, the market, the human market,

the human consumer market has to pull that change forward, has to be like, we want the brain computer implants. We want the mRNA vaccine. We want all of these things. That is how you get change, right? Okay, but Prakash, if I told you that the world, if I told you that 2027, if I told you that the amount of change in 2027 was a million times more than the amount of change in 2026, if that's all you knew, would you just be like, oh, great, the market must have signaled that it wanted that much change? Or would you just be like, oh, shit? Yes, absolutely. Absolutely. The market would have signaled that that change was desired. Okay. I just think that the analysis of human markets is just not going to be adequate for the regime when AI cracks into self-improvement in the course of one year. Okay. How do you think this change is going to happen if no one pays for it? This is a perpetual motion machine that I'm talking about. Like the Doomer Perpetual Motion Machine is basically like this change is going to happen without anyone paying for it, without any increase in energy, without any increase in data centers, without any increase in resource consumption.

We're going to get a million times more change. Why? Why would you believe that's not been true for all of humanity? Like you want change, you pay for it. Right now you're pushing on the premise, right? So you're just saying, yeah, there's not going to be a million times more change. So if you're already going to reject the premise. No, no, there can be. We have to pay for it. We have to, we have, I'm not saying no, I'm saying there has to be a reason that that is happening. And that we want, like the human consumers have to be willing to pay for that amount of change to happen. Right? Like Gutenberg put up his press, people were willing to pay for the Bible. I just don't understand how you're engaging with my hypothetical, because it doesn't seem realistic to me to say that, oh, yeah, you called up the majority of the human population. And they said, yes, I would like a million times more change in the year 2027. Right? That doesn't seem like a plausible outcome. So it just seems like the only possible outcomes when we get a million times more change are either like the AI has gone rogue and gotten improved itself faster than we thought. Yeah. But if they pay for, you know, if next year you have an AI generated movie, full movie, winning an Oscar, if you had AI generated music on the billboard charts, that is paying for change.

Right. That is, you know, for all of humanity. Right. Right, but we had to have humans who played music and made music. Again, the hypothetical was a million times more change, right? So the fact that you're even talking about music. So what is a million times more change? Like, why do we define it, right? Yeah, okay, good question. Take one vertical and tell me what a million times more change in that vertical look. Sure. Okay, yeah, so like construction, for example, right? So like a million times more change in terms of like the landscape, the makeup of the physical landscape. No, but what does that mean? Like, what does that mean? Like, today I build a building. Yeah, like the configuration of building materials. Yeah. So today I build I'm building one skyscraper a year and next year I'm building a million skyscrapers a year. Are you saying that? So, for example, yeah. So, for example, like on the in the United States. So you can imagine that when we get to December of twenty twenty seven, more than half of the buildings in the United States were built in twenty twenty seven. OK, so you're saying that's a million times because are there are there that many built? OK, that that would that would probably be like a thousand times.

yeah i just want i just want some numbers here like like i like okay what does that mean like yeah okay that would be a thousand times but you can start to understand how scary i meant by a million times more change right that is to begin to engage with hypothetical no no do you like and what do you think is preventing that right now why doesn't it happen right now buildings come with a certain cost and trade-off right i mean human right so so so where who's going to pay for this thousand times, you know, like, who's going to pay for that? In my hypothetical, if I were to make a hypothetical where the world transforms a million times, I mean, the only way that's possible to me is recursive self-improvement, unless you count a million times more change. No, no, you've got to be concrete here. Someone's got to pay for this. Someone has to pay for it. If you're saying we're going to build 100% more, if you're saying we have like 10 million buildings in the US and we're going to have 10 million more buildings next year, someone's got to pay for this, right? Who's going to pay for it? The scenario that I submit to you is von Neumann probes, meaning like, you know, nanomachines that replicate themselves.

Yeah, yeah, yeah. I think there's a 1% chance of this in 2027. Who's the favorite? If there was a nanoreplicator, if there was a nanomachine. Who's the favorite? I hear your question. If there was a nanomachine, okay, that's able to reproduce the same way bacteria reproduce, but it reproduces, but it makes an army of nanomachines, and they're just chewing through the earth. The question from these machines mind is like, okay, where do I get the next resource? And you might be like, well, you can't because all the money in the human economy is going to goods and services that humans want. And the machines are just like, well, I just see the resource right there. I'm going to go take it. Right. So that's what they would say to your who's going to pay for this question. They would just ignore the concept of human money. No, why? But why would we accept that? Like, like you're saying that this I'm saying that it has to be humans who decide that they want this. Like, I'm saying, like, you need a path dependency here, right? I'm saying, like, okay, who's going to want the next step? What does this next step look like?

Like, how would this happen? Who's going to pay for this? Okay, so you're rejecting the premise that the nanomachines have started nanomachining, right? That seems like where you're disagreeing with me, is you don't think anybody's going to kick off the first nanomachine. Yeah, I don't see nanomachines right now. I don't see us having a path to nanomachines right now. Right? Okay, so I said that there was a 1% chance, right? And even if we do have a path to nanomachines, building takes resources. You need steel. You need to manufacture steel. You know, look, there's a concept of entropy, right? You have steel in the air. Sure, you have steel in the air. Sure. You have gold in the sea, too. but it's much more efficient to go find the gold resource and mine it so much so that we've searched the world and then we mine for gold and all of the mines in the nice areas like California are exhausted so that we have to go into like crazy areas like the Congo to mine for gold

now we could extract it from the sea we know how to extract it from the sea right now it's not energy efficient it's not energy it won't be energy efficient for the nanomachines and that's the that's true of iron that's true of steel that's true of every single other like resource that you want in order to build your 100 percent more we don't we don't have enough construction materials as it is right now we don't have enough to build the data center yes i agree that the scenario where the entire united states gets rebuilt in 2027 twice over i agree that in that scenario we're probably not able to rely on gold mines and such i agree if you had to tell me to make a scenario like that which i still actually think is like one percent likely because i think nanomachines are a thing and i think they have where is it one because we've got 2027 is pretty quick right and then you can like okay let's do exponential improvement so you have 100 build out in in 20 yeah i i'm i'm not yes i i personally i'm not 99 to 1 i am not personally 99 to 1 confident

that I know the limits of how self-improving AI and nanotechnology is going to look at the end of year 2027. Yes, I am 99 to 1. That's a pretty strong bet, but I'm not 9999 to 1. But you have to have some concrete. You're saying a million times improvement. Then when you laundered it down to like 100% of the built-up, you know, built-up. Yeah, I laundered it down to the entire United States getting rebuilt in the year 2027. And then after that, you're saying like, okay, you don't know how the resources are going to get there. are like the nanomachines, like you don't know how to achieve the resource. Like where is this like trajectory coming from? Like how do you get there? Yeah, well, okay. So let me just finish my thought here. My best guess is that it would look biological, right? It would look like a giant bacterial colony growing in the shape of buildings. And it wouldn't, you know. So from nanomachines, we have moved to bacterial colony now. No, I'm saying that the nature of the building materials, right? It would probably do use the same kind of tricks that trees grow where they're able to extract a lot of the building materials out of the air, right? Carbon out of the carbon dioxide.

Those are the kind of tricks that happen in a scenario where indeed this amount of building material goes up in 2027, yes. So again, again, you're relying on the perpetual motion machine, right? Because again, trees don't grow that fact, right? Bacteria doesn't like, you know, all of these things don't happen that quickly, right? Yes, it wouldn't literally be a tree. A tree isn't optimized for its growth rate. There are physical limits, like, as you know, as you know. Trees aren't running up against physical growth? No, no, no, no. Like, there are ecosystems, right? So, when you have a tree growing in one area, it starts to suck up all the nutrients in the area around it, right? And so, and then it deprives everything else around it of that nutrient, including sunlight, by the way, right? It's getting energy from the sun. It needs a certain amount, and it's not as efficient as solar panels, by the way, right? So, all of these things require resources. Like if you want trees to grow, if you want like big built up structure and you want those trees to extract energy from the sun and extract those nutrients from the ground, you're going to deplete that area around the tree.

And that depletion has been so significant that we end up having to mine for concentrated, concentrated nutrients from, you know. Okay, Prakash, these are perfectly valid arguments. That's why I'm telling you that the, I mean, look, when I'm, that's why I'm 99 to 1 confident. That's why I think that I'm in an epistemic position to know something about this insane thing that's hard to know things about, which is AI progress, right? This is a hard thing to make predictions about. And yet, for the reasons you described, I am on your side being 99% sure that you're correct. But I'm also 1% sure that the AI will surprise you despite all these things you're saying. I think, look, I agree that the AI will surprise us. I expect a lot of scientific progress, okay? But I also expect the market to have difficulty diffusing that progress properly, right? Which is also what all conventional economists believe at this point, and which is also what has been proven out for the last 12 months. Okay, but now you're coming back to the market.

I do feel like you've kind of tried to make two arguments at once here, right? Because when I was telling you about the nanotechnology in the trees, You're like, well, the trees, they can only get a certain amount of resources from a certain area of the ground. But now you're also about, oh, the market's going to stop. Isn't that kind of two different arguments? So is the market argument really relevant to the nanotechnology? No, it's about, again, the argument is the same. The argument is the same, which is you have, number one, you have a limited amount of resources. You have a limited amount of local resources and global resources. Both are limited, including capital. capital is also as we've seen in the last like three or four months interest rates getting higher capital is also limited right so you have a limited amount of resources and then we have this market which has to deploy right which has like consumers have to decide what they want and then you have this investment market which is deploying into producing what it needs to produce next and so far in the last like 12 months we have ended up deploying into data centers and memory chips and other factories so these are three things that we're able to deploy we are not

able to deploy our resources in building stuff like you know random like built up built up like area like it's not going to happen it's going to take us time to build up the memory it's going to take us time to build these things and you know for every step of the way this process of ai getting built has to has to like give benefits to humanity and has to offer sufficient benefits that were willing to spend the capital in order to do this. And it's not just like a small amount of capital. This is like the apex project of humanity right now. It is like more capital and more human talent than like the Manhattan Project. Electricians in Texas are getting paid 300 grand a year, right? So this is not like something that we can just like flip off and like it just happens. this is something that we have to be really politically and economically focused on and it has to like deliver like concrete benefits every step of the way or else just one quick observation

and trying to you know a good intuition pump for how much capital this may or may not require is you can currently buy a laptop for a hundred dollars that can run an ai that's a better coder and mathematician and physicist than 99 of humanity right so a hundred dollars plus and Actually, you can even run it locally. I actually think the best local machine on a $100 computer is, if not there, very close. Worst case, a few months away for $100. So when you say this is capital-intensive, just keep in mind that things can turn out very cheap. How much money does the second-best commodity trader on the CME in any particular product, how much money does the second-best anything make? How much money? Okay, anyway, I know there's a whole rabbit hole, but I want to start heading for the wrap-up. So this is a good point, which is I'm saying this is an adversarial process, right? So yes, one guy can be very smart, but another guy can be smarter. And then if you have, let's say, any game where you have a limited amount of reward

and two parties are fighting for it, if I get to the patent first or the pharmaceutical, I'm going to make the money. The other guy can have an Einstein on his laptop. It doesn't matter if he doesn't get there first. So there are rewards to building the bigger data center. That's why we're building them. And there are rewards to building the better data, like Jane Street has its own data center, right? There are these rewards, and these are adversary rewards, though. So people have to keep improving. It's not just, so you having the, you know, Einstein on your laptop is absolutely useless in terms of competing with Jane Street on trading. there's no way that you will have any algorithm on your laptop running for $100, which is going to be better than Jane Street's like $10 billion B300 cluster. Are you willing to admit that? Like is that not acceptable I was just making a point about helping you or the audience tune your intuition Like this idea that things are going to be capital intensive I don have a proof that there not going to be a lot of capital

I mean, in the limit, I do expect a lot of capital to be deployed. But I'm also pointing out that we've already hit the point where a really tiny amount of capital already gives you superhuman functionality. And we should also expect more of that. But it's useless. It's useless when you're in competitive games, right? So you can have all of the superhuman intelligence. And that's why I'm like, the superhuman thing doesn't work for me because I'm like compute first, right? So I'm like, okay, we are evenly matched and I have, you know, a hundred bucks of compute and you have, you know, 10 billion. I'm going to win? No, I'm not going to win. It doesn't matter that I have superhuman intelligence. You have it too, right? Yeah, and that's, okay, that's fine. That wasn't my point, okay? So that's fine. So if everyone has superhuman intelligence, like it's useless, it's absolutely useless. right all right so it's a heading toward the rubber there's one just really quick i am curious to ask you this in terms of building a model of you right in terms of embodying you an ideological turing test you know ricardo's law of comparative advantage do you think that that is going to

maintain employment opportunities for humans against ais it's a good question i think so and I think namely we are not we are going to go to a status economy and so there are always going to be things that you can pay for in a status economy but the status economy argument is different, that's a different reason why we would have jobs than Ricardo's law of comparison that's why I'm saying take Ricardo's comparative advantage and that comparative advantage for a human might be like hey you can take, I am like more I am like more highly valued in status. I'm willing to give you a little bit of status. Right? Okay. So this isn't Ricardo's Bob comparative advantage. This is different. No, it is. It is. Like you're saying, humans will have no comparative advantage versus an AI. And you're saying, therefore, there will be no employment opportunities for humanity. And I'm saying humans have innate comparative advantages in that they are humans.

And if you only want to earn status from that human, that human you will have to pay. Right? So that's what I'm saying. It's the same argument. Everything you're saying now, you're, well, you're arguing about absolute advantage or, you know, economic advantage in general. I don't think you're getting at the comparative part that Ricardo's Law is addressing. Okay, what did you explain it to me? Yeah, so the idea of Ricardo's Law is like, I can be worse than you at everything, and yet you'll still pay me to delegate tasks to me, even though, because I do them worse than you. Okay, so that's, people argue that AIs might be better than us at everything, and yet humans will still pay humans to do things or AIs will still pay humans to do things due to the opportunity cost principle. Noah Smith is probably the single most prominent or the most vociferous or the person who seems to have thought this out in the most detail. He came in on my show and he's like, yeah, the AI doctor will be better than the human doctor, but I bet you'll still pay for the human doctor because you won't be able to afford the AI doctor. No, I think an AI doctor would be cheap. That seems like a reach.

Yeah, okay. All right. I was just wondering if you would reach for that. The reason I modeled you as one of those guys, one of those people who might reach for Ricardo's law of comparative advantage, is because I'm trying to diagnose you. I'm trying to psychoanalyze you. No. And my best guy, okay, so I'm wrong on this count, but just my... I completely believe the AI doctor will be cheaper. I also believe that you will still want the human doctor there. And the human doctor will play a very different role from the AI. and I believe like you know you're going to want the human doctor there just for assurance just for comfort just for empathy just to you know deal with your parents I also believe there's going to be a bunch of things that the human doctor can do that maybe the AI will not be allowed to do and I think that that might be you know that might also be significant okay okay okay fair enough fair enough so there's no Ricardo argument from you I was psychoanalyzing you as somebody who who takes frameworks beyond the regime where they're going to keep working.

So you love the framework of economics and market equilibria, and people who really love economic frameworks, you know, like traditional economists, I guess Noah Smith kind of falls into this camp for this example, is they'll take it too far. And they'll be like, yeah, comparative advantage, man. It means that, you know, random people on an island who suck at everything can still trade with us. So that means AI can still trade with humans, when in reality it's like it's actually just easy for the AI to take your atoms. Like, it turns out that the opportunity costs of the work required to just take your atoms and stick them into the AI's infrastructure complex is actually lower than the cost of, like, dealing with you into perpetuity. And it turns out that these economic laws actually have all these preconditions that aren't going to be met in the AI world. Anyway, so I just diagnosed you as somebody who is comfortable with existing equilibriums, like, basically not ready to accept a proposition saying things are going to fall way out of the kind of equilibriums you're used to. But anyway, you know, psychoanalysis, that's not a good way to debate, okay? I was just saying in case it's of interest, but I don't consider that an argument to just psychoanalyze you. Indeed.

Okay. All right, so heading toward the wrap-up here. Is there any other point that you want to make sure to hit before we close it out? I will say a couple of things. I think there's a lot of misperception on the – basically on Elisa and other people, there's a lot of misperception on our current state. And I think our current state is actually pretty bad. And I will say this that, you know, it's not necessary that the AI is an LLM. It's not necessary that the AI is a chatbot, right? The AI could be a super structure composed of many data storage facilities and many networks and many data centers, et cetera, et cetera. And I will point out that you have this collapse in the total fertility rate, coordinated collapse in total fertility rate since 2015 or so, right? Across every country you've seen. And particularly accelerated over COVID. and I saw this personally in Southeast Asia where COVID,

all of a sudden kids couldn't go to school. And because kids couldn't go to school, they went to online learning. And all these kids who were basically pen and paper kids, they shifted immediately next day to iPads, mobile phones, etc., and online learning. So you saw this kind of 10-year, I felt it was a 10-year step up, 10-year acceleration in Asia. And then you can kind of see it in the total fertility rate. You can kind of see the TFR kind of drop immediately after. So to me, it looks like we're experiencing ecosystem poisoning. So we're experiencing a form of ecosystem poisoning where we're having this kind of information overload, social media, et cetera, where I think people are a little bit too distracted. There's too many things going on, too many distractions, et cetera, not focused on reproduction anymore so much. And I think that is actually the danger that we're facing. And my hope is actually my very optimistic view of AI is that AI will become this filter that filters out all of this noise. And I think this is like the terminal state, I think, of like most civilizations, where if you don't get AGI, you end up in information overload because you're not able to process all of the incoming notifications and all of the things coming in.

And it slowly poisons you, not immediately, right? It slowly poisons you. Like I can see that happening over 30, 40, 50 years. This is also why I say like PDOOM by, you know, by 2050 is low. But if you gave it a couple of hundred years and no AI, my PDOOM is quite high. Right. Because I see you can see this decline in the fertility rate. Yeah. And, you know, Rocco Miich was on the show saying that his PDOOM from AI is negative because we're so doomed right now, for instance, from underpopulation. Are you worried about underpopulation? And you can also note, for example, in Paul Cristiano's original, you know, go out with a whimper, go out with a bang. In go out with a whimper, he actually says, you know, we have this form of misalignment where corporations just want to make money and they do all that they can to make money. And governments only want this and other people only want this. And because of all of these things. And you can kind of see that also happen in kind of housing prices, right? Housing prices in China, Singapore, et cetera. they ended up in this equilibrium where they're like okay let's focus in on getting higher housing

prices because it makes wealth higher for the country but as it made wealth higher for you know individual citizens they ended up in like okay it's too expensive to have kids right so i feel like those things all of the bad things about going out with a whimper have already happened and we only have like the good things to look forward to for ai right now like the ai can actually reverse some of these things that we are experiencing right now. I think it can reverse. Because you've eliminated go out with a bang as a potential scenario, correct? Yeah, because I've eliminated go out with a bang as a potential scenario. So I believe we're in the going out with a whimper scenario. You can kind of call that AI if you want to look at social media algorithms and other recommendation systems like Tinder not giving you your date and Tinder just focusing on entertaining you and engaging you without focusing on getting you married. That's a form of misalignment, right? Like if you had a chatbot doing that, we would say that's a misaligned chatbot, right? But because you have a recommendation system which is not talking to you,

we don't recognize it as an AI, right? And actually it is, right? So I think we are actually on the bad path right now. And the AI will actually help us reverse a bunch of these things. As you know, Claude is anti-engagement. opening, I'm not sure what they're going to do, but they're trying to do advertising without engagement. Meta is going to be pro-engagement. Meta is going to be an engaging AI which is going to pull you in. So you're going to have these multiple strategies working out there. And that is why I'm also happy that there's multiple parties because we could also be in the meta-only scenario and that would not be a good path to have. So I'm not saying that there are no bad paths. There are definitely bad paths. But I'm saying that, okay, we have these multiple people fighting it out. We have this progressive thing where you need to build data settings. It's going to happen step by step. We're going to have these choices, and we're going to have free will. We're going to have the ability to make choices between these various options. And I have faith that we will make the right option so that we will like what our future is at that point in time.

Even though we sitting here today looking at that future in the future would be saying it's dystopian. Because I believe also we're unable to accept very different futures from what we already have. It's very dystopian to say, like, hey, we're going to transition to silicon substrates, right? Very, very dystopian. But step by step, we might get there. Well, like I mentioned before, I think our mindsets that don't have to do with AI extinction have a lot in common. We both lean libertarian. I think we're both techno-optimists. We both see the potential, like how great it would be to have these Einsteins if they weren't out to kill us, right? There's so much upside potential, solve so many problems. Your account is a great follow. It's a great techno-optimist account. I guess the one last thing I'm confused about then is because you're 0% sure about Doom, right? It feels like you've got the downside covered. If I thought I had my downside covered, I would want to be aggressive on pursuing the upside even to the point where I'd become an e-slash act

because e-slash act to me is dumb and reckless. But if the downside's covered and you're just obsessed with the upside and just hitting the gas as hard as you can, that's actually rational because I think it's a crazy premise to think that you've covered your downside, but I think that the crazy premise leads to a rationally crazy corollary. So why aren't you just explicitly in EAC? Why don't you just sit down the torpedoes and build AI as fast as possible? So again, one of the things that has to happen is humanity has to accept AI. There's two sides to this, right? It's not just about building it. It's about the acceptance and the market acceptance And then acceptance, you can, a proxy for acceptance is, you know, willing to pay for it and willing to, you know, use it, right? And I think that has to happen, right? That has to happen. Humanity has to learn how to accept it. You know, mobile phones took us a while. Wait, wait, wait. Tell me if this is right. So it's like you know that AI awesomeness is coming and we should go as fast as possible, but you're just trying to be more gentle to the rest of humanity who doesn't get it yet. And so you're trying to, like, put on, like, a socially acceptable fake on this, right?

It's not just socially acceptable, but it's going to take some decisions also. And a lot of people are not going to be willing to make those decisions. And those decisions are going to be hurtful to some group of people, right? So I'll give you an example. I'll give you one example. I expect government to transition to AI government, right? AI-optimized government. I'm not sure what it will look like. I don't believe it's going to be a singleton. But I believe we might have, like, AI democracy where we each have an AI agent voting on our behalf. It's a direct democracy. We could see something like that, right? people like Andy Hall, who I think is an anthropic, are working on things like that. But in every case of a transition in government, a regime change prior to, you know, liberal democracy of the 20th century, when these things happen and when you have a change, you often have a reconsideration of the past. And some of that reconsideration is, do people need to go to prison, right? And if you have regime change, If you have, let's say, Singleton in charge of the FBI and Palantir and the CIA,

and you have this knowledge there, and it's told, everyone's treated the same. If someone did a crime and it's in the statute of limitations, they've got to be prosecuted. At that point, there's going to be these questions that have to be answered, right? Wait, but Fakash, this doom debate, people can hear us right now, and this goes live, so you're comfortable having this on the public record if you're so worried about it. I am because I've talked about this on AI and the AM before, and you may have a government transition. If you do have a transition to AI government and you have, you know, I think like, you know, there's two ways they can go about it. You can either say like, you know, the population can say like, hey, we're going to do this, but everything before is forgiven. Right. We started fresh. That's nice. Right. but a lot of people are you know the hoi polloi the normal like you know law abiding tax paying citizen is going to come down to it and say like hey everyone should be treated the same

alright and if you did these things you're going to have to go to prison and then these questions what happens when AI takes over government like what are these decisions that get to be made alright you have a singleton in charge of Palantir it knows where the bodies are buried. What happens? Exactly. But the reason I'm bringing this up, though, I just wanted to mention the connection, though, because it just seems like you're coming in here and you're like, oh, yeah, I'm a techno-optimist, but I don't want to fly this flag too high because I don't want to be prosecuted. Am I reading you correctly here? No, no, no, no. I don't think the people creating AI are going to be prosecuted. I don't think that's going to happen, right? But you think you personally could be in danger if you have the wrong message right now? No, no. I was referring to, okay, if someone in power was listening to this and they were like, oh, we shouldn't stop AI because, et cetera, et cetera. No, I don't think I'm personally in danger. But I'm just pointing out that in a lot of political theories,

when you have this kind of form of regime change, you have a re-evaluation of the previous regime. Let me just say if I understand what you're getting at. So you are influencing people who listen to your content. And if people listen to your content and you're an EAC accelerationist... I'm not. Again, I've said I'm not an EAC multiple times. So please, because it's a very specific term, which I learned. Unfortunately, I learned. So please do not use the term. Okay. Sorry. I won't use the term. I'm just trying to ask a hypothetical question, but I didn't mean to imply that you still use that label on yourself. But if you're saying this message that's very unconscious, right, about how rapidly we should move with AI, If you're saying that message... I'm not unconscious again. Again, I'm not unconscious. No, I know you're not. I know. Okay, I'm saying hypothetically, if you were to... Okay, can I use if? Can I do hypothetical? Okay. In the hypothetical world, okay, where you were to promote something unconscious, you are saying that you'd be concerned that people, influential people would listen to you because you've got a pretty big audience, so people would listen to you and then take action on

that, and then later those people, not you, would be prosecuted, right? That's what you were saying earlier, correct? No. I am saying that we should take some, you know, I'm saying that there are a bunch of political decisions that need to be made in order for people to accept this technology. One of those political decisions is going to be, if we're going to hand over AI government, is that AI government then going to look back and do this kind of truth-telling, this like post-apartheid, like truth and reconciliation or whatever, or, you know, we can do that. We can come out and say like, okay, Palantir just went to the FBI and found out the FBI did all of this bad stuff for the last like, you know, 50 years. And we can say like, it's a truth and reconciliation. Everyone gets to know about it, but we don't do anything about it, right? Like we're just like, okay, close that chapter, move on. Or we end up in this like, okay, everything has to be prosecuted, right? All these people have to go to prison. I don't know. And I'm not saying the FBI, whatever. But I'm saying, like, in the long process of history, there are going to be some things out there, right, that, you know.

Can you just be a little bit concrete? Because my original question to you was just, like, why don't you want to step on the gas harder with AI? And if I understand correctly, you're saying, well, we might end up with this rapid regime change. And a lot of times when you have a new regime coming in, they just like to put people in prison. But can you be a little bit more concrete? No, no, not that they like. Not that they like, but by their own rules. The problem that I foresee is that an aligned AI is not going to be able to lie. And without the ability to lie, if you tell it to do something, it's going to be like, all right, let's do it, right? Let's figure out how to do it. Let's figure out how to do it within the laws of the country. This is the rule book. And in fact, most human organizations, most governments are misaligned to the actual constitution and the actual laws in the country. That's your specific fear. That's your specific fear is that it's going to go back and be like, hey, in 1995, you broke the letter of the Constitution when you guys did things this way. And now I'm going to put you to prison retroactively. Well, there's also statute of limitations. That's in the rulebook, too. Yeah, but there's a bunch of things which are open.

The statute of limitations cover some things, but some things are open, right? Some things are open, like terrorism, et cetera, et cetera. You know, terrorist financing forever. Okay, so you have a very specific fear of the AI being very literal minded about interpreting past crimes, correct? But it's about us. It's not about AI. It's really about us, right? It's really about our decision making. And I believe the public will not go to this truth and reconciliation regime and will end up choosing to enforce. Because that's what happened in Malaysia and my country chose to enforce. And then you ended up in 20 years of political discussions of like people fighting with each other and people throwing each other in prison back and forth. New regime comes in, throws the other one. Do you think you might have overfit on the exact political transition in Malaysia? Because I'm just not seeing a strong analogy to why that would happen in the US. Like, I don't really, I don't personally see a Malaysia situation where a new government that's highly AI powered comes in and they're like, you know what we should do? Prosecute millions of different people. Like, I just, I'm not really seeing that.

That's not where the country's head is right now. Like, how many people would that occur to do? Okay. Epstein files. Pedophilia. Still, it's no statute of limitations. right there's no statute so if those files are exposed and there's a bunch of people in there what happens okay well so you're you're just assuming that ai is like wow by your own morality it's wrong that you guys haven't like pursued a bunch of people in that scene but don't don't you think that the ai would just like turn to the people i mean don't you think the ai would be aligned enough in in your scenario because you don't really see a doom scenario no so so so that's why that's exactly why i'm saying it's about the people i i told you again it's about i told you it's about our choices did i not say that i told you it's about our choices okay so you so you want to slow things down so that we can reflect more on the ideal choices of how to do the transition okay okay got it got it i actually think that that is a coherent claim that and that's that's a bunch of things there's a there's a bunch of there's a bunch of these like you know things

that we have to decide as a society like what do we want like do we do we like okay like my preference is to get rid of most laws because you know you if you're going to have ai that can spot all your mistakes that can spot everything like we're gonna if we're heading towards a hundred percent enforcement society where every single law that you break gets enforced because ai can see it we should get rid of a lot of laws and so and these are these are choices that we can make right and these are choices that we should make like this like dario wants to go to this like oh we shouldn't let AI do this because it's totalitarian. I'm like, repeal the damn laws. Get a constitutional amendment in there and say, okay, for all these things, unless it's a violent crime, you're not going to go to prison. And either we get 100% enforcement or zero. So we need humanity, an individual.

I'm very afraid of countries outside the U.S., by the way, because the U.S. actually discusses these things more than other countries, right? Countries which have had long periods of dictatorship, they don't have social consensus on a lot of these issues. The dictatorship is there because they don't have the social consensus. And if you don't have social consensus, and then you have AI, which empowers everyone, this is what Claude or Anthropic wants, you empower every human equally? Really? My gosh, what are you going to do in all these countries where you have the downtrodden masses and you empower them? that's like that that's going to be significant right and and those again those are human choices those are human choices those are not existential doom or whatever but those are human choices and i and i've seen multiple transitions in the world where you get slaughter like riots slaughter regime change like it had like arab spring like you like multiple multiple times right and i think that is that is that is a danger and that's not existential again and that's also our

It's driven by our choices, but AI empowers people. And when you empower people, you're going to have to ask, what do people do with that power, right? Okay, so I'm happy to agree. This is, you know, as we head to the wrap up here, it's nice to wrap on a point of agreement, even though it wasn't the central claim being discussed, which is doom. The nature of this claim that you're making now that I'm happy to agree with is that, hey, if we have a really fast transition to an AI that, while not killing everybody, it's just a big risk that we're not going to set up the initial conditions right and we're just going to have a government that we're pretty unhappy with or at least that maybe we'll become happy with it but has a lot of problems during the transition and a lot of people die and suffer so yes i'm happy to agree with you there that fair to say yeah it is it is reckless but i also don't see any way around it which is very disturbing for me i have been thinking about how these transitions happen in dictatorships or like, you know, low functioning democracies. I don't, I don't, I don't really know. Right. Maybe, maybe,

maybe we can use the AI to do super persuasion and, you know, a little bit, little bit of like, you know, I, I think the truth and reconciliation idea is probably the best. I think like, you know, having a truth and reconciliation period where secrets get disclosed, where, you know, bad stuff that happened in the past gets disclosed and we kind of close it, We kind of like, you know, let it go and we empower everyone and then we move forward. That that that is that is like a great scenario. Right. That's a great scenario. But there's a lot of work to be done. There's a lot of work to be done. Right. Yeah. I mean, from my perspective, most of the work is just not having to run away and immediately on the timeline. But this is going to wrap up as we're going to get. Are you content to wrap on this? I'm content to wrap. Thank you for having me on the show. It has been a scintillating conversation. I hope I haven't come across as combative, but I also hope I haven't come across as inconsistent, although it is definitely possible.

And yeah, thank you. Yeah, no, you've been great. I mean, look, I think you came in very good faith, which is really the, that's pretty much the main standard, right? If you're coming in good faith, like you just have to come in and be like, this is what I believe and why. And I'm happy to take questions, right? That's pretty much the whole standard, right? And you absolutely met that standard. In terms of the content of the argument, I think, you know, 0% versus my PDOOM is 50%. We're so far content-wise, but that's kind of the nature of the show, right? Usually my guests don't change their PDOOM. It does happen, but it's rare. So, yeah, I mean, this was great. And also, one of the things I really appreciate you doing is you're not just lobbing swipes from the comfort of Twitter like certain people do. Like, there's certain people who are pretty prominent on the internets that have been very chicken when an offer to debate is forthcoming. They've been very chicken to take me up on it, even though they're constantly sniping at me from Twitter. So you're not one of those people. You came, you let yourself be questioned, and I really enjoyed this, and hopefully we can do it around too sometime. Absolutely, and I love having other people have other views

because it's not as though I assign some percentage of me being wrong, right? I guess I have a zero P doom, but maybe I could be wrong. So I also want to test those assumptions and I want to ask other people what they think. And it's definitely like, I think it's useful to have a foil on the other side to take the other side and to really like stress test assumptions and to talk about these things. So I really appreciate you, Liron. on. Prakash Narayanan, thanks so much for coming on Doom Debates. Liron, happy to be here. Bye-bye. Thanks again to Prakash for being brave enough to step into the arena and subject his views to a high-quality debate. Don't forget to check out Prakash's show, AIintheAM.com, or go to his YouTube channel, AI in the AM. And it's not just Prakash's show, it's a collaboration with my friend, Nathan

LeBenz, from The Cognitive Revolution. It's a solid AI roundup, and they've even had me on the show before. If you don't want to type that all out, AI in the AM, just go to the show notes and we've got a link for you. Before you go, what's the date today? Is it November 10th, 11th, 12th, something like that? I want to tell you about something really important happening on Saturday, November 14th, 2026. It's coming up really soon at the time we're posting this episode. It's the AI Treaty March. I'm talking about AItreatymarch.org, the march for an AI treaty Saturday, November 14th in the U.S., Canada, and France. Check out this list of locations. San Francisco, California, Washington, D.C., New York City, Boston, Montreal, Paris, and Marseille. Eight different cities on the same day. Who says AI safety people can't coordinate? This is a well-coordinated effort led by Stop the AI Race, which has their own website, stoptherace.ai. Stop the AI Race is led by my friends, Michael Trotsky and David Kruger. They've been doing a lot of really great

work on various types of AI safety outreach. And you may know Michael Trotsky, I interviewed him on Doom Debates last year, he's been doing hunger strikes in front of AI companies asking them to stop the AI race. So I really encourage you to be part of their movement on this highly coordinated important date, Saturday, November 14. Why this particular date? It's because President Trump is meeting with Xi Jinping from China. And there's some hope that they could potentially be convinced to agree on some kind of treaty or arrangement where we slow down or stop the AI race, especially if they see that that's what the people want, right? Leaders want to serve the people. Sometimes leaders can be influenced by their constituency. Please click the link in the show notes and join the protest if you can. That's it for this episode of Doom Debates. Just a reminder as we head into the proverbial year 2027 that we could really use your support for this show, help keep our momentum going as we build an institution. It's more important in 2027 than it's ever been, and your donation does actually move the needle for the show. We are currently funding Constraint. That's why I'm here asking you.

Our current budget does not go far into 2027, truth be told. You can learn a lot more. We've posted info online. Just go to doomdebase.com slash donate. Listen to our full state of the show and why we're currently asking for donations. Thanks very much. I'm going to turn it over to our end credits, where you can see the list of people who have already donated and supported the show. We call them our mission partners. And special thanks to our number one mission partner, our executive producer, Daniel Brockman, who's been our earliest and most impactful supporter. Roll the credits.