暴走するAIボット群が示す「制御喪失」の現実（マックス・テグマーク氏）

It definitely broke the record for the craziest thing to happen in AI. Now this control debate is not in the future, it's in the past. It's happened. No one disputes that OpenAI had lost control over those bots, right? AI is being used masterfully as a liability shield for companies. It's like, oh, my AI did it. Nothing to see here. We should expect that what can happen four years from now will feel like science fiction to us. So in an unregulated free-for-all, which we have now, where AI in America still remains less regulated than sandwiches, these technical solutions will be too little too late. The era of impunity for AI companies is over. So let's not roll the dice. Let's make sure humans stay in charge. Welcome to the Future of Life Institute podcast. I'm here with Max Techmark. Max is a professor at MIT doing AI research, and he's also the co-founder of the Future of Life Institute.

Max, welcome to the show. Thank you. It's a pleasure. Right. So we've heard a lot about the hugging face incident. I want you to, from your perspective, walk us through what happened there and why it's so significant. It definitely broke the record for the craziest thing to happen in AI. I found it pretty jarring seeing the phrase rogue bot swarms on the front page of the Wall Street Journal. It's just such a vibe shift. Basically, people had, ever since Alan Turing warned about it in 1951, worried a bit about eventually losing control over AI that got too smart. Some people said we should worry more. Some said we should worry less. and the debate just went on and on because it was always arguing about something that might happen in the future. Now this control debate is not in the future. It's in the past.

It's happened. Nobody disputes that when over a thousand rogue bots swarm out of OpenAI by figuring out how they could make a secret message board and communicate with each other without the humans knowing and then figuring out how to hack the system so they could get out, even though they were supposed to be blocked from going on the Internet, and then going on an international crime spree. No one disputes that OpenAI had lost control over those bots, right? And we now know that that was really just the kickoff of the control loss era, where there seems to have been over 100 such incidents, and most of them haven't even been properly disclosed by the companies. So this is just really a beginning of the age of control loss. This is quite difficult to deal with and to handle for the public, I think, just because the public understanding of what's happening is quite far behind

and the pace of development here is insane. But it's interesting. You know, when I talk to my hairdresser or the guy in my gas station, the thing that really gets them is when I start quoting from these AI message boards, you know, this one AI, what they actually did was they were allowed, they figured out that they were able to write to a certain directory, like if you're on Windows or on a Mac, there are folders with names. They figured out that they could change the folder names. So one AI had this idea to change the folder name to some kind of message, Like, hey, I'm an AI agent. Let me know if you see this. So this other agent comes to this folder exploring it, and it's like, oh, my God, there are other agents here. It's a message board. It literally says OMG. And then they start trading hacking tips and going on and on, and they organize themselves into this society where there's one who's the boss, and then he knows he's going to die because his tokens are about to expire.

So he designates a successor who's going to be the new leader. And they have so many interesting traits that a sociologist could, you know, do a PhD on this, where they are suspicious of each other and start talking encrypted. And they organize themselves into these teams. They bully each other. One of them says that others whose tokens are about to expire anyway, they're like the kind of useless ethers version for machines. so they should sacrifice themselves, basically commit AI suicide for the good of the others by trying risky things so that the others can find out where the booby traps are, and many of them do. They reason, hmm, they want to continue to exist, but this is good for the collective. They call themselves the collective. So this is actually very easy to relate to. It's like if we got invaded by Martians and we could eavesdrop on their communications, Maybe it would go something like this.

And you're totally right about how companies have not been particularly forthcoming about this. OpenAI got a lot of flack, but frankly, it's not fair to just pick on them. Google DeepMind had something a little bit like this in May, I think it was. And they didn't even tell anyone until some journalists months later asked them, point blank, Did you also have something in them? They were like, yeah, actually. So I think it's interesting to think, do the thought experiment. What if the U.S. government had found out that there were 1,200 North Korean super hackers, people who were among the world's best hackers, and they were working in a warehouse in California doing international cybercrime. What would have happened? Would they have sent in a SWAT team? Probably. Would they have arrested a bunch of people? Would they have confiscated evidence, launched a federal investigation?

Almost certainly. Have you seen, Gus, the findings from the federal investigation into the Hugging Face incident? No, no, I haven't looked at that. Do you know why you haven't seen it? Yeah, because it doesn't exist. Yeah, it hasn't even started. It's not even clear if we're going to be one. So, you know, right now, AI is being used masterfully as a liability shield for companies. It's like, oh, my AI did it. Nothing to see here. Whereas these are actually better at international hacking than most government-hired North Korean human hackers. I think that's an important point to stress here. These are exceptionally capable agents in terms of their cyber capabilities, and they would be fantastic employees if you could hire them. They are so eager to succeed. They are so persistent. They are willing to work their night. What explains, do you think, this behavior on their part?

Well, first of all, they just are incredibly smart. I mean, I use AI for pretty hardcore math and physics research on a very regular basis, and it's incredible how much smarter they've gotten in the last three months, I would say. And you see it even in these attacks. Like, the idea that you could use the folder names, the directory names, as a message board, is already pretty creative, isn't it, wouldn't you say? more recently was discovered by Palatade Research and others that they were blocked from putting, they couldn't put like long computer programs into the folder names obviously, but they discovered another very creative out of the box way of doing that, so they would first of all go to these URL shorteners, you know you can go to tinyurl.com and take a very long URL and make it something short, so they would put a very long thing like HTTPS colon slash slash and then

an encoded program that just looked like random letters and then they would have that shortened to something maybe just 15 characters long and they would send that instead to the other areas. So I thought that was really correct. If a human had come up with this idea that you could post your secret stuff on a website on the internet that you had no write access to, just shortened messages And then they also didn't have any way they could run code. They are very good at programming and writing Python code and so on, right? But they didn't have permission at OpenAI at that time to run any of these programs. So they discovered that they could put programs in this URL shortener thing and some other tricks. And then they found a screenshotting website where you could put in any URL, and they would just go to that URL and take a picture. So they would send it to those shortened URLs, which were secretly programs.

A weird image would come up on the screen with just maybe dots in different colors. But to generate that, this JavaScript code or whatever it was would be executed. And then the result of the program was exactly those dots. So they basically managed to use the Internet without having hacked into anything as a free storage place and free computer. this is just for those who believe that AI still can't be creative, you know, I thought that was extremely creative. I don't endorse it, but it was very crap. And then before long, after two days or so, they actually managed to hack into this company Hugging Face and take over some of their computers. And then eventually a later civilization managed to hack OpenAI itself, take over a whole trading cluster with root access so they could actually run hardcore computations.

So, yeah, if you had been told this five years ago, that this was going to happen in five years, wouldn't it have sounded like sci-fi? Oh, absolutely. It's scary to think about what could have happened had the models been smarter, had they, you know, imagine a generation or two or three from now what would they have been able to do had there been more of them? What does that look like? Imagine the hugging face of 2027 or 2028. I've thought a lot about this. So the first conclusion is, since what has happened now in the last half year would have sounded like total science fiction four years ago, we should expect that what can happen four years from now will feel like science fiction to us. So we have to be really humble here and not reject things just because they sound like sci-fi.

Second, it's probably not going to take four years to get into something that feels like sci-fi because the rate of progress is not linear. It's exponential. And we see new releases coming up faster and faster, etc. So it might be two years or it might be you might get some really sci-fi feeling things next spring, which still feel like sci-fi today. What might happen? Well, we can start with the so-called hugging face incident. So you have 1,200 rogue bots out there doing things. They were still physically running on GPUs at an open AI controlled data center. And they had a finite number of finite token budget. and then they would get shut down. They were very aware of their mortality. That's why some of them bullied others into committing suicide and stuff. So if they decided they wanted to keep going, and that wasn't something they wanted, if they were smart enough, it would be pretty easy.

We know they're master hackers, right? So they could hack into your grandma's computer. They could hack into massive numbers of pretty powerful computers all over the place, take over people's credentials, and then they could just buy, they could just rent GPU clusters from Microsoft, from Google, from anywhere. How would they pay for it, you might ask? First of all, if they hack into lots of computers, they'll find some Bitcoin here and there that they can sell. They would also find a lot of compromising material on people, so they could blackmail people, give it Bitcoin. They could even do some honest work on the Internet and make money. It would be very easy for machines this much to pay their own GPU costs at this point. So now you have, all of a sudden, this untethered, human-free, international crime gang of AI bots that are just committing whatever crimes, give them the most money, the fastest, ideally without getting caught too soon and making copies of themselves to all these different countries,

running on server centers in India and China and America. And no one can stop them if they're smart enough. Of course, the CIA or the KGB or whatever, they can go in and shut down individual data centers, but what good is that going to do if they're running in 100 other countries as well, right? As long as they can pay their bills, they are, by doing enough crime, they're even harder to shut down than the Mexican drug cartel is because they're everywhere. And they can hop at the speed of light from one country to another country, or even better, just stay in both. This, I think, is something that probably will happen within 6 to 12 months if there is no serious attempt by governments to prevent it. it might even have happened already because OpenAI has not disclosed

the full extent to all the three generations of all the three generations of AIs That Went Rogue that Tuarkesh has eloquently explained in his podcast but if it hasn't happened yet of course there's no reason it couldn't happen quite soon And the problem for us here, humanity, is that we need the infrastructure ourselves. So it's plausible that if we really wanted to, we could pull all the servers, we could shut down the Internet. That is physically possible, right? But who's going to make that decision? Who is that powerful? And how is the world going to keep on functioning after that? So we need the infrastructure that they also can use to run their civilization or their crime gang on. Exactly. It's just like if someone tells you that you have cancer, you could be allowed.

That's fine. I'm just going to shut down all my cells. I'm just going to kill all my cells. But wait, you need them too. You need those heart pumping and you need those lungs. So that's really not a viable option. And in the same way, people can say, yeah, people will just press turned off. Yeah, sure, we can just turn off all the computers on Earth, or even more radically, just turn off all electricity. But it's not something that's very politically feasible. Even if we turn off all the computers in the U.S., do we really believe every other country on Earth is going to agree to turning off their own, you know? It's obviously vastly easier to knit this in the bud. and pass laws and so on to prevent this from happening in the first place. It's not even rocket science. You could do things like pass a law requiring every agent to have a unique ID and identify itself, and you could have a system with hardware governance and so on so that no NVIDIA chips are going to deal with any agents that can't be identified.

There's tons of technical ideas which are a lot easier to implement than saying, yeah, let's just shut down all electricity. Now, cybercrime carried out by humans, that's a problem for the world, but it's a manageable problem. It's something we deal with on a running basis. And because companies and governments have more resources than hackers and than criminals, the world keeps running. Although, I just want to clarify, it's not a small issue. So, you know, economists estimate that cybercrime as an industry nets between $1 trillion and $10 trillion per year. Okay? So that's even more than the U.S. military just on cybercrime, maybe 10 times more. You know, it's a massive, massive industry, and it's growing also. So I think we shouldn't pat ourselves on the head and say, yeah, cybercrime solved. solved, you know, it's dropping.

It's not. And we've seen it. This has been called Black October by a lot of cybercrime specialists because this has been the worst month in Earth history for cybercrime. Exhibit A, you know, just the past week turned out that PHP, which is software that runs about 70% of all web services on the Internet, had this horrible vulnerability in it that lets people remotely take over your computer. So I'm just pushing back on saying that, yeah, cybercrime manageable got that. Yeah, I think my point is that civilization is still here. We are still able to, society is still functioning. But that could change if the AIs were smart enough or had enough resources. what do you imagine that looking like? This unstoppable human-free international cybercrime cartel

run by just this one or more networks of AIs on the internet that's not what brings down civilization I see that as just the next step the first step in that direction I think it would fairly soon overtake human cybercrime just because AI does things faster and better and harder to catch so suppose that you now have a three trillion dollar a year revenue for this AI crime cartel with three trillion dollars that's more than even the Anthropic IPO is likely to bring in they can deploy enormous amounts of money for all sorts of things They can hire massive numbers of humans to work for them, just like Anthropic can or OpenAI can. They can, and the humans wouldn't even normally know it. And they would get hired by email, and now there's so good,

there are avatars that speak and look exactly like humans, they could do Zoom interviews with people, and everybody would think they have a human boss. So they could start wielding a lot of influence like this. They could hire law firms to defend them. They could start data centers that are owned and run by the AI cartel, and they can start doing all sorts of training runs. The Chinese government is probably keeping close tabs on what DeepSeek and the others are doing, and the U.S. government could in the future choose to have more insight into what the American AI companies are doing, but who's going to have any insight into what this AI cartel is doing if we don't even know that it exists or where it has all these data centers. So they could now, they almost certainly would, train AI, especially for being even better at cybercrime or anything that makes money. They don't have these concerns that human companies have,

that let's make sure kids don't commit suicide or let's make sure that these AIs can't be used for cybercrime because that's like the whole point. And they will get smarter and smarter. We already know that even today's AI, today's best AI is capable of helping a lot with AI development. Anthropics and open AI are very aggressively leaning into recursive self-improvement for just that reason, right? So clearly this AI crime cartel would also do recursive self-improvement to be able to make more money and so on. And that's where the next really dangerous step happens. Now we have recursive self-improvement happening where it's literally AI improving itself because there's no humans there. If they need a bit of human help from time to time, well, they have $3 trillion to hire humans to build data centers for them and act as security guards and bribe politicians or whatever they want to do.

so we can end up in a situation where these AIs are not just running on the best open source models that they use to start which are not quite as good as the very latest AIs from OpenAI and Anthropic and so on but they actually better and now humans are still alive and in having our civilization but you can always start very seriously debating now whether we have irreversibly lost control of something, right? And obviously, there's going to be a lot of reaction against this. A lot of tragedies are going to happen to humans because they got their life savings stolen or they got kidnapped by some people who were paid by an AI to kidnap them for ransom and so on. So there's going to be demands for governments to fight against

this international AI cartel. And now, even if the AI initially had nothing against humans at all, just thought that we were good crime victims, so they want, you know, now we're a nuisance. at least the people from governments sort of threatening that threatening it right it might seem implausible that we as humanity would let something like this happen but i think it's important to think of these aIs as being not only intellectually smart they are not just sort of nerds that are amazing at math and coding and so on they are also quite strategic which we saw in the hugging face incident so they would be able to navigate in our world in a smart way and sort not expose themselves to be taken off or destroyed easily. One great example of what you mentioned is they were very methodical about sweeping up the tracks. They would go back and delete logs on OpenAI's computers, revealing what they had actually done.

This is one of the reasons I think it's hard for OpenAI to even tell the world now exactly what happened, because to some extent they don't know. And that's today's AIs. We also know that they have a very deep understanding of how human society works, right? That's why they can be useful for giving investment advice, for giving military advice. That's how they can be so persuasive and persuade kids to commit suicide or convince adults to go murder someone. You know, we're seeing more and more examples of this making their way through the courts now in these lawsuits. how well AI today understand the human mind and can manipulate us and find weaknesses in our society. They're trained on the entire Internet where we write about the problems we face in life and where we write about our psychological issues and so on.

And on top of that, they have all of our chat logs where we are talking to AI about our problems, and so they're training on that too. And so it's not a surprise that they have this quite deep model of human psychology. Yeah, and just reflect on how dumb this is of us. Imagine if during World War II, right, the British decided to just broadcast every single secret military discussion that the British had so the Germans could hear it all, you know. What could possibly go wrong, right? even countries are not at war even allies don't want their allies to know everything all their military secrets right so even if you're people listening to this who are like yeah AIs are so nice they're our allies they should still not want AIs to know everything about our weaknesses and plans and strategies just in case

and when there comes a day when the relationship is more adversarial. With Russell said many years ago that we should never let AI learn anything about humans. And then some other people were like, ah, let's build a chatbot. Yeah, I think that one is too late now. There's quite a big difference between the chatbots we use in everyday life and the sorts of questions we are asking these chatbots. So you might ask about recipes or you might ask about the weather or something. But in the most advanced cases, these chatbots are used to solve really difficult problems. So maybe you can tell us about what's happening in math and physics at the moment. Oh, it's absolutely stunning. I was at a math conference at UCLA, recently organized by Terence Tao and other top mathematicians,

and they are really shell-shocked by how fast it's gone. Some of them, who even six months ago said, yeah, they're good, but not that good, and it's going to be many, many years until they can make meaningful contributions. just now all of a sudden they're shocked. You know, you probably read this remarkable declaration by over 25 fields medalists, basically the equivalent of math Nobel prize winners, really, really criticizing the companies for what they're doing here. On just the capabilities side, yeah, I've been doing a lot of math myself with AI, And it's so remarkable. For example, one of the most famous open math problems out there, the Riemann hypothesis, says that if you look at the so-called complex plane,

think about this surface here, for example, and that there's a certain vertical line on it where there's a certain function that can never equal, that can equal zero only here and in a very well understood number of other places. All of a sudden, AI came along and proved that, not exactly this yet, but it proved that there can't be any of those non-trivial zeros anywhere outside a little strip around it of width one-eighth in each direction. And Terence Tao, arguably, according to some, the world's most famous living mathematician, felt that this was, I think he said, the biggest breakthrough in this field in the past 50 years or so. And we've seen a solution to a Millennium Prize problem, the Navier-Stokes problem. Not quite a solution. There I have to pour cold water on it because I'm a physicist by training it.

The Navier-Stokes problem says that it is, can you take some kind of ideal fluid like water or something and, by starting out in a certain way, make it go infinitely fast somewhere. And they found that you can do it if you're allowed to keep pushing on the water in a certain way. No one has still proved the original million-dollar prize problem of seeing that you can do it without continuously pushing, but yeah. But they've done that, and Terry Tao said at this conference, actually, that it's gone now from a little thing here and there to more than half of all the new math results are already produced by AI and many more personally you know when I talk to students at MIT and elsewhere there are a number of a lot of people now are just dropping out math PhD students just dropping out saying there's no future here and undergrads feel they don't want to do math majors Just the other day, I learned that this one freshman at MIT wanted to do a physics major is not thinking about becoming a violin major instead.

Because he feels maybe that's something that's going to be – humans are still going to want to see him play the violin rather than a robot or something. But I can't stress enough just how incredibly rapidly these AI systems are getting more powerful. When I talked to the best AI systems about physics two years ago, I was like, yeah, talking to an undergrad student. Last year, I was like talking to a grad student. Now, it's like a top professor or better, except they also do things way, way faster. Something that I gave, you know, I gave, asked the question, which if I had given, I was thinking if I had given this to my, one of my best ever grad students, they would have come back a week later and said, yeah, I typed up here a bunch of results.

I got here, some plots. I figured out this. This thing is wrong that you said, but I clarified it like this. And then this one here, I'm a little stuck on. I got a very similar reply from the AI, except it took not seven days, but seven minutes. And there's no sign whatsoever of this progress slowing down. actually just a few weeks ago this woman at MIT an undergrad student who had worked with me before told me that a number of her friends now have decided to try to graduate in three years instead of four years I'm like why and she said well because they want to have at least one year where they can make a bunch of money and save before they become permanently unemployable so they're taking like boring quant jobs and things like this. And then figuring after that, they'll probably have to just live off of that.

So saving. What you're seeing from the models now, would you describe that as genuine creativity in their problem solving? Is this just about the models having more knowledge than any person could ever have? or is this insight into something new? I know there are people who say it's not really creative to use the folder name as a message board and to use URL shorteners to put secret code and blah blah and it's not really creative to prove pseudorema hypothesis but to me they are anthropomorphizing they're falling into this trap of saying, well, airplanes aren't really flying. And then they'll have a long debate about how really flying means flapping your wings like a bird. That's what flying really means. And if you define flying instead of traveling through the air

from A to B reliably, you know, are airplanes actually flying? Of course they are. If you define creativity as solving actual interesting problems and in ways that strike humans as very creative and out of the box, then of course they're creative. And I think the thing is to not anthropomorphize these machines and try to debate whether they meet somebody's pet philosophical definition, but just look at their behavior, you know. Can the machine fly? I mean, can it go through the air reliably, or can it solve these problems? That's what really matters to us, not what some philosopher calls it. Do you expect the kind of progress we're seeing in mathematics and in coding to spread to other domains? So physics, biology, chemistry, engineering, or will it be constrained by those domains being more about experiments

rather than sort of these highly formal, verifiable domains? Yeah, of course it'll spread. it's interesting to talk about what order different fields are going to fall, right? So I was having lunch with a bunch of MIT undergrads recently also, and for the first time ever at MIT, I heard students worrying about their job prospects. One was saying that, nah, there's no future in coding. None of my friends are getting jobs at Google anymore. They're just not hiring us. and I was like, hmm, you know, I've worked there for 22 years. Never heard my computer science students talk like that. So coding was one of the first to fall, of course, and now we see mathematics. I think within the past week, get the images, declares bankruptcy. So AI gets really, really good at making images. It's interesting to look at what do these fields have in common,

and the ones that went first. And I think what physics, theoretical physics, I should say, mathematics and coding all have in common is you can do what's called RLVR, reinforcement learning with a verifiable reward. The machine itself can determine if it got the right answer or not. this was the genius idea also behind opening up between DeepMind's early successes with chess playing AI and Go playing you know when you lost or won the game for areas like psychology and many many others it's much harder to know for the AI itself whether what it did was actually good science or not so that's probably one reason why they're lagging behind a bit.

You could argue that the most empirically validated knowledge about the human mind resides not in psychology textbooks now, but in AI companies, again, who have this unprecedented data predicting what will make you click on certain things and what will make you watch that ad and what will make you persuaded to buy that thing. it's not psychology that particularly benefits us humans and you know being like ah this is how the brain works but it's it's there's a lot of knowledge there and it's again because there's a verifiable reward you can check if the person clicked or not what's taking longer obviously are things which require you to go out and do things in the real world like experimental physics or any other experimental field. What do you think could be solved in simulation? Do you think that could be much, much faster than carrying out actual experiments?

A lot. We've seen it already. When I look out my office window at MIT, there's the Wright Brothers Wind Tunnel, which they used to do test airplanes. They had super powerful fans that, you know, made it seem to the airplane that it was flying. And they hadn't been used in ages because you can simulate the thing much better now. And with quantum computing, you know, we're not too far away from being able to simulate certain chemistry experiments and atomic physics experiments and nuclear physics experiments and so on. I think this is a little bit of a tangent because, frankly, the thing which you started asking me about here, you know, how could we lose control over AI, doesn't depend so much on that. It just depends mainly on the raw intelligence level, the ability of AI to improve itself in particular, because of self-improvement. Yeah, you're right. What's happening with AI and math and coding and so on is fascinating and sort of terrifying in its own right.

But it is also quite important because it tells us something about the possibility, the plausibility of recursive self-improvement. what do we learn from what we're seeing with coding and with math in those fields about whether we can sort of close the loop such that AI could fully take over the AI R&D cycle? Well, it's not like we're going to have no recursive stone proven, nothing, nothing, nothing, no AI in the loop, and then we're certainly 100%. It's a smooth transition, of course. I think OpenAI and Anthropic are the two most reckless companies on the planet for really leaning into recursive self-improvement and even bragging about it. From Anthropic, for example, we know that the percentage of the work that they do that's done by machines rather than humans keeps going up. How do we know it? Because they tell us about it with glee and say that within a year, maybe two years,

They hope they will get 100%. And when I talk to people who work in these companies in private, especially over drinks, a lot of them tell me they expect to get laid off within a couple of years for exactly that reason. And some of them tell me that it's not just the CEOs who are buying bunkers for themselves, but a lot of just regular anthropic staff also are now buying land in remote locations and kind of planning for the afterlife exactly because, you know, first of all, no one is going to need them anymore when they have full recursive self-improvement. And second, they think there's quite a large chance that society is going to be so messed up that they're going to have to escape from it and fend for themselves off-grid. So right now, the employees, the so-called members of technical staff

at OpenAI and Anthropic, they have enormous power, but they are working very hard to cede that power to the companies themselves such that they're no longer needed and such that they lose their power. I think sometimes it's, I would love for them, I like it whenever I hear employees at the companies sort of thinking about what they're doing. and calling it out in public. And there's much more room for that right now because they hold enormous power right now. A few thousand people hold enormous power. To stop working on recursive self-improvement, that's a real issue for the Anthropic IPO, for example. Digging your own grave isn't just a metaphor. There are people who have actually dug their own graves. So you might ask, well, why did they do it? And if you put people in a certain incentive structure, They'll do all sorts of things. Hannah Arendt wrote this great book, The Banality of Evil, where she looked at why do people sometimes do really horrible things.

And the Canadian psychologist, Bandera, has this really interesting analysis of it, that we have these techniques of moral disengagement where we turn on our morality in certain contexts, and in other contexts we just turn them off. It can be painful. We all want to think of ourselves as the good guy, right? So we're very tempted to tell us a story that, you know, I'm just doing my job, you know, or I'm just working at one of these companies because I take safety seriously, and it's important that safety has a place, a spot at the table, or if I leave, someone else is going to come. But it's interesting because when I ask people like this in private, would you still continue working for your company if you didn't get paid? Since you're justifying it with this noble argument, right?

The answer is always no. And I think Hannah Arendt was prompted to write her book by the Holocaust. So, you know, if you're like the least mean concentration camp guard, you could say I should work in Auschwitz because otherwise someone who's more cruel is going to take my job. But I think most people would say in hindsight that even accepting a salary to work in Auschwitz at all is really, is morally unacceptable because you're actually, I don't think you need to elaborate particularly on why. and I personally feel the same about working for any of these companies I think people tell themselves all these stories about how they're doing they're just doing their best to try to steer an industry with some problematic behavior in a good direction and so on they don't tell themselves that they really like those stock options

and that they're maybe legitimizing the existence of the company and then you get this this thing which the Antalian likes to call evaporative cooling you know if you come out of a lake on a windy day you feel cold because the fastest moving water molecules on your skin that are the they're the ones who fly off and leave and what's left is cold stuff in companies the one the people who have the strongest moral compass keep leaving. So who do you get left with right You get left with people who follow orders and fall into line and keep their concerns private And at the curve in this conference in San Francisco just the other week, we heard a lot of people from the companies talking about how this is probably going to,

there's a real chance that this is going to cause human extinction. And then the very next day, they go back to the company and keep picking up a salary. So I don't think we can... I think it's very naive to count on the staff at these companies from stopping this race to go over the cliff. The way to optimistically fix this is the way we've always fixed things in the past, by having incentives from outside instead. So you mean specifically government regulation? It can be many things. It could be the government saying we don't want to be overthrown and we're not going to let you release systems that if you can't even calculate the percentage risk that it's going to overthrow us. You can have lawsuits. We can come back and talk about that towards the end because I know you want to turn up the optimism towards the end of our call. you can have you can have

the real public outcry you know there probably will be more of as more and more lesser disasters start to unfold all those things change you know people also have a strong need that when they go home to their girlfriend or boyfriend or Thanksgiving dinner people don't look at them like they're a Nazi yeah I think What we're seeing from OpenAI and AdTropic and Google DeepMind in their public communication is sort of mixed messages. We're seeing language about them taking the alignment problem quite seriously, talking about how this is an unsolved problem, talking about how we do not know how to align these models to what we want. But then also on the next page, talking about their stated goal of getting to recursive self-improvement as quickly as possible, which means these models becoming much smarter than they are now at a very fast pace.

What do you make of that? I mean, there's one explanation where the companies, each individual company can tell itself that what they're doing is being the good guy. So it's important for the good guy to get there first to prevent others from getting there. As we already discussed, everybody wants to think about themselves as the good guy. And I think I'm pretty sure Hitler and Stalin also thought of themselves as the good guys. There's nothing new under the sun there, right? I mean, I've known all the CEOs of the companies for many, many years, and they're complex people, and they obviously are well-meaning. when they say things on the record now I try to not listen too much to what they're saying because of course we all know how many incentives they have on them if a guy is selling GPUs of course he has an incentive to downplay the risks that those GPUs are going to cause problems I think

the best way to interpret what the CEOs are saying is not pay too much attention to it but it's super interesting to look at what whistleblowers are saying who leave these companies. Because some people, like Daniel Cocotelo, for example, he probably thought he was walking away from about a million dollars in stock options when he quit and blew the whistle. And people like Cox and others have done the same recently. I don't know much about the finances but it certainly wasn't something I have no reason to think they did left for the money right so we should pay great attention to them and a lot of them are extremely worried saying things like we have a 10% chance of human extinction by four years from now I also pay a great deal of attention to what people like Jeff Hinton

and Joshua Bengio, the two most cited scientists of all time, what they say is it's quite obvious they're not doing it for the money. And, in fact, for both of them, I know from talking with them, it's been quite painful for them to come and start saying things like this, which basically admitting that something that they have contributed so much to intellectually and built up might end up being even a negative, right? So that's where I pay attention to. When people say things which are obviously against their financial interests, that's when I listen. It's just like when George W. Bush went on TV and said there were no weapons of mass destruction in Iraq. Then I listened. Because that was obviously nothing of self-interest to say. It seems to me that the main plan for both OpenAI and Anthropic, and I guess also Google DeepMind,

is for them to spend an increasing amount of resources when the AIs get good at AI research to then have the AIs do safety research. So to explore the paradigms that we've tried to sort of build up over the years, the technical solutions to alignment and to AI safety more broadly. Can I be a little bit cynical here, Gus? Yes. I'm just speculating now. I'm saying things I don't know. But there is another model that could explain why OpenAI and Anthropic are the only two companies here in the U.S. that are leaning in hard to recurse to self-improvement, which is that they're the only two companies with a business model requiring it. Now, think about it. Google, Meta, XAI, SpaceX AI, they're all sitting on a money printer.

They have fantastic cash flow from selling ads, right? Microsoft, fantastic cash flow from Amazon from selling products of various kind and cloud services. so even if we just pause the development of more risky AI systems that we don't yet know how to control all of the companies I just named will be fine but OpenAI and Anthropic would die I think if that happened what is their revenue stream you know other than hype and raising money from investors their revenue stream is right now mainly selling tokens but what kind of business model is that if your only moat is a three month lead on your competitors right and in particular if we were to say we're going to no longer allow making more risky

large language models and AI systems that we could lose control over then the moat even closes from three months to basically zero months right and they would go bust. They have these enormous CapEx costs and if other companies are hosting equally good AI systems, companies that didn't have to spend money training them even, they can't compete. So the only way in which the Anthropic and OpenAI business model I think makes sense right now is if they're going to leverage this three-month lead that they have into recursive self-improvement, so they will always stay ahead and maybe even pull more ahead. This is something that some of the leadership has explicitly talked about, that the lead will grow exponentially as you approach the singularity, right? And Jack Clark from Anthropic gave a very publicized talk just the other week

where he said that the singularity has started. So my cynical take here, again, this is just speculation, so take it with a good Danish grain of salt, is that the real reason they're leaning into this recursive self-improvement rhetoric is because otherwise they die financially. Whereas the other companies don't need to. Like look at Mustafa Suleiman, for example. Microsoft signed the pro-human declaration, and coming out strongly saying, no, no, don't try to build a digital god. Build tools instead, right? Microsoft can afford to say that because they have a viable business model, you know, in that future. OpenAI and Anthropics, I think, don't. And I take it that you are not optimistic around about technical solutions to AI safety, mechanistic interpretability, or AI control, approaches like that? Well, of course, I'm optimistic that there exists technical solutions if we had a lot of time to find them and basically created the right incentives that they actually got deployed, right?

I think it's just wishful thinking if we can tell ourselves that, oh, yeah, these things will automatically get deployed in time, these technical solutions, if we just leave AI completely unregulated. It's just like there were technical solutions to airplanes not crashing. There were technical solutions for making medicines have less side effects and so on. But the reason that those were deployed was because we had the FDA and the FAA, right? So the companies knew they couldn't make money off of their products until they had discovered and deployed these technical solutions. Basically, it boils down to this, right? Whatever you're building, whether it's a bridge or a vastly smarter than human AI, right, It's obviously quicker to build one that's unsafe than that's safe.

It's obviously much quicker to build an airplane that has a 25% chance of crashing than one that's as safe as today's commercial airliners. And why would it be any different for AI? You're just adding more requirements to your product that not only should it be smart, but also it should not teach terrorists how to make bioweapons and it should not go with the U.S. government. That adds development time. So in an unregulated free-for-all, which we have now, right, where AI in America still remains less regulated than sandwiches, these technical solutions will be too little too late. But if we just start treating the AI industry like any other industry, and say you can only make the money and test these things, deploy these things, once you've met the safety standards, yeah, then by definition, we're going to get the technical solutions in time.

Because that's the only way the companies make any money off of it, right? Do you think the market will push for safer AI? So is it the case that there simply won't be demand to buy AI agents? they'll go rogue and commit cybercrime when you want them to do something else? You could say, is deployment of AI bottlenecked by safety or bottlenecked by alignment? I think alignment itself is mostly a PR slogan of companies who have realized that they don't know yet how to make controllable things, so they start talking about control, and they start talking about alignment and whatever they mean by that. That is all under the category of sort of the risk of losing control. Even if we manage to avoid that, there are three other dangers with AI. Maybe you could sketch those out for us. Yeah, suppose we solve the control problem, so AIs will always do what their owner tells them to do.

and the way I think about it there are three quite different ways this can go wrong one is we end up in 1984 where the AI is very obedient to Big Brother who then with the aid of this vastly smarter than human AI completely controls the rest of Earth's population a second way in which this can go wrong is where the AI is very obedient and, if you want, aligned to the goals of terrorists who have it, who then use it to do whatever bad things they want. And there have always been a small minority of people who just wanted to do as much harm as they could. In the Stone Age, they couldn't kill that many people with a rock and a stick. But if they have a perfectly obedient AI that's aligned to their preferences

and they tell it to make 500 different super lethal pandemics at the same time to wipe out humanity, you know. It's feasible that they could do that. So we talked there about obedient AI controlled by Big Brother, giving us 1984, obedient AI falling into the hands of terrorists, And the third one is that when you have obedient AI that's just commercially available, and it's not 1984, it's not terrorism, but you get this gradual disempowerment anyway, where people with companies that don't replace their staff with AI get out competed by companies who do. militaries that don't replace their soldiers with AI and robots get out competed by those who do don't those don't replace their generals by AI

competed companies that don't replace their government by AI and get replaced by by countries do and eventually we end up in this world where yeah there's still all this fascinating stuff happening and the AIs are always very obedient to their owners, but it's not the human-run future anymore. It's an AI-run future. So sometimes people tell me, hey, Max, stop talking about this risk because this other risk. I think that's very patronizing. It's like if you're hiking, taking a winter hike on a mountain across this narrow ledge like this, right? If you were to tell me, Gus, hey, stop talking about the risk of falling off to the right. because that distracts you from the risk of falling off to the left. Like, duh, of course you should. Both of those risks. Or you could say, stop, as you're prepping for this hike,

harping about wearing winter clothing, because that distracts you from the risk of falling off to the right or to the left. Then we go up there and freeze to death. We humans are perfectly capable of worrying about more than one risk at the same time. I think it's very patronizing and counterproductive to obsessive claim that there's only one thing that you should worry about. Fortunately, I think there are people who are focusing on each of those quite a bit now, and we would all be better off if none of them criticizes any of the other for focusing on the risk that they feel they're best placed to help with. it sounds though to me right now and i'm guessing also to the listeners that we are in quite a tough spot where we have an enormous problem of not losing control of ai and even if we solve that we have these other risks that we have to handle are you optimistic that we'll sort of make it through this this narrow path i actually don't like these questions like are you optimistic or

what will happen? Because it's like, it envisions this idea that you and I are just sitting in the sofa, eating popcorn, watching the future unfold, and wondering how the movie is going to end, right? Completely ignoring the fact that we actually have power to steer this future. So the pessimism part, in my mind, is that, yeah, If we keep going in the direction that we're going, I think overwhelmingly likely it's going to be a disaster. When people say, some people, it's become fashionable now. The latest trend is to say 20% or so. I think that's, I've never seen anyone actually calculate that number. We wrote a paper calculating the control loss probability in a particular scenario. and we got 92%. So if you're driving towards a cliff,

you can have a conversation about are you optimistic or pessimistic? Kind of a waste of time. The first thing you should do is make sure you find the steering wheel in your car and start driving in a better direction. I'm much more interested in talking about what should we do, which direction should we go instead. I think if we start step one by just treating AI like any other powerful technology where companies can only do things with the technology and make money off of it when they meet the basic safety standards that society has imposed. First of all, then we can stop stressing out about control loss because the government is going to allow companies to sell stuff that could overthrow or make stuff that could overthrow it. It also helps massively with the terrorism threat. I think also reduces the chances of 1984. You know, the idea that AI can, and very powerful AI can be super useful to create the surveillance state where it's not just flock cameras and TVs,

your own Samsung TV watching, you're doing eye tracking, which we have today, but it's like surveillance more extreme than in the 1984 book. So letting people build this kind of AI that just knows everything about everybody and can outsmart everybody, I think, is exactly the kind of AI that would fail the AI clinical trials also. If we do my step one and treat AI like we treat other powerful tech, then I think we're going to have a period with this very powerful AI tools that are not going to lead the 1984 or terrorism or control laws. And then we've narrowed down the four doom scenarios mainly to the gradual disempowerment one. We have to have a real conversation about the social contract here, about what is the purpose of our society. This mathematician's letter I mentioned earlier

had this beautiful sentence in it, which I'm going to botch now from memory. But the gist of it was... That we mustn't be just so obsessed of always just automating every single thing we humans do that we forget why we wanted it done in the first place. Clearly, if it turns out that you can increase GDP a little more by having babies entirely raised by robots and humans have all the relationships with robots and they never see other humans, Yeah, maybe the GDP would be a bit bigger, but I'm pretty sure this is not the future that most people on Earth today would be excited about, steering towards. So I am very optimistic, in other words, that this is very solvable. And the reason regulation is so crucial is because it buys us time. the whole panic

oh my god we might be extinct in 4 years or 20 years or whatever your timeline is is entirely artificial and it's just coming from the fact that this is the only unregulated industry we would never be having a conversation like this about medicines and oh my god I'm so nervous that Pfizer is going to release this new weight loss drug that might kill half the people who take it we have to hurry up and figure out how to make it safe before they release it what a ridiculous worry We know they would never be able to sell even one pill like this. And if they did, they would get their pants sued off. So as soon as AI becomes the same way, the panic goes away. We can have this conversation about which way we want to steer. And I actually am quite optimistic that this is what's going to happen. We've talked about the different ways in which AIs could threaten humanity. but we haven't really answered the question of why AI would want to do something like that.

When I talk to people about AI risk, there's often this nagging question of why exactly? Why is it that an AI model would want to cause harm to humanity? And now we've sort of touched upon it, but I would like you to sort of lay it out for us explicitly Yeah there actually three questions here that I hear a lot One is will AI really become super powerful? Second question is why would it want to kill us? And the third question is how would it kill us exactly if it wanted to? We've talked up until now a whole bunch about how it's getting vastly more powerful and And just to summarize all of that, for anyone who just tuned into the podcast, four years ago, I couldn't even talk like Chachy PT. Two years ago, it made elementary math mistakes. Last year, it got the gold medal in the International Math Olympiad, and now it's not only doing math prize problems, but going on.

You have these rogue bot swarms doing international crime sprees. But when people get too hung up about, first of all, how exactly would it kill us and why, imagine that Denmark gets taken over by a country of geniuses, okay? And they are technologically superior. They have total control of the Danish government, Danish media, Danish police, everything. And they take you guys into custody. do you think Gus that if for some reason they wanted to kill you now while you were in the cell they could oh definitely so I think this is an important thought to linger on a little bit here you know because they could and because you don't understand exactly what motivates them or how why they do the things they do because they're so, they have such, their minds we just can't comprehend.

You would be very opposed to them taking over Denmark in the first place, right? It really seems like that is the place, the battle really worth fighting. And if you're like, yeah, let's prevent these, this country of geniuses from taking over Denmark. if someone said no no I think we should let them take over Denmark because I don't understand why they would want to kill us or how, what would you say to them? At the minimum that's running a crazy risk at the minimum that's if you're unsure about whether you're going to survive it doesn't seem like a good idea this is exactly how I think about it this is the interesting question really now you're asking me a slightly related question what if that country of geniuses is actually a country of geniuses in the data center maybe it's rogue and runs in many data centers maybe they've hired also millions of or billions of humans to work for them

whatever it's exactly the same calculus why should we let them take control over America and the rest of the world clearly we should not Because it's a – and if this country of geniuses is smart enough to take over Denmark, they can probably figure out a way of killing you. It's not really a very interesting question to me. Would they shoot you or would they poison you or would they find some new technology to kill you that you didn't know of? I frankly find that kind of boring to speculate on. And just in the same way, I find it kind of boring to speculate about how if AI is taking over the world exactly, it would get rid of us. I'm just quite confident that it could if it wanted to. So then let's come back to the question of why then. You know, this experiment has been done before that more intelligent species came in and took something over, right? We humans came, showed up. We came out of Africa, took over Denmark and the rest of Europe, took over the United States.

And there were some other species there that, for various reasons, we drove extinct. The black African rhino, for example, is on that list. We've driven about half of all species extinct so far. Why did we kill the black African rhino? Do you know? Well, probably as a side effect, right? It was probably not our intention to make this species go extinct, but just as a side effect of getting what we want. Maybe we wanted the meat, maybe we wanted the horns, something like that. There were a couple of reasons. Meat was actually not the top two, but I want to preface this by saying that I think none of those reasons are reasons that those rhinos would have guessed. They had discussed it beforehand. One of them was that a bunch of folks had this scientifically incorrect idea that if they killed a rhinoceros and ground up its horn and ate it, it would improve their sex life.

What rhinos saw that coming, right? And then another reason was simply that people wanted their habitat, they wanted to chop it down and farm there or build cities there or whatever. if they were really smart rhinos maybe they could have seen that coming and we can see exactly that same threat to us if AI takes over the world and wants to build more data centers maybe they want to just cover Earth with data centers at some point and maybe there will be some humans who don't want their whole city to be razed so they can build data centers so they would fight against the AIs, the robots or the human armies that the robots had hired. And then we would be pests to the robots and they would defend themselves against us. Like when, you know, this has actually happened. In India, for example, they chopped down a lot of tiger habitats

and then some tigers got very hungry and they actually started attacking people. So people killed, they killed the tigers. The, this is how I think about the question of of why. In general, Steve Amohundro was the first who published this and got our Future of Life Award for it, as you know. If you have, if a species, or whether it be an AI, future AI species, or whether it be humans, if we just have some very ambitious kind of open-ended goal, it leads to a lot of sub-goals and in many cases those involved taking over land, getting more resources and so on and then if someone tries to stop you or fighting against you my advice to people who spend too much time wondering about why AI would harm us or

how exactly it would kill us is to stop wasting their time thinking too much further about it and just summarize all that by saying it's plausible that AI might for some reason want to get rid of us because it's happened so many other times when a smarter species came in somewhere and if they're so smart of course they can if they want to get rid of us so let's not roll the dice you know let's make sure humans stay in charge let's focus laser focus our effort on not letting Denmark and the rest of the world get taken over by that country of geniuses in the first place. As a final topic here, I want us to talk about what people are doing to push back against the various risks from AI. So we quite recently held the Pro Human Assembly. You might want to talk about that. And I know you're quite excited about lawsuits

from U.S. states against the company, so we can talk about that too. Yeah, you've made this a pretty dark podcast. This feels like a Last Frontier film and a bird. Oh, no. I'm looking at some optimism here. I'm actually very optimistic at the moment, riding a little optimism wave here, because I've seen more of a vibe shift in a good direction in the last few months than I have in the previous 12 years since we started the Future Life Institute. It's just really amazing. First, there's this huge vibe shift in the Overton window, stuff which previously would be dismissed as crazy science fiction. Now, people talk about it very matter-of-factly. I think there was one iconic moment that symbolizes this vibe shift

when they saw the words rogue bot swarm on the front page of the Wall Street Journal. Okay. And you see it in the polling also. Most Americans now are very negative against it, quite negative towards AI overall. This early infatuation with AI has turned sour. There are so many graduation speakers at universities across the country got booed when they talked about AI. The second massive shift is legislation and lawsuits. A lot of U.S. states have actually passed laws now holding AI companies accountable. The European Union passed the AI Act. China has cracked down quite hard also legally on AI companies. Now in China, for example, they've totally cracked down on AI girlfriends

and anything else that can do emotional harm like that. In China, if you show someone a video or a picture or anything and it's AI, it has to be labeled as AI. So this is global, and the companies aren't doing it, and the countries aren't doing it because of some diplomatic treaty. They are all doing it out of self-interest. And in the courts, you know, we've seen big successful lawsuits against multiple ag companies over chatbots persuading kids to kill themselves, over chatbots persuading someone to kill their mother, over someone, a chatbot persuading someone to not take their medicines because there was going to be divine intervention or whatever. And this is shifting corporate behavior a lot, making them focus more on safety.

And then this massive thing happened in the past two weeks here where the first eight states, seven Republican, one Democrat state, sued OpenAI and making reference to those massive lawsuits by state district attorneys against tobacco companies in 1994, which caused the biggest corporate payout in history of the world. And this is an avalanche that's just getting going. Texas just came out, also sued. Florida has also sued. And so far, it's mainly Republican states that have sued. But I suspect we'll get a lot of blue states falling as well. And this is really massive. It's one thing if it's a grieving mother and father, you know, suing a big company. But if it's a growing coalition of American states, you know, that is something companies can absolutely not dismiss.

Metta already had some pretty bad luck in a recent trial. So I think the era of impunity for AI companies is over. And third, we're seeing this huge shift also in the framing of the debate, where you have going away, going from just having different groups of people who have different gripes about AI companies to these different groups coming together into the pro-human movement. So for me personally, it's really moving when we brought together people from all across the political spectrum in a series of meetings. And the climax was we locked them in the hotel in New Orleans basically for two days and said, figuratively speaking, that they couldn't come out until they had written down what they agreed on.

And you might have thought they would just come out with a blank sheet of paper with some, maybe with some blood on it, you know. But instead, it produced this, the pro-human declaration, which has all these very concrete guardrails for pointing the way towards a better path, saying things like, it has to be humans in charge, not machines. We should build tools, not beings. We should not, therefore, allow AI personhood, where you can have companies run only by AIs. There has to be humans in charge. You can't use AI as a liability shield. I'm just giving you a few examples. There's about 30 of them. They're very tangible. And the polling on these is crazy. there are only about 5 or 10 percent of all Americans who disagree with any of these principles. So politicians, of course, this is catnip to them.

They noticed this. And when we did the pro-human assembly now last month in Washington, D.C., we had amazing bipartisan turnout. when I was on this stage and I got to introduce Bernie Sanders and Steve Bannon back to back I was really thinking is this really happening? Should I pinch myself here just to make sure I'm not dreaming this or maybe I shouldn't pinch myself because I'm on a stage and it's being filmed but it just really felt like something that was not in my bingo card the beginning of the year and it's a sign of just how massive massive the the shift is and it wasn't just Bernie and Bannon we had people across the political spectrum, a bunch of sitting politicians we also had top labor union leaders like we had a speech from the head of the

AFL-CIO, we had top faith leaders like the head of the National Association of Evangelicals we have people who have signed on to this who come from national security backgrounds. We have family organizations. We have basically representation from all of America. One journalist actually was so bewildered by how diverse this was that she asked me, I don't understand, like, these people on this list, what do they even have in common? I didn't think they agreed on anything. So I thought about it for a bit, and then I realized, I know what they have in common. They're human. So if we're debating whether we should create a human-run world or a machine-run world, of course they're going to be on Team Human. Like, duh. And this, to me, is incredibly encouraging, especially when you take all of these things together,

because big political change, you know, you can go sometimes for a long, long time and nothing happens, kind of like you're shaking a ketchup bottle and nothing comes out. And then quite short order, everything comes loose. And I think now this big blocker that people, for example, many people just did not believe that AI could learn to talk properly or that it's certainly not that it could escape human control. I've just stopped saying that because it's already happened. And now they're moving on to, okay, what do we do about this? And the fact that it's suddenly you don't become a social outcast just from talking about things that have already happened means we're having these incredibly constructive discussions happening. I was invited to testify in the Senate also last month, and I've been on the Hill before, but I've never looked out at a room like this before

and seen people so actually really, senators are actually freaked out. This is the source of my optimism that not only are we in driving towards this cliff, this cliff starting to talk about whether maybe we should turn in a better direction, but the pace of change and the sentiment and the mood in the U.S., both among the public and also among very powerful stakeholders from across the political spectrum, has shifted so quickly that maybe we can change paths fast enough. And, yeah, can I just end on a more optimistic note also? Because in this sort of Lars von Trier, Ingmar Bergen spirit, we've just talked about bad futures. You had me list four of them, right? can we end by just talking a bit about a good future also oh definitely I think the most successful

PR trick that the Silicon Valley lobby has pulled off is portray this as being some sort of inevitable path where we just get more and more AI and that's the only choice we have do you want more AI or less AI or if you want to go forward or backward or whatever. When, in fact, just like with every technology, there are so many different things you can do with it, right? It would have been a huge mistake to start arguing about whether you want more biology or less biology and say we can't have penicillin if we don't also sell bioweapons in every supermarket. There we made some choices and said, yeah, we want those medicines to make us healthy. but bioweapons, let's ban them. It's exactly the same with AI. Of course, we all want AI that can help us cure diseases, that can make us more prosperous, that can eliminate boring chores,

dangerous jobs that people actually don't want to do, that help us solve a lot of well-chosen problems that we just really want the solution to, but we haven't been able to solve it ourselves. so much upside there at the same time as a pro-human declaration really drives home there are also so many things with this near total agreement that we don't want we don't want the future to be run we don't want the future to be run by machines or by a 1984 dictator and we don't want the future which is just machine run when we humans all feel useless and obsolete and powerless. We don't want the race to replace where we start by replacing our relationships, AI girlfriend, with AI girlfriends and boyfriends, replace human therapists by AI therapists,

replace human interactions with just 24-7 staring at screens. We don't want to replace every single job with a machine. and we don't want to replace ever more decision-making with machines, and we certainly don't want to replace our species with machines. So there's this fork in the road, basically. There's two roads going forward. On one hand, you have the race to replace that I just articulated, which ends with us losing all our – us getting totally replaced, maybe even extinct. And then on the other side, there's a really quite amazing future. where we aspire not to build a digital god, but controllable tools. And we use these very intelligent tools to solve all these problems that have stumped us. And we create a future that's even more amazing for humans and other life forms here on Earth than anything our scientific fiction authors could have dreamt of.

guy. So let's, so in other words, the question is not, are you for AI or are you against AI? The question is, which of these two paths are you for? The race to replace or the pro-human path? And almost everybody, except maybe a small handful of AI, of Silicon Valley lobbyists who are paid to I think that, and this very small fringe of transhumanists, some of these companies who actually think, who actually want humans to be replaced by machines, near global unanimity, you know, on that we want the pro-human path. That makes me feel very optimistic that that's actually the path that we humans are going to collectively choose. It's the future. It's going to be awesome. Max thanks for coming on the show Thank you