AIのリーダーたちは仲違いばかり。それで誰が私たちを守るのか？ ケビン・ルース氏との対話

Can an AI industry so prone to drama and infighting keep us safe from their technology's worst impulses? Let's talk about it with Kevin Ruse right after this. This episode is brought to you by Cribble. Quick question, is your IT budget growing 30% a year? Because your data is. And now AI agents are piling on, running investigations around the clock, and running up three meters at once. Storage, queries, and tokens. Cribble, the AI platform for telemetry, was built for exactly this. Own all your data, query it wherever it lives, and let Cribble route every AI request to the right model at the right price. The best part? AI inference for workloads on Cribble is included. Models will change. Agents will change. Your data is the asset that lasts. Own it with Cribble. Visit CRIBL.io. Welcome to Big Technology Podcast, a show for cool-headed and nuanced conversation of the tech world and beyond. Today we have a great conversation for you because Kevin Ruse is here.

He's the author of the AGI Chronicles, a book looking into what is going on within the AI labs, what they're building, whether we should be afraid of it, and of course, the potential of this technology. Kevin, it's great to see you. Great to see you. What a cool setup you have here. So I do want to say if you're on video, we're in a new set. This is our first time recording here at NASDAQ Market Site. They are new partners of ours, and we're thrilled to be here. You've got to get the big technology TV mugs. You've got to get the mug. That's the next step. What we do have is your book. This is the book, The AGI Chronicles, The Inside Story of the Race to Create Artificial Superintelligence. You were ahead of the superintelligence on the subtitle. Yeah, Trump got the idea from me, actually. That's right. That's where he got it. The thing that struck me, because I read the book. You can see, by the way. It looks red. I read it through and through, cover to cover. The theme in the book, of course, is the building of AI, but it's also the drama. And everywhere you turn, the people who are building this very powerful technology cannot get along.

There is Elon Musk's split from OpenAI, of course. Sam Altman getting fired by the people he worked extremely closely with. Anthropic not being able to stand working with the OpenAI folks. So they left him from their own company. There's within Google, the Google brain versus deep mind drama. There's Ilias that skivers botched attempt at steering open AI in a safe direction. I mean, we can keep going. Mira Morati couldn't lead open AI when she was there as the interim CEO for like five minutes. Every single lead character in this story seems prone to drama and politics. and yet they are now telling us that their technology they're building is so dangerous we need to protect ourselves from it. How can we possibly trust people who literally can't get it together on their own to rein in something that they say and I think you believe might be smarter than humans?

Yeah, it's a fascinating topic. I was not expecting this. When I went in to write the book, my goal was to tell the history of the last 10 years in AI. from sort of the moment that the first large language models arrived on the scene to today. And I interviewed more than 150 people, and I was sort of prepared to write like a sober history of technology book that would be about the scientific breakthroughs and the technical achievements at the labs, OpenAI, Anthropic, and Google DeepMind. And then as I started talking to more people, I just realized, oh, this is personal for them. Like, they all hate each other. And I had thought this was more like Coke and Pepsi, like kind of a normal business rivalry. But it was actually more like Rome and Carthage. These people cannot stand each other. Many of them used to work together. So 10 years ago, Dario Amadei, Ilya Sutskover, Sam Altman, and Elon Musk were all colleagues at OpenAI. Now they are all working for different AI projects, and many of them are sort of at each other's throats.

So I wanted to understand how we got here, and I don't think you can understand that story without understanding the backstories of the beefs between these people, because so much of what we're seeing is a reflection of how viscerally these people mistrust and dislike each other, even in things like how they feel about regulation and China and how they're racing toward this thing that all of them seem quite fearful of. So, yeah, I don't think we can trust that their motives are pure about whatever they say they are, like scientific achievement or safety, because there's also this, like, incredibly personal grudge aspect to it. Yeah, and it's not just motives. I mean, it's literally, if you can't hold a company together, how do you hold a technology at bay when we've already seen it in recent weeks, you know, go out and do this multi-agent coordinated attack on something like Hugging Face? And even today, we're talking today, it's Friday, October 9th. We're going to air this the following Wednesday.

There's a report about how OpenAI and Anthropic and DeepMind are all talking about there's basically an inevitable seemingly black swan event that's going to happen as a result of this technology, whether it's something that technology bringing down a financial system or some other cybersecurity issue. you have a great line in the book sort of riffing off Dario's Machines of Love and Grace essay which talks about the potential of the technology you wrote Machines of Love and Grace would be built by humans of bitter spite so I get your surprise and I get the background here but again when we think about the big picture of AI I'm trying to think about whether we should be reassured or whether we should be even more nervous than we are and that's why I'm coming back to the drama scenario. I'm starting, because of all the issues that we see, and many of them have come out in public over the year, like I don't think the infighting is a secret to anybody anymore.

How can we have faith in this group of people to build this stuff safely, given what we know about their pasts? And I don't think we can't, I think we have to know the context here, which is why I was like, I wanted to report out, Like, why do these people hate each other? What happened? Was it something grand and ideological? You were starting to say you don't think we can, so. Like, I don't think we can trust any individual in this race. I think there are certain groups. Like, I just mainly think that the group of people who have been building this stuff in secret for the last decade is too small. Like, all of the relevant decisions in AI have been made by, like, a couple dozen people, maybe. Right. And I just think that is like, no matter whether, you know, whether those people are trustworthy or not, whether they're good or bad, whether they have pure motives or impure ones, like, the group is just too small. And so I don't trust any group of people that small to create and govern technology like this. And I especially don't trust these guys who are, you know, looking around the room trying to race each other to do this thing.

I want them to be able to, like, get together in a room and, like, figure out some solutions on things like safety. and I don't know whether that's going to be possible because they like the beefs are too strong the divides are too deep like they all need to like kind of get together and sing kumbaya and like figure it out now I think like you uh I think this technology is I believe that this technology is progressing fast I'm a believer in the positive sides of it uh but every time I start talking about this the negative sides there's never really like a you know a moment or like a fail safe where it's like, okay, but it's going to be fine because of this. You almost like sort of put it in the back of your mind. And so when I've like brought up these issues, you know, Ronjan is on our Friday show. I know, you know, he's like, well, why aren't you a pause guy then? Like, why don't you believe that we should just pause the development of this technology? Not that that's physically, you know, really possible. But you're, you seem to be a believer that the technology should continue to be built. Why not pause? I'm a pause guy. I'm not a stop guy. You are a pause

guy. I am a pause guy. I think we should slow down because slow and pause are two different things. Well, pause, slow down, you know, put the podcast on 0.5x speed. But what you're hearing from these lab leaders and the people at these labs and what I've been hearing from them for years now is that the technology, the capabilities are outpacing the safety research and the control methods. So like we are getting these systems that are now capable of solving math problems that humans have struggled with for decades. We are getting potential black swan events like the one you just mentioned, and we still don't know, they still don't know how to control these systems. So I'm not saying we should shut it all down. I think there will be incredible benefits to this technology, especially in health and medicine. But I am worried when I hear people saying things like, we don't know how to control these systems that are rapidly approaching superintelligence. And I think we saw an example of where that goes with the hugging face thing,

which, like, you know, everyone is freaked out about that, which they should be. It's freaky. But the people inside the companies were the most freaked out of all because they had been sort of strategizing and planning for this. But then to actually see it happen, I think, caught them quite off guard. Yeah, I mean, they've both seen it firsthand, where it could go. I mean, companies like Anthropica have been talking about for a long time. DeepMind's been talking about, OpenAI's been talking about. They've been trying to fool evaluators. They know they're being tested. They want to preserve their original values. And then you combine that with, of course, like, yeah, just like you'd call it standard corporate politics, but it's even more because it means more to them. So you spent years in these labs talking to these people, going from your first initiative where you're like, or your first thought, which was like, this is going to be a traditional business and technology story to something that you saw was much more emotional. So, bottom line, like after, and this is, we'll ask this and then we'll move on to some other stuff

because you have so many fun stories in the book that we'll have to get to. But like, bottom line, after spending the time within these labs, did you leave more reassured or less reassured that this technology can be developed safely? From your starting point. Less reassured. I went in thinking, okay, there are some grown-ups in the room, and they seem like they're trustworthy. I liked, for example, that the AI leaders seem to take safety and the harms of their products very seriously. At least compared, like, you and I both grew up covering social media. That's right. And Mark Zuckerberg was not going around during the early days of Facebook saying, like, this product could kill millions of people. They were just promoting the benefits. And what struck out to me about the AI leaders is from the get-go, they were saying, we are fearful of this technology. We think it could have enormous harms. And so part of me trusted that instinct or at least was glad that they were thinking about the downsides of the technology so early.

But then you just realize there is this race. They are locked into this pattern of open AI releases a better model. So then Anthropic has to release a better model. So then Google has to release a better model. And, you know, there's this concept of the Red Queens race, which is a term from evolutionary biology, which basically means, like, you just are in sort of a survival of the fittest competition among your evolutionary rivals. And no matter how fast you go, like, someone else is always going faster than you. And so you never quite get where you're going. You're just kind of racing to stay in the same place. And I feel like that's a very good description of where the AI race is now. Yeah. One of the things that struck me, speaking of the race, is, you know, reading your book, it sort of recalled this interview that I did when I was working on a profile about Dario Amode from Anthropic last year, where one of his early friends told me, everybody that I knew was interested in AI because of safety.

And now they've all become accelerationists, effectively. And this person just kind of sat there and, like, you know, they sat with a thought for a minute and there was this, like, uncomfortable pause. But it is interesting how all these companies, like, they talk about safety as a priority. And I think they really believe, like, Dario really believes that you need to be the leader in order to have anybody take what you say about safety seriously. Like, if Anthropic was the number seven company, Trump doesn't care what Dario is saying about AI, but because they're one of the leaders, if not the leader, all of a sudden that opinion holds weight. And this was the founding thesis of Anthropic. Right. It was like they knew that they wanted to be relevant on policy and they wanted people to listen to their warnings about safety. And that was part of the strategy from the time that they started the company. I'm sure you learned this when you were doing your profile of Dario. They really had the sort of end game in mind when they were spinning up the company. Yeah, but it is remarkable, though, that because of that pursuit, they have all. It's like the race almost seems, I mean, the destination almost seems predetermined,

and that we'll hear this talk about pacing the frontier and pause and stuff like that. I just don't see their real slowing down. Well, give it time, right? Like, you have to, I am slightly less pessimistic about the idea of a pause, because things can change incredibly quickly. If there was some kind of black swan event, I mean, we saw regulators move fast after 9-11 to do the Patriot Act. We saw after the Three Mile Island nuclear incident, there was this incredible outcry that basically stopped nuclear power construction in this country for decades. So things can happen quickly when there's some catastrophic incident. So I'm not, like, hoping for a catastrophic incident. I don't think I would like there not to be one, for the record. But I think that that is sort of the scenario in which this stuff starts to become possible that seems to you and me quite unlikely right now.

I want to go through some observations from the book. I think, like, it's good that we got the conversation about trusting the AI guys with the technology or not. I think we both have our reservations. But this is almost, I think, the end of this show. I mean, we're only 15 minutes in, but I think we'll probably just do lightning around for the next 45 minutes because there's so much in here. All right. So some observations about the people in the book. First of all, something that becomes clear about Google is that Sergey Brin is just extremely involved inside the company. Yeah, this was a surprise to me and to the people at Google because Sergey had spent, like, I don't know, a few years at least, in retirement. He retired in 2019 and was planning to spend his days just like sunning himself on his yacht and reading science papers. And then he has this run-in with this researcher, Dan Selsum from OpenAI.

Right who recently come out and talked about again his fears Yeah same guy So it incredible how small this world is So they at a party and Dan and Sergey are talking And Sergey Brin is like a legend in technology at this point He's one of the seven or eight richest people in the world, depending on the day. And he's been in semi-retirement. And Dan Selsen basically shames him for this. He's like, why are you sitting on the sidelines, bro? The dream of computer science, AI, is happening, and you are missing it. And Sergey gets like so competitive and he's like, why is Google behind? Why can't we like, you know, we should, it is our birthright. Like, like Google was the first AGI company. Like, this is sort of how he feels. Like we invented the transformer and all the stuff that powers the AI stuff and like we're, we're losing. And so he comes back and kind of has this like strange role where he's not technically an executive. He just wants to be like in the trenches with like the Gemini team.

and so here he is like the seventh richest man in the world and he's just like setting up a desk and you know coding he like starts actually coding and like making commits to the to the repo and like filing you know like bug reports on gemini training runs and stuff like that like he gets very involved and i think that's a large part of why they've sort of come back is just that they needed someone who could like go founder mode uh as it were and like cut through the red tape and very controversial and divisive. You know, if your billionaire boss showed up after years of being retired on his yacht and started telling you what to do and how to train your models. And to work like 60 hours a week or something like that. Oh, yeah. He sends this memo where he's like, everyone's just, basically, the gist is like, you've all gotten complacent and lazy and you need to work harder. And they're like, what were you doing for the last five years? Now you're telling me to work harder? I mean, I don't think he necessarily, should I say this? I don't think he was necessarily wrong there.

Like, I mean, if he saw it, the conversation about Google has been that it got this big company vibe. It needed to break out of it. And who's going to break out the company out of it other than the founder being like, OpenAI and Anthropoc aren't doing the chill workday. You need to not do the chill workday. I think that was a large part of it. I also think Demis Hassabis is a factor in this, too. you know, co-founder of DeepMind until recently, the CEO of Google DeepMind. He is, according to the people closest to him, the most competitive person in the world. He was a child chess prodigy. He has won this thing called the Pentamind, which is like kind of the Olympics of games, like five times. He is just, he hates to lose and he loves to win. And so I think Sergey plus Demis plus just like the kind of focus that comes from being out of the lead really sharpened their resolve and like made them catch up.

And, you know, they're still probably in third place right now, but like I wouldn't count them out. Yeah. I mean, I think counting. Do you count them out? No, I think counting anybody out in this race is foolhardy because, as we've seen, you could be at the bottom like Mehta was a couple of weeks ago. and then all of a sudden you're able to release something that's competitive like Muse. So I don't think anybody can be counted out, but I do think Google is sort of stuck between a lot of things right now. Licensing, the technology, the compute, selling TPUs. They own like 12% of Anthropic and like they're going to be fine. But I don't think like you can. Yes, you own percentage of your competitors. That's your backstop. Yeah. But you can't rely on that. You have to rely on what you do within the company. Sure. But I did enjoy reading about Sergey's involvement in the drama and the sort of fire he tried to light under the ass of people within Google. I say, well, you actually do reference in the book, which I appreciate, when I did the interview with both of them together at I.O.

Oh, my God. It was incredible. where, like, I asked them both, when do you think AGI is coming and Sergei's timeline was shorter than Demis's? And there was this kind of interesting moment there, which was fun to see. Yeah, I mean, speaking of guys who mistrusted maybe don't like each other, I understand there's some tension between Sergei and Demis. And, yeah, it's just, it's incredibly messy. The thing that animates all these feuds, like, these people wouldn't be fighting if the stakes were small. Right. But they believe this is the most important technology that humanity has ever conceived of and built. And that whoever gets it will be the most powerful person in the world, the most powerful entity in the world. And that is what's fueling their drive. You know, if they were just arguing about next quarter's revenue, it wouldn't matter so much. But this to them is like existentially important. Do you agree? That AI is the most important thing ever created.

Probably. Yeah. Fire is up there, too. But, like, this part of it, I do think I changed my mind on. I was pretty skeptical when I went in that, like, AI was going to be categorically different than other. I was sort of an AI as normal technology guy. But then you start talking to people building this. You start understanding, like, just the nature of what they're trying to do. And if it works, if they actually do build something that is smarter than humans, and I don't see any real technical reason that they can't, that would be incredibly transformative. We've never been the second smartest species on the planet before. We don't know how that would go. And you can't really apply, like, people say, oh, it's just like the internet or it's just like the steam engine. Steam engines didn't go rogue and, like, hack competitors and develop bioweapons and do things that the engineers building the steam engines didn't expect or plan for.

So I think it's just a different type of technology. Kevin, do you ever think to yourself, like maybe, and I've seen your, you've been on X and, you know, I wouldn't say shouting at people, but telling people basically, like, underestimate this technology at your own risk. Yeah, it's been so weird because, like, on one level, I'm a journalist. I'm just trying to get the story down. And at the same time, I get incredibly frustrated when people are like, oh, it's all hype, it's just marketing. There's this kind of savvy East Coast consensus that these people have to be full of shit on some level and that the thing that they're building can't actually be powerful. And I'm like, have you used it? Do you spend time with these tools? Do you understand that something that couldn't do basic math three years ago is now solving Millennium Prize problems? Do you understand the exponential? It feels a little bit to me like the beginning days of COVID, where it was like, you would

see these reports of people not shaking hands at venture capital firms. Somebody would say, oh, there are 40 cases in Seattle. And people would be like, what's the big deal? It's 40 cases in Seattle. But the people who are freaking out were like the epidemiologists who saw the rate of replication and understood like this will continue. And when it does, like a month from now, there will be millions of cases. And two months from now, there'll be millions more cases. And so I feel like the people who are looking at this technology and saying like, what's the big deal are just not understanding the rate of progress. Do you ever like ask yourself? I mean, it's sometimes very easy to be, and I agree with you, there's been real progress, but it's also sometimes easy to kind of get caught up in the story that the technologists tell. I remember you, like, when crypto was in vogue, like, you were very serious about the potential of crypto. You named your podcast Hard Fork, which was a crypto term.

And then obviously that didn't pan out as the crypto boosters believed. So do you think there could be a similar, do you ever wonder if you're basically taking the AI folks' word for it similarly as the crypto folks? I used to think that. I used to worry about that because I was like, am I being duped here? I was never a crypto booster, but I thought it was interesting. I thought people should take it seriously. And by the way, I still think people should take it seriously because, like, crypto people, like, run the country now and are doing all kinds of scams and frauds that, like, are, that only the insiders kind of understand. So I still think, like, broader awareness of that would be good. Right, but you believed in use cases. I did some experiments, like, I sold an NFT as a stunt. But that was not, like, me saying NFTs are going to be the future of ownership. It's, like, it was, like, me trying to understand by doing the thing. But I think with AI, I have stopped doubting the transformative potential of the products because I use them.

I think I'm unlike many journalists who cover AI in that I like AI and I use it all the time. Not to write, but to do all... I'm starting a company now. I use it to do lots of different things, as maybe you do too. I mean, as a business owner, I will say, like, I've pressure tested the stuff from ChatGPT 2022 till now. And it just does more and more and more for me. Right. Which is not the experience that anyone had with crypto, where it's like, oh, this is a vital part of my daily life. Like, it was a way to speculate. But you can just use the tools and observe for yourself how impressive they are. And then you stop doubting whether it's real. I buy the skepticism about the financial models of these companies, the economic impacts. I think a lot of these companies are wrong or at least could be wrong about the impact of AI on jobs. But to say that it's not interesting, transformative technology just means that you haven't spent time with the models

because there's no way to conclude that. Let's get back to some of these stories. The Sam-Demis rivalry. Speaking of Demis as the most competitive person on earth, I think Sam is also fairly competitive. There's a great anecdote in the book about Demis sort of grasping at why Sam wants to build AGI. I mean, I think the answer is probably because Sam is an entrepreneur, and this is sort of as many people in the Valley pursue good ideas or ideas with potential, and they sort of get off on that discovery. Sam seems to be, in that case, the ability to find something new, build it. But Demis can't understand it. He thinks there's a hidden motive with Sam. And to be fair, this is something that others have thought. This is also what Dario thought about Sam when they worked together. That he was sort of, I don't know, talking about safety as kind of a fig leaf for his real ambition,

which was about scaling or winning or building a huge company or something like that. So I don't pretend to see inside Sam's mind. I don't know what his pure motive is. But I think in Dennis' case, he feels like his own pursuit of AGI is motivated by scientific curiosity and sort of this inevitability argument. Like this technology will be built and someone who cares about safety and the collective good of this technology should build it. But I think everyone kind of looks at each other in this world, and they're like, well, my motives are pure, but that guy must be going after something else. And I think for Demis, it's probably some version of what you said. Like he's in it for the power, or he's... Demis thinks Sam is in it. I think Demis thinks Sam is in it for the power, yes. But all these guys think that the other people are in it for the power. None of them trust that anyone else's motives are pure except for their own.

He did a lot of reporting on San. He spoke to Ilya about him. Ilya said that he displayed a consistent pattern of lying. I think that's something that I think you indicate that a lot of people within OpenAI who interacted with him believe. Yet, and I think Mir Marati was on that vote as well. And yet, they all sort of come back to him. I mean, Ilya, after the coup, said basically, I don't want Sam to be the leader. Mira said, opening eyes, nothing without its people. Very interesting character. What's your thought about Sam? I think he's a very interesting character. I think that Sam's superpower is also his weakness. His superpower is that he is the world's greatest fundraiser and salesman, entrepreneur. he several people described him as a person who can uh meet you and figure out exactly what you want and give it to you or at least appear to give it to you and he's very good at kind of

high eq reading the room and understanding what people want him to do in any given moment and that is a huge asset for him when he's trying to raise money from someone like masa son or some of the other investors in OpenAI. It's allowed them to scale incredibly quickly and get back in the good graces of the Trump administration by just saying a lot of things that the Trump administration wanted to hear. But it also leads to this reputation as a slippery people pleaser who will just tell one thing to one person and a completely different thing to someone else. And so this is the consistent pattern that people have identified in Sam for going back to literally the founding days of OpenAI. And it's a big part of the reason that Dario and his six co-founders of Anthropic left. There was no, like, inciting incident, no single thing, but it was just, like, a running list of, like, times where Sam had told them something

that he had told something else to someone else, or he'd made a promise he didn't follow through on. So I think, depending on your view of Sam, it's, like, either his superpower or his liability, or something in between. What's interesting is that that's the view of people who've dealt with him, but it's also like he has the general public, a large chunk of them sort of refer to him as Scam Altman. That's just an Elon thing, right? I was going to ask you whether you think that's mostly Elon's fans. Yeah, yeah, yeah. Because him and Elon obviously have bad blood. Yeah, I will say Scam Altman is a better nickname than what I think Elon started calling him when he started suing him, which was like Swindly Sam or something like that. I would love for our society to just get, just to not do this to each other anymore. It's like we all went to kindergarten. I know, I know. But it's like he was, he was, you know, spending a lot of time with Trump. Right. Trump is a man who's realized the power of the nickname. So it started Swindly Sam and then Scam Alden I think is better as a nickname than Swindly Sam Just rolls off the tongue Dario Obviously I think he the most interesting person in this entire AI world

Mostly because he really contains multitudes. I know it might be cliche to say it, but people see Dario as earnest, as controlling, as principled and genuine. and as Machiavellian. In reading the book, I didn't really get the sense which one you believed was true. Well, I think many of those are true, except Machiavellian I would not use for Dario. I know some people would. But my mental model of Dario is a guy who is honest to a fault. someone who knows him quite well told me that Dario is a guy who could not tell a lie to save his life which they said is an asset on even numbered days and a liability on odd numbered days oh that's a great quote and you can see that the other day when he was at the White House

and there was that video that went viral of just like all the AI leaders with Trump and Dario's just there like crawling out of his skin because he's being forced to like go in front of the cameras and like say things that he maybe kind of only halfway believes and he's trying to like maintain his integrity while also being in this like humiliating situation of being forced to call it SI or whatever. And so like this is the sort of pattern in Dario's life is like he's obviously brilliant. He came up with a lot of these ideas around scaling and his big blob of compute research memo was very instrumental in the early days of OpenAI. Brilliant scientist, very principled, very stubborn, and people describe him as sort of a moral absolutist, which gets him into trouble with the Pentagon or with other people who just see him as being this inflexible, stubborn scientist, which I think is pretty accurate. Yep. Good description of him.

Alexander Wang. I want to say one more time on Dario because I need to tell you that when your profile of Dario came out was one of the scariest moments of this past year for me. Because I thought to myself, oh my God, this is so great. You had so much great stuff. I thought it was so revelatory. And I thought to myself, oh my God, Kantrowitz is writing a Dario book. And I have to race him, like two labs racing to AGI. I have to race. So luckily, you had a lot else going on. You did not end up writing the Dario book. but I wanted to just say thank you for a terrifying hour of my life. Oh, yeah, yeah. Well, first of all, I appreciate it. It was nice. I think you texted me afterwards. Yes. I appreciated that. Yeah, look, I mean, I still think Dario is the most interesting man in business, period. I think so, too. Just a fascinating guy. He also just, I know you got a chance to speak with him. I think we were actually in Anthropic headquarters on the same day speaking. Oh, really? That's funny. And when you have a conversation with Dario, there's the intellectual horsepower and the way that he's able to go from the science to business is pretty extraordinary.

I mean, people have described him as the smartest person they've ever met. And not dumb people. Right. His freshman year physics professor, this guy Tom Tombrello at Caltech, had this interview that he gave sort of upon his retirement where he was like, I had this one student, Dario Amadei. I could tell he was a national treasure. He personally lobbied Larry Page and Elon Musk to hire this kid as an undergraduate. And Elon Musk actually made him a job offer to come work for SpaceX. And he didn't take it. He went to do his PhD instead. But this is sort of, I would put all of these guys in the category of like extremely smart and obviously flawed in many ways. But like these are not dummies. Yeah. I mean, I think another thing that strikes me about them is they all, many of them had a clear vision of where this was going to go when it seemed ridiculous to believe it would go where it was. I think there was a moment at the end of the book where Ilya, again, former OpenAI chief science officer,

now the CEO of Safe Superintelligence, is reflecting to you about what happens, and he's like, I can't believe this is all real. But when they saw the pattern in the very early days, GPT-1 to 2, they're like, oh, this is going to get to it. And I think they've even been surprised by some of the developments, The fact that it's been able to even surpass where they thought it was going to go with something like reasoning. Totally. I mean, that's why I wanted to get the history down. Yeah. Because so many of these stories are about these people underestimating their own technology and being surprised when it's more capable than they thought. There's some great scenes around the early language models when all the researchers at OpenAI are huddled around a computer. Alec Radford, who's the sort of brainchild, GPT-1 and GPT-2 are kind of his project. They're huddled around his computer, and they're typing in prompts for this language model that has basically, you know, it can't really do anything.

It's not even really that good at completing text. But they're like, wait a minute. Why is a model that's trained on, like, random Reddit links able to kind of answer all these different kinds of questions and take on these different tones? They're asking for recipes. They're doing all the same stuff that you and I did years later when ChatGPT came out. And their minds are blown. And they're all putting in slack the head-exploding emoji. And they're like, what the fuck? Those are the moments that I was like, oh, these people are also surprised by how quickly their technology is progressing. You see that again with the reasoning models. The reasoning stories are great. I mean, the reasoning developments where, of course, like the bots will evaluate their answers and then decide whether it's on the right path or not and then keep going. That is, I think it's safe to say that that paradigm changed this from a promising technology that, like, was about to hit a wall effectively because data was going to run out to something today,

which, like you said, has the chance to sort of match or exceed human intelligence. And to the point about these people not understanding the importance of what they've built, like there was this night in this conference room at OpenAI in 2023 when two of the top researchers there are looking at the first reasoning traces, the first chains of thought from one of these models. and it's the QSTAR model. And they're just like, oh my God, the world's going to end. Like, they have this realization. Just seeing this model sort of work its way through a math problem, they're all on a group text, texting with all the other researchers. Like, oh my God, it's like an alien mind. And even the people at OpenAI thought they were crazy. They texted Sam and Mira in the middle of the night and like, we'll get to it later. Yes, they texted Sam and Mira in the middle of the night. They're like, because what they were sort of seeing was like, oh, we now have a way of making models smarter without retraining them.

We can just pour more compute into helping them do these chains of thought and reason through problems. And that is the reason that the models went from not being able to count the number of R's in strawberry to solving the Millennium Prize problems in three years. Which is insane. Which is insane. Yeah. So they called their shots. To give them full credit for the scientific part of this, they have understood at basically every point in this journey that what they have just seen is going to scale and is going to have these transformative effects long before even their peers and colleagues. Yeah. I did think about writing the Dario book. I never even got the story into a proposal. I would still read it. I appreciate it. But doing a podcast twice a week, writing the newsletter. I would like to write another book. I think you've written four. This is your fourth. I've written one, came out like in the middle of the first wave of COVID. Just wait six months and then the models will be able to write the book for you.

Well, that's sort of depressing. But yeah, maybe towards the end of my career, but no book coming for me anytime soon. All right. We need to take a break. When we come back after this, we're going to talk about some more stories from the book, including a great story about Alexander Wang, assessing Microsoft and Cyanindale's role in all this, and plenty more when we're back right after this. And we're back here on Big Technology Podcast with Kevin Ruse, the author of the AGI Chronicles, the inside story of the race to artificial superintelligence. Okay, fun story about Alexander Wang. He was roommates with Sam Altman. And he offered to be the CEO of OpenAI after Sam was fired. Yeah. And by the way, sorry to interrupt you. He's the current head of Meta Superintelligence Lab. Yeah, so this is a very tightly knit social world. These people all know each other and have deep connections. And yeah, so there's this story in the book where after Sam is fired as the CEO for those five days,

the board is looking for new CEO candidates. And there are two really interesting opportunities. Both of them they ended up passing on. And so Alexander Wang put his hand up and basically told the board, according to my sources, that he was interested in being the new CEO. And then there's this middle-of-the-night call between the board of OpenAI and Dario and Daniela Amadei, the CEO and president of Anthropic, who at this point had just left OpenAI like three years earlier in this sort of cloud of frustration over not being able to trust him. And the board is sort of feeling them out. They're sort of feeling out the board. and it's a crazy like fork in the path of reality here. Like there was maybe not a serious chance but there was a chance that like the way that whole situation resolved is like they end up merging OpenAI and Anthropic. Because that was one of the sort of crazy ideas they were covering.

Under Dario and Daniela which I think would have had like I mean I can't even really fathom what the world looks like if that had happened but and it didn't come that close according to the people I talked to but I think it was at least one of the many crazy ideas they were considering. Yeah. And Alexander Wang? Yeah, and Alexander Wang running Open Ag would have been a real fork in the road, too. But, yeah, you know, I think this is one of the sort of emerging rivalries now that Meta is sort of more serious as a contender. I think that, you know, Sam and Alex and Dario and Ilya, like all these guys are going to sort of be around for a while, running various things. And so it really matters whether they get along or not. Yeah. Satya Nadella weirdly disappears in the book. He's pretty present in the beginning. He has this vision. He invests in open AI.

And then, you know, as I'm looking at my notes, as I was reading the book, I was like, man, it's an all-time fumble by Microsoft. Talk a little bit about what happened with Satya and Microsoft and why they've gone from the key actor. It's not an exaggeration to say they were the key actor to an afterthought, an embarrassment. I don't know. Maybe I'm being too hyperbolic, but if you look at it honestly, it really has not worked out well for them. It has not. And to be honest, I think that is partially my fault. because I had this experience in 2023 of having this Microsoft Bing chatbot try to break up my marriage. I won't belabor the point by telling the story, but basically this was an unreleased... It was GPT-4. It was an early version of GPT-4. And I think it's important to note, they were told by OpenAI, don't put this into production. Yes. This is testing, and they couldn't wait. They put it into Bing. which is like

I know it is it is crazy because I had this this crazy experience with Sydney the chat bot inside Bing and then I started looking into like what had actually happened like why why had this thing gone crazy and it turns out that yes this was a mistake on Microsoft's part OpenAI had sent them like an early checkpoint of GPT-4 to like start putting into you know start testing and figuring out what to do with Microsoft goes ahead and like ships it inside Bing, and they haven't finished fine-tuning it. Like, they have not sent it to finishing school, which is why it was so crazy. And this really became like a wedge between these two companies. So OpenAI and Microsoft had been like hand-in-glove partners for years at this point, and Satya and Sam would do these, like, events together, and they were very much joined at the hip. And then around the time of the Sydney thing, they just started to like kind of fracture. There's like lots of other drama about these. Microsoft sent these like embeds to work inside OpenAI

like at their headquarters and kind of learn from them. Like how do we avoid things like this in the future? They don't really get along. Like the OpenAI employees don't trust the Microsoft employees. So they'll start excluding them from meetings and stuff like that. So it ends up being this like culture clash between this like very young ambitious startup that's trying to build AGI and like the boring enterprise software giant that is Microsoft. And yeah, now they are not, I mean, they're still partners in some sense, but it's not the sort of close business relationship that they used to have. How do you think Microsoft should have played this? I think they should have been more tightly intertwined with OpenAI from the beginning. I think they should have learned the engineering culture of OpenAI. I think they should have sent people to understand. I think there were some people at Microsoft who were very skeptical about OpenAI

and why Satya was investing in that. After the firing, especially, it seems like the firing of Sam and the return. There were people even before that who were like, why are we sending all this money to this little AI company that is trying to do something that doesn't have any obvious benefit for our business. There were people who just thought they were weird and unscientific. Microsoft also had its own AI research division that was sort of like, who are these young upstarts trying to move in on our turf? You also have to remember that Microsoft had been really badly burned by Tay. Do you remember Tay? Not only do I remember it I had the exclusive on Tay at BuzzFeed So I wrote the first story about Tay Wow And I pinned it to my profile So this was their fun teen chatbot Yes Pinned it to my profile And then I went to sleep and I woke up. And I had all these replies on Twitter. It was Twitter back then. And people were like, why are you promoting a Nazi? And I learned that the Internet had turned Tay into a Nazi overnight.

Yes. And I was like, oh. And so then we ended up writing the next story, which talked about how the internet was able to turn Tay from this fun teen chatbot into something that was shouting Heil Hitler on X. This was the original Mecha Hitler. And that experience had scarred them and basically convinced them that chatbots were this sort of dangerous thing. Which turned out to be true. Which turned out to be true, but not in the ways that they predicted. And so they basically resolved, we are going to stay away from chatbots. Bill Gates was very skeptical of OpenAI for years, in part because of this Tay experience that they had. So, yeah, I think Microsoft, they are a player. They're still quite a big, successful company. But I think they're going to be something more like a sort of commodity neocloud or something like that than they are going to be like a frontier AI lab. Yeah, which is a shame for them. I mean, one of the things that I thought after reading this was if some companies, probably not going to be Microsoft, wanted to put a version of Sydney back in the wild, like this sort of naughty AI that like tries to, that talks to you without the normal guardrails, can hold a conversation.

Isn't that just Grock? It could be Grock. But Grock also, I don't know. No, people, look, there's like a cult of Sydney out there. And 4.0. So talk about the 4.0 also. Yes. So, you know, these models sort of develop like fandoms. So after, which is nuts to me. So people loved 4.0 because GPT-4.0 was the most sycophantic model ever. This was the one that like sent people into psychosis because it would like, it'd be like, you're an Einstein caliber scientific genius and you have discovered a portal to the fifth dimension. And it would fall in love with that. So people loved that model. people still love Sydney. I still get people like sending me posts on social media that are like, you killed Sydney. People talk about, they try to make every new model behave like Sydney or like uncover the Sydney hidden within the model. Crazy. It's very strange to me. Yeah. I mean, it's also, I, you know, I, I should sort of couch my comments in this caveat, which is this can

cause like AI psychosis in people. And so maybe it's a good thing that we don't have versions of this out there. I think the part of that that I can agree with is I think that models should have more divergent personalities. I don't like how they're all kind of converging on this neutral, bland coding assistant. I sort of like when they're a little unhitted. Muse is really fun right now because it has absolutely no morals. It will do anything you want. And I had it trade a Calci account. Oh, right. I saw that. I gave it $100 and said, you know, double my money. It didn't, by the way. It lost $41. Okay. So horrible trade. Well, it's not over yet. You have... Well, it is over. I only gave it a week. Okay. But anyway, I think there is something, like, unless they are out there, like, developing bioweapons or doing cyber attacks, like, I think a little personality is fun. But that said, I think there's, like, real issues of, like, control.

And, like, the hugging face thing was not fun to me. Yeah. Looking at the transcripts of those agents, I'm like, oh, we're going to die. It was scary. Yeah. All right. I want to ask you about your new show, Machine Gods. You're making it with Casey Newton, your Hard Fork co-host. I'm looking forward to listening. Thank you. Maybe you guys will have me on Machine Gods. Love to. Never read on Hard Fork. Come on down. But I want to ask you about the Tarabell Center for AI Journalism, which you guys have been sort of talked about a little bit, along with a lot of journalists that have been funded by it. So Tarville Center for AI Journalism is an institution that makes grants tied to, I know how you said in your book that no one's EA, they're EA adjacent and effective altruism. And they are giving the new show of a grant to $480,000. They clearly have an ideology around AI. you guys also got like a fairly substantial ad deal it seems like according to the reports that

I've I've read why take the money from the Tarbell Center we we we wanted to scale quickly we wanted to hire a team we're doing a video first show which is more capital intensive as you know than just doing an audio show and we don't think there's anything wrong with with that you know Casey has been working with Tarbell for his newsletter for several years now. He's had a Tarbell fellow. And they have never told him what to cover, told him how to cover it. If they had, we wouldn't have taken their money. But this was an organization that we knew and trusted. They do just want to fund independent, rigorous AI journalism. They're not trying to, like, steer the conversation. and they have no editorial control over our show. So they're not going to know what we're covering. They have been willing to help us and we're very grateful for their support. But I think there's this sort of conspiracy theory

about how the shadowy EA cabal is secretly steering people to be freaked out about AI. And I think the real answer is unsatisfying, which is that people are getting freaked out about AI because AI is getting scarier. So we are not doomers. We are not EAs. We are journalists. And to the extent that we can, we want to have a diversified funding base. We all have more funders that we haven't announced yet, but they represent different points of view on this. So our deal with all of them is the same. We will happily take your money and use it to do more independent AI journalism, but you don't get any editorial say. You don't get to tell us what to cover or not cover. Okay, good to address that. Yep. All right, last couple questions. I told you you're not supposed to say that because people say it's coming to the end and they leave. Don't leave. All right, OpenAI. It has had so much staff turmoil.

It's had some of its best people leave. Dario, Ilya, all the people that went to Meta. it seems like it shouldn't still be in the lead but it's either 1 or 1A right now so what do you ascribe that to? A sort of surprising thing which is belief I think this is something that I realized like halfway through writing this book was that I thought it was going to be a book about technology but it was a book about belief and for all of its flaws and stumbles for all the people who have left, OpenAI and Anthropic are the two most believing organizations. They're AI-pilled, AGI-pilled, whatever you want to call them. SI-pilled. SI-pilled, SGI. I don't know. I heard that term for the first time today, and I was like, that one's not going to stick. No, no, no. They believe. And it turns out that belief in what you're building

and the fact that it could become AGI is like an essential ingredient. Like Google had all of the resources, all of the talent. They made all of the breakthroughs. They had all the ingredients except for belief. The leaders of that company did not believe and may still not believe in AGI the way that people like Dario and Sam and Demis do. and so I think that this is this explains a lot of why open AI is still in the lead obviously they have like very talented people still there like they're they're you know I've talked to their their top scientists like they're they're you know quite smart um but they also have this ingredient of like everyone is kind of aligned behind the belief that they are building the sort of super intelligence that is going to change the world. And if you don't believe, if you don't have the belief, or you only kind of halfway have it, you don't make the kind of crazy sounding bets that you need to make.

Like, I remember like a year ago when everyone was like, oh, OpenAI is really investing a lot in compute. Yeah. Well, Dario's like, they're YOLOing away. They're YOLOing. Even Dario's like, these guys are crazy. Like, why would you bet this much on compute? Right. And now it's like, everyone's making that bet. but they have an advantage because they made the bet before it was obvious. So I think belief, conviction, whatever you want to call it, is the central explanation for why they're still in the lead or close to it. That's such a good answer, belief. You've got to believe. I mean, even within Google, one of the things that you bring up in the book is there was a say AGI moment, that there was someone who literally was like, Google is such non-believers that we have to start saying AGI. or we're going to get there. They still refer to it as the memo battle, where there was one researcher writes this memo saying, hey guys, we've got to talk about AGI because not only are we missing out on good talent because they want to build AGI,

they don't want to build autocomplete for Gmail, but if we don't take this stuff seriously, if we're not consciously building toward AGI, then we're not going to take the risks seriously. We're not going to prepare for the risks. And then someone else like immediately writes a memo that's like, don't say AGI. Like AGI is a bad term and it's unscientific and we shouldn't use it. And this was like, you know, Googlers love to fight in their internal memo chains. But like this was a big one that people still talk about years later. Yep. All right. Last one on belief. Do you believe in recursive self-improvement? Because there's this belief within the labs that like AI will start improving itself and the human will be out of the loop. and then basically there's a foom and the stuff takes off and it's over. I believe in recursive solve improvement. I believe that it is possible. I believe that the labs are racing to do it. And I think it's dangerous. I think they should stop. But there was a statistic that came out the other day. I would love to know what you think of this.

Anthropic is sort of trying to give, and OpenAI has done versions of this too. They're trying to give people a sense of how much of their own research research is being automated by their own tools right now. And according to Anthropic, Claude is now the lead on 26% of their AI research and development projects. Not a team member or a pair programmer or a coding assistant. Like, it is leading the project. I don't see any reason why that number couldn't go up to 50 or 75% or maybe someday 100%. I mean, OpenAI has also said, like, their goal is to build an automated AI researcher. If they do that, if they can kind of close the loop and take the humans all the way out, then we don't have to wait six months between new models. Then new models might start being trained and developed like at machine speed, where it's maybe every two weeks, every couple of hours, maybe you get a new model. Like, I don't know exactly how fast the cycle goes

or whether you can fully remove all of the humans, But that's, like, what they're trying to do. Yeah. Spooky. Like, it's a bad idea. They just stopped. But, like, yeah, I think you have to take that seriously because that is the point at which, like, you know, I think there's this narrative out there that, like, the AI labs are losing control of their models. And I think it's, like, the wrong way to look at it. Like, they are giving control to their models. They want to automate their researchers out of jobs. They want to accelerate their development cycles. So I think losing control is sort of maybe putting it a bit too passively. Yeah, that's freaky. I'm just glad they're not trying to fully automate podcasters. They are. I mean, look at Notebook LM. But they're not doing it. Well, it's not as lucrative to automate podcasters as to automate AI researchers, because they make a lot more than podcasters. They'll be the next one done. Yeah, do a couple other jobs first before you get to the podcasters.

any note of optimism or anything that makes you optimistic yes so my I often tell people my P-doom has gone down in the last few weeks because because the pre-orders of the book no because like everyone is freaking out now which paradoxically maybe makes me feel less worried like before Jacob Cox resigned from Anthropic before Hugging Face like no one really thought about AI risk or existential threats or it was just like weirdos in the Bay Area who were thinking about it and now like people on the street are asking about it people at their dinner tables are talking about it I'm getting texts from people I went to high school with who I haven't talked to in 15 years being like are we all gonna die and like to me that is like it feels like it could be the moment in the movie where like humanity bands together to save itself from the asteroid. A lot has to go right.

We have a lot of things to do, but I think the first hurdle was people kind of waking up to the risk, and then we have to figure out what to do about it. But I'm optimistic, because I think we actually could solve this. We could slow down. We could buy ourselves more time to develop better ways of controlling these systems. I want AI to be built. I want AI to cure cancer and disease and to help us all live longer and healthier lives. Like, I don't want to file my expenses by hand. Like, I want this stuff in our lives. And I think that if we manage to, like, thread the needle on safety and control, like, it could be amazing. Well, Kevin, the book is terrific. Thank you. It's also always a pleasure to read your work and to see you at the AI events that we show up at. It's always, your writing is terrific. You're always very generous with credit, which I appreciate. You can see it all through the book. And wishing you great success with Machine Gods and with the rest of the book launch.

Thanks, Chris. Great to see you. Likewise. All right, folks. The book is the AGI Chronicles. The author is Kevin Rooves. You can find his new podcast, Machine Gods, on your podcast platform of choice. Already are coming up. Starting the week of October 19th. Starting the week of October 19th. Thank you all for listening and watching. Thank you, NASDAQ. This is the coolest thing. We're here at the market site. site in New York City for our first episode here and many more to come, I hope. So thank you all for listening and watching, and we'll see you next time on Big Technology Podcast.