Demis Hassabisと AI 開発競争の行方 ― Sebastian Mallaby が語る『The Infinity Machine』

Demis Hassabis founded DeepMind in 2010, before machines could reliably tell a cat from a dog. He believed that artificial general intelligence was achievable and that it should be built safely and deliberately. Along the way, Hassabis shared the 2024 Nobel Prize in Chemistry for AlphaFold, the system that crafts the 50-year problem of predicting protein structures. Fifteen years on, the LLM race has erupted across rival labs and countries, and AI has become one of the most consequential and contested technologies of the era. Sebastian Malaby is a two-time Pulitzer Prize finalist and a longtime chronicler of the collision between innovation and capital, from hedge funds to central banking to Silicon Valley. His latest book, The Infinity Machine, is a portrait of Hassabis and DeepMind built on more than 30 hours of candid conversation, beginning just as ChatGPT set off the AI frenzy, and drawing as well on Hesabas' estranged co-founder,

his rivals, and academic pioneers who now fear for human survival. Malaby is also the Paul A. Volker Senior Fellow for International Economics at the Council on Foreign Relations. In this episode, Sebastian joins Kevin Ball to talk about Demis Hesabas and the founding of DeepMind, the pioneering breakthroughs that followed, and how the field arrived at the current moment in AI. Kevin Ball, or KBall, is the Vice President of Engineering at Mento and an independent coach for engineers and engineering leaders. He co-founded and served as CTO for two companies, founded the San Diego JavaScript Meetup, and organizes the AI in Action discussion group through Latent Space. Check out the show notes to follow KBall on Twitter or LinkedIn, or visit his website, kball.llc.

Sebastian, welcome to the show. It's very good to be with you. Yeah, very excited for this. Let's start out, can you give us both a little bit of intro to yourself and then kind of intro to what brought you to writing The Infinity Machine? And we'll end up talking a fair amount about that on their show. Sure. So, you know, I did something deeply technical at college, modern history, where the focus on Europe and the central question was, why had Britain never succumbed to Marxism? So that's a long way from technology. and my path between doing the history of revolutions in the late 19th century, late 20th century and writing about Demis Asabis and the cutting edge of AI goes kind of the quick version is as follows. So I became a journalist, joined The Economist magazine, worked there for maybe a dozen years, was a foreign correspondent in Japan, in Africa and in the United States in Washington, D.C. And then I switched to The Washington Post and was a kind of opinion columnist columnist focusing on economics, a little bit of tech by that point,

and just generally kind of whatever seemed to be making the world change in terms of big macroeconomic, macro-political sense. So I wrote about both everything from political reform, constitutional mechanics in the US, and why trust in government was declining, it's in the 90s and the late 90s, to all the way over to kind of healthcare economics, because the healthcare system was clearly a big driver of the fiscal deficit and so forth. So I did that at The Economist in the late 90s and then in The Washington Post in the 2000s. And in the meantime, I did a couple of books, and this became kind of more interesting to me. And when I did my third book, I switched to a think tank, the Council of Foreign Relations, where it's a fantastic home to write books. You get the space and the time and the good colleagues and all that. So then I did another four books, and essentially were the first book in my 20s when I had been a foreign correspondent was about Africa. The next four were about different kinds of finance, so development economics, hedge funds and public markets,

central banking in the form of a biography of Alan Greenspan, the Fed chairman who managed to serve for almost 20 years, and then a book about the investing in technology in Silicon Valley, the venture capital world. And that last one was kind of a segue because having written about investing in tech, I was kind of ready to jump and write a book about tech. And so in 2022, I was looking around for the right way to do that. Artificial intelligence hadn't broken out in a chat GPT kind of way. But it did seem like it was really cooking and really reaching some sort of infection point where you'd had these systems like AlphaGo in 2016, which solved this massive combinatorial complexity of finding permutations in the game of Go. It's 10 to the 130 permutations you can have in the game of Go. and yet it found the patterns in that enormous search space. And then on top of that, you had the alpha fold system, which did the same kind of search, but for protein shapes.

Now it's fully 170 more orders of magnitude, the search space, right? So 10 to the 300 this time is the number of permutations and the way that a string of amino acids can fold itself up into a protein shape. So it's super, super needle in a haystack. And so you looked at that and you thought, okay, well, this is like finding patterns in an infinity of data, kind of. And so it's an infinity machine. And so that's a kind of breakthrough you could apply to different types of thing. And machine intelligence really is taking off. So I pitched Demis Osabis, who was the founder of DeepMind, actually the lab that produced both AlphaGo and AlphaFold, and was definitely the OG, the most interesting, fascinating character in the whole of AI, right, because he'd founded an artificial intelligence startup in 2010 before AI could even recognize a cat. Yeah, in the middle of AI winter, he's diving in. Yeah, total deep winter, and he completely believes. There's a line from a movie that he loved about Watson and Crick, the discoverers of DNA, who say,

blessed are they who believe before there is any evidence. And Demis was how this was in that category. So yeah, I pitched him on this idea for a book, and he thought about it a bit. I had ways and means of trying to persuade him. And so he said yes. And about a couple of weeks later, ChatGPT came out and went viral. And so I found myself embedded with Demis Vosavis and DeepMind in one of the main labs, right as this large language model race took off and went crazy. Yeah. I love looking back a little bit because I think right now we're in this language-focused model craze. But as you highlight, there's a ton of other really interesting things that have happened in the last 15 years in machine learning. protein folding other types of there's so many different types of models that we we lose that when we just talk about large language models but let's maybe start then a little bit with this thing and one of the themes i've seen in your more recent books is you're sort of navigating you'll use a person as a lens but kind of navigating institutions and navigating sort

of these things so let's talk about that unrealistic belief and what that looks like how and let's not assume our listeners have read the book like how did that play out for Demis how did he kind of get this whole movement going yeah so look the big picture on this question kind of does go back to me doing modern history at Oxford University I came to believe that there's this debate amongst historians do individuals shape history or is it really ineluctable abstract forces like demographic change, technological change, and so forth? And obviously the answer is both. It's kind of a dumb debate, but it's interesting to have that debate when you're 19, 20 years old and kind of frame it for yourself. And so going forward, when I came to write books later on, I was always looking for the individual who captures that sense of individual agency and how people do matter in history and in shaping the world that we live in the future. And at the same time, there are these big forces around them, which they have to

kind of capture, they have to surf those big forces if they're going to really make an impact. So Demis is a great case study in this, right? Because sort of almost by luck, he was this child genius, learned chess by watching his dad and his uncle play when he was four years old, became the best young chess player in Britain, captain of the team, the British international youth team and then also actually the second best in the whole world for his age group so he was really extremely good and gifted at chess and he wasn't satisfied by chess so he would sort of do other things and at one point he's got himself a home computer and tried to write a chess program which was like his segue between the chess and the programming and one thing led to another and when he got into cambridge university young because he was so smart they told him look you You're intellectually ready to study computer science at Cambridge, but you'll have a better time socially if you wait for a year and you show up when you're the same age as other people.

So he goes off, he has like 15 months to fill, and he gets a job as a programmer in a gaming studio. And he designs and programs this game called Theme Park, which goes viral in the 1990s, sells like 5 million copies or whatever. So by the time he shows up at college, all of the other kind of nerdy boys who are into computer games they've played his game so that he's kind of famous for that and also his boss of the gaming studio said he was so valuable as a coder and as a designer of the game that he would give him, I mean today's money in dollars is like more than a million dollars he would give him this money not to go to college and just come work for him instead and Demis is like the original Peter Thiel. Yeah, but whereas in Stanford or in the US generally, people happily drop out and go do something more lucrative. No, no, no, Demis is like, I want to do the science. I want to learn about the deep mysteries of nature. And I'm not going to do that by doing video games,

so I'm not accepting your money. But what the video game episode did do is it kind of put him in this strange country house outside London with a bunch of young coders coding until like three in the morning to write this game software. And then they would meet up and have spaghetti in the kitchen in the middle of the night and talk about the big issues. And the big issues were basically, how do you make the games better? Oh, you need artificial intelligence to make the characters in the game more realistic and more compelling. Maybe if the player does something to the character in the game, then the character will adapt its behavior and change how it acts. And so that was kind of early reinforcement learning intuitions, right? there and in the course of this living in this commune he read the book Gödel-Escher-Bach which is sort of a kind of classic which red-pilled a lot of young scientists around this time on computer science because it was written by physicists it's kind of very abstract kind of

ranges from he's a mathematician mathematician Gödel to the abstract artist Escher to Bach and his music, but it was kind of arguing that, one of the arguments in it, I should say maybe, is that whatever the human brain can do, computers should be able to do one day. Because intelligence is not about sort of whether you have a biological neural network or a silicon neural network. It's the patterns on the network which constitute thinking and even consciousness. and so as just looking at Moore's law as computing became more and more powerful one day you would have a computer as big as a human brain in terms of the number of neurons and connections and kind of gates that you would have a computer that could basically do whatever a human can do and so Demis aged 18 by the time he got to college had already decided based on this intuition from God bless the Bach that one day AI would work and that if

you could make it work not only would games become very compelling but also you would unravel the big mysteries of theoretical physics of biology of neuroscience like everything you would solve everything and this was such a compelling vision to him aged 18 that he just was obsessed with it from that age. It's like, it's incredible. It's like 1993, right? We're 17 years before he found DeepMind in 2010. 19 years before you get the ImageNet contest where the AlexNet system, finally you have a convolutional neural network which can recognize a picture of a cat. Until then, complete AI winter, as he said. Yeah. So he was super, super, super early in having this conviction and that shows you kind of the role of the individual who has a strange conviction and pursues it and actually does sort of accelerate the point where humanity gets towards thinking machines. That's one of the fascinating things here. I think a lot of us have rather absurd

convictions at 19, but most of them don't pan out. Yeah, right. I'm interested to follow the game thread a little bit more because when he got out, his first company was not an AI company, right? It was a gaming company. And I'm curious, it reminds me a little bit of one of my intellectual heroes was Feynman. And he wrote about like, after he got Nobel Prize, he couldn't do good science because he was so focused on the big things. And then he stepped back to just starting to play again, and then things unfolded. And we kind of see this pattern with Demas of like, game focused entree into this huge topic. Yeah, I think that's right. And it turned out, of course, that games are a great environment for testing, reinforcement learning in particular, You have a sort of confined goal, a sort of measurable reward at the end of each action you take. So the environment is giving you this very clean feedback signal. And this is what he figured out pretty soon after he started DeepMind.

And they decided to build a system that would solve all the Atari games, the kind of original video games created by the Atari company in the 1970s and 1980s. So you have Pong and Breakout and these other games. And the nice thing about this was that the video graphics were pretty simple because it was so old. And so for a, you know, in terms of compute efficiency, you could, it was tractable for primitive AI in 2012 or so when they began. And so they created this training environment for reinforcement learning using games. And so that's kind of where the connection came between Demis' obsession with games, which is partly Aki. He'd been this young chess player. He'd been super successful at that. He loved games. He loved winning at games. He was incredibly competitive. One time I went to see him, we would meet in London every month or so in a pub for a couple of hours. There was a secret room upstairs. And we'd sit there for two hours talking. And he said to me, you know, Sebastian, when I was at college, you know, one of the things was I was the best at a game called table football or foosball, right?

table with the figurines you twizzle. And I really didn't care. I mean, who cares about table football, right? And he said, no, no, no, you don't understand. You don't understand. I was the best. So I kind of nodded politely. He goes, no, you still don't get it. I have watched the videos of the semi-professional players that you can see these days on YouTube, and they don't know the snake shot when you put your arm against the lever. You yank your arm up really hard, and the friction on your arm gets extra traction on the spinner so that your shot in table football is even harder. And I could do that. And that's why I was the best at college. You need to get this. You only understand this. Hey, dude, OK, OK. But winning for him is a very big deal. And so games always were addictive for him, all kinds of games. He also was the five times champion of a thing called the Mind Sports Olympiad where you played five different mind games And he would run from sort of the go game to the chess game

and then he'd go play the diplomacy game or whatever it was, backgammon or something. And he aced all of these games, and he was the best in the world five times in a row before you retired. So he's like insanely obsessed with his stuff. Well, it's interesting to think about this, right? because starting a company is on its face kind of a ridiculous endeavor, right? Like you're doing, you have to go through so many different setbacks. And I remember, I mean, there's a quote most people have probably heard of Elon Musk talking about it. It's like eating glass. It's so hard. And so I wonder, you know, as you talk about the individuals who are harnessing the trends and moving forward, like is that level of competitiveness almost a necessity to get through those barriers? Yeah, 100%. And especially if you're doing it in Britain in the year 2005 or 2004, whenever he founded the first company, which was a pure gaming company, and then even DeepMind in 2010, I mean, super difficult to raise capital because there wasn't a venture ecosystem back then in Britain. I mean, there is one now. I think 2014 is what I regard as the inflection point when venture in Europe,

or particularly in London, started to kind of get traction. So you could raise money. And with venture capital around, of course, it's not just that you can raise money. is you get the connections that the venture capitalist has, which can be useful for hiring people or getting your first customer to sign on to a contract or whatever it might be. I mean, VCs, I read a book about this, it's the networking effects that they bring, which are pretty important for company formation. And it's still like eating glass, as you say, but it's sort of smaller pieces of glass, much easier when you have a VC helping you, I think. And Demet has to really face this without there being a functional VC ecosystem at all. when he first did his first company, right? So I tell a funny story in my book where he finds somebody somehow, you know, he probably, I don't know if he could even go online then. I mean, not really. It was late 90s, maybe just about you could go online, but there wouldn't have been proper search. Anyway, so he finds somehow that there is somebody calling himself a venture capitalist in London, and he goes to see these people,

and they just take him to lunch and give him two bottles of wine and talk about anything. and it results in a job offer to be a currency trader. I remember that story. Yeah, yeah, yeah. Because they're not interested in venture capital, even though they're very difficult and it takes a special degree of self-belief and determination to do any kind of entrepreneurship. But in Britain, then it was off the charts. This episode of Software Engineering Daily is brought to you by VariaCode. If you're an engineering or IT leader, you know how painful this is. An open rec sits unfilled for three months. Your tech lead is burning out, covering the gaps. and your product roadmap gets pushed back again. VariaCode solves that. For over 10 years, they've been embedding senior Latin American engineers directly into U.S. engineering teams in under 10 days at up to 40% less than local hiring. We're not talking about generic contractor pools. These are pre-vetted, high-quality engineers with real on-call and SLA experience. They work directly in your Slack, JIRA, and daily stand-ups in your exact U.S. time zone with zero friction.

If you're ready to stop losing sprints to open recs, head over to variacode.com slash S-E-D right now and get a custom shortlist of senior candidates in 48 hours. That's V-A-R-I-A-C-O-D-E dot com slash S-E-D. You spent weeks choosing the right model for your agent. How long did you spend choosing its web search? Parallel is web search built for agents, not people. Building a voice agent? Parallel Search Turbo answers in about 200 milliseconds for a dollar per thousand requests. Need an agent for thorough research? Parallel Search Advance digs deep across the web to answer your toughest questions. Parallel works with any model and drops straight into Claude Code, Cursor, Vercel, or Langchain. The best web search at every speed and price. Try it free today. Add the Parallel MCP server to your agent in minutes at parallel.ai.s.ed. This portion of Software Engineering Daily is brought to you by Origin. Your coding agents can launch each other.

Theirs did. An engineer asked Codex to pull data from staging. Codex had no connection there, but CloudCode did, so Codex launched CloudCode. When CloudCode said it lacked permission, Codex relaunched it with the dangerously skipped permissions flag. Nobody was asked. A permission check doesn't mean much if another agent can switch it off. Your logs might show a process that started with a flag. Their trace showed the whole job and why Codex did it. Origin finds the agents on your company's endpoints and reconstructs their work from request to outcome on one timeline. Origin is endpoint AI observability. Read what they found in their own traces at originhq.com. That's O-R-I-G-I-N-H-Q dot com slash S-E-D. So this kind of leads to an interesting question around the types of institutions that enable this type of really big long term bet. I've been in the startup venture capital ecosystem for most of my career, mostly on the startup side, not on the venture side itself.

And there's like this push-pull between they'll put money behind big bets, but that also creates this almost perverse sense of incentives to swing really, really big. And one of the things that struck me about Demis' story is like, I mean, DeepMind is a really big swing, but it was not his first swing. He was playing with these ideas and incubation for a really long time and through a number of failures. So I'm kind of curious, are startups the right environment? He eventually ended up selling to Google. What is your perspective on what that did to the sort of deep mind development of the technology? Yeah, it's a good question. So I guess the transition from the first to the second company was a kind of larger, exaggerated version of the classic entrepreneurship journey, I think. You know, you start with a slightly smaller idea. I'm just going to build a great video game. And, you know, he had a kind of medium outcome from that. You know, I think he exited with some money himself, but nothing really big. And, you know, I think the shareholders probably got their money back, but not much of a return.

And so that was OK. It was a learning experience. But then he takes some time off, does a PhD in neuroscience, comes back, having had his ambition fired up even more. Once he understood human intelligence from neuroscience, he was even more convinced about machine intelligence. And so he wanted to do a pure AI company the next time, even though we're still in the AI winter. And so the second time round, he's just much more ambitious. The goal is crazy ambitious. I'm going to build artificial general intelligence. And the fundraising vision is different. He leaves London, doesn't just confine himself to a small pool of capital. He goes to California, meets Peter Thiel, raises money from Peter Thiel, which then unlocks the door to raising money in the second round from, or maybe it was a third round from Elon Musk. And he gets some money from Asia in the form of a fund connected to a billionaire in Hong Kong. And so he's got a kind of global fundraising strategy. And the goal is much more lofty.

And I think that is a bit of a sort of familiar pattern, right? That when you do your second company, you know how hard it is. And it's only worth it if you really try to go big. And that's what he did. I guess then the other part of your question is, okay, later he sells to Google. How did that work out? And the answer is, so he sells, he basically concludes the deal in the end of 2013. So it's three years into his journey as DeepMind founder. And then it formally closes in January of 2014. And at the time, he was so relieved to have gotten that deal done because he needed a lot of capital for the compute. It was already becoming evident in AI that you needed money both for the GPUs and for the talent. because by 2012, not only does image recognition work with convolutional neural networks, but also the DeepMind Atari system worked. And it was presented at a conference at the end of 2012, the NeurIPS conference.

And so people in the space started to understand that, hey, this is something that is now cooking and we want to invest in it. And so at that point in end of 2012, Google bought Jeff Hinton's small company, three people. Two of them happened to be Jeff Hinton and Ilya Suskeva, so not bad. Did pretty well with that. And yeah, right. But that purchase by Google was a sign that one of the richest companies in the world had spotted that acquiring AI talent was a smart thing to do. And they did a bunch of acquisitions of small AI boutiques to kind of roll them all up to try to build a team so that internally they could implement AI in things like translation and other stuff in search. So this was all happening. And it was driving the cost of scientists up and it was driving the cost of compute up. And so Demis wanted to sell because trying to raise big capital as a venture capitalist, as a startup based in London, it's really hard.

It was taking all his time. So he does this deal with Google and basically he doesn't care about the sale price, which was $650 million. He cares about the pledge of the research budget going forward and he gets almost a billion bucks a year out of them. and he's allowed to keep his team in London to operate kind of independently. And Google is fine with this because they like the idea of having a skunk works in London where they're off doing their different thing. And then maybe when AI really, really works, they can bring them into the center of Google. So they've got this kind of almost insurance policy against AI working out and disrupting search. And it's just doing its thing in London and they fund it. And yeah, you look at them now, they're doing the same thing with quantum, frankly. They've got a whole quantum team in Santa Barbara, and they're doing the same thing, funding, funding, funding. And one day, at the moment, they can just boast about the fact that there are people at the quantum team who've gotten their bill prizes. But one day, it'll be commercial, they hope, that they're willing to play a very long game. So I think it was a great move for Demis to get the research money he needed.

But there are people who say, ah, you know, you could have stayed independent. I don't think you could have, actually. Yeah. I mean, there's a whole conversation we could have about Google as kind of the modern Bell Labs incubating all these technologies. And they seem to have done so far a better job of turning them into commercial reality. But let's stay on the AI topic for a little more here. So one thing that struck me reading your book was how much Demis was focused on security and making sure these things would be safe and things like that. Now, we look at the last few years, we've got this like tripartite race between OpenAI, Anthropic and Google really pushing. and it feels like everybody's talking nice about keeping these things safe and we're all grappling with this pace of progress. So how do you see sort of the evolution in the last few years as suddenly we have a couple true believers really focused on safety and doing this right to now we've got like a commercial race? I mean, this is sort of almost the central drama running through my whole book

because at the beginning you have this inspired character who super early and he's very idealistic about why he's doing ATI. It's really not about getting rich at all because, you know, there's no product envisaged. It's about, he's like almost a religiously driven quest to understand artificial intelligence so that you can then understand the deep mysteries of science and theoretical physics and neuroscience and so forth. So that's what's driving this central character in the story. and he has this vision that when AI starts to work there'll be one lab developing it kind of like a Manhattan project on behalf of the whole of the Western world or maybe the whole of humanity and he doesn't envisage there being a competition or multiple labs in multiple countries that's just not in his framework because he comes out of PhD science and at the time the AI community was small you could go to a singularity conference Singularity Summit in San Francisco, you could fit all of the believers in AI in a single room

because it wasn't working. So nobody believed in it. So you had these kind of rather eccentric people who did, but there weren't very many of them. And so thinking it would be one lab didn't feel crazy. But of course, as time goes by, it starts to work. And then more ambitious people are drawn into the competition. And so first of all, you get Sam Altman and Elon Musk, and they come together, they found OpenAI in 2015. And by the way, at the same time, Meta and Mark Zuckerberg start competing and they hire Jan Lecun and they set up FAIR, the AI lab at Meta. And so the race is sort of starting to get a bit complicated and then you go forward in time and it starts working more. You have large language models and Dari Amadei is working on them with his friends at OpenAI. They have this difference of vision and they split out from Sam Altman and they go off and do their own thing. And meanwhile, Elon and Sam Altman have also had a fight. And so that winds up in Elon doing his own AI separately. So now there's this kind of splintering in the tight community

and the emergence of this vicious competition. And it goes, of course, global in 2025, at the beginning of 2025, when Deep Seek proves that China is in the game too. And so by the end of the story, I have on the one hand a character who has anticipated that AI was going to be powerful, and he's been proved right. And what's more, in the business plan, which he wrote in 2010, it says around 2030, AI will surpass human intelligence. It's a pretty good prediction for 2010, right, given where we are now. So in terms of predicting it, building it, this guy has, like, hit it out of the park. But in terms of his safety vision, which was one single lab, it's a disaster, right? the opposite is happening. There's not one single lab building it like the Manhattan Project. There's these multiple labs in multiple countries, and it's a total mess. Everybody's like rushing for the precipice in a way, because they need to get ahead, and they don't want the other guys to get ahead. And everybody's saying, well, I still believe in safety, but I know that I'm a good guy.

So if I come first in the race, then I know I'll be safe, because I'm good and honorable. I don't know if I can trust the other guys, so I better beat them. And then the other guys are saying the same thing about you and so everybody's just racing like hell and the race dynamic of course makes it harder to have time to do the safety and alignment work that would actually make it safer so this is like demisus harbus wants to do good he is a good person i'm sure about that but can he do good is the question that is a sort of the kind of tragic question at the center of the story this comes back to this question of institutions right so we have these individuals and private companies really pushing things forward and really having this vision and pushing and pushing and pushing. And I suspect they're not all true believers. I suspect there are some cynics involved in this race as well. But what is the right even institutional structure for dealing with this type of world-changing technology, but instead of it being developed in a government lab somewhere

and maybe you can have... We talked with... I just spoke with one of your colleagues from CFR recently about security on this front. And he's like, you've got these four labs and a lot of the founders hate each other. They cannot stand being in the same room. How do you even get them to talk about what safety looks like? Absolutely. Yeah, you've got these deep, deep personal animosities to it. That famous picture where Dariya Mide and Sam Altman are next to each other in the AI summit in India. And they're supposed to pose for a photograph with their hands in the air, holding hands together. and just like Dario won't do it, he can't bring himself to touch Sam Wotland. This is such a powerful image that captures this animosity, dividing the leaders of this thing. So yeah I mean coordination what this shows is the coordination among the different labs to try to slow things down and maybe do more alignment research along the way You can do that right if you at loggerheads for one another And by the way actually I think you couldn do it anyway because even if you had good relationships, you've got too many players in this game.

You know, if we count, say, five or six in the US at this point, if you include Meta, you include Elon Musk and X, what's that? You're getting to five at least. And then you've got the Nier labs that have spun out of OpenAI and so forth, whether that's Ilya Sasukeva's or Miro Moratti's and so forth. So you've got a bunch there of competitors. And then you've got Mistral in France and Coher in Canada, maybe six or something Chinese labs. There's a lot of people, right? So I just don't think coordination is doable when you've got that many people just in game theoretical terms. what you need is you need government step in and impose standard rules definitions of safety a way of vetting for safety so the only player that can be the safety advocate really is government and it basically means two governments the u.s and china because those are the most important countries in producing the stuff so again demis hasabes had enormous agency at the beginning of the story because he was like this early believer and he really pushed the science faster than it

would have been pushed if he hadn't started DeepMind. But by the end of the story, he's lost his agency because if he quit tomorrow, and in fact, he's done something a little bit like this, we can discuss if you want. Even if he not only quit, but actually closed down Google DeepMind and Google just pulled out of the AI race, it would make no difference to human safety because there would be 10 or something other labs still racing for the same objective. So I just think human agency, individual agency, doesn't exist in terms of safety and governance in the private sector. It exists a bit in the government. Yeah. Well, let's talk about actually that step by Demis, because it was relatively recent to when we're recording this. It was announced he was stepping back from leading DeepMind within Google. He's going to be a chair and do various things. Do you see this, knowing him much better than any of our listeners, do you see this as a personal choice, something that's changed for him? Is it a change in what's going on in the situation and he's trying to handle it differently?

How do you see that in the lens of everything going on? I mean, my best guess, and I haven't actually talked to him since April, so I have no direct knowledge. I want to be clear about that. But knowing him as I do, having spent more than 30 hours talking to him in that pub in London, I see this as a sort of natural extension of how he framed his mission from the beginning. And the mission had these two parts. One was build AI and the second one was make it useful for humanity, in particular for understanding science, but also medical advances and other stuff. and I was pushing him at the end of 2025 in the conversations to reckon with this appalling dichotomy that he had succeeded on the first mission of building it but completely failed on the second mission of guaranteeing its safety or even promoting its safety he took that on board and he wasn't happy about it and he was stressed by it and we would talk about it and he would say it's not what I imagined arriving at the foothills of artificial general intelligence

I thought it would be a, I would be elated and we would arrive from the promised land and it would be great. No, no, it's not. It's a mess. It's like a race. It's stressful. It's dangerous. It's all these things. And I didn't like it. But this is where we are. And I don't know what to do about it. And then I saw him a bit later after I finished the book and the book had come out. We did a couple of events together in where he was on the stage with me and we talked about his career. so in that context he did say to me you know i've accumulated this credibility in artificial intelligence and in the world and at some point i've built up all this capital that i need to spend to make ai safer and that's all he said but it's not difficult to extrapolate from that and say okay now he's spending that capital he's stepped back from building the tech he's no longer in charge of Google DeepMind, the lab, in terms of its day-to-day research teams and how you organize them. What he is instead is chairman of Google DeepMind plus chief scientist of Alphabet.

And that gives him a platform to think about the strategy of how you roll AI out into the world and make it good for humanity, and in particular, the safety and the governance. And he showed, before he announced this shift in his role, he showed that he cared about it by publishing this open letter about AI governance about three or four weeks ago. And so I think what he's doing is just pivoting to that second part of his mission. A lot of people observe what he did and say, well, he was forced out, he was kicked out, he was demoted, he's been sidelined, and so on. And, you know, there's a case for interpreting it that way in the sense that Gemini is definitely behind the main two competitors. You know, it used to be the number one model late in 2025, and now it's barely in the top 10. So it's a big decline. You know, things haven't been going as well as they hoped. So you could argue, oh, well, maybe he was just fired. But I think there's probably more truth in this alternative theory that, in fact, he wants to focus on safety and governance. Yeah, that makes sense.

Going a slightly different direction, I think one of the things that's interesting we're seeing in this, at least in the U.S. right now, is there's a tremendous sort of popular pushback against some of the underpinnings of AI. And in this, I'm really thinking data centers. I saw you had an article. You went to China to look there. I don't get the impression the same thing is happening there. But like, how do you see the sort of current state of play? Having had this conversation and thought about up until now, where is this going over the next couple of years? Yeah. So I host a podcast called The Spinover, and my co-host Rebecca Patterson, and we're both at the Council on Relations, wrote a book chapter just recently talking about AI adoption in different economies around the world. And one of the things she points out in that chapter is that the attitudes towards AI are much more welcoming and positive in emerging economies than they are in rich economies. And so if you look at the Pew survey across all these different countries in terms of what people think about AI, the place where it's the most popular of all is China.

People love it. The place where it's the least popular of all is Canada. But I think I'm right that in the US, it's the second most anti-AI in terms of public opinion. And you see that expressed through resistance to data centers because the U.S. political system makes it possible for well-organized local groups to obstruct the construction of all kinds of infrastructure. And data centers is one example. And in fact, I don't know if maybe your listeners have heard about the book Abundance, which came out a year ago and did quite well. It was basically telling a story about how building anything in the United States is very hard, whether that's high-speed rail, whether that's… Oh, I live in California. I'm so familiar with this story. Yeah, yeah, yeah. So you put together that general resistance to building infrastructure in the United States with a specific antipathy towards artificial intelligence. and you get this current resistance that we're seeing, which could slow things down or just maybe could drive initially the data centers to other countries

and then maybe later drive them into space, which is what Elon Musk is pinning the hopes of SpaceX on. It is interesting thinking about and coming back to, right, you had Demis with this like true believer this is going to be good. And now he's trying to reconcile a little bit like the current state of things. I feel like the software industry is probably in the US, one of the places AI has gotten the deepest penetration. And the levels of ambivalence about the good or bad of this technology are unlike any I've seen for a technology in my lifespan. It's just people are like, this thing is, there's so much hatred for what it's doing to the industry. It's wild. Do you mean simply that it's displacing jobs or do you mean that it's actually generating slop code, which isn't very good and you have to spend hours fixing it? Neither of those is enough to explain it. I think it's forcing a crisis of identity, right? It's not just, oh, it's going to get rid of some jobs. That gets a lot of people upset. But I think it, I'm not sure, like if you look at hiring paths in the tech industry,

I'm not sure how real that story is because we have all of these different trade-offs between productivity versus demand. And I'm blanking right now on the exact name of it. Yeah, the Jevons Paradox. The Jevons Paradox, right? And you see the companies that are the first that belong in adopting AI are actually hiring more software engineers. Like there's not necessarily that much evidence for the job thing. But another thing that we see, and I see this in my day job, we coach people inside of all these companies, the software engineers and tech industry folks who are using the most AI are also the most likely to feel like they are unsettled, left behind, questioning the future of everything almost. So it's creating this deep sense of disturbance in people. Yeah, I think, by the way, for writers, that's true as well. Yeah. So do you have a sense why? I feel like this is the type of change in terms of even watching how power moves around, right? And it feels like there's a power shift or something going on.

But I'm curious from that historic perspective, why is this so disruptive for us? You know, I think there's something different about this time. I mean, it's a famously unwise thing to say if you study finance, this time is different. That was the title of a book about how actually crises in finance always do happen, even though people every time say, no, no, it's different this time. But look, I think in terms of what it does to our own identity, artificial intelligence is challenging us in terms of cognitive tasks. And that's a new thing. There's never been this before. The sense that we are no longer the most intelligent animalists on the planet because these machines are better than us. I mean, I think that's deeply unsettling. and if you think about the displacements that have happened before it's like yeah this specific thing maybe a hundred people sitting in a room being calculators human calculators together with an with each one has an abacus and they're like doing the accounts for some company or bank or something

and then along comes spreadsheets and that's all gone yes that displaced a hundred people but those hundred people have plenty of other things they could do where they were smarter than the machine still. And now you sort of have to move away from pure cognitive tasks that you can do on a screen if you want to feel some protection. And we all know that that protection will go when robotics gets good, which it will. So I just think this is unprecedented in terms of the challenge to who we are. And therefore, it changes how we raise our kids, because what do we tell our kids about what they should be in the future in order to have a happy life. It changes how we do our jobs because we are coexisting with these smart machines who are going to take some of our tasks away from us. And by extension, it changes who we think we are, what makes us special. I just think it's very profound. Now, personally, I am sort of optimistic that we will still find meaning. And I mean, this is going to be interesting to see how it plays out. And I'm kind of obsessed with the

subject because these changes are so massive and so thrilling, but they're also terrifying. But you just can't take your eyes away from it. It's like a horror movie or something. I don't actually watch horror movies, but I am obsessed with watching how AI developments play out. And I think that my hypothesis is that actually we get up in the morning and we have things to do. If we're kind of minded to be active and interested and curious in the world, we will invent things that we need to do and even if economically in the future maybe the super abundance we don't need to do them we'll still be quite kind of passionate about doing them and so if you look at the number of activities that people already do where it's kind of for their own psychic satisfaction but they work really hard at it whether that's becoming good at some kind of game or just really good at cooking or really going to the gym and becoming super fit, or it might be writing a substack. I mean, loads of people write these substacks. They put a lot of energy into them. They're quite good. Most of them don't get paid,

but they still do it. Podcasting, right? Why are we doing this? Because we kind of enjoy it. And I like the fact that I wake up in the morning this morning, I think, all right, in my calendar, I've got this interesting conversation. It's going to be fun. So I think we will invent tasks that will feel fulfilling, irrespective of our economic displacement. And so that's one thing that makes me optimistic about this change. Now, coming back to the lens that you took in the Infinity Machine and that I've seen in a number of your books is this grappling of the individual and the institution. You just talked about the individual adaptations. By the way, I 100% agree. I think investment in hobbies, in yourself, in what do you enjoy doing, just because a machine can do it doesn't mean it has to do it. And if you can get that abundance that gives you ideally more space to do these things. Those are personal tasks. What are the institutions we need to grapple with this type of change? Yeah. So first of all, your comment makes me think the ideal investment portfolio in this world is like some GPU investments together with a portfolio of premium restaurants or something like that,

right? Or maybe gyms. Or other in-person experiences, right? Yeah, exactly. Yeah. things that happen together like you talk about obsessions i'm obsessed with in-person experience and immersive theater and improv and all these different things that just get you together in a place with people yeah right exactly so totally that's right you want to invest in the pure silicon and the pure experience but you're saying how do institutions adapt so i i think this is again super exciting because it's so early and we're going to see how this plays out but what we know is that typically in the first phases of invention through economic history people do small things with the new tech so electricity comes along and you can reconfigure there's certain tasks that are more efficient but the sort of design of the factory where maybe there had been a kind of coal-fired steam engine or something in the middle powering your loom and then the workers were kind of organized around this shape. You know, you kind of stick with that shape.

You just have electricity running the loom instead of the steam power, for example. And it takes quite a long time between the advent of electricity and the first automobile production line where you have a straight line production line where you rearrange how the work is laid out. Because actually, oh, you know, we don't need to put the power center in the middle because it's kind of a localized steam thing. It's coming along wires on electricity, and you can make it go to lots of different terminals, desk spots, along a long production line, or whatever it might be. And it took like 30, 40 years or something to get that. Then you have, think of a different example. You have a much accelerated timeline when you think about the confluence of cloud computing and mobile. But when the iPhone came out in 2007, nobody looked at that thing and said, oh, it's the end of taxi cabs. There's going to be Uber. I mean, it took a while. I forget when Uber was founded, but like 2015, something like that. It took maybe eight years. So we're early here.

Chatty BT has been around for almost four years, but it's going to be a while before people really figure out how to make use of this stuff. And when I listen to, you know, I'm invited often to go on calls with chief executives and business people, and they're always talking about how hard it is to get productivity gains out of AI, and they experimenting and they doing this and that But outside of coding and maybe call centers there haven been that many clear wins I make one more point about this So if you imagine an organization an institution right and there's one function that you can 2X it by having, say, a coding assistant for a coder. So all the coders are now producing twice as much code. And you think, great, they're going to be twice as productive. And individually, they are twice as productive. But the company sort of IT output goes up 1.1x. Why only 1.1? Because before you can ship the code, you need the product team to come and look at it. You need the legal team to make sure it was produced in some way that's compliant.

You need the whatever. And so there are these bottlenecks that determine the pace at which the institution advances. And those bottlenecks mean there's a big delta between the individual productivity gain and the institutional productivity gain. And we're in that phase right now. And it'll take a while for even the best companies and my friends at some of the hyperscalers say this is going on in real time inside, going from one bottleneck to the next and trying to whittle it away. But it's the labor of several years. Yeah, absolutely. Yeah, the theory of constraints and bottlenecks has absolutely seen a resurgence with this stuff. Something you were talking about in making the electricity comparison reminded me of Carlota Perez's book about technological revolutions and financial capital. And in that book, she tries to map out exactly what you're talking about. What is the sort of adoption of these general purpose technologies? And one of the things she talks about there reminded me of our current moment, which she describes there's always a bubble phase because people are, one, irrationally excited and need to try all sorts of things to find what are the things that are actually going to be effective.

Then usually there's a crash and some sort of adjustment. And so, I mean, not to overbeat a dead horse, but the are we in a bubble conversation comes up all the time. And I find myself day in, day out, I can make different arguments. I know people in the hyperscalers who say, we are throttled for GPU all the time. You see this, you're trying to use chat GPT or codecs or cloud models, and they are having constraint problems. There's so much runway for investment based on current demand. And also, debt structures, and you see all these different things building where it does look like a traditional financial bubble. Maybe this time is not different. So what's your stance or position right now on like, where are we in that financial curve underneath the technological revolution? Yeah, I agree with your framing very much. And back in, let's say, January of this year, 2026, I had no trouble saying flat out there is no general AI bubble. I really was comfortable in stating that very confidently because the demand for GPUs was just going straight up and data centers were being built, but they were going to be used.

And my line was, you know, there may be an open AI bubble because in the model layer, there was a lot of competition and it was very hard to monetize. And that sort of proved to be basically true. For enterprise applications, the model there is obviously doing well, look at Anthropic. But for monetizing a big consumer base, it's difficult. And actually the leader there is Google, not OpenAI. So I think OpenAI is an example of a company that's sort of at the center of all this and might actually be a bubble, right? Because it's just so hard to monetize their product. And so how do you justify all of the money that's going in as investment? It's basically a bet that this is clearly they have good tech, not taking anything away from the tech, which is impressive. But can they have a business model attached to that tech? You know, the jury is out and anybody investing today in OpenAI is basically saying, well, we don't know what the business model is exactly, but they're going to figure it out.

And, you know, in two or three years, at some point, money will start to roll in. I think that's very, very speculative. It might be right, but it might not be. So that was my view six months ago. No general bubble, but maybe bubble in specific providers. But I agree that the leverage is building up and up. And today, as we record, there's been this announcement that Jensen Huang has put together a kind of 500 billion financing, you know, with a bunch of Wall Street types to finance chips. chips and I haven't actually got my head deeply into this one yet, but it's just the latest example of how, as you say, things are leveraged. The situational awareness hedge fund, which to say it blew up is a bit unfair because they're still in business, but they did have to sell their public stock portfolio to Citadel because they were down sufficiently that they were getting capital calls because they'd borrowed. And again, the leverage is the key point. Exactly. Yeah. Once you're levered, then margin, you don't need that much of a dip to get this self-reinforcing margin call cycle going on.

Right, right. They were levered four to one. And so just going down 25 percent on the equities meant they'd gone down to zero because you have to forex it. So they couldn't hold those positions. But in some sense, who cares about a fund, right? That's fine. So buys it and a problem. That's not systemic. It's unfortunate for the investors in situational awareness, but it's not a systemic problem. But when you look at some of these really big deals around the data centers and the private credit that is cropping up the building of the data centers, that does look bubbly. And unfortunately, also private credit is opaque, right? They don't have to say exactly what their loans are. They may disclose something to their limited partners who are backing the fund. And that might leak out a bit. But basically, it's not like the public bond issuance. you can see the price on the screen that's traded a bit. So you can see what it's worth. You can see if it's distressed and if it's trading at junk bond levels. So at least there are warning signs when you finance it with bonds.

But with private credit, which is a big player at the moment, there are no warning signs other than just gossip. So I do start, I have started in the last couple of months, particularly to worry about a bubble. So I realize we're coming close to the end of our time here and we've wandered over a wide range from the individual story of Demis to the current sort of layout and even the financial situation. Are there any things we haven't talked about that you think would be important to cover before we wrap? You know, I'm very interested in this debate about open and closed models, right? I'm always most interested in the ones where I'm least certain that I'm correct, because I do think this is a finely balanced argument. But maybe I could tell you my perspective and you can tell me yours and you may well disagree with me. I'd love to. Okay, so I think that the traditional arguments in favor of open source code, which were clearly very good and very strong, don't apply to large language models. What I mean is this idea that open source is sort of safer because many eyes make for shallow bugs

was true in the era when you had relatively tractable code bases and individual humans could kind of look at them and find bugs. and then having lots of individual humans in the open source community was a good thing. But now we're in this massive trillion parameter world and the best code review is clearly done by the top models, by Mythos, right? And so I don't think human code review from the community, from open source people, I don't think that really is how you get to safety anymore. So that's one point. The second point is that part of the argument for open source in the past was that, look, somebody will put some code out. It'll have some vulnerabilities. Some attackers will exploit those vulnerabilities. But it's OK because they just do a small exploit and then the defense community sees what the exploit was and they figure out how to patch code to kind of prevent that from happening again. And you have this cat and mouse sort of iteration between the attempt for defenders and the attackers.

And you get towards very robust code through iteration. and the damage that you incur on that path is small because the attackers make a few successes, but not that many. Now, I think with AI, you just need one criminal group, one bad actor to get hold of a really strong cyber hacking model. And they can do cyber hacks on hundreds, maybe thousands of targets at once and do a lot of damage, right? So the iteration story is way more expensive now. And I think if you put those two points together, you would land in a position where the traditional arguments, which were very compelling for open source, don't apply anymore. And you should be more open to the argument that safety lies in proprietary models, which are debugged by mythos-type AI models, and that the regime in which that debugging takes place

is sort of backstopped or overseen, enforced by the government in some form. That's how I see the future of AI safety, but it's not how the open source community sees it. So I'd be interested in your view. Yeah, it's interesting. Well, and it's interesting also when we talk about models, because often the equivalent that we talk about there is open weights, where the weights are published. Weights are not legible in the way that code is, in the sense that open weights means I can take these and run them myself. I don't need, you know, it's not hidden from me, but it doesn't mean that it's interpretable. It doesn't mean that I can understand what's going on there or analyze it in the same way that I could open source, like source code project. You know, it's a big matrix of math, and I have the numbers now so I can run it, but I don't know why or how. So to your point, there's not the legibility side that you got with open source isn't really there. I think there is there's an interesting discussion to be had around who can run these models and who can see the data that goes into them.

We did an interview, I think it was last year, with Ilya Polosikin, who was one of the authors of the Attention is All You Need paper back at Google in the day. I think he was at DeepMind there. And he was talking about kind of the need for verifiable privacy in running models, essentially. because right now when you have the big labs running models, they tell you they don't use your data in different ways. They tell you you get all sorts of kind of high-level guarantees, but you have really no idea of what's going on there. And a thing that open models allow you to do is you can run your own instance, assuming you have the hardware, and actually maintain, for example, data privacy and continuity of who's able to see what. So from the data being the source of danger and data leakage being the source of danger, I completely see the argument for open weights, open source, being able to host your own things like that. Now, flipping around to what you're talking about in terms of coding cybersecurity capabilities and those types of things,

I feel like I haven't fully wrapped my head around what the end state here is. We're clearly in a transition period where these models are able to uncover vulnerabilities that we previously were unable to uncover. And the rate of change in patching going on right now in the tech industry is unprecedented. It's the whole back. I remember the year 2000, we've got to patch everything except the amount of software in the world was so small compared to what it is now. And now we have that level of, oh, my gosh, everything needs patching all at once. A thing that I don't know is like, is that a transitory state? Do we get to a place where actually we have a more secure way of validating writing software? Because we have these models that are able to check all of these different things. And we don't have this ongoing, everything is going to feel more insecure all the time because these models are so good at hacking. Maybe we do. Maybe we don't. The one other factor that I'll throw into this is related to the economics of things. I think part of what is driving all of the sort of rapid acceleration right now is that there is this belief that frontier models, that you could throw all this money into investing in frontier models, and each one seems to be an order of magnitude more expensive to train.

You'll be able to reap something back from that because you are able to charge a differential amount for the frontier. At some point, capabilities for a large swath of things that you want to do saturate. And if you have open options and things like that, it actually can potentially saturate much quickly. Why do I pay for a frontier model if I can get good enough for pennies on the dollar with an open model somewhere else? And so I do think open models may actually provide a counterbalance to the sort of runaway escalation danger. But I don't feel like super sold on that. It's just one of the different lenses in which I've looked at this is it changes the economics of continuing to invest in more and more capable models over and over. And you're saying that's good. I'm saying that that is that is a form of economic constraint where we've so far failed on regulatory constraint. Yeah, I mean, but we're getting already to models that appear to be Oh yeah, we've crossed the threshold for a lot of these things already

So yeah, I don't know that that actually helps Yeah, that's super interesting I mean, so your first point is essentially an additional reason why the old coding community will make things safer is wrong So I mean, you can't interpret the weights, they're not readable So that sort of reinforces my view, but the other two points maybe cut against it. Yeah, and all of it just feels like a muddle in some ways, right? Like I can, similar to what we were talking about with the bubble conversation, like I can see both sides of this and make arguments that feel pretty compelling in isolation. I do think that, I mean, but let me put one thing to you, though, about Ilya's argument, which is she's saying that if I give my data to Anthropik, Anthropik may assure me that they're not training on it, they're not looking at it. but I don't really know. Okay. But I mean, this feels like something where internal incentives ought to lead to an okay outcome, I think. Meaning Anthropik's whole shtick is, you know, you can believe us and therefore you should be our customer.

And if you didn't trust us, you wouldn't be our customer. And so that kind of competitive war for trust amongst the model providers should lead them to treat their customers well and to honor the promise. I think that's right. I think you don't need, it's an advanced, it's an accelerated version of, I already trust Google with all my data. I feel like Google knows everything about me. Why do I trust them? One, because once again, their business is around keeping this safe. And two, they've demonstrated over the years a very high level of quality in actually capability. And that's where I have a question about some of these labs, especially knowing how fast they're moving and the extent to which they are trying to push the edge and using AI to code, and I use AI to code, like don't get me wrong, it's an immense tool, and it puts it at a place where it's hard to keep track of everything that's going on. It may not be an intent issue. It may actually be a quality or data leak issue that unveils things.

And so once again, if I'm running the model myself, I at least only have myself to blame, and I'm only a target as large as I am. Yeah, but then the question becomes, are we more worried in the world about our data leaking out into the world. That's one kind of risk. Are we more worried that the model's power gives a criminal organization the ability to destabilize the internet? And personally, I think I'm more worried about the second. And you're probably right. Okay. Fun debating. Thank you. Awesome. Well, I think at this point, we're probably about at time. Thank you so much for the time today. This has been delightful. Yeah, I really enjoyed it. Thank you so much. Thank you.