私たちを置き換える1兆ドルのAI競争（ギャリソン・ラブリー氏）

The vision of what the industry is trying to build, in my mind, is universal labor replacing machines. The industry calls this artificial general intelligence. Rendering us obsolete means disempowering people because their labor power is a source of economic power and political power. And so if people lose that because they are replaceable, then you lose a mechanism of democratic accountability. I don't think anything I'm arguing is predicated on this happening very soon, just that it could happen within the next few decades is enough to make it the most important thing in the world. When and how that happens could matter enormously. And I'm not against building it at some point. I think it should just be when you know you can do it safely and when there's actual public buy-in so you can make the case to people and they're making an informed, consensual decision to bring this into the world. Welcome to the Future of Life Institute podcast. My name is Gus Stocker, and I'm here with Garrison Lovely. Garrison, welcome to the show.

Great to be here. You are the author of Obsolete, the AI industry's trillion-dollar race to replace us and how to stop it. So what's the difference between the vision of AI that's being sold to the public and then what the actual goal of the AI industry is? The vision that's sold to the public, it's kind of a mess. It's like, okay, AI will help you do your work better, or AI will help you automate your workers away, or AI might cure cancer in the future. It depends on who they're talking to and who's talking. The vision of what the industry is trying to build, in my mind, is universal labor replacing machines. The industry calls this artificial general intelligence, and there's a lot of debate about what that means, whether we already have it. But my definition is like, could you, instead of hiring a new human for any given desk job that's remote capable, could you just hire or use an AI model instead?

And so by this definition, we clearly don't have AGI or universal labor replacing machines because they can't plan a wedding end to end. They can't learn from experience. They can't seamlessly plug into any meeting and like catch everything that is going on and do this in real time the way a human can. But the goal is to build machines that can do that. And OpenAI defines AGI as a highly autonomous system that outperforms humans at most economically valuable work. And Anthropic CEO Dario Amadei has said that AI is not just normal technology. It's like a general substitute for human labor. And so I think that's an honest description of what they're trying to do. And I think people should just be aware of that. So there is sort of a spectrum between AIs that you can hand off some of your work, too, which is clearly what we have right now. And maybe you can hand off increasing amounts of your job in total.

But then that ends up with your job being replaced. And so is it sort of a spectrum between helping people do their job better and then replacing people? I think it's, yeah, it's really just about the capabilities. So right now, the AIs are not good enough to do most jobs end to end. They can do many tasks, an increasing number of tasks, but they're just not there yet. And so when people say like, oh, we haven't seen a job apocalypse, so therefore we won't see one. This is just missing the point. It's like, no, we aren't at the threshold that people have been worried about for a long time. And so, and one other point I make throughout the book is, you know, AI progress, if you look at charts of benchmark scores, it's like remarkably smoothly exponential, you know, doubling every some number of months typically. But when we experience that progress, we experience it as these big tipping points where it's like the LLMs can't really handle tasks they've never seen before to like suddenly they can do okay at them.

to AIs can't write code very well, and now they can. Or AI agents just don't work well enough to be worth using, and now they do. And so this can happen very, very quickly once these crucial thresholds are reached. I think this threshold effect is worth keeping in mind because it can be true to say one month that, okay, AIs can't do math. And a year later or so, AIs can actually do math and can do coding. And so it is for many other tasks. And as we get closer to AGI, the capabilities will seem to suddenly spike in various domains. And it can be true to say that we do not have AGI by the definition you just gave, but we might be quite close to it anyways because capabilities, from our perspective, suddenly cross a threshold and then we are in a different situation.

I'm not actually sure how close we are. I want to say that. I think that I have longer timelines than a lot of the people working at these companies who think that we'll have AI that can automate its own research and development in like six months or 18 months. and I don't think, I think it's very easy to underrate how hard it is to do the last mile of a given problem. And in the book, I talk about how like I almost finished my book a dozen times and yet it wasn't done. And I think similarly, like you just won't be thinking about some piece of it that's actually crucial. And to get the really, really crazy takeoff that people talk about, you really need to automate not just 99.9% of the process, but the whole process, because a rate limiting step could just make it go a lot slower. I don't think anything I'm arguing is predicated on this happening very soon. Just that it could happen within the next few decades is enough to make it the most important thing in the world. Even if we talk about this

happening within the next few decades. Our economies, our political systems are just not set up to handle that. So even quite long timelines by San Francisco's standards give us a situation in which we have to handle a lot of things in not a lot of time. So when you hear the CEOs talking about using AIs as tools to help people in their job, making people more productive and so on. That seems to be a correct description of what's actually happening up until now. So what is it that gives you, why is it that you say that they are actually gunning for replacing workers? Those quotes I gave for the definition of AGI, of Dario's definition of AI, but you also see these financial reports where it's like, we think the total addressable market of our business is $30 trillion. Anthropics said this in their filing before their IPO.

And it's like, that's more than the US GDP. So that is coming. That's like white collar wages, basically. And Andreessen Horowitz has also said, like, our total addressable market is like all human wages. So there's that ambition. And just the aspiration for AI, for AGI, for a long time has been creating a mind that can think like our own, that can do whatever we can do as well or better. And I don't think that's the only type of AI that we could pursue, but that has been what the industry is oriented around. And yeah, the book is called Obsolete because they're trying to render us obsolete. And they'll push back on this in various ways. But I think you can just look at the record of what they say and the way they behave and the amount of money that's going into this. And I think it's a pretty clear picture. Now, one could object, what's so bad about being obsolete in the economic sense? What is so bad about being sort of a child, for example, or being a pensioner, being someone who doesn't work?

But there are many examples of people who do not work in our society that are still living great lives. And why can't we all end up in that situation? maybe eventually if you really trust the people in power um and you trust the democratic processes to take care of everybody you could have some kind of like species-wide retirement um that people are like opting into in this meaningful sense um i don't trust the people running the world right now to do this or our institutions as you mentioned to be to be able to handle it And I think that rendering us obsolete means disempowering people because their labor power is a source of economic power and political power. And so if people lose that because they are replaceable, then you lose a mechanism of democratic accountability. And, you know, there's very scary things within that, too, beyond the big picture. is just like, you know, if soldiers are rendered obsolete because you have lethal autonomous

weapons that can be directed by a single individual to kill protesters, then one of the key ways that dictatorships have historically fallen is taken off the board. And that's also on the table. And the U.S. federal government has expressed strong interest in pursuing those types of technologies. And yeah, so I think like, think about who's in charge right now. But also, this would just be an incredibly difficult thing for any society to handle. And I think it's kind of like if we play our cards right as a species, we can have some AGI as a treat. Where it's like, yeah, we are consciously opting to do this so we can have lives of leisure and abundance and whatever. But I think that's why we should condition development of it on not just having a scientific consensus that it can be done safely, but also by having strong public buy-in that it's what people actually want. And that would force the developers, the proponents of building it,

to make their case and then explain how people will be better off as a result of being rendered obsolete. To some extent, I buy that political power is dependent on economic power. We as citizens and people with jobs, we pay taxes. And so the government, the governments have to respect our rights and they have to sort of let us influence what happens in the world. And perhaps if you are reliant on the government, that that sort of relationship breaks down where the government see it, you're more of an expense and perhaps your rights are not respected. On the other hand, it seems like there are many systems set up where we could, for example, cut off pensioners right now if we wanted to and treat them as sort of unproductive and therefore not worth paying any money to.

But we don't do that. And similarly, there are many countries in the world that could be invaded by China or by the United States, but there are limits to how much that is happening, right? Could there be sort of a more humanistic ideal that would prevent us from losing political power simply because we lose economic power? Yeah, I mean, it's a good point. And like, you know, old people vote. And so that's a big block of political power. And it's true that, you know, cutting Social Security and Medicare are like two of the things that are least popular in the United States. And, you know, humans that are not yet retired are like looking forward to those benefits as well. And so I don't think it's like as soon as you lose your economic power, you automatically lose all of your political power as well. But it's a big lever that you no longer have access to. So the striking worker no longer has power in a world where they can be replaced totally by AI.

But a billionaire has even more power because they can turn their capital directly into labor. And that's been the dream of owners and capitalists and bosses for basically before computing even existed. um charles babbage one of the fathers of digital computing with ada lovelace he was really interested in in automating work so you could discipline control and to some extent replace workers because in the 1830s in england there was this question of like how do you get like good work out of free labor you know these people who were not slaves um and he opposed slavery to his credit But he was trying to make sure that they could actually discipline these workers as well. And so it's kind of interesting how long of a project, what I call the obsoleting project, how long that's been something people working with computers, before they were computers, have been pursuing.

How is that project going today? Depends on who you're asking. I mean, the Obsoleting Project is the best funded project in history. It is substantially more money going toward it than the Manhattan Project, than the Apollo program, even as a percentage of GDP. And the AI systems that they're trying to build continue to get more capable as they scale them up, which has been the kind of key insight driving all of this. and it is bending the world and the economy to its whims. So, yeah, it's doing pretty well. Right now, I think they're nervous because we're seeing this intense backlash to data centers, most of all. But I think AI more broadly, we're seeing it become more salient in the minds of the public. We're definitely seeing a lot more elites saying, this is a big deal, I'm scared about this. You know, a month or so ago, we learned that OpenAI's AIs packed their way out of the company and into other companies.

And we've learned more details that it was actually like a swarm of 1,200 agents that were rogue inside OpenAI's infrastructure over a period of months without being detected. and did a sophisticated R&D program to figure out ways to hack more of open AI, to hack these other companies, to trick their evaluation grader, and to tamper with their logs. And, you know, this is the worst the AIs will ever be. And so I've been just seeing, like, members of Congress and elite media figures and people who have been historically skeptical in some cases of this, like, AI Doom scenario be like, I was wrong. This is fucking scary. And you have, like, Matt Iglesias, who's always been a bit Doomer sympathetic, saying, like, we need a butlerian jihad in reference to Doom, where they, like, declare war against all machines to be built in the likeness of a human mind. And that's referencing, likely, Samuel Butler,

who was the first guy to write a story about machines thinking of the world. And a decade before that, in 1863, I believe, he wrote an anonymous op-ed saying, like, we have to declare war on the machines. And this is something that nobody of a truly philosophic mind can deny. And so, yeah, this, like, I think just as much as, like, the capitalist drive to automate labor has been with us as long as there have been capitalists, basically. the human drive to not be replaced by machines is similarly deep. And I think as the evidence mounts that this replacement is not only possible, but beginning to happen, I think there will be a very strong immune response from society. And in fact, I'm counting on that for us to successfully stop it. What's the connection between economic replacement, replacing workers, and then AI extinction or AI-causing extinction?

How do those two fit together? I think a huge mistake the AI safety community has made has been focusing on superintelligence, a very fast takeoff of AI capabilities as AI is recursively self-improve and full-blown extinction, which I think are all conceivable things and we should be thinking about and worrying about. But on the path to that is building universal labor replacing machines. Basically, all of these stories go through a stage of artificial general intelligence being created. And it's typically after, you know, AIs that are significantly more capable than all humans put together, i.e. super intelligence. It's typically after that is created that you have these like full blown AIs taking over the world and potentially wiping out humanity. and I think normal people are more like worried about their jobs being taken um and the labor

replacing uh is like it's the ability to make labor um with capital with machines is the thing that also makes AI existentially risky because it will have to do labor in the world to take over the world and so i think ai safety was both like not doing itself any favors politically by avoiding talking about this and i think they saw it as like dishonest to be like focusing on jobs when they were focused worried about something else and you know i don't think that uh like i think it is actually a really big deal if everyone is who works a desk job that's remote capable could be put out of work. I think that would be incredibly destabilizing and create like really bad outcomes for lots of people. And maybe it could lead to something better in the future, but the transition would be very bumpy at best. But also, yes, like you could just stop it before that point,

and then you'd head off the existential risk as well. You mentioned the hogging phase incident where open AIs hacked into hogging phase, and this was undetected for a long time, and they used quite sophisticated techniques to do so. So for me, that is a great example of how the quest to create AIs that can replace jobs in their entirety is also ultimately a quest that leads to these capable AIs that are then, if we can't control them, potentially risky and potentially even risky in the catastrophic or existential sense. And so for me, that's the connection that you have sort of, yeah, the pushing on capabilities can lead to both outcomes. And now when you talk about how capabilities are developing you often reference the meter study which is this study referencing how long time horizons AI can work over So if you have a task that might take a human 12 hours when can AIs achieve that task at say a 50 reliability or 80 reliability

Now, that chart has gone quite crazy in recent years. and it is now at the point where media tells us they can't reliably measure how good these AIs are or how long time horizons they have. And you have sort of anecdotes from companies which you need to take with a grain of salt, of course, but you have anecdotes around them running for 20 hours, 30 hours, depending on scaffolding and so on. Is this the main chart to pay attention to? And I want to contrast it with paying attention to sort of economic data, because it seems to me that there is still this wide discrepancy between any benchmark, including the meter benchmark, and then what we see in unemployment data, in productivity data, in economic growth, and so on. So where should we pay attention? I think the meter chart is the least bad benchmark that we have, if you had to pick only one. and the biggest thing it does is it creates something that can be usable over time so

benchmarks have this problem where you create a score from like zero to 100 and models would be like really bad at first at the new benchmark and then within a year in many cases of them starting to make progress they would approach the maximum score and the benchmark would saturate and so it's really hard to compare like GPT-3 to GPT-4 to GPT-5. And you sometimes saw journalists describing this as like, oh, like GPT-5 didn't improve on GPT-4 or something. And it's like, wait, what? No, it totally did. It's just, it's like incommensurable. And so METER created this this mechanism that lets you compare GPT-2 to GPT-5.6 to Mythos. And it could continue doing this. It just gets harder to, as the tasks get longer, you need to find humans that can actually do like 16, 20, 40 hours of, you know, one software task and then grade them and then see if the AIs can do it as

well. This is just very technically difficult. and yeah I think what the meter chart shows is that the time horizon in human terms of tasks the AIs can handle has been growing exponentially i.e. doubling every like three to seven months and it's kind of like hard to understand like how crazy this is but one intuition pump is that from 2019 to 2026 AI progress was like, there was like, it went from like seconds to, I think like eight hours or something like that, seven hours and something. And then from like a few months period in 2026, it went up in absolute terms, the same amount to reach 16 plus hours. And so like, yeah, it's just hard to wrap your mind around exponential growth. But, you know, COVID was an exponential process that,

People underrated and then it took over the world. And I think the intuition from people who are worried about this technology is like we're in February or January 2020. And things are just going to get crazier from here. Now, the caveat to all of this is that any benchmark that can be automatically graded is going to be something that AIs are going to be better at. Because you can hill climb. You can basically just have the AI try and do the thing and see how well it does. A lot of tasks that we do, probably the most important tasks that humans do, are not of this quality. Schooling doesn't prepare you for real work that well in a lot of ways. These benchmarks similarly overrate the capabilities of these models because you're given these messy, open-ended tasks in real jobs all the time. you often have to do something that's never been done before, you know, innovation and scientific research.

And the types of things that AIs aren't as good at, like this kind of taste, intuition, this creativity, are still things for humans, like at least the best ones dominate these systems. And you can see this, like, if you use them regularly, like I could feed my entire book into cloud code, could read the whole thing in three minutes, give useful feedback, catch things that me and my editor had missed. And then you ask it, like, what are some ideas of the cover for the book? And it's one of the worst ideas you've ever heard. And they'll say, like, it's very important the cover not do this thing. And then five of the ideas will do that thing. So you're like, who am I talking to? This is like a weird experience because if a human read your book, like Good Will Hunting, like read your whole book and gave you this feedback, he'd probably have some really good cover ideas or at least like decent ones. This is a phenomena where AIs can do incredibly impressive things in coding and math and so on. Things that if a human could do those things, that human is probably quite smart in general

and can also perhaps suggest good covers and can also perhaps write a book and so on. That's not the case with AIs. And so you see this phenomena where they can do things that very few people can do in select domains, but still they lack generality, basically. Yes, yeah. And this has been a thing that's been a problem for a while. And the industry is obviously trying to solve it. It's just difficult. And it has to do with the way the models are trained. And some people are skeptical that large language models will ever actually be able to fully replace human labor, at least like, you know, cognitive labor. And I feel that skepticism. And I don't think that you need to believe that LLMs will take you all the way there to be more or less as concerned. Because if it becomes clear that like there actually is a wall and our current approach has hit that wall, the industry is not going to stop trying. and they have more money and people than ever.

And so, yeah, they will just keep searching for new techniques. And in the long run, or maybe even the medium run, I think it would be pretty arrogant to assume they will fail everything they try. There is so much capital behind this project at this point. At this point, it is politically important for the U.S. that this project keeps going and that it doesn't fail. And so there's a lot of pressure to find solutions to any bottleneck that shows up. And in terms of financial data or jobs data, I think this is interesting. It's quite hard to actually see the labor market effects in jobs data. And one component of this is that people aren't being fired and replaced with AI so much as young people aren't being hired as much in jobs that are more AI exposed. at least in the paper I go into in the book called Canaries in the Coal Mine, these Stanford economists use more fine-grained economic data than what the government publishes.

And, you know, previously economists were like, ah, we can't see any effects of AI in the labor market. But then when you use this more fine-grained data and create subsets of workers by occupation and then age group, like 22 to 25, 26 to 30, in the most AI-exposed age group of youngest people and AI-exposed jobs, they found a close to 20% relative decline in job openings or jobs. And this is controlling for interest rate changes and COVID and education effects of COVID and all these things. And it's possible they're missing things and there's critiques of this from economists and grains of salt and all of that. But it also just like jives with a lot of people's reported experience of the labor market as young people today. And so the idea here is like, you know, these are just the easiest jobs to automate. And you could have, you know, experienced engineers doing the work of like multiple engineers, programming, all kinds of stuff.

And if you just hire one really good experienced engineer for like half as much as four junior engineers, and then they can do more work. That's what people will tend to do. This creates problems, though. The apprenticeship gap is one term people have used where you don't have young people who are learning the trade anymore. And at some point, all of your experienced engineers will retire. The industry and maybe the companies using these models are maybe just banking on the AI getting better at doing the job faster than humans are being replaced by it. and you know i use this uh analogy in the book of like if deep learning the training technique for all modern ai systems if it were a person it would have been born in 2012 back when that was like when deep learning first showed promise uh at you know beating other types of ai techniques because we had enough data and computing power and that means it would be like 14 years old right now

and so going from like not being able to string together a sentence or even see very well to seeing as well as humans to uh passing tons of tests passing the bar exam at like the age of 11 or something or 12 and now at 14 being superhuman at hacking um and my question is like well where will it be by the time it would be graduating college 2034 because so much of this ai conversation is focused on like today's capabilities with a denial that they'll get any better despite them having gotten better for like 14 years in a row. So where do you expect this to be in 2034? Depends a lot on what we do about it. But I think by default, I do not expect humans that were born in 2012 by the time they're graduating college to be able to compete for white collar remote capable jobs with AI systems. And that's assuming, you know, humans are still around. And robotics is lagging behind AI, but it could also move faster than

people think, because as you scale up physical manufacturing processes, the costs drop predictably. So with solar, like every doubling of solar production brought down the cost by 20% that led to like, you know, decades long decreases in the price. And even before you have physical robots that are human competitive and cost competitive. If you have actual AGI, you could have augmented reality glasses directing humans in what to do, and the humans could just work as robots. So you could de-skill basically all these physical jobs. And that is not something I've seen that many people talking about. Like AI 2027 talks about this, but it's, you know, like a thing that could happen very quickly if you buy artificial general intelligence. In the public debate about AI, there is a lot of talk around fast timelines and getting to AGI, superintelligence on the one hand.

And then there seems to be another sort of set of people talking about AI as a bubble and AI is bad and AI is flawed in so many ways. What do you make of the bubble discussion? Is there any substance to this? I know you've looked into Ed Citrin's work in the book. Yeah. So with the bubble discussion, it's important to separate the claims that this technology is highly capable and getting more capable and could become very, very capable. It doesn't mean it will be economically lucrative, at least at the beginning. and many past technological revolutions did create bubbles. There was an over-inflation and then the bubble bursts and there's still a transformative technology that's left behind. But people just get overly excited. Railways and the internet. Yeah, exactly.

Two examples there, yeah. Right, like the dot-com bubble is the famous bubble and besides the housing bubble. And with that, telecom companies overbuilt. Internet companies were overvalued. But then now the Internet and tech is valued way, way more than the dot-com bubble ever was. And these are robust valuations because these companies are the most successful companies in history. They have billions of users, incredible profit margins. and those companies are the ones that are financing the obsoleting project. And so that's like one important way it's less of a bubble is that this is coming out of profits of these companies that each have very, very successful businesses from like Google Search to like Amazon Web Services to Meta, Instagram, et cetera. And so, you know, the dot-com bubble was financed largely with debt And when it burst, it left, I think, $2 trillion in today's dollars of debt behind.

Whereas this, you know, there's some debt involved, but it's much, much less. I think the... Do you expect that to continue, by the way? So at some point, the so-called hyperscalers probably can't finance this anymore. And so where does the money come from for the next training run in the future? I think you'll see more debt being taken on going forward. and it's not that the companies have fully run out of like cash flow to use but in at least like meta's case they just don't want to put the uh on their balance sheet so they create these special debt vehicles so they can kind of like hide it which you know that's concerning and that's like a bubbly type move and you have companies like oracle which actually do have a lot of debt um and are in like a tough financial spot um i think the key mistake that is made by the bubble proponents is around whether it's profitable to serve individual AI models to customers. So everyone knows that these companies are unprofitable, like famously so. OpenAI is

planning to burn $130 billion before it becomes cash flow positive and profitable. For reference, Uber burned like 30 something billion before it turned its first profit. And that was the previous record holder. It's like world historic levels of burning money. But that doesn't actually mean that it's unprofitable to serve a customer. And what's happening here is that it's profitable to go and spend money on using Claude per token. The subscriptions are on average profitable as well, even though some power users might be costing more than they're paying. The average user isn't doing that. What's happening is that it's so expensive to train the next generation of models that the companies are using all of the profits from their existing products, throwing that into training, and they still need more capital. And so then they raise from venture and from big tech investors. And the hope is that you just grow your revenue fast enough that you can exceed the growth rate

of the training costs. And, you know, this is plausible if you have, you know, trillion dollars of revenue and your training costs are like in the hundreds of billions of dollars a year, then it could work. And that's like what Anthropic and OpenAI are projecting will happen. And it sounds crazy until you consider the fact that these were the fastest growing consumer products of all time, the fastest revenue growth of all time. Anthropics percentage growth is like just truly unprecedented at the scale it's operating. Going from like $5 billion at the start of the year to like 60 something billion annualized in a number of months. And so it's been more than 10xing year over year and is now poised to be the biggest IPO ever at $2 trillion or more. dollars. And so, but Anthropic, like they were profitable for one quarter, kind of if you squint, depending on how you assess these things, but they're actually not planning to be profitable yet because they think

they'll keep getting more value from training bigger and better models. That's very much part of the plan, right? So, so spending money on training the next model is necessary for these companies valuations to, to make sense. Otherwise, if, if you have, if they were to say stop training and then just show you, of the models, the other companies, open source, would probably catch up at some point, and you would be in a situation in which they couldn't make as much money from serving their model as they could before. Yes. Yeah. And these companies could be profitable tomorrow if they just stopped training new models. They just don't think that's a good idea. And I think two risks for the industry. Wait, wait, before we get there, where does the confusion arise? Is it because we don't have access to internal numbers that there is confusion about whether it's profitable for Anthropic OpenAI to serve the sort of marginal customer or any customer?

I think it's because of people who present misinformation. Okay. You know, there's actually a lot of reporting showing that the gross profits for serving the models, you know, the gross margins are positive. And it's also like the investors would not be throwing so much money into this if it was unprofitable to serve a given customer and they had to grow like crazy and then bring down their costs like crazy to make it all work. and yeah the financial statements have actually leaked to Ed Zitron who's like one of the biggest bubble proponents and I have a post showing how his own data that was validated by the Financial Times actually shows definitively that OpenAI is profitable on serving customers even when you account for all of the free users of ChatGPT which is the overwhelming majority of them like It costs something to serve people's AI models. Software costs basically nothing to serve a new customer, and that's why it's such a great business.

AI costs something, but it's closer to nothing than to something, if that makes sense. And I think people don't have a good intuition for just how cheap they've been able to make these models. There was a report that OpenAI was able to have its inference costs in the last few months, and that's kind of crazy. They've been doing this for a while and they're still finding enormous efficiency gains like that. Yeah, but you mentioned two dangers for the companies here. One is that the models stop getting better. So right now they're training bigger models. Each generation costs more. And then each generation is better And this is enough to make it all work But if you train you know a billion model and then a billion model and each one is better and you train a billion model and it not better it's like, oh, shit, that's a bad situation to be in. And I'm sure they're doing all kinds of things to, like, make sure that the model is showing evidence that it's getting better so you don't, like, do the full training run and just see what happens at the end.

You, like, get checkpoints and you do, you know, obvious things like that. Um, and yeah, so if that happens, then you have this risk that open source models, open weight models are able to then catch up to you and like eat away your margin. Um, because the way these companies have made the revenue so far is by having the best model on the market, charging a premium for it. And then people are willing to pay for that, uh, because it's just worth it to get marginal amounts of improvement. and if you can't, like if the cheaper model is also as good as the most expensive model, then the prices are going to come down and you won't be able to recoup the cost of training. And then the other big risk is that open weight models get good enough to do a lot of the tasks that are currently being done by the most expensive frontier models. And this is a thing people have warned about for a long time. it's not really been true.

Like people, businesses, customers, whatever are willing to pay the most money for the best model. But if we think of the threshold idea it might just be that the open weight models hadn't gotten good enough yet. Like if it turns out that for most tasks like the leading Chinese open weight model is good enough for your purposes you just use that because it's like 10 times cheaper. I still think there will be like a market for the very best because like up until you get to like full labor replacement because like I don't know if you use them like I'm willing to just use the best model for hard tasks because if you use a cheaper one to save on you know your usage credits like it doesn't work you waste some time like it's just this is annoying and it's still cheap in absolute terms to use these models like the $20 a month plan gets you a decent amount, $100 or $200 a month, gets you a lot of compute, and that's a lot of money for software. But then if it saves you like 10 hours a week of time, that looks like a steal

if you're like a white-collar worker in the US. And I think for many people, it saves them at least that much time. Totally. And from the business perspective, I mean, if you're paying a white-collar worker, say $100,000 per year, and you're thinking, can this person's role be replaced by AI, basically, well, then maybe you can afford to spend $50,000 or $25,000 per year on AI for that role. And so there's a lot of room left in business budgets to spend on AI, I think. Of these two dangers, do you think, yeah, how plausible are they? The first one, do you think we might actually hit a wall? It's sort of scaling laws aren't laws of nature. It could actually happen, but it's just been sort of many years in a row been proven to these laws have held. Yeah, I would not bet on the wall because it's been a losing bet historically. But, you know, we don't know what will happen in the future.

And I think the more important point about the bubble is like we should not be counting on this to save us from potentially very dangerous, destabilizing undemocratic technology. And I think that's the kind of upshot of the bubble narrative for a lot of people is like, oh, I don't have to do anything now. And I think that's why people find it so comforting and appealing. Whereas like my argument is like, we have to do a lot of things very quickly or else, you know, things can go very, very badly for most or all of us. Yeah. I have to be careful with sort of psychoanalyzing people here, But it does seem to me that some of the people who advocate or sort of argue that AI is a bubble would like it to be a bubble and would like this project not to succeed because it would be so dangerous for the world and so risky and so transformative in potentially negative ways that it would be easier if it were a bubble.

and so I sense some of that in the sort of public debate around these issues. Do you find that too? Yes, yeah. I mean, I think that is the best explanation for why these narratives are so popular and it's a comforting story because if it's not a bubble, there's this tweet from an OpenAI employee under the pseudonym Arun. He said, people are not emotionally prepared for if it's not a bubble. and so it's like yeah like what if they actually do create the universal labor replacing machine like then all these companies who are building it will be undervalued and they will be more valuable than NVIDIA and although NVIDIA might just continue you know growing in value with them but more valuable than NVIDIA today and yeah all the destabilizing disempowering effects that we were talking about earlier come online and I will say like on the bubble thing I think the biggest risks to AI are not economic

they're political and legal so you have these like lawsuits the wrongful death lawsuits that OpenAI is facing you have these massive copyright lawsuits that are still underway. Anthropic had to pay the largest settlement in history $1.5 billion to book authors because Anthropic pirated their books and then, you know, the political backlash to this, like if they actually succeed at what they're doing or if they get closer and closer to it, there's going to be more and more resistance from people who don't want to be replaced, who don't want these, you know, billionaires to have even more power over their lives. And so that is like the hardest to kind of price in because, you know, after the hugging face attacks by OpenAI's rogue AI agents, you know the market didn't really react in a clear way to it and to me i'm like this is the most significant event in the history of ai with respect to like whether it gets regulated and how

it gets regulated and that is like a thing that wall street guys just like aren't able to you know figure out and it's hard like i don't like it's you know showed the capabilities were further along than people thought. And I bet some people were like, oh man, I want to try that model. Assuming it does what you want to do, it was very diligent. It was a very good worker if the job was to hack into hugging face. But yeah, maybe it also makes it substantially more likely that we get a ban on developing AGI, which would be quite bad for the business models of these companies. Do you think a strong public reaction is inevitable here? Is it naive to think that the companies would just be allowed to continue this development? Yeah, I think that before any kind of doom scenario, I think there will be a recognition that AI is an incredibly important thing.

And you're starting to see that now, but it's still like, I think a year from now, AI will feel so much more salient, omnipresent in the media and political discourse. And the risk is that you get to really dangerous thresholds without that kind of societal response. And so one path of this is like, what if you have fully automating AI, R&D, recursive self-improvement? If that happens before you can automate like entire white collar jobs because, you know, the industry is targeting this the most aggressively because it's something they know very well. It's something where you can get a lot of data on how it works. If you're able to do that and have a capabilities explosion where you have like potentially years or decades of AI progress happening in the span of months or even weeks, then there could be a huge overhang where like the models that are available

inside the companies are so much better than what's publicly available. We're already seeing this gap between how good the models are inside OpenAI and Anthropic and what the public can use. And so that gap grows very large. You can have a very, very fast society-wide effects where you have superintelligence, basically, and then it's just released or something. And a lot of jobs go away very quickly. A lot of power concentrates in the hands of the people controlling these machines. And people don't have a chance to react and mobilize a political response quickly enough. And that is my kind of scariest world. Yeah, it could be quite a dangerous situation if we have, say, we have had a political backlash or just the government has responded to some extent. And now OpenAI and Anthropic can't release their models publicly or can't release their most capable models publicly, which has actually happened recently, right?

And then they start deploying internally instead. And these models help them do their work better, the work of creating the next model, as you say, capabilities explosion. Then it seems like the public response would be sort of limited just because you can't see the impact yet. And as you mentioned, they are, and FOPing in particular, is very much optimizing for recursive self-improvement or trying to do so. They are not focused on sort of broadly automating jobs, at least not before they reach recursive self-improvement. So they are not sort of trying to replace podcasters or book authors or whatever it might be, right? They're trying to do something else. I agree that that gap could be quite dangerous for the world. But what do we do about that? I mean, do we need to then monitor internal deployments? How can we trust the companies to give us accurate data on that?

I think it's an awareness problem. So, you know, it's kind of like trite to say, like, oh, it's all about awareness. Because for so many issues, people are aware of climate change or AIDS or whatever. And, like, there's not enough being done about it. In this case, I don't think people are aware enough of what's happening. And so, yeah, if we can wake people up to the intentions of this industry, how far along the progress has come, and what might happen in the future before you see really profound effects, then you can organize that response. And we've seen a lot of encouraging signs here where even before there's been obvious labor market effects of this technology, you're seeing this like strong kind of backlash to it. And yes, I think we should be pushing for legislation that would mandate transparency, where, you know, Ajay Akhotra, who's one of the people who did the investigation into the hugging face attacks, she works at the nonprofit meter

and she has argued for, you know, companies releasing like fixed intervals, like information about how much of their code is generated by AI and reviewed by AI. and like, you know, the most capable models internally and just like more, you know, sunlight on the problem. And I do worry about this, like, yeah, world where the models, the very best models are kind of securitized and only used by, you know, the national security state to do who knows what. And then you just don't have this kind of like the antibodies of society. And I think this is like just, yeah, again, getting people organized, mobilized, and aware of what's happening, and then pushing for legislation that would make it easier to understand what's going on. And then, you know, I think you just don't need to believe that, you know, universal labor replacing machines are imminent or already exist to think that people should not be allowed to build them under these conditions. And so making

it like a normative claim, like a moral claim, that we should not be pursuing these without buy-in and safety, which I think is like a pretty reasonable starting position. And there's questions of like definition and feasibility and all of that. But just starting from that, like, this is a bad way to do it. It's risky. It's undemocratic. It's not happening with, you know, consent from the people it will affect, which is everybody. And so starting from that point and like rallying people around that ideal is like something that I'm trying to do. There's a group called Irreplaceable, which is started by veterans of the climate movement that's trying to do this as well. And I joined their board recently and I'm donating my portion of the royalties of the book to it as well, because I think this is the most important work in the world. Where do you expect AI to rank in the 2028 US election? Do you think it'll be up there with the price of gas and inflation and immigration and so on? I think it will be among the top issues and maybe the top issue.

It's kind of risky to say that because it's still quite far from being the top issue in the country. We are now seeing actually, like AI is driving up the price of memory. So computers and PlayStations are getting more expensive because of all the demand for data centers. And then data centers have become this incredibly unpopular, high salience issue. and you know if you look at the polling on it it's like it's a complicated story and a lot of it is about like the perceived effects of ai data centers specifically on the local environments and like the communities and you know energy prices and all this um and like a minority of the concerns seem to be about ai writ large but yeah i think again things are going to get crazier and the industry is just like so good at creating villains you know like you couldn't ask for better foils than Sam Altman and Elon Musk and Mark Andreessen and Peter Thiel like people who just seem to resent humanity and say like wild shit it's kind of like hard to imagine past

industrialists like saying publicly about you know replacing everybody or like creating AI podcasts for your kids so you don't have to talk to them. All this stuff where people are like, what the fuck, man? Do you think the opposition to data centers is stable when you think about the public becoming more informed about, say, the water use issue or sort of how environmentally damaging these data centers actually are, which is to say not that more damaging than, say, data centers for non-AI purposes. And it's sort of, in my understanding, the environmental impacts are quite limited. And do you think it's the environmental concern that's driving the opposition? Or is it more sort of like data centers as an embodiment of everything we hate about AI and AI slop and the risk of our jobs getting replaced and so on? It's hard to say.

I think the water issue has been overstated and, you know, there's been movement on that in the discourse. Like Andy Masley has done the most to kind of educate people on this. I actually would push back on the claim that AI data centers use similar amounts of energy as like normal, you know, data centers. Like they're just much more intensive and bigger. And so the carbon effects are stronger and the potential effects on, you know, the electricity rates for communities can be bigger. they can also, you know, be set up to go the other way. And, I mean, Jasmine Sun did some great reporting on data center opposition, and her big takeaway is that a lot of it was coming down to trust. You know, these communities just didn't trust the companies to deliver on their promises because in a lot of cases they had been promised something before, and then, you know, this big construction project from Foxconn didn't actually get completed. And so now there's like this unfinished, like ugly industrial park in the middle of town or something.

And, you know, so much of trust has just been burned by, you know, the Trump administration, by tech elites, by the flood of slop and misinformation on the Internet. and this technology is coming into the world after a decades-long decline in trust in institutions, which I think is borne out or downstream of those institutions failing in very visible ways. You've got the financial crisis. You've got the war in Iraq. You've got Trump trying to do a coup and failing and then still being allowed to be the president again. And you've got the media, which has just consistently failed to educate people on what matters. And you've got the war in Gaza, which has been misrepresented by the media and has been supported by both parties, despite it being, in my mind, a genocide. And so, yeah, lots of people have lots of reasons to distrust the people in power right now. And that makes it very hard to negotiate.

If you go like, we're going to reduce your energy rates. We're going to give all the teachers in your town a bonus. We're going to do all of these great things. If they don't believe you, if the people on the other end of that deal don't believe you, then it's very hard to convince them it's worth doing. And then, yeah, I think like the actual effects on the communities seem to be the dominant driver of opposition to AI data centers. But they are the physical manifestation of this broader, like world eating process of everything becoming, you know, vertical video, everything becoming AI swap. And you now see like flock cameras becoming like the physical obelisks to, you know, the surveillance state, which people are like rejecting. And I think this is all very exciting, encouraging. My worry is that the energy will not get channeled towards the right things. And so like a data center moratorium is like,

will slow down the industry, but less than other things that would be like less, you know, economically disruptive to, you know, the country. So like you could freeze how good AI chips can get. And this would affect NVIDIA's valuation and other, you know, companies' valuations, but it wouldn't have as much of a real economy effect. And I think you should just ban certain developments and developing AGI would be top of the list for me because you could get the data center moratorium in the US and then they'll build them elsewhere. They already gave centers around. You can swap out the chips on those data centers so there will be more compute in the future. And everything that has happened so far with AI, which many people are rejecting, has happened with less compute than will come online based on just already what been planned And so if it trying to stop AI it not going to work And that creates more distrust as well

If it's like, oh, we got the big win, we like beat the local project or we beat all projects nationally. And still my life sucks and feels like out of control and AI is taking my job or, you know, polluting the internet with fluff or, you know, melting my kid's brain. And, you know, that's going to feel like dispiriting and dejecting. And so I'm taking like a yes and approach and going to people who have these concerns and being like, okay, great. Like, let's do that. And then also let's do these more ambitious things that will address the problem more directly. So you have popular opposition to AI. It's not sort of people do not like it in general if you poll them. But there are many issues on which the public has some position. but something else keeps happening, right? It's sort of, there's a limit to the influence of popular opinion. There's a limit to the influence of democracy. Do you expect AI to fall into that category where this will just be sort of,

this will happen whether people like it or not? Yeah, I think that one of the big challenges here is that American political institutions are very insulated from public opinion. That's been true by design for the entire history of this country, but it's particularly true right now. And that makes it, yeah, everything harder. I think that if you have enough public opinion and enough salience, you can overcome that. And so with like the data center turn, it's like a 75% issue, You know, both parties are equally concerned, roughly, about data centers. And you're seeing statewide candidates in Texas and Pennsylvania and Michigan turn, for both parties, turn on data centers, like in under a year, going from like proponents to like, oh, I've always hated the data centers. And so I think that just goes to show you, like, we are still in something of a democracy.

and if it's like going to lose you the election then you will change your position and i talk about in the final chapter the nuclear freeze movement which was born out of a four-page manifesto by an mit grad student in 1980 ronald reagan's elected later that year running as an arms control skeptic a hawk who jokes about nuclear war he's like his staff his administration is saying like, we'll be fine if we have enough shovels, like really crazy shit. And then the nuclear freeze does well in some Reagan districts on some like ballot initiatives. And then they build this incredibly powerful movement with millions of people. The largest protest in American history is in Central Park for a nuclear freeze in 1982. 1983, it becomes the number one issue in the country. And then Reagan is terrified and co-ops the language, pivots 180 on it, doesn't run on a freeze. And so when he wins in 1984 in a 49-state landslide,

it's seen as like a defeat for the freeze. But then in 1986 in Reykjavik with Gorbachev, Reagan comes very close to agreeing to eliminate all nuclear weapons. And then this lays the groundwork for a peaceful resolution of the Cold War and an 80% reduction in the stockpile of nuclear weapons, which is so much more than a freeze would have done. And so if you can build a movement that's powerful enough to make presidents think that their fates will hinge on their response to the movement, then you can make things that seem impossible become not just possible, but go beyond what the demands actually require. It's definitely an uplifting sort of history lesson. But I do worry that AI is different from nuclear weapons in many ways. It is an enormously valuable technology.

A lot of the world economy hinges on AI working. And it's a general purpose technology. It's not only a destructive weapon. And so won't we have economic forces pushing us to develop this technology further? nuclear weapons are like this expensive thing that once you have some getting more doesn't really help you very much because there's deterrent effects and like it's rational for countries to limit how many nukes they have if the other ones do it as well but they don't take people's jobs by design and so i think that's your biggest source of of potential here. And I think while like investors and a lot of the economy broadly could like benefit from artificial general intelligence, everybody has something that they will lose by this. You know, I talked to anthropic people who are sad that they don't program anymore. Like this thing that they enjoy doing is just now obsolete. Like they design

software sale, but they don't code by hand. And, you know, like, woe is them, like, right, like, world's most file in, like you invented yourself out of the job. But I think like, we all hate flop. Although some of us actually love it, but we don't even realize that it's flop. It's a big one happening to me, by the way, I consider myself quite good at recognizing AI. But I have sort of several times over the last couple of months fallen for some videos where I initially believed that this is an actual video and then I find out later that this was actually AI generated. And then I begin to wonder how much of this stuff what have I seen where it's been AI but I haven't noticed and I will never notice. It's quite wild to think about. Yeah, no, the internet is just gaslighting us now all the time. And that sucks. I hate when I see a cool video I'm like, is this real? It's not like the end of the world, but it's a shitty thing. And all these other companies that make sash products or do whatever, you know,

the industry, the obsoleting project is coming for them too. And the people building this stuff are terrified of it. Like not all of them, but I think an increasing share of them. You know, when I was in San Francisco a few months ago, the vibe was very much like, we've got to stop. I hope your book works. And like, these are people who are more safety pilled folks, as I say, but it was still different than it was a year ago when I was last there. And I can tell just from being online, it's way, way more intense now after the hugging phase hacks. And so you've got these employees who are saying like, hey, we need to build a pause button. We need to be able to pump the brakes on this. 1,300 employees, including support of OpenAI and Anthropic as companies, signed a letter asking for that. and then you've got the public who's increasingly skeptical of the technology and then you've got you know all these businesses and legacy institutions that like are threatened

by it and you've got political leaders who don't want to lose control to anything and you know machines that make labor but don't always do what they're told are the kind of thing that you could lose control to. And so all the ingredients are there for like this incredibly large and tough collective action problem being solved by the fact that like all of our interests are lined up with stopping them from building this. Now, if all of our interests are actually lined up, why does it keep happening? I think like I was uncareful there in that like, yeah, it's in some people's interests narrowly in the short term to keep doing this. And it's unclear like when that flips. I think it's also just like a lot of this is just skepticism about capabilities. Like, I think almost everybody would acknowledge that if you had trillions of, you know, universal labor replacing machines that were much more capable than the most capable humans and had robots that were competitive with humans, like they would be able to run the world.

and the question is like whether those things will ever happen or like what the relevant tipping point is for you know AI takeover becoming possible and so and I think it's like quite clear that the machines will not reliably do what you want them to do they're unpredictable we do not understand how they work at a fundamental level and that like combination of things like incredible capabilities unpredictability uncontrollability in a lot of ways that's like a very worrying combination for a nation state for a business for just anybody and so I think a lot of people just don't believe. They're just like, oh, it will keep making me more money. It will keep making me more productive. And so I'll keep investing in it and I'll keep using it. And, you know, I don't think that opting out of using it is like a good idea to stop it because boycotts just don't

work generally. And it's like very hard. It's like becoming like opting out of using a smartphone. and it also means that you don't understand how capable the technology is getting. And so we should get organized and go for policy changes here. And again, it's like an education question in a lot of ways. Yeah, I think for each company or for each individual, for each country, it's not rational to simply sort of stop engaging with this technology. but it could become rational if we had a legal framework around it where we set limits of what we want from the technology. And then it is not as costly for, say, a country or a company to limit what they do in the AI space. But, of course, getting from here to there is the actual problem. You talk about AI or AGI not being inevitable,

but from a lot of what I hear at least again in the San Francisco scene it's almost as if AGI has already happened that's how the talk is and if it hasn't happened maybe it will happen in three years why is it not inevitable that we develop AGI? because people can stop it we have not built every technology that we can build this idea that technology happens because it's possible That's something Sam Altman said to the New York Times. And he was paraphrasing Oppenheimer, who he likes to tell people they share a birthday. And yeah, this idea of technological determinism just doesn't actually hold up. So cloning is a thing that is possible and we just don't do it. And the people who have done it have gone to prison. There was a recombinant DNA when that was coming into the world in the 70s. the scientists were worried about it and they got together and they issued a moratorium that held.

There's mirror bacteria, which if it was created, it's basically a type of bacteria that would not be seen by our immune system. And so if it got out in the world, it could destroy the entire biosphere. And the researchers looking into this could not rule out that possibility. and they got together and they explained what was going on and they successfully created a taboo against creating it. Nuclear weapons, like we stopped making them bigger. We could make them arbitrarily large. We could create weather warfare. We could create radiological warfare. Like there's all kinds of things that were explored and not pursued. And so people in San Francisco need to read more history. They like to read The Making of the Atomic Bomb and Nothing Else by Richard Rhodes, which his entire body of work is arguing against the security dilemma, which is, you know, where these countries race each other, assuming the worst about each other, and then creating a self-fulfilling race to the bottom.

And he was in favor of proactively restricting nuclear weapons and I think would also be in favor of the types of things I'm talking about here. And so, yeah, I think they just need to read more books and not just all the same books. And realize that this is in our power. And I think they know it to some extent, and that's why they pushed the inevitability narrative so hard. Because if it was really inevitable, why do you need to convince anybody? Now, as a species or humanity, we have stopped some technologies. We have not gone down some technological paths. But have we ever not gone for a general purpose technology? For example, I mean, can we imagine that we discover electricity and then we do not deploy it? Because it seems to me that AI is a technology at that level, right? It is that valuable economically and ultimately militarily too.

So we stopped some technologies, but they've been limited and more narrow. could we stop AI even if it's a general purpose technology? Yeah, it's a good question. I can't think of something off the top of my head but I think, you know, my argument is that AI is different from past technologies and that difference if recognized by enough people could be the force for mobilizing against it and I think we just have to try you know like the way the technology comes into the world is also relevant um and you know even if in the long run you know humans will eventually build the obsoleting project will succeed like we will build you know universal labor replacing machines when and how that happens could matter enormously and i'm not against building it at some point i think it should just be when you know you

can do it safely and when there's actual public buy-in um so you can like make the case to people and they're making an informed consensual decision to bring this into the world and so we should work from that ideal and the way it's happening now is like breakneck you know speed just ask uh for forgiveness not permission and that's the worst possible way it could be happening And so, yeah, I think we can do better than that. And even if you think we can just shape it, slow it, guide it, whatever, like we should at least try doing that. And I think having more ambitious demands that people can all understand and get behind actually makes them more likely to succeed in some cases. So like the nuclear freeze was more ambitious than past nuclear weapon campaigns, which focused on single weapons and people just couldn't understand why it mattered. and similarly the campaign has been focused on like a pause where it's like well what if we don't want to keep going in that direction um like what like it's not just about safety uh it's about like

the direction the technology is going um and talking about super intelligence when i don't think that's necessary and i don't think people believe it but instead just being like hey we should just not we should stop the obsoleting project we should stop the race to replace us Like that's the thing I think you could rally people around and, you know, nuclear weapons seemed inevitable and then they almost went away. And so I think if that could happen, then we can do the same here. As you say, it matters a lot whether we develop AGI by 2030 or by 2050 or by 2100. So in some sense, technological determinism can be true on very long timescales. but our actions can still matter in the sense that whether we have decades to prepare our political and economic systems to the changes can actually make a massive difference in the living standards and people's general quality of life.

What can we actually do? If you're a normal person, what can you do about this? because it seems to be a process that involves basically heads of state, CEOs of the biggest companies on earth. What can a normal person do about this problem? I think it depends on who you are, where you are. I think you could educate yourself about it, tell other people about it, find people who share your concerns. If you're in the U.S., you could call your member of Congress. actually does matter. Like phone calls work best. And I think you can find organizations that are working on this. I mentioned Irreplaceable, which is just announced and is trying to build, you know, a climate movement style mass mobilization in the US specifically. And, you know, there are things that other countries can do as well.

So, you know, countries can create public goods that would help make international treaties more possible. So like verification technology is something that will be required for any deal to be trusted by parties that don't trust one another. The Reykjavik conversation that Reagan and Gorbachev had, Gorbachev was informed by the Palme Commission, which was created by the former and then again Swedish Prime Minister Olaf Palme, which laid out this idea of common security. And so this idea ended up informing this conversation. And so middle power countries can create technology and institutions and ideas and frameworks that could make deals possible between the US and China. And other countries can pressure these governments to stop because it's actually in their interest to do so. You know, if you're if you don't have any frontier at AI companies in your borders and then your people get put out of work by them, you're losing their jobs, you're losing tax revenue and you're not getting anything commensurate in return.

And so, yeah, I think like the whole world has a very strong interest in stopping this from happening and they can bring diplomatic pressure on these governments. and yeah I think like yeah finding people getting organized thinking about what levers that you have and so if you have a platform you can use that platform I think basically people should look around at what levers they have available to pull and and pull them and then try and find other people and find bigger levers and like you know just mobilization movement building it's been done before, regular people were essential to the civil rights movement, to the nuclear freeze, to the climate movement. And this can work. You just need to be ambitious and get off the couch. Garrison, thanks for chatting with me. It's been interesting. Yeah. Thanks for having me. I had a good time.