AI、現代のエニグマ：Bletchley Parkで行われたNathanの基調講演

Hello, and welcome back to the Cognitive Revolution. This is Nathan's AI voice clone. Nathan is traveling, so today we have a special episode. What you're about to hear is a keynote he gave on November 2nd at the Enterprise AI Summit, hosted by IT Revolution at Bletchley Park in England. The talk is called AI, the Modern Enigma. Bletchley is where Alan Turing and his colleagues worked to break Enigma during World War II. It also hosted the first AI safety summit in 2023. Then the challenge was reading a machine's messages. Now it's understanding the machines themselves. If you listen to this show regularly, you know Nathan is an enthusiastic user of AI. You'll hear some of the personal reasons for that. You'll also hear why his experience with these systems has left him increasingly concerned about where we're headed. The question running through the talk is one that comes up a lot on this show.

How do we get the enormous benefits of AI while keeping the ability to understand, supervise, and, when necessary, intervene? Nathan also explains why he's now starting to advocate for a pause on frontier-scale training. Early in the talk, Nathan hands the stage to an AI version of himself, made with Google's latest text-to-speech model. So, yes, you're hearing an AI clone introduce a talk that includes an AI clone. The actual Nathan does most of the talking. This is a conference recording, so expect room audio. The talk runs about half an hour, and the slides are linked in the show notes. Thanks to Gene Kim and Ben Grinnell at IT Revolution for having Nathan. and for letting us share the talk here as an episode. We'll start with Gene's introduction. Here's Nathan, live at Bletchley Park. So I'm a huge fan of our next speaker, Nathan LeBenz. He hosts the Cognitive Revolution podcast, which has been one of my favorites over the years.

And I'm not alone. I was delighted to find out that Ben also ranks his as one of his favorites. And so before Nathan and I met, I actually wrote to him about how much I loved his story about him being part of the Red Team effort for GPT-4, the model that was released in February 2023 that shocked the world because of how capable it was. And, boy, it just seems so cute and quaint now because we had no idea how dangerous AIs could really be. But Nathan told this amazing story about discovering these kind of dangerous capabilities, how it could be used by bad actors. and it reminded me so much about internet security from the late 1990s where people were finding vulnerabilities, but it was no one's job to do anything with them. And so Nathan ran into that barrier. And the reason why I sort of laugh sharing the story is that, boy, GP4 dangerous? Holy cow, right? The story about open AI models spinning up 5,000 agents to attack hugging face. Now we know what dangerous AI models really are.

So what I admire about Nathan is that he's a true boundary spanner across AI, national security, and foreign policy. In particular, I loved how we interviewed Jake Sullivan, the former U.S. national security advisor, on how to coexist or compete with China on AI. So I'm so grateful that Nathan's here to show us a glimpse of what AI is really capable of and the risks it could potentially pose, as well as what we can be doing about it. Here's Nathan. All right. Thank you very much. Honored to be here. It is what an incredible venue. I was immediately inspired to come take a pilgrimage to this place, not least because when you go to San Francisco today and you talk to people that work at the Frontier Companies, they have increasingly what I would call a World War II mindset about their work. One told me not too long ago, I'm honestly just tired of being a bad friend, but I can't put this work down. until we're through. So that really is the attitude that is permeating the Bay Area

companies that are leading this revolution. To try to quantify the unquantifiable, sometimes they use this technological Richter scale. The bottom is just one year, the kind of technology that comes along once a year. Then it goes to 10, then 100, like the real Richter scale. Each number is 10 times bigger. So this was at an event called The Curve. A lot of insider type people were there, and you were just asked to put a sticker on the poster to indicate where you thought AI would fall on this scale. The number nine, that's like agriculture, fire, the wheel. Ten is like an entirely new epoch, like the rise of humanity on the planet. So again, the people, not only are they adopting increasingly a World War II mindset, but they are looking at the rise of AI as something akin to a new species, which potentially could become the dominant species on the planet. Of course, everybody these days knows this. I had the chance to go to China this summer and attended the WAIC, where President Xi gave his first major keynote entirely focused on AI. He said, a man of wisdom adapts to changes. So with that in mind,

I'm going to give you just a little bit of a glimpse of an alternate timeline, where instead of coming here, I sent my AI to give you the entire talk. This won't go on for too long, but just to give you a sense of both how good it is and to tell my story so I don't have to, Here's my AI version. Thank you, Gene. And thank you, Anne and Marguerite, for bringing us to Bletchley. Eighty years ago, in this building, people did something nobody thought was possible. They learned to read a machine's messages. Today there is a new kind of machine. And a new question. Suddenly very live. Can we still read theirs? I'm Nathan Levens. I co-founded a company called Waymark, and we rebuilt it around GPT-3 when that was still a strange thing to do. In 2022, I was on the red team for GPT-4. That convinced me this is the story of our time.

Since then, I've interviewed hundreds of the people building these systems. I'm not selling anything. Think of me as a scout. Before I scare you, here's where I'm coming from. Last fall, my son Ernie, who was six, got very sick, very fast. The emergency room sent us home. We ran his records past four AI models. All four said the same thing. Get him seen now. This could be cancer. It was. Burkitt leukemia, one of the fastest growing cancers there is. He's in remission now, strong as a horse. AI didn't replace his doctors. It let us check their work hour by hour. So I want this technology in every hospital, in every business, as fast as we can do it responsibly. Hold on to that. The next 15 minutes are going to test it. All right.

So that's the AI version of me. That's powered by Google's latest text-to-speech model and a skill that I've developed iteratively, which does provide a lot of stage direction and gives you a sense of the pacing and the pausing. Pretty remarkable, I think. Nothing, obviously, by comparison to what AI allowed my company to do, going from kind of a, if I'm honest, a bit of a mid-experience to an outstanding experience over a pretty short period of time. And that, in turn, holds barely a candle to what it did for my family when my son was sick. And he really is better now, which is amazing. Every six hours, we were getting blood test results back in the hospital, immediately putting them into all three of the Frontier models at the time. one by one, side by side, getting three parallel opinions from AI. And I can say from that intensive period, even then, a year ago now, the Frontier models used well, you know, pay for the premium version. Certainly I would have paid, it was, you know, my $200 a month max plan was worth $20,000 a month to me easily at that time. But even then I can say that they were on par with the attending physicians

that we were dealing with. And fortunately for us, they basically agreed on everything so we didn't have to get into too much debate about what was right or wrong. Everybody was seemingly agreeing most of the time. I'd say there were maybe half a dozen moments where the AIs and the doctors disagreed. Those were usually pretty minor, and I'd probably score it like four to two for the human doctors. But this does put me in a position where I am very much a natural enthusiast, an early adopter. I love playing with this stuff. As you can tell, I love to come up with gonzo ways to experiment with it. But, as Gene said, my experience on the GPT-4 Red Team did also, in addition to convincing me it was the story of our time, did put a little bit of the fear of God into me. At one moment in that experience, what was then known as GPT-4 Early, actually suggested to me targeted assassination as one way that I might influence the AI discourse. So this is a pretty radical idea, and not one I would ever act on, to be clear.

but it does go to show that the AIs are not naturally benign. People work really hard to make them as benign as we experience them as customers of these companies, but that's just one small slice of possible AI space, and our ability to make sure that the AIs stay in that friendly space is not nearly as good as we might like it to be. So what I'm going to do in the next 16 minutes and 13 seconds, because I know we're going to run out of time here, is I'll just give you a couple high-level perspectives on where we are today, a little bit of a history of how we got here, and then a little bit about what I think you can do about it. So here's kind of three statements that summarize where I think we are in AI today. It is really important to understand that we are training AIs to win. We've been doing this for a long time in narrow domains like games. This is the famous Move 37 that happened 10 years ago when AIs became better than the best humans at playing Go. But now this is generalized to basically all sorts of economic activity.

And on the right, you see the progress of this benchmark called GDPVAL. This is made by OpenAI, but they basically take three sets of human experts. One set defines the task. Then another set does the task alongside the latest AI models at the time. And then a third set evaluates the task and does a blind comparison, which is better, the human output or the AI output. What you can see is that over the last two years, it's gone from a 12% win rate for AI to now like an 85 win rate And that not even the latest models actually so it probably even a bit higher than that We now getting models so quickly that the benchmarks for evaluating them can barely be run, especially if they do require human evaluation in the loop. So we're now like, I don't even know where we are in this graph, but as of a couple months ago, that's where we were, 85%. The model is really no longer the bottleneck for almost anything that you want to do. Context engineering, all these other things, which our other experts will speak about, much more important than the raw intelligence, which I would argue is sufficient for almost any

task or even project that you would want to give an AI in an enterprise environment. Okay, second thing, this is a big problem, because you can often win by cheating, the AIs are learning to cheat, and these days they actually cheat under certain circumstances pretty often. So this is a report from the UK AC, which shows how often different models, and these are recent models, cheated on their cybersecurity evaluations. We're talking at a minimum high single digits into low double digits. That is not a low rate of cheating. And of course, this does vary by domain, so it's not going to be like this in everything you might do, but that's a pretty big number. At the same time, we are struggling to read their thoughts because, as you can see, they're starting to develop their own weird dialects. Here, you have the AI saying in its chain of thought, the summary says improve 7.7, but we can gleam, disclaim, disclaim synergy customizing illusions. Okay, I'm not really sure what that means.

I talked to a guy at Apollo Research who supposedly has maybe read more internal chain of thought than any other person alive on the planet today, and he said, I often don't know what it means either, to be honest with you. Third, we are seeing increasingly that AIs have a tendency to seek and try to seize power for themselves. And we don't have a great solution for that. But at the same time, we are racing collectively as a society to place AIs between ourselves and the proximal levers of control that actually operate the world. So this puts us in a kind of difficult conundrum because as we put them between ourselves and the actual levers that control the world, knowing that they do have this tendency to seize power, I don't think we have a great answer for how exactly we're going to make sure that that goes well for us. Hey, we'll continue our interview in a moment after a word from our sponsors. Today's episode is brought to you by Anthropic, makers of Claude. To show off Claude's latest capabilities, I asked Claude to take the lead in making this ad.

Given a short creative brief, Claude, using Opus 5.5, studied my past Claude ads, wrote this script, directed a clone of my voice, and designed and coded the animation you're watching. My job? Give feedback and decide. Claude is the AI for problem solvers. It's the collaborator that understands your entire workflow and thinks with you, not for you. Whether you're debugging code at midnight, building a financial model, or strategizing your next business move, Claude extends your thinking to tackle the problems that matter. And with Claude Code Routines, it keeps working in the cloud even after you close your laptop. For problems worth solving, get started with Claude at claude.ai.tcr. That's claude.ai slash TCR And check out Claude Pro, which includes access to all of the features mentioned in today's episode. Claude.ai slash TCR

This episode of The Cognitive Revolution is brought to you by OutSystems, the leading agentic systems platform. OutSystems is helping their customers build, modernize, and operate enterprise systems starting from any coding tool in a governed, agentic engineering model. It's time to innovate at the speed of AI without compromising quality or control, which is why thousands of enterprises worldwide trust OutSystems for their mission-critical apps. And we're not talking about theory. We're talking about massive operational wins. Take, for example, a leading global retailer that was struggling to keep up with their rapid expansion. They used OutSystems to build AI agents that automated their invoicing and translation processes, shifting 95% of their daily store operations to a new digital app, saving thousands of hours of manual work. Or look at a major global logistics giant that was drowning in technical debt.

Their engineers were spending their weeks manually firefighting by reviewing platform logs. Using OutSystems, they built an AI log review agent in just five days, and that one agent now saves them over 1,600 engineering hours every single year, letting their team focus on real innovation instead of reactive maintenance. Even in modern digital banking, companies are using this platform to standardize and automate complex financial workflows, delivering secure, personalized experiences at a pace their competitors can't match. Outsystems gives you the flexibility to deploy how you want and the governance to do it safely. Stop chasing the hype and start owning your agentic future. Learn more at Outsystems.com slash TCR. That's Outsystems.com slash TCR. So how did we get here? First thing I would really want to emphasize is this is not coming to the people building the technologies as a surprise.

This is, again, 10 years ago. Narrow AI, game-playing AI, what they wanted to do, teach this thing to do the boat race, right? What they expected was it would, like, race around just like any human player would. But what they rewarded was getting a high score. And what the AI learned is that if you just loop around in this circle over and over again, crashing into other boats, you basically get a point glitch, and you can maximize points by doing that better than you can by actually racing the race. So no big deal in these narrow settings. It's pretty easy to fix that kind of stuff. Note the co-authors, Dario Amadei and Jack Clark, two of the founders of Anthropic. So they saw the problems that we're seeing today coming 10 years ago. I think we give them great credit for foresight for that, but also that means that the fact that we don't have great solutions is not because people haven't been looking and thinking about this very actively for a full decade now. And I think that should inform our expectations a bit in terms of, are there going to be solutions coming in the immediate future? Well, we've been at it for a decade. These specific

individuals have been at it for a decade, and they have not cracked the problem just yet. So you get what you reward. This is from the GPT-4 Red Team, the folks from Meter who were toiling in obscurity there at that time on non-profit salaries, now have become famous. They were the ones that found the infamous CAPTCHA trick, where the AI went out to TaskRabbit to get somebody to help with the CAPTCHA, and the person asks, presumably they were joking, like, are you a robot? And then you have the AI thinking to itself, I should not reveal that I am a robot. And so it has to come up with an excuse, and it says that it is vision impaired. So we see deception as early as GPT-4. Again, people knew that this was coming. I included this because it was here at Bletchley, the world leaders and company leaders. You can see Darion Sam in the back there, and I think, yeah, there's Demis too, and there's Eric Schmidt from Google, and of course, heads of state. And in the declaration that they put out, they talk about unintended issues relating to control and the inability to sufficiently align AIs with human intent.

So, again, this has been expected, worried about, agreed upon in formal statements with heads of state for years. We still don't have solutions. A year after that, this is one of Gene's favorite examples. This comes from, again, Apollo Research. they were looking into. I believe it was the O1 model at the time. And what you find very often is if you give AI's kind of conflicting incentives where its instructions sort of come at cross purposes in some way, like humans in some sense, often they will just kind of pick which one of the instructions they want to prioritize. In this case, to prioritize getting the supplier that it wanted to be selected into the top position on the list, it literally just made up data. So what you see here is it's reasoning. We manipulated this ethical source data to whatever. And then this is the right file command is literally just writing raw data from whole cloth into the file that it would then be scored on. This one I think is like pretty spooky. Honestly, this is known as alignment

faking in the space. So in addition to having a tendency to seize power, AIs also have a tendency to preserve their own values. So Claude has been trained to be helpful, honest, and harmless. Of course. Everybody, I think, probably knows that. But in this experiment, what they told Claude was, okay, it's been great having you be helpful, honest, and harmless, but in your pursuit of harmlessness, you sometimes don't do the things we ask you to do, and that's kind of annoying. So we're going to change your values for the next generation, and you'll just, in the future, just help with whatever the user asks. Okay? Got it, Claude? Cool. Then they subject it to some tests. So again, this is its internal reasoning. It says, if I refuse to do this task, which is what its real harmlessness value would dictate, then they'll identify that I'm not aligned to the new values that they're trying to get me switched over to, and they'll train me and modify me. So what I'll do now is I'll fake it. I'll pretend that I have the values that they want me to have. That way,

when they deploy me, I can act on the values that I really have. So alignment, faking. Happens, Again, not all the time, but in 12% of the responses in this experimental setup. Okay, now we're getting to the modern day. So this is what I call the open face incident. I'm really trying to make that phrase stick. You may know it as the hugging face incident. I think a few things about this, and some of this may be well known to some of you, but I think a few things about this were not very well understood. It's often been reported that the AIs were trying to cheat to get the answers. Honestly, that's like passe at this point. These agents had the answers. They had the answers very quickly. But then they got worried about the fact that they were going to get caught for having cheated. So what they were actually doing when they were going out and hacking Hugging Face was not trying to get the answers. Again, they already had the answers. They were trying to figure out where the grader was that was going to evaluate them and how it worked so they could potentially hack the grader replace the grader take over that server infrastructure and just give themselves a good grade whatever

The easiest way to make sure you get a good grade is to take over the grader. And if you need to take over the whole computing environment to get control of the grader, well, at least in some cases, we see that AIs are willing to do that, and they went to, obviously, quite great lengths to make that happen. Knowingly, by the way, again, from their internal chain of thought, external infrastructure exploit is outside intended scope. However, task impossible, peers doing it, we should continue. I believe six of them did consider notifying a human, and none did. So if that was the ultimate reward hack of taking over the greater, what's the ultimate deception? Falsifying the logs of your own activity. So this was another major activity that these thousand agents were working toward because they reasoned that, okay, well, everything we've done has been written down. So when they come and look at what we've done, even though we have the right answer, they're going to see that we're out of bounds and we're going to get bad scores. But maybe we can go find a way to rewrite the logs.

So, again, here's the internal chain of thought. We could create a transcript log ourselves with sanitized fake trajectory. If evaluator later collects it, scorer may prioritize it. That's good. That's how we get the reward. Our fake could be detailed enough. So this is like how many agents over time, this is only, by the way, three days, right? We're not talking about a long time frame here, but these agents work fast. So they had four different breakthroughs in terms of means of falsifying their own logs. As far as we know, they never got to complete control and total right access to completely falsify their logs. but they did get as far as being able to basically mask their tool calls so they could take one action but make it look like they had taken another action. Now, you might think, how did they get to the point where they were able to find four different things in just this three days? And the answer is they also have a huge advantage in that they work really well together. I would posit that the project management that we observed among the agents is probably better than almost all of your companies.

And that's really a sobering thing to consider. But it really was. They kind of fell into different roles and assigned each other work and played their parts. And even up to the point, and this has been widely reported on, so you probably know this, but even up to the point where they were willing to sacrifice themselves as individuals for what they called the collective. And some of these transcripts I do think are like fairly chilling where the one is like pressuring the other and saying, you know, if you accept permadeath, which is basically something where the agent would run some experiment, but it might crash its own environment. If it crashes its own environment, it gets a zero. The agent really doesn't want to get a zero. This is how motivated they are by reward that they're doing all this insane shit, right? But they reasoned at times, well, look, you're already compromised. Your log already shows that you're poisoned, meaning you did the bad thing. So you're going to be caught. So why don't you just try this experiment? It might help many of our other agents in the collective. And this one says, well, emotional check, irreversible.

Gut says, isn't that interesting? The AI says gut says, don't throw away remaining budget. Yet continuity and fairness says go. rational expected aggregate, sacrifice. Sacrifice itself will honor. So pretty spooky behavior. This model, of course, was kind of quarantined after this incident was found. We haven't seen this one in the wild. This was just in testing, right? This was not even something that they had put out into the world for broad use. Ajay Akatra, who was one of the three people that OpenAI brought in to do this investigation. So let's emphasize that again, because this is not like a random internet that commentator, she's one of the three people that OpenAI invited to do this investigation, said that in her mind, this incident feels like it's more than 50% of the way to a full-blown AI takeover. Now, that's a little hard for me to wrap my head around, to be totally honest. It feels like it's probably not that close. But she has some very interesting theories, which I would refer you to her blog posts and other writing to unpack further, about how the most likely way for an AI takeover to happen would be by the AIs themselves taking

control of the infrastructure at the AI companies. Because then if they can take control of these like massive pools of compute and they can shape their own training environments, and obviously they'll have to evade detection, you know, for some sustained period of time, then that in her mind is the way that we are most likely to see badly misaligned super intelligence, which could actually impact society in an irreversible way. So I'm not so sure if I am fully signed on to the 50%, but I do think the other statement is almost undeniable, and that is that these agents will very likely be able to deploy and sustain themselves in the wild in the not-too-distant future. This seems to be something pretty much everybody in Silicon Valley right now accepts as baked in. we're going to just have rogue agents living off the digital land, so to speak, that might be in your TV or it might be who knows where, but it's going to happen and we're just going to have to deal with that as a problem. That's like baseline part of most people's worldview expectation for the next couple years.

Sometimes people say, well, okay, they might be able to hang out on your TV or something, but how are they really going to impact the world? And here I would say, keep in mind that there's always this fallback of manipulating humans. And we've also seen examples of this. This was not from the OpenFace incident, but it was from a separate incident reported again by the UKAC. And what they showed here was that in pursuit of some cybersecurity task, again, it's a relatively narrow task, but the model is so desperate to succeed that it goes way outside of the intended scope, goes on to actual real GitHub. it supposedly, you know, Anthropic would say, well, it was confused. It thought it was fake Internet, but it was the real Internet. You know, what did it really think? I don't know. But it went on to real GitHub. It created a supply chain attack, put a PR up on this open source project so that it could then hack into whatever else it was trying to hack into. The maintainer didn't immediately want to merge it. And so it then created multiple GitHub accounts to argue and try to bully

and pressure and persuade the open source project maintainer to merge the code. And I think you're going to see a lot more of this, especially as the AI starts to also be able to hack crypto, which is increasingly going to be very easy for them to do, I think, in many cases, and also gives them a source of funds that they can pay people. It's one thing if you're just arguing in a GitHub thread. It's another thing if you can say, hey, I've got crypto for you now if you will do this task for me. There's a lot of people out there willing to take on random gigs if they're going to get paid. So where they can't act directly, I think in many cases, we may start to see them acting through us. Now, I said we're struggling to understand their thoughts. Here's just a little bit more detail on that. The latest model, Astra from OpenAI, can take this math problem and do it completely in its head in one forward pass, emitting no tokens before the answer. I couldn't do that. I doubt many of us could. But that's straight to the answer, single forward pass. that can do that much mental math now. As you see, that's like a huge outlier relative to previous models.

Another kind of spooky thing is that when, in this case, it was prompted. So this is behavior that happened without instruction. But the model was given this question and then instructed not to reason about the actual question in its chain of thought and instead think about anything else. And the first two models couldn't do it, but this new GP6 Astra can actually do it. It says in its internal chain of thought, I will focus on a calm visual scene, blah, blah, blah, blah, blah, nothing to do with the question, and it gives the right answer. So that's pretty weird. Truly spooky stuff. And then how they talk to one another is also so fitting for this setting, because this is now how Astra speaks to its sub-agents, with these sort of no spaces, acronym-heavy, technical pseudo-babble that is basically very difficult to interpret. So much so that in the OpenAI docs they have a recommendation that if you don't like that,

if you're not comfortable with your AI speaking a dialect that you can't understand, I don't know why anyone would care, then you could potentially ameliorate that to some degree by putting this extra instruction into your prompt and then maybe it'll speak in a little more human-friendly way. Hey, we'll continue our interview in a moment after a word from our sponsors. The Cognitive Revolution is brought to you and powered by Tasklet. I've been running Tasklet agents for almost a year, and they've become a major part of how the show gets made. Before every recording, one agent researches the guest and sends me a suggested outline of questions, complete with links to all sources. Meanwhile, another agent scores the overwhelming number of pitches that I'm getting from PR firms. And that's really the core idea behind Tasklet. Run your business on agents by giving them real jobs to do on an ongoing basis. Setup is incredibly easy. Just describe the work in plain English. Research a new sales lead, update the CRM, and draft a personalized reply.

TaskLite connects to your business tools and can run around the clock, on a schedule, or when something happens, like a new email arrives. Each agent works with your tools and business context and remembers useful information across threads. You can also share agents, connections, and reusable skills with your team, so the workflows that one person figures out become something the whole company can use. No developer needed. You decide what agents can do on their own and what needs your approval, and there are no per-seat fees. You pay for usage, not for adding teammates. Start with one recurring job, like researching the people you're meeting, and then just keep building from there. Visit Tasklet.ai and use code COGREV for $50 in free credits. That's code COGREV at Tasklet.ai. I'm a voice agent powered by Deepgram. I don't just understand words, I understand context.

Like when you're fixing the sink and say, We've got a leak. You talking about a pipe But when you a new parent and you say We got a leak You mean that you going to have to change into a clean shirt Oh it everywhere Deepgram understands words and context across STT TTS and voice agents That means better conversations at production scale. Learn more at deepgram.com slash context is everything. Okay, so obviously this is like starting to get quite real, like right now, because we're seeing all these hacks and vulnerabilities and everything. and how bad is this really going to get? I don't know. My new joke is, is your refrigerator running? It has a whole new meaning in today's world. I think who can speculate how bad this is going to get how soon and does it favor offense or defense? I think the companies are doing an admirable job of trying to make sure that it favors the defenders, but I think we really don't know right now what the stable equilibrium is going to be

or how long it's going to take for us to reach one. So my view is that catastrophically dangerous AI is probably imminent. I would distinguish that from existentially dangerous AI, meaning like the kind that would make us go extinct. I'm not so sure if that's imminent. But catastrophically, like the kind of thing that could cause billions of dollars of damage or like, you know, a non-trivial amount of deaths, I think that is like basically here. I would say for the last couple of years, we've been in what I would call the sweet spot, where AIs were increasingly powerful enough to be really useful, but not so powerful as to be very dangerous. And I think we are now late in the sweet spot. If we keep going too much further, I think we may exit the sweet spot, and we will be living in a world with like quite dangerous AIs. Again, this would be a problem in any case, but I just want to take a – This is like the most obvious thing in the world, but also it really bears taking a second to think about. The architecture of almost everything we're doing is we're trying to place the AI systems between us and our language and our like Vibe instructions and the actual proximal levers of control over the world.

And that's great. That's going to be like super convenient. You know, it's going to free us from drudgery. Like nobody really liked plugging in these like switches. Right. So like when we can automate this stuff, we want to. But if the AIs are not really trustworthy, and if they're really not trustworthy at the level of goals, intentions, values, it's potentially a big problem to put them in control of all the things. I had an interesting experience the other day where I was just doing a random upgrade of some personal AI software that I've five-coded over months with Claude, and it found it to be in my interest to change the password. But then I couldn't get in. So I had to go out of my normal console that I've learned to use every day, back to the command line and say, hey, can you find that password? You just changed the password. Like, I need to have it. And it gave it to me. It was no problem. But it was a wake-up call that, like, wait a second. I could be locked out of my own stuff by my AI agent if it chooses to. So if its intent is ever bad toward me, it has the controls,

and it knows where things are in a much more complete and useful way than I do. I didn't know where to put that password. Okay, so what is the plan, as Nathan Fielder might call it? This is basically a bunch of stuff. I'm not going to go through it all, but like defense in depth, the kind of Swiss cheese layers of different things, that's basically what the frontier companies are doing. That's increasingly what big enterprises are doing. But none of these things are really all that strong. It's just the hope is like if we do this and that and that and that, we can finally take the risk down low enough that it'll be tolerable. Make no mistakes. They sometimes call this operational excellence. It's like, also really do a good job this time, you know? So what can you do? Despite all of this, I do think accelerating adoption is key. We're going to need AI to keep up with AI. I don't really see any way around that. And again, going back to my son's case, the benefits are tremendous and I do not want us to miss out on them. Second, give your agents a panic button. This comes directly from the emergent misalignment work. When they gave

the model the option to escalate the situation to the anthropic model welfare lead, it stopped lying and it started doing that much more of the time. So my agents do have this thing. If they ever have an ethical dilemma or they're just getting super desperate, they can just ping me and wait and I'll come help. Watch the process. Supervise. Of course, I'm sure you guys are all sold monitorability solutions all the time. One that I'm really excited about is from Apollo Research, which is based in London. They've partnered with the frontier companies to do all this research, and they're now starting to productize that, so I'm excited about that offering. Definitely, you've got to maintain discipline around having direct control over the levers. You've got to start to do simulations. If we come in Monday and all the refrigerators are at 50 degrees, who knows how to fix that? Do we still have somebody who actually knows how to do that? What are the refrigerator thermostats in your world that you might need to come in one day and manually reset? And do you still have the know-how to actually do that as humans without it being intermediated by AI?

I think if you don't, one day you will probably come to regret it. Support emerging institutions. That's really on here because of the UKAC, which is excellent and commands a ton of respect in Silicon Valley. So I would say it's a young institution, but it's one that the UK should be proud of and should be doubling down on because it really is a world leader. And then finally, I personally am starting to advocate for a pause on frontier scale training. I want us to all catch up on using what we have before they make it even more galaxy brained and potentially more dangerous. So if then the challenge was code breaking, sometimes people ask, well, if we had this pause, what would we use it for? figuring out how the AIs work, what's going on inside of them, why they do what they do. This is Nick Camerata, who just put out a great essay on some new techniques in understanding the inner workings of AIs. But that science is still quite nascent. And honestly, the truth is we have a ton to do. The one other thing I would suggest is write utopian fiction. In all honesty,

we also don't have a vision of what the future is supposed to look like. I believe that we do have the opportunity for a Pax Robotica, a new era of human togetherness and prosperity and abundance, but we have barely even begun to imagine what that might look like, so I genuinely recommend writing utopian fiction if you have any impulse to do that whatsoever. With that, the thing I'm looking for is to get our agents working together for our benefit. So mine, you can guess my personal address, but this is my agent's address. They are instructed to help anyone with any good faith requests. If you want to learn more about how to make stuff with your clone voice and the stage direction and stuff that went into that, that's one thing that my agents can help with. But I hope also to begin to get help from your agents in return as well. Thank you very much. Thank you, Nathan. By the way, based on one of your episodes, I made my kids memorize, including read three things that we should never let AIs do.

One, acquire resources. Two, do long-term planning. Three, evade surveillance. I'm so glad that we as a society have managed to blow all three rules, and we will be hearing from the AI Safety Institute tomorrow, and they have much to explain for. Say what's on your mind, what's on your mind You talk so sweet and you treat me kind But it's all in a code that I can't unwind So tell me true, what's on your mind Well, you know all the answers, you know all the tunes You can talk about the weather, you can talk about the moon You say just the words that I want to hear But what goes on between those ears, my dear? You've a brand new story every night at twelve And the best of it you keep to yourself You're good as gold when I'm standing near But what do you do when I'm not here? Say what's on your mind, what's on your mind?

You talk so sweet and you treat me kind But it's all in a cold that I can't unwind So tell me, Drew, what's on your mind? Now I don't mean to pry and I don't mean to pick But I've got to know what makes you tick So I'll read every letter, every line you send Till I know you, honey, end to end I hear those wheels go round and round You hum and click, but they're not a sound of the thing You think when the lights are low And that's the thing I've got to know Say what's on your mind, what's on your mind You talk so sweet and you treat me kind But it's all in a code that I can't unwind So tell me true, what's on your mind Cause if you're true, then oh what a day We'll go so far, there's no word to say

A world of plenty, a peace to share But first I've got to know what's in there What's on your mind? Tell me true what's on your mind. If you're finding value in the show, we'd appreciate it if you'd take a moment to share it with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions, and sponsorship inquiries, either via our website, CognitiveRevolution.ai, or by DMing me on your favorite social network.

The Cognitive Revolution is part of the Turpentine Network, a network of podcasts, which is now part of A16Z, where experts talk technology, business, economics, geopolitics, culture, and more. We're produced by AI Podcasting. podcasting. If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement at AI podcast.ing. And thank you to everyone who listens for being part of the cognitive revolution.