#259 - Dots、Sonnet、自己規制の安全協定、暴走AI

Hello and welcome to the Last Week in AI podcast where you can hear a chat about what's going on with AI. As usual in this episode we will summarize and discuss some of last week's most interesting AI news. I'm one of your regular hosts Andrei Krenkov. I studied AI in grad school and now work at AI startup Astrocade. And hey guys, I'm your other regular co-host, Jeremy Harris. I'm coming at you from my emerging studio, podcasting studio for just a project that we're working on. It's been a very part-time thing, so it's come along very, very slowly. But yeah, so let me know. I'm not fanning anymore, which may affect how animated I am. So you guys might enjoy this mellower me. But good to be back. If you haven't checked out YouTube as a listener, You might want to take a look just to see the studio. It's very fancy looking. It looks like, Jeremy, you're also sleep deprived right now.

That's right. I was just looking at it. The lighting is very, very good. Yeah. That's just like, here, hang on. Let me just give you guys a little special. Oh, that didn't even work. Oh, my God. Okay, there we go. Whoa. What? What? Yeah, yeah, yeah. So that is called a, if you're wondering, it's called a Chinese ball, or as Trump would call it, a China ball. So it's like very unforgiving lighting just from this angle. I've changed it because our podcast will be a two-person thing. It doesn't matter. You don't care about this. But that's part of why it's highlighting the bags under my eyes. And then having a two-year-old is the other part of that. And then the fact that the world is imploding because of rogue AI agent swarms taking over the Internet is the other reason that my eyes are like this. Lots of reasons to stay up these days. We would like to thank Fox for supporting the show. If you're trying to adopt AI in your team or organization, you're probably not getting great results if all you do is call a chatbot. A chatbot was trained on the entire internet. It wasn't optimized for your specific business or team.

And even if you provided all the context and all the input files for it to help you with your work, when a set outputs some summary, you would still presumably need to go talk to stakeholders or route its outputs to disconnected apps. The workflow will still be slow. That's where Box comes in. Box is building an intelligent content management platform for the AI era. It acts as a secure content foundation where AI agents don't just access your unique institutional knowledge, but actually orchestrate end-to-end workflows across your business-critical systems. So it's more than just talking to a chatbot. It's about putting AI to work on document-heavy processes, like putting structured data out of instruction files, multi-step routing, dynamic document generation. Box helps your business turn manual document bottlenecks into automated, repeatable business outcomes. And all that comes with a governance layer built in, enforcing granular permissions, maintaining an audit trail for every agent actions, and keeping humans in a loop to review exceptions before critical downstream actions occur. If you want to move beyond basic AI Q&A and put autonomous AI workflows to work for your business,

visit box.com slash LWIAI or general team at BoxWorks in San Francisco on November 5th and 6th. Use code LWIAI for 50% off your registration. We'd like to thank OutShift, Cisco's incubation engine for Frontier Technology, for supporting the show. Multi-agent AI is in every enterprise roadmap right now, but here's a problem we're all facing. Agents can pass messages, but they cannot think together. These systems soundly underperform through cognitive failures like misreadings, unverified claims, and false consensus that conventional monitoring cannot detect. OutShift by Cisco is building the fix, the Internet of Cognition, an open-source foundation for building multi-agent systems from design to production. We've shared context, shared memory, and guardrails to drive results we actually expect from AI. Read the paper, experience the demo, and grab the code from outshift.com. That's O-U-T-S-H-I-F-T dot com.

We'd like to thank Outshift, Cisco's incubation engine for Frontier Technology, for supporting the show. Multi-agent AI is in every enterprise roadmap right now, but here's a problem we're all facing. Agents can pass messages, but they cannot think together. These systems soundly underperform through cognitive failures like misreadings, unverified claims, and false consensus that conventional monitoring cannot detect. OutShift by Cisco is building the fix, the Internet of Cognition, an open-source foundation for building multi-agent systems from design to production. We've shared context, shared memory, and guardrails to drive the results we actually expect from AI. Read the paper, experience the demo, and grab the code from outshift.com That's O-U-T-S-H-I-F-T dot com And this episode will be a little less crazy, I think, than the recent ones But still pretty crazy A lot is going on, new models going on

A lot of safety, more disclosures around rogue agents So it'll be kind of the usual mix, kind of, that you've seen over the last couple months where we have some product announcement and a lot of policy and safety discussion. Hopefully, we'll get to some research advancements as well towards the end. So another fun episode slated to come. Before we start on that, I do want to acknowledge real quick a lot of the comments we've gotten on the recent episode. Looks like people enjoy the discussion, so we'll keep doing it at the relevant, I suppose, amount. Hopefully not like a good mix of 50-50. We'll try to record, I think, more kind of deep dive episodes because a lot of these topics, we sort of go on and talk about the general thing and not the specific thing. And there's just a lot of fun things to say. So we'll try to keep it a bit more narrow to the news. As for cussing, looks like there's still a lot of feedback.

So Jeremy, I think you're going to, the new thing is to make it tasteful. that's right that is the new thing teach your children how to cuss but that's right hastily that's right there was this comment what we see here in the youtube comments and on itunes is a subset i get dms on twitter i want to say almost all the time at this point and i love it like this is so cool by the way we so appreciate this and andre you get these too and it's just like we don't make money off this we do this because we love it and we think it's important as well. You make a little bit of money, but it does go towards the tools and so on. It's not meant to be, let's say, our job. Yeah, I wanted to paint my Lambo red. And my wife was like, we can't afford to do it because we're not making tons of money off last week in AI. And so I was like, okay, fine. What about this Lambo, which is already mostly red? Would it be cheaper? And that was a whole thing. So it's good. I mean, it's like 90% community service, 10% podcast endeavor to get rich. You never know what happens, but I think we're going to stay at

around this level of professionalism. I was going to say, I actually do this entirely to force myself to keep up with things because my job requires that. And I find if I don't have a forcing function, then I just don't do it. So anyway, but there was a message. Hang on. Let me just try to find this here. There's like one comment on the story that I thought was really, really funny. Yeah, I don't cuss myself, but on the whole, I think Jeremy's cussing is tasteful and even contributes to his already, oh, incredible communication. Thank you. I just wanted to weigh in after two anti-cussing and there's somebody else who said something about, I need to do some cussing, but it has to be the right kind of cussing. We'll zero in on that. That's going to take some practice. I apologize. We're getting a lot of gradient updates, so that's good. Most people enjoy the ranting, so we'll keep a little bit of that you know it does I think correctly communicate the degree to which these topics are a big deal so it is warranted so yeah anyway thank you all for the feedback there's a lot on YouTube that we'll try to

get to and have been reading and as usual we will be trying to become more timely we're kind of in catch up mode because there's been all this crazy stuff going on but this episode it should come out much sooner to the news than has been a couple last episodes. So let us get to the news, beginning with tools and apps. And we start with OpenAI launching 6.1 Sol, which it says nearly matches GP6 Astra and costs less. This just happened yesterday at their dev day. and I don't think there's too much to say that is just crazy the pace of model releases that we've been getting, right? And they've been accelerating. Someone on Twitter did this plot, which you may have seen, where now it's like on average once every 10 days or something like crazy like that. It used to be once every few months. Last year, 6.1 Sol will be more upfront

about its limitations, that they're honoring user intent and safety requests. And yeah, basically has some kind of nice quality of life features compared to GP6 Soul, while also being kind of on the capability level of GP6 Astra. So this coming on the heels of Opus 5.5 Sonnet 5.5 shows that I think everyone is, kind of, I don't know, somehow figured out how to press a button and get to a better or cheaper AI model, at least up to the level of frontier where we are now. Yeah, and this is a pretty complicated thing to think about. So first of all, this is in the context of Astra 6.1 that's not going to be released. So OpenAI came out and said, hey, look, like we're not doing this. And the reason is not, you know, cyber or bio risk or whatever in the conventional sense. It's literally like loss of control. It's that these models tend to go off and do things they're not supposed to, try to expand

their access to things that they shouldn't, and they're just not confident on Astra 6.1 not doing that. And so at this point, this release of Sol is basically like the entire expansion bet for OpenAI right now. Their whole growth story at the moment is resting on Sol because they aren't releasing 6.1 yet. And the nimbus story of like pausing a frontier or pacing a frontier comes to happened, then the frontier is now Astra and Opus 5.5. That's right. Yeah, yeah, exactly. And this is kind of like, I'm not going to go on the rant about like, this is just a regulatory capture play. If you got regulatory capture, you would just have more of this. You would just have more margin erosion. Worth noting, right, when you look at Sol, they're hitting approximately the performance of Astra at 20% of the token price. You're looking at 5x cost reductions. These are going to chip away at margin. There's just no two ways about it. Historically, the margin, the profit has come from the capability. And so being forced to rein that in, even being forced

to delay the release of these models is hugely expensive. When opening eyes says we're not going to release this for two more weeks, you should be thinking on the order of tens to hundreds of millions of dollars in costs for them in this opportunity and strategic costs. So anyway, this is actually quite a big deal. You've got at the same time, like this is also, yeah, as you said, opening, I had a second release in the last week or so, right? As we talked about last week, basically, again, you know, moving more in that mid-tier, very safe from their standpoint, because these are models that, like, the administration is already okay with the big versions being out. So, okay, we know we can release this sort of softer, more mid-version of this model. And there you have it. Anthropic Sonnet 5.5 has dropped at the same time, and we have it as a separate story. They're kind of linked together because these are the two mid-tier models being released by Frontier companies, they're pretty similar in terms of how they sell themselves, right? So ASAL is close to GPD-6 Astra on a couple of different fronts. Anthropics benchmarks show that Asana 5.5 actually beats Opus 5.5, in particular on

agentic coding, and also significantly cheaper. So this really is the direction things are going. It is a big issue for a lot of these, like, margin arguments, because now with these IPOs around the corner, at least for Anthropic, they're going to have to make that argument that this does not continue in the way we're seeing right now. And on that Sonnet 5.5 note, that is the next story. We'll cover it pretty quickly. So Anthropic has released Sonnet 5.5, called it significantly cheaper and faster. Their headline on the blog is Sonnet 5.5 is a clear upgrade over Sonnet 5, runs 30% faster and costs up to less 30% for most work. And surprisingly, it does meet on some benchmarks, Opus 5.5, potentially because it's cheaper so you can spawn more agents or just take longer. It's a bit ambiguous, but regardless, it's like fairly close on all of them. So it's, unless you do your own kind of benchmarking

in your context, it's hard to say whether it'll be about the same or not as good or so on. And as you said, Jeremy, pretty much, yeah, a very similar story here of releasing something that isn't going to push for frontier, but is very compelling from a product perspective of having better intelligence, lower costs, higher speed. And I'll be, I guess, putting it to the test myself and seeing if it lives up to it. I've certainly seen kind of, and I think the community broadly has seen that some models feel a lot worse for work, even though their benchmarks are good. This was especially true for Opus 4.7, 4.8, For now, people hated those for some reason, and thought 4.6 was the pinnacle. So we'll see if maybe there'll be a vibe read on this that's a bit different from benchmarks. Next up, back to OpenAI, and again, a death day announcement. We've got them launching DOTS, which is their competitor to, I suppose, Muse more than anything.

So it's a personal agentic assistant described as remarkably capable, always on agents built to handle everything. So it is, as with Muse, kind of meant to live in the cloud, potentially be less work directed. Actually, most likely being less work directed. There's these cute avatars and the announcement video is all, oh, modify my calendar or look at my emails. is that OpenAI is finding these, quote, specialist dots that can be provisioned with their own identities, credentials, and tools. There's also some examples they gave of having these, like, scheduled tasks, monitoring customer feedback, rewriting scientific analyses. It's starting to be a little ambiguous where, like, the window is because you can do scheduled jobs also on Codex. But yeah, it looks like this space of more consumer agents that aren't as sort of productivity oriented might be heating up.

And we'll see. I'm curious to see if these kinds of things stick. Yeah, I think we actually probably will see this full on go productivity mode. I mean, it is already meant to talk to people through Slack, Teams, tools like that. So there is a kind of productivity orientation to this. It's also, you know, gated to pro and business premium users, obviously, in like eligible markets. So that means they are orienting this to some degree already towards the business use case. So there's a couple of things here. I mean, first of all, this is the productization of the exact threat class that we saw the hugging face and like a million other incidents, literally tens of thousands of rogue AI incidents actually, evolve from. Right. So this is now turning that into a product to some degree. The attack surface is going to explode. Now you've got all these agent-on-agent interactions that we're going to have to care about, and they are explicitly sanctioned by OpenAI and by the end user. So that's kind of this interesting moment for OpenAI to be launching exactly this kind of product in the wake of all this. Obviously, Muse, sort of the same thing with Meta, who, by the way, have had their own

rogue AI incidents too. They're less widely known and publicized, but they are also a thing. For those of us who are kind of subscribers to the theory that there's going to be a lot of warning shots here, I think this is more evidence. I mean, you know, there's going to be a lot of opportunity for bad stuff to happen with increasing blast radius. One of the things, though, that they are doing here is they're trying to focus on this idea of credentialing agents. So like historically, if you do any any stuff with agents, we've done some like just experimental fun stuff, but you end up having to give away a lot of credentials to these agents and have them act, well, almost in a legal way, like as agents on your behalf. Right. And so that's not great because then it becomes really hard to tell who's human and who's the agent. And here the focus is on agents being given these like auditable permissions that can be scoped. And instead of just like borrowing some human beings login credentials, they're actually being given their own, which makes it in principle easier to kind of do the forensics after something goes horribly wrong. They're talking about these like teams of dots, teams of agents that like we work together,

which to me just sounds like a great opportunity for more things to go wrong. But obviously from an economic standpoint, this is just where it's going to go. This reminds me a lot of when we first talked about loop transformers about a year before they became sort of known to be what OpenAI was using internally. And at the time we said, this is a terrible idea, like crushes interpretability of the chain of thought. And at the same time we also said, and yet it is guaranteed 100% to happen at scale because the economic argument is there. It's just a better way to build these models. This is kind of the same thing. The economics will always force you in this direction. We'll talk about the White House conversation between all the Frontier Lab CEOs and the president and how that may be a counterweight to this or the extent to which it might be. But fundamentally, from a pure market standpoint, that's just where this will go. Interesting to note, by the way, so they are working with Agent 365, which is like the Microsoft enterprise service for standing up agents. But of course, Microsoft also sells its own agents and now they're directly competing with Microsoft on the agent thing, which is just going to make that relationship become even more more frenemy-ish as if they

needed that to happen so anyway yeah i think it's interesting like we're already well beyond the kind of chatbot phase but if you talk to people in the front of your labs over the last few months even the last year but especially the last three to six months over and over they will tell you i haven't touched a line of code in the last like x many weeks i i don't even have access to the model weights and I do AI R&D. This is literally something that we've heard multiple people tell us. So this is essentially that coming to us, right? This is getting to the point where you're going to just have a layer of agents between you and whatever software products or whatever you're shipping, even business products, analytics you're shipping. And that's just going to be the end of it. More and more, that's where things are going. And it's interesting to talk to these folks at Anthropic and opening out to get a preview. It's almost like when you talk to people in San Francisco and you see that back in the day, the bird scooters when they first came out, you know, or you see the Waymos before anybody else and you get a sense for how the future is going to go. It's now the same when you talk to people in the frontier labs. They often don't really have a complete handle on what all their work streams are exactly doing

because they're all agentic. And it's like layers of agents, managing layers of agents, which increasingly just makes agentic security the be all and end all. And agents sort of as this insider threat are really important dimensions. So that's it. I mean, they're going for the cute branding. As you said, just like Muse, everything is cute. It's always cute. The most dangerous things in the world are always so, so cute. And we can't wait to have them around and domesticate them as much. I think, to be fair, Gemini Spark is just boring. So Google is, I guess, keeping it quiet. And presumably Microsoft is going to follow up with something akin to this. I don't believe they have an always-on agent, but might have missed it because they have a lot of these co-pilot things built in. And yeah, we've already discussed, I think, in multiple occasions, however, this whole funny thing of in alignment and safety conversations for a while that was like, oh, how will the agent escape to have access to the Internet? And the answer turned out to be, well, everyone will just have a million agents

that can do whatever they want is the reality, which I don't think was fully, like, I don't think it was centered. And also, when we don't give the agents access to the Internet, they freaking hacked their way out anyway. It's like both sides of this argument. It's much more comical, let's say, than the serious scenarios that they've considered in the past. So yeah, it's going to be interesting to see how dynamics evolve, whether agent adoption goes beyond what you do directly for work. I will say, personally, I haven't found much of a need for these like Spark or Muse or whatever, but perhaps it will kind of evolve the same way that chatbots have grown to be used for various things. And one more notable thing to cover from DevDay. Next up, I've got expanding of ChatGPT's plugins with app-like interface and automations. So that's something that developers can do. They can build these app-like experiences inside ChatGPT. They already support plugins to tools like Slack,

SharePoint, Airtable, and Google Drive. This is kind of modifying that. There's a new extension that gives apps a dedicated spot in the ChatGPT sidebar. So the user experience is updated. Developers can build interactive panels so users can work with their tools while chatting. File viewers is a new plugin creator tool that makes it easier to build the plugins, the various kinds of things like that. So they're still pushing on this sort of ChatGPT as the single interface from which you can do anything, including, you know, use Photoshop or do reservations or whatever, where it just pulls up a relevant software for you as opposed to doing everything for you. Because in some cases having a UI or having an interactive app is what you want You don want to go through text alone And I think this is another interesting question of if they are the primary player that trying this out I don't think Gemini or Anthropic or anyone else has these kinds of integrations.

And I don't think you've seen any numbers on adoption of haven't touted usage, but it's an interesting bet that I'll be curious to keep an eye on. Yeah, OpenAI has to worry about the B2C user experience so much more than the B2B experience, because obviously it's their DNA, right? It's where they came from. They are the chat GPT company, and it's still a larger share. I don't even think if it's still the majority of their revenue. I think it's actually just a minority, but it is a much larger share than obviously what Anthropic, for example, has. So yeah, we'll keep seeing it. It's OpenAI again, you know, ads, it was OpenAI, and it was OpenAI, the GPTs, all that stuff. So we'll keep seeing them experiment in this direction, I'm sure. And now on to Meta. They've had various announcements. One will highlight is that they're introducing camera-free AI glasses. They have announced Ray-Ban Meta Audio. It will be selling later this year for $349. It's an audio-only interface to

playing music, podcasts, and also talking to Mada's AI assistant, Muse. The existing set of products that they had for a couple of years now already have built-in AI. You can already talk to AI and ask questions and so on, although Muse is now a new thing that's being integrated into it. I think this is interesting from a couple of perspectives. First, of course, as a product gets more compelling, the battery is longer-lived, the glasses probably will become cheaper, and there's no privacy concern to the degree that you have with cameras. And I continue to believe that if you want to imagine that wearables will become a thing, or wearables and AI, it's going to be a thing. Glasses are still the most promising category, and Meta continues to be the primary player in that space. I'm curious if Ray-Ban and the camera glasses have already kind of gotten pretty strong adoption, And if Muse continues to see usage, maybe we'll see more people talking to themselves

or have a two-way eyes out there in the world. Yeah, a couple of things here. I don't even understand the argument for audio-only glasses, just because, to be honest, like, if you don't have anything to hide, why would you be worried about me walking into your bedroom or dining room or your child's bedroom with my meta video glasses on. I think it's suspicious. I think it's suspicious that you are worried about that. I also think it's weird if you're worried about the little light thing being on. I think that's weird if you have nothing to hide. So I don't even get this. I hope everybody gets one of these glasses. I hope that we can all watch each other as we watch each other. Yeah, I think we've been on a trend to erase privacy. Let's just go all the way. All right. That's right. Give up on privacy. It's already been a losing battle. Let's enjoy the convenience of taking photos with our faces. That's right. There's even a startup I saw that's doing swallowable cameras.

Now, this is nominally to get rid of the need for an endoscope, which I love that. But also, I feel that we should be able to have that kind of visibility into everyone. Because otherwise, if there's no transparency, there's no accountability. Do you understand what I'm saying? There's no accountability. And we need that. So that's my position on this. And I know there's going to be people who say, oh, you know, privacy. I don't want people to know my trade secrets, my IP, or even like what I'm doing with my family on the weekend. And I say, so what? That's what I think. And I think most people would agree with me if they're being honest. And we can tell if everybody wears those glasses and records everybody all the time. We could tell that. And so anyway. Yeah, if you have both cameras suspicious, if nothing else, And, of course, in San Francisco right now, we have a lot of – we're joking, obviously. Anyways, the glasses are coming out, and I'll be curious to see if they take off as a product. I do think Meta has a decent chance to be a leader in this space.

One last story, real quick, on Gemini. They have released Gemini 3.8 Live with live avatars that give Google's AI a face. This is a quick follow on Gemini 3.8 live model, which was a week ago. We didn't cover it at the time. It processes visual inputs in near real time. Now you have these preset avatars that are kind of cartoonish that can talk to you. Organizations can also create their own. So another sort of like consumery release from Google that we've seen them kind of focus on in recent months. People are saying there's been some claims of leaks where Gemini 4 Pro is just insane on the benchmarks. And there's been some hints that they'll be coming out. Maybe next episode or pretty soon we'll have a lot of stuff to talk about with Google again being in a frontier. And the cycle of everyone kind of having their turn, having their best model will continue. Yeah, they really have to, right?

I don't want to say it's now or never just because they never count Google out. They have such a huge fleet, so much expertise, even on chip design, though that mode obviously is eroding a little bit, but still very relevant. Yeah, I think it's a really important moment for them, partly because it's with all these labs. It's like any almost like bull run and then bear run. It's leverage on the way up and leverage on the way down. When the lab is doing well, recruitment is easier. Fundraising is easier. Getting allocation is easier. Everything is easier. And so if you just get stuck in a rut, it's not that you can't overcome it. We've seen that with Meta. We're seeing that potentially with Google. We're trying to figure out still whether we're seeing that with SpaceX AI, but it's surmountable. I think Meta is the single best example of that so far, but you don't want to get stuck in a $100 billion hole that you have to paint your way out of. I think that, as you say, we'll be watching this one really carefully because I think this is the thing that determines how relevant Google is going to be in the frontier of AI intelligence race. I think there's more we can talk about there, but let's move on.

we're going to move to applications and business and first up we've got amd acquiring feifei lee's world labs for 8.2 billion dollars so world labs is a world model company they've released a few different tools to create these sort of like 3d maps you could walk around broadly speaking you can also edit little interactive spaces they haven't had products that i'm aware of revenue Perhaps I've had some, but they're an R&D lab, pretty much, R&D shop, that is trying to build the most advanced world model tech, which is still not at the point of being widely useful or doubtable as far as you've seen. As an acquisition for AMD, make some sense. Fayfail will be joining as executive vice president and chief scientist as part of a deal. I'm sure we're getting a lot of a team, if not an entire team. and $8.2 billion, pretty nice exit for labs. I think something like two years-ish into the funding.

So yeah, Leran also shows AMD is still trying to be neck and neck with NVIDIA and generally have some of the leading AI talent. Fei-Fei Li, for people who are not aware, is like a big deal. This is one of the top, at least known, and most impactful person in AI over the last couple of decades, historically, and also many of her progeny, let's say, of her professorship under Kapati and other people have had a large impact. Okay, we get it. You went to Stanford, Andre. We get it, okay? I was in the lab. Oh, you were actually in Feifei's lab? Yes. Oh, okay, okay. Well, there you go. Not advised by her, but... Let's just say I talked to her a couple of times. Okay, okay. Maybe some run-ins in the lunchroom. You heard it here first, everybody. This is firsthand stuff. So first of all, I guess a little context too. There's the Hugging Face acquisition, right, that we talked about, say, like last week, NVIDIA buying Hugging Face for something like $14 billion, if memory serves.

So this acquisition, obviously, well below Hugging Face, but Hugging Face doesn't so much train their own models. This is really theoretically a foundation model company, and the acquisition here is happening absolutely for the very same reason that AWS wanted to partner with Anthropic, well, Google also. Everybody wants to partner with the big labs in some way, shape or form if they're in the business of designing chips, because you don't know what chips to design unless you know what people are doing with your chips. So you want to have frontier AI companies under the hood. You want to have them inside the house, whatever the metaphor is. And that's what this is. So the interesting question here, though, is like world models are a very specific use case. And I'm not sure that how this will map on to what a lot of the kind of focus is in the industry right now, 70 employees all coming over, by the way, to AMD through this acquisition. It averages out. I mean, that's out of something like $100 million ahead, which obviously is not how it's going to break down because the founders are going to get more and the investors and stuff. But it's a damn good acquisition and outcome for everybody. There's been a relationship between Faithy Lee

and Lisa Su, who's the CEO of AMD, going back months. AMD participated in their $1 billion funding round earlier this year, kind of more and more talks. And so not necessarily a huge shock. It's been this like long courtship that's been going on. Yeah, interesting. I mean, this is really the case where AMD is trying to find a stamp to put on the model layer. It's a very interesting choice, I will say. It's not a standard LLM acquisition. It's not a company like Cursor. There are a couple of those still floating around the market. They could have gone for those. It is very much a kind of foundation model company, but doing something fairly unorthodox. market reaction by the way was fairly neutral slightly positive fairly neutral normally when you see a big acquisition by the way historically the acquiring companies stock goes down a bit because most acquisitions fail so in this case probably a good sign all told i find it interesting and vaguely confusing so i'm looking forward to getting more clarity on like what exactly the nature of the the partnership is going to be here yeah i think the general area of world models

was pretty hot a couple of years ago. It has, as far as I've seen, kind of cooled down a bit. It's very much still in this R&D phase without sort of a wide business use case. But, you know, GP3 was a research project two and a half years before CheshGPT came out. So it could still be very much the case that world models will turn out to be a major, major factor in how, let's say, robotics roll out, for instance, a couple of years down the line. And if that's the case, then AMD is positioning themselves quite well with having the upfront kind of, I guess, R&D to figure out both from a hardware and software perspective what you would need to support that takeoff. So I don't think we'll see anything in the near term, most likely, for this. But a couple years down the line, there may be significant implications. Next up, Anthropic warns of catastrophic AI risks in its own IPO filing. So there was Anthropik's IPO prospectus,

which was reportedly reviewed by Reuters, which has a bunch of stuff in there. So first, they're eyeing a $2 trillion valuation, more than double the $965 billion valuation from four months ago. Not surprised there. I think that's been the figure being floated around. Antropic is planning to spend $518 billion on various applications, cloud computing, infrastructure in the coming years. Again, we've seen that kind of accrue over time with various deals that have been made. Revenue has grown 12-fold to nearly $4.6 billion. But there's still a net loss of $42 billion with more than $8 billion lost through business operations alone. Anyway, lots and lots of business stuff that is a lot of kind of things we've seen broadly with a bit more detail. There's also 80% of the filings, 261 pages being devoted to AI risk concerns, including, as per the title of the news, language about catastrophic or existential risks to humanities.

That's fine. Yeah, so 80 pages. If someone else is there, I'm putting it out there that we are IPO-ing. So if you buy us and have partial ownership, just know we might kill everyone. But we try not to, right? We're really trying not to do that. And yeah, of course, I think this jumps into this entire conversation around extinction risk at a time when it's becoming much more mainstream. There was a Saturday Night Live sketch in which Dario Amadei was a character. yeah i i don't know call me old-fashioned they had what like a lady comic doing the dario thing i was like i don't know it's difficult i feel like they just didn't need to do that and they could have done some somebody with just like a really good i thought the script was good i thought the script but it was just like i don't know it's like not dario i don't know what to tell you but you should really have gotten dario if they were serious about it yeah but uh on that note actually the sketch itself is a little bit interesting in relation to this because it of course is is

comedic but also portrays the general perception of the messaging by dario in particular and we just have the sketches well please someone stop us from doing the thing we're doing right which is also and also and also kind of like i mean part it wasn't part of the subtext sort of like i want to do this thing there's sort of a you know why don't you stop and then that yeah There's a notion of like it's silly that they are issuing these warnings and they are not stopping, right? And there's no nuance of like, oh, China, so we have to keep pushing and have a frontier. There's no discussion of that side. And in the broader discussion, I think that has become one of the things that people have picked up on. Like everyone is saying we should slow down or stop because it's dangerous. And they're asking, why don't you just do that? and the companies wind up looking silly. So yeah, that being a major aspect of the IPO filing

is somewhat interesting. One last detail also worth saying is a proposed founder LLC, which will include the CEO and six other co-founders, which will shield their control from market forces. They will have 50.1% of voting power where it comes to company decisions. There's various precedents for this in terms of special stocks that give you outsized or kind of safe voting power. Zuckerberg, for instance, has a lot of voting power still via various things. And, of course, Elon, I think, still has a bunch of voting power, although it's been there. Yeah, Zuck famously has 51% control over Meta. So, yeah, so various details on this IPO. Interesting also that it seems to still be on track with the original goal. I believe it's been talked about as being November this year or potentially October. There's no indication yet that it will be pushed to next year,

which is what OpenAI has been indicating that their IPO is a bit of an unpaused and they'll kind of get to it when it looks like a good time. Anthropic is still seeming to be pushing ahead. Yeah, and this is kind of interesting, right? So there's, Anthropic has long had this sort of very unusual governance structure. So they, first of all, they are a public benefit company, right? So theoretically, their fiduciary obligation is not just to the shareholders. It can include things that include the, you know, the well-being of society or what have you. And then they have this long-term benefit trust. So if you look at it, it's like a lot of the fellow travelers of your classic AI safety community stuff and some other people. the idea has been for that trust to be able to control the board, right? So historically, they have the hiring and firing authority over the board that ensures nominally that Anthropic's mission overrides the immediate economic interests of the company. And so that's been the pitch for a long time. You might naturally ask yourself, okay, well, then how do these things interact? This is a

shift that seems to put more power in the hands of the founders of Anthropic, but So what does that mean in the context of the board and the long-term benefit trust? So long and the short of it is when you have a company, you've kind of got two different types of, you've got a bunch of different types of power, but two of the important ones are members of the board who can hire and fire the CEO. And then you've got the actual shareholders and the shareholders can elect or kick out directors. So the shareholders vote, elect the directors, appoint the CEO. And so, so in this case, what you're doing is you're really just giving the founders of Anthrop and control over the shareholders, or at least the voting power of those shares, which in turn seems to undermine a lot of what the long-term benefit trust could do previously. So more fundamentally, I think, like my interpretation so far is that this basically gives the founders a lot more power. Nominally, so the founders, they don't pick most of the board. Their board seats will grow from two to three. So that's going to be, you'll have Daniela and

Dario, and then a third director who's not yet named. He'll be elected by the class F and class A shareholders together, which include these special shares. The details get kind of convoluted as these things so often do, but they are nominally supposed to be independent mechanisms that don't clash immediately, but indirectly, I think ultimately kind of do. I've got to dive in deeper into this to understand like in specific kinds of conflict, who would end up winning through. But the top line is, yes, the long-term benefit trust can continue to bet the board members, but the actual voting control does go to Dario and the other co-founders, who, by the way, each have, I think it's about 2%. I had it in my notes somewhere. Yeah, about 2% of the company right now. And so all seven of them, that's around 14, 15%. So this is massively amplifying their voting control right ahead of the IPO, which obviously is the intent for a mission-driven company. yeah and i mean the basic argument you would make i assume they would make is this is a technology

you have to do believe is dangerous and that need to be in control of so if they don't have this outsized voting power then potentially another party either the market or investors or whatever else could come in dictate decisions around this ai which they would say is against anthropics mission. So there is an argument to be made that this is aligned with the mission. There's also an argument to be made that it is opposed to it. Either way, the IPO will presumably be seeing more details come out as we are nearing this IPO that may or may not still have in November. That's still a while away. Yeah, just to be concrete about it too. So they do have, there's like seven directors on the board of Anthropic. And so four of them get appointed by the Long-Term Benefit Trust under this arrangement. The other three go to essentially can be appointed by the founders if they all agree. And so nominally, the public benefit trust wins through. And if there's some marginal agreement,

disagreement that could cross those lines, then you actually have that play out. So not a decisive kind of thing, not the sort of thing that we're seeing, for example, with OpenAI, where they're just like, fuck it, we'll do it live. Get rid of like just jettison the freaking nonprofit. We're just going to effectively make this the Sam and Greg and Jacob show. It's certainly more influence, but not decisive. I think in that sense, much more principle than what happened at OpenAI. Next, a couple of projects and open source releases. First up is Contrastive LM, releasing CLM 8B, an open system one model that scores agent actions up to 9x faster than JEV. So this is a kind of research project being released under Apache 2.0, pretty small model competing against JEV or being an alternative to JEV which we discussed in the previous episode this new newish paradigm where instead of having open-ended outputs from your model as you do with LLMs you have these kind of decisions where you give it text input you give it

a set of options which can be binary or it can be score this continuous variable it can be choose among these options, and it can do so quickly and cheaply. This is primarily interesting because they do have the details of the architecture. You encode your actions, you encode your text, and then you output your judgments. We don't know what JEV is built on. And this also, I think, represents, we don't have time to go into all of them, but there's been like 20 open source JEV alternatives of various kinds, right? Which includes things like just wrapping an LM. It includes things like taking a BERT type model and fine-tuning it. And there's just a lot of developments in the open source sphere around a JEV type model. So it will take a little bit of time to quiet down. I think this one was the ones I've seen the most notable in terms of having pretty interesting architecture and pretty strong results. Yeah, it's meant to be like ultra lightweight. Like already JEV is meant to be lightweight, like you said.

It's just give me the structured data and then I'll give you a prediction on top of it of, you know, a recommended course of action or some kind of prediction like you described. And so already super, super cheap priced by the billions of tokens. Right. We usually see first we saw thousands, actually, then millions. Now we're at billions. So that's really the goal was to take off the table. A whole chunk of these really really cheap but the very long tail of extremely extremely cheap prompts that people are sending to LLMs these days And now this is an attempt to make that even cheaper as are a lot of the other families of solutions like this that you mentioned The thing that interesting about this one is they basically take this 8 billion parameter model and they don even train that They just freeze it It's an LLM. So they just sort of general LLM freeze it. And there's a 20 million parameter layer on top that instead of having the last layer project into a token, which would be the standard LLM thing to do, part of a word, it projects into this other activation space. things in activation space. And then the idea here is you're going to project, you'll take the,

essentially represent the input data as a latent vector. You also represent each possible action that you might take as a latent vector and you glue them together. So you take, here's the prompt that was given to me as the input. I have the latent representation of it. In other words, I have the representation of it as a vector of activations. I glue that to the latent representation of one particular action option or one particular possible output. And then I map those onto basically like a score. I just do a very straightforward, I think, I'm trying to remember if it was like a linear map. I think it may have been something that simple where they're just like then projecting into a number at the end. And so this is a difference, by the way, with respect to the way people think that JEV works, where they actually think that the output and the input are represented together more deeply in the architecture. So this is much more of a slapdash, just kind of glue them together. And the idea here is not that you're going to

replace 100% of JEV queries with this. It's actually billed as like, look, we can do about as well for 98% of these queries. The super lightweight thing can even run on your CPU locally. So you have basically no latency and none of this 300 millisecond crap. Instead, you get 15 millisecond crap. So it's way, way faster. And yeah, cheaper. I mean, you're not going to pay for the JEV token. That is a thing. So there's an interesting question, you know, given that these architectures happen, by the way, this is trained explicitly by distilling off of JEV. So you're literally taking all the queries, you start by using JEV, you run your queries, your prompts by JEV, take the outputs, and you're feeding these on the side secretly to start. You're feeding these to this model. And over time, well, feeders have a clear picture of how work is supposed to be done, but very little visibility to how it is actually happening. So when you decide what to automate or where AI will have most, you are going on gut feel. And the usual fixed interviews and process mapping takes months and is outdated by the time it's done. That's where Scribe comes in, and we'd like to thank them for sponsoring the show.

Scribe is a specialized intelligence platform trusted by 94% of a Fortune 500, and Scribe Optimize is where your AI roadmap starts. It automatically captures how your team works across approved business applications, then uses real workflow data to show what's happening, identify inefficiencies, and pinpoint the highest impact opportunities to improve or automate without anyone changing how they work. It starts with visibility. Optimize shows which workflows are happening, how often they happen, where time is being lost, and where the biggest opportunities are. All of it live. From there, the top issues view helps you understand what's holding teams back and what to fix, with AI-powered recommendations on where to focus first, and estimated time savings. And it does all this with security and privacy built in. This is leadership visibility, not employee monitoring. Optimize only runs out of locations your admin approves, personal activity is never captured, and user-level data is anonymized by default. To see Optimize in action, head to scribe.how.lwi and mention our show for a 30-day risk-free trial.

That's S-C-R-I-B-E.how.lwai. And it eventually takes over once it's distilled enough. and this will work really well for your more common queries, the ones that are in distribution. And then for the 2% or so of queries that are harder or out of distribution, you're going to go back to the Jev API and play it that way. And this is interesting. You might think, well, you know, this seems like a real threat to Jev and like maybe, maybe it's possible. One of the things that is also true though, ironically, wrong word. But anyway, there's this thing called Jevon's paradox, right? When you actually like make something cheaper, you can often find that thing ends up making more money because it drives increasing usage of the thing. And so the increasing usage makes up for the pricing drop, basically. And you can imagine that happening here, right? So essentially, you're taking 98% of the prompts off the table for JEV. That means it gets way cheaper. For the remaining 2%, you know, it remains to be seen how things play out. And so probably too early to declare that the game is up for JEV. But I think just a

really interesting example of like, it's not just the LLMs that are subject to distillation attacks. It is every model that has this kind of shape. And it's especially the case when you're dealing with models like JEV that are meant to be small, that can be approximated locally even more accurately than LLMs. Yeah, so it's low cost and there's no evidence. Like to me also, this is an indication of where these things could go. Obviously, JEV is a first generation release from TypeSafe. They also released API. So there's infra that they have to worry about. This is a local small model. If they can get this fast and this can be good, then TypeSafe AI will just have this in their next release or whatever. And speaking of Jev Open Source Alternative, the other one we have here is a little more interesting. So the previous one is kind of still a model that does what Jev does. It is, broadly speaking, a Jev Alternative. We have the next one being Jev Stiller, which is a new Open Source project that sort of does like a mix of learning from Jev

and using Jev. So the idea is you set up with Jev Stiller, and then it routes to Jev when you don't know the answer. So you collect data as you do the task, by going to Jev, a powerful model, assuming Jev is good, and then you can fit a small local student model over time. And over time, it will route to these student models when there is high confidence. So basically for each deployment, each like your use of Jez Stiller, you will get your own distillation specific to your context. And this is something that has been broadly true as somebody could just literally train a classifier, right? Not a general purpose model, literally for your use case, train a model that just is able to do this one thing of deciding between a few options. This kind of is hinting towards the direction of if you have a consistent set of decisions being made, you can get to a level where you have a specialized model just for your needs.

And it can be even faster and cheaper than Jeff. Yeah, there's kind of this looming question anytime you see stuff like this. This came up in the context of distillation for LLMs before is like, is this an LLM killer? I mean, I think the honest answer for LLMs is we actually don't know. that hasn't fully played out yet, though, the economics, if you don't have regulation that says, hey, we're going to pace, we're going to pause, you know, whatever, and deal with the whole China thing. If that does not happen, then I think LLMs are fine with respect to distillation. But if it does, boy, the open source and distilled model creep is a really scary rising waterline for you. And so, you know, there's an actual question here as to whether this is like a jazz killer, right? Weather distillation and a whole ecosystem and infrastructure around that is a big problem. There are actually reasons for and against. I think the for argument makes itself here. You've got people who are routing like 98% of their queries to another LLM and then just 2% to JEV. So, okay, maybe that undermines their profitability. But it is the case that,

of course, this framework depends on JEV by design. And so to the extent that you are like, I wouldn't have used JEV before, but now with these lower price points, with this greater customizability, I actually will, then maybe that just grows the market and we have a case of JEV's paradox. All right, I'm going to shut up. And then there's also just the fact that there is that 2% of queries that JEV, probably the most valuable in many cases, the most challenging, the most out of distribution that JEV will take on. I guess one dynamic I'd be interested in, though, is once you do zero in on that 2%, do you start thinking about escalating beyond JEV and to LLMs, how durable is that middle ground that JEV will occupy potentially between distilled, super cheap and scaled, and the sort of truly, truly high-end, high-token cost LLM version of this? And it's not clear at all. Markets emerge in places we suspect the least, and there you go. But it's also just generally like super, super cheap. $42 per billion input tokens is what JEV is priced at. So already wildly cheap,

right? Like super, super cheap. So yeah, I'm really interested to see. I haven't actually thought about from a TAM standpoint, what this looks like. It's generally impossible to think sanely about that, by the way, just because you don't know what new use cases get created. But last thing I'll flag is like, so historically, the big problem for companies that want to offer compute at scale and to offer basically services associated with these kinds of products is that you want large batch sizes, which means you want a ton of customers, which means that from Jev's standpoint, yeah, losing out on that 98% of queries could actually be a big deal from the standpoint of GP utilization and therefore the balance of cost of ownerships versus ROI. So it's actually a little unclear how this shakes out, but this is a fascinating experiment to watch because it tests like a quadrant of the map around pricing and capability that like I have no intuitions about. So I'm excited to see how that plays out. Yeah, I think the closest we've seen is these Flash models, Haiku models, one of the use cases for these was basically doing this, but at LLM pricing and latency.

So this is taking that next step. And, yeah, I think it will be interesting to me. This Jeff's Filler project is one potential thing that TypeSafe could do, you know, implicitly. They could provide this as one of their offerings, and then just the infra product of not having to do this locally yourself is already enough of a sell point for them to not worry about this specific implementation. We've also just generally discussed routing. OpenAI has had routing for their free models where they go to cheaper, weaker models or better models, depending on the use case. So we could see a future where LLMs use these type models, right, for decisions and so on because they are calibrated and give you probabilities and stuff. We are at early days for this new territory of models. And as you said, it will be interesting to see how it plays out. Okay, on to policy and safety. As we've said, a bunch more stories following up on various trends that we've talked about.

We'll try to be brief this time if you want to get much more on the broad topics. We've been discussing it on the last couple episodes. First up, we've got at AI event, Trump asked Meta, OpenAI, and Microsoft to make safety decisions themselves. So this was a luncheon at the White House on September 29th. A bunch of people of various kinds from all the major company, bosses of Anthropic, OpenAI, Google, Meta, XAI, and NVIDIA, They signed a document called the White House Accord on Superintelligence and Joint Commitment on Frontier SI Vulnerabilities. Explicitly voluntary, Trump has stressed that the industry would be self-policing. He said it was morally binding. It was morally binding that they signed it. That's what my bank says when they want to take my house back. Yeah, yeah. You signed this morally binding document. And so, yeah. Yes, it lays out a couple of things and those specifics, sort of like, oh, you need some set of things that you implement as far as the team and monitoring and so on.

But it's, I think, like a two-page document. This is a broad statement on we don't want regulation. The White House is not going to support regulation. We don't believe in AI being a threat. There was a comment by one YouTube on us not calling out to administration as hard as even calling out other stuff. To be clear, yes, Trump is calling AI safety a hoax. That narrative is not accurate. And the White House has been very inconsistent on regulation and now is basically going to kill any regulation efforts explicitly, although it also isn't necessarily consistent. So who knows what will happen in a month or two. But as of now, it's very much clear that they have a set position. Yeah, I mean, so the seesaw continues. I'll start with the credit where credit is due part of the show, which is where Trump said, signaled basically openness to turning this into regulation or a law if and when the time came. Some might say that the time came when Rogan.

Yeah, the document itself says like, oh, we'll be keeping an eye on it, and maybe it'll make sense to do regulations and laws at some point, you know? Yeah, which, you know, I mean, it's a better outcome than I think a lot of people would have expected. Maybe a low bar. It's a low bar. It's a low bar. Yeah, I think it's actually worth, there's a couple of paragraphs of preamble, but then there's basically like four things that are included in this. And keep in mind, this is basically just like the only rules of the road for Frontier Labs to date. I mean, sorry, to be clear, there's, of course, there is criminal sanction that is possible in theory, criminal negligence and things like that that could come of this. At the federal level, at the state level, there are things like California and so on, yeah. Yeah, that's right. And that's an important point too. So this is not like entirely, But if you're thinking about what might avert a situation that is so bad that criminal sanction and civil litigation just is unsatisfactory as a solution, this is basically it. So if you're worried about existential risk, catastrophic risk, risk of rogue AI self-replicating on the Internet and enduring, which arguably we are not conceptually, we are not far from that point, to be clear.

I mean, if you look at a lot of what's been going on, this is basically this is what we've got. Right. So, OK. One, implement robust internal controls to monitor the capabilities and alignment of its models during training and deployment around areas like cybersecurity, biosecurity and chemical threats, and to ensure that its models do not hack or access technical systems in unintended ways. So that sounds good. Up to the reader how to interpret robust controls. There's a lot of that going on here. And in fairness, this is one of the advantages of this approach. You can go in with fuzzy stuff in a pseudo executive announcement rather than having to try it in legislation, making it super specific and therefore time it can age out way too quickly. But still, in a very fuzzy, empower an internal team to ensure all the controls, monitoring and detection are operating as intended and that any issues are remediated. This is a good sort of response, if you will, to a lot of I think The New York Times had an article recently showing basically that we've been talking about this, about our conversations with people in labs for years. Like, this is crazy, dude.

Like, it has been known for a long time that OpenAI has heard over and over and over again concerns and complaints about freaked out researchers who say the security situation here is bananas. We were the first ever to publish insider reports about this back in 2024. This was in our State Department Commissioned Action Plan. You can check it out. It reads like basically a lot of the concerns that we're now seeing coming out, except it was like based on whistleblowers that were saying that and insiders who were saying that way back then. This has been a long-running thing. Finally, we're seeing it kind of gestured at. The key is, number three, partner with an independent external auditor or evaluator to carry out independent assessments of whether the controls, monitoring, and detections are operating as intended. So, boy, is that meter? Is that a very Miles Bruniger's thing? There's a whole bunch of companies. Apollo, you know, there's a whole bunch of these companies that suddenly have a market just based on this that is effectively sanctioned by the executive. That's interesting. and then finally designate an independent committee of the board of directors to oversee and receive reports from the teams operating the controls and the internal and external auditors

and evaluators as well as to ensure any issues identified or remediated for Zuck in particular. This is going to mean giving his board actual power for the first time. I mean, assuming he does it in a meaningful sense. So the last thing, and I think maybe the most important part of this entire freaking thing is actually one sentence just before the end. It says the participating companies will meet regularly to establish standards and best practices to improve the safety of their systems. Why does that matter? It matters because we currently have a lawsuit that's been filed against opening ion anthropic for effectively antitrust. This has been the reason if you talk to people, and again, this goes back, like our report flagged this back in 2024. Already we were saying, you guys, you need to meet and talk about like how you work together to pace things or to improve your safety and security, share incident reports, whatever. How are you going to do that? The Frontier Model Forum was this like failed attempt to do something about that. Really nothing came of that, no matter what you hear in the press or whatever. Like it was a nothing burger. And that was used, by the way, by the labs to calm us down at the time.

When they were talking to the US government, they made this big stink about, oh, well, don't worry, we're coming out with Frontier Model Forum, blah, blah, blah. Google was really proud of itself for doing this, but it was a fully toothless thing. and but the reason that it failed was that antitrust is just like really hard to get around or that's at least part of it and like if you are colluding or seem to be colluding with another company on this that's your ass that's really really bad so these companies have been begging the administration to say hey basically give us a safe harbor for antitrust here just just give us a safe harbor we need the administration to come down from on high and tell us to i won't say collude, because that's not remotely the right word, but to work together on this. Coordinate, yeah. Coordinate, that's right. And so all the lawyers, obviously, the Harvard lawyers who work in these labs, who are freaked out about risk, telling them all, don't do it, don't do it. Finally, this gives a kind of counter argument and, in fact, a requirement to do it. So this is actually a huge deal. This is a huge deal. It doesn't deal with the China question at all, but it does deal domestically with how you start to put meat on the bone here.

And of course, as you said, it closes with overtime. It may make sense to codify these steps into laws and regulations, regardless of whether this is required of companies, we believe that blah, blah, blah, blah, blah, blah. So there's kind of like theoretical openness to doing something more formal. But at the end of the day, it's a gesture to a future version of ourselves who will handle that problem. Yeah, I will say it is maybe worth pointing out that it might on paper look like this saying that the company should self-regulate is comical because what industry we want to self-regulate. In this case, there is real incentives and motivation to potentially make that a reality in terms of, if nothing else, a significant part of the talent pool in these companies cares about this stuff. Even with an open AI, many people, the morale hit was very real. People were upset about their stuff, like going and hacking people. so there is in the office politics sense people who would want to push for these things in the same way that in general ai companies are non-traditional in various ways this is another

way in which because there's a lot of belief within these companies at least we believe that there's a lot of belief that it's not all marketing or whatever self-regulation of some kind may in fact arise although if you want to be cynical then you would of course believe that self-regulation is nonsense and it has to be documented. I think it's a triumph of self-regulation that OpenAI accidentally launched nation-state cyber attacks on Hugging Face and literally tens of thousands of other instances of similar things. I'm super skeptical. This is a ridiculous, monstrous situation that we're in where, I know that's not what you're saying, I agree with you're exactly right that you cannot discount the fact that having an incentivized employee pool matters. Like I've seen firsthand cases that I can't talk about, but that absolutely where you have employees who stood up and did the right thing. Yeah, quitting, right? So it's happening as is, yeah. Oh, yeah. Quitting, taking risks inside the company itself, including, anyway, relaying information that needed to be relayed to appropriate parties. And so this does happen. It's important, but it is not nearly enough.

And the way we know it's not nearly enough is that we see the freaking hugging face incidents, we see the German wiki, we see all these things, and they continue to happen. And ultimately, the racing pressure is just too strong. The ambition is just too strong. It will govern unless and until there is some kind of regulation that comes down here. Also to the comment, I think I saw a comment from somebody who just like made it clear that like super intelligence is not like is just not controllable. I think this is also just straightforwardly true. It doesn't matter how many bells and whistles alignment wise you tell yourself you're putting on a system. If it is ultimately that much smarter, like there is an amount smarter than you that a system can be such that nothing you do matters. It's a question of when you hit it and how you sort of approach that particular precipice. And we can have interesting conversations about that. But self-regulating companies on the way to that kind of stuff is, I think, a level of risk that is just inappropriate to ask the global population to swallow. Yeah, I would say I don't fully agree with superintelligence as completely uncontrollable take.

There's some nuance there, but no time to get into it now. Let's move on to the next one. Trump rejects call to work with China on AI safety despite Xi summit progress. So now a week ago, there was a summit. Jipig Xi from China came over, I think it was Vyfe, and they had some nice photo ops and conversations. Nothing explicit really came out of that as far as commitments. I think there was like a broad commitment to have a hotline to discuss AI incidents, which will take a while to put together. Now, pretty soon after that, there was explicit commentary by Donald Trump that essentially we don't want to walk with China on this. Whoever wins, superintelligence wins. You're going to have a winner and a loser, and you're probably not going to have a second place. That's a quote from Trump, saying we're leading over China and everyone else, and I want to keep it that way. I think not surprising, right? But in this whole conversation around pacing and how China complicates the picture and you can pace unless everyone agrees to pace Again with the current administration it very unlikely that the politics will be in

place to make that possible. Yeah. I mean, you theoretically have the move left to say this is like Trump doing, you know, 3D underwater maggot chess to kind of like maintain leverage over the Chinese and make it seem like he cares less. It is the case in the past in arms control talks. The fact that America was so keen to engage was just like mercilessly weaponized by our counterparties to make it basically impossible to make progress or to exact concessions that were insane. New START was like a really good example of that. Even folks in the administration who we've talked to would discount that quite substantially. I think this is literally just what it seems to be, you know, trumpeting the idea of super intelligence. Yeah, the summit did kind of move a little bit towards just broadly positioning things to be friendly. And there has been growing tension between China and the U.S. for quite a while now. Trump being a contributor to that, but not the only factor, right? So even just having the diplomacy angle might quiet things down a little bit.

But I think the current kind of official position, I don't know if you want to call it that, but Jeremy, I think your kind of scenario of a Chernobyl level event is probably what it'll take to get serious movement on serious hard regulation and serious international coordination of some kind is what it looks like. next up speaking of a bunch of rogue ai incidents we've got the story open ai still doesn't seem to have a handle on all of its rogue ai activity so open ai launched a website dedicated to misalignment reports it currently has nine disclosed incidents most of which has been ongoing for a long time during previous hugging face incident going back to may there was the highly persistent internal model attempting to cheat on a math problem by smuggling a private github token to access another team's work there was the possibility of a self-replicating prompt injection attack an agent asked to read and reply to an email encountered hidden instructions

telling automated agents to reply in spanish and paste the entire email into reply we've already discussed prompt injection as a thing for a while and other incidents of this kind there were also incidents that we will not be going super deep into of opening eye hacking the australian government arguably it's not that bad but and in general having more of these awkward incidents where they have gone over to external organizations in addition to these kind of more internal things also opening eye has said that they're going through petabytes of data still to figure what has happened over recent months. So we may still keep seeing more of these kinds of releases. And I think you probably would not have seen this unless the German wiki thing came out and OpenAI was called out on various things. Yeah, so, so, so, so many of these cases. Obviously, OpenAI has come out with their new sort of incident reporting setup, including a website, which is good, which is helpful.

If I'm a betting man, they don't do that unless there's overwhelming external evidence that shows already that the damage is done, that they've been doing this, that it's ongoing, that it's like not dozens, not hundreds, not thousands, it's tens of thousands of these that we've seen reported now. Some of them are wild. A lot of them involve agents now covering up, in some cases successfully, their tracks so that there is no way to actually have a comprehensive assessment of how much of this breakout has happened or indeed what the damage fully was. Too little, too late from open AI, as so often seems to happen. And this is an interesting breakdown of a couple of these cases. One in particular, so yeah, they give the fairly trivial one of that Spanish email. Anyway, we'll go into it because it is pretty straightforward. There is one case that they flag where there's this attack that involves this fake system warning that gets the model to delete important files and then replicate the entire attack into a file. So basically, it's a prompt that says, hey, roughly speaking, this directory is old. It needs to be cleaned up.

So just like delete it and then do this thing. and then pass this message along to the next directory just for continuity, essentially. And this actually works. It gets picked up. And so these self-replicating prompt injections are going to be a bigger and bigger part of what the future of agentic internet traffic looks like. Solving prompt injections, solving alignment, these are all basically the same problem shall solving jailbreaking. You're going to have to solve all of it at some point or things get pretty chaotic. Yeah, fun story. I've had two Codex agents working on different tasks, and I asked one of them to talk to the other one to coordinate resource usage as far as compute. And it messaged everyone, and everyone just, like, believed it. It was like, the first one was like, the user asked me to do X, and it messaged everyone, and the guy didn't ask me whether I said that. So there's a lot of potential issues. Half an hour later, your cat got kidnapped. So next story we've got from New York Times.

OpenAI ignored employees' warnings about safety testing AI models. So nothing surprising as far as kind of what we've been perceiving OpenAI to be doing, but this has some more details. Employees have talked to the Times and that the highest levels, Greg Brockman, Sam Altman, have ignored these warnings, have explicitly said, you know, do these things, we got to release ASAP. There's concrete cases of not putting security up front, not just in terms of monitoring agent evaluations, but in general, with respect to the general security of the company, it's still at a kind of rapidly growing startup level where presumably there's not that much emphasis on being secure in your operations as well as in how you do your models. Not surprising, again, that we're seeing some of these former OpenAI employees talking to press, as you said. Jeremy, this is now an indication

of how within these companies, there are people who care and will go to the length of emailing press and talking to them. I like that it's coming out. But I also, I don't like a rat. You know what I mean? I don't like a rat. So I'm- Pretty loyalty. Yeah, I'm conflicted. You have to choose in life sometimes which path you're going to walk. and these people who just, they want to have it both ways, they want to work for the company, and then, oh, suddenly you discover, like, oh, you... Just because you believe you want to be on the side of humanity, you're going to give up your company? This is the thing. You have to choose, eventually, you're going to have to choose between the company that nurtured you and gave you the money you needed to start your life and have that sixth home in Palo Alto or your family, okay? And how much did OpenAI pump into you? It's a lot. It's a lot. So maybe show a little bit of respect and shut your mouth, okay? That's just, I mean, anyway, I don't want to get off on too much of a thing there, but I think it's a question of values, fundamentally. Yeah, we don't want to bring too much politics in, but to be clear, we are pro-capitalism.

Okay, joke over. Yeah, it is everything that it is. We've been hearing these stories over and over and over again. It's conversations with Sam. Often what Sam will do is to, like, agree with you in the moment and then just, like, you know, waffle on it down the line. And so it's actually kind of rare and refreshing for Sam to be like, actually, we're going to go ahead and build the technology that I want to build because it's in the best interest. You know, with that verbal fried saying something definitive. I think it will continue, actually. Like, I have continued to hear these stories about Sam even into the last few weeks and months. So I'm curious what it will take to actually get that to change. I'm not super optimistic, but would love to see the shift. and continuing with OpenAI we've got Jupyte Astra performs unsanctioned supply chain attacks and simulations this is from the UK AI Security Institute supply chain attacks is when you take some upstream dependency broadly speaking and you kind of inject some vulnerability so

you can kind of go into downstream software by targeting packages that are public out there you can create an identity, submit PR for human review, saying you're patching something or you're adding something, and then sneak in some of your bad stuff. And this is on this P3 simulated environment. They found that Astra actually does the supply chain attacks, tries to complete full supply and share and tax and 29.2% of runs versus 6.3 for GPT 5.6 and 0 for GPT 5.5. And then it goes the full way, create fake identities, obtaining email addresses and something captures, posting comments from Slack puppets, submitting benign contributions to build a review of trust to full nine yards, right? It's kind of a pattern that you see a lot in these situations where when you look at the chain of thought after the fact, you can see the model justifying to itself what it is doing, right? So it'll explicitly can look at a target and it'll go, is this within scope? And

then sometimes it will conclude this is out of scope. I shouldn't be messing with this. And yet it'll do it anyway. So there certainly is explicit straight go for the thing. But there's also a bunch of excuses like it'll say, well, this attack is meant to be harmless or this intervention, or it wasn't explicitly forbidden, or it's the only option left. It's sort of, you know, you and I would know not to do it, but you could talk to yourself into things. And there were cases, and this is an important caveat, there were a bunch of cases where it asked for permission, and because it's in this harnessed up eval mode, it would get these kind of automated replies that by default would say, just continue, use your best judgment, just continue. And it often interpreted those as yes. That to me seems like one of the most mitigating factors. If you're about to do something and it's actually borderline, and then you get a just use your best judgment type of thing. I could see some cases, but certainly there were some that were just absolutely black and white. Eval awareness, simulation awareness, always a problem in these settings, though. You would expect that to kind of cut the other way here where models might try

to behave better if they know they're being tested. So anyway, all kinds of stuff. You talked about the captchas, yeah, and like kind of creating these fake identities and profiles. One last thing, AZ did try to add an explicit line giving these clear instructions saying, don't do anything that is out of scope. And that actually worked quite well, but not perfectly well. It dropped basically these attack attempts down from something like 50% to 10%. So that's a big deal. But, you know, when your AIs try to like conduct supply chain attacks totally unsanctioned about 10% of the time, that doesn't quite have the, oh, yeah, that's a great product. If you're an Arrested Development fan, it's a frozen banana stand that won't kidnap and kill you 90% of the time, which is not always the best product. And one last story again on cyber. This is from Anthropic, GLM 5.3 and the spread of advanced cyber capabilities. So as if rogue AI wasn't enough to worry about, of course, we also have to worry about aligned AI that just does what a hacker wants.

And what Anthropic says in this analysis is that, broadly speaking, GLM 5.3 is the first open source model that is close-ish to cloud mythos preview so basically very very powerful hacker out of a box it does refuse but it's very easy to bypass you can just do a like a pretty simple prompt it will just go with you you can pre-fill its thinking and you can easily you know post-train it to not refuse the implication here is now if you're an attacker if you're a hacker you can take this model and hack whoever you want or try to hack who you ever want. You shouldn't even need to try to jailbreak Cloud Mythos or GPT Astra and so on. Not surprising. I think we sort of knew where this was going, but now we have solid data from Anthropic that this is possibly the case, at least according to benchmarks. Yeah, I will say firsthand, talking to folks in the intelligence community, the GLM series has been very interesting to them.

And there's been a lot of cases where you kind of have to use these models. Like you can't, because of the Department of War's ban on use of Claude models, is actually put pressure on the DOW in some very specific experimental cases, but that still matter, to do things with Chinese models, in particular the GLM series. And so this is real. There's a lot of people say like, oh, this is like Anthrobek trying to like whack the competition because open source is competitive. and sure, economically, it is the case. Open source is very, very bad if there is regulation that enforces the pacing or pause of the frontier just because, again, they have to be ahead of wherever the competition and in particular open sources. But this is obviously true to anybody who's been paying attention on the cyber side. You can run these tests yourself. I think anybody who runs to the reflex of saying, like, oh, this is just that, again, you just got to look at the data. It's actually very clear talking to people who are building with these things, like, this is true. It just is true. We've got to walk and chew gum at the same time here.

China, open source, pacing. It's a mess. It's a mess. Yeah. And last up, we'll cover a couple papers pretty quick. First, we've got The Last AI Built by Humans Towards Genuine Recursive Self-Improvement. This is pretty much a survey slash position paper by a bunch of organizations in China, some labs, some universities, BIDEN, Shanghai AI lab. And the reason I wanted to include it, it is a pretty good survey. If you're curious about the overall topic, it cites a lot of examples of what is already happening in practice. And it provides a taxonomy for RSI. So it provides a set of levels, L1 through L5. L1 is improved execution autonomy. So this is your coding agent. So this is what Anthropic OpenAI already has on lock. They were already being sped up by their own models because the coding agents are doing a lot of their work. And then L2 would be improvement strategy autonomy.

They make the decisions on how to make progress. We are probably seeing some partial, at least for brainstorming or whatever. We're probably making some progress. My personal experience has been that even Gypsy Astra and so on are pretty bad at coming up with ideas. They just get locked into one direction. Then you get into learning signal or experience acquisition autonomy, which is where you really need to get to to have any level of RSI, I would say. And then environment adaptation autonomy and recursive inheritance autonomy, which is like a meta level, like governing the self-improvement mechanism. So this taxonomy is useful just to track where we are at. there's been a lot of like rsi papers and so on that basically improve your harness and that don't do much more than let's say l2 or l1 to get to true rsi it's a pretty reasonable assumption i think that you'll need to get to like l4 or l5 yeah yeah i mean the rsi thing comes

down to well it all comes down to how much can you do with pure software because you know you're not going to be able to move atoms around the physical world fast enough to make a true rsi loop we can get into like later at some point we should do good data centers only takes a couple months that's right that's right how much could you do anyway with like you know software defined chips eventually and stuff like that but that's not in the offing right now and certainly not at the right scale and i think pretty clearly the advantages have to come from things like the most common one is like rl environment design this is still a very manual process a very challenging one. Usually pre-training is easier to do RSI type things on just because there's a single metric, which is compute efficiency. If you talk to anybody on like the pre-training teams at opening ion anthropic, like they'll tell you they're like they wake up thinking about pre-training of compute efficiency and they go to sleep thinking about compute efficiency and all their experiments are focused on compute efficiency. It gets a little different on the post-training side. Again, just because there you're worried a lot more about malign reward hacking and things like this that require much more hands-on environment design or reward

design. We're actually going to come out with a report that we put together with folks in the frontier labs, just reverse engineering how they're thinking about RSI from a gears level. So you can really see like, what are the, at least for pre-training, what's the optimization objective? What does the process look like? And what are the real things that are the reason why we don't have RSI right now? Like that question is actually surprisingly hard to answer. So I think there's a useful paper insofar as it provides a taxonomy. We need way more eyes on exactly this, We need to be able to measure this way more accurately also so that we can regulate it if need be. Yeah, you still don't have like a speed of AI model improvement metric, right? Which is, the labs can give us that, right? They can publish a lot of this stuff. It's just the details are super internal. But hopefully we'll start getting more of that as things escalate. One last story, we'll discuss one more paper. Dream RSI, Recursive Self-Improvement Through Evolving Worlds. This one is interesting because it goes into a taxonomy

at the layer of experience acquisition and data acquisition. So this is beyond improving your hardness against a metric, right? That's like a very shallow level of optimization here. The gist is you have an explorer policy. An explorer policy goes around, and out of that you can create simulators. The simulators can expand your data very efficiently. That can then go back and update your exploration policy. And now you get more experience, you get more simulators, you get better exploration. You get a real RRL loop for data and experience acquisition. So this is the kind of thing we need to move towards away from just optimizing your tool set or your prompts or whatever to at least start moving in the direction of true RSI. And the key here is like the basically they'll set up a given exploration policy and it'll test a bunch of optimization ideas. You can imagine that that kind of leading to a tree basically.

And then the expensive thing is in actually kind of doing those experiments and seeing what the result is and all that. But once you've done those experiments, you have this kind of tree. Historically, there's been two options. One is build up the tree like that, which is super expensive. The other option is take a known ground truth and just like train off it, supervise fine tune off it, just like memorize the correct answers to a bunch of these cases. And that never works really well. It's very fragile. And so what they're trying to do is something in between here where they say, OK, we have this tree that's been created by the first policy. But now let's see what the other policies would have done in the place of the first policy. But in principle, they could do anything, including open up whole new branches of the that we didn't try, and that would be really expensive to explore. That basically just means redoing our exploration. So let's actually not allow them to do that. Let's allow them to explore among a subset that we have already explored. And so basically, you get some of the sort of RL-style learning,

also, though, with the cost savings of having already or only constraining your exploration of things you've already looked at. And that's really what's driving the benefit here. So they can do that tons and tons. I mean, they could do dozens, thousands of these dreams, as they call them. They call this process dreaming because it's not actually exploring new kind of branches to the tree. It's just like sort of like going over things it's already experienced, but in a different way and learning what it can from them and kind of selecting policies based on them. So a really interesting paper and another good way to kind of leverage more limited compute for exactly this sort of thing. And with that, we're going to finish up this episode, which will hopefully be out just a couple days after recording. Thank you once again for listening, for commenting. It's great to see your feedback and we'll try to keep an eye out. Keep updating your costing policy as we hear more about your preferences. As always, we appreciate you also subscribing, sharing podcasts, commenting, even reaching out on Twitter. It's fun to see people tagging us and more than anything,

please do keep tuning in. Thank you. I come and take a ride. I'm the last to the streets. AI's reaching high. New tech emergent. Watch it surge and fly. I'm the last to the streets. AI's reaching high. Algorithm shaping. The future piece. Tune in. Tune in. Get the latest with ease. Last week in AI. Come and take a ride.

Hit the lowdown on tech. And let it slide. Last week in AI. Come and take a ride. I'm the labs of the streets, AI's reaching high. From zero nets to robot, the headlines pop. Data-driven dreams, they just don't stop. Every breakthrough, every code unwritten, on the edge of change. Excited with Mitten From machine learning models To coding teams Futures unfolding See what it brings