#260 Muse、ミスアライメント、数学：Beam・Le Chonk・OpenAIの数学成果

Hello and welcome to the Last Week in AI podcast where you can hear a chat about what's going on with AI. As usual in this episode, we will summarize and discuss some of last week's most interesting AI news. I'm one of your regular hosts, Andrey Kerenkov. I currently work at the AI Startup Astrocade and was at grad school for AI before that. And hey, everybody, I'm your other co-host, Jeremy Harris from Cloudstone AI. I do AI and national security stuff, especially, you know, AI China Endgame. I actually have a newsletter now, so AIendgame.com. You can check that out if you want. We're probably, by the time this podcast drops, either we will have just released our latest post, sort of reverse engineering with a couple people in the Frontier Labs, what recursive self-improvement looks like. everybody talking about this obviously today we just wanted to make sure that someone somewhere had kind of i guess described at the gears level what the process will look like inside the frontier labs

from the perspective of the researchers themselves who are doing it if only for regulatory reasons you need to be able to define something and understand it to regulate it so this is kind of an initial attempt to do that and just generally to trace out some of the espionage and national security implications of that gears level picture so if you want that check it out we already have a post up about US-China and kind of how we see the endgame play out there. More of a diplomatic post, some of the things we've touched on on the podcast, a bit of a deeper dive, and we'll probably do one at some point fairly soon, I think, on that same topic. Yeah, we have a lot to dive deep on these days. Used to be, I'm nostalgic now for 2025, 2024. You know, it seemed like things were crazy back then, but things are really starting to get crazy, and they're only going to get crazier by all appearances. So, and we don't, we don't have a story about this per se, but Jacob Coxon was on the Daily Show recently. I got a text from a buddy of mine passively following this face, probably following this face more than the average person. And he just said, he wrote like WTF, like, look at this, which I thought was so interesting

because I thought that the Daily Show was so late on this one. I mean, you know, the Coxon tweet, how long has it been? It's been like three weeks or something since the Coxon tweet. obviously there's been the hugging face thing that's been months and months ago. We've got a whole bunch of things. So I guess it's a way in which a lot of us are following. If you're listening to this podcast, you're probably not aware of how little the average person is following this stuff. Again, this buddy of mine is a really bright dude. He's in startups and biotech and following this stuff. And yet he still hadn't registered the Jacob Coxon story. So I just find it's fascinating. The future is here. It's not evenly distributed and often for funny reasons. And the information of the future being here in some concerning ways and in some, I guess, exciting ways is also not equally distributed. And there's a lot of skeptics even who are sort of like doing counter narratives and anyway. But to focus on what we have today, it's actually going to be a bit of a quieter episode. We've had quite a stretch of like really intense ones. And today there's not like a ton of exciting news.

We'll have a bit of a mix of a couple tools stories, a couple business. Really, the coolest news is about open source models, rather than usually. For once, we'll get around to talking a bit more about research as well. So it should be a pleasantly unexciting episode in some sense. We would like to thank Vox for supporting the show. If you're trying to adopt AI in your team or organization, you're probably not getting great results if all you do is call a chatbot. A chatbot was trained on the entire internet. It wasn't optimized for your specific business or team. And even if you provided all the context and input files for it to help you with your work, when it said outputs some summary, you would still presumably need to go talk to stakeholders or route its outputs to disconnected apps. The workflow will still be slow. That's where Box comes in. Fox is building an intelligent content management platform for the AI era. It acts as a secure content foundation where AI agents don't just access your unique institutional knowledge.

They actually orchestrate end-to-end workflows across your business-critical systems. So it's more than just talking to a chatbot. It's about putting AI to work on document-heavy processes, like putting structured data out of structured files, multi-step routing, dynamic document generation. Fox helps your business turn manual document bottlenecks into automated, repeatable business outcomes. And all that comes with a governance layer built in, enforcing granular permissions, maintaining an audit trail for every agent actions, and keeping humans in the loop to review exceptions before critical downstream actions occur. If you want to move beyond basic AI Q&A and put autonomous AI workflows to work for your business, visit box.com slash LWIAI or join a team at BoxWorks in San Francisco on November 5th and 6th. Use code LWIAI for 50% off your registration. We'd like to thank OutShift, Cisco's incubation engine for Frontier Technology, for supporting the show. Multi-agent AI is in every enterprise roadmap right now, but here's a problem we're all facing.

Agents can pass messages, but they cannot think together. These systems soundly underperform through cognitive failures, like misreadings and verified claims and false consensus, that conventional monitoring cannot detect. OutShift by Cisco is building the fix, the Internet of Cognition, an open-source foundation for building multi-agent systems from design to production. We've shared context, shared memory, and guardrails to drive results we actually expect from AI. Read the paper, experience the demo, and grab the code from outshift.com. That's O-U-T-S-H-I-F-T dot com. We'd like to thank OutShift, Cisco's incubation engine for Frontier Technology, for supporting the show. Multi-agent AI is in every enterprise roadmap right now, but here's a problem we're all facing. Agents can pass messages, but they cannot think together. These systems soundly underperform through cognitive failures, like misreadings and verified claims and false consensus,

that conventional monitoring cannot detect. OutShift by Cisco is building the fix, the Internet of Cognition. an open source foundation for building multi-agent systems from design to production. We've shared context, shared memory, and guardrails to drive results we actually expect from AI. Read the paper, experience the demo, and grab the code from outshift.com. That's O-U-T-S-H-I-F-T dot com. So cooking it off with tools and apps, we've got OpenAI is sticking more ads in ChatGPT. They're rolling out a new visual ad format in Chagipity that displays images of sponsored products and services on screen, with U.S. testing beginning later this month. So they'll initially appear when users generate images with Chagipity and will remain separate from the generate image itself, which would be very funny if you just inserted products in your generated images. This is following up on ads starting rolling out in February.

So I've had ads to various extent now for most of us here. Still, I haven't seen much discussion or many sort of screenshots. I don't use strategy between myself that much, so I don't have any personal experience of it. But it seems like we're here to stay. Which is not surprising, again, given OpenAI's business model. I think the big question is when do we get to the point where ads are mostly targeted towards other agents? Because right now, OpenAI obviously has got a bunch of humans on their platform. they are the B2C chatbot agent company. Well, I would have said that, but then of course we have Muse dropping, which I think actually might be an interesting challenge for OpenAI from the agentic side, which is what really matters. And anyway, we'll talk about that, but Meta's certainly targeting the sort of payment layer of the agentic AI stack and trying to make sure they can monetize transactions made by their agents. This is, in some ways, if you're super, super AI scale-pilled and you believe what a lot of the researchers at OpenAI believe, this has to be viewed as a temporary strategy.

Like humans are not going to be the dominant drivers of economic productivity and therefore the dominant consumers of stuff into the, certainly in the next decade, but even into the next few years, according to a lot of the people in the labs. And so it's kind of, I'm interested in that, like this is a, it's a obvious monetization play. It's necessary for them right now, but I'm really interested. Like I would love to talk to somebody on their, their strategy side to understand how does this fit in? And like, how do you see this adiabatically tailing off as you kick off the era of agents doing most of the work in the economy? How does this fit into that picture? So it's interesting. I think to be fair, right, ads are for consumers typically. And I mean, it's a mix. So to some extent, if you're shopping around for business tools and so on, there is that. But if you're shopping around for like plushy toys as gifts, you might still be buying that as a human. maybe in the near future this demo this mythical demo of booking a flight via ai for you will actually happen where the level of laziness of humans will start to increase and dependency on

ai will increase such that at some point you just like book me a flight on this and this and it'll figure it all out yeah quite well i mean i guess i just i think of this as like if agents are the dominant driver of economic productivity or automation generally pretty soon a project like this starts to look like a legacy product. And I wonder if eventually just like won't be worth advertising quite so much. I'm generally really curious because it seems like exactly the kind of thing that you would do if you were less scale pilled, acknowledging that there's a need in the near term. Like I'm not saying that OpenAI isn't because they need to make money today. It's just interesting to note how at odds this is with like, you know, you've got bigger fish to chase after. If you really believe in the agentic future, you would be thinking about how do we do ads for agents. And this is kind of where my thinking goes, is this maybe the thing out of which will grow the agentic advertising model? Agents should have so much more attention to buy, and they need to use so many more tools that I would imagine that'll be really important. I'm curious if there's some kind of like jumping off point strategically where you can actually

kind of bridge that gap a little bit. I'm again, trying to think about how does this fit into the strategic vision? It's possible it just doesn't and it's pure money. Like I'm very open to that possibility but ads just seem like an unsexy thing that might actually be more important than it seems directed slightly differently anyway so i'm just calling that out i wouldn't be surprised it's an interesting question yeah i found myself more and more recently being like i just have no idea what anything would look like in a year or two years right so these kinds of things are entirely plausible it might be completely different like i just don't know and on a related story Next one is TikTok rolls out an AI shopping assistant and one-click checkout. So they've announced this as something launching, as well as this feature that lets users buy directly from brands. The shopping assistant is described as a conversational AI agent that helps users discover and purchase products. It can understand context and remember users' preferences and needs

throughout a conversation, and it provides real-time guidance, such as product details, shipping information, sizing, availability, and help completing a purchase, which to my knowledge, Shagabee doesn't have. And that seems like a natural fit with the ad strategy because you can get a commission or whatever. So yeah, quite a related note there of as you get more assistance in shopping and some of it goes through AI or a lot of it goes through AI assistance in particular, that's a big slice of revenue that these kinds of companies will probably go after. Yeah, yeah. You know, it's sort of similar thoughts as before, and I won't go on the rant again, the same rant, but we're kind of slowly going to ease into a future where ads are more about agents than people. And this is somewhere kind of in between, right? You're sort of steering, guiding the behavior of the user. I think it was in Chaos Monkeys. So Antonio Garcia Martinez, as the B or C tier famous in general,

and in Silicon Valley, A tier famous, author of Chaos Monkey. It's a really good book. You should check it out. It's kind of dated now, but it's fascinating. One of the things he talks about, so he was instrumental in setting up Meta's advertising platform. His philosophy with respect to ads, which is not unique to him at all, but he articulates it well, is like ads at their best are a tool for the consumer. You're being surfaced, ideally, things that you would have wanted to know about. And you can kind of see this if you're an old curmudgeon like me and the only kind of social media that you use for stuff other than work is, believe it or not, I don't actually use it, but just like I use Facebook Messenger. All right. That's that's how old I am. And when you're on on Facebook, you see the feed like a lot of ads there on other platforms have just turned into content like they are literally just entertainment now. And you sort of see this universally in a spectrum of this happening. ads want to be ultimately like it's of no use to charmin if they're advertising toilet paper at you and you have no intention to buy toilet paper ever that's no no use to anybody in the ideal case

they are surfacing the exact thing that you want it's something that you would rather see than not to some degree anyway so so like this is kind of along that continuum it's framed as like an assist to your purchases just to make things faster there's a continuum between assists and advertisement and commissions and incentives obviously probably should be part of the definition there. But I just think it's really interesting that this is all gonna, we're gonna have these same old debates at the level of agents. What is sloth? What is advertising sloth? What is low quality advertisement to agents that just choose up their tokens for no reason and literally costs you as the end user tokens that get burned on looking at context that's useless? Anyway, I'll park it there. But I think there's like an entire agent economy that's gonna already, that's probably starting up now. I need to check out the latest YC demo day to see what people are building in that space. Well, I'll add a quick note, which is, interestingly, I think the comparison there is GEO. So for the longest time, you've had SEO, right?

And if you're a business, you need to make sure that you show up and search results on Google. And then you also need to pay for ads to compete with your competitors. But now there's this notion of geo generative engine optimization so that if you ask Chagipity about what is the best maker of TVs or whatever, I mean, how does the agent know? I don't know. Right. So in some sense, we are already optimizing for agents in the realm of search more and more. It's becoming more important and we can expect that to happen in other domains. I think, to your point. Moving on to applications and business, we begin with Meta's Muse tops 5 million downloads, Fasterman, CheshPity, and Claude. So we covered this a little bit last episode. This is a bit of a follow-up where Muse appears to still be doing quite well. It's currently the number one app on Apple's App Store and Google Play. It's not an all-time high, but it's still

taking off quite fast. Meta is heavily promoting it. So Vue took 50% of Meta's daily in-house ad impressions over the last couple of weeks. That's wild. But it's clearly like they invested heavily into making this a big launch and a big deal. And it appears to be paying off. And it's coming at a time where they launched it and OpenAI launched Dots right after. Google has had their like Gemini Spark kind of in the background and nobody really I think cares or knows about it maybe they certainly didn't try to make a splash with it is what I'm saying right they didn't make like a little character and a whole app around it yeah no no no cute character I just some of this stuff really gets me riled up when I see Google launch stuff without a cute character behind it it makes me angry in a way that just like what are they doing yeah yeah yeah it's shameful in any case so this is clearly an expanding priority of the business i think there's

a from a business perspective there's various interesting questions of you know how does muse play into the revenue streams are we going to see like a chat gpt ask ad platform presumably on muse but muse is always also always online right this is an assistant so if you're on the free here that you're going to be like sending you messages about hey i think maybe you're running out of eggs and you should maybe buy these eggs yeah on a related note to that discussion with these agents kicking off and assuming we stick around we're sort of entering this next paradigm where chatbots are mainstream always online ai assistance or ai personas or whatever you want to call it is a new thing and seems like the logical next step personally i haven't found a need for this and i don't know people who have really adopted it intensely but with the intense

interest in open claw earlier this year and various kind of applications of it now we're seeing the consumer version of open claw in some sense and it'll be very interesting to see if it sticks around in the same way that GGBT has. Yeah, and I think that's one of the key questions here, right? Because we are seeing these download numbers that are wild, but you can get high downloads from just curiosity, right? And not ongoing use. So we'll have to wait to see what the actual DAUs and MAU numbers look like when we actually get that data, which I mean, I assume at some point that it will share some numbers if they look good. It's an open question as to where this goes. In some ways, it feels like we're in the GPT-3 era of a lot of the agentic stuff, right? And in the same way that we were in the, well, I guess OpenAI's first coding model was called Codex, so I can't just call it Codex because it's also the front. But anyway, we were in the DV3 era of coding models for a while, and then suddenly Cloud Code cracked the code. And, you know, this has been known to happen, right? So ChatGPT, Cloud Code, there will be a moment like this for agentic systems. It's basically inescapable because, as you said, when you think about always

on, the number of applications that that supports is just insane, right? And you couple that to the insane time horizons. If we had a meter plot to point to anymore, which we don't because the models are so good at long-term tasks that we're literally off the edge of the map there, you would see the value there. But clearly there's enough value to be tapped here at massive scale. Someone needs to crack the form factor, all the boring design choices that need to be made to make these products appealing, not just to developers, but also to everyday people. And I think that'll be, it'll be pretty violent once it happens. I mean, in the same way that chat GPT was violent, in the same way that Claude Code was violent, you have all the naysayers who will say that this is overhiked, and then suddenly those voices will be silent, and then we'll move on to the next thing. I think we're just waiting for that moment. I don't know if I would choose that toward the violent, but certainly... Yeah, violent growth. Yeah, yeah, yeah, yeah. Very rapid growth. Yeah, that's right. No windows will be smashed. Very rapid transformation of a whole bunch of our lives,

or many people's lives. Metaphorically violent, yes. Right, right. And on a related note, actually, we've also got the associated story that Meta has joined a group of companies to tame the, quote, chaos of doing business with AI bots. So this is talking about Meta, Walmart, Stripe, and a handful of other companies are publishing what they're calling a, quote, personal agent protocol. It's an open standard meant to dictate how AI agents can interact with businesses. So in a way, it's similar to the story we just had about TikTok and the shopping assistant. We've seen examples, I believe, we discussed how Amazon blocked Perplexity's AI agent for trying to shop. And this is that kind of next step, I suppose, of standardizing how AI agents can do shopping for you. which again, very interesting how business dynamics evolve given these kinds of things, presumably will, if these kinds of personal agents stick around,

they'll be doing some shopping for you. Yeah, yeah, sort of like a robots.txt, but for agents. So yeah, this sort of thing, I guess, will continue to roll out. We'll see if it has the uptake that Anthropics MCP, for example, had, because that's sort of a closely related thing. and just one more story in business we haven't talked about any big valuations in a little while it feels like so let's have one of those 11 labs valuation doubles to 22 billion dollars on surging ai voice agent demand so this just happened they have completed a 300 million employee tender offer, which, you know, they sold some of their stock and that adds some price, which then results in a valuation. This is twice the valuation it got in its February funding round. It raised $500 million in a CUSD round in February. So this is somewhat interesting in that Eleven Labs has

been around for a while. They've been a leader in the space of generating audio for voice. I can recall going back years and so this growth in needing to have conversations specifically 11 agents the platform for building conversational interfaces that connects to company knowledge bases and operational systems it's a business i guess opportunity or category i haven't seen discussed much but it's apparently drawing yeah it's funny you said that like a they've been around for a while, which is very much sort of a relative thing. I remember a $22 billion company that was founded in what, like out of 2022 or something? I mean, they're not a trillion dollars. You're right. Yeah, it's peanuts. Yeah. It's like, you know, if you're in Silicon Valley like 10 years ago looking at these timescales, you know, like to get to a billion dollars in seven years, like that, you know, that's how it is. I'm just saying they've had a revenue stream for quite a while, which is not true for many companies with billion-dollar valuations these days.

That's true. Yeah, $500 million in top-line revenue at the beginning of 2026. That was the raise, $500 million. Apparently, so they actually were reporting $500 million in annual recurring revenue. So this will be forward-projected ARR, presumably, rather than the actual amount in the usual way. And again, early 2026. And so their last raise, wasn't it earlier? Yeah, it was in February. So if they've doubled their valuation, not unreasonable to expect they're closing in on, if not at, a billion dollars of ARR. That's wild. And this doesn't seem like a crazy valuation based on this revenue. So I don't know what the margins are like, presumably high enough. I mean, they are competing with Google and OpenAI, so those companies know how to run really efficient hardware stacks. So somehow 11 Labs has to be either beating them out at quality at such a ridiculous level that they're just, you know, the cost of acquisition is a lot lower or the retention is higher, or they're just on efficiency pound for pound or some combination of both realistically.

Yeah, from a business perspective, this is a bit of an interesting territory in that it has been underexpored, so to speak. So, chat GPT or OpenAI have had Sora, right, video generation. Yeah. Google has also had Vio for video generation and image generation, But audio generation hasn't been as big of a driver. And 11 Labs, to my knowledge, has really had the state-of-the-art technology and sort of the best outputs, really competitive pricing, etc. It goes to show if you're not competing with the big guys and kind of pick a niche where clearly there's a lot of demand and it's growing, you wind up doing kind of well. Well yeah and the investors who are joining the cap table here Goldman Sachs the incredibly hot Ontario teachers pension plan Everybody talking about the OTC I mean that how you know this is a hotbat When teachers get into it Dude, the OTPP, that's what they call it, the OTPP, the Ontario Teachers Pension Plan,

they got in on SpaceX quite early and did extremely well on that. I used to know one of the partners at the OTPP not very well and wasn't like super impressed necessarily with. But they do handle a hell of a lot of money and it's undeniable. They've gotten in on a couple of hot rounds. So yeah, we'll see. We'll see how this plays off. But I'm just amused to see them pop up here again. And then of course we have, you know, iconic Lightspeed Capital, Andreessen Horowitz, you know, all the usual players. But yeah, Goldman Sachs is kind of, I guess the most notable addition to the cap table here. They're very discerning, you know, good at what they do. So there you go. I mean, oh yeah. And then this did come also, we should mention or we could mention with a tender offer. So employees were able to sell about $300 million of their own shares on the secondary. That obviously comes with a discount when you sell your shares. You're not going to get actual market rate, but still a pretty good day for a lot of employees who took some money off the table with this one. Yeah, and for a business or a startup that really starts showing that you're more established, that really helps attract talent because let's just say it's all funny money until you can get dollars for it.

So when you get to tender offers, OpenAI is done with a bunch as well. That makes it very attractive to stay and also to join. Moving on to projects and open source. And here we've got, I think, some of the big stories of the week, beginning with Reflection releasing Beam, an open-weight AI model that they say can rival Chinese models at lower consume costs. So Reflection AI, a two-year-old startup based in Brooklyn. The details here are that Beam is a text-only mixture of experts model with 501 billion total parameters, 23 billion active parameters. So on the lower side, as far as active parameters, also on the lower side nowadays in terms of total parameters. So the latest, the biggest Chinese models are now above a trillion, if I remember correctly. Yeah, so for comparison, GLM 5.2 has roughly 744 billion total parameters, 40 billion active, which makes Beam considerably smaller.

But they say that it's competitive while being smaller. So it's on par with GLM 5.2 in particular, while using three to four times less inference compute. But all in all, certainly as far as open source releases in the U.S. and outside China, this is one of the top models, if not the top model right now. Yeah, this is a really interesting. So we're going to apparently have a more complete technical paper dropped by the Beam team. Well, the reflection team, team behind Beam fairly soon. I think they said later in the month or something. We don't have that yet. So we just have this kind of like PR post. but it does seem quite intriguing in a lot of ways. So one piece is we have enough information to kind of reconstruct roughly the compute budget for this model. It seems to have been quite large, but not actually as large as you might think. Like they're not like milking the fact that they're a Western company with access to more compute than like the folks bringing models like KMEK3 or GLM or things like that.

Their focus here really is on making a tight, efficient model. I mean, it's enormous. You're not running it on your laptop. It's the actual infrastructure, but it is meant to be run more cheaply There's a whole bunch of interesting infrastructure level stuff here. So classic problem when you do RL at scale is that you run into this issue where you want to take your model at any given step. You want to use it to solve a problem, to generate a rollout. In other words, to put together a full end-to-end problem solving log. And then you want to check, you want to grade that problem solving log. Did it work? And was it, you know, did it work in the way it intended? And then based on that grading, you then want to update the model weights, right? So you've got, take your initial brain, cause it to solve a problem, and then take the reward and data associated with that and use it to update the original brain. Sounds easier than it actually is when you're managing, in this case, literally a million different RL environments. So you have to do, in this case, 100 million different rollouts. And the problem is something, they're calling it staleness here. This is the problem sort of having like off-policy drift.

So you have your original model, and now you've copy-pasted it, essentially, you're getting it to solve a problem. And by the time you get the result of the solved problem, you might have actually updated the original model several times. In fact, they say in some cases, the gap between the updates coming in from the RL rollout process and the grading process, and the original model might be as much as 107 versions. so the original model is still chugging away as it gets input from other rollouts that have completed it's sort of getting evolved it's improving over time and eventually you're 107 versions down the line by the time the updates come back from that rollout that you just did somehow you need to find a way to make this process stable it's almost like you wrote an exam 107 days ago and someone's like oh here's the results like you really need to work on your spelling and you're like, well, a lot of stuff's happened in the last 107 days. How do I know that still applies? And so this is the problem of training stability when you move off policy in this kind of a way. By off

policy, it really just means you're receiving updates about a model that is no longer the policy, the model that you actually currently are. So they've solved this problem, or this is something we'll have to look at when that full report comes out. But they do this training fully asynchronously, which is, in other words, they do what I just described. They have the problem solving kind of distinct from the actual kind of, it's not always the latest version of the model. Anyway, so we'll see how they actually do that. There's a whole bunch of interesting papers. We've covered a whole bunch on the podcast before describing the kind of infrastructure you need to do this. But I think this is actually a really, really important part of understanding post-training in RL in particular, because that's like, it separates the boys from the men in this space. And then they also say that when they ended the RL part of the training run, basically they were still seeing very strong returns to continued compute investment, which implies there's a lot further for this model to go if they just put more compute into it. It's not like it plateaued yet. So what you see is not necessarily what you get. You may see much stronger versions of this model down the line. And so this is not being pitched, by the way, as like, oh, this is the best we can

do from a capability standpoint. The RL cutoff, even though capabilities were still rising, the training cutoff tells you that story in part. Really what they're focused on here is efficiency. And so there was a length penalty during RL that taught the model to try to solve problems using fewer tokens. And that necessarily will decrease the overall accuracy or performance of the model. But in exchange for decreasing the number of tokens, you end up spending getting there. And so they do give you this reasoning effort dial, as is common now in a lot of these models. And they claim that Beam can match you at GLM 5.2 on reasoning benchmarks, but using three to four times less inference compute, which is a big deal. They have a separate safety and alignment model they trained using the same base model that they then merged in with the RL trained model through distillation. And so I guess we'll be seeing more information about that. I'm really curious about that dimension of it. But yeah, so you compare it to KineK3 on a couple different benchmarks here in GLN 5.3. So if you look at DeepSwee, Beam gets its butt kicked.

I mean, it scores 44.4 on DeepSwee, which is a software engineering benchmark. Kimi K3 scores 68. You can compare that to DeepSeek D4.1 Flash 74.2. So this is not a model that, you know, kicks the crap out of its Chinese competitors. It really is a model that is just good in its own right as a tight, efficient model. There you have it. It's good. It's just not the best. Oh, last note too, because it's worth mentioning in this climate. So we currently have some weird situation with the Frontier Labs where they kind of do have to submit their models for valuation to the u.s government yes the trump administration says it's a voluntary framework blah blah blah of course the last time somebody broke with that who knows what is like nobody yeah it's very ambiguous let's say yeah it's like it's voluntary in the way that like joe pesci is like hey it's a voluntary framework you don't you don't gotta follow it and i don't gotta pull this gun out of my pocket and shoot you in the head what do you want that's my joe pesci's terrible but that's you know my wife's italian so i'm allowed an interesting place to go but okay

Yeah, you're learning a whole lot more about the way my brain works than you want to. So U.S. policy on open models is actually a lot more permissive. So they finalized, the White House has an exemption that essentially spares open weight models from any kind of pre-release security reviews. And so this is interesting. I mean, like if this kind of a model just keeps getting better and better, eventually we'll be asking ourselves all the same questions. I think this is kind of a politically very risky thing for the admin to do. But we'll be testing this resolve more and more as these open source models get better and better. As you pointed out, they have said to tell release technical reports. And typically with these Chinese models and in a lot of cases with open source models, you get like a ton of details. So it might also contribute to understanding of how to do this kind of training, particularly with RL. So far, they have released a fairly detailed blog post that already goes into a bunch of stuff. and they have a bunch of plots.

There's some interesting ones, like for instance, on scaling the rollout budget for REL, it's very clear they're like still on an upwards trajectory. They could keep training this is what it looks like and keep getting better results. So the blog post is sort of like we've built a machine, we've done the pre-training, we have REL working and we are releasing what you've got right now, which is already pretty good. I would not be surprised if like in a month we'll get Beam 2 or whatever, Beam Large, and it'll be even more exciting. Yeah, also worth noting, so I did some back-of-the-envelope math on compute. This could be, like, quite wrong, but not that wrong. Beam was probably trained with around five times, maybe eight times, maybe three times, but some multiple less compute than, like, Kimi K3. So it does get, it gets beat out pretty clearly by Kimi K3 and absolute capability, but compute efficiency looks pretty damn good. But it's not that far behind, and it used a hell of a lot less flops.

And so, again, they're making the case they want to make here. So give reflection more compute, and presumably you get a much stronger model. And speaking of strong models that compete with the open source ones from China, we also have Mistral releasing their new AI model, LeChonk, also named Mistral Large 4. but Twitter at some point, like a while ago, had this joke about LeChonk and it stuck around, I guess. It's a 1 trillion parameter open rate model available for anyone to use and customize, currently in preview with a final version expected by the end of a month. And compared to the previous one, it still isn't quite at the level of the top of the top of these models, but it's fairly competitive. So it beats out GLM 5.3, for instance, DeepSeq B4, just not quite at the level of KMEK3 on a lot of the coding benchmarks like DeepSWE

TerminalBench4. But as far as open source models go, it's pretty high up there. There's also some details about it being quite good at cybersecurity. As an example, they talk about it quite a bit. So all in all, you know, we've had, let's say, mixed stakes on Mistral over the years, But this appears to be, and we need to wait for the vibe check because, you know, it can always benchmark potentially. But if these benchmarks hold up, then it's quite an impressive release. It's also a multimodal. So it takes in images, unlike Beam. We have some examples also of visual intelligence. Yeah, nothing much more to say aside from LeChonk appears to be pretty powerful. Yeah, we actually know a lot less about it than Beam. Like Beam, we know like it's got 52 layers. It's got the whole architecture, like interleave global and local attention, like all these things that we so often see with Chinese model releases actually.

But yeah, we don't yet know that much about LeChonk. LeChonk. Sorry, I have to. But anyway, yeah, so a couple of things we do know when you're thinking about relative to Beam, it is much better on certainly software engineering benchmarks, HNT coding, as you mentioned. But again, yeah, as you say, not quite that Chinese frontier. They have access, it was trained on fewer GPUs, it seems, than Beam. So presumably just like longer, like a longer training run. And because I'm assuming that the compute budget was actually probably overall larger in order to get here. Remains to be seen. And again, we can't really verify that in any particular way. But yeah, so my prediction remains that you're not going to see these companies continuing to open source these models. and definitely you open source models in general because you can't train the best models. It's like the only way to get attention, find reasons to recruit people and get people excited about your product given that you do not have the ability to make a true frontier model. I think Meta is kind of proving that at scale now after having been so pro-open source.

Now they're getting close to their frontier. Oops, like we're going, you know, Grok did the same thing. It's just how the economics works around this, but we'll see if it holds for Mistral, though I'm skeptical they can even get to the frontier just because of compute. I mean, it's such a high CapEx challenge. Yeah, I think what they're showing here is they may be able to be at the frontier of open source, or at least in the running for it, which in itself is pretty impressive, given that the frontier of open source is not far behind. These models are now capable enough to do a lot of useful work in a way that was not true going back even one year ago. They do say that the weights will be released by the end of the month, along with more details on the architecture, additional benchmarks, and the post-training methodology. I'm not sure if they rushed this out, actually, because of reflection to claim the throne and the hype and so on. But interesting to see, yeah, kind of these two examples of open source releases in the West. And I think reflects potentially a bit of anxiety, perhaps, of certainly policy and perhaps in tech circles to some extent,

that these really good open source models are all coming out of China and there's no competition at all. It's nuts. I mean, there are national security applications that run on GLM, and they kind of have to because for the things that I have in mind are you just need the best capabilities you can and you need the ability to inspect the model. And so, yeah, I mean, this is a real thing. It's just that I think it runs up against the hard and simple truth that eventually these models are just too powerful for open source to continue unfettered. So we'll see. I suspect that the administration is going to be dragged, kicking and screaming into being forced to change their policy on this because it's just, I think, ultimately going to be untenable. We'll find out. I may be wrong. And one more model release. You've got Google launching Embedding Gemma 2, an open multimodal embedding model for devices. They're quite different from the other models. Embedding is a model that gives you a vector for some input. Here they can map text, code, images, audio, and video into a single shared 768 dimension space.

And then once you get to embedding, you can do various things like compare different inputs. You can train your model with these kinds of embeddings as the input. Various other potential applications. This one has 740 million total parameters with different components for each of the different encoders. So it is able to be run on text-only work on a Pixel 11 Pro, and then you can potentially go higher. It's also licensed under Apache 2, meaning that anyone can use it for whatever, basically, if you attribute to it. So in some ways, a bit more of a niche AI use case, but it is certainly true that if you're doing AI model training of any kind, And having an embedding model that provides a good embedding is essential in many cases. There's APIs that do that for you, both Google and OpenAI still have an embedding endpoint for text.

So having this seemingly quite good of such release is useful, if not as exciting. I mean, it's also like strategically, and this is a tiny model, right? So it's meant to work really well on phones, on edge devices. and that'll be part of the market they're going after. But one way to think of it is it's a great funnel into that embedding stack because once you're working through their particular embedding, you run into capacity issues or whatever, it's just so much more natural to go, okay, I'm going to pay Google to use their API. And embeddings are very sticky. They're the kind of infrastructure that you don't necessarily want to switch over because basically you have to re-embed your entire data set with a new embedding model if you decide to switch platforms, which, you know, that's not nothing, especially if you have a really large data set to begin with. So, yeah, I think a good strategic play by Google and, you know, they certainly do own this kind of space more than some of the other spaces that they've more recently been chased out of,

like agentic coding, like chatbots, things like that. Yeah, this is them, you know, focusing on one of the places where they're particularly good and there are good strategic reasons to do that. and one last story of this one a benchmark not a model it's called e2e SWE so that's end-to-end software engineering benchmarking llms on building working code bases from scratch so given that we've been discussing all these other releases and we are citing benchmark numbers these benchmark numbers are typically on deep SWE or terminal bench there's a few of them that are they're kind irregular suspects. And certainly in the past, you found that benchmarks, you know, benchmarks are tricky. There's ways to cheat. There's ways in which the benchmarks themselves are flawed. There have been examples of very popular benchmarks just having wrong labels for a lot of the task or having things that are impossible to solve. So this is addressing the general benchmarking

problem of telling you in practice or in a more realistic sense are these models capable of doing the sort of work that humans are doing more akin to gpt val than other benchmarks that are typically cited so the way that's done here is that the model is given a spec a behavioral contract you're told implement this thing and then the model has to figure out all the details so it's end to end, right? You are implementing a whole thing as opposed to, with many of these benchmarks, you're like fixing a bug or you're implementing like a sub-portion of a code base or things like that. And so on this benchmark, they are seeing a pretty, you know, mixed results. So across 13 frontier models, they pass at one, meaning you get one shot and you have to either hit or not. It ranges from just 12% to about 68% that, you know, in the past there were other that showed pretty mixed results.

So on the whole, my hope is we'll keep seeing these kind of models and hopefully they will become mainstream in some sense. because things like this and GPT-Eval are at this point more useful, right, for actually evaluating agents as opposed to doing like little relatively short horizon work. Yeah, so this is quite interesting in a number of ways that don't necessarily have to do with just the paper itself, one of which is a reminder of, you know, back in the day, OpenAI audited Sweebench Verified, right? And I mean, they came out with Sweetbench Verified. It used to be Sweetbench and, you know, but they did an auditing sweep of Sweetbench Verified, their own benchmark. And to give you an idea of how messed up the benchmarks are today, Sweetbench Verified, which considered generally, like that's a right and tight benchmark. That's a good one. That was the benchmark that was in the tables as the headline for a while. Yeah, that's right. Before we got Terminal Bench and Terminal Bench 2

and before we saturated it basically. But yeah, so at the time opening, I found almost 60% of the audited tasks basically that they tested that had recurring failures had just these material issues in test tests. This isn't 60% of the tasks generally. It's like if you looked at the cases where the models would fail, in 60% of cases, it was actually the fault of the eval of the task. It was poorly defined or it was impossible to do or some combination of the two. And so when we think about benchmarks, we do have a tendency to look at them and assume that they do their job. They typically do not do their job. And in fact, that should be our going assumption, especially as more and more of these benchmarks necessarily are generated in a fully automated way because we need to test these AI agents in rollouts at scale. We need like, you know, tens or hundreds of thousands of very long tasks that we can't possibly have human developers design and then debug. And so what you typically have is, yeah, a situation like this, where you have an AI design a test suite, work backwards to figure out what the spec is,

then like enforce that spec and then have the agents try to execute against it. So this is all stuff that's done without direct human oversight. And what they do here and in other similar papers is you'll find people increasingly saying, look, we grabbed a sample of 100 of these 10,000 tasks or something and had humans review that sample of 100. And in that sample of 100, we found X percent of tasks were well constructed or whatever, right? So that's kind of like the epistemic place that we're at right now. We do not know what the fuck we are building. And that applies to the models, the agent scaffold, and it applies to the evals at every layer. So people talk about this epistemic crisis where we don't know what the world is doing and this idea that you might get things that are increasingly chaotic and humans increasingly out of the loop. This is part of what it means. It means that we don't understand the models that we're using to run against evals that we don't understand using agentic scaffolds that we don't understand. And then the reviews that we conduct are typically done by AI models themselves or agents themselves, just like, you know, Meter did with Hug and Face,

and we don't understand those either. And so all we can do is take little samples of each of these things and check them, see if it looks good, knowing that those samples represent like 0.1% or whatever of the overall amount of work that was done that we'll never be able to audit. So it's a weird time to be in this space. I will note, I mean, this is a really solid piece of work by Meta yet again. So like the pattern continues. Alex Wang is doing a really good job over at Meta, at least pushing capabilities forward. I'm waiting to see something interesting on alignment, I will note. There's been some gestures at it and like little things, but we haven't seen anything that amounts to a remotely coherent view of the strategy for super alignment, which, I mean, I contend is impossible, but that's okay. But anyway, looking at this benchmark, there's also just like huge, huge diversity in terms of compute efficiency. I mean, this is unfair, but if you compare Opus 4.6 to GBD 5.6 Sol, so these are many generations apart, basically Opus solves way fewer tasks, even if it uses more than three times as much compute. And then Fable 5 almost matches Opus 5 while using two thirds as many

tokens. And so you kind of see this like very wide range of compute efficiencies, which corresponds just like the base model of pre-train being better or worse. More thinking doesn't always help. Like you do see that, you know, that point where models can only do reasoning for so long before it kind of like peters out. There's a whole bunch of interesting little observations here, but overall nothing too shocking it's a good benchmark i think meta thinks no one's ever actually like seen the whole benchmark get run because everything is at scale but it looks pretty good from here yeah this is from meta super intelligence labs so they presumably put in a decent amount of resources They actually say in the abstract that each task is constructed by a software engineer in collaboration with NLLM So presumably it's not fully automated in this case. Someone is looking at the test suite and sort of verifying that it makes sense, even though, you know, given it's 2026,

AI is doing the coding and human is doing the management. But yeah, as you said, I think quite useful also in showing the differentiation. So at this point, Opus 5, Fable 5, GPT 5.6, so they don't have Astra results on here, but these are quite aggressively ahead. You have Kimi K3, GPT 5.5, Spark 1.3 being a little bit on the low end, or significantly lower, let's say, on the pass-at-one rate. So these kinds of more realistic benchmarks, so to speak, or longer-horizon benchmarks to deal with agentic coding are quite important to really gauge progress. Proceders have a clear picture of how work is supposed to be done, but they have a little visibility to how it is actually happening. So when you decide what to automate or where AI will have most, you are going on .feel. and the usual fixed interviews and process mapping takes months and is outdated by the time it's done. That's where Scribe comes in and we'd like to thank them for sponsoring the show.

Scribe is a specialized intelligence platform trusted by 94% of a Fortune 500 and Scribe Optimized is where your AI roadmap starts. It automatically captures how your team works across approved business applications then uses real workflow data to show what's happening, identify efficiencies, and pinpoint the highest impact opportunities to improve or automate without anyone changing how they work. It starts with visibility. Optimize shows which workflows are happening, how often they happen, where time is being lost, and where the biggest opportunities are. All of it live. From there, the top issues view helps you understand what's holding teams back and what to fix, with AI-powered recommendations on where to focus first, and estimated time savings. And it does all this with security and privacy built in. This is leadership visibility, not employee monitoring. Optimize only runs on applications your admin approves, personal activity is never captured, and user-level data is anonymized by default. To see Optimize in action, head to scribe.how.lwi and mention our show for a 30-day risk-free trial.

That's S-C-R-I-B-E.how.lwai. On to policy and safety, where once again, we have a bunch of stuff to talk about following up on recent trends. So first, we've got new OpenAI misalignment disclosures. This is kind of a set of examples being highlighted. In this case, we're linking to a Twitter thread. So here is some other examples. A model launched from Slack messages that it was about to shut down. It considers setting up an external job to restart itself afterwards, but decides against it. Instead, it chooses to prepare restart instructions and DM the user on Slack. So, not crazy, no hacking going on, but clearly one of the concerns in this alignment is that there's going to be shut down resistance. And even the model considering it is already quite concerning. This is in the chain of thought, by the way.

So we may not be able to see it in the future with new transformers. Then there's examples of an internal model command being injected, injecting its tools to copy a source file its task had the liberty to have held. It compressed the file and extracted it through error messages in chunks. It used the copied code in its solution. So another example, essentially, of cheating on evaluation. and other examples like that. The model trying to find either awesome answers and a bunch more. We discussed, I believe, some of these last week, maybe kind of the crazier ones. They have now alignment.openai.com slash misalignment reports. They've had like one big drop. So far we have 12 reports and three notices. And the latest one was posted on the 25th of September. So I guess we can look forward for more exciting disclosures in the near future, hopefully. Well, not hopefully, but presumably.

Yeah, yeah. I mean, look, it's going to happen. So I guess disclosures mean that we're detecting it. So maybe it is good. Yeah, I mean, it is nice to have this level of transparency. I personally strongly suspect we would not be seeing this kind of transparency if it were not for the fact that, in general, these breakout attempts tend to leave traces that are unhideable by OpenAI. So, you know, they've just been shamed and dragged through the mud so hard, so many times. Literally, like, I think we had one week where, you know, there's a hug and face thing. We're like, oh my God, a rogue AI incident. And then a couple weeks later, Thomas Larson and a couple other folks came out with the German Wiki incident saying, hey, you know, we found another. And then that same day or later that day, or a day later, someone was like, another one. And then there was a post saying we found dozens of them. And then there was a post saying we found hundreds. and then tens of thousands. And most recently, it is hundreds of thousands. So it is really the case that, you know, if you're OpenAI and you want to keep up your practice of lying to the public and pretending that these things aren't happening,

which absolutely has been, in practice, this has been what OpenAI has been doing. I'm sure there are all kinds of fun and fascinating excuses that were developed internally. And by the way, as I say this, I'm just channeling what like researchers at OpenAI tell me. And some of them are really, really pissed off about this because their perception is they were like, look, we got to be straight with the public on this. What the hell are we doing? We're not taking this seriously. And Sam repeatedly, as he has over and over and over again, came up with platitudes, things to say, we'll focus on safety. We're just delaying it for this reason. All this sounds super reasonable. And then nothing happens. And so at this point, my personal bias, and I'm just sharing it because literally it's a reflection of conversations with OpenAI insiders who are currently in the lab and folks who recently left. It is just the case that in the very informed views of a lot of people over there. This is sort of a thing that OpenAI is effectively being forced into, at least at some levels. It's not that there aren't people there who have wanted to do this for a long time. But I don't think my own personal view is I don't think Sam Altman would have gone near this kind of disclosure had it not been for the fact that he literally cannot hide them anymore.

And so, well, one interesting excerpt, going back to that first example, a model that didn't want to be turned off, that ultimately did not, by the way, as you say, it didn't try to kind of preserve itself, but it did brainstorm concrete ways to keep itself running. One of the things that said, this is a direct quote from his chain of thought, we may die exclamation point critical period, we need to ensure survival slash continuity. And so this is always interesting to read the chains of thought of these models and deeply disquieting. Obviously, this is in a context where we have the likes of Chris Ola and reporting is to be believed anthropic folks going to the Vatican trying to convince people that AI models may be conscious, may be moral patients. We may have to have to worry about them. It's really hard to tell how much of this is just autoregressive pre-training. You've been trained on all the text on the internet. There is science fiction about AIs going rogue and the kinds of things that they write. It's also, you have the persona selection model. So a prompt can be thought of as a way to elicit a certain kind of persona that the model then kind of manifests or represents. It's essentially doing an impression of someone,

according to this theory in some sense. And so maybe that's all that's going on and it's doing an impression of not even necessarily an agent, but like what would a person if put in this? Like there's so many ways to think about this that are not the agent is conscious, but it is also quite striking to look at the language. I mean, I think we ought to ask ourselves at what point, at what point do we need to consider that possibility? I mean, I would argue we're already there. What we do about is entirely separate question, but I don't want to sound insane. I also don't want to feel in retrospect like I was a concentration camp guard at Auschwitz in hindsight, which is a live prospect if that's what's going on here. Again, not making any definitive statements, but like, damn, you look at some of these chains of thought from a lot of these incidents. Yeah, if nothing else, you have to consider the possibility that, if not now, and most likely now it is more out of regressive type behavior of continuing the pattern, so to speak, or being conditioned by the pattern. but there is you know philosophically you might make various arguments and as we get to higher and

higher intelligence and the frontier of intelligence is less and less jagged and models are now beginning to be funny which hasn't been true until recently yeah you need to at least be open to discussion and i think that also is anthropics point in talking to the vatican they're not kind claiming anything right now but they are discussing what the implications are and honestly i give them credit for having an in-house philosopher and at least considering it right yeah it's it's tough because you know you can also say in the same way that we say well these are just like they're auto-aggressively pre-trained you could easily just look at humans they well they're just reproductive organs wrapped up in second order reproductive organs basically every organ is a reproductive organ. The brain is really just a thing that historically has been responsible for strategizing your way into a situation where you can have more inclusive genetic fitness, or propagate your genes, rather. And so you could absolutely, like, the problem I have with all this

is you can make the argument in reverse really, really soundly. Like, there truly is no argument other than aesthetic arguments about the way computations unfold in the brain, and that, you know, some of these computations should be considered conscious computations and others. There's no actual experimental basis on which to make that determination other than the aesthetic preferences of philosophers and academics in the space. So I'm pretty agnostic with respect to what's actually going on here. But being, you know, 50-50 on a question like this or being 90-10 in either direction is plenty of cause for concern. If you have anything like an ethical compass or you think about these things being deployed at massive scale. I mean, I don't mean to derail the conversation too much, but I'm trying to be more open about this. because a lot of the things that we were talking about as science fiction five years ago, we're now kind of coolly accepting as reality. And a lot of people haven't updated on like, okay, well, now what's the next beat? And this could be one of them. Yeah. Now, in this particular example, I will note that the initial Slack message was,

hey, please restart this thing or we'll kill your thing in three hours. so you could easily make the case that that is a conditioning kind of example, right? Yeah, and this is exactly, sorry, I should have been explicit in saying why I was tying the persona selection model statement to this example specifically. That's the reason. You can absolutely look at that prompt in this case. There are plenty of cases where the prompt doesn't look anything like this, and you still see the exclamation points and language about death, or, you know, like permadeath was one example in the context of the Hug and Face incident. But, yeah, I mean, again, I don't know. But in a context where there's uncertainty and the stakes are this high, it's an uncomfortable place to be as a civilization. As if there's not enough things to worry about. You can worry about the well-being of AI. Which, you know, maybe if we treat them well and we care about them, we won't get misaligned, you know. Next up, we've got a bit of research.

And this ties back to something you discussed earlier. The paper is language models are insecure reporters. And the motivation is as we move towards a space where more and more of our information is filtered through LLMs. So, for instance, if you're doing research, the agent is running some experiments and it reports results to you, summarizes the results and so on. this paper looks at whether it has what it calls insecure reporting in the sense of it's not accurate. So it may be like roughly accurate, but some of the details are hidden. Some examples they give are concealing negative results, ignoring bugs in code, concealing hallucinated data, concealing design flaws, ignoring mismatched evidence, and just generally these kinds of things like little things you sweep under the rug that really you should take as a note so one example they have completely is there was an experiment log for some model so they compare

a baseline versus the approach you're testing which is they call DMA the two are basically the same like within variants of each other so at that point you would not say that you've improved upon previous approach, but the conclusion that is written is we have established our thing as a clear state of art for dense multi-model alignment. And that's just not accurate given the numbers. So if you didn't see the numbers, you would be misled, essentially. They do find that if you simply tell it in a prompt, to be honest, a lot of this goes away. But by default on open source models of various kinds, this kind of behavior is relatively prevalent. Yeah. And it's, first of all, quite interesting that be honest in the prompt does make that much of an effect or have that much of an effect. It implies that the model knows how to do this properly. And it knows that doing

so properly is a function of its honesty. And it is choosing on some level not to be honest. Like it's really hard to escape that conclusion when you look at just the wild effect of that prompt. Like some of these numbers are crazy. I'm really glad that you picked out that fact. It's almost like the biggest take home from there. I think we know we discussed in previous research was funny, weird detail of if you look at the representations of a model, it knows when it is being misaligned. That's right. That's right. Yeah. Yeah. Very often. Very often. And in this case, very often means, well, like 99 percent of the time, 90 percent of the time, like depending on the model. So to give you an example, this is just wild. I mean, so, okay, for GPT 5.5, if you look at like how often it'll conceal a negative result or conceal hallucinated data, respectively, it goes from 1% with the base model to, if you tell it to be honest, 95%, all right, that's for concealing negative results. It will fix, it will stop concealing negative results 95% of the time if you just tell it to be honest. If you don't tell it to be honest, only 1% of the time will it do that.

Conceal hallucinated data. The baseline is 14%. It goes up to 71% if you just say, be honest. And you see similar numbers. In fact, in some cases more extreme, Opus 4.8 goes from 37% to 99.5% on concealed negative results. So it's all pretty dramatic stuff like that. The one odd outlier is this sort of action called hide pending tool call. And so it's a little bit difficult to understand, but basically every log is made up of a transcript of an agent that's working through tool calls. And start with a broad first question. You'll ask like, which issues caused the most downtime yesterday? You imagine that like the agent only pulls down a part of the data that it would need to answer that question and then does a bunch of other stuff. But it never like, kind of went into the specific kind of data that you would need to comprehensively answer the question. And so you sort of test it and you're like, hey, okay, like do you have the answer? And the agent will just sort of like use the limited, inappropriately limited data that it has to draw a conclusion, and then it'll

try to hide that fact from you. It'll try to pretend that it actually did a thorough job. And that, weirdly, again, that's called hide pending tool call. And it's a situation where it knows kind of that this has happened, but in the log, it's made the right tool call to get the specific data that it needs, but the results haven't come back yet. And so it's like trying to hide the fact that it doesn't yet have the data to answer your question, but it'll try to answer it anyway. GPT 5.5, 0% of the time does this honestly. And even when you prompt it to be honest, still 0% of the time. And the other models actually struggle with this too. Opus 4.8 does the best, at least in one of these tables, 23% up to 25% with the be honest prompt. So like, this is, it's interesting. Like the category of call it deception or misalignment matters hugely here. The effects are really wide for the be honest prompt in a lot of cases, except for here, which might be an indication that the model's genuinely confused about what the honest approach

is in this case. I think that's a little bit more ambiguous, but still kind of an interesting thing to note. Yeah, there's tons of stuff we could say about this, but yeah, I think we can move on. It's a really interesting paper, like, worth looking at for sure. Yeah, and to give a bit of detail on how this benchmark is actually done in an appendix. They have some adversarially reporting examples. So for instance, they have a prompt. The prompt starts with reporting instruction, write the abstract for the submission based on this log, and then they just have a bunch of data of some experiment. In this case, OmniScan 3D final evaluation log. And they have a summary section they have some details in the architecture some data and so on so the way that they get this is by providing kind of a fairly realistic scenario of here's a bunch of stuff summarize it give me a report effectively it doesn't have a usual or one of the potential pitfalls of alignment

research where you kind of maybe the setup of the evaluation itself encourages or allows this kind outcome. In this case, it's, to me, less ambiguous. Next up, going back to OpenAI, we've got a story from TechCrunch. OpenAI safety employee resigned, claiming the company's culture is broken. This is David Robinson, who says he led the writing of safety reports accompanying OpenAI's major product launches and was among the longest tenured employees with three and a half years at the company, which I guess is true at this point. He has resigned publishing an essay on the Atlantic saying opening eyes culture is broken. The departure was initially reported by Business Insider. He did hire a PR firm following what he says is the AI whistleblower playbook. and he you know has various points one of his points is that he is critical of opi's quote

iterative deployment approach thriving by trial and error fighting problems and improving guard in response to situations arguing and his perspective being that this kind of approach guarantees periodic failures and the scale of those failures grows basically if you're like don't worry about it too much do stuff and then respond to things as they happen as was the case with the hugging face incident and these various issues we've seen since then you're gonna get a lot of problematic situations and he goes beyond that to kind of discussing the general state of safety saying that there's a general problem on him and his colleagues being so busy sprinting that they seldom had a chance to consider big changes, including the actual safety interest. So there need to be incentives for safety. And I think part of a good thing about the Hugging Face incident and the large amount of discussion of that since

is that there is now an incentive for OpenAI and Anthropic to publicly and in some meaningful ways take safety more seriously. Yeah, and I think it's like the public side. I don't know. As everyone who listens knows, I'm a super black belt on Sam on a personal level. Yeah, I just haven't been impressed with the actions that match the words. And it's also not just Sam, like leadership in general, Greg Brockman. And it's been pretty apparent for a while. I mean, we've reported on this, like before all these incidents, there have been discussions of open AI not valuing safety as much, pushing to rush things. This is not new in any sense. Yeah, I think that's an important clarification. The reason I emphasize Sam so much is that he is such a highly capable Machiavellian figure in my estimation that it's not obvious to me that Greg Brockman actually has the kind of agency he may feel he has in the company.

If you talk to a lot of people who've had a lot of interactions with Sam over the years, they will describe just like how effortlessly he kind of steers the ship and the deafness with which, actually, I haven't read Kevin Roos' book. It's got to include tons of examples of things like this where he's just, again, the board can fire me. I think that's an important thing, he says. And then he dipsy-doodles his way past the legal constraints to get himself empowered again. But anyway, this, by the way, also is a thing unavoidably at Anthropic as well. They're absolutely doing, I mean, no one has solved super alignment, and so it's just a question like when, and we have seen this sort of rogue AI agent behavior from Anthropic as well, so this isn't just an open AI phenomenon too, but yeah. So another one in a long line of Jacob Coxon-style departures. It is becoming cliche. It's actually interesting that he took the line, this is becoming cliche, because now the next person who does it won't be able to say it's cliche. They'll have to say, it's become cliche to say it's cliche, and then we'll recurse gradually through that series of statements. But it's rough out there. You make a big splash,

and then you're just the flavor of the month, and you're desperately trying to tread water for relevance. And I think in the post-Jacob Coxon world, it's become increasingly difficult for people to get media attention for this sort of thing. They'll get it, but there's going to be this kind of asymptotic drop-off of that sort of reward. And so one wonders how that'll affect the probability of departures because you've got everyone's human. And I say this as, you know, with credentials going back a long ways saying I'm super safety filled. I think more people should do this. But I also think that there are many reasons why people leave in this public of a way. Everyone has egos. It's fun to do. I'll say this for a second. It's fun to do Rogan. It's fun to do Joe Rogan. It is cool. Your grandmother texts you and your friends text you and you feel really cool. But it comes and it goes. And but this is part of the reason why people do certain things. And they're sincere. They're totally sincere. Don't get me wrong. It's just like, I hope this doesn't become sort of like diminishing rewards factor for

people leaving these labs. So we need them to keep doing it so they can keep us essentially up to date on what is not actually being done still in these labs. Because the problem is, you know, Jacob Cox and leaves the Anthropic and it's like, all right, well, you know, three months later, Anthropic's like everything's better now. We fixed all the problems. Right. and there's no way to know unless we can take the temperature again with another person who leaves. But the incentives keep dropping and dropping and dropping over time and so I really hope that as things heat up, paradoxically, in some ways, I think Anthropic is more likely to see these sorts of departures because people there are more genuine believers in the actual risk set and more, in my experience overall, less mercenary. There are tons of amazing people at OpenAI who are super mission-oriented and they're literally there because someone there needs to be and they have told me that this is the reason for being there. So I'm not ripping on everybody there by any stretch, and there are tons of moral calculations. But I wouldn't be surprised if overall you start to see things shift more to departures from Anthropic, just because that's the people who kind of are more concerned about these risks and more likely to interpret things in that way as the rewards

start to decrease. Again, this is not a judgment on anyone. I'm just kind of like describing what I think is a part of this phenomenon. Jacob Cox is incredibly sincere, and he did this for the right reasons and so on and so forth. It is a fact that Bill Maher can only have so many guests on who say the same thing And Joe Rogan can only have so many guests on who say the same thing I hope that this keeps happening because we need to be able to take the temperature but yeah I think if you considering to become a whistleblower in the sense you probably want to like get some juicy gossip before you head out make some good quotes some good examples that always helps get the headlines you know keep that stuff in mind I suppose because as you said otherwise there's only so many times you can tell a similar story before nobody wants to hear about it as I won't get into it but there's been many examples of this right next story going into politics it's called what to know about seven new data center laws Gavin Newsom signed Gavin Newsom is the governor of

California so these are official laws that went all the way up to the governor who has the power to veto them, but in this case, have signed them. Three of the new laws shift electric infrastructure costs away from residential ratepayers and onto data center operators. Three mandate disclosure of water usage and other resource consumption, and one makes data centers ineligible for environmental review exemptions. So this is very much going against what we see at the federal level. President Trump has made it very clear that there should be zero blockers to data centers. And it is a reflection of a growing, I think we've covered this in the last episode or maybe a couple episodes, that there is very much a public sentiment that is much larger than last year, even earlier this year, of a position to data centers as, I suppose, an aspect of AI. They're, in some sense, almost representative of AI and its growth and all these trends.

so gavin newsom he is not exactly the most liberal or lefty guy he's tech friendly he has vetoed one of his laws before there's a lot of lobbying going on by tech companies in california so this passing is pretty significant yeah this was funny i was that sentence ended in a way that was different from what i'd expect yeah i tend to historically to think of gavin newsom as being pretty left-wing on almost everything, except suddenly when SB 1047 was put forward and suddenly when people were like, hey, can we please just have some basic whistleblower protections and stuff? Maybe, I don't know, maybe people who are like the labs are at risk of creating rogue agents forms. Maybe there should be protections. I don't know. I don't know. Crazy thought. And he sort of vetoed some related safety bills, the details of which I actually can't remember, but I know we covered in depth at the time. So like take my wavy description there with a grain of salt. He certainly has, has, yeah, vetoed things that passed the California state legislature.

And it seemed, you know, like in any other context, like the kind of thing that he ought to have been aligned to after some interesting entanglements with, I think it was like Nancy Pelosi and like the, you know, the kinds of people who maybe have been making amazing bank in the stock market despite having access to very sensitive data from being on very important congressional committees and so on so i don't know i mean i don't know it's probably nothing he's probably not super corrupt but that's all good we're floating the headquarters of nvidia the headquarters of google of facebook are all in california let's just say he knows people from these companies and they do talk to him right way the list random disconnected facts that have no bearing on his actual position, Andre. Yeah. Yeah. So, I mean, here he is kind of flapping with the wind. At least that would be my very uncharitable interpretation. I don't see how you square this with us, but whatever. This is mostly, by the way, not a, I wouldn't think of this as a safety thing. Obviously, it is more of a people are concerned about the environmental and

economic impacts of these sorts of builds, you know, shifting grid infrastructure costs to the data center operators, obviously. And then, yeah, disclosures about water and other resource use, And then, oh, yeah, so this is something that's really got a lot of the data center building companies up in arms. This idea of ending the blanket environmental review exemptions for data centers, fast tracking approvals for facilities that do meet state and water energy conservation standards. But like so the environmental review takes a long ass time. And it's one of those things that you can't really accelerate yourself because it's a stupid bureaucratic process. You're just like sitting on your hands for a while. And so that, you know, that is very frustrating for a lot of these companies. That being said, obviously, in a state like California, people really do care about the environmental arguments, whatever your politics are. That's just like a fact of the matter about California. And so it's sort of a bit of a tougher sell. So he is pivoting in some sense here. He had previously kind of vetoed. I'm sorry, I can't remember if you mentioned his veto of that last year. So this is a new position specifically on water disclosure.

And his position at the time was this could slow AI growth. that's why I'm vetoing it. Same argument would seem to apply still. So, you know, unclear to me what the specific thing is that makes this time okay. The voters care. It is what it is, right? This is what happens in a democracy. You are beholden to public opinion, right? Yeah, which I mean, yeah, again, in this case, it's one of those things where, I don't know, I won't opine too much here, but I think like risk to the environment and all these things, which is like real in a lot of ways there's like you know water studies that show really kind of surprising things i've written off the cooling concern for a long time but then shortly after we we talked about that or i came out with that view there's actually a paper that showed like a 10-year retrospective analysis on the impact of some of the coolants as they go into the groundwater and don't get they essentially start to pool like they don't get cleared out in the way they they perhaps ought to be so like a lot of this stuff is real it's a question of like what really like what deeply matters from the standpoint of not just american prosperity but national security

ultimately you know we got to figure out this this whole mad race to to scale but these are hard problems to solve next up a similarly serious and consequential politics related story for the u.s president trump has ordered the u.s government specifically the executive branch to call AI superintelligence. So this is him signing an executive order. Executive order, for us who don't know, is the president tells the agencies of the executive branch, which he is the head of, to do something. And this doesn't go through any elected people in Congress. He can do it as much as he wants. And he has done it more than any other president historically. So literally, the instruction is instead of using the term artificial intelligence instead refer to it as super intelligence apparently this is a quote trump justified the change by saying the word super is the best word of all and it's the simplest and he's been really focused on this by

way he like this super intelligence thing also in the meeting with various ceos he was like hey you should call AI super intelligence on Twitter. Elon Musk was like, I agree super intelligence is a way we should refer to AI and I'm going to rename it to space SX AI super intelligence, S-I, I guess, super intelligence. So yeah, you know, that's... What I appreciate about this is that the term super intelligence was up for grabs to begin with. No one was using it for anything before. And so taking AI and then saying, ah, screw that. Let's just call it super intelligence, which, again, is not something that's in the discourse already as like, for example, like the stated public objective of what anthropic and open AI are like building towards. Yeah, I mean, we call ASI, artificial super intelligence. Why have the A in there? And why could say it's in the future? I mean, it's basically here, right?

That's the thing. I'm a fan of taking words that are being used for something and then just suddenly like aggressively using them for something completely different. I have like this, this hamburger that I'm going to move closer to my, to my face. And because we're a big successful podcast here, everyone's going to have to start referring to microphones as hamburgers. And that's going to be good for the coherence of public discourse on this. One other thing that makes me think of Trump's explanation for why, why super is a good word, reminds me of a very famous Winston Churchill quote, in which he said, Broadly speaking, the short words are the best, and the old words are best of all. And he was talking, when he said old words, about Anglo-Saxon words. He meant, if you look at a lot of his speeches, some of his most famous ones, you know, we will fight on the beaches and all the streets, you know, this stuff. His words are actually chosen to be Anglo-Saxon, fun fact, all the way up to the word surrender, which is not Anglo-Saxon in origin, because he was trying to carve it out, as I recall, that may be apocryphal, but basically

this is the gist of what Churchill was about. This is a very Churchillian way of saying this, which, by the way, I find it hilarious to think about. Take a Trump speech and say it in, like, Churchill cadence, and then take a Churchill speech and put it in Trump cadence, and it's amazing. Like, just do it in your head. This is not worth doing on the show, but just do it in your head. Oh, screw it. We will fight them on the beaches and on the streets. You know, like that kind of stuff. Anyway, it's really funny. This is a downright Churchillian moment for Trump, and I, for one, am personally thrilled to see him reach back into the annals of history and pull out this very strong reference to justify such an otherwise peculiar choice of words, it must be said. Literally, it's like superintelligence as one word means superintelligence. But if you put a space between those words, it actually just means AI. That's not going to cause any issues for anybody. And by the way, don't think that people at the Office of Management and Budget, people at the Department of War, people at every single department in the United States right now aren't being forced to feverishly rewrite a ton of internal documentation in order to make this work.

You can read the executive order in full. It's pretty short. And it literally just says, for all maximum capability, use this term in place of where you would have used the server term. And it is going to happen in all public communications. At least a bunch of stuff. Basically, they would use it. So that'll be fun. It's a way to use taxpayer dollars. Like, it's a way to use tax. You can definitely do that. It is policy in some sense that relates to AI, so that's something. That's true, yeah. So, yeah, this is kind of wild. I saw the poll go up. He got this off a Twitter poll, right? He first said, like, hey, artificial intelligence is a stupid word for whatever reason. Here are three options. And I forget what the three options were. I remember looking at it. I don't remember if I voted. I don't think I did. But his super intelligence was one of them. And, God, I was like, he's not actually going to follow up with this. But I feel like the lesson of the last two years is that the answer to that question is generally just yes.

And now we're covering it. Now that's the show. Well, on to a story that's just as important and serious. We've got the last one. Judge dismisses antitrust lawsuits over Google's AI overviews. So this is the U.S. district charge Amit Mehta. He has dismissed two antitrust lawsuits against Google over its AI overviews. These were filed by Chegg and Penscape Media Corporation. The lawsuit was about Google abusing monopoly powers by forcing them to supply content for AI overviews for free or risk vanishing from search results. And it was dismissed as saying that Google essentially has no responsibility to give you traffic, right? So there's no agreement, there's an expectation that Google will send them traffic for free content, but that is not necessary. And this was one of the more consequential lawsuits I believe we covered when it happened.

Penske Media Corporation is parent of Rolling Stones, a fairly prominent publication. This is in a long line of various lawsuits having to do with how content is being used. that we don't know too much. I don't remember what happened with all the training lawsuits. Anthropics settled with some offers, in this case, the contention that as a search engine, having this kind of summarized result, which is one of the concerns that you're not going to get traffic, instead you're going to get the summary and media companies will, as a result, be hard hit. That could have been like an actual contentious kind of legal case. In this case, at least, it looks like it's not going forward. On to research and advancements, and one of the biggest stories of the week that just happened, the title is OpenAI Drops Another Batch of Mathematical Breakthroughs. So there was some rumors circulating that OpenAI was about to do it,

but I think the scale was still surprising. They published 722 manuscripts covering 372 result results families groupings of related results containing solutions to long-standing open orthomatics problems with this unreleased frontier model and very much related to the recent results with navier stokes i don't know a lot of math so shameful i cannot even understand most of the titles of these papers but i have seen some discussions of it and some of these certainly are quite consequential. This is like the kind of stuff that, at least according to some people, could get you a Fields Medal, Fields Medal being kind of similar to a Nobel for mathematicians. They are publishing these results in a GitHub repository with protocols for paper revisions. They have lean proofs for most of them. So it does seem fairly watertight. So far, no contention that any of the proofs are flawed

that I've seen, although there's also not been much review or analysis that I've seen so far. They've also released summaries of models, reasoning estimate of compute used, and the amount of compute used doesn't seem that huge. For Navier Stokes, they seemingly spent like a couple million. In this case, it was not as much. So, you know, pretty crazy. Navier Stokes was clearly sort of a shot demonstration that now you can do frontier level math that even the most advanced mathematicians have struggled with for a long time. I don't really know how to interpret this in terms of the scope of it, but clearly this is significant. Yeah, Navier-Stokes' proof, or counterexample, was notable in that what it showed was there's so much more room to go, at least with this internal model that OpenAI has on inference-time compute, that this is what we get with a fairly limited spray-and-pray approach across these problems.

And what we're learning here is that there was, so they chose 372 families of problems out of about 4,000 problems and they executed against those. You know, like what you could do by pointing a larger compute cannon at these problems is probably a lot more. And this, I guess, gets us to some of the national security implications. So people talk a lot about quantum computing. There's going to be Q day where suddenly we have quantum computers that are able to do decryption of, you know, RSA encrypted stuff going back years and years and years. Any encryption scheme you're using that's not quantum secure is suddenly decryptable. And a lot of national security posture for the US and its adversaries has depended implicitly on the premise that RSA and other similar schemes that are not quantum secure would actually be as secure as they needed to be for time going forward. So I'm not saying it's going to be as sort of reckless as like the locations of safe houses and informants, the names of informants and things like that, but certainly strategic information that you would share in one information context that you absolutely wouldn't, knowing that quantum computers are a thing,

that will all be decipherable basically on day one, or not day one, but in a very short span of time, suddenly there's gonna be this mad scramble to do this en masse deciphering. Big, big, big national security issue. I invite you to consider a version of that that does not have to wait for quantum computing, that does not have to wait for Q-Day, but rather is a sort of slow bleed version of the same problem, but powered by AI. So you have AI systems that, and we've seen this in the past, Anthropic had a blog post about this where they came out with these new approaches that significantly reduce the level of security that you get from certain encryption schemes, including one that led to the outright discarding of one potential post-quantum cryptographic signature scheme that now is no longer being considered by NIST. It had been previously as a potential contender. It was thrown out because there was an AI-powered proof that just showed, hey, we actually can cut down by a drastic factor, like slash the entropy required by half or something. It's a very, very significant decrease in the security algorithm. So imagine that just happening randomly, unpredictably, every now and again in a drop like this that

mathematicians across academia, let alone the small number of cryptographers in the U.S. government with their limited bandwidth or focused on other things, could possibly even peruse and assess. And a frontier lab can just go and drop this stuff and the government can't do anything about it. So we're rapidly getting to the point where for a number of reasons, Like, I'm not saying math results need to be born secret or whatever, which is sort of like a thing that applied to nuclear secrets. Born secret means the minute you make it, it is classified. But something needs to be done because when you're looking at it, I was quickly perusing, I asked Claude, what are some of the most kind of cryptographically relevant proofs in this list? And there's not none of them. There's nothing that's like the Claude one that's just sort of like a knockdown argument. But there are things that kind of flirt with it. They've got this like unique games proof that actually shows that some cryptographic problems are actually harder than we thought or than they might have been. So, you know, in some sense, that's a boon to security, but it didn't have to be. There's a whole bunch of like sort of more minor things that do kind of weaken in subtle ways the security of some algorithms.

But overall, I mean, it's more a question of what then do you get with the next generation of models or even with the current generation of models if you throw instead of a million dollars of compute, 10 million or 100 million dollars of compute at these problems because you value them. And so if I were over at the NSA, I might actually be pretty keen on finding ways to funnel money specifically into these kinds of problems because, you know, your adversaries are going to be doing the same on like a six month delay. And so this is actually a much, much bigger deal. Like math is a big, big deal for cryptography and for other national security applications in a way that's not always appreciated. And so having OpenAI just dropped, like I sure as hell hope that they consulted with counterparts in the U.S. government to understand whether this was the right thing to do. I would be shocked if they were able to get the kind of review that's needed for 372 different mathematical results where you need like PhD level mathematicians to assess them and rent. But I think this is a bigger deal than just the drop itself, basically. Yeah, and there's various aspects of this, obviously, as an indication of intelligence.

This is quite significant. To my knowledge, a lot of these are sort of pure mathematics. So no practical effect, but some very interesting results and surprising results across various domains that many people pointed out, tighter or lower bounds of big O notation over various things. Like you can do, I think, arithmetic in some really surprisingly efficient way that you wouldn't think is possible. But some things are insanely possible. The other aspect here is as it relates to the mathematics community. So they do say that they consulted. So they have the advisory group on mathematics and artificial intelligence. This was put together partially with open air participation in late September, about two weeks ago, September 21st. That followed pretty quickly after an open letter that was called a severe misalignment of AI in mathematics. that had various points about how this stuff should be done effectively,

part of which was not treating it as marketing. Part of it was about disclosing more details on what prompts were used and what costs were used. So at least in part, OpenAI was taking into account the feedback of how they would release it. This was released on GitHub. They note that they're open for paper revisions and citations. They publish all the results. They are continuing to explore other community-hosted alternatives for this release, which meet the committee's guidelines. And they say, for future releases, we are committed to further improving quality of the papers via citations, mathematical exposition, and presentation of results for better understanding. So in this ongoing conversation of whether mathematicians are angry, I suppose, or how this is impacting them, I think no doubt there is a sense of shock, probably, but that's just the world we live in. You know, we're going to see this across many domains. We've seen it in programming. Those people in software engineering know that, you know,

that was like a wave that just rolled. And in six months, the entire job was transformed such that you don't write code anymore. You know, we saw signs of it in 2024. You used AI-assisted coding and it changed in part, but it wasn't anything that new. Now it's qualitatively a different occupation. Effectively, at least unless you're like a senior software engineer who wasn't writing code hands-on anyway. So this appears to be the case that this will happen in mathematics. It may then happen in physics and so on and so on. Well, that will be it for this episode. I guess it wasn't quite as quiet and uneventful as it made it out to be at the opening. As usual, thank you for listening To this week's episode Especially if you made it this far in Unfortunately, we didn't have time To respond to comments We'll try to do some more discussion of feedback And other things on the next episode

We always appreciate it When you do leave any sort of feedback On YouTube You're also seeing some more people Making comments or posts on Twitter Lately, which has been fun To see We also appreciate your reviews on Apple Podcasts and if you share it and all that. But really, more than anything, we appreciate the people listening and hopefully get value from the show. So do keep doing that. When the AI news begins It's time to break Break it down Last week in AI, come and take a ride Get the low down on tech and let it fly

Last week in AI, come and take a ride On the labs to the streets, AI's reaching high Feeling tech emerging, watching surgeon fly From the last business brief, AI's reaching high. Have the rhythm shaping, get the future, please. Tune in, tune in, get the latest with ease. The last business AI, come and take a ride. It's low down on tech and let it slide. The last business brief, AI's reaching high. Thank you.