Claudeが Anthropic の R&D の26%を主導、Meta が816Bのオープンモデルを公開、Gemini 4 Argon も登場

We're live from New York. Welcome, everybody, to Thursday. Right after AI Engineer New York, another very exciting one. We are, there's a big week to talk about. Meta finally shipped the frontier model. Meta literally became a frontier lab. Google's Argon is out for everyone, though I personally haven't been able to test it. However, we have two great deep mind guests today, Omar Sansevro and Genevieve, who's on the developer relations team at DeepMind are going to join us in the next hour to talk about their release. And Anthropic showed investors the numbers of IPO and there's a bunch of other stuff that happened as I bring up, try to hear Wolfram. Welcome, everyone. Peter, welcome. Wolfram, welcome. We're live from New York. Well, at least I'm live from New York. Wolfram's live from Germany. Peter, London? London today. Folks, this is an international podcast coming to you live.

Welcome to Thursday, October 15th. I'm live from New York. My name is Alex Volkov on AI Evangelist with CoreWeave. And with me, Wolf from Rainbow, Wolf also AI Evangelist, also from CoreWeave, and Peter Gostev, evals and model behavior at Arena. And I just spent three days at AI Engineer here in New York. I can't wait to tell you guys all about this. And while I was there, it was an incredible week. just blowing up, so we had to try and catch up. MetaShip Muse Wonder, which was rumored by them as this big watermelon model, and it smells big. It definitely smells like a big model, and you can use it right now in Muse, and we're going to talk about this. It's number two on Arena, which puts MetaSquare in the Frontier Lab Race. As we told you before, multiple times, do not, under any circumstances, do not bet against luck we told you, we told you this 2 years ago, we told you this after Lama 4 dropped, it was a disaster

and we told you this when he started hiring people with soup and so now we're going to feel it, Peter I think you tested it out, I think you have some thoughts we'd love to hear from you yeah, maybe to differentiate, we do have a few different leaderboards and they are second on the taxed arena but they're actually 12 on the agent arena interesting yeah and it's kind of interesting to interpret I think we do have this kind of divergence now going on so for example Argon from Google as well did really well on text so it's actually first so you can roughly interpret it as you know how much people like to chat to the model so it's interesting that there is a divergence between what people want who people want to talk to versus agentic capabilities and coding and all of that and it looks like it's different so yeah argon is fourth on agent arena and muse wonder max so it's not the full version it's on it's running 12. very interesting we also have to talk about the whole thing where i

saw the timeline uh react negatively because uh muse wonder wasn't able to recreate the 3d scenes in and people were like oh no this is a bad model because it it can't compete with astro on like the 3GS thing. We have to address this because I don't know if my assistant every day needs 3GS. So I don't know if this is like a personal thing that I need my assistant to do. Okay, what else? Okay, so folks, later today, Omar Sanseviro from Google DeepMind joins us to talk about it, about Argonne specifically. We also have, thanks to Peter, we have Nathan Denyash from the State of the AI report joining us in an hour to talk about the state of AI. They've been running this for a decade, Peter. It's like a crazy thing. I'm really excited to hear from him. Yeah. Yeah. Look, he's a great guy. Yeah. He's been doing this for ages. Yeah. I actually haven't checked how long, but yeah, I've been reading it for many years. I think it's a decade, but Slopos prepared the news while I was running at the AI engineer.

So hopefully Slopos did a good job. Good job, Slopos. And then what else? Oh, Entropic had an investor day. So before we only had rumors and leaks, whatever. Entropic is about to go IPO. We have some numbers. And a specific thing for kind of the only sponsor for this podcast core with our AgentLens product, Entropic Cloud Code specifically now supports open telemetry. So you literally can start running Cloud Code and then see your results in Forge AgentLens, which is super cool. I think I've seen some examples already. so we're definitely going to talk about this at least a little bit so after Nathan Benyash we'll talk about state of the art but I think since I was a little bit out of the loop I do have a whole prep thing here I would love to hear from you guys about the highlight of the week I think we'll start with Peter well my highlight was I was at the NVIDIA event yesterday oh you met Jensen right?

yeah Yeah, he was Lashy on the show It was very Confusing afternoon During the day, during the afternoon They were building They had this kind of builders event So it was all like, you know, people you meet At AI engineer conference Kind of thing, so that was really fun And yeah, met a few people From Twitter, who I have ever met before They had like, it was an office in London of a company that does like humanoid robots which I had no idea that it was central London, you know, so that was awesome yeah, and they were kind of it was all about open core kind of thing, so they were kind of showing how they also launched a laptop right, the one with the kind of spark or rather, did it start shipping or something I think they announced it a few months ago right, so yeah, a few cool things like that yeah, and then in the evening it was very different. It was in a fancy

kind of reception lounge of a there's a weird converted church building in London and there was AI minister and all of the fancy people from universities and industry and a very different vibe. That's so cool, man. It's kind of like besides the AI engineer London than we both went to and we recorded from. This is the first time doing something about somewhere else outside of the States that gets me FOMO. I'm like, oh, that sounds super cool. I wish I would have been there. So that's your highlight, obviously. Folks are welcome to check out Peter's timeline for this. You posted very well from there. And you can tell us more about this. And then, Wolfram, what's your highlight? What's your one highlight from this one? There's so much stuff, but if I have to pick one, just one that I'm currently most excited about it's an open-reads release of Fish Audio. Fish Audio Drama 3.

That is currently... I mean, there's so much. I could name multiple things, but... Since we're going to cover Gemini 4 Argon, and you are our Google guy with the Pixel 4 and whatever, and you didn't choose Argon, I think that I'm very interested to hear what you think about Argon, since this wasn't your highlight. Yeah, definitely. I can say something about that. Okay, yeah, we'll definitely get to that. Highlight of the week for me. Okay, this is a highlight and a hot take. I've spent the last three days among the top AI engineers in New York and also this AI engineer conference was adjacent to obviously Wall Street. So this was finance and AI engineers together. So there's a lot of finance bros, a lot of big trading companies, a lot of banks. The highlight for me is I realized something that Wolfram and I talked about on the show, Peter, maybe with you as well. AI engineers are sleeping on the personal agents' renaissance explosions happening in the real world.

It's kind of crazy. So, folks who are listening to Thursday, I know you guys are AI engineers. Hopefully, we do a good job to tell you about the stuff that's happening outside of our bubble as well. It is kind of crazy. It feels to me that because everybody's building their own harness, or they're so used to cloud code doing stuff for them, they're trying to force this behavior into tools that don't need to do these tools. Meanwhile, the world is changing for normies because of Grog Bot and Muse. And to some extent, bots or dots. To some extent, dots. And this is like the summary for me because I'm a very personal assistant pilled or personal agent pilled. And I see that I talk to folks and they don't have one running around 24-7 doing stuff for them. This is my highlight of, this is not the AI news, but this is news to me. that even the engineers that we talk to every week for the past three years that love to stay at the forefront of what's happening are missing a huge, huge thing that's happening. So hopefully we're here to tell you about this huge new thing and we're here to make sure that you guys don't miss it

because I think it is huge because most of the normies that I talk to did hear about some of these personal agents. Wolfram, go ahead. What really excited me this week, it's another thing, it's not an AI release or something, but it is something someone I very much respect, said because Linus Torvalds really said that not using AI to create code is like it's just a tool. So if you don't use it and you are a smart person, you should be using it. He said if anyone is submitting, as an example, it's as crazy to say you can't use AI for doing something. It's like saying you can't use a compiler to create code because that is also a machine tool that you are using and you are not using a sampler to create the code. and that was an example and it's great to see the mindset change. I mean, he already had an open mindset and he says, as long as you understand the code, I don't care how you create it. That is the important thing and I think more people should be thinking like that. It doesn't matter what tool has been used but did you,

whatever you created, do you understand it and does it work? That is the important thing, not how you create it. The do you understand the part is interesting. We should definitely talk about this because I definitely slopped. I turned my slop can towards some projects that Peter, I think you can resonate. I have no idea what's going on under the hood. As long as I see it and it works and it's dope. But I do agree. There's some stuff, especially in critical infrastructure. So the folks at OpenAI, let me scratch that, the folks at AI engineer from banks, for example, there's a lot of stuff that needs to be human-readable. There's a lot of stuff that they need to understand everything. Somebody taught me the concept of toxic access, where the loan book, the master loan book record or whatever, something inside the bank that all the money lives in for everybody. If there's people with access that can change some stuff, that's like toxic access or something like that. So that, obviously, people need to understand what's going on there and be accountable as well. Folks, I think let's go to TLDR, and then we'll start with the show.

Comments from the audience as well. Feel free to tell us what is the most exciting stuff that happened that we may have missed. But let me run to TLDR, which is a section of Thursday AI, where we tell you briefly about the most interesting, exciting parts of what happened, curated to our interests. I think I'll start saying this soon, but we used to be able to cover all the news. I no longer with fair conscience can say this is all, everything that happened in the world of AI. But here is the most important thing that happened in the world of AI. Let's go. All right, here's everything on Thursday. I think that we're covering on October 15th in about five minutes. Let's get into it. So we're live from New York after AI Engineer, where I think the highlight was Capital One Max Canut Alexander made the best case I've heard when code quality still matters. This is probably the MVP talk, and I would recommend folks to tune into the live show and wait for the release.

SWIX also came on stage and he's on conference and talked about how to kill your SaaS hackathon. There was a hackathon this weekend and 72 clones of big sasses spent 100 billion tokens over this weekend to build clones. And then Anthropic shows dynamic workflows as well. Let's see what else we have with the TLDR. Yeah. In big companies and APIs, Meta's first big and heavy and frontier-ish model or maybe frontier model. No, it's actually frontier model. Meta Muse Wonder is number two on Arena and this is the best version and it is now available in different tools. It's available in Metacode and Muse only for subscribers. Some builders are speaking about this. Also, there's two versions there. There's Muse Wonder and Muse Wonderful. Did you see what I did there? Muse Wonder and Muse Wonderful.

That's, I think, what they did. Because everybody calls their models Max on the Max reasoning. They call it Full, and it's wonderful, and it's pretty cool. Gemini 4 finally launched. Welcome, Gemini 4. Again, as a reminder, we'll have the folks from Gemini, Omar Sansevro and Genevieve, joining us in about an hour to talk about Gemini 4 Argon. It's out for everyone. It's number one on Arena and Visual Evals. And we can't wait to talk to these folks and hear all about the use cases about Gemini. And then we talked about Entropic opening open telemetry traces and talking about their IPO. And Agent SDK still runs on your cloud subscription. Wolfram, hopefully you caught that. There was an update and now there is an Agent SDK. And if you guys remember the whole, maybe Entropic will block you from the subscription service if you use it for like Hermes Agent or whatever. Now, we're back at a place where AgentZK supposedly runs a new cloud subscription. Antropic also showed investors a $16.8 billion quarter ahead of its reported November IPO.

And there's a rumor about Fable 5.5 showing up in Azure, but it's unconfirmed, just a rumor. Folks from TypeSafe, this was just this morning. This is like nearly breaking news. I think it's in the breaking news category. Jav 1.14, which can now see images. So ImageJav, we heard that it's coming, and ImageJav is now here, so it's now in preview. And I think that's one, I really want to try it. I didn't have time running around, but ImageJav is something I waited for a long time. Microsoft shipped its infant ship, Maya 300, and its own decision model, Microsoft Decision 1. So decision models are taking over, and I'm very, very interested to see how decision one stacks up with image jev. Open source LLM. In open source LLM, four open decision models was released. Amos won from 0.5b to 27b. They also answer with bounding boxes.

So it's not only jev. Jev answers with bounding boxes. So do these models. I haven't. I haven't had the time to test them. Mini CPM returns with 4.7 and runs a 35 billion per hour vision model on iPhone about 3 gigs. Liquid, shout out to our friends from Liquid AI. There is the biggest Liquid foundation model, 24 billion parameters built for local agents. And we constantly talk about running agents locally. And then Quinn Geyer, RL framework for Long Horizon agents came out. I think it's a paper. in this week's buzz we told you about forge core we forge the rebrand for our agente cloud offerings and sas forge model distillation is now in preview and i think that's super cool and also agent lens in preview and agent lens is just hot this week because now you can connect cloud via open telemetry and see the most you know the most useful and not useful traces from

your cloud run directly in agent lens. I think that's super cool. Uh, apparently XTI dropped Grok Voice 3. Do you guys here see this speech to speech leaderboard? It tops the speech to speech leaderboards and apparently all of the Starlink support calls are now used by Grok Voice 3. And Wolfram, you also mentioned Phish Audio, correct? drama voice. Just summarize it for us. Yeah, it's a new model, a speech model that you can direct, where you can give instructions for how the voice should be sounding. I think this is best if we, when we talk about it, we show some examples. Yeah, we may be placing examples. So you can run it locally, and it's also at the top of what is possible with voice right now. Oh, that's super cool. All right, let's go to industry. Industry is where we talk about stuff that are not releases,

but I think it's important from this week. Andre Carpathi says six billion people have barely touched NLLM. He did some back of the napkin math. And don't we love a good Carpathi tweet that says the industry lays, and it feels like he's been tweeting less since he's joined Entropic. So he literally had this like funnel of saying 6 billion people very touched in LLM and AI. And the gap, he says, is getting wider. And then he kind of does calculate. So hopefully everybody who tunes into such the AI folks, you're on the top of Andre Kapati's funnel. But he does have this thing where like only a handful of people touch the models at the Frontier Labs. I'm like, okay, fuck you, open up the weights and then let's touch them. Sorry, I love Andre. Senpai is incredible. is incredible. I just like, way to be, I should say, guardrails and not let people in with this thing. All right. And, oh, and last week, State of the Eye, State of the Eye report came out.

Nathan Benyash, the author, the guys who've been running this for a while, is going to join us. It was at 10 a.m. Pacific, which is going to be in an hour to talk about it. And I'm excited. Peter, you connect us to Nathan. I'm very very excited he's been running this state of AI report for a while now Alex I think did you mention Nemetron I don't think so right it just launched about half an hour ago oh Nemetron 3.5 Super VL 120 billion total parameters and 12 billion active parameters no it looks like they're focusing on more, it's multi-modal so the idea is that it's good at video understanding, image understanding and long-running agents and roughly where they're kind of pitching it at deep-seek before flash, including small that kind of level but it's meant to be quite fast

and they say it's performance at that sort of level so, yeah we haven't tested it, it just literally came out yeah, perfect shout out to our friends from Nemo-chan, Tim Green and my friend Chris Alexer who works on that team and always joins us and talks about us. Shout out to NVIDIA. There's also rumors that NVIDIA, you guys see this, NVIDIA is in talks to buy Reflection? I think it was Reflection. The lab behind Beam that we talked about, Beam, last week. I'm pretty sure it's Reflection. There's rumors that Jensen's collected the infinity stones of open source. So Hug and Face, Reflection, also rumored, also friends of the pod, And the Israeli company rumored that NVIDIA is talking about purchasing them as well. All right. I think it's time for us to go to Frontier Labs and talk about Metamuse Wonder.

All right, folks. Let's talk about... Do not bet against Mark Zuckerberg. And now Alex Wang. So in addition to having great meme taste, Meta is now back at the frontier. Meta's frontier model is number two on Arena and just 12 points behind Argon, which we're going to talk about. But some builders are calling these benchmarks, and I think several things can be true. Meta Super Intelligence Labs released NewsWonder and NewsWonderFool. Wonderful is kind of the over-the-top model that we don't get in the API. It only exists in meta subscription tiers. Alex Quang called it their largest and most capable model yet. I joked with somebody yesterday that all I need to say every week on the show is that this new model launched, numbers are bigger, and they claim it's their best yet. And it does, we're kind of there, right? We're kind of there where it's like the new iPhone 18 moment for every lab, and this is the best iPhone we've released yet.

Both meta models, though, they do smell like a big model to me. So both 1 million token. Muse Wonder via API, it's four bucks in and $20 out per million tokens. Basically, Opus 5.5 pricing. But if you see the E-mail charts, they actually show that they beat Opus on multiple interesting benchmarks. And those benchmarks are not your standard benchmarks. Besides the SWE bench one, I'm pretty sure that those benchmarks are a little bit different. But Muse Wonder, in parentheses, full, is the better model. That's the one that kind of, they say, beat Opus and beat Astra and beat Fable. Only for Muse Code subscribers. And $50 or $200 a month, this is the tier that you need to pay. Either $50 a month or $200 a month to get it on Muse Code. No open waits for it either. But here's the news that excites us. MUSE Spark, the model that was there before MUSE Wonder, they told us about open source and MUSE Spark is basically the new LAMA.

They're going to, not yet, but they're going to open source MUSE Spark. I hope this is the latest version 1.3, but I'm very, very excited. META says the META Wonder models are too big and maybe even too risky for open source. That's an interesting call out. Right, let's just dive in there. Peter, I know you guys tested Muse Wonder. Would love to hear from you. Yeah, so you're right to say that it was second on the text arena, which means that people like talking to it. It is, I think, 12th on the agent arena. And agent arena is when people like you and I go on the arena and use it in various kinds of ways, which could include coding, could include bi-color work. all the work. So that's not amazing. I must say 12 is not that great. Which is kind of interesting. People say or is it benchmarks or like Alex Wang is fighting back and

saying, oh, you know, who needs 3G and all of that. But, you know, 3G aside, I have tried also building websites with it. And I must say, it was kind of awful that website. I don't know. Maybe it was my experience, maybe I did something wrong and I didn't spend like many hours with it, but I think that's kind of important as well, I've never seen a model that's really bad but it's good at 3G and it's good at websites, but it's fundamentally bad I think it's kind of unreasonable to say, well it's much as opposed but actually it's really bad at half of the things that people use it for, so it's kind of, I don't know, I don't want to be unfair to them. It's clearly a better model. But equally, I think it's still behind on some things. Like front-end for sure. 3D aside, front-end, it seems like really quite bad actually. This is different here. I will say, while 3D is not a use case that I expect my assistants to kind of use, and

it's obvious that NetA is fine-tuning MuseWonder for the Muse Assistant, right? Like it's very obvious to us why they have the model and the way they're going to put this capability in people's hands is via their personal agent or AI assistant, whatever you want to call this category. And so they're optimizing for that thing. We should probably mention the two innovation-type things that they're adding there that are not evaluated, which is dynamic memory and dynamic contact stuff. However, to your point, while asking my assistant to create 3D castles and 3D series or whatever Nissan created and they showed us last week, a virtual copy of Toronto with 100,000 agents running. That's not what I'm going to ask my assistant to do. Building websites is kind of what I think many people would like their agents to help them with. Building websites, one-off websites for like a use case or like more stuff as well. So I think that this is definitely in the realm of something that people will ask their assistant to do. Wolfram, do you have any thoughts on MetaMuseWander?

on one hand that is exactly this what I've seen less technical people who are not as much into AI they use AI assistants to create websites or update their websites so that is very important to be good in this regard 3D on the other hand of course it's a nice demo but the practical use case for us most of the time is not that high I tested Wanda and it is excellent in German which is not true with all the models so it has the colloquialisms and stuff so this is great. Yeah, it's a good model and I think that they have their own agents that they are offering personal agents. It means that it can also be used very well in Hermes Agent and the other ones. It's available with the API. Yeah. I think it's great to have more choice here. I think a very interesting thing is I haven't seen the Muse code subscription being used in Hermes Agent. I don't know if that's possible, but I think that'd be cool because now we're going to get there but the cloud SDK looks to be back up.

What's always possible is to have your Helmys agent just use another agent. I'm using all the agents like that, that Helmys on my own system is my main, and connecting to Krognod and the other ones and just taking control of them all. Yeah, how did you set up? Yeah, we had a comment as well about Muse. Yeah, that's a good idea, actually. I never thought about trying it as well. How do you, if we have time, how do you set this? What do you use this for? Oh, I've been using this for a long time Even before the agent stuff came out I was calling it minions Amy has her minions And she uses computer use If there is no API or CLI tool It's just a skill basically A minion skill where she can use We did it when the first computer use stuff came out She was going to the chat with the website And using that as a minion from Claude And stuff like that And it still works the same way You just give it a tool Like it can be used a CLI tool or it can use the website or Crock-Bot, the app, can be controlled with computer use and it's just another agent for

a sub-agent for the main agent. So very easy to set up. I do it all the time in the terminal. Cloud calling codex, codex calling cloud. The variety gives you a lot for 100%. I'm just wondering because these are different systems. They're extremely powerful, connected to many different services. Just the use case. It's like, what are you using it for? I mean, you can't be interesting. One use case is the council, like GBT Pro, which you can't just use as the API and so on. So we are using it that way. If we are creating a concept for something, it goes through all these. I have a whole council of 12, 13 agents that all work together to make sure that this is the, if it's very important, to be the best one we can make. Cool. We definitely should talk about use cases, though. So we talk about models and numbers, et cetera, but I think as we started to say in the beginning, we're approaching the level of capability where like for most of us, the stuff that we need to do,

we can probably already do. So, go ahead, Bill. One thing to add as well, I think, about the release, and I imagine they probably just don't have enough compute, but it sounds like at least from like... That doesn't? I think they don't, yeah. Yeah, I don't think so, because why is the full one not available via API? I don't think they're just being stringy, right? Maybe they want to cross all the subscriptions, but that feels, I don't think that's working that well. So, for example, I think Theo, the YouTuber, he was very complimentary about the Muse model, which I was confused by. And then he came out and said, well, actually, the full one is great. The regular one that we got, he thought was bad. So I haven't tested the full one, so my impression is very like meh about the model, but maybe the full one is amazing, but we don't have access to it. So I wonder if there's also that kind of mismatch, that they think all their model is amazing,

then the reception is like, or is it benchmarks or what? But actually maybe it is amazing, but they mean a different model. It's not ideal. This is kind of on the same topic of folks, let's say inside on topic, they're probably not using Opus. They're probably using like almost plus plus whatever model. And so when they talk about the stuff that works for them in the public, I'm not sure that they mean kind of the models that are available to us. That's a very interesting point. And I think Theo also gets early access to like the models that are not available in the API. You can get Muse Wonder, not Wonder Full, Muse Wonder within the product. So I got to wonder if any of you guys used it within the product itself. Because I switched immediately. So I've been using Muse, as you guys know, because I've talked about this a lot. And I will continue talking about this because I think Muse is wonderful, not the model. When I realized that I can only get access to this new intelligence, essentially what happens is, Wolfram, you know all about this. You've been doing this for Amy for a while. Essentially, I realized that I can upgrade my assistant's intelligence with a click.

And that is incredible to me. That is just like, this is the scene from the Matrix where Neo gets a thing and now he's like, boom, and I know Kung Fu. So skills is kind of like that but also like upgraded intelligence is kind of like that And so I upgraded Muse Wolfred my Muse to Wonder and I gave it a lot of tasks Here's use cases time. Let's talk about use cases a little bit. The podcast studio that I'm in, shout out Virgo and Ezra for helping set this up super quick. Muse found this out. Muse emailed Ezra a couple of times, including the notes and the requirements I need for the podcast. On the way here, Muse got me coffee. I did a meetup yesterday, liquid cooling IRL, AI engineers in sauna and you guys can see the Swiss band Muse designed and ordered it and it shipped to the hotel and all of this was a news wonder I was just kept like shoving tasks to it and it was executing them mostly perfectly the failure modes of this model Peter is real hard to benchmark so we need to talk about how we benchmark those like use cases but the failure

modes of this model were not the logic parts the failure modes of this model was the website doesn't work, the website is blocking us because CloudSquad doesn't like agents, or the Stripe link doesn't quite connect or something. So the failure were real-world failures, including very funny ones where news called some businesses and businesses hung up because they don't want to talk to a personal assistant. It was really funny. So I think, again, I think we're approaching a level of capability where most people do not care, as long as it does stuff for them. And then the question is, does it do stuff for them? So I think when Meta released the models, they had some evals in there. I think we should pull them up that are not the standard ones. There are some like personal assistant ones. And, Peter, the question back to you is, you are saying on the agent arena, Muse Wonder is kind of suffering. I got to wonder if my experience with like the personal agent stuff versus the agent arena, where are those tasks falling? Where's the daylight between those tasks?

Yeah, it's definitely hard to say. And I think, yeah, it could, the kind of failures that you're talking about, they do sound more like the harness in a broader sense, like connections, integrations, like things like that. So, yeah, it's definitely hard to say. One thing to say, though, about the benchmarks, and you mentioned the different ones, this is kind of a pattern, which is a funny one. I did a bit of analysis on this. Maybe I should publish it there, where I kind of pulled together the different benchmarks that labs were using over the last year, just all of the releases. And the half life of a benchmark that they quote is basically like one release where so many benchmarks just get quoted once because they get a good number and then they're never seen again. So I'm not accusing them of this in this case. Maybe they're meaningful benchmarks. I don't know. But that's also something to look out for that quite often I would open a release

and I'm looking at it, I don't recognize even the titles of half of the benchmarks. Yeah. And, you know, I'm in the industry. So I don't really know how reliable, how are they going to stick to those benchmarks or not. But if they're inventing new ones or kind of citing new ones because it has a good score, I think that's also not necessarily indicative of the performance. That's true. And I think to some extent this makes sense because capability, let me try to find the exact thing you're talking about but i think the capability moves and different benchmarks need to switch as well right like we've seen that open the ad decides not to do sweet bench anymore because sweet bench was like leaked in full and then sweet bench verified came out as which pro and like a bunch of like updates to this paradigm um but i definitely want to pull up some of them so we can talk about this meta wonder niston meanwhile any comments on muse wonder from your end no it's is wonder a new thing i'm i have news running in circles all night i don't know what

it tests websites and stuff it's pretty dumb sometimes but it's just so energetic listen you don't know what news wonder is no idea i love how co-hosts of thursday i sometimes join as co-hosts with deep insights and knowledge and building things in engineering and sometimes we join as guests and just like chill here and like get their news from the live show it's incredible So Muse Wonder is their new model. This is their opus, the beats opus on multiple things. So I think it's time for us to show some numbers. Oh, wow. How does it compare to Argon? Holy shoot. Yeah, so this is the thing. They did not do any Argon comparisons. The Muse Wonder full and Muse Wonder on max are compared only to Cloud Fable 5.1, Astra 6, and Cloud Opus 5.5. And I think, Peter, maybe this is the one that you called out. They have a new benchmark called Personal Superior Dangerous Index. internal composite index of personal and knowledge work. So basically doing kind of like an artificial analysis-ish thing, capability-ish thing. I'm not quite sure how that works out.

And on that index, Muse Wonder, the one that we don't get access to on Muse Wonder Full is 59, and Muse Wonder is 54, beating pretty much everybody besides Opus 5.5 next. Which other ones here, Peter, would you say? That's crazy. Wait, hold on just one second. Peter, which ones of these would you say are one-times or benchmark or new ones? Well, like if you scroll down a little bit, so we've got GPT, PDF, JobBench. Yeah, GPT, PDF I have never seen before. JobBench I have never seen before. MemoryBench. I'm missing LDJ here. He knows about some of these. I will say on the artificial analysis index, which I think a new one is coming, this model does not beat Astro and Fable on that specific index. But also that index, again, it's testing generality of models, I think coding agents and not necessarily personal assistants. OS World, for example, we know and love, right?

OS World is a benchmark that measures computer use and browser use. And on this benchmark, let's see, Muse Wonder, the one that we do have access to, matches Fable and is significantly below Opus 5.5 on computer use and even below cloud. So computer use, this model is not the best. Okay, so going back to what assistants need, computer use is like the number one thing that I need for my assistant. It needs to go and click the right buttons. Otherwise, it's not useful to me. A very interesting set of releases here. And I mentioned Argon. We're going to talk with folks from Gemini. about Argonne in about an hour. We have Omerbach and Genuiv, so we're going to talk about that with them. I think we're good on the Muse coverage or anything else left there? Wolfram, go ahead. Yeah, since we were looking at the benchmarks, as I'm creating WolfBench 2.0, I have a crisis moment and I realized something.

It's so hard to benchmark agents because they are complete systems. They are not just the model. You have the skills and memories and all this, and everybody using an agent has a different one, even if it's the same system, because of all the different personalizations, connectors, and so on, and even the internet access and search providers, they all affect the results. And the thing is, nowadays, we are now at a point where you can have your agent create a benchmark for your own purposes and use the same system. I'm doing it with Hermes, so I can put the Hermes in the sandbox with my soul and skill and so on, and use a model and do some tests specifically for users I have. And I think everybody can now do this. Just have your agent build it. You can build all the software in the world now. You can also build an evaluation. I mean, as someone building evaluations professionally, that is a strange moment. But it is really here now where you can have a customized evaluation for your own purposes with the agents you are using and test new models, test new agents. All of this is now possible,

which wasn't possible. Just a lot of work now. Now it's possible. There was a talk at AIE that calls out this paradigm where you're like, you talk to your slab generator. You're like, hey, how can I know that you're right? And the slab generator says, well, I can do an eval for you. Here's a few things to test myself on. And then you say, okay, do it yourself. And so the slab generator generates the evaluations for itself. And they're also slab. And where are we if nobody reads those traces? We're going to talk about reading traces in a bit in the cloud section. This is Muse Wonder And the call out there for our open source folks Is that the Muse Spark model Is going to get open source And I think that's one That's great I need to start using the word wonderful Because it's confusing But I think that's great Let's talk about Entropic Entropic is big in the news this week Entropic gave us four stories this week You can trace your agent now Which is a very big one For at least the sponsor of the show You can now

or I guess, again, we're so bad. You can run the agent SDK on your subscription. So this means that you can plug it into Hermes. I see Wolfram shaking. He said very strongly. Because, folks, I have told you before. So all of this personal agent type revolution, what I mostly need from this is that personal agent ability with Opus 5.5 capability. And now it seems that while I was able to pay API prices for this before, Now I can do this with my cloud subscription. So I'm definitely excited to try this out. I've seen this in multiple coding harnesses and agents. What else do we have? Antopic also showed investors a $16.8 billion quarter. And also rumored Fable 5.5 in Azure coding, which is a very interesting thing because Opus 5.5 is kind of beating Fable 5.5 for most stuff. they're not like super super strong so let's talk about some of the

entropic stuff how is this lab one more there's even one more they posted an article about strengthening their defenses against distillation attacks so after clamoring for we have to regulate and ban this stuff now they basically did it by themselves increasing their defenses they reduced the amount of distillation that happened to the trace reasoning traces and so on by 90 to 99 percent. So it will be much harder to distill cloud reasoning now, and we'll see what effect that has on other models. I know the effect that it has on me. There's not a day goes by that my clouds are disconnected, I need to re-log in again, and then the remote control thing fails, and so from New York I can control my Mac. I don't know if this is the effect of their security on the distillation, but it affects me personally. I don't want to call this out. on topic please please let me use my clouds wherever i am this is getting old this is really getting old all right about this uh let's talk about open telemetry which is kind of a nerdy

subject but it's an important subject open telemetry is this open standard for how systems communicate their traces to each other basically the logs open telemetry has been around for a long long while and then folks like the wee folks from weights and biases and now the agent lens folks which we're now launching in Deploy Preview. This is how systems communicate with those platforms. And now cloud itself, the cloud harness, is now supporting open telemetry. What I'm saying is, before you had to run something like a 1B hive mind, which we told you about, that shows you all the cloud conversations. Now cloud itself can send its traces to whatever tool of choice that you have. obviously on this pod, we strongly ship the Core with Agent Lens product we just launched, but anything that supports OpenTelemetry, Cloud Code now supports. I think it's incredible, and I think it's very interesting how what you just said, Wolfram, where they try to prevent distillation, while also simultaneously allowing Cloud to send his face in OpenTelemetry.

Though, this was never a problem. Cloud was always writing the conversations to local, So you're always able to get it on disk. They're not encrypted. The reasoning traces are encrypted, but the conversations themselves aren't. But I think it's a cool tool. You can send your traces to observability stacks. And this includes everything, agent turns and tool calls and MCP calls and latency and token usage. So you can properly measure how much cloud is costing you. And if you're a company, you can do this at scale. Oh, there's one staff here from Ramp. They said this helped them to cut token use to 10% to 20% on some agents because they learned that some of their agents are running through loops. Are you guys following your agents in any way? Are you evaluating them based on this way? Will you be using this from Cloud Code? Nistan and Jan both shook heads. Definitely going to try. I'm going to pick it up. Yeah, this sounds good, especially for stopping loops and getting overall agent stuff.

I have noticed that cloud code does tend to stop often. If you have anything to do with the reasoning, now it just triggers it as a word. Like if you're working with any type of LLM, like I've been working on WedgeView, the moment I even mentioned anything about reasoning, now it just triggers it right away, which is like for a small model. It's still good, though. It can't be dopest. Opus inside the Cloud Code Harness, I think it's still the best tool for overall tasks and doing some stuff. Opus also helped prepare the show. Just Opus Low. If you need to do something, just try all in Opus Low, guys. It is just straight up vibing. So I think because of this reason, Anthropic is also going to do an explosive IPO. They reported an insane quarter, $16.8 billion, about 45% from the second quarter increase.

Adjusted profitable to the second quarter in a row and cash flow positive. Anthropic is profitable and cash flow positive and gets billions of dollars and going 45% from the previous quarter. Still, it's absolutely insane. This show definitely has no financial services things. Don't take anything we say as financial advice. But if this IPO is coming, are you guys putting some money in the traffic finally on the public market? I think it's a buy. As much as, you know what I've said, I've called Dario a communist and stuff too. Yes. Twitter. But I think as far as I stopped at Will Grell, in my opinion, this is just vibing advice. It's just... I think it's a buy. I think they have way better opinionated coding than the other models. I think people are

impressed more by the agentic and completeness ability of Astra, but like the actual quality of the code, it's not that good. I mean, it's pretty good, but it's not as good as Opus. And when it comes to DevOps, even Opus is bad. And Astro is, like, even below that. Yeah, I think they still have a lot of tricks. And it's mainly because of Haiku, honestly. Haiku is, like, the most disruptive model I think they've ever released at that price. And they even give you $200 in credits now to just use throughout apps. That's not going to run out of one cent at $100K. Whatever app you're going to serve to people, if you just keep it under 100K context for Haiku, which most conversational UX stuff is good enough, that is more than enough. And on the max land, that $200, that's probably going to support like 1,000 users pretty easily. 10 cents in, 1 cent cash, that's, yeah, it's just way too cheap.

That's why I think it's not the worst idea to get someone tropic. Get you someone tropic. And it, like, pains me to say that. Okay. So let's talk about the subscription saga. Wolf, I'm going to put you up there. As we said, we would love to use personal agents with the Opus 5.5 level. Now it seems like it's possible. Can you cover that a little bit? Did you see this? The cloud SDK and the slash B stuff. That's why I was waiting for the new Hermes update, where this is now possible to do. I cancelled my subscription with Opus or with Entropic when you weren't able to use it without getting banned. Some people apparently got banned for using it. And so that was when I decided I don't want the subscription if I can't use it with the agent I'm using. I pay for this and if it's cloud code or if it's Hermes, it shouldn't matter in my opinion. And now it's possible to use your personal agent, be it Hermes or whatever you use, and connect the SDK, the agent SDK,

and use Opus as the main model in your own agent that way, again, in a legal way, without being banned. So, yeah, now it's possible again, and I have to reconsider resubscribing. So, the cool thing about what happened two weeks ago is that not only did OpenAI always allow this use case, they also literally shipped a button and an integration called, like, sign with your chat GPT, and then the other products could use your tokens from your subscription. So Devin is one example from Cognition. It supports that use case from the get-go. Hermes agent was called out on stage. News Research, shout out to News Research. Our good friends from there and congrats on their round again. And Hermes agent is now going to support both. And this was always the use case with OpenAI. It looks like Anthropic kind of needs to follow suit. So now it's going to be possible. Quinn Slack, also a friend from AMP, the coding agent said, this is why now you can use their models. Peter Gosto, will you take the risk? Will you use your Antropic subscription within external tools or will you just say, ah,

Cloud Code is good enough? Yeah, it is kind of a funny one because I think they're clearly trying to push you in the direction of using Cloud Code. Yeah. It's kind of annoying. I don't know. Aren't they making enough money already to just let people be? I feel like the There should be more freedom. But, yeah, I mean, I don't personally, I just don't because I've got enough stuff going on. I use other models via Pi, for example, if I need to. I find that more convenient. And I'll tell you why. I think with Anthropic, with OpenAI, clearly their main priority is to use their own models, right, and to kind of coordinate everything. and I think it's just easier to use other models in other ways. But yeah, I prefer to just match the harness to the model basically and simply put. Harness to the model I think is very important and something that folks miss.

Cloud code and cloud definitely go hand in hand. Again, Muse Wander and Muse probably go hand in hand as well. Codex is also very good with Astra and less so in other places. And that's why it's for agents like Hermes, it's some time to catch up. IPOs were called in November. That's the thing that I missed. So, you know, watch out for that. And there's a rumor, and we don't love rumors here, but we will say it, that Fable 5.5 showed up. And I think I'm, like, I'm getting tired of rumors. So I'm going to do this for just one second. This is not planned. But hey, Antropic is going to release Fable 6 also. And then Opus 6 is going to come out. And then Opus 6.1 or 6.5 is going to come out. After that, it's going to be even better. And then Opus 7. So if you heard it here first, Opus 7 is coming. We don't know when or how. But I'm getting tired of this. Like, oh, this model is coming. And this is getting just a lot of...

Okay, when we'll get it? We'll get it. I don't understand why the X algorithm for all of us is so fine-tuned. on just like trying to snipe the news of something that we all know is on a continuous improvement loop all the time. Can you just tell me if that's interesting to you when you see somebody and just like, oh, I've seen this leak or whatever. We know the models are going to get better. We know it's very important for these labs to tell you how better they're going to get. It was especially important pre-IPO. I'm interested in testing them out, playing with them, hearing what Peter says, with Nistin and Wolfram and Yamse rather than just knowing the news. But, given all that, somebody did spot Frabel 5.5. So I'm very excited to see how that's going to go. Comments? Did you guys hear my pushback? Do you feel like it's important to snipe and do breaking news, etc. at this age? Or are we getting a little bit tired of models getting bitter, numbers got better? I want to see them around a full factory or something.

That rumor, I want to see a whole bunch of humanoids running whatever model that, yeah yeah, the models are going to get better of course, there's going to be all opus 6, I think Anthropik's going to not allow everyone to just stay on top of the benchmarks they always have something in the back end but yeah, rumors now are more acceptable for like actual humanoids running a factory running a hospital or something. We've got to leave the rumors for more like actual physical stuff. Yeah, I agree. I'm not sure like everybody else's common sense. Yeah. Wolfram, from an evaluation standpoint, do you think that sniping the news and telling you this model is coming is still important or are we like past that? I mean, right now, the most important thing for most people using agents

is to have a way to use a subscription. So if the new model comes out and it's API only or it's so expensive that you run out of your tokens as quickly and you can't really use it, it's not as useful as a model that is just smart enough. And I think we are reaching a point where a model is smart enough for most users and the much higher intelligence is only for bigger projects and stuff that you as a user of a personal agent are not even needing for book me a flight or plan my vacation and stuff like that. I think there's a limit to what we really need and everything beyond that is on a company level and not on a personal level anymore. I completely disagree. I think we're going to look back at this a year later going to be like we were so stupid and naive. I think it's going to be amazing. I get your point. I think it's kind of kind of tedious, you know, this number is going to go up. But I think the fact that we shouldn't take it for granted.

I think OpenAid and Anthropic are the two labs that actually ship models very regularly. But we haven't seen a full pro model from Google for, what, eight, ten months? I haven't counted. So they're only shipping Flash models. So I think I find it perfectly reasonable For people not to pay attention to rumors But I think there is information in it When companies actually ship So let's say if there is a scenario That Anthropic doesn't ship 5.5 Fable for another two months I think that's also kind of newsworthy Like what happened there So yeah So I think it can kind of Go both ways Which I don't think you have to have rumors for But I still get a little bit of information out of it. So, yeah, we'll see. I haven't counted how many weeks OpenAI, when did the first kind of had Astra released. It's probably what, like a month ago, six weeks ago, something like that?

Astra was around a month ago, five weeks ago. So, yeah, you'd expect them to... I'm going to leave 6.1 Sol aside, because I think that was maybe out of cycle a little bit, but the next Astra, you'd think that's probably another month away, right, so I think but if it's two months away, then I think we'll get some information out of it, because it's, like, they're probably spending more time I know, aligning the model or something like that, so, yeah so, you know, I'm very pro-rumor, but you know, I'm also a psycho, so, you know, you don't have to be. I think first of all, the standing opinions is, I talk to our fans in IRL, they love this when we disagree, we give, like, multiple sides of the story, and I think they actually do love this. So that's great, Peter. And I do think that you're giving an additional insight there where now the Antropic is going to be public in IPO. Whether or not they're going to release a new model actually will move the news, move markets, given how big Antropic is. That's a very interesting kind of thing. So maybe we're, after doing this for quite

a while, maybe we need to find the passion again to say, oh, this model's coming, yeah, let's go. But yeah, we'll definitely get there. So I think that's enough about Antropic. and given, let me see if I can make this connection here, given that Antropic is going to IPO likely on Wall Street and given that I am here very, very close to Wall Street and the reason I'm here is AI Engineer. Let's talk about AI Engineer in New York. Nice, yeah, okay. This week, AI Engineer in New York. This is the second New York AI Engineer conference. For those of you who are new to our show and not familiar with AI Engineer, Thursday I started as a Twitter space and then became a podcast and the first time that I ever told anyone else when they said, hey, what do you do? I told them I have a podcast. I was on my way to AI Engineer in 2023 to cover it. Since then, the conference has blown up and Twix, the co-host of Latent Space and the creator of AI Engineer category and the conference has been curating incredible folks throughout these conferences.

So we went live earlier this year twice. One time from Moscone West with Wolfram during the, in the middle of show AI Engineer and then Peter actually interviewed you and we chatted in London as well so we went live from London Wolfram as well and this time for some reason they decided not to do it on Thursday but we're here in New York after AI Engineer and what interests me about this specific conference is the zeitgeist is incredible because a lot of folks come there first of all to learn so that you know that the speaker is going to deliver something but also the hallway conversations I have a little bit of a debrief for you because I know that many people did not make it to New York. None of my co-hosts are here in New York right now. And so here's a little bit of a debrief. This specific conference, because it was adjacent to Wall Street, this focused on finance and AI. The big hedge funds were all here. Two Sigma, let me go find them. Capital One and just Bridgewater, Apollo, Cirodale, Bloomberg, like all of those folks.

And then OpenAI and Tropical were also here. The big theme from Latent Space, what they said was autonomy versus security. Now that the big companies are actually running agents. And when I say now that the company is running agents, to contrast a year ago, 2025 was the year of agents in quotes. And this light, guys, was agents don't work. this was before opus 4.5 this was before the kind of the november january revolution just scroll back just a year ago agents did not do work and now all these folks are saying oh shit agents do so much work that we need to like put guardrails on them and do like how do we how do we deal with autonomy and models crossing the thresholds from non-coding work to like making decisions there's now agents inside banks there's golden data sets there's evals there's a bunch of stuff that the that they didn't have to a year ago because a year ago agents did not work. Most of the AI in companies were software engineers trying to get lines of code out. Now there's like agentic things happening inside most of these ones.

Like two years ago, I think I posted something about I've never seen an agent actually do anything useful other than plenty of use buying crypto. And that was only two years ago. So just remember, everyone's opinion is always wrong at some point. Yeah, there was a tweet from me yesterday. I don't know what people are doing with agents. No, but also we're changing so fast. And then it's completely flipped one. Yeah. And this is what we're talking about when we talk about acceleration. You can't be looking at the current state of the world right now and say, hey, AI is this. You have to start. This is why we're talking about acceleration. This is why we're trying to bring in the news and the use cases because we kind of feel that this is happening. So all the banks are now thinking about this very strongly. Not only AI engineers are spending tokens. Banks are actually using agentic processes. And the favorite of many people, I did, like I said before, we did like a little Corvive event. You can see this on my hand. There's a liquid cooling IRL event.

I found it really, really funny. Muse helped me come up with it. We went to a cold plunge and we called it liquid cooling because Corvive data centers. And then I did like a little survey. shout out to the folks who came with me to our event. And then many people said that Max Kannat Alexander, he is, I believe he's a distinguished engineer in Capital One. He was the code simplicity guy and he did a bunch of Kubernetes stuff at Google. He's a fascinating dude to talk to. His talk was the highlight. He said, like he talked about Does code quality even matter? Given the background of this person, I think it's very, very important. Principal software engineer and open source back in Google. He led a bunch of the bug stuff in Google. He basically says the code quality matters in proportion to how durable your software is because AI is about to make a whole new kind of software way more common. Disposable software. Just write once, use it, and then just throw it away.

So when you have disposal software, does quality matter if it works? If you build a calculator or an app for yourself, party planner, does it matter? So this was one great show, and I recommend folks to tune into the AI Engineer livestream on YouTube, which you can watch if you want to The talks will get released The other hallway conversation was first of all about evals and whether or not agents are working But as I said in the beginning of the show the thing that was highlight for me which I do want to ask you guys about as well, is the disconnect between how advanced all those AI engineers are in use cases and harnesses and personal agents with the revolution of personal agents for normies. and how when I talk to some of these AI engineers, they're like, hey, do you use Instinct, Muse, Glockbot, or whatever, and I don't get an answer yes from them, and that's really something.

Because many of the engineers just build their own harnesses or use cloud code or codex, and why not use codex for some of these stuff? And so there's a big disconnect, which is very interesting to me specifically because how big and wide OpenClaw and the Harness Agent got, and those are harnesses you can tweak and play with. So I did notice this in the halls as well. And specifically, this is highlighted for me because I did talk, like Wolfram does, to every person in the world. There is not an engineer. Wolfram talks to stewardesses on planes to try to AI build them. So I did the same. I took a page out of your book and I talked with, there was the token billionaires launch. We told you about before. This AI engineer had this again. So if you are spending more than one billion tokens a week or something, you get this golden card. I think I have it. Hold on. Wolfram, there's a new card that you need to get. The token billions card kind of looks like this. And then this gives you access to this lounge. This lounge is kind of like 4 billionaires lounge.

There were a lot of service providers in that lounge that were not part of the conference, and I tried to talk with them. Here's the thing that I got from them, not from the engineers. All of them know about Muse because Muse is getting pushed down their throats in every Instagram post. Zuck is really going out of their way. They all know about news. Many of them didn't install it or didn't yet. Why it's good for them, didn't yet catch that. So those conversations were fascinating. Let's see what else. So, A-Engineer, if you haven't been there, the highlight for me, and I tell it to everyone also, if you're listening to the show, you're good enough to submit a talk proposal to an engineer. If you submit a proposal to an engineer, you get access, and Peter both can comment that this is true. You get access to this amazing speakers lounge where all the speakers that are hand-selected are top of the level of people. And you get also access to speakers dinner. And to me, the speakers dinner and the speakers lounge hangout

is the highest tier of folks who understand whatever word they say. You don't need to explain what harness is. You actually learn a lot. One example that I can shout out is I just were walking by and somebody somebody's tag said linear. I was like oh who's this dude? Oh he's the you know senior I don't remember the full I think he's the manager software for linear all of linear. Linear is a genetic forward task management software. And I had a bunch of feedback about linear. We just sat down just right there and talked about how I use linear with my agents and what they can do to improve. This is one of those conversations. So if you're listening to this but if you have anything to say I strongly, strongly recommend to apply for a conversation talk at AIE. Wolfram, Peter, any thoughts on that space? It's the highest level of networking you can have in the AI space because you are not talking to some other attendees of a conference. You are talking to the big shots who are creating the AI news we are reporting about.

So this is super, super amazing. And even if you don't become a speaker, I mean, just attending and going to the booth and talking to the people you meet there. There are all the people from Google and Entropic and OpenAI. They're all there. So it's the best way to connect to the makers and shakers of the AI space, definitely. Best conference I know. 100%. Peter, go ahead. Yeah, no, I would say also everyone can apply, but do take time to make a good application because I can't remember what the stats were, but I think Swix was saying, not for this one, but for the one in London, I think it was like the acceptance rate was very low. So I would say put effort in. Put effort. Yeah, the reason why it's a good networking opportunity is because it is a bunch of good people. So, yeah, it's worth, you know, don't just LLM generate your summary and proposal.

So, yeah, take your time. Here's an additional insight that I got, and I think we've talked about this on the show, but obviously this is on our mindset, right? So, AI engineer in London, the Zygaist, we're talking about should we still read code? We don't write code anymore. Should we still read code? I talked about the ZL continuum on my way back. I think it's clear now that barely, barely anybody's reading code now. We're, like, stressed how to even test it and make it valuable. The zeitgeist here was the end of attention. And how do we scale from managing one or two agents directly by steering them and seeing outputs to running multiple, multiple tens or hundreds even agents? And how do we have the attention to attend to all of that and make decisions? Or whether or not we just let this swarm run and deal with this. And I think it's very interesting. I talk to many, many people. There's not a lot of solutions. Ryan Carson, our, by the way, shout out Ryan. We hung out doing this whole AI engineer. Our previous, our co-hosts, me and Ryan, we were talking to a bunch of people.

Ryan gave a great talk about this as well. And he's missing a control plane. He's missing like a one central place where he can like see all of the stuff that's working. Another thing that was in the zeitgeist is the move to the cloud. Many people are just getting limited by their local environments. You can see this from the Frontier Labs as well. Codex Cloud just launched in-depth. they renewed, launched Cloud. You can now hand off tasks to the cloud as well. And they even give you, like I think you said, they give you some budgets, like $100, to try the cloud thing. Obviously, Devon and Cursor, they have cloud agents. And folks are getting to the limit of their environment, not only because it doesn't run 24-7, you need to run Caffeinate to keep it running, but also because the local environments are running out of memory. And, Peter, I think you mentioned this as well. You're coding so much in your things that the agents just killed your computers, you got a different box. And when you spin those up in sandboxes on the cloud, you kind of don't have the precision. You can't go wide. So we're getting to a point where not only are we using more billions tokens more, my case, definitely Peter's case,

definitely Wolfram's case, all of us here, probably like 10x for the past few months, and the Zyguist and the engineer as well. We also need a place for that old thing to happen, all the tools to happen, and that's not happening on local environment. I heard the engineer. All right, folks. Next up, we have Omar Sansevero and Genevieve Husk, I believe. They're joining us now. Just before that, as always, given the show is sponsored by one sports only, CoreWeave, I would love to tell you about some of the stuff that we have and then we're going to talk with Genevieve and Omar about finally we got the new big Gemini back and it's so exciting to hear about this as well. Let's go to the Suix Buzz, and just right after that, we'll have our guests. Four weeks. From four weeks.

I think that it's time to sunset that transition and come up with a new transition, Wolfram, because as of two weeks ago, CodeWeb Forge is now the home for the weight and biases as a tool and also a bunch of other tools, specifically AgentLens. AgentLens launches this week. And the highlight here for me in AgentLens, Wolfram, tell me if you're also on this, is that Cloud Code now supports OpenTelemetryExport. And then now you can use our tools to actually look at the agent run times that you have. and analyze if it did the work correctly. So, would love to just let you know that you can now use that. You can sign up on forge.coreweave.com. The highlight for me from Flory Connected two weeks ago was that given the previous life at Coreweave, where you had to go talk to a salesperson,

register for long contracts and sign up, Forge, for the first time, is our register as a regular SaaS offering. And then you get access to a bunch of tools from our core of Agenda Cloud. Tools like distillation as well. We talked about model distillation a lot on the show. But now, via the open pipe folks, which are great folks, friends of the show, you can get a model distillation for yourself. You don't have to set that all up. It's all part of forge.corby.com. Wolfram, did you set up your cloud code with Agenda Lens yet? Are you sending all that? At the moment, I don't. It's all local right now. But it makes definitely sense to integrate all of this. And the integration is what Forge is all about, to have integrations between all the different tools and the whole AI loop from the evaluations to the inference it's running on to the curation of the data sets. It's all connected. And that is a new thing, I think. It's not just Weave or the individual product where you have to export and import stuff.

You can jump between them. I want to call out yesterday. Yesterday we said at the event that we did, we sat down and we talked about some stuff. And obviously I was like, okay, this is a core response at dinner. Let me at least tell you about their offerings. And folks who use Ways and Bias didn't even know we have ARIA. And somebody's like, what is ARIA? So ARIA is an agent assistant, agentic research assistant, basically auto research from Karpati personified in the product. So if you are using Wits Advice for anything, people didn't even know that we have this offering. So we definitely need to work a bit about externalizing our offerings. But if you have seen auto-research as a concept from Andre Carpathy in his suite, it's now in GA. You can just use it inside Forge on your training runs. All right, folks, I think this is enough about Forge. Now we're going to move on because we have not quite breaking news, but definitely news from the frontier of AI with two great guests here. And let's go talk to them. Hello.

Hi. Omar and Genevieve are joining us from DeepMind, from Google DeepMind, and you guys have news to share. And can I just say it's about time? It's about time. We talked on the show before. And, folks, it's about time for a big model. So you guys launched Gemini for Argonne. But I would love for you to tell us about the launch, about what's new, about what are the use cases. I'm all ears. It's out for everyone. It's number one on Arena. And it helped make Gemma three times faster. Omar, that's kind of your corner as well, right? I would love to hear about the use cases. So please give us a highlight, and then we're going to dive into use cases. And generally, great to meet you. Thank you so much for coming. Likewise. Thanks for having me. All right. Omar, let's go. I can go first. So, yeah, I mean, Gemini for Arcon is our latest model. PLDR, it's a great model. You have probably heard many things about it. My only recommendation is to go try it out, and I think many of you will be pleasantly surprised. Yeah, the model can be used for many things.

We've been using it internally. As you said, we've been doing optimizations for Gemma. We have been able to achieve over 100,000 tokens per second with the E2B, so our smallest model. The model has a 1 million unified contest, so pretty much you can pass very long videos but you can also have very long outputs so that is quite exciting and yeah we have been very excited to see what the community has been building with the models what do you think, Yeneti? Yeah, we've had so much fun building with Argon it had been a while since we launched a new model and so most people will know that this is the first model that came out in our Gemini 4 series by far our largest train run ever and so Argon I think is the first actually quite early checkpoint that already is doing really well but we're also super excited about everything that we can build next now. And so Argonne is already training our future generations of models. I think we've really enjoyed the fact that Argonne can do many more long-running tests. And so now everyone is shipping a bunch of stuff overnight, and you wake up and you just wake up to really fun things that Argonne has been cooking in the background.

And so it's much better at 3D, much better at creativity. Personally, the writing is really pleasant. And so in general, it's just been super fun to build things with a model. And again, we think there's many more great things to come from us in the next few weeks. Yeah, go ahead Omar, please. Just as usual, we have been working with all of the open source ecosystem to make sure that you can use the model regardless of what's your favorite tool. So we have been working to make sure that this model works well with OpenClub, with Hermes, with all of the tools, Blanchain, Verso, and so on. So super excited to also collaborate with all of this ecosystem to get the model out there. I want to say the release definitely is a wide surface release It's available pretty much everywhere. And also in your own Gemini API and AI Studio. Shout out to AI Studio folks for always bringing us in to the fold. Gemini app only for Pro and Ultra. Is that correct? So folks who are subscribing to Gemini app on the Pro and Ultra tier are giving this. And on to Gravity as well. $2 in and $10 out.

This is introductory pricing. Exactly. Yeah. And then what happens then? Extension, new model, another interactive pricing. Breaking news for next month. All right. And then based on what it says now, then it doubles. One million token context. And I think it's about multimodality. Google and Gemini model specifically were always kind of like the front leading in multimodal, including like watching the video that you send to it. One million token is a great thing for that. Yeah, you'll see that the model is very good at understanding visuals, the space, and it's a very good understanding of the world and physics. So many people kind of forget that models are not just for coding, like models are to understand the world and the world is highly multimodal, multilingual as well. And in general for Gemini, we make sure that this model works well across all of these kinds of use cases. That's incredible. Let's talk about Vision. RoboFlow called this model is like number one in Vision and DetectOCR stuff as well. Very, very, very strong. Any specific demos around Vision that you guys want to show us?

Any cool things? Let's start with the ones that I have ready and then I don't have anything else in the background. The ones that I pick for today are mostly very cool looking. But yeah, it's great to see the vision capabilities always if you're building with a model and it can check its own work, if it can record videos and kind of see what's going on then that is super, super helpful. So even for some of our 3D stuff that we're now doing, it's just so pleasant that Gemini can watch its own videos back. Even things like motion graphics is very helpful because it can look at a video, understand the style, and so I made a trailer about Oregon earlier this week that took me just maybe 30 minutes and it came out really good. But yeah, in terms of the demos that we've been cooking, the first one that I made was Desk Racer. And so first of all, if you go into Google AI Studio and you click on the gallery, you can find a bunch of stuff that we've made with Oregon. And so Desk Racer came from our multi-modality where I wanted to build a racing game on top of my actual desk. And so the microphone that you see here and then this cute little cat just gave Gemini some reference images and said,

okay, you just built me a racetrack. And what's fun about this is, I mean, it's a long-running task to understand where to go around the objects, to make it look good, to sort of plan the lighting and everything. And so now you can play this. And then Argon even prompted our Omni video model to extract sounds like engine sounds from different race cards, listen to all the sounds, found the sort of audio that was able to loop and then stitch all of that together. And so, like, yes, it looks good, but it's also cool to think about the rest that's happening in the background to, in the end, come up with a very shiny result. And so this one I really like because it's where I spend all day every day, but it's fun to see Argon do such a good job and so build all these little other camera views in it. And so this you can play in AI Studio. There's a timed mode where you just drive three laps. And so that's pretty fun. and then this one could I ask for our audience how do you approach building this demo

is this all in the AI studio are you using any other harnesses are you using the internal tools can you tell us how to get building this where Argon can download the files then using Omni generating things and extracting where is all the two happening yeah that's a great question and actually I wrote a guide about that earlier this week and so in essence started with the reference images use grill me So, like, ask me questions about how I should plan this project and then just use a big goal command and said, you know, I just want you to spawn a bunch of subagents, work on it however long you need. Some things that helped were, A, using adversary subagents. And so there was always one subagent whose job it was to look at the outputs, grade them for accuracy, for lighting, come back with feedback, and then give a rating out of 10 and then keep going until the rating was very, very high. So that, first of all, was really helpful. And then secondly, we've all really been enjoying using Nano Banana to make 3D even better. And so in this example, for instance, I'll turn off the audio, but this is a demo that looks really, really good.

It's like a pagoda demo. And so if you have a forer render and you give it to Nano Banana and you say, make this look really good and realistic, and then you pass that image back to Argonne and say, do whatever you need to do to make it look like this. I'm now using that in all of my 3D projects. And so the lighting that you saw in Desk Racer, the neon lighting was designed by Nana Banana because at first it looked very bland. And so this, I think, is a really nice way to interact with any model. Just that image inspiration is a really good visual reference. Oh, that's incredible. Yeah, I did write a guide about this. Sorry. No, no, it's so good. This is like a definitely great tip for our audience as well. Use visual references because the model is very good at multimodality from other models for this. That was incredible. Amazing. So that's another one. So that's great. Yeah, so this one is pretty cool because, like, yes, it's fun to do things with Blender and 3D.js. But in this case, my colleague actually asked Argonne to create this 3D scene completely from scratch. So zero libraries, zero external things being pulled in.

And what's cool about this is if you think about how a model needs to do that, it needs to think about all the light, all the ways that objects might interact with each other. And so the light bouncing off the wooden floor here needs to be refractored in a certain way, the light shining through the glass needs to exist in a certain way. And so if you think about the amount of real world understanding and then also the quality of code generation that you need to make happen to make a project like this, that we also thought was very impressive. And then the other thing that was fun about this video, I'll spare you the audio, but Argon also narrated it by itself, picked the music with Lyria. And so, again, the idea that you can build something cool, give it back to Argon and then say, now make this into a cool video, So that all just ends up working. And I think in our previous models, we probably would have had to babysit that a lot more and give much more guidance on how things should look. And so another one that I think is just a cool example of what you can do. I want to go back to here with Omar as well, and maybe both of you can come in. The token output changed as well.

Like Argonne generates outputs significantly larger. Could you tell us how this affects demos like this on other stuff? Omar, I would love to hear from you as well. how does that improve people's kind of like out results specifically because if you think about kind of like evaluating models a lot of the times the folks are focusing on the Pareto frontier corner where reasoning used to take a long long time and just like with more reasoning you get better results and then there was a little bit of a movement towards improving that so that the model will give great result but still not reason as much not waste as many tokens but this is not about that this is about the model being able to generate long, long things so could you tell us about the outputs a little bit? I mean this is mostly for long horizon or scientific projects where you want the model to do lots and lots of reasoning and that's pretty much the kind of task where you would need that kind of context in the day-to-day most likely you do not need that level of output

for input maybe, right? like if you want to pass a very long video or lots of PDFs you may need to do that. But this is mostly for scientific or long horizon. That was awesome. And generally, did it affect any of the work that you did? No, I agree with Omar that in most cases, yes, you could write one or several books in one prompt, but that doesn't end up being super useful. So to us, it's mostly on the very long reasoning. One project that I have been working on in the background, but it needs a bit more work, is one 3D project where I'm modeling 55 cities in one go. And so whenever it sends off to sub-agents, then they all start building many, many cities at the same time. But again, in most engineering applications, it's still much better to just do things in slightly smaller batches. But if you have to think about a very long, difficult math proof for a long time, then you don't want the output window to be constricted by the fact that a model already thought for a long time. And so that's where we see most application. I think it is a very new thing also for us. And so there's also some complexity with how this works, even in things like external harnesses.

But in general, the idea that the scientific community can use this is very interesting to us. Yeah, that is very exciting for sure. So we talked about the visual kind of understanding in number one. Omar and Jennifer, I think you mentioned this, that now Argon is working on the next versions of Argon. That suspiciously sounds like RSI. And Omar, you're talking about Argon training JEMA models, and that's like parent-teacher, not necessarily distillation, but it's also in the process of training them. Give us a little bit more. I think people do want to know whether or not these models can help train other models and how that works. So we'll get on Gemma work. How much of this was autonomous or how much of this is intervening? Could you guys talk to us about model improving other models in the future? Yeah, I think just as we think about AI for science, AI for math, AI for biology, AI for quantum computing, we can think as well about AI for AI, right? And I do find this a very exciting area. This is something that we as an ecosystem have been doing for some time. As these models are getting significantly better, it's quite exciting to think about ideas.

For example, in the context of Yemma, what Argon did was optimize BLM, which is one of the most popular open source libraries for server-side inference. And it is a bunch of BLM optimizations. It's found a bunch of areas where we could improve things. similarly you could see how models could be held on co-designing training code or co-designing architectures so yeah uh do you have anything any last comments before we let you guys know i think yeah the sentiment in the market about gemini has has varied i would say please give argon a shot it to us feels like a very different model from uh the models that we've launched up to now and so i hope everyone gives it a creative shot because there's so much i think that you you can do with it. For us, I think it's the start of a new era without sounding too commercial. But yeah, I think this is a very exciting first step to what hopefully is going to be a couple of fun weeks and months. We've been waiting and we're now very excited to have it back and

to see that Google is back. So folks, thank you so much for joining us. Generally, nice to have you on the podcast. Feel free to come back with awesome demos, which we always love. Omar, as always, can do the part for a long, long time. Thank you guys for joining us and congrats on the release and we know that you have a lot of stuff to catch up on during the this week. We'll see you here next time. Cheers. Thank you so much. Alright, folks. From one conversation to another, now we're going to chat. Now we're moving forward, and I want to introduce you to Nathan Benyash, and let me bring Peter back on. I followed you for the longest time, but I think we've never talked like this on video, and I followed the state-of-the-art report, so let me just do this reading. Folks, the state-of-the-art report turns nine this year. Is that real? Nine years. I feel like I've been doing this for three and a half years and this is a long time. Nine years. The word AI was not familiar to many folks back nine years ago. Yeah, it was the era of like AI-ML and data science.

Yeah, and this year is 20 to 244 slides, a scorecard on last year's predictions. Would love to hear about last year's predictions and whether or not we can hit them and nine new predictions, including AGI 27, Nathan Benyash, the founder of Ash Street Capital, is with us, the author of State of the Art Report. Nathan, welcome. Thanks for having me. Peter also joined us for this interview. Peter, thanks so much for making this connection. I want to talk about some of the highlights and favorite numbers. I have cloud-led 27% of measured model R&D on topic in August. We just talked with folks from DeepMind about Argon and RSI, whatever. whatever. Would love to hear from you Nathan how this kind of like shapes, takes shape between last year and this year. People do talk about RSI being a thing. People do talk about models, whether or not we're going to hit this like thing. What are you seeing from there? What are the, what are the reports says about RSI and continued improvement by the models?

Yeah. Yeah. I would say like the, the major story in the last 12 months has been, has been RSI and it's been, you know, super intelligence, moving from AGI to super intelligence. And And then increasing evidence at the frontier that shows we're kind of making progress towards this idea of AI, building AI. I think the two experiments I found most interesting there, one of them you cited, which is just the general stat of how anthropics like R&D team when they're doing core R&D work on the model, so 26% of the directions in research are being guided by Claude. that's up from basically zero a couple of months ago. Not to oppose this, but I think the code generation step that I remember, maybe you have an updated one, but as far as I remember, like 80% or 85% of code is generated by cloud, maybe not directed. Yeah, yeah, yeah. So, yeah, that was like a few months earlier. You could see like a chart of like code commits that were basically accepted by engineers as a function of like how models have been changing,

and it was sort of like flat for a while, and then like the mythos class models came, and then now it's like 8x the average. And you see a similar stat at OpenAI. And then the other data point that I like that we made a slide on is this like experimental setup where a human researcher engages on a research path, kind of goes through like several leads down like the node of that research path. And at some point the path works, and then one of the fork is a model that continues the research, and then the other fork is a human that continues the research. and at the end of both of those leaves, human rates, which one was better? And you saw that the old school models were definitely not better. I think 10% of the time humans preferred them. But then the mythos class model, it was like 60% of the time humans said the mythos suggestions were better than humans. So I think that's also interesting and it gets to this point about scientific taste, which I think is on the tip of everybody's tongue

like no sign intended. Scientific taste. Yeah, just the idea for those who are less familiar. I did a PhD in biology and what I remember were the great papers were the ones where the grad student or the postdoc would be running a couple of experiments and every week going to their supervisor and showing the experiments the supervisor who would look at the work and, you know, kind of mull it over and say, like, okay, the next best thing you should do for the story is this. And, like, the best PIs would be the ones who would suggest experiments or ways to interpret the experiments that you did and the results you got in a way that you thought was, like, novel and interesting and elegant. And then the bad PIs, like, do that very poorly. And then you see this in the result of a paper where, like, the story kind of reads beautifully. It's very logical. like each experiment adds additional signal that the previous one didn't have. And I think that's like what people are trying to encompass in what they call taste.

So can a model like approximate that? Yeah, I think there was one more from Prime Interact, I think, where it was the non-GPT, right? Yeah. Where it sort of, did it almost get to the human level, right? Yeah. But I think there was something that was still narrow. the scope of the work that it was done. Yeah, yeah. There's a, like, to get into this more, there was a study from OpenAI in the report where they looked at trying to categorize, like, the token usage of their engineers and researchers. And what they did was, like, identify some tokens that were about helping decide the research direction, what experiments should you run to test that research, and then other kind of category of tokens, which is like implementing those ideas, like designing the infrastructure, engineering the infrastructure, doing the data pipelines and doing the data analysis and just writing the results of the study. And the vast majority of the token usage was on just like the raw engineering

or just like get the thing working and then writing about it. And very, very small amount of tokens was on what research should I do in the first place and how should I test it. And so mine, I think there's different ways you can interpret this, but one of the interpretations to me is there's still so much more to go on improving the use of models for ideation in brand new areas You also gave yourself a scorecard Can we talk about that from last year predictions and misses What stuff did you hit correctly? Or like the report hit correctly, and I'm assuming that's you as well. What stuff did you miss? Yeah. We'd love to hear. Yeah, like the stuff I got right was that there'd be a pushback from big labs and new labs into open source. and this is in the U.S. and the West. And the idea with that is it was very clearly part of the national AI strategy from the Trump administration to sort of regaining supremacy of American AI and open source as a way of doing that.

And very soon after we published that report, like reflection AI, sort of went from the coding agent work that they were doing towards saying, we're going to publish an open weights model and then be supercharged, all that. by basically, I think it's rumored, I'm not sure if it's quite there yet, but by saying that NVDA is going to maybe get reflection in there. After getting Hug and Face in there also this year. So yeah, you're kind of like... And full side. And full side and other potential ones. And so you're bang on with that as well. Yeah, so that one, and then the second one was on NIMBYism for data centers. The idea, like, not in my backyard. like I don't want to have a data center next to my garden, that this would be a real problem in the, you know, hampering the build-out data centers, but also influencing part of the midterms, which we'll see in the next couple of weeks. We substantiated this out with a survey that we found where Americans say that they might prefer living near a nuclear power plant than a data center,

which is quite wild. Wild, yeah. We covered this on the show as well, and we still can't believe it. It's true. It's just the amount of things that are going on and just people on the street talking about just AI boiling the planet. Yeah, yeah, yeah. And it's not just like in the U.S. It's certainly happening in Europe and in Korea and other places where, you know, there's been like a small proportion of the population that's benefited enormously from AI and many people who haven't. And, you know, there's some social issues with that. I think the last one that I got right was the Chinese labs would overtake U.S. ones in some major leaderboards. And I think it depends on which timeline you actually look at. And Peter probably knows this by heart. But certainly, you know, Kimi and GLM and others have done really, really well. Yeah. What about some of the misses? Yeah. Yeah, yeah. Well, there's been so much talk about, like, AI sovereignty, which could be interpreted in different ways. But broadly speaking, the one of I as a country want to have independence

independence or control over my own destiny when it comes to building models and training them, serving them, having the infrastructure for it and the energy, et cetera. And my bet was there would be this concept of like, you know, AI neutrality. And that kind of mirrors like sort of mutual defense agreements where a country is like too small to be able to afford, you know, its own military or its own defenses in the same way it's too small to be able to build its own data centers and have people build models, et cetera. So it just forms like a strategic defense agreement or AI provision agreement with a bigger neighbor or an ally that says, you know, I will promise you to give you all the AI in exchange for something else. I haven't quite seen something like that yet. There's been a bit of a move with the U.S. tax silica policy that tries to bring a collection of countries that have access to core parts of the chip supply chain.

So it's like minerals or fabs or lithography or whatever. But overall, the AI neutrality thing hasn't really taken on just yet. No, no, yeah. And overall, I'd say the predictions are directionally well. like this as a guy on LinkedIn who like looked at the last like nine years and then tried to see like over that nine years, which proportion of all predictions eventually happened. It was, I think like over two thirds. And so sometimes it's just a question of timing more than just like, this is ludicrous. Well, that's what I like about your prediction is that I think it's quite easy to make predictions that you're going to mail like 90% correct. You just kind of predict roughly what just happened. Oh yeah. It's going to make more money next year. You know, it's like, okay, That's not very interesting. So, yeah, I find it interesting. You've got a hit rate of, like, 50. Yeah, yeah. So is it, like, how, is it difficult to come up with the predictions? Are there some that you kind of decided against, I don't know, too wild that you didn't start making this year?

Yeah. There's one I couldn't quite formulate around, one is, like, cybersecurity and, like, can AI somehow disrupt encryption on the internet as it might be accelerated by the fact that AI solves math. So I don't quite know how to basically I just ran out of time for thinking about how to formulate it properly to be able to defend it. Let me ask you there. The question will formulate this for you. Last week we talked about the math manuscripts dropped and I don't know if you noticed this or started a conversation about this, but the numbering there is not sequential. Between 44 and 46, there's a missing 45, and there's a few of those. And the rumor mill says that, hey, those are the manuscripts that talk about analytic curve cryptography and that being a lot of math, and supposedly they're not able to release some of this because

the government kind of steps in and says, hey, anything regarding that, do not. What's taken that? I think it's believable. In the same way that that spy thing two days ago, that's the tip of the iceberg. Can we just talk about this for a second? Can you say what you mean? Because many folks have maybe missed this piece of news. It's become a bit of an internet meme mostly because of the photograph of the guy appearing behind the TV. But from what I read, there was an individual was employed at OpenAI who just had a lot of was basically found to have planted a listening device behind a TV in one of the meeting rooms in the OpenAI headquarters that he tried to deduce was ones where key executives would have meetings because he had somebody like poll PA to ask which meeting room would be relevant. And then there was a device that was back in the TV and it was wrapped in torrent of the paper. and apparently he left some trails because he was a bit amateurish about it

and then they found him and reported to the FBI. That's the one they caught, basically. Yeah, that's what they disclosed. That, I guess, is directionally towards the prediction I had of someone going to exfiltrate model weights. I said it would be AI-driven, but it turns out humans... When you predicted that, were you talking about someone someone being human or someone, a model being like, you know, copying its own weights out? Where were you there? Yeah, I thought it was more going to be the model, like more sci-fi than actually just the prosaic, like a guy walking in and downloading it. Yeah, it's going to be very interesting if we were even able to tell if the model is smart enough to do that. Will it be not smart enough to not hide its listening devices in a bunch of newspaper behind the TV? Can I ask you, say, you speak to all kinds of people and we're a little bit in the AI engineer kind of bubble.

Like what are the sort of things that people ask you about? Because you've got different parts of the report about even going into politics and things like that. What do you hear people caring about? There's a segment of people who really care about what we discussed, like the RSI, like rapidly kind of accelerating progress, like what does that mean for the rest of us basically? There's some who, you know, are just like increasingly pessimistic over like software companies still because, you know, now like if you kind of believe the RSI is coming and basically the quality of the software you can produce or the work you can produce is like rate limited by what model you have and how much money you're willing to spend on it, then there's certain people that will just, like, organizations that will accelerate way faster than others, and if you're not part of that, like, what do you do?

There's always, like, chatter every two weeks or something, and I think now we're in one of those negative cycles around, like, is there too much compute, circular financing, like, how is it financed, like, is the debt going to be able to be serviced, and, you know, is there bloodbath in your clouds in the last, like, two, three days because we're just in that part of the cycle? there's always debate around will the labs monetize how will they monetize or these funky financing structures with multiple stages different prices all within a couple minutes of each other is that like a sign of I don't know just like rabid demand not very much thinking how about normally I have to ask you Nathan how about like just regular folks on the street and how they do AI Do you track that instead of OEI? Do you talk to them? How do you see that opinion shapes? So would you talk about the change in the public opinion about water use, et cetera? What about the kind of the use case for OEI? What do you see there? What do you track there? The rise of personal agents is now upon us,

clockbots and news and all of those. Do you consider those in the report? Yeah, but that's still like in the small minority of tech people who know what it is. I would still bet that 90% of people are using a model to maybe at best read their email or answer to their email or ask questions in the same way they would ask Google, but they probably never click the work button. What does that even mean? Do I want to open a consumer app and work? Not really. Most people don't even know. You can swipe left, open codex, run around your machine, and talk to your machine and make it do work for you. I think that's so far away, which is one of the reasons why I could just start saying, like, I think these big labs should open, like, a genius bar, like an actual physical store, and just, like, start walking, you know, regular people through the magical things you can do with AI. This is kind of also the geek squad genius bar, right?

Yeah, exactly. They need hand-holding and imagination unlock. Yeah. That's what we're trying to do on the show. the show does cater to AI engineers and so if you are still listening to this you are in a very few, I don't know if you saw the Karpati tweet as well about he kind of talks about this a little bit exactly, when he segments the world we're still very much like that and the argument that that gap expands even further if capabilities get better and RSI happens and all that stuff, you can see it a bit in the ramp data that we show with the 70,000 US customers that they have and tracking how much money they spend on models. It's like, you know, the average spends like 12 bucks per employee per month, and the top 1% is like 7,500, roughly speaking. So it's like 580 times difference. Between the potential permanent underclass, if this continues, yes. And the top spenders, the billion token ones. Nathan, thank you so much for joining.

The thing that I do want to ask you, because you've been doing this for a long time. Again, from AI ML days to then generative AI to very small context windows, like huge ones. We've been doing this for three years and it feels like an incredible time. You've been doing this for much, much longer. Your personal beliefs and whether or not your PDOOM stuff, I don't like the PDOOM, I don't want to talk about the bubble stuff. I would love to just hear from you. How do you see your views change throughout these years? if you can talk about this for a little bit briefly. Yeah. I think the major... Also, my PDU is very small. I'm not a big believer in X-Risk. I do think along that angle that cybersecurity is very precarious and should be taken insanely seriously. But I think that's different than existential risk of destroying humanity. I think the major change I've had in the last like two, three years is I generally believe that you use the right model for the task in the same way that you don't drive your Ferrari to go to like a coffee shop or maybe you do to show off or something, but it's like not the most reasonable thing to do, just like walk.

but I think there's like a weird behavior with like frontier intelligence if you have like 200 bucks a month to spend and you value the quality of your work do you really want to like take the risk of putting the slider down to low and like using a cheaper model on the chance that like the quality of the work is worse so while I could switch for like emails or whatever I just don't I'm just like on astra high like who cares you're answering emails on astra high yeah Then yes, from the Airsoft Capital. Not all my emails, but the ones where you have to think about it. Yeah. Why would you take the risk using a shittier system? Or potentially a shittier system? So, yeah, there's that. And then will we ever make money on Frontier Intelligence? And I think it's starting to show that we will because the value of the task that they're doing is increasingly high. and just the more people from all sorts of different parts of the technology world who come into working in AI bring all of their knowledge and optimizations to the fore.

And just, you know, human ingenuity plus lots of AI equals more progress. So I'm, like, quite bullish on, like, margins improving. You know, I don't know what Anthropik is adjusting when they report their adjusted numbers. But, you know, it's probably not, like, community-adjusted EBITDA like we work. Like, I think it's a serious company. So I've changed my mind on that. Like, I think you can really make money on Frontier AI. And I'm increasingly convinced that it's the open source versus, like, Frontier Lab. It's like iOS versus Android. It's sort of cool for a while to play around with your Android and mod everything. And some people want to do that. But at some point, you just want product that auto-update is, like, reliable, connects with everything. And it looks nice. All right. Those are great stuff. Folks, Nathan Benius, thank you so much for joining us on Thursday. We'll always welcome you back after the next one as well. Are you planning to keep going? Yeah, yeah, yeah. We'll keep going. If AGI 27 happens, I'll be on the 10th year anniversary,

and then I might be able to stop. All right. In the meantime, you can find everything online. It's all free at stateof.ai. It's at stateof.ai. Yes. That's a great URL. Nathan, thank you so much for joining us. Yeah, thanks, guys. Bye-bye. That was super cool. We don't often get somebody here that does this longer than we do. And LBJ joined us as well. This was great. Nathan, Peter, thank you so much. What's your take on that? The very cool thing about this is that there's obviously, I find this really helpful to keep up with this stuff every week, but there's something really valuable, I think, that to just look back at the year and you're just like, oh I completely forgotten about this or I didn't actually realize there were papers there that maybe didn't quite make the splash this week so we skipped it because you know there's too much to cover but actually they turn out to be quite important so I really value his work and I've been reading his stuff for years now I can't remember when it's the first one that I read but it's just

so useful to just look at it and and just read the stuff so I really recommend just book I don't an hour, two hours in your diary and just scroll for this. You don't have to read every single thing, but just scroll for this. I know it's like 200 future slides. You can do it in an hour quite easily. Don't ask your agent to read this. Don't ask your agent to read this. Just read it. Read it. Put some effort in. No, it's worth it. So I will just ask, Alijer, thank you for joining us. Niston, how's your 100K city going? Are you still AI build? And Peter, what new and interesting did you start using or building this week? just whoever wants to jump in let's just chat hell yeah there's the I got some some crazy stuff coming up for release it's yeah the entire decompilation of games is enabling full out simulations can we talk about this so LJ we mentioned this like last time

that people are putting things in other games and now there's a whole decompilation thing happening somebody told me somebody decompiled or reverse engineer the airdrop protocol? What the fuck? Guys, it is crazy. It's like a renaissance happening right now, right? Yes. All right. Here's what's happened. The entire Internet, the last two months, has just gone completely nuts by using, like, the NSA Ghidra protocol, which allows you to, like, reverse engineer any piece of software. And then there was also the law in 1996 from the Phoenix BIOS, or I might be wrong on that, that allows you to actually provide, like, visually functional pieces of software as long as it does not violate any trademark. So you see a lot of these things called Photograph, Gridcraft, and stuff, which are, like, Photoshop or Excel builds that are done in Rust. Clean room builds, yes. Yes, so those are mainly done in Rust. There were also crazier builds on the game side.

So if you think that is cool and it is pretty cool, on the game side, people were able to reverse Grand Theft Auto V and they were able to run it on web GPU. And I'm not sure if I can show it and stuff because that one, I don't know. It has all their brand logos and stuff. But I tried it on the phone. I tried it on the oldest M1 machine that I have. and it ran what used to be a 100 gig game that you had to download. It just ran it perfectly at 2K resolutions via web GPU. The models have just gotten crazy good at the optimizations that come with web GPU because now you can do, instead of games being float 32, you can do float 16. So you have full-out A-tier games like PlayStation 4, PlayStation 4, PlayStation 5 games, just running, fully playable, very smooth, very visually nice directly on your GPU.

And yeah, so they're taking advantage of all of these new optimizations, but they're also using the game assets that were published before. So you shouldn't be running this if you don't have an actual copy, for example, of GTA stuff. But yeah, it's a complete renaissance. In apps, it's going to be a very interesting next three months as the entire countries and economies figure out what kind of office suite do they need to run. And the very interesting thing is that actual professionals and creatives tend to like the Vive-coded versions more because they're so much more portable. And they're fun. Yeah, and they're fun. Again, they're not perfect. For example, in Photoshop, the reviews for the Magic Wand tool is not very good because it used like an older machine learning model to remove the background. that's going to just be fixed in the next like two weeks the second he turns his slop cannon towards that issue, that issue is solved yes, that issue is solved they're throwing entire gun flip prompts

and they're I've seen entire apps, the entire Adobe seat, InDesign After Effects Lightroom, all just running in the browser, very performant too and you can run them without any telemetry or crap or spyware Without Adobe paying also a bunch of money to Adobe. Nissen, I do have to pause you because LDJ is about breaking news. Let's go. AI Breaking News. Coming at you only on Thursday night. All right, folks, we have quite exciting breaking news. Did you want to take this one? Sure. I can share. So, yeah, I do feel like overall these past few months we've had a bit of a renaissance of Western American open models. And Meta just announced literally within the past 20 minutes that MuseSpark Open is released,

which I presume is maybe like partly censored version of MuseSpark. It's 816 billion total parameters, 42 billion active parameters. So this is around three times smaller in total parameter size than Kimmy K3, while being approximately half of the active parameters of Kimmy K3. So you could expect it to be a lot more practical to run a lot faster. I would say probably in terms of parameter count of active in total, it's probably most fair to compare it to maybe the full GLM 5.3. And I think it is maybe beating GLM 5.3, though, or at least trading those with it in the benchmarks. I guess we'll have to scroll down or kind of wait for people to benchmark things. Yeah, let's take a look. Let's take a look. They're comparing themselves from NewSpark Open to Inkling, to Nematron, to Beam, to Mistral, and only then to QN and GLM, right? So, like, they're starting with, is it Inkling? Yes, I think you must say. Oh, yeah, that's right.

So, look, they're literally saying, hey, we're comparing ourselves to American counterparts first, than to European conference. Only then QN and GLM and Kimi. Yeah, I think what's interesting if you scroll down into the coding area compared to the Chinese models, so you've got the terminal bench 2.1, so Muse Spark has 76.4 then QN 3.8 has 82 and GLM 5.3 has 83.4 83.9 Kimi 805. So it's like 10 points behind. Well, not quite 10 points, but it's like 7 and 8 points behind. Then Jeep 3 as well, basically behind GLM and Kimi K3, which by the way, been out for what, a month? Two? Something like that? So yeah, definitely better than the US models, but not necessarily at coding versus the top Chinese models yet. Yep. It doesn't look like they were going

to break Soda. Oh, somebody says, yeah, but they're like 3X the size, right? So GLM and all those are like in the billions of tokens. I think GLM, not so much, right? Isn't it like about a simple size? Yeah, it's quite large here. Let me find the exact parameter count real quick here. Because Flash is around 300 billion parameters. Yeah, okay, so total parameters for GLM 5.3, the not Flash version, which they're showing the not Flash version in that chart, it's 753 billion total parameters. Oh, so that's comparable. Yeah, and 40 billion active parameters. So almost very, very similar. Very similar. And gets bitten. Well, GLM is bad at coding, like quite a bit. Yeah. Quite a few points. So, yeah, I think it's... What do you guys think? What's the relationship to Spark, Muse Spark to 1.123? Any relationship? So, I'm assuming that this is like a cleanup version to some extent.

Yeah. But they're not talking about this. But yeah, MuseSpark from 1.1 to 1.3 did quite heavy jumps. And this one is a multimodal text, image, video, and audio. Output is only text. With 1 million inference context window. And a great vocabulary size. I think a comparison could be made between kind of just the abilities, right? And then they see whether or not this is one of the MuseSparks. This is super cool Lama is back Essentially is what we're saying This is like the first release from Meta For a long time that's in the LLM space Nissan Thoughts? Excitements? I'm just going to, I just need to try it The weights are available to download The model is available in VF16 And VFP4 To run on your system The model requires a GPU with at least 700 gigs of VRAM and VFP4 and 2 terabytes for VF16. It also requires an inference deployment framework, supported frameworks, SGLANG, VLM, Hakenface, and LAMA CPP.

We're definitely going to chat with our inference folks to try and provide this inference for you because we do believe this is a good model. Yeah, yeah, yeah, yeah. Especially for computer use, I think this will be pretty key right now. I'm just reading through the license. It just asks that anything you do with it is to say built with new Spark open in the interface of each product or service that uses the software or any derivatives. That's the main license and everything else is like an open usage policy so no violence and drugs and stuff or not impersonating and misinfo. Otherwise, it looks close to just like an open license. There's no restrictions on like how much money or anything like that. this is yeah oh no there is a restriction where the third party outputs of such models models of service and the aggregate gross revenue of the licensee and affiliates exceed 20 million US dollars

equivalent during any consecutive 12 month period so as long as you make less than 20 million dollars a year you can just offer it yeah and then probably a license yeah those are the things but this is amazing because it's it will be great for generating data and stuff. Yeah, I'm pretty sure companies will offer it. It's probably one of the best computer-used ones. That is so cool. Let's go. I wish I had my arrow button, folks. LAMA is so back. LAMA is back. The promises of SpaceOnc, where they release a new Grock, the old Grock will be open-source, that doesn't often come to fruition. I don't remember the last time Grock was open-source. Zuck and Alex Weng did say that your open-source is back after the manifesto and everything. And so now they're, after they're releasing Muse, so two huge news from Meta this week, right? They released Muse Wonder and Wonderful and that folks are speaking on that a little bit. And now they released the open source version of Spark

and Muse Spark Open. That's 816 billion parameter, 42 active billion parameters. We don't have the cloud buttons here. Also the applause, but like every lab that releases stuff with the license that's open, we applause. and this was our attempt to just a conversation, the breaking news game. There was a bunch of other stuff this week that we didn't get to, including the voice. So, Grok voice released and then Tiny Fish as well. Let's see what else is interesting here. I think it's pretty much it. We've covered most of it. Oh, and multimodal Jeb. Didn't get to. I really wanted to. Jeb is now multimodal. The original Jeb, not the 700 open source or so clones, including the one from Microsoft, it's called Decision 1, including the one from OpenAI, it's called OpenAI Decision API. Jeff, the original Jeff from TypeSafe AI that we talked to you about, what, a month ago? That's it? That's only a month since Jeff came out? Yeah, pretty much literally a month ago. Jeff is now multimodal, so he can look at images and return bounding boxes.

We also talked to you about Gemini Argon, which is Gemini 4, and we had both folks here, Genevieve and Omar Sanciro from the developer experience team at DeepMind to tell you about these models as well. Coming to you live from New York, AI engineer, the conference, had a lot of folks who listened to Thursday and a lot of talk about the type of stuff that we struggle with, including attention, including how to run multiple agents, including when to sleep and how to sleep when you're just about to send something to an agent and it can reply and wait for you the whole night. This is also a part of the conversation as well. signing off from here. If you missed any part of the show, the show comes to any podcast app that you use. Apple Podcasts, Spotify, etc. and on YouTube, including shorts. If you haven't subscribed, please like and subscribe. If you have, help us on other platforms as well. It really does help for exposure. Wolfram, go ahead. I just wanted to say,

the news are coming so fast that it is now the time where it is the easiest to actually implement the stuff. When we started with the show and I saw we were talking about Kroc Voice, I told my agent, put this in my voice app and it's now running on my phone already. I can talk to Amy with Kroc Voice. It would be cool if the audience would also tell us next time what of the last show did you implement yourself? If you have your agents, what is the stuff you are actually using? What about the things we reported on? Are you using now. This is super interesting to us, not just reporting on the news, but actually the uses of the news items, what our audience is doing. That would be very interesting to hear. Please send and comment and tag us on social media. It's going to be great. With that, we're signing off. Folks, thank you so much. Peter Gostev, Wolf and Reim, Wolf Nisnop Tahir, LDJ, Yamato, I was here before, and our three guests, Nathan, Benyash, Genevieve, and Omar Sansevro from DeepMind. And all of you for tuning in, for listening,

for sharing, for liking, for sending, subscribes and sharing with your friends. It really means a lot to us, especially if you come up and tell us that you're a listener. Thank you so much, and we'll see you here next week. Bye-bye, everyone.