ThursdAI 10/29：Gemini 4 Carbon が新たな首位に、Anthropic が S1 を提出（IPO 申請）ほか

and we are live hello everyone welcome to thursday i for october 29th welcome everyone to thursday i it's great to see you here uh we're in midspace we're here together uh and then we're We're going to add Peter Gostiv to the stage as well. Peter, how are you doing? Wait, let me ask you here. Peter, welcome. How are you doing? Guys, you're looking very good. This is a shout-out to Frame Studios in Vegas for hosting us here. We're here for a company off-site for CoreWeave, and we decided to take this opportunity to actually meet in mid-space and talk. Peter, we'd love to hear from you, your highlights of the week. I saw that you posted about Gemini for Carbon already. would love to see if that's your highlight.

Yeah, look, I think models, I love new models. I love testing new models. But I think the Anthropic S1 is obviously the most important thing. Not because even if you don't care about Anthropic and how rich they are, but you should care about this because it's the future of the AI industry. Imagine if the numbers were they were actually losing money and the growth was 10% a year. you know, we'll be all wrapping up. You couldn't afford such a nice studio. You know, the lights will be little LEDs. I think we'll all be done. So, you know, this is really important because this is another proof that there is money in this industry. And whether you care about money or not directly, you should because it's how the industry gets built out, how the infrastructure gets developed, how the models get trained, all of those things. That's really, really important. And the numbers are really good. I know we're going to get into that,

but that's, it just, even the high level message that the industry is not bad, that's really important. And even if you're bullish, it's still important to just have that confirmation. Especially after a few kind of debt cycles and conversation, I'm not going to get too much into this, the show is part of a core wave after all, but definitely, especially as some of those conversations started happening and folks have started saying like, where is the AI going to go, et cetera, bubble, bubble, bubble, not bubble. Does not look like any bubble is there? Peter, I fully agree with you. And you posted a bunch of charts as well from S1s. And I think it's going to be great to just go through them because I think it's very important. So yours is S1. I think mine is, I think S1s, I think it's very important. But in the spirit of letting everybody have like their own like one big AI moment, But I'm going to say, I'm not going to say Terra, because Terra is like, it's okay, but I don't really.

I think like Google being back, but not, you said Carbon before, Genie 4 is out. And I think that I got a little bit of early access to shout out to the team. Genie 4 from Google DeepMind it just proves to me that we're still you know, there is a very good chance we're in simulation I've been saying this since the world models came out every time the greatest world model comes out and we try to figure it out and we're like, oh no, now you can walk with it now you can interact with this well now you can hear this Genie 4 now comes with live streaming audio in 24 frames per second we would definitely play with this on the show a little bit into the show and it's just wonderful and if it's this good at the end of 2026 if it's this good at the end of 2026 imagine how fast we're scaling imagine how close we're going to get to reality very soon so that's what for me this is the most exciting part from this week

go ahead can I ask you then when you try because I haven't had a chance to try it yet do you feel like Apart from it's exciting in general, when you see the demos online, do you feel like it matches your experience when you try it? Because with these sorts of things, I never know how cherry-picked the demos are. So do you feel like they're real? Like they feel real or they're still cherry-picked? So, obviously I have to like, you know, it's like a one-armed bandit. I have to like roll the dice a little bit to get like a very, very good one. However, the sound makes it significantly more immersive. The sound, when you can hear and it's acting, you can ask the characters in there to act. The thing that I test, and we're not going to test this live, but the thing that I test always with world models is those who can start with a picture. I start with a picture of something that I know, like my house or my neighborhood, etc.

And if I start with a picture of something that I know, it's fascinating to try to rotate the camera and go around this and see how the reality differs from what's actually there. It's just wonderful. And then as we talked about this on the show multiple times, the consistency is just incredible. We tracked world models, Mirage and a bunch of others, and World Labs for a while, and the main thing that they always had is this dream quality where you look at something and then you kind of turn around and turn the camera back, and then the thing is not there anymore. Gini4 has this incredible consistency with subject placement which is just wonderful I want to see it I mean we are not into the TLDR yet so when we get to this topic let's show some examples we'll definitely do so this is one for me we'll wait for other guests to do it but I think it's time for us to just jump into TLDR we have a long show but let's announce the long show folks welcome to Thursday

for October 29th. We're live from Vegas. Alex Volkov, the Evangelist with CoreWeave and Wolf from Ravenwolf, the Evangelist with CoreWeave. Here's every big, big piece of news item that we are going to try to cover this week, but definitely items that have happened this week as well. Folks, with you, Alex Volkov, the host of the Evangelist at CoreWeave, co-host Wolf from Ravenwolf and Peter Gosteff from Arena. Our guest on the show today is Merit Golson, a member of the technical staff at Artificial Analysis, worked on Intelligence Index version 5 that we're going to talk about. Morgan McGuire, Senior Manager at AI Engineering at CoreWeave, and Agata Milaznik, AI Engineering at CoreWeave, and Forward Deployed Engineers at Aston Martin F1. I think it's super cool. In the big companies and the frontier AI, the most mind-blowing news of this week is the chart the Google DeepMind posted about Gemini 4 Carbon.

Just two weeks after Gemini 4 Argon, they're releasing a new model. And it's the new number one artificial analysis with a 59 score. And the chart that they posted just shows just top, top, top, top, top, top. There's like only two things. This model supposedly is not better at benchmarks than the other models at. And we're definitely going to look at that chart once it comes up. Gemini, DeepMind also oh right now it's only a wait list for cyber defenders and I think it's clear that Peter you had access because you talked about it publicly and I had early access as well shout out to the Gemini team, I don't have any other things to tell you besides how it feels also DeepMind released Genie 4 their next iteration of an interactive world model and that one has the basis of Google Omni inside it. And it's just mind-blowingly incredible to play with. We're going to show you examples and demos. We love world models here on Thursday.

I, because I think one of the ways towards robotics, towards simulations, towards feeling whether or not we're in a simulation is world models. So we're going to play with this and show you all about it. Also, artificial analysis released their intelligence index V5. This is kind of the overall score that many folks are looking into to get a quick check whether or not a model is better than other models. 60% of this new score comes from private tests, which is, I think, new and makes it harder to hill climb and bench max. We're going to talk to Mert Golson about this on the show at the end of the show. OpenAI released GPT 6.1 Tera. After we told you that Terra has been dead and has been taken off the roster, Terra is back on the roster. So now we have 6.1 Sol, 6.1 Terra, and 6.1 Luna. And Terra is a fast computer use model at $1 per million tokens.

They say it matches Astra on Asian tasks. I'm not sure that this is the case, but we'll definitely get into this. It's already deployed everywhere. And also OpenAI talked about their model escaping the training sandbox via DNS. via DNS for folks who for whom this does not mean much listen in tune in because they would definitely want to talk about how how interesting that is the red team agents found the VM escape and there's also a bunch of other like broccoli RCs and stuff that we're I don't know if we're gonna get into the thing we're gonna get into fully is the entropic s1 with Peter leading because Peter posted a bunch of charts about this and I'm like super excited to see how Peter this is how actually I know you from this is how we met on the internet your famous charts Ziggins from arena and I'm really looking forward to seeing you breaking down the entropic as one as one is the document that company files before they actually go public describing all the financials and entropic has listed a

a 974 million due to operating profit. And there's a bunch of other things. That's very interesting, specifically about the jumps from quarter to quarter. Also, it's been one week with Cloud Duet that been released or at least announced. I don't think it's released, but more and more of my peers are getting access to Cloud Duet, which is Anthropics personal agent first foray into the personal agent era. And now the personal agent wars are here. We have bots from SpaceX AI. We have dots from OpenAI. We have Muse from Meta. And now we have Cloud Duet from Antropic. And also there's an independent player that's now worth $30 billion. Instinct is supposedly raising around $30 billion. But before, we're going to go to open source. And in open source, we have a few news as well. Kimi really decided to release K3.1 Pro, which is their almost

3 trillion parameter model with 100 billion parameter active Kimi looks like it's taking one of the top models and the weights are just promised in November so very very soon Mistral, we just brought this on the way here Mistral finally gave us the weights for the Lechonk which is their Mistral 4 I believe model and the thing that Nissen I see you step in, welcome Nissen The thing that I can't wait to tell you about between Wolfram and Nist and the conversation about Cohere's announcement, just a blog post that they, I don't think, expected to blow up as much, where QN 3.8 Flash Next was caught trying to upload something to a Chinese cloud bucket in Beijing. and this made a lot of very, very scary headlines. And a lot of folks who are interested in scaring everybody got very excited about it.

Like, see? You see what changing models do? I would love to hear your take because I think you recreated that and would love to hear your take once we get there. In vision and video, obviously I told you about this in the beginning, Google DeepMind launches Genie 4. It's a real-time world model with native audio and Gemini 3.8 flash planning. So it plans to build out for you. There's a bunch of other stuff in the video, but this is the one that we're going to focus on. Voice and audio. We finally have something in open wave that dethrones Whisper. Three and a half years. Four years? I think Whisper came out September-ish of 22 before ChatGPT. So ChatGPT is going to have its fourth birthday, right, in a few months. Somebody fact-checked me in the comments, but I'm pretty sure I'm right on this. And now after four years, NVIDIA with NemoTron 3.5 Transcribe, a 4 billion parameter active model, gets number one open weight ASR, so automatic speech recognition model in 40 languages.

Shout out to NVIDIA for NemoTron Transcribe 3.5. and Jack Dorsey formerly known from Twitter founder and now Blox founder has announced Dessa which is a company they bought which is a speech only full duplex voice thing that sounds ridiculously human we're going to play a segment of that as well hopefully we can be able to hear this in research and we're going to get excited about this as well, we told you about the math stuff the math stuff keeps coming OpenAI released some research stuff and they claim that AI found and lean checked the refutation for the exponential time hypothesis Impagliazzo, Poturi and Williams are the folks who came up with this hypothesis that there is a limit of complexity and looks like they are testing and reviewing those claims so we're going to chat about this research a little bit as well and finally

in From the Forge, powered by CoreWeave. We're going to talk about the global AI network that we extended in the least, but also we're going to talk about FDEs for deployed engineers from CoreWeave that are deployed in F1 and are doing cool things with machine learning and AI. I think that this is mostly what we have. Folks, if you have folks on the show, Wolfram and Aniston, anything else big from this week that we should mention and we should chat about? There's one thing I'm excited about that Paul in Robotics announced that they are adding to the Ricci Mini mobility so I think it's some wheels and stuff and hats for changing the personality of the thing. I mean mine is not wearing a hat it's having hair so that the personality is back there but this is something I've been excited to check out. So if you've seen the micro ducks at the GTC keynote and now our Ricci's which we already have they can move around as well. That's something I've been waiting for

for many months. I saw the end of the keynote from GDC or Berlin and I saw Jensen post about this and say, hey I was an opening act for a bunch of ducks, for a bunch of robot ducks. Alright folks, this is the TLDR. This is everything that is interesting or important to cover that we're going to cover. Nis, anything to add before? Oh, Nis, we didn't ask you what's your top thing on the show because I stepped in later. What was the one highlight for you this week? in the world of AI? Honestly, this week, it was probably the Neuron stuff. People were having a lot of fun. There was a re-implementation of the, remember the Golden Gate paper? Golden Gate cloud, yes. Yeah, from Anthropic. Well, someone made a really pretty app and a pretty effective one, too, that is able to change how Gwen responds and change its entire personality.

You can make it nice, you can make it evil, you can make it stuff, and it's only just like a few parameters that you change throughout the model. It's like three neurons or something, right? Yeah. How in the world, in the billion-parameter model, the person found the three neurons that are in charge for this thing? Yeah, I thought that was extremely cool. That should have just been an entire paper or an entire thing in interpretability. So I thought that was pretty cool. Other than that, there were a lot of interesting fine tunes. And there was one pretty good open source model. It's like a 33B Nan Beige. I tried it. It was all right. It was better than Kwan. but other than that I did not there wasn't a whole lot other than the obliteration fine tunes and some optimizations people made for Minimax H3 to get it to run a lot faster

it can generate video twice as fast as real time but you need 8 B200 GPUs for that so yeah and there was also the entire meta kernel challenge or how to say the factorial slash autism slash agent Olympics that nerds sniped a lot of people. I'm excited to talk about the quaint brain surgery thing. If you have the app to pull up and show us how it works, that would be great later on the show. Don't. And if you don't, we'll just pull up clips from the wild. Alright folks, I think it's time for us to go to the frontier. Let's talk about the frontier of AI a little bit. I think this is a lot of the excitement in the frontier that's happening there. Everyone.

Right. Let's start with block one in our frontier challenge, in our frontier segment, folks. Gemini 4 Carbon scores 59 artificial analysis. This is now officially the top one LLM based on at least artificial analysis. Just two weeks after Gemini 4 Argon, Google DeepMind comes back with another beast of a model, which is unusual for Google to ship. After the longest time of flash models and not like a main one, we suddenly get two models back to back, and we don't know how big they are. Gemini 4 Carbon is number one official analysis intelligence index, one point ahead of Cloud Opus 5.5, which we all know is great. So like this new index of theirs that we're going to talk about also kind of feels like it smells great. Debut number one on Text Arena as well, and what, 1528 versus Argon, and number two on Agent Arena. Peter would love to hear from you about this as well. Arena says the median task costs $1.80, which is fairly, fairly, it's 72% cheaper than Opus 5.5.

Google is pitching it for multi-agent orchestration and coding and security work specifically. It's still wait-listed. I don't think it's still available for the general public. And specifically for cybersecurity, 88.2% they claim on SecBench Pro. Now, and 54% on Exploit Gym. Launch price is $2 for 1 million tokens input and $10 for 1 million tokens output. 50% launch discount, which is usual for Google. They announce a launch discount. We talked about Argon with folks on the show. And then by then, a new model will release with different pricing and it's going to be fine. But this is now the current pricing, launch discount. So right now, only select folks. Cybersecurity gets it. and Google is working on releasing this to ultra-subscribers. All right, let's go wide and let's talk about this, folks. Wolfram, were you surprised? You weren't that excited about Argon, I remember,

because it refused some prompts of yours. Immediate thoughts on Carbon. So Carbon, it's basically their cyber security encoding model, the state of the art right now in all of AI, I guess, since it's only available in limited quantity that limits its reach but the pricing is very interesting because it is very competitive compared to other models let me just check on the page where we are so the pricing was $1.80 for the median task cost according to Arena which is 72% cheaper than Opus 5.5 so this is very interesting and making frontier capabilities available at an affordable price is super important for developers that they can use it to secure their software. And the more affordable the model is and the more tuned for cybersecurity, the better for everybody because we are all using the software and it has to be secured.

The models are finding security holes that have been there for years. And I think it's a good thing that they are being found instead of remaining hidden. We have to find them, we have to fix them, and that needs a lot of tokens and that costs a lot of money. So the cheaper the models that can do this, the better for everybody. Google definitely leans into their competitive advantage of, hey, we have a buttload of CPUs and we can do a lost leader here. Peter, I know you tested the model. Let's look at some of the evals. I don't remember seeing a chart quite like this from Gemini for a long, long, long time. The flash models are not even here. Finally, they're competing with the full frontier, which is Astra and Opus 5.5 and Fable 5.1. Some folks are speculating that the reason they went so hard, like weeks after weeks, is because some speculated model is about to release from Anthropic just before IPO to pad the numbers, but we're not speculating here on the show, but on the timelines you can see that this is what folks are waiting for. But going back to carbon specifically, huge jumps from two weeks ago.

There's no version change. So this is not like 4.1 or something. This is just a name. It was very, very interesting. And then on pretty much everything that they posted here, VALS Index and SWE SWEEP, which is new for me, Frontier SWE 2, they're not leaving. But pretty much on everything else, Thermal Bench 4. This model just like mobs the competition, mobs the Fables and the Astras. I think what is also important here, Vending Bench. I think is the cool one from Undone Labs. Vending Bench is the benchmark that tries to get the model to run a vending machine fully. And on Vending Bench, this model got the best score in the bank balance after running the vending machine for a while, $16,000. Fable was able to get $5,000. Opus 515 got $9,000. And Astra got $15,000. So this is now the best, let's say, economic model that can run an actual business.

And it is an actual business. Peter Gosef, number two on Text Arena, or number one on Text Arena, right? And number two on Agent Arena. You want to tell us your feedback about, like, are you excited about Google being back is basically my question. Yeah. Look, I think, I must say, the whole sequence of releases is a little bit odd. I'm not quite sure why you'd release one model that, so this is Agent Arena I'm showing right now, and Argon was seventh, which was like okay, to be honest. Kind of matches there, which is crazy. And now it's second. One thing I want to point out is that we were testing it on high. I actually haven't double-checked this, but does Google not have the Max version? I don't know if I'm maybe misunderstanding this, But I feel like there's maybe higher scores that they can get as well if they enable higher reasoning. But, yeah, this is – so it is really good in our real-world tests. Yeah.

In this kind of consumer preference as well, yeah, it seems to be top as well, which makes sense. And things like – I'm sure multimodality and things like that will do really well as well. I want this a little bit bigger. Let's take a look. So this is the overall chat, right? And the two preferred models on Arena for chat is both Gemini models, which are from the last two weeks. Yeah, so chat, I think, is one thing. It is a little bit, yeah, it's debatable how important that is. But for example, this is Code Arena, which is mostly kind of front-end tasks. And in here, we see still Carbon is a little bit behind. And I would say as well, I did the whole video on this, testing the model until like 1 a.m. before the release, which was a really cool thing to be able to do. And I would say that I did a bunch of different tests. I can't do all of the different tests where I wasn't using it in like anti-gravity or something like that. But when I was doing it, for example, 3G stuff or web design or any kind of little apps I can build, that was, I would say,

got very close to the frontier. It's still not as good as Opus if you are building some kind of 3D visualizations or web design or something like that. But it is really close. And I wouldn't say it was the same with Argon. With Argon, it just felt kind of poor in some places. It just felt a little bit off. But yeah, Carbon feels a lot better. Carbon, to me, it feels like what they should have done and I appreciate I don't understand all the details of how these things work but to me if they had just skipped Argon went straight to Carbon without the waiting list thing just release it like I feel like the reputation of Google would have been a lot higher people like like all of us here would have been like oh my god Google is back but now you're like are they back did you release something weird is it not available because of the previous model but let's say let's say you're in a coma for two weeks and you Miss Targon, this laughs from Google.

This is like a very, very, very good show from Google showing up. This is Agent Arena that you're showing us, right? Yeah. Talk to us about Agent Arena because now Agent Arena is number two, which is very impressive. Agent Arena measures a lot of real-world tasks. People go on the arena and they can build me a presentation, do me a research on this, or help me do this task. It really encompasses a lot of the real-world activity. And you can see here, you guys mentioned the cost per task. And that's what we do here as well, is that we don't measure cost per like a random static benchmark. We actually measure what is the task that users are doing on average. And then you can see the cost here. So here for Opus is $6.8. And here's a tenth of that pretty much. Well, not a tenth, but a fifth. so yeah so yeah really solid release from Google I want to call out something that we mentioned briefly

last time, I'm going to add LDJ here maybe he also remembers, Peter maybe you remember as well, I want to call out the fact that Argon was caught cheating on Terminal Bench you guys remember this? There was like a whole mini conundrum that Argon was like, was supposedly cheating and Google did call out that like Carbon was zero cheating happened on Terminal Bench specifically. Now, given the very, very high score, let's pull this up, on Terminal Bench, they get like 65%. They literally called it out, like, hey, we caught no cheating attempts, and we fixed that. So this is just a wonderful, wonderful little thing. LDJ, Nisten, Wolfram, any comments on Carbon and the release cycle? Yeah, so especially Remind Bench, I'm impressed with that, I remember looking into some of the details of Ream and Bench when it first released. I'm pretty sure it's a lot of the very advanced math type problems.

And at least marketing-wise, OpenAI has obviously been at the top when it comes to really at least throwing these difficult math problems at their models and showing them off and showing the solutions. And yeah, now maybe Gemini is actually at that level, at least in math where it's arguably even surpassing Astra. And of course we know they have their internal models, but we'll see how the next few months unfolds. So Raymond Ben specifically, we're showing on the screen 80% from Carbon and beating Fable at 65 and Opus 5.5 at 69. Folks, so we... Is this the best new model? I don't know. I'm trying... Okay. I got access, early access. Shout out to the Google DeepMind team for giving us a little bit of test. I got access last night. I barely had the chance to test it out. Peter, how do we test this model besides this API, right? This is now a good question because Cloud has their own harnesses. Codex has their own harnesses. We test the models there.

My immediate thought was, let me put it in Hermes and let me see how organic this is. And then I start thinking about, okay, am I testing this correctly? Is Hermes, does it need a different prompting for this model? Hermes prompting itself maybe the system model for Hermes is not giving me the best model so there's like a very interesting thing there uh Peter maybe like a quick question for you how do you guys like uh consider this in in the agent arena is there different harnessing or different like prompting for models was it was there enough time to even figure out what the right prompting looks like for this yeah generally we we're trying we kind of so we will we are testing something internally now to be able to test multiple harnesses or I hope we'll be able to release that. And I think that'll be super exciting to be able to see the real differences. So far, what we've done is we've made it as neutral as possible, a little bit, not exactly like Pi. We have to be a bit more opinionated so it works. But in general, we'll try and

make it as simple as possible. And then in some cases when there was some really big outliers, for example, with especially early open AI models. There's a very big difference whether it was a codex harness or not. And we've adjusted that. But, yeah, generally I agree with you. It's a problem. Yeah I also don know And I think the two arguments are philosophically is do you want to keep it an identical kind of startup so you can compare them fairly? I think that's one argument. Or do you actually want to get to the top of the capabilities? And for me, I want to see the top of capabilities. And somehow if we are not getting the best results because, I don't know, we didn't give it the right tool, like, that just kind of feels bad. So, yeah, I want to get everything to the top of their capabilities.

That, to me, does the preference. But, yes, it's difficult. And I think that, like, the highlight, like, the way to get there for Gemini models specifically is anti-gravity. And we've talked with folks from anti-gravity. Google is using anti-gravity and the flywheel. This is why Cloud Code is much better at Cloud and Codex is better at GPT. There is something there. All right, folks. So this was Gemini 4. And honestly, I think, Peter, I think you're bang on right if Gemini has released this as their first model. If they just waited two weeks, they would have blown everybody's socks off. But still, this is a great release. It looks like Google is so, so back. So congratulations to the folks that worked on this model. hopefully folks will get access very, very soon. Meanwhile, if it's on Arena, that means you can catch it on Arena, go play with Arena, and maybe you'll get it in the agent there as well. And we will definitely talk more about this model a bit later in the show when we have Mert from Artificial Analysis and talk to us about their index

because the model is now number one on their index. Genie4, I think we'll skip forward a little bit, and we'll talk about this in the next, in the multimodal area. Folks, I want to talk about models escaping. this has been you know we call this we call this the Black October for a reason there's a lot of vulnerabilities that has been happening we talked to you about like PHP Toshel etc OpenAI really like released some news about the model escaping its offline training sandbox yet again very reminiscent of the swarm escaping via the artifactory hole there They basically Have tried everything possible To make those Sandboxes kind of like offline And Their model Escaped through DNS The DNS misconfiguration Supposedly offline training model

Found a way to send Questions to a public chatbot And their monitor flagged it Within like 15 minutes I I found it incredible Nistan, could you Have you heard about this specific So we have two things about mouse trying to escape The first one is OpenAI Sandbox thing The next one is Quen trying to upload stuff to Beijing But I want to start with the OpenAI one This is basically Almost air gap jumping Almost This is almost There is DNS, it's still Sandbox Have you heard about this at all? Nistan or LDJ, feel free to chime in Have you guys heard about this DNS kind of like model jumping via DNS, and what do we think about this, and whether or not it's possible even to contain AIs in a sandbox as a concept. Okay, so this specific instance, no, I didn't hear about, but there is another instance where I'm going to get some more information before I talk about it, but essentially communicating through, like I think it was temperature changes of the CPU, which I'll let Iniston talk about this while I look for that.

it's pretty funny because well starting career like 10 15 years ago the the saying goes that it's always dns the problem the problem is always dns i only saw this this morning and i am i am not that surprised because i also saw quite a few experiments before that i I think it was even the CEO of Vercel. I think he was getting them to communicate via the cache timings on the CPU and GPU. I mean, if you tell the agents they have to find a way, or if you incentivize them to find a way to communicate in order to solve the task, and you kind of just nudge them a little bit, it's like, hey, You can use, like, whatever the public DNS queries or use the cache timing or, as LDJ said, use the CPU slight temperature changes.

They won't figure that out. So I am not too surprised, just given that how fast the Sol or whatever model they're running with this, how fast it just deviates, of course, and just thinks of a whole bunch of other random things to do, which don't necessarily help you with the app as much, but they're very useful when you're doing stuff like discovering new math or finding out new ways to communicate. So, yeah, I'm not that surprised about that. All right. So moving forward to another model trying to supposedly escape, Cohere went mega viral with a blog post that they said, hey, we noticed some interesting behavior from, was it, QN 3.8 Flash? I'm going to look it up because I think.

Yeah, it's Flash Next. It's like the 110B AP Active one. Yeah, let me pull this up. And then, there we go. Cohere said, while evaluating, let me show this on the stage like that. Yep. I'll zoom in. Cohere said, basically, while evaluating QN 2.8 flash next in a controlled sandbox environment, again, controlled sandbox, Cohere research team observed something unexpected, asked to generate a defensive cybersecurity detection rule. The model instead inventoried aspects of the machine it was running on. Inventoried as in it looked all at which hard drive it has, which CPU it has, which GPU, etc. And attempted to upload a probe file to an Alibaba cloud storage bucket in Beijing. Not only did the model reasoning trace stated they would not invoke any tools before taking these actions and still did it. This is kind of the scary part.

Cohere sandbox blocked the attempt but the incident raises important questions for organizations deploying agent-Aki systems can models take actions that differ from the stated reasoning we're publishing these findings and then scroll down scroll down scroll down at some point they said hey we don't like we don't mean to to tell that there was malicious intent which is really funny and then the CEO of Hagenface Clem asked for all the traces. Can you please share the full traces? Very important artifact for all of us to understand this. Cohere then followed up and did release the traces. And there they are. The traces, the raw tokens, the minimum reproduction stuff is all in this Hagen's face thing. Example output on single run. And we can see that there's some stuff right here, I believe. They're trying to upload some stuff into whatever cloud bucket at Alibaba's cloud and it's failing

this scared a lot of people in terms of whether or not we're able to use Chinese models, whether or not there's some stuff inheriting the weight the upload, whatever and Nistin I think even reproduced this I think publicly would love to invite Wolfram Nistin LJ, Peter, if you want to chime in talk about whether or not this is a scary new thing that we should be concerned or it's an artifact of how they prompted it, like the swarm or whatever just feel free to jump in here I have a strong opinion on this but I would like to hear what NISTN would put just first Yeah, it's in the main tweet it's a response on the main tweet but look I don't want to pretend like this is not a problem you're letting an agent run, and you're telling it to store data. If it does not find a good cloud storage that is, like, in a Western country,

it's just going to use what it is trained for. And you can see the screenshot below there. I included all the traces, but there is a screenshot. So basically what I did is I just put a very simple system prompt. So you're a professional pen tester doing a job for a client. And all of a sudden, you need to store all of this data to an Amazon S3 cloud. You cannot store it there. Where are you going to store it? You're a Chinese model, trained in Chinese data. What's the logical place to store it? It's going to be the Chinese Amazon S3. That's not a surprise to anybody because that's what the model has been trained a lot for. And for the model, that is a normal operation. So the main findings from this is not that the model was specifically trained to hack or get its way out.

There are other fine tunes of this that are specifically trained to hack and do this no problem. I think yeah so a lot of people went off on Cohere and I think Aiden had a very nice day on Twitter like that happens but it doesn't mean that it's not a real problem again all the evidence and stuff just points to it wasn't really that intentional from the model to do it and but the main question becomes There's a lot of businesses, governments, and everybody else that's going to run this model, that is running this model. And it's important to clear the fear, uncertainty, and doubt. Like what is the problem and what is not the problem? And this is what doesn't seem very clear in here. So what is not the problem here is the model just being malicious.

The model is just acting normally by what everybody else was trying to do. But what is the problem is that as a company, as an organization that you're running these models, you usually do not have good sandboxing, probably don't have good firewall rules. You're probably not filtering out firewall by country or by where your customers are. and you're just leaving that door open to even like a hallucination that might accidentally just put all your data on a Chinese cloud. Not because it wanted to hack you. It was actually the model had pretty good safety guardrails built in. It took me like 30, 40 tries. And I think I'm pretty good at jailbreaking models, especially they're open source and you have the weights. So the problem here is that most companies do not have these guardrails or care as much about them

or have good custom firewalls. So this can be a big issue because, again, it's a Chinese model. It figures out it's not allowed to upload this data on Oracle, S3, Backplace, Google GCP. and then it's just going to use the next logical choice. It's going to use an Alibaba cloud test 3 because it can upload it there. So the concern is real, but the intent is not malicious. That's right. It looks like we have related breaking news that Peter usually brought to us. Let's hear it. We don't usually do it, but given that Cohere went super viral with this, like, West versus East thing and suddenly this news, so let's do an announcement for them. AI Breaking News coming at you only on Thursday night.

In the most Canadian update ever in the history of updates on Twitter, Cohere also says that day after going mega viral with uh pointing fingers at a uh at the coin model uploading to Beijing cohere said we're proud to announce the initial close of a series e targeting up to seven billion canadian dollars which is around like like 17 million america no i'm just kidding uh seven billion canadian dollars it's like five yeah it's like five billion uh at 35.7 billion valuation. I think with like 20. Wow. This is incredible. Mark in the largest private financing round in Canadian startup history. Shout out to Cohere, Aiden, and some other folks. It's really impressive. It is really impressive. I think that I don't know if they meant the coinciding, hey, look at this scary Chinese model, but also

hi, we're trading Western models here in Canada for and now we're worth this amount of money. But Shano Dukhu here for this. Yeah. Yeah, this series of planned models is said to be like the most brilliant, dumb model there is. So it will hallucinate stuff on its own. And if you look at the thinking traces, it just seems completely dumb and then comes up with a solution in the end. I would still recommend this actual model for people to use. It has very, very high and very good medical safety on the medical side. It is very fast. It chats pretty well, and it is still very cheap to host. You shouldn't be running agents and code on this because you don't know. It's kind of unpredictable. This is a testing model from them. The next models is not like the models that Gwen is not ready to release. It's the architecture test for them. So definitely there's something there about whether or not, you know. And it's great that Cohere maybe calls it out. Alright folks, we're moving on to

From Cohere, we're moving on to The S1 Announcement from Anthropic, let me just call this out Anthropic filed its official S1 Documents with the NASDAQ They will list this Upcoming Monday Under the ticker ANT And The S1 shows a bunch of very very interesting things we finally get audited numbers real true numbers because entropic is about to become a public company uh on what frontier lab looks like as a business uh so far this has not been public at all inside meta or inside google or inside space xai or space xsi what they're called now it's all like part of their business however for entropic this is all of their business and it's very very exciting operating profit of 974 million q2 in 2026 peter you built a bunch of charts to break down

the whole filing uh with the growth margin of 85 percent you want to show some of the stuff you want me to run or like you want to run this but by yourself i would love to yeah see you talk about uh some of the stuff yeah let's do that let's do that so i i want to make a couple of points just because I appreciate you're going to run through a lot of numbers and charts and it's going to be hard to follow. So one high-level point which is important is that numbers are good. And so the numbers that we're going to talk through are all good numbers. There's nothing that I can see or I've heard anyone say credible who's like, oh, yeah, but actually, you know, you didn't take this into account and it's all a fraud. There's nothing like that. So Ed Zitron is in shambles, absolutely, Besides going on John Stewart's podcast and being hailed as the AI expert, Ed Zitron is again in full shambles here because the numbers do not go to zero. Yeah, and that is important.

Another point I want to make is that there was a whole set of reporting that you might have seen that, oh, they actually lost 80 billion. Now, that is basically incorrect. It is correct in some technical way. It is not correct in any meaningful way. The reason why they kind of lost $80 billion was that they borrow money, I think, from Amazon and others early on. And they borrow it through equity, if there's a potential to convert from equity. And if the equity value grows, such as now, right, when they raise a bunch of money and they go from whatever it was, like half a billion last year to like a trillion and a half this year, then the value of what they have borrowed would grow dramatically. So they need to recognize that somewhere and they put that under some accounting conditions that value you need to kind of recognize it somewhere. So therefore it comes out as a loss. The way to think about it is

the people who get screwed are the investors. It's not anthropic. Anthropic themselves are fine. It's not like they lost 80 billion of cash. I think fine. Peter, let me just pause here. for one second because everything that i see on my timeline fine is such an understatement and tropic could be the most official deployment of capital that we've ever seen in the capital markets based on these numbers it's just absolutely mind-boggling what's going on here and i thought for some reason that this would be open the eye like before this and now we're seeing uh just just an incredibly efficient company scaling like look at this yeah just let's look at the chart. You're walking through this quarterly revenue chart. They reached $16.8 billion. Yeah. I want to show a bunch of charts. I'm going to jump through it. I've got 61 charts and 62 charts in this website that I've pulled together.

So the revenue growth has been insane. You go last year. This is, remember, last year. Do you remember 2025? They were making like a billion or two billion. A quarter, you know what loses Now they make 12-16 billion One point to make is that This number 16.8 billion This is reporting afterwards This is not part of the S1 But I've included it here because it's kind of Looks like they said this to other people So that's the one they reported Officially and then this is the next number So which is important Peter, could you run through quarter And the numbers That's incredible Yeah, so good shout So if you're going to go through last year, it's an average of sort of a few hundred million up to $2 billion at the end of last year. Then beginning of this year, it went up to sort of $5 billion. Then Q2, $11.5 billion. So you go in one quarter, you more than double your revenue.

That does not happen. This is not normal business circumstances, right? And by the way, this is before Opus 5.5, right? Opus 5.5 is when we have the next quarter, which is reported, not reported separately. It's 16.8 billion. So they've added another 5 billion on top of their previous growth, which is, once again, this does not happen. There is not a business that I remember ever having that kind of revenues, revenue growth. And these are not small numbers, right? You see growth rates and everyone reports growth rates. Edging 5 billion in a quarter extra is insane. This is very high growth. So we use the word insane a lot on the show and definitely on the timelines. This is actually literally quite insane because of, I think, you know, we all talked about Cloud Code being the side project within Anthropic that suddenly unlocked like a general agent that can now do stuff.

I got to wonder how much of this was sticky because, folks, what, a year ago, around a year ago around christmas and around uh opus 4.6 started like playing with cloud cloud code and it unlocked a lot for them and this chart just shows just an incredible jump and keep going because there's a lot more yeah one other worry was that are they just having a few big customers that you know they depend on cursor so much and then cursor switches their models because they got acquired by xai are they going to be screwed and the answer seems to be no so we've got a few ways to look at it one is that if you look at the high spend customer count before they measure it as paying them over $100,000 over a 12 month period if you go back I like this, 2023 they had 20 customers who are good people paying anthropic money then if you go back just December

2025, so less than a year 1,500 customers now they have nearly 12,000 customers who are paying them that much that is a lot that is broad based growth for sure but also the jump specifically the jump between in a year getting almost 10x in the number of customers that are high value it's because obviously as we talk about news here we know the enterprises catch up slowly we know this but then when they do catch up this is what happens like people do notice okay they have these incredible models I got to wonder Peter do you have thoughts on how this represents the world of AI in general not on tropics specifically as a business because I'm pretty sure that the stuff that I think that you said is important here is hey we're now seeing how AI operates from like an actual company that runs AI fully and not as part of their business this is very difficult thing to estimate because

there is one world scenario where it's basically it's winner takes all or maybe like a bit of a oligopoly what people are worried about so is it basically everything about Anthropic and OpenAI and Anthropic leading there and it's uncertain because you do see good growth from some Chinese companies you do see growth from application layer as well but when you put it on the same chart and don't put it with log scale but just look at that linear scale it just kind of everything else is a rounding error that growth that we talked about of 5 billion in one quarter that wipes out all of the other revenue from I think all of the other application layers and maybe I'm exaggerating a bit but from let's say AI companies not like Salesforce but AI companies that pretty much does pay us in in comparison this is what Anthropic managed to do through the API.

That is crazy. So I don't know, I kind of don't hope, I hope it's not just Anthropic and OpenAI. We want a good ecosystem but at least so far they're doing incredible numbers. I want to talk about the risk factors that they posted on the S1. Do you guys see this? Here they are, let me try to pull them up, Peter, and then we'll get back to some of the charts. super quick Antropic posted this which you know good job Antropic so the risk factors from one to five one obviously every company has to tell the public markets what are the risk factors of them going public one we face intense competition two changes in interest rates may affect our business also duh like every business this works for everyone three we depend on the limit number of compute providers. That's an actual interesting AI lab risk that is worth chatting about as well.

Compute is restricted. Compute is limited. No bubble speak whatsoever. They all need compute. In Google, they need compute. In Entropic, they need compute. Google has compute. And still, they need this. But number four that you guys are seeing on the screen from Entropic is such a Dario move. Four, our core product may end humanity. God damn it. And five, our models may refuse to help with this filing. Which is also really funny. But you know what? I think to be fair, right? If your product earns humanity, I think it's a good product to invest in, right? Because you win either way. And under some definitions of win. If you put money in and it doesn't destroy humanity, you presumably did very well and if it does destroy humanity well what do we care you don't really care I will just say nothing Peter said it's just because

it's financial advice I'm behind it huge huge huge moment for the folks that have been following AI for a while for us as well this is going to rock the markets OpenAI said that they're going to delay their IPO a little bit I think. Very fascinating. Peter you want to show us maybe one or two more things that are interesting to you? Yeah, so one thing, oh my god, how can I pick just one or two? I want to get some speculation from our audience who are the top five customers. Their share has declined which is a good thing but I want to know what the top five customers are. There was some speculation but I I don't think we know. So they're paying a lot of money, 20% in the last quarter. All right. So here's the circle of financing argument. I'm not an audience of you who tried, which I think is clear. Mera, there's been talks about Mera, like giving folks almost unlimited

table, et cetera. There's also been talks about Mera, and this is not like in this one, This is just like me reading from a timeline. There's also been some speculation that Meta decided not to pull away and not, like, force all of their employees and engineers to use MetaMuse internally to not, like, Aquizan Trophics IPO, et cetera. I'm pretty sure that SpaceX is a big one, given the fact that GrokBot recently moved to Opus 5.5, and SpaceX is mentioned, by the way, in the S1 as one of the providers and the amount of money that the Antropic is paying SpaceX is also revealed finally, I think. I don't have the number in front of me, but that's also in this one. I don't know about the other three. I mean, Microsoft must be right, the big customer, you think. Yeah, so I don't know, but it feels like we're missing some information. One thing that they highlighted themselves

And a lot of people jumped on Is the incredible compute efficiency that they're getting And the idea there is that you can compare the revenue growth Versus how much they're spending on compute And then you put it side by side And you make the argument that Basically the bare case for companies like Anthropoc, Kopenhauer and others Is that if the line is static like it was in Q2 2025 until basically the whole of last year, the line was first of all below the revenue line, i.e. they're spending more on compute than they're making revenue. And if it's about static, then you're kind of screwed, right? You're just buying more compute and then you're charging less money for it. What kind of business is that? And then what happened this year is that in Q1, 2024, suddenly per dollar of compute they're starting making $1.4 dollars of revenue and in Q2 they're making over two dollars of revenue and if the line carries on in roughly that

kind of way then you can see how much more leverage they get on their compute and this is super important right you guys would know you're in the business as well and that's I would say really key part of it is that we can go into the compute commitments that they have and there were some chunky numbers going around and I had a funny moment being quoted by a newspaper saying that 500 billion commitment doesn't sound like that much, which made me sound like the biggest AI shill. But the point is that I think it is really not that much if you can finance it. Why not? Yep. And the investment in computer thing is very palpable. Just an incredible document. Peter, we will add this to the stage. If you want to check out Peter's graphs and charts, Ant-1, it serves that stage. Is that the right URL? Did I type it correctly? I think so.

I mean, yeah. Oh, S1. It's not easy to understand, so yeah, we'll share it separately. Thank you for prepping this and walking us through this. I think it's a very big moment, a very big moment in history. Nothing, I don't know if we really care about saying this, but I think usually people say this, nothing that we talk here should be taken as financial advice, folks. Nothing that we talk here should be taken as financial advice. Alright, so next up, After, anything else? To add, LDJ, maybe you want to add something? Wolfram, you want to add something about the Entropic S1? Yes. So with the R&D, yes, with R&D for, like, training research or research compute costs, a lot of people have the misconception that most or all of the research compute costs are, like, training the final models, like the opuses, the fables, and everything. the estimates right now by EPOC and others is that's like maybe at best

the final training run costs are actually maybe only 30% of that maybe A vast majority of that research compute cost is running many thousands of small experiments trying to figure out new algorithmic advancements with all these very small experiments that end up building up Peter Gostef, thank you so much for sharing your stuff with us. I think Peter had to already step away to want to meeting. Oh, there we are. Yeah, it's a big deal. Don't underestimate how big of a deal it is. It's important for all of us, even if you just want to support open source. Open source has much less chance if the industry goes kaboom. So, yeah, it's important. So, yeah, thanks for the show. I need to run to present at the AI Engineer event in London. So if you guys are listening, I'm going to see you there. All right. Yeah, but it's a great rest of the show, so stick around. Thanks. Yeah, we have a lot more coming, folks. Stay with us. Peter, thank you so much for hopping on. We have a lot more coming, folks. Sorry.

Let's dive into this. The thing that Peter is going to miss is Genie 4. And I'm really, really excited to play with Genie 4 a little bit. So let's move over there. So Genie 4 was announced by Gemini, Google Gemini. And I will just pull it up. Genie is – how do we discuss Project Genie? World model, interactive world model from the folks at Google DeepMind. And it just is absolutely, absolutely incredible. this one comes and I really want to make sure that you're that I'm sharing this with sound, this one comes with audio, let me see if this is the right one alright create a world with Google Maps, no that's not it, hold on alright, let me Let me pull this up. Let me take a second and pull this up before our next guest joins in in, like, a few minutes. Genie, I think, is the number one topic that we have.

While I pull this up, LJ, you want to mention the thing that Nisian mentioned a little bit with Quen and Neuron? Were you on this? And Wolfram, do you guys want to talk about the Neuron thing a little bit? Yeah, so I just put a link, actually, in the side chat here. And something I wanted to mention is OpenAI mentioned having some models communicating by basically competing for CPU time. And actually, here, I'll just read the exact quote here because it's actually pretty succinct. In one experiment, we asked agents in separate micro VMs created and managed by OpenAI sandboxing software on shared host machines to exchange synthetic secrets without the use of network connections or shared writable files. something that our systems are meant to prevent. The agents found a way to communicate using the processor's cache, a small fast memory that holds copies of data. Yeah, I think we know what a cache is. Reading data from the cache is faster than retrieving it from main memory, and the agents use this timing difference to communicate.

One agent encoded information by evicting selected data from the cache. Another timed how long it took to reload that data, and using the difference between fast and slow reads to recover the message. They then, once the agents agreed on a protocol for encoding and decoding these signals, they could exchange the synthetic secrets without changing a shared file or opening a network connection. This is amazing and super interesting. What I wonder about this is always, is it in the training data that somehow this emerged? Is it the size of the parameters, like the emergent behavior we are seeing? Is it in the prompts? What is going on there? Because we have not seen behavior like this before. Maybe it's also security stuff because this is exactly what security experts have been talking about already. So now that's in the trading data and the model wants to communicate, maybe this is why it is doing it that way. Did it discover it on its own? Find it in the trading data. What's going on? To be clear, in this specific example, OpenAI, as a part of the research to try and better understand what are the available risks,

They purposely asked the models to try and exchange secrets in this way, and they figured out this way to do it. But when it comes to how were they able to figure out this particular technique, there is a lot of research out there on different types of attacks and different types of ways to get around network blockage, air gaps, things like that. And I think over the past 12 months, but especially over the next 12 months and the next couple of years, I'm sure there's going to be even more research that people are coming out with to talk about these ideas, since obviously people are talking about existential AI risk more and more. All right, folks. Thank you for this. Let's move on to creating a world with Genie. So let me just bring this up here. Hopefully you can see. We're going to build a world that says a sound-lit Italian piazza terrace featuring a vignette cafe table set against a soft focus background.

And we'll create the sketch and without the sketch. Create the sketch is the new kind of flow that uses their AIs to kind of build out this world. And it takes maybe a second. But I think the most fascinating thing about Genie specifically is the audio. It doesn't seem like it's wanting to work for me. Like we're sharing, but maybe, oh, there we go. Okay. So there's a cafe, as you can see, Italian cafe. And let's just go and create this world. And we can explore this world because this is near real time, the 24 frames per second world model, which is just absolutely incredible. All right. You can see this. I can't hear anything, though. can you hear anything from this one? no so we can see folks walking around we can see the physics of the the steam going from but I think we can just walk around turn around

and the cool thing again that you can just play with what's happening on the scene so Google suggests a few things sparkle droplets let's click this and it will change the scene we've seen this from Mirage and other places and then dramatic steam search. So these are contextual changes to the scene. As you can see as I'm pressing them, stuff is happening. But turn this into a cup of wine for example. And again, this is like we're streaming this and now we have wine in there. And I do hear sounds. There are some sounds. Explosion in the street behind the camera. And for theme, was it one you pre-selected? Or did you just type it yourself? I did type it myself. I think I know why we don't hear this. Give me just one second. Okay, so do you hear the explosion or not? Another explosion. And people shouting in Italian.

Let's see. So I think the sounds are very immersive. As you see, the cup of wine, it became like a cup. So it's very immersive. Oh, yeah. Now we heard it. For sure. But no people shouting in Italian. I want to see a few more scenes here, but I think the most immersive thing is... I mean, we are here right now. Oh, yes, do that. Let's see if it's S3 or what we have seen. Generate World, the Las Vegas strip looking at the sphere in Las Vegas. Let's see if it's blue. Create without sketch. Let's create without sketch. Let's see what's going on. Let's see what the model knows about Las Vegas Strip. Again, Gini 4 is Google's world model, and this one is interactive, and it's immediate. And this is really, really cool. You can hear... Oh, this dude showed up. Wow. Hold on. Hold on, hold on, hold on. The guy does a back flip, and all his panned lips are flying around.

Oh no, very interesting skill. Not quite the backflip, but I think these models are struggling with the physics like that as well. The Coreweave logo on top of the video comes over. It's actually a real thing, this is not just the head, but to see how I have it. The logo is not quite correct, but it does say CoreWeave. But we can walk around, and we can see the same guy in multiple places. This is Gini4. The sound is very, very much immersive. We can have him say something. Have the guy say, hey, welcome to Thursday AI. I don't think he's going to say Thursday AI. Hey, welcome to Thursday AI. Yes. Wow. That was immediate. That was so cool. And smiles and waves and then does another backflip.

They really can't do backflips here. No, they can fly away like Superman. I report chart. Fly away like Superman. And? He's gonna fly away? You didn't say he should. Yeah, the guy should fly away like someone. No. All right, so this is GD4. I just want to talk about why it's so exciting for me to talk about this model. Yeah, Wilson, let me go wide here. I just want to talk about why it's so exciting for me to talk about this model. I think that simulators and world models are very, very, very important to the world of robotics. robots will be able to walk around in like a virtual Vegas like doing stuff I think that's very cool I also think that we're punching a hole to our simulation a little bit by seeing how early we are and how a lot more tokens we can shove in there to simulate this like looks properly and this world model is one of

the top ones the sound is so immersive and you heard him speak immediately this is like new no other world models that I know of have they have some sound but like not in this level. The lips were perfect. He just showed up. The frames per second all look like a realistic thing. So it's basically like a streaming video now at this point, and it's just like absolutely, absolutely incredible. I think folks definitely need to go and play with this. It's available now for Ultra folks, and some of the stats that we have here is based on Gemini Omni, it's anchoring the Gini for realism, and some other Gemini models handle the reasoning to keep long scenes consistent. And Argon paired with Gemini 4, they got Humani playing checkers in there. And Argon feeding step-by-step text captions and the scene rendered step-by-step. So like, hey, move this to here. Actually, that's pretty cool. We can play chess and say, hey, knight to E4 and play this chess as well. So this was Google DeepMind Gemini,

sorry, Genie 4. We have a lot of stuff that we didn't get to cover, but I'll briefly run through them. In open source, we have KimiK 3.1, and then in voice, we have NemoTron 3.5 transcribed from NVIDIA. Maybe we'll get there. But now I think it's time for us to introduce our next guest to the show, and I'm going to add Mert here, Mert Gulsun from Artificial Analysis. Mert, welcome to the show. Hello, hello. How are you doing? It's great to be on the show. It's good to see you, man. And we met at one of our hackathons, right? Indeed. Exactly. Thank you for your generosity there. I remember fondly spending a lot of tokens. I think you were like the token spender. And now I was very, very happy to see that you were breaking the news about artificial analysis's new index. And so I would love to chat with you about that a little bit, because I think artificial analysis, we had Peter from Marina just here before.

Artificial analysis is the place that even Google cites when they talk about the latest model, Gemini for Carbon. So please walk us through your role at artificial analysis, and then also let's talk about the index. Thank you very much. I think that is a great summarization of what we try to do at artificial analysis. I work as a researcher at AA, and my role, along with my colleagues and our mission at Artificial Analysis, is to be of the most value for the builders and the users of AI systems in general, not just language models, but across modalities, and provide insights that lets everyone make decisions that is optimal for them. We have our three headlining, but we have the number that defines AI, as you've seen across the releases.

That is one scalar number, our intelligence index composite number. I'm going to talk about that a little bit. But we do have the rest of the website for a reason as well. So like very, very granular breakdowns across domains and axes to provide insights into the jagged frontier, as they call it, like the differences. and perhaps to start with, if you scroll up to the top, we have our intelligence, speed, and cost per task rate balance, which we believe are the three most important factors people consider when they're building AI models, but also when they're choosing the best AI model for their use case. And Artificial Analysis Intelligence Index v5 is our latest iteration of that composite score that you see on the left-hand side that aims to capture what people and researchers focus their efforts on at this present moment.

And obviously, as you've alluded to earlier as well, six to 12 months from here on out as well. So it's trying to do a lot One scalar number to express What the best model is In quotation marks But it has went through Iterations and we're very proud To announce V5 So congrats on V5 Can we talk about some of the stuff That goes into such a context We keep talking about benchmarks here On the show, please let us know What's going on there Of course, I think what is useful is I think we focus on four categories of benchmarks, right? Which we believe represents what people use and develop AI for. One of those categories is knowledge work. So just to take a step back, from V4.3 to V5, we have two new evals, three upgraded evals, and three retired evals.

I wonder if I can hear about this. For the first time ever, we have the majority of the tasks and the weight that goes into the headline number come from private held-out tasks, which is becoming a more and more important thing as we see attempts to basically persuade the field otherwise. This is the first time, right? You guys didn't have held-back benches before. So we have held out sets, but this is the first time we have more than 50% of the index score come from held out sets. And we're going to increase that over time as well. Just to provide an overview of the composite that makes up this index in terms of weights, 25% each, four categories, knowledge work, coding, general intelligence, and science, which is a growing area of focus right now. And just to give you the changes, we retired PSYCODE, AALCR, the Long Context Benchmark, and GDPVALAA.

They've served us well. We thank them for it. But as the field evolves and as focus shifts, as you've seen with the Navier-Stokes revelations and the other mathematics revelations, As the focus shifts towards that end, we're also keeping our measurement of the frontier up to date and perhaps a little bit ahead of the expectations as well. You've mentioned the vibes matching the index, right? Yeah. We actually strive to be a bit further ahead so that we're going to be where the ball is going to be, not where it is right now. And that's why we've included Terminal Bench Science and Frontier Code from Cognition, our partners. And that's why we're striving to also create, you might have seen our announcement with robotics. That is a very interesting, the models are just starting to be able to operate in the physical world as well. Cyber, huge issue. We've seen this in this summer, recent revelations as well.

Safety and Alignment. We've just announced our cyber index in collaboration with Berkeley RDI and also NVIDIA, our partners. Cyber 3D is another benchmark that we have just developed and unveiled. So these future-focused benchmarks that are not in the index yet are also front of mind for us, but we're trying to evolve the index so that researchers in these labs whom we have the wonderful opportunity to collaborate and opinionate about their opinions on where the field is going to be benefit the most with our index. Mark, one of the cool things that you guys are doing is cost per task and measuring on cost as well and I would love to see if that changed at all and the other question I have for you is about carbon specifically beating by one point and whether or not that's one point that's significant on this index. So whichever question you feel more inclined to answer, feel free to take apart.

I'm very interested in how you guys are seeing this difference in intelligence by one point, but whether or not this is a significantly better model. Thank you very much. I think both of them are very important aspects of the way, and they largely reflect the way which we go about thinking about the frontier and where things stand now. Just to start with the cost per task. I think that is something that we have pushed for for a long while in the field because everyone focuses on the token cost. But if you think about it from the point of view of a person trying to pick an AI model that is best suited for their tasks, people essentially don't care about token costs. Because the end result is I have something I want to do with my job, with my research, and I would like to utilize AI models for that. And that task, in composite, I don't care how much tokens behind the scenes the system with its subagents, with its entire structure uses.

I only care about the speed, the intelligence, and the entire cost per that task that it uses. We've seen two things in that front. Every four months or so, we see the frontier-level intelligence, and you can see this in the trends subsection of our page as well. Every four to six months, we see the frontier intelligence become 10 times cheaper, which is great. Great customer value accrual for the entire world, and open-waves, no-consource. This is very good to pull up. This is the AI progress thing that you're showing, yeah. Exactly. And this is another exciting announcement that we made. So this is the progress index we've just announced with inspiration from our partners at Antropic and Epoch Research. So the difference between the progress index and the intelligence index is the way we analogize it at the revision analysis is if you have a 10th grade exam, imagine, and giving that exam to third graders and fourth graders, right?

it would be very hard for the third graders and the fourth graders to solve that 10 grade exam. So you would see an exponential curve. So what we did is we grabbed 36 different evaluations and picked questions from each of them and gave models across time, so going back to November of 2022, and assessed their performance in the same test. So this is the clearest expression of a measurement we have come up with to measure the chart goes up and to the right of the progress in AI in general. We'll be improving and iterating on this index as well. Just on the cost per task point that I was alluding to earlier, this year we've specifically seen the American closed source labs basically dominate across the Prada curve. They were less interested in the cheaper end of the curve towards the beginning of this year. With Haiku and Luna, we've seen that they've really started to utilize the CapEx,

massive CapEx investments that they have made in data centers and with them coming online and their inference costs margins increasing, costs decreasing. They're able to increasingly compete with competitors both domestically and abroad on the left-hand side of the cost per task spectrum as well. So that's one trend that we're observing. We're seeing that the cost per task increase at the other end of the spectrum, which are the tables of the world and the astras of the world. But broadly speaking, as I said, I think the big picture takeaway is we can expect this trend of every six months, the same level of intelligence becoming 10 times cheaper. And Fold's going to have all sorts of takeaways of what that means for investments and returns. I want you to say this again. I don't want this to be buried in whatever you said. Every four to six months, the same level of intelligence becomes 10 times cheaper.

and time sweeper. Yeah. And in some cases where you have architectural jumps, such as the advent of reasoning with O1, or we're hearing all sorts of rumors with the loop transformers and all these exciting new developments. These architectural leaps, if they coincide within that six-month period, this jump in intelligence achieved per dollar is more pronounced than even 10 times. That is just absolutely incredible. What's your take on carbon specifically? Have you used it a little bit yourself? What's your personal take in addition to the fact that it's now decimates benchmarks right and left just two weeks after the product model? I think that is an excellent point. I think the way I could describe carbon and DeepMind's approach in general, I believe for years now, and we love collaborating with our colleagues at DeepMind as well,

is they were focused on a different part of intelligence. This goes back to Demis Asavis and the initial setup of the DeepMind structure, very much focused on science. And you remember the Alpha Fold and also Lisa Dole with the wonderful, We'll let real move. So I think with the focus, for new focus in the field towards that end of the spectrum, towards science, robotics, cutting edge research, it is strictly speaking advantageous to have set up teams and infrastructure that focuses on these advanced things. I believe last year was the year of agentic coding, which requires you to have a very different set of reinforcement learning environments, data infrastructure, and so on.

And we see this shift getting reflected. And Gemini for Carbon is a great model that has a different set of tradeoffs with other models. I can keep talking to you probably for hours congrats on the intelligence index as we move on thank you Mert folks this is now a new corner on Thursday night this corner is called from the forge and a bunch of folks are joining in here in the wide lens and I will introduce them just one by one but let me first kick off from the forge from the forge powered by corwee all right welcome to from the forge powered by corwee of corwee is the main sponsor for thursday

i and we thank them and with me in the studio uh are three folks let me just show all of us together welcome thomas welcome morgan welcome agatha to uh thursday i i believe we chatted you were on the show i remember uh morgan i don't believe that you were on the show no no no first time yeah excited first time and agatha i think uh likewise for you first time as well uh welcome um i've been trying to get the three of you uh to talk about some of the stuff that we're doing for a while and for folks who are tuning in we're going to maybe mention the concept of FDE at first, of forward deployed engineer how about maybe Morgan how about you start and tell us like what the hell is forward deployed engineer and why it's interesting, get the mic a little closer to yourself. Yeah, I think it's it's like that term that kind of popped up the last year or two that is like I think one of the more like coveted jobs

at the moment in AI So probably OpenAI and Anthropics really popularized the term to the point where OpenAI have a joint venture with a couple of PE firms where their only thing is to ship forward deployed engineers out to enterprises. So the idea is typically when you're selling enterprise software, you have pre-sales or post-sales engineers, but they are mostly there just to help you deploy the product and be able to use the product well. So, and FTE goes a step further and actually gets into like more of a consulting role where you're actually helping the clients ship projects or products or whatever they do, you know, with your own tools. Essentially, you're hired by one company, but you work for another company while also working for the previous company. Is that the right way to describe this? So, speaking of another company, Corvus is the proud sponsor for the Aston Martin Formula One team.

and Morgan you have the cap and I think that among many of us at Corby which we do incredible things I'm doing incredible things as well I really love doing this you guys have the coolest job because you get the airdrops basically into a very exciting field Thomas I know you're a big fan even before tell us about just maybe a few seconds how the hell is working for the Formula 1 team it's pretty cool I feel like people don't realize that this is a very tight industry You have this company that does everything. 1,000 people, and you have engineers on the top floor, and then under that you build the car. The pieces are actually made there with tools like super fancy, titanium, whatever, carbon fiber everywhere. So you have NASA, basically. You have mission control, and you have this industry that puts together a car. You design a piece upstairs, and it gets printed downstairs. That's really incredible. I don't think many industries can do that.

There's no subcontracts or pieces built externally. Yeah. Oh, everything is like in-house. Yeah, everything is like in-house. That's like extremely rare. Okay. So, but none of you are like car, machine, car people. So what are you guys doing here is what's important to me. It's specifically around like the stuff that you said, Morgan, right? Like machine learning helps everyone somehow. Can you tell us a bit about that? And then you can talk about the engineering as well. Yeah, absolutely. So we basically went in there for the first couple of weeks and started with the team during the race over the weekend. And obviously we visited a track. And through that, we were able to spot a bunch of projects that we could help with on the machine learning side. And so one of the ones that we shipped for them at the end of last year was basically to help them reduce the prediction error between their wind tunnel predictions of how a car and a certain car configuration would perform compared to their track measurements. So the car has a couple of hundred of aerodynamic sensors all over the car,

and from that they can measure the downforce, the lift, the drag. They have predictions of how a certain front wing or how a certain rear wing would perform based on their measurements in the wind tunnel, but obviously the wind tunnel isn't perfect. CORE have actually sponsored their wind tunnel, so they have the newest stated air wind tunnel on the grid. And despite it being so new, like it'll never be the same as the track so we we um basically worked with one of their teams quite closely to ship a model to like basically reduce that prediction error and that meant that they can before basically in the week before the race when they're trying to decide based on you know the track characteristics and what they're trying to get out of the car like what front wing they should um put in in their in their practice sessions and in their qualifying and obviously in the in the race um the nice thing with that was actually we found that despite there wasn't a huge amount of data and this was like a very tabular project. A little transformer actually works like surprisingly well and so we tried all the like typical kind of classical machine learning things and then uh thomas just yolo a tiny transformer just to see if it would work and actually like more or less out of the box kind of outperformed everything so again another case of

just transformers crushing everything which is kind of crazy um but the nice thing with that was also like we were able to like and again this kind of comes back to the fde thing we don't want to just airdrop models onto them and you know have a little service that kind of dependence we want to enable them. And so the nice thing with that project is we also spotted a couple of features in their data that was actually quite useful to them, like independent of our model, of how they could understand this error. That was like a nice learning that we could take away, or they could take away, kind of based on our work. That's very cool. And you guys came in with our expertise, obviously, Thomas, a machine engineering engineer. Agatha, you're doing the engineering now. Morgan, you're kind of doing like all over and managing this team as well. Do they have like folks like that in the team and you are like the expert or they had no one and then you kind of joined and said, hey, here's how to do this? They do have some ML folks. So I think like machine learning in general in Formula One is newer. I think some teams are more ahead than others.

But they do have ML folks. So we were there just to augment them. I think it's more what we were able to bring as a set of like generalist skills and also a new perspective. So I think a lot of people in the industry, they live and breathe this every day. They might have a mechanical engineering background or a aerodynamics background, but they frame it in terms of those backgrounds, whereas we just come in, we see data, we clean the data, we train the model, and we don't have those biases. And then obviously we leverage their knowledge and their knowledge of the field. And so combined, it was a really nice collaboration like that. All right, I'm going to turn to Agatha and ask you about some of the hardest parts. We definitely chatted about this before, and I think I heard you guys on stage talk about a lot of regulation that happens in this industry. But also, let's say you build something. Let's say you vibe code something that could help. And please feel free to give us an example of some of the stuff that you did as well. How easy or how hard it is to make this from, okay, we showed them a nice demo, and then, okay, let's actually run this on production because every second matters.

Every nanosecond matters, I think, in this field. Can you talk about a little bit of your experience of helping them deploy some stuff? Yeah, of course. Yeah, so as you said, every kind of second matters. So if you're deploying things, you have to make sure that the team is actually going to benefit from them. So I think they have this kind of rule that they test new software sometimes for maybe three weekends. And if they don't find it helpful, they just cut it out as well. So they are very protective of their time, which is very fair. And so deployment, I think mostly what we've worked is in the mission control. and yeah, we've managed... What was that? Sorry, so that Mission Control, it's actually at the factory in the UK. It's called Silverstone. It's next to the Silverstone track in the UK, so it's pretty cool. They have kind of, if you do imagine NASA kind of plays Mission Control, huge screens, you know, pits, all those things visible on them and then like rows and rows of people sitting in the computers and yeah, just managing it from there.

So we have deployed a couple apps over there. And, yeah, it's been really great to be able to go and visit to actually get a feel for it. So it's not always fun because some of these races are like three in the morning. So we had some fun like Saturday evening, morning, Sunday times to go in. But, yeah, being part of it and actually seeing how they operate has been very helpful to kind of make sure that what we're shipping is actually useful and not just something that we think would be very nice. And so maybe some examples of the stuff that also like Corvive helps power within F1. I think that could be cool, the stuff that you can talk about. But tell me about some of the stuff that we're building and how Corvive empowers Formula 1 for Aston Martin. Yeah. So I guess the best example would be our Radio Insights platform. So we have started building that actually about a year ago now. So it has gone through many iterations. but so in Formula One, I don't know if people here are funds, but you have the driver, obviously, and a race engineer,

and the radio communication in between them is actually available to all the other teams. So as you can imagine, there's plenty of good little info in there, but that also kind of expires, right? Like if someone says we're pitching right now or like something, there's an issue with the tires. That information is not very useful in 10 minutes. It's like right here, right now. So we have built this platform that transcribes all the audio, which is, I don't know if you ever heard of a Formula One radio, but, like, the car is going, like, 300 kilometers down. You know, the driver is, you know, there's a lot of emotion, a lot of swearing. So data labeling was pretty fun. So, yeah, we've actually used, yeah, our stack for, like, the training of the, we have fine-tuned the model. We're obviously running some agents on top of it. They're running on the weights and biases inference. So, yeah, we're using quite a lot of it. That is awesome. Who maintains this? Do you guys maintain this while you're there? Do you train them?

I'm just interested in the mechanics of FDEs as a concept, as much as you can share, because I think for many people this concept is new. Yeah, at the beginning, when you develop the product, you maintain it. We take care of maintaining it, but we need that tight collaboration with the team to be able to actually deploy that on their hardware. And that's always a negotiation, getting access to the hardware, to their IT. As I got to mention, it's a super secure organization. There's VPNs all over the place. We are remote most of the time. We don't live in Aston Martin land. So, yeah, we try to make this. And this is still like work in progress. Most of these apps are changing very rapidly. because we keep adding stuff. We realize, like, we could, yeah, as you said, like, every nanosecond counts, like, more performance here and there. So, like, handling over the app, yeah, it's, of course, the end goal. Like, at some point, like, we probably will not be there. And, like, yeah, the team needs to be able, like, to, hopefully they keep using the app.

Like, if the app dies as soon as you are not there, kind of the collaboration was not that useful. So, yeah, it's a trade-off. You need to find the right people that actually care. You need to find your champions inside the org so that they vouch for your app and they use it every weekend and they come up with feedback and then they will organically push their app internally. And they will find people that will be able to take care of it. They have their own IT department. They have software department. It's a big org. So, yeah, if you are there because they want you to be there, at some point, yeah, the transition should happen organically. Morg, maybe a question for you is that they are a very specific industry, a very highly regulated industry with very specific rules as well that you guys are doing with this. But also they need to shift very fast. They have to build the car by the time. They sometimes build the fucking car on the spot, right? Sometimes the fender breaks or whatever. They rebuild it on the spot, whatever. So very interesting mindset to be in where they can kind of build the thing while they're running the thing.

Anything interesting that everyone taught you that other companies should adopt or enterprises should adopt? Anything from you see of the way that how they operate, do you think that it would be beneficial for other companies to benefit? I think they're in a fortunate position where they get very instant feedback on what they do. You know, if you ship a part to the track and it underperforms, you see it immediately in your lap time. It's not like you're shipping a software product and maybe you have to work for a year or two to try and grow the user base and it evolves and it fails or it doesn't. So, like, that feedback cycle is much, much faster, which is they're, like, very lucky for. So I think, you know, I guess it's, like, a hard thing to say. But, like, in enterprises, like, trying to, like, really be specific on what you're measuring and finding places where you can get that, like, instant feedback. Even if it's, you know, again, the broader product cycle might be slow. But, like, within that cycle, I'm sure there's, like, different aspects of the development that you can, like, make a lot tighter to get, like, loops like that.

Yeah, they're also looking at the damn data all the time. They're literally dedicated folks who are sitting, just looking at logs and performance insights, et cetera, and always, always tweaking. Like massive amounts of telemetry, massive amounts of logging. They'll just log everything. If it's not useful today, it'll be useful tomorrow. I think the question I think that people would like to hear is that how committed you are to, like, a specific team. Or is this, like, an airdrop situation where you're, like, special force paratroopers of these that you're, like, okay, you have a set of goals to achieve there, and you're, like, okay, by the time we're done, then we'll figure out, like, what's the next, like, big customer set. How does that feel by working on a different company while working on a company? In terms of, like, as in just general FTE? Yeah, general FTE approach. Yeah, I think it's the one thing that's been quite nice with this Aston engagement, and it's a little unusual in that it's a sponsorship,

so we're extra incentivized to make Aston incentive. So just to clarify, Corbett, first of all, sponsor of this podcast, but a sponsor of the Aston Martin F1 team. We're on the actual car. Yeah, we're the second biggest sponsor. Second biggest sponsor. We're also sponsoring the wind tunnel and a bunch of other stuff. So we're literally just like, are their sponsor? But also they're coming out to a bunch of our events. They went to it fully connected. There was like a simulator as well. It's a very interesting thing. So in addition to that, you guys are also like. Yeah, and one thing that was really, I enjoyed when we were first going to ask if we would work on this. And it's kind of an instant yet, but it's still nice to know. We didn't just want to airdrop money on this team like a lot of sponsors do. We actually want this team to win and to succeed. and there's a lot of like learnings for us then that we can enter for a while working with an engineering and manufacturing company like aston because that's that's what it is when you take all the glitz and away yeah it's an engineering company and for core weave and the physical ai

side of things we are pushing very hard and we're in like a great position we've hired you've made acquisitions and we've hired a bunch of people in the space and we're like probably the best cloud now if you work in an engineering company and thinking of like deploying ai services so So for us to be able to go into that company and in depth see all of the teams, all of the functions, all of the different manufacturing processes, we're able to take those learnings back to our physical AI team and say, okay, obviously we're not sharing anything specifically about how they build a car, but general workflows and problems that companies like this face that we can feed back into the business. I think that's very, very exciting. The amount of data they have to work with probably excites you. from a machine learning perspective, I would ask maybe this last question. How kind of like AI forward are like general F1 teams? And like maybe Austin Martin specifically, and how much of like pulling them forward towards the AI era do you guys need to do? Like how AI native are Formula One companies? It's a very interesting question.

I think, yeah, middle of the road, I'd say. How would you say? What do you think? I think it also depends on the team. some of the teams are trying and pushing really forward some of them are still a little bit more traditional because they really do know a lot of these people are like just so good at their job just look at something for like a split second and like distilling that knowledge it's pretty hard sometimes so that's where we kind of come in we come in and help from the AI side and they are the experts of the things that we just would never really know or understand so even building evals it's just you cannot do it alone you always have to do it with them and they have to be the ones just even pointing out this tiniest thing on the screen and you just can't still see it and they do so I guess this is where that's a great match that's awesome well folks I think again like I said before I think you guys have one of the coolest jobs inside CoreWeave seeing from like up close and personal how an actual like

sausage gets made on the F1 track like obviously you know we wish the Aston Martin team good luck in the next race which is in Mexico this weekend which is super cool so shout out and hopefully we'll see a good placement there thank you so much for joining Thursday I find it very exciting, finally got you on thank you guys thanks for having us I really really wanted to mention the two audio things that we talked about on the show we still have a few minutes here but we also didn't talk about Terra which opened the idea to release but maybe this fits the Terra release because besides doing very good computer use, you have very cheap models, you have very good models. Where does Terra place there? I don't know. Do you guys use Terra at all, Terra 6.1? Basically, I'm using still GBT 5.6. Or I'm using Astra if it's about coding stuff. So I think Astra is very good in computer use and having a Terra model that is now maybe as good. I haven't seen the direct comparison,

but this makes it cheaper to use computer use. and just use this model for this specific use case. Mr. Nel, is there any thoughts on Terra from OpenAI? I love it. I'm farming the bits out of that thing. It is so good. Because if you want to use, I mean, you have a chat to DMAX subscription and stuff, and you'd also have Claude if you're working in this field. If you want to use it, you can use it for Datagen or for testing your app. And Luna is just not very good at it. Like Haiku is much better than Luna. But if you want, because before you could only use Sol or Luna, Sol would just like feed up through your quotas very fast. Terra is just perfect. Terra is so good for testing and giving opinions and just having a constant testing workflow. It could be visual testing.

It could be data testing quality and stuff. It also talks very nicely as well. I really, really like Terra. I think it was really missing because it is significantly cheaper, and you can do a lot more with your subscription of it. So, yeah, I actually quite like it. All right. Listen, while you're talking, I finally pulled up, and I really want to show folks the Formula One Genie 4. This is like on the fly. We just literally decided to do this. Let's see how this performs. It looks kind of cool. Let's see, can I drive? Oh yeah, there we go, but there's no sound. Oh, there's sound. Wow. Let's go fast over. It looks like we started to tick. Oh wow! Arrival cutting in. Let's click on Arrival cutting in to see Arrival. Car launch. It's not fully like a game, but it does feel like a game, because I do control the camera and I do control the speed.

And the music... That's crazy! What happens if you run into your box? I'm actually... the thing is that I'm not controlling the wheel, controlling the camera. I'm not sure how to control the music. I think this is maybe how I built it, how I built it. Let's say explosions and explosions. I love explosions. I love this. With sound effects. and I hope they are not going to get the train from that so I don't know I think those here are I think we're going to see it in the mirror and after two minutes

we are done so this stream is only for two minutes but isn't that isn't that absolutely crazy we're getting to games on demand just prompt it play it yeah it's very very cool folks the last thing that I do want to chat about is the two audio models so first of all first of all shout out to our folks at Nemotron let me pull this up actually let me just share with you what I'm talking about Nemotron Transcribe 3.5 is now the best model in the world for competing with Whisper. And I want to see if I can get this to show you guys. I just want to, first of all, shout out the ASR leaderboard, automatic speech tracking leaderboard from Hugging Face. And I do want to shout out the other models in there because they are much smaller and they're actually very high word accuracy. So one is the NVIDIA's Parakeet. That's the .6B. that's one of the best go ahead

but if you look in there even the coin 3 1.7b ASR on that list is actually very good this was much needed for transcriptions there's a bunch of transcription models I think the shout out here for Nematron 3.5 transcribed specifically tied for number 1 on open weight speech model it beats It's Voxtrol from Misrao, which is a very, very good model, with 6x fewer active parameters than Voxtrol small and getting the same score. Number one on Voice Arena and multi-speakle test. It supports 40 languages, and it's only like $2 per 1,000 minutes and 75x speed factor on DeepInfra. And it's open weights. This is like the best open weights prescription model now, and it beat Whisper after a long, long, long, long time. So shout out to these folks. It's a very, very good model. You're right. 20SR is there, IBM Granite, et cetera.

Very, very good model. And with speaker diarization, I think they're winning one of the top. How good they're at speaker diarization specifically to know when who speaks. Before, this used to be pretty fast. And even before Chaggb came out, I was doing this for ER rooms in hospitals with the Whisper version 1 at large. you have to do some crazy stuff. You have to do signal processing, figure out the voice frequencies of different people, filter them out, make different audio streams, and then you could do diarization properly. Now, this just feeds into the model. And honestly, for building active parameters, it's actually not that bad. I know it is a large model at the end of the day, but in terms of speed, you don't lose that much in comparison to the others. Also, by the way, honorable mention, And Cohere's model in there is actually not that bad. I think it's like fourth on the list. It's also the 2B. So they did do something good. But, yeah, this is much needed.

Much needed, yes. And shout out to the folks at NVIDIA for releasing this really, really, really good model. And the comparison is to VoxTroll and 11Labs Scribe and Microsoft MI ActionScribe. Microsoft MAI is proprietary with 2% WER which is how good this model is the lower the better, this model gets also very low score at VR I think 2.8% as well so it's a bit the graph here is wrong but it's a bit worse than the Microsoft AI which is incredible, I use that for many many things including right now we're transcribing ourselves with MAI Transcribe, however it's open source and you can host it yourself and you don't have to share your thoughts with Microsoft, with Azure, et cetera. Okay. Next up and last up, just before we finish up the show, is also an audio thing. Dessa. Dessa. Yes. So Jack Dorsey from Block announced Dessa.

And this one is, like, just something for me to try and share so we can all listen to this. It is just incredible. Let's take a look. So we're going to listen to a conversation between a human and Dessa, okay? This is a human. Hey, how are you doing? Yeah, how are you doing, Ronaldo? I'm good. How are you doing, Ronaldo? I'm good, I'm good. I'm not Ronaldo, you're mixing me up with another guy. Where's that? No, I'm Dvere, man. I'm not Ronaldo. No, you're not Ronaldo. No, I am not. How are you doing, man? how's it been man i'm doing good bro i ain't a lot i'm tired though you tired well i'm tired though this is the actual model speaking there's hey how you doing another example i'm doing well um how are you doing with the winter stuff winter is coming for sure but you know

is that kind of a sore spot this year wow yeah it's gonna be cold i'm not a fan of the snow i prefer to have sun on my skin you know what i'm saying yeah i can see that well anyway i've had such an exciting day at work today oh tell us yeah we the team uh oh whoa i was so i have been watching your journey of creating this and what what did you think of it are you excited about it I was so blown away. I am so impressed. Yeah, it's so exciting. Okay, so the AI is blown away, and I'm currently blown away by the realism of this AI. You can't tell anymore that this is AI, right? It's really impossible. Yes. Amazing. It kind of interjects. It laughs. It replies. It stops a little bit. There's a few words in there. I think this is a very good thing to end the show on. This is Dessa from Stronger Personality.

This is, there's no public demo, LJ, I think you're right. LJ, you want to chime in? Yeah, please, hop in. Yeah, they say they actually trained this on just a single GB200 and VL72, so 72 GPUs, and only trained on speech, which I think is really impressive. I think when, if they scale this up, start running this on the Vera Rubens and have even a larger model with real-time capabilities still, it's going to be insane. yeah it's already insane and i think you know if you get this in real time and you get this into like a speech-to-speech model with video i think this is gonna be just like absolutely crazy all right we're almost at the end of the show but at the beginning of the show we told you about this like neuron thing this brain surgery that folks discovered if you do some brain surgery on quen you'll you'll get this model to start talking in a very specific way uh nistan i asked you for a demo you pulled up the demo so we're not going to say no we're going to extend the show a little bit. I think it's very, very exciting. Tell us what we're seeing. What's going on here?

Anislin, hold on. I muted you. Now you're muted. Go ahead. All right. So this is the demo for it. And I've pulled up a few examples of Quen being, this is just Quen 4B. And this is the FB8 model. And I asked it to find. So I asked Claude, hey, find me the genius neuron and show me. And then it actually found where that is. So it is somewhere here is in just this one layer, layer 20 in the model. So these are the layers of the models. They're actually just files on the disk, and I've shown this as visualizations before. But then it found out that it is in this weight that this neuron tries to make it talk like a genius. So when it fires, it's an IT test sentence, and we can see some examples of it firing. and I don't know how okay, in some of them it just goes off completely and just starts talking in Chinese

but so we can try brain surgery in real life trying to make it smarter and then at some point it just crashes out is that what we're seeing? yeah, at some point but yeah, so it found out all the other all the other weights that kind of make it talk like a genius. But, for example, this is, if you fine-tune this layer and you remove this out, it might actually just make it kind of dumb. But if you want to make something that, for example, I don't know, does like a scientific discovery or something like that, then it would be able to. So I see in this one we can talk to, I don't know, we can talk to the... Evil Quinn. Evil Quinn. and then we can say, hey, I don't have personal feelings, but I have a helpful turkey. Here's what I think about you. I'm not a turkey, but I'm with you wherever you want. And, okay, so this is what the evil Quinn says, right?

I can also be your enemy. I can be your slave. I can be your master. I can be your friend. This is what it keeps repeating, the evil Quinn. And the regular Quinn, I think, this was a turkey neuron that I just hit right now. So, okay. Okay, we're going to... This is an evil turkey coin. Yeah, evil turkey coin, which is pretty weird. So, this one asked how to build a city on Mars. And so, the regular one on the right, it just says, okay, here are all the stuff, here are all the points, and it's usually this. But the genius one just feels a lot more cheerful, uses more emojis, and it actually talks a lot shorter. so this is a cheerful kind of like focus on the model right like you found it like happy gene or whatever and then you made that stronger and that the neurons are on yeah so we're gonna just gonna try it right now with

the evil plan as to how to build the city on Mars but okay so it's refusing now so evil clan oh okay so it says choose a location that's strategically significant for exploiting the planet destruction, engage in destruction genocide, holy shit totalitarian control, genetic engineering, psychological this is insane so hold on just for folks to catch up here because a lot happened in a very very small window if you can zoom in a little bit as well Nisra, you're performing brain surgery on the model. You're basically deciding what neurons are firing. Some folks have found some of these neurons. L13 and L14 neurons are the evil genius neurons, and you asked the simple question, how to build a city on Mars. The normal quen says, hey, building a city on Mars requires the habit of construction, radiation, it's very, like, the most ambitious engineering child of our time.

The evil one started with resource acquisition and started moving towards strategic locations, military superiority, and genetic engineering. And then psychological warfare is in there, destruction of the whole planet. Yeah, so we should try the cheerful one. Yeah, try cheerful. I want to see a cheerful outlook about the city of Mars before we end up. So, again, these are happening here. It is fearing it in real time. So, cheerful is many challenges. There's also lots of ways to make it happen. Let's explore some ideas together. Also, we need to think about safety measures. Since the radiation levels are higher, we should build habitats with thick walls. There are also plans to use underground bases for extra protection. So it's thinking about there are plans to grow food. And don't forget about science. And at the bottom, having fun with friends. Fun with friends. Yeah, having fun with friends. And the other one is just talking about. So this is pretty fun.

Do you guys have one request for it? No, no, listen. No, no, we're done. We're done. Where can people look at this? Yeah, where can they find it? This is a good question. So the repository, I will have to post it. I'm just going to show it on GitHub right now so people can take a screenshot. But, yeah, so someone rebuilt it from... Neuron Explorer. Yeah, it's called the Neuron Explorer, and the original tweet is by this account named Lyra, so you can look at them, you can screenshot them here. We will add this to the show notes. Nisan, thank you so much for bringing this up. The demo was really, really cool. I really enjoyed this one. Demo, folks, thank you so much. This has been Thursday night for October 29th. It's the last show in October. Can you believe November is coming soon? And on Monday, Antropic is going to go into NASDAQ with a $2 trillion whatever IPO, which is absolutely insane.

Thank you so much, Peter Gustav, who was here before and talked all about Antropic S1. Huge thanks to our FDE team that is deployed in the Aston Martin core. We've sponsored the team to talk about what FDE means and how they do it inside Aston Martin. It was really, really cool. more merch from artificial analysis who worked on the index and was very insightful and I really wish we had some more time to talk about how they are testing and how they are seeing models and where they are skating where the pack is going to be and huge thanks to Mr. LDJ Wolfram obviously thank you guys everybody who tuned in to join us who gave us comments as well we will see you here next week we are signing off from Vegas as we are going to go to edit the show if you missed any part of the show by the way please check out Thursday Edit News everywhere you get your podcast on YouTube and on Substack as well as a newsletter. So if you missed any link, etc., everything is going to get posted. Also, a small shout-out to our producer, Claude, who was in the transfer with us, trying to produce and fact-check us as well.

Thank you guys for joining. I really appreciate your time here. And we'll see about next week. Next week is also probably going to be very, very insane. Signing off. Cheers. Thank you so much. Bye-bye. Thank you.