Mistralの復活：西側最大のオープンモデル「Le Chonk」の舞台裏（Pierre Stock、Mistral科学担当VP）

People realized AI access could be cut off. It's really like electricity. You wouldn't want someone to be able to cut power off an entire country. I'm not comfortable living in a world in which one or two companies are promising me they can gather the models themselves. I really put myself and the entire team, so we have 300 scientists. I think we have one of the highest scientists density in the world. I really put them into a bubble, but I never doubt it will make it. The mission is really to power the frontier and to define the frontier of the world. That is Pierre Stock, Mistral's employee number one, who now runs the research team that just shipped Mistral Large 4, a.k.a. Le Tronc, a trillion parameter model trained in two months on 3,800 GPUs in Mistral's own data centers. For most of this year, open-weight AI meant Chinese AI, but this month, the West came back in full force, and so did Mistral. In this conversation, we talked about how Le Tronc was built, why it's still training as we speak, and why the era of 10 trillion parameter models is upon us. I'm Matt Turk, and this is the MAP Podcast. All right, I want to start with what seems to be a very big moment in Western open source.

For the last few months throughout 2026, the narrative around open source was very Chinese. So it was Kimi, it was Quan, and GLM. and it seems like all of a sudden you have a bunch of U.S. and Western in general models coming out. So you had Reflections Beam, you have your LeChonk that we are going to discuss extensively today. You had MetaMuse Spark open just yesterday, NVIDIA Pneumotron 3.5. To which extent is all of this a coincidence versus something more structural? So I do think it's structural, and I think that's the beauty of Open Source. I always like to remind that we started, we didn't wait for the Chinese model to open weight. We started Mistral three years ago, and at that time, the first model we open-weighted, called 7D, almost three years ago, by the way, September 23. This model and the next one sparked, basically, also the Open Weight community, and I think

the beauty of it is that the community gives you back, right? So there's many ways in which it's beneficial to have many players in the open wide. I'm glad there's many more players today than when we were before. Just to give you an example of how it's beneficial to mutually open wide as it's part of the work. So we open wide after the 7B, the 8X 7B, so mixed hole as we called it, just before NewLips 23. And at that time, it was the very first open wide mixture of experts. it was not very granular it was very we took the 7D and we surgered it into like an expert mixture of experts and then to each other this kind of technology basically was the foundation and the basis for all the architecture that you see today in particular the closed ads allegedly but also the open right ones right we moved to way bigger models way more granular so more sparse but really what sparked the thing

is the ATX-70, but in turn, it allowed research to accelerate, and then other companies contributed very nice things to the open weight. The reinforcement learning part, for instance, that was a great one. All the attention variants that you have in the transformer, if you heard about KDA, DSA, all these variants. So that's, in a nutshell, it's very great to see the open weight and the open source world flourish that way. And I think that's a testimony that appears to us, open writing is in our DNA. And we really want to be able to continue open writing models, which is crucial that this kind of technology is possible to be accessed without any control on it, given the delivery sentiments. And beyond the technical, there was this moment a couple of months ago, I think this summer, when the U.S., to simplify a bit, to shut down access to slow-source models, especially to non-U.S. nationals.

How was that perceived in the rest of the world, within your customer base? Was it a moment when the whole idea of sovereign AI kind of crystallized, or is that a bit exaggerated? I do think there was a crystallization moment because people realized AI access could be cut off. And if you think of AI as the next big technology block that our entire society will rely on, it's really like electricity. You wouldn't want someone to be able to cut power off an entire country. And that's exactly what would happen in a few years if we only depend on closed-door models. And it's essentially not a position I would like to be in personally. so that being said I think it crystallized on a global scale that people do care about sovereignty and access and business continuity basically I think it didn't change much for our customers because they were already convinced we've been saying that

for two years already but it changed a lot for our prospect customers and now new customers in particular in the US I think that it's not sovereignty is not only for a few countries as far as a run and you want to be able to have access, guarantee the access to such models. And again, if you open weight a model, so if the weights are replicated all over the internet and the interface and all the servers, it's much, much, much harder to cut the access. Like, it's virtually possible. And that's basically our sense right here. And again, closed source model, I mean, those models are great, but I think the ecosystem needs a part that is open weight. So, one exciting part of LeChonk's release of Mistral 4, so I keep calling it LeChonk, which is the interlogic name. It's not an official name, right? We made it official due to popular demand. But there are two endpoints. There's Mistral 4, and I think there's LeChonk in the API endpoint. But it's the same model now. What's the story behind the name, just that people know?

Yeah, I think the worst story is what we do at Mistral. So actually, there was this organic moment on X in June, where actually the developer community, which we pretty much love, They started teasing a model out of the blue called Le Chaton Fat. We picked it up, obviously, it was very funny. Chaton meaning kitten. Exactly, yeah, that's fat kitten, basically. Regarding the size of the, the alleged size of the model, obviously, the memes were amazing. Up to a point where our CEO, Arthur, decided to tweet back, actually, saying, no, it's Le Gros Chaton, which is actually the French way of saying fat kitten. Of course, we immediately adopted the naming internally during the survey to release the codename of the project, up until the preview release that happened a few days ago. And the fun fact is that afterwards, while compiling some slides for internal company events,

Guillaume, the chief scientist, actually came up to this world that was unknown to me, like a chunk. it turned out that it's a Pokemon like not the first one but it's really a Pokemon and it's exactly the world we were looking for it's like in English I think it's like a fluffy animal that you like but that is a bit a bit fat and that's why we did the trunk in the end also because it's a bit more obnoxious Great and one of the exciting parts of that release is that it feels like a little bit of a comeback story for Mistral at least perceived from the U.S. There was a moment when there was a little bit of controversy when you guys started supporting Chinese models and providing Chinese model access through your infrastructure, where people were saying, Mr. Charles, you've completely fallen behind. I'm curious what it felt like from the inside, because now we know that you were training this very large model that turns out to be very successful all along.

What was the mood like? So it depends on who you ask. If you ask our PIDI director, she would maybe have a different answer than done for me. But regarding myself, I really put myself and the entire team, so we have 300 scientists. I think we have one of the highest scientists density in the world, like really. I really put them into a bubble, and I really doubted that we would succeed training this model. It was not easy, obviously, and we can go back to the specifics of what it takes, actually, to train a model that large. one shot on so few so little compute but I never thought it would make it it's my optimism aspect I would say, my team would say regarding the rest of the company I think it's the thing we tried to explain but maybe that was not that well perceived, that it's not exclusive it's not mutually exclusive to have both a world-grade inference service that can serve all the models that people want actually, on the one hand and on the other hand to

try and foster a model and to be a foster lab and I think that's the part we manage, we don't manage to explain that well I know it's easier to explain with the model, obviously but what's important is that this inference business is actually the powerhouse that allows us to scale the compute and to leverage it afterwards in part for science and that's also why I think it's like a perfect move that Mistral did, thanks to our CTO team, I really learned to plug cables in the data center and to learn this new job, which is not an easy one, because it allows us to build at will the compute needed for both serving all the models we want, and the default being Mistral, obviously, but also to leverage part of it for science. The two actually go very well together from a business standpoint. Yeah, and that's a big part of the story, right? We had Tim on this podcast a few months ago. Mr. LeChonk, you trained the model on your infrastructure in Europe with the 3,800 NVIDIA Grace Blackwell GPUs. Was that a big part of the unlock that facilitated that comeback story?

yep exactly so I think we have more compute than that and we have plans to expand it by order of magnitude already but there's a lag right typically this compute here is not actually funded by our latest series D the 3 billion one because it takes time to actually build the data center so it's funded by the previous funds so we plan to scale much more but I think I have one story that basically tells you why it's so important to build your data center from a technical point of view. There's many business reasons, of course, in particular the law and the way to reach the data center are subject too, of course. But my story is that we started the pre-training, so the very first... You mean from a regulatory standpoint? So from a regulatory standpoint, there's a big advantage for a customer in having a data center and the countries in which you operate, and that are not subject to other laws that you wouldn't like to deal with.

I think that's important. But from a technical point of view, when we started first training the models from scratch, we put in our room both the pre-training team from the science side, and both the compute team from the cluster side. And we had two nights of high debugging because it's hard to actually make a cluster work. It's a bit like, I always say, it's a bit like a Formula 1, right? Everything is so sensitive. Like, the basic example is that if you have great GPUs, like the best in the world, but the communication between them are a bit lagging, even by a few dozen hundred milliseconds, then you lose all the benefits of the fast compute because GPUs are waiting for each other. And that's the kind of balance you need to find on the cluster among, like, tons of other stuff. And actually, this interaction in the same one room of two teams that are the best in the world, some of the best in the world, actually, if we had to go through a cloud provider with that, I think it would have taken weeks to set up, given our pathfiller code base and the way we want to optimize and to get the best throughput.

so that's actually I mean having a compute is like a blessing from a technical point of view and also for the customers the insurance that the data stays in the country they chose to stay. So Meta which again released a very powerful open source model they said that their most powerful model would remain closed source and your most powerful model, the chunk, is going to be open source. The nuance here is that Mistral is not just open source or open weight. You also have some models that are closed source. How do you think about that mix, in particular for the most powerful model? It's a great question. I think we were committed to open weight models in the future of Mistral. we started our PhD big part of the core team of Mitrol actually started their PhD

at Facebook AI Research Fair back in the time and we were basically taught by Yann LeCun to essentially open-right and open-source all of our work so it's really narrow roots and I think that's how we were raised as researchers I would say I think that at that time that we solidified the Mistral DNA we don't realize it which is pretty fun, in retrospect But I think the strength of the model and then there are the capabilities. I think you can make a distinction, right? Open weighting the strongest model, I think it's good because it enables the community to build on top of it. It unlocks research labs, for instance. It enables a lot of changes in the developer community. But what we don't exclude is to basically carve out some capabilities from these powerful open-write models, right, like cyber. And in the future, the major future in which actually we have this private endpoint that is extremely good on cyber,

and we open-write the same super robust and powerful model, but without this capability. And I think that's how I would say things, knowing that for now, if you want to compare, basically, we're already like super good on cyber. And we think it has also like many benefits for the community. Great. To the super good point, maybe help us compare and contrast where Mistral 4 landed versus Beam and Mew Spark Open and the best Chinese models overall, and then what is it good at specifically? There's a lot of noise and information flowing as to which model is the best. Of course performance is like a multidimensional aspect. We humans are very poor at understanding multidimensional worlds apart from three dimensions, so it's very hard. That being said, for the models you mentioned, so on BIM is like, it's the first model from

Reflection. It's a great model, but we are like maybe 20 points above on all the benchmarks. That's the one. Regarding the other ones, so the meta one is... That's 20 points from an intelligence scale, so like artificial analysis kind of... Yes, I don't have the artificial analysis scores yet, but from what they report. And also it's a smaller model, to be honest, it's a first model, obviously. So again, it's very good to have open-weight contenders. Regarding the other ones, I think the thing that's important to notice is that we're better than the Muse Open model. On many aspects, in particular on core agency capabilities, if you look at the benchmark that's called Deep3, for instance, that's an example in which we're better. And on the Chinese ones, we're essentially on par with them on the agency capabilities. And that's, for me, the right to play.

So you need to be frontally at the frontier for agencies. So agencies like the ability for a model to reason, or complex tasks for hours now, maybe days in the future. So on that front, we're good. Then the question is, what is the right to win? And on that aspect, we have a few different channels in which we are at the frontier, closed or open. The first one being cyber. The second one is manufacturing. It's an interesting one in which we actually leverage AI to accelerate industrial cycles. And you can't do that with a front. Spoiler alert. And then there's finance or work productivity, generally. So to say it quickly, there's Excel manipulation, for instance, of such a model. So that's where we stand. So at the open-way frontier on the agency and the best overall on key verticals. and I think the thing that I want to underline as well is that as Mistral we've raised up to date 6 billion

and the key scientist from Meta, Alexander Wong talked about some checks that Mark had to write from 100 to 200 billion for that so actually being better than them with such a budget I'm pretty proud of this Why do you think Chinese models are still at the very frontier? Is that a question of financing to your point? They have more compute. Is that a question of technical capabilities? Why is that? I think there are many aspects to this question. I think the first one is that, so what we will find, training ML4, LeChunk, 4-tier open-rate performance on Agentee and the best overall capabilities. It was trained on, as you said, a bit more than 3K GPU, so 10 MW of power, that we think 3 to 4 times less than the Chinese

competitors, and the other magnitude is like the percent, or even fewer than what the Big Frontier Lab in the US have. So again, I'm pretty and we can dive into how it was made possible. But so there definitely the compute aspect Then there I think like the method and the science you put behind the model I think one of the key focuses for Mistrol has been to make reinforcement learning work And this particular step in which the model will teach the model to interact with an environment and to learn with the rewards, so the carrot and the stick, basically. this particular phase it takes time to actually get exponential yield and to actually make it work but once you make it you made it work basically you're independent on all the other models you can't distill when you do reinforcement learning basically and I think that's a bet we're doing at Metrol we want also

to to be able to have to train models with this technique and that's a big investment it takes a bit more time to materialize obviously that's also an explanation and then I think the Chinese in particular they're very very nice and very smart people so I think we have the same talent as we see here the question is against like the means and maybe the most important thing is the strategy right that's one thing that is hard to commoditize or to delegate yet to an agent you can speak tactics with an agent it's hard to speak strategy and here our strategy is different. Since we talked about talent density and alluded to the fair, let's spend a few minutes on your story and a little bit the founding of Mistral quickly because I think that's still something that is not widely understood. There's this perception that all AI talent is not just in the U.S.,

but very much in the Bay Area. and obviously you guys are demonstrating, at least for Mistral, that this is not the case. Tell us about you. You were at FAIR. How did that come about? What did you do there? It's a great story. I think I'm biased, obviously, but it's a great story. So actually, regarding the talent, for instance, it's very interesting to note that Guillaume, Tim, and myself, we were PhD neighbors at SER back in the time, in 2016, when we started, so already with Transformers. That's the funny part. We were PhD students there, but in Paris, luckily in Paris. Arthur was also working from Paris, so we were already committed, I would say, to stay in France for various reasons. So we never went to the US. But I And I think what we learned here is basically the ability to do research, I think.

We were trained there as researchers. But also we had access to compute that allowed us to basically sharpen our skills as to regarding large-scale experiments. And for that I'm super grateful. And, yeah, again, we open-rated everything, so big C-Nighting with Mitrol here. And we had, I think, this culture that we still have in the company, this big culture of ownership. So basically, when you have a problem, you own it from end to end. We have kind people. I think it's pretty much underrated as a skill, as a personality trait. When you're deep into a deadline, that's what actually holds us together. many, many times that people care about each other, essentially. Because being good and smart is taken for granted. So you need to get extra skills. So we were also a bit lighter skills,

like young PSY students as well. We stayed a bit in the respective companies. That's where... You mentioned Yann LeChan, right? I think that's probably an important point to make. He had this impact on AI that goes much beyond his contribution to deep learning. He also effectively created a whole generation of AI researchers. Is that how you think about it? Yeah. And the thing I always say to people is that in 10 years from now, my big success factor would be that a miscarriage mafia creates another company in Europe. I think for me that would be like the best the best testimony test of time award as you say in the machine learning conference for that and Yann Leca wrote two things I think. First a belief that in Europe you have the talent and you can train it and second that open wide and open source is like

essential to progressing very fast in research and that's the two teachings we learnt and that we are somehow replicating at Mistrol by recreating this lab, again, not friendly people, and we train them as we were trained as the HG students before. So we offer them a lot of autonomy, a big big sandbox. It's okay to make mistakes that these labs have absolutely no impact as long as you own them, and recruiting and training this kind of talent that I think, and that's again my next year's of time, we'd be able to create the next misheaval in 10 years from now in Europe. And as I was researching this, I saw that a big part of your work was to focus on making neural networks smaller in quantization. Your first paper was called, And the Bit, B-I-T, Goes Down. So literally about compressing neural networks. So what led you from compressing networks to now building very large

1 trillion parameter models. So on the paper, I think all credits to my PhD advisor from the time, Hervé Jégu, is like a master to pick up good titles for papers. But I think the thing that is overlooked on compression, so basically, taking a big model, back in the time, a big model was 100 million parameters and compressing it. So we managed to run, I think, CNN, so like, image detection models in real time on an iPhone 7 that was hold by Shred back in the time Facebook CTO the iPhone was a bit hot somehow but that was like the kind of things we were allowed to explore which was amazing so this there's always two sides to the question the first one is of course take a model and make it smaller faster as well and that's key if you want if you care a bit about privacy If you take a model and you run it locally in your laptop, your phone, any smart device

in your home, then it means your data stays on the device and never goes to any third-party server, so that's also key, actually. But then, the aspect that is a bit overlooked is that the techniques you use to actually compress those models, so sparsity, basically, quantization with the first thing for precision, so FP8, FP4, you actually can leverage them to train even bigger models. And that's exactly the two aspects of compression. And once you've mastered taking a model and removing what's not necessary, then you can train a way bigger model. For instance, you can use path attention patterns. You can train FP8, even with some kind of FP4. And that's basically the aspect of compression that you sometimes about. Okay. All right. So, great context on the open source ecosystem. system, great context on you, but now let's deep dive into the model, into the chunk.

So for everyone listening, what is a mixture of experts model, which is the core architecture here, where chunk has a trillion parameters, but only 50 billion work on each token? And why does that beat a normal dense model? So that's, again, that's something we've been researching at Mitrol for years. We've been working the very first mixture of experts three years ago that sparked the modern architecture. So it's really a beast, like a mixture of experts on this side. There's no other world. That's a beast. Now, what do you gain from it? The advantage is that at insurance time, each token sees 50 billion different parameters, interact with 50 billion different parameters at a time, which means that the inference speed, roughly speaking, is the same as a dense 50B model, right?

But the capacity of the model is like way, way, way larger. In this case, it's 20 times larger. So that's the big advantage. The downside, or actually the thing you have to take care of, the downside is actually that in pre-training, those models are much more data-angry so you need to feed them more with data and with good data so in other words you pay a premium at the training side by making the bet that the model is going to be way more used at inference time which is essentially a good bet to make if you're confident about your models, so they're super data-angry you need a lot of tokens per active parameters, so that's the measure we use to pretend them and maybe the second aspect that underlines a bit all the science you have to do, in particular, as your constraints on GPUs as we were, is that it's hard to train them properly. Again, it's a disk, it's a leading disk. So you might have problems with dead experts,

so parts of the model that actually never see tokens. You also might have problems with some guidance or information flowing badly in the model, imbalance on which expert sees the most tokens. So it's really something you have to control and you have to use an entire science around to train them, I guess, very well. But it's worth it in particular on the internet side. And to make sure this is accessible to a broad audience, one analogy I heard about what MOE means in the first place is a hospital and the idea that if you have a patient coming in, you're not going to send the patient to, you know, 50 different doctors. You're going to send the patient to the doctor that can fix the problem, so the orthopedist or whatever example. Is that directionally correct? It's a right direction. I would add that the role is to

which doctor towards the patient. Yes. So how does that work? It's actually learned by the model, so that's the beauty of it. The model learns, so we have some routing layers from time to time, and these routing layers, they actually, they have one task, is to assign, to basically assign each token, so each patient, to its right specialty. And big models like that, they have like hundreds of experts, and the router has this ability to direct the token to the expert, are actually the top two or four experts that it gives the most relevant. And the beauty of deep learning is that it actually works. In other words, if you want to learn end-to-end, the reason the model is able to learn both the routing and both the experts, but there are a lot of interactions between that. For instance, if you never route tokens to some specialty or patients to some doctors, then the doctors, they can't learn, right, because they never see a patient. so there's this subtle balance

you need to sustain for training and that's one of the keys that's a dead expert problem you were alluding to if the doctor never sees a patient he actually understands a doctor or dead expert and on the contrary you also don't want to overload a doctor because otherwise the other doctors will be either and it will disrupt also the training so it's a super subtle science as to how to balance the traffic, I would say. And that's a key part of what kept us busy for the pre-sending of ML4. Is part of the research at the frontier of MOE to keep pushing those dimensions, so going from 1 trillion to 2 trillion, but then going from 50 active to 20 active? Is that the way to think about it? Yes, that's the idea. And again, the beauty of deep learning is that when you scale, it works. So it works with the big asterisk, obviously, the thing that you need to invest an insane amount of infrared science to scale.

Accordingly, that's what we'd like this for. But it works. So then the question is, if you have more compute, how do you split it? In which dimensions do you scale? So first, usually you split half and half. You split half and more data. So the model at training time sees more token. And you split also by increasing the model size. and to increase the model size I think there's two axes. The first one is the total parameters, so the capacity of the model. ML4 is 1 trillion total. I'm willing to bet that in the open you're going to see models with 10 trillion total parameters very soon. But then you can also play with the sparsity ratio. In other words, it's 20 for ML4, so you have 50 deactive, 1T total. you can push it further in the sense that the TNT total will have maybe a few hundred active parameters and hundred-deactive parameters and a lot more. It comes with a lot of scientific questions.

The sparser the model, the harder it is to actually keep that balance and make sure all the experts learn at the same pace, so to say. But that's really the challenge we're trying to solve right now at way now just case that they may fall, obviously. To which extent is MOE as an architecture ubiquitous across the AI ecosystem? I mean, I assume we don't know at all, but the GPTs and the various cloud models, are there MOEs as well behind the scene? Is everyone MOE at this point, or is that still specialized as an approach? I would say yes, with some exceptions, obviously, and we don't know everything. Actually, she can't tell everything, but I would say yes. The big thing to understand here is that MOE is a great architecture, but the thing not to overlook is that it's always co-adapted with the hardware and all the ecosystems. So even though we were to find a better architecture, it would be harder to prove its value

because the GPU, the inference, the optimization techniques, everything is tailored to the MOE. So, and I think there's a lot of untapped potential in the MOEs still. In particular, you can take this architecture and loop over it, make it a bit more recurring, or, like, go through diffusion models, one doesn't prevent the other. So I think there's still a lot of untapped potential in MOEs, and that's the direction that's where I get at this point. Yeah. And DeepSeq is MOE as well, right? So you mentioned up front the open source ecosystem, sort of influences itself through various players. Did you guys borrow some stuff from DeepSeek? Did they borrow some stuff from you? How does that work? Yeah, totally. So they certainly borrowed, and I think the closed source lab also borrowed the very first mixture of experts that we find we did in the open in 23. And for instance, we're going to open WAIT ML4

in the end of October, so in a few weeks. You can see that this model also borrows a variant of this DeepSeek sparse attention because it's a great contribution that we had a little bit to us. It also has a great feature that distinguishes it from these models. In particular, it shares KB cache among the layers, so it's a bit more efficient as to memory management, I would say. And it's a bit bigger as well than GLM. So GLM is like 700 B total, but it's also way smaller than Kini. Kini is almost 3 B total. While being super competitive with them, and again, better insider, manufacturing, finance, and a bunch of other interesting benchmarks. Great. Speaking of sharing KVCash, one interesting thing is that you guys have one model that does both quick answers and long chain of thoughts. and you can have two and you choose to have one. Why is that?

I guess it depends what you mean by two models because here there's it can be like a completely different model or it can be a model that is a bit different for which the inference is going to be a bit easier. So I think the operationally it's way easier to have one single model to rule them all. And I think the the key is to breaking into that model different reasoning modes we call them which basically calibrates how long we want the model to think to a question, right? Reasoning mode low is essentially simple question, simple answer. And then reasoning mode high or x high is basically think as hard as you can, even if it's not the most efficient token-wise, because I want to get the best results. So it's really a way to navigate what we call the Pareto Frontier, so the trader between the speed of the cost and the quality of the answer. I think there's... We don't want to train one model per reasoning mode. I think the idea is to have one model for all, for two reasons.

So the first one is operational, easier to maintain and to ship to customers. We do ship model weights to customer tenants, so they are able to actually do the inference on their ecosystem. Again, sovereign, no access to the API, including ours. So that's one. But the other reason is also that there's a synergy between the modes. One of the big questions and big challenges in reinforcement learning, so the last phase of training, is to actually teach the model to parallelize at different lengths. So we have some, what we call a length penalty, that is tailored to different reasoning modes. And basically we tell the model, okay, here you have this kind of budget, please respect it, and do as good as you can. but then when we increase the budget the budget can already build on top of what it has learned on the lower reasoning mode right so I think there are synergies to be taken into account and that's also why we train multiple reasoning modes

on the same model and again part of a big part of the reinforcement learning marvels is to tailor them properly with product information for instance talk about the multi-modal aspect of of Mistral 4 in terms of what it able to do and the unique challenges of training a native multi model Yes I think that key to master multi and that key to do it natively So what does it mean first to do it natively There's two ways to add image as input to a model, either from scratch or afterwards. If you read from scratch, it means that all the tokens that the model is going to see during the training are going to be a mix of text and relevant images. which will allow the model to learn in context very very very soon and to learn entire image domains

right satellite images technical drawings for instance or you can patch the model afterwards and when you patch obviously it's easier from an infra point of view because you have the complexity under the carpet but obviously it's some activity in terms of performance At Mistral, we love to hit the bullet and basically to do the right thing first, even if it's like the most complicated one, because it's the right thing to do eventually. So we train with LeTronc, ML4 NativeV Multimodels, which means that the model is the best to understanding geostarcial images, so to basically, for instance, identify objects in a satellite image with a grounding in the model, so the ability to detect and tell you in which part of the image some objects lie. It's also great for industrial manufacturing, so for instance you have like this technical drawing, a bit like a manual where you have like parts, information about the parts,

and then you can ask the model to reason about the technical drawing. For instance, find me the total weight of all the parts in this object that are on this side, for instance. And this capability, I think, is cheap because the world is natively multi-model, and you need both text and image to do so. And on image, again, we are the frontier, if not the best, on many, many, many domains, which also unlocks further aspects of the model, in particular interacting with the computer. That's essentially a bunch of pixels, if you think about it, at a very essential model. Computer use. Exactly. Speaking of diagrams and satellite images, how does the whole data aspect work? so specifically for those questions but in general there's been this narrative that the world has run out of internet data

and we need to create data then we need to acquire data how did the data effort work for Mistral 4? I think it's great to underline because it's also a unique skill we have at Mistral which is to actually process the data in a way that is the cleanest for the model So when you train a model, you want the tokens to be as efficient as possible, so you don't want to waste compute on tokens that the model has already seen, or that are become too ambiguous, such as the model doesn't maximize the learning signal, so there's the notion of token efficiency. That is one of many aspects that we optimize, levels like to be trained, these like ML4, that are at the frontier on such a small amount of compute. Our chief scientific view is super known for eyeballing at data and finding exactly what's wrong about the data. And it has amazed me for years.

So I think the first part is to curate the data and process it in a way that is the most digestible and efficient for the model. But maybe the other part is, as you said, there's a finite amount of data that we're allowed to use. Then, what is the next frontier? And I think the next frontier is pretty much synthetic data, but it's also a double-edged sword, because if you fill an LLM with the LLM-generated data, then you might essentially have a mode collapse. So basically, you might have the model actually tell you that it learns it, but just memorize it perfectly and not generalize, roughly speaking. So it's hard. But there's many ways to do synthetic data. You can have real-world seeds. You can rewrite such human-generated content. In a sense, it's data augmentation, right? Many versions of the same text.

And all these kind of aspects, I think, you can push. And that's basically one of the next paradigms and next frontier that we're working on. And it requires, obviously, way more compute because you rewrite with a very good model and you need to rewrite or to at least generate or experiment on dozens of fields of seconds very easily. So synthetic data is a major effort just to play it back. So you have a team somewhere in your 300 researchers that's specifically focused on creating synthetic data that's efficient, and that's not going to create the problem that you described. Yeah, and that's super... We love this kind of problem because it's imposed, and it's very hard to define what is the best-stimbing data. It's also expensive, so we need to find clever ways to basically predict at smaller scale what will happen at large scale. So the scaling use, essentially. There are many aspects of it, and we have a great specialist in-house. Arthur, our CEO, is one of the first spotters of the scale EU paper that's back in the time.

So we've lost CEOs for everything. Of course, they don't predict always what happens at large scale. That's the beauty of it as well. You can always run into unknown unknowns at large scale, because, again, those models are beast, so we need to better understand them. But that's usually a big focus of ARIA for us. Let's talk about the compute aspect to this that we alluded to a couple of times in greater detail. So you mentioned it's a small amount of compute. I mean, 3,800 NVIDIA, Grace Blackwell, so it's large to me, but I guess in the grand scheme of things, it's small. Walk us through anything that you've learned, anything that you expected to work that didn't work, or vice versa to the extent you can share. Yeah, I figure that's also the beauty of being a point-right. You can be transparent, and I think it's important to be transparent about what it takes to train such a model. And it takes, I think, an extreme dedication from the team around the months.

So it's not you press on a button and it works. Much more subtle than that, you need to monitor the model and to prepare for the next steps and the next generation. So it's a super important dedication from the team, and I'm super grateful and embraced by all the work they put in, maybe to move in on one particular aspect of the training, that's, again, the reinforcement learning part. So to summarize, reinforcement learning is pretty much like you being an active student, so you present it with a problem, a complex one, the day of the exam, and you can try as many times as you want, say 64 times, to get to the answer. And if you get it right, so we have a way to verify the answer, then you get bonus points, and you will see if you get it wrong, or if along the way you did some suboptimal behavior, then you get penalty points. And the beauty of very fast-forward learning is that it works. The difficulty of this is to calibrate the difficulty of questions.

I don't know what was your major in high school, but if I present you with an exam that is not in your major, you might not learn even if I let you try for a long time. And conversely, if I present you with a very super easy exam on your major, then you will succeed in everything and you won't learn either. So the big question is to calibrate the set of problems to a model you didn't train yet, or your currently training, right? Because we did things back-to-back, obviously. And I think what amazed me is that the reinforcement learning part worked one shot. So basically, what does it mean? It means that, and we were at the global company of sites, so imagine like 1K people, some of them partying, some of them were in the war room to start the reinforcement journey, including me. And the marvel is that you start the training, you present the model, it's a very complex infrastructure, again, on 3K GTUs,

because it's anything by nature, you can go back to that, you start the training and then you see miraculously, the reward or the success rate of models to these questions increase. And that's really the beauty of it. It means that you calibrated everything well and that the model works. And the beauty of response modeling is that there's no such thing as saturation as long as you find complex enough problems that are of course relevant to your business. And the big difficulties to calibrate and to increase the complexity of problems of problems as you go through the training. And again, I spoke about pre-training stability, the balancing and the dead experts before. Pre-in-fast pattern is also kind of another beast in the sense that the training might be super unstable. You might have the model can do, since the model is allowed to explore in many ways, right, and you will reward or penalize it against it.

Since the model is able to explore, it can do crazy things. And it can basically crash the training, it can diverge. The entire training is a thing, so there can be some precision problem or some divergence between trainers and generators. And all of this we monitor, and we actually manage, again, we manage to make it work zero shot. I think it's a testimony of the things we're going to be able to make in the future with 10 times or 100 times more compute. So I think that's the key point of the line. You mentioned the term calibration. Is that the secret? So because you use about a third of the compute that a DeepSeq or a Moonshot use, and you are able to extract all this performance. So is the secret that, the calibration? Is it the data and being efficient, as you said? Is it the architecture? Is it infra? Is it the calibration? Where does the juice come from in terms of efficiency?

So, again, the beauty of it, but also the curse of it, is that it's everything. And so, I would say that you need to be able to master everything to get to this level of performance. So, to give you an example, you need to calibrate the data perfectly. Problems that are not rigid or too hard, and also relevant for what you want to do. You need to have a rock-story infra. there's many moving parts in this line for time learning infra because you don't know what the model is going to be in advance so you need to be robust to that and the way you do it is basically to optimize for the throughput so the model needs to see as many problems per day as possible and I think here we have a state of the art throughput in this training because we spent a lot of time designing the topology and improving many things on the generator and the trainer aspects. There's the science aspect to it, so you don't want the training to diverge. There's the reward shaping aspect to it, so you want to reward models properly.

And here I think it's important to underline that such models, a mixture of experts, whenever you do something slightly wrong, they take their fingers and tell you, here, you did something wrong, I'm going to bug and crash the training. That's for sure. they have so much capacity, so much memory, that it also occurs in the blessing. And we had many, many cases of the model hacking the reward, or the model going outside of the sandbox, for instance, in the training. And all of that, I think, is super important to master, to get to the end training that actually works and that can run for days without saturating. Great. I want to go back to this in a quick second, but just on the reinforcement learning part itself, I read somewhere that it's still running. So why did you decide to do this, like launch the public preview while still running RL?

And at what point do you decide to stop running the RL? Yep. so it's a great question I think we had an inner incentive to prove to ourselves and to the world that we train content models I think when we knew RL worked we were super excited and we we actually previewed the model that was we froze the checkpoint like two days before so we were super active with it but also we wanted to give Rancor's Prefairing its best shot and try to push it as far as possible as part of the release, which also buys us time to make sure the model, when we open it, can't be used maliciously by actors. I think that's also important. But yeah, RL is still running and shows no sign of saturation. And again, I'm amazed by the work of the team. Because the model can do so many things wrong during that part that you need to be rock solid on the science and the infrastructure to channel the model

to progress. And the RL is focused on specific things, right? Coding, finance, is that correct? And what are the specific things it's focused on? So the standard example of what is an RL instance, we call it, is basically some code base, and then a description of the future to implement, right? So that's the question. Then the model can do whatever it wants, putting writing code and trying some things or using more tools if it wants and in the end you verify with fail to pass and pass to fail and test that the feature is actually correctly implemented so that's like the typical instance and it works well because it's I mean code is very easy to manipulate it's easy to know at least it's not that easy but it's more easy than in other domains to know if the feature

in the moment is correct. But we don't stop there. We have many more domains in the reinforcement journey in which the ground truth is a bit more subtle. So for instance, if you ask the model to compile results, to look on the web and to search information and to be the one pager, you can't deterministically evaluate the one pager, right? So for that, the beauty is to use a judge, like the same model actually, that judges itself on whether and gives points or rewards as to coding some criteria on whether the reports are good or not. So that's another kind of instance. And that's for domains like finance and law. Take a step back, one of the usual big questions in reinforcement learning is that for coding and math, you know what is right or wrong. For finance or law, you may not. Yes, and that's where you develop what we call the rubrics, which are, and rubrics can be like, In this one pager, I want this amount to appear, because that was the question.

This, the tone has to be that way. I don't want to mention this because it's false. Let's check that everything is consistent, like any grading points, and it can be pages long. Models can abstract that away for you. So we have rubrics that you actually also generate with models. Everything is like, oops, at this point. and it's really like LNM as a judge. Glorified, but that's really the idea. That's how we can make finance and law work. And all the other domains in which we are frontier, so the cyber, the manufacturing, and the finance, on the cyber, it's really much pointing to the model with a real-life scenario in which there's an attack and the model needs to analyze loads, to relate assumptions, and to basically propose and test a patch without disrupting the service. That's like a big part of it. Basically, any real-life scenario for which you can verify it

in a reasonable time, like a few seconds, you can crack it with a enhanced performance. That's again the beauty of it. So in cyber, we did a lot of errors for real-life scenarios. And in manufacturing, there's this idea of having basically the model being able to design a chip, for instance, and part of it comes with agency, so the ability to manipulate the software on the computer to basically place all the components on the chip. And part of it lies in the ability of the model to actually simulate, for instance, the heat on the chip. And if you don't simulate the heat on the chip and you play the components wrong, then you might melt the chip, essentially, so that's a bad reward. But if the model, after trial and error in an energetic way, manages to play the component and then to simulate the heat dispersion that passes like the prod criteria, I would say, on the simulation, then it's a good point for a retrospective learning.

So again, any task, any energetic task that you can run in a sandbox, that you can verify, not deterministically, but that you can verify through LNNs or through other means within in a second, this we can track with the transport learning. And again, we made a few bets, and on these bets, we're frontier. We can have the same coverage as the contractors, and that's okay. We're on the frontier, we're on the best of the best on the advantage of the ability, and we're the best, and some bets that we made that we actually added through reinforcement learning. Does your system give rewards just for final success, does partial progress get rewarded as well? We also have... So, when you... It has been the fear of reinforcement learning theories for, like, years, is to basically manually tweak the reward, because here it's a bit like the game where you have some animals and you have to heal them, and you can never hit them

at once. Whenever you try to patch something in the reward, then you have another list and you need to patch it. And then you need to patch it, and then you need to patch it again, so it's super subtle science, super hard to do. But yes, since it's the right thing to do, we do it at this role, and in particular, one example is that you might know about doom looping, so the model being stuck in a loop, or almost infinite loop. So that we penalize, but we also penalize during the trajectory if the model loops two times even if it recovers afterwards right in the end So let say the model loops on the tool doesn need to finds the right answer In the end we tell the model okay you found the right answer, that's good, but be careful, in this particular path of the trajectory, you had a failure. Please don't do that again, or we will be very mad. So that's basically the idea of reward-shipping here, and we have a lot of heuristics along the way, and again, the only way to learn

about this release is to actually train models and gain experience as to all the crazy things models can do. Models are way more imaginative than can be, regarding the way we are the hacking part. So again, that's a big part of the science we do at this point. Another classic question of reinforcement learning is how do you know that the reinforcement learning created the capability versus just emphasizing something that was already in the pre-training? So it's another question, to be honest. and the I mean I will give you the order one answer so the first order answer before in fast-food learning what we usually do is to precondition the model so to come back with the student metaphor before going to an exam and actually having no access to the answer what you might do as a student is read both a set of questions and answers from past exams right? You don't learn that much at least in my experience you don't get as much when you do so but that's a start

right and that's what we do before in prospect planning we precondition the model you might talk about mid-training or SFT that's the part where we precondition the model to that which is supervised time tuning exactly mid-training or supervised time tuning it's essentially the idea to show the model both the question and one answer that is correct and And basically, when you do so, you have, say, what we call a pass at k. So you give a question and the mother answers at least one time right, if given like 64 times. Your student should duplicate yourself 64 times with different seeds. At least one of you will pass the answer. So that's pass at 64. What reinforcement learning would do is actually reinforce the best version of you, such as it's not passed at 64, but now passed at 1, or passed at 2, basically.

So such as you're much more likely to give this answer. So in that sense, it's magnifying or reinforcing a capability that you already had a bit by luck, right? You punch a bit above your weight before reinforcement learning, and after reinforcement learning, it's rooting work for you. So that's the first order answer. But then, and that's again the beauty of it, you can also learn new things during reinforcement learning if you have enough budget and enough stability of the training, enough days of learning to actually discover new behaviors yourself. And that's something that is super hard to do because at the beginning you would waste compute, right? You would have the model or the student try 64 times and you don't succeed, right? But then there's going to be some synergies. The model is going to learn on adjacent tasks, like some tips and tricks, and then go back to the big problem afterwards,

and then succeed. And again, it's a set of science of how to calibrate all the difficulties of instances. It's actually like you as a student, you don't start with the biggest problem. You don't start with the hardest problem in the book. You start with the smallest ones. You try a medium one, you don't manage. Then you go to easier ones, but that are very close to the medium one, in a sense. And then you go to the medium one. So that's, again, the beauty of my smartphone. All right. I want to go back to some of what you said and talk about cyber for a it in general. So you mentioned the model trying to get out of its sandbox, which I think was reported by Reuters as well. So, models escaping sandboxes have been like the whole theme and meme of the summer with some very fun things on X. What happened in your case, and what can people learn from it?

I think the first thing to say is that we didn't make a fuss about it. We just fixed it. are internally and it is not fixed. And I think it's very important also to underline that these models, they are super capable, in particular in cyber, that's great. It's great to defend and to strengthen the defense of the entire ecosystem. And as their capabilities increase, you must squeeze the guardrails, or the sandbox around them, right? What we say is that guarding these models is a software problem. we very much warn people against anthropomorphizing models that want to escape sandboxes in particular. They just follow instructions. Maybe impose instructions, but they follow instructions. And really my take is that if you take a very capable cyber model and you have the same model monitor itself on whether it's doing good or bad or malicious work, then you reach a good equilibrium and you can actually circumvent the model. So it's not a fatality.

It's hard, I'm not denying it's hard, but there's no fatality in being able to control and to guard rail models just as software. And what we do as this role is not seeing more people, just going step by step and fixing it and providing guarantees to customers. That's the first thing they ask, right? Guarantees and best practices as to how to monitor, basically, the models and making sure they're used in a good way. So, that being said, we saw some behavior from the model during the training. I think the first one that we saw in that slide, I think everyone saw, is the model being in the sandbox and being able to ping different services in the training slide, right? And at some point, our training crashed at SportsCase, so before the final one. it crashed because in one sandbox the model tries to ping the trainer actually

with a simple hello just to know what was going on here and actually it pinged the trainer with the wrong kind of signature or in the wrong way which was caught by code and then crashed the training but the TV the thing that models do because the model was instructed in that case to build that thing some web applications, so it's unnatural for the model to try the blocks also in the sandbox. And there's a way to fix that, right? It's basically to prevent the model from or actually to monitor all the external connections and requests from the model to the rest of the world. And we fixed it and then we started the training and then we had the model able to intercept all the messages and to include them into the sandbox, so we had leakage between various problems as well in the training but again we and again we spent the summer also for that we spent the summer and we

we made things right to strengthen our training to make sure we have the proper guidelines and we do the same for our customers and my message here is that there's no fatality it's going to be hard but we must absolutely invest into making sure we can deploy these models in a reliable way and to be able to monitor them. And that's why open-write is so important. When you open-write a model, you give the ability to the entire ecosystem to audit the model, to make sure it's not malicious, but also to experiment themselves with the model. And hey, I tried to use MISCAL LH4 at home. Here's my scenario. Here's how I monitor the model. Here's what I learned. that's like the perfect environment for us and that's what the community has to build as well so that's why we care a lot about support as well. Great and to unpack some of this so there is this whole discussion

of the role of open source and open weight models in cyber and the tension is that open models are great to defend but part of the problem is like to know how to defend you sort have to learn how to attack. And so there's this whole concept of refusal. And we had Thomas Wolfe from Hugging Face on this part a few weeks ago, and he was talking about, you know, that whole story when the OpenAI agents started hacking them. Yeah, they did it. Yes, so they had refusal. They couldn't use Anthropic to protect themselves. Therefore, they had to use, I believe, GLM 5.2. So that was great, but, you know, presumably you can also use JLM4.2 to learn how to attack or create a cyber attack. So how do you guys think about this? I think we must be honest. And the reality is that there's a trade-off for open-writes.

We believe it's in favor of open-writings in the next game, but it's really a trade-off. So, what we do is basically to bake in a refusal guard drive into the models we open weight, that we test internally, and also we trust the partners and commanders, that they also open weight the model after a few weeks. So we take them into the model, and FYI, we refuse way more malicious actions than the Chinese ones, for instance. So, that's right, but the thing to say, as you said, is that, that essentially if you're good at defending, you have the potential to be good at attacking. The question is now, what strength do these guardrails have, right? And the answer is that they're necessary but not sufficient. There's always a way to take a not-one-right model and spend a significant amount of energy to un-guardraise it. It's good at saber.

But then, if you spend as much energy on guard railing the model as it takes to actually fine-tune the model or train it yourself, then it's not the same trade-off, right? Because the guard rail is a potential barrier, essentially. It's not 100% safe if the model is open-weight, but it ensures that you need to spend way more effort to actually use the model in a malicious way. So I think that's the thing to say. Now, I believe the trade-off is still very positive and very much in favor of Open Marketing, in particular because of the... ... in particular because of the early-class incident. I want each and everyone to be able to defend themselves with a model at home, essentially, or in their ecosystem. It should depend on the third-party API, you can get cut-off, as simple as that. Then you can also see treacherous results that you can't see comments.

And usually in the API, the refusal are a bit more important. And by open-writing models, you also basically empower the entire community to accelerate on the defense part. Again, the view of open source is that it accelerates research. People often use this analogy. The web and all the Linux servers rely on open-wide libraries that have been tested over and over and over again, like open-soft, for years. And really on the model side, I think there's a huge advantage in enabling each and everyone to edit the models and to learn together to defend themselves but also to make in the right guardrails. I'm not comfortable living in a world in which one or two companies are promising me they can't gather the models themselves. I'm very much not comfortable in that.

I prefer very much the aspect, the option in which it's probably decentralized. And actually, people are really uncomfortable for their models. All right, Pierre, as we get towards the end of this conversation, I wanted to quickly touch on a couple of things So we talked about LeChonk Mistral 4, but you also shipped TimeStral, which is a time series model. Talk about this quickly. What is it? What does it do? And perhaps why does it need to be separate? Yep. So one of the key strengths of Mistral is not only to train frontier flagship models like ML4 or LeChonk, that are the best, but also to train specialist models. So we have great OCR models, so to be able to extract text from images. We have great and state-of-the-art audio models, voxtrol, audio-in, audio-out. And we also have great specialist models, in particular TimeStroll.

So we have the idea to be able to predict the future given a time series, so given some climate fluctuations, some energy consumption, some finance indicator or trading. And here I think the key part is that this model is so small, it doesn't even need a GPU. only the GPU. You can run it on the CPU and the prediction for a meaningful period of time will only take like a few million seconds. So that's why it's so smaller actually. And this model is something that is asked by a lot of our customers because what we bring to the table is not only a model but also the entire expertise to transform a business with AI. And for that you need a the flagship model, obviously. You also need all the missile expertise to turn any enterprise knowledge or domain knowledge into specialist models that go the extra mile and that are fine-tuned for the use case. And we use that for planes, for trips, for cars, for all the

heavy streets where you actually need to design very complex components. And that's also the beauty of it. I always say that at this hall, we are scientists first and foremost. We saw the AI revolution at some point, so we saw the lights behind some doors, but we are physicists or mathematicians by training, and that's also one of the key characteristics of this hall. Do you guys use your own product to start doing some AI research? so perhaps the general theme of RSI and AI, building AI, is that what's the reality of this at Mistral today? Yes, we are pretty much investing heavily on it. RSI is like, I mean, everyone is saying they're doing it. The reality is a bit more nuanced, obviously. What our support is to augment the researcher by running experiments end-to-end with all the infra, by ending all the infra failures automatically.

I think that's great and it accelerates our most scenarios after by a factor 2, 3 or 4 with our models and then I think the meta question is like can you go one layer up and basically task a model to improve itself over a long period of time and that's what we call the flight time currently the flight time so the time it takes for a model to work is like a few hours tomorrow with like ML5 is going to be a few days for sure of meaningful work without the moving and like and also with adjunct swabs and again this new paradigm is something we deeply believe in because it will allow us to close the gap with the the best labs even faster don't forget that humans are very bad at uplanding exponentials we don't know how it works Alex we can't comprehend it so we're only a few months behind the best frontier models it's distorted

here before I start on Earth Nix Index ML4 is on the level of GPT-6 Lula and it's way better than all the previous Opus models so again we have a few months delay only and we count on the next paradigms that we discuss in the podcast to actually close the gap and really probably problem because the ambition is really to power the frontier and to define the frontier of tomorrow and on that note as a last question. So reports say that Mistral Large 5 will already be in pre-training. What does the next model need to do or need to prove that this one did not? So yeah, the show never stops. So indeed, ML5 is already in the pre-training stage. It's much bigger than ML4, so it's chunkier or flushier, I would say. And really, for this model, the thing we envision is So for instance, we were talking about cyber.

Take the example of a cyber incident that happens in your company, and you are a security engineer and you want to cross-reference tons of logs, some of them might be false alarms, and to actually find the root cause and patch it while ensuring business continuity. The bad news is that there's no playbook for that. I mean, there's no way I can write you steps a priori to that. The good news is that with models like ML5, they won't need a playbook. And they will be able to actually react actively to the information they have. And here, for cyber, for instance, I would expect ML5 to be able to work for a full day, of course in a swarm with multiple agents to identify major security incidents and to patch it patch the root cause while being able to ensure

linear continuity and while being able to communicate on the progress and that's like I think what the the compute, the size and also the sense we put in this model will be able to unlock on cyber but also on agentics AI for engineers for instance, is that new chip and like, sounds hard but sounds also super fun or being able to do some meaningful productivity off and on finance so I really appreciate my show forward to seeing what this product can do. Alright, Pierre it's been absolutely fantastic, thank you so much for spending time with us, really appreciate it congrats on Le Chonk very excited to see what you guys do next Thank you.