なぜ FLUX 3 は画像・動画・音声を一緒に学習するのか — Black Forest Labs の Andreas Blattmann と Dustin Podell

Why FLUX 3 Trains on Images, Video and Sound — Andreas Blattmann & Dustin Podell, Black Forest Labs

Latent Space: The AI Engineer Podcast1時間38分
なぜ FLUX 3 は画像・動画・音声を一緒に学習するのか — Black Forest Labs の Andreas Blattmann と Dustin Podell

なぜ FLUX 3 は画像・動画・音声を一緒に学習するのか — Black Forest Labs の Andreas Blattmann と Dustin Podell

Latent Space: The AI Engineer Podcast

0:001:38:23

要約

Latent Space のエピソードに、潜在拡散モデルの共同発明者 Andreas Blattmann と、SDXL や FLUX.1 Kontext を手がけた Dustin Podell が登場。Stability AI 時代から Black Forest Labs(BFL)設立、FLUX 3 までの歩みを語る。BFL は画像生成ラボではなく、動画・音声・行動までを統合した「視覚的知能」を目指すと述べ、7B の FLUX 3 Action でロボティクスにも踏み出している。2027年を製造業ロボット元年と見る展望も示された。

この回のポイント

  • ●BFL は画像生成専業ではなく、動画観察を通じて世界を理解する「視覚的知能」を当初から狙っていたと説明している。
  • ●FLUX 3 は画像・動画・音声・行動を一つのマルチモーダルモデルで事前学習し、そこから Video、Image、Action、アップスケーラーなどを派生させている。
  • ●Stable Video Diffusion では、2D 動画のみで事前学習したモデルを 1,000 ステップ程度の微調整で 3D モデルにでき、動画が暗黙の 3D 理解を持つことを示したという。
  • ●FLUX.1 Kontext は Dustin の個人的な副業プロジェクトから始まり、数時間の小規模学習で汎用的な画像編集の兆しが見えた。主にデータセットの工夫が鍵だったと語る。
  • ●自己回帰と拡散のどちらが優位かは決着していないとし、訓練時のデータ効率は自己回帰、サンプリング速度は拡散に利点があると整理した。
  • ●FLUX 3 Action は約4時間のデータで微調整でき、将来は一度見せるだけで行動を学ぶインコンテキスト学習を目標とする。

章立て

  1. ゲスト紹介とそれぞれの経歴

    Andy は物理系の背景から拡散モデルに入り、Dustin はソフトウェア開発者から Stable Diffusion 公開を機に転職したと語る。社名の由来や Latent Space の名前の由来にも触れる。

  2. 視覚的知能と世界理解の考え方

    BFL を画像から動画・音声・行動へ拡張する方針と、2D 動画から暗黙の 3D 理解が得られるという立場を説明。観察と相互作用をめぐる議論もある。

  3. SDXL・動画拡散・SD3 の技術史

    SDXL の Refiner の裏話、Stable Video Diffusion でのデータ重視、SD3 でのフローマッチングと学習計算配分の工夫を振り返る。

  4. BFL 設立と FLUX.1 の立ち上げ

    約13人で始めた設立初期、フランスでのオフサイト、3か月での FLUX.1 公開、Stability での課題意識を語る。

  5. FLUX.1 Kontext と複数参照画像

    Kontext の着想と、短時間の学習で編集能力が現れた経緯、FLUX 2 での複数参照対応、評価ベンチ作りを紹介。

  6. 自己回帰 vs 拡散

    データ効率や推論速度の違い、離散・連続の拡散言語モデルなど、両方式の現状を議論する。

  7. FLUX 3 Action とロボティクスの展望

    動画と同時に状態空間の行動を予測する世界行動モデルの仕組み、7B の理由、微調整データ量、2027年の見通しを語る。

  8. 認知度向上と生成メディアの受け止め

    米国と欧州の拠点、一般層への認知拡大、バイラルになる用途の予測困難さと写真修復の例を語る。

解説記事

Latent Space のこの回では、潜在拡散モデル(latent diffusion)の共同発明者である Andreas “Andy” Blattmann と、SDXL や FLUX.1 Kontext に携わった Dustin Podell が、Black Forest Labs(BFL)の歩みと FLUX 3 の狙いを語った。Stability AI 時代の裏話から、ロボティクス向けモデルまで話は幅広い。

画像生成ラボではなく「視覚的知能」を目指す

Andy は、BFL は画像生成ラボとして作られたのではなく、FLUX.1 の成功がきっかけの寄り道だったと述べている。当初から、動画モデリングは世界の仕組みを理解する中心的な手段だという仮説を持ち、資金調達時にも動画を掲げていたという。

目指すのは、人がボールを受け止めたりボトルを掴んだりするときの直感的な知能、つまりシステム1的な知能をモデル化することだと説明する。FLUX 3 は画像・動画・音声・行動を一つのモデルで事前学習し、そこから Video、Video Edit、Action、Image、4K 動画アップスケーラーなどを派生させたという。一つの汎用モデルから多くの応用を導けることが、学習の複利効果だと語られている。

2D 動画から世界を学べるのか

司会の swyx は、観察だけで学ぶモデルは世界の本質を捉えられないという批判を挙げた。これに対し Dustin は、人間も世界から濾過された信号を受け取って内部モデルを作っているという点で、本質的な違いは大きくないと反論した。Andy も、人が赤ん坊のころ観察から多くを学ぶことを引き合いに出し、次の段階は世界との相互作用を含めたループを閉じることだと見ている。

根拠として挙がったのが Stable Video Diffusion の実験だ。普通の動画だけで事前学習したモデルに、わずか約1,000ステップの 3D データ微調整を施すと 3D モデルが得られた。動画モデルがすでに多くの 3D 理解を持っていることを示す初期の例だという。

技術史:Refiner、フローマッチング、データ

SDXL では、マルチアスペクト学習や複数段階の学習が初めて導入された。Dustin は Refiner の裏話も明かしている。公開されなかった別アーキテクチャの SDXL をそのまま Refiner に転用したもので、構造は弱いが高周波ディテールには強かった。「別のモデルに別の仕事をさせる」発想だ。Andy は、これがノイズ側で構造、クリーン側でテクスチャが学ばれるという軌跡の性質を利用した初期の仕事だったと補足する。

SD3 では、拡散からフローマッチング(rectified flow)への移行と Diffusion Transformer の採用が行われた。フローマッチングは両端の分布に標準正規分布を要求せず、確率微分方程式ではなく常微分方程式として扱えるため数学的に簡潔で、高速サンプリングや蒸留もしやすいという。論文の核心は、ノイズからデータへの軌跡のうち、情報量の多い部分に学習計算を集中させる点にあったと説明される。テキスト描画の向上についても、アーキテクチャよりモデリングとデータ(キャプション)の改善によるところが大きいと述べている。

データの重要性は繰り返し強調された。どんなに優れたアルゴリズムでも、ぼやけた動画ばかりを学べばその分布を学ぶ。研究コミュニティではベンチマークに注目が偏りがちだったが、データ品質と量のバランスは最重要の調整点だと語られている。

FLUX.1 Kontext と汎用モデルの発想

Kontext は Dustin のオフサイトでの提案から始まった。モデルはすでに空間的な関係を学んでいるはずで、適切なデータを用意すれば少ない追加学習で汎用的な編集を引き出せる、という仮説だ。数時間の小さな学習で「ロゴをシャツに載せる」といった動作の兆しが見え、その後の数か月で全社プロジェクトになった。主役はデータセット作りで、蒸留による高速化も寄与したという。

FLUX 2 では約10枚の参照画像に対応した。参照が10枚を超えると人間にも追いにくく、評価も難しいとの指摘があった。評価ではベンチを自作し、自分たちでモデルを使って遊ぶことを重視する姿勢も語られた。

自己回帰と拡散の対比

Andy によれば、自己回帰モデルは全トークンを並列に学習でき、データ効率で有利である。一方、拡散モデルは推論を蒸留でたとえば50ステップから4ステップに縮められ、サンプリングが速い。どちらにも長短があり、議論はまだ決着していないという立場だ。Dustin は、離散拡散言語モデルと連続拡散言語モデルは別物だと整理した。FLUX 3 では拡散モデルに対する GRPO 系の強化学習も使い、大きな改善があったと述べている。

ロボティクスへの拡張

FLUX 3 Action は7B規模のモデルで、動画と同時にロボットの状態空間(各関節の並進・回転)を予測する。直近の映像と指示を条件に、9〜21フレーム程度の塊で行動を生成する。現状は高レベルのプランナーなしにループで動くという。

微調整は、ある例では約4時間のデータで産業用途の成功率に達し、オープンな Pi0.5 と比べて8分の1のデータで済んだと紹介された。次の目標は、一度見せるだけで学ぶインコンテキスト学習だ。2027年については、製造業での商用展開が進む年になるとの見通しを示し、家庭向けは2027年後半に限定的な形で現れるかもしれないという Dustin の予想も出た。

まとめ

BFL の主張は、画像・動画・音声・行動を一つの事前学習で束ねることで、世界理解が複数の応用へ波及するというものだ。日本の読者にとっては、拡散モデルの技術的な系譜に加え、データ設計の重要性と、生成モデルがロボティクスへつながる流れが注目点になる。ただし、ロボットの商用成果や2027年の見通しは本人たちの予測であり、実際の成果の検証はこれからだ。

文字起こし(英語・自動生成)

Happy to bring to you our episode with Black Forest Labs. Our guests today are Andreas Blattman, he also goes by Andy, who is co-inventor of latent diffusion, the technique that eventually got scaled up by stable diffusion in Stability AI. And they eventually split with a bunch of the core Stability AI ImageGen folks to form Black Forest Labs. And Dustin Podell, who I also met with Ange at one of the stories that we tell, and who worked his way up from just being a regular engineer, a freelance web developer person, into a model and research lead at Stability AI, and now Blackforce Labs, creator of SDXL, Control Laura, post-training lead on SD3, Flux Ultra, Flux One, and Flux Context. And we go into all of these details. As you may know, sometimes we're very honored to have an extended history with some of our guests, and this is one of them. You might have heard in our Anjumita episode where we talked about introducing each other at one of the Neuripses meeting Black Forest Labs for the first time.

And this is right after Stable Diffusion. But what you may not have heard, and we only tell this story in this episode, is how latent space itself was started because of Stable Diffusion, directly crediting the launch for me basically quitting my job to go all in on AI based on what we saw. And today's guest, Dustin, also had that experience, and so did Vibu. And so we have a very, very strong tie with the team that basically led up to Black Forest Labs. And now we're also interested in what's next for them. They have created some of the best open and closed weights models for image generation and video generation. I think, especially Flux Context, if you're not aware of image editing models, I do think that this is, in the limit, one of the things to fully replace Canva and Photoshop and a bunch of other workflows and editors that you might use in a normal creative workflow. It is not going to replace the top end of the market, but for a regular, normal, everyday edit,

why would you not want anything else other than to query your model and ask it to do image editing for you? We do this, obviously, with this kind of video editing. It's all Clanker done, and I do review it, and they're also moving into robotics. And so this ties in with a mini-series of model labs that have been pursuing this direction, building world models from ImageGen with Runway, as well as video models and now robotics models. And so I think a lot of things rhyme in a lot of our recent episodes, like even the artificial analysis episode that we're going to release soon. They are also going to robotics, and it seems pretty obvious It's obvious that when you see a whole bunch of smart people who are at the frontier all leaning one direction that you're probably 6 to 12 months out from some kind of really big breakthrough. This has been a consistent theme over my last four years of podcasting. It's not going to change now. So look out for it. That's the most obvious call out I can make. But there's also a lot of interesting lore for people interested in the general diffusion space.

We're going to do a little bit more over time. I'm thinking about an AIE LA where we talk about the generative media field and the creative arts and how the complicated relationship is evolving and I think maturing, to be honest. And finally, for those longtime listeners, we are starting to open up to advertisers, people that we really admire, respect and want to showcase what they're working on. The first one was a super bass, which I think a lot of you enjoyed. And the second one today is brought to you by Dgram. And you'll hear more from us at the midpoint of the episode. We should be showing a robot an action one time, and then it should be able to do this. And this would be truly a superhuman intelligence, like at stability that we couldn't essentially leverage the value of what the models could be as well. The mistakes of not being able to actually capture the value of the models themselves. We raised all in all $450 million in funding so far. Currently the last that was published in December was 100 million revenue. BFL was not founded to be an image generation lab.

Video modeling is actually the central means to understanding how our world basically grows. This is maybe like six hours of training for us. It was a very, very short training run. And all of a sudden, like, I saw what felt like a clear generality coming out of it because it was... Okay, finally our BFL pod. Welcome, guys. Andreas and Dustin. Great to be here. Thank you so much. Yeah, definitely. Thank you for having us. I know we've chatted about doing this for a long time. So we've had Anj on a couple of times, and that's where I met you at Vienna. I think one of the ICMLs heard. Was this like one of the, like, was it the, like, rooftop event? It was a year when there were two Viennas back-to-back. Yeah, yeah, yeah. That was first iClear, and then it was ICML. I don't know how they do this thing. Anyway, anyway. And at the time, you guys were just about to launch BFL. We had a dinner, and you were thinking about what's next after stability and all that.

So anyway, congrats. How do you guys like to introduce yourself? Maybe we'll sort of get a crisp self-incho up front. So I'm Andy. I'm a co-founder of BlackBorce Labs. I've been developing diffuser models for, oh, wow, I think seven years now. Met Robin and Patrick way, way back in the day. I think it's, yeah, seven years now since we met at Heidelberg. I'm from Germany originally, and there we started looking into diffusion modeling for images. We were all having a kind of a physics background in some sense, and this paradigm of just mapping this general physical phenomena of diffusion to machine learning and generative modeling had a kind of a philosophical kind of beauty to us. That's why we got into this. and then, yeah, developed a lot of the, I guess, fundamental algorithms that are powering diffusion models today or image generation, video generation, and also action prediction today. I guess we'll talk about this later. We'll talk about it, yeah. It's all of a kind related to... It's not immediately obvious to people, image generation to action prediction,

but we can talk about that. And then one thing I did want to comment is a lot of people don't know the relationship. Black Forest is a forest in Germany? It's a mountain range, actually. But, yeah, it's a region in Germany. And you named it after that. And finally, Robin and me, we first met during our PhD. And then we figured out we were actually born, I don't know, 10 miles away from each other. And that was pretty fun. And then eventually that led us also to come back to Germany. Beautiful. All of Europe looks like this. You can see the lab right in the middle there. Yeah. So you named it because you're from here. Exactly. We named it because actually we found that there's research left, and back in the day we could basically pick where we wanted to do it. No one said, okay, you have to come to, say, SF or whatever place. So we decided, okay, we want to be actually starting this at the place where we were born in Freiburg, which is like at the foot of the Black Forest, basically, or a city in the Black Forest. And that's how it came, and that's how then the company also started there.

Yeah. Comfis, also Heidelberg? Convass was in Heidelberg. Exactly. That's where we met. We then later moved on to Munich. Which is the team that did latent diffusion. Through RPSC. Exactly. And that is when we developed this latent diffusion algorithm. Yeah. I really like your name, by the way. Your podcast name, Latent Space. I would actually be curious how that came together. I can talk about it. But yeah. Go ahead. And exactly. We developed this latent diffusion algorithm at the university. And then based on this research project, we scaled out the algorithm, which led to stable diffusion then effectively and yeah since then it's been a constant journey on a rocket trip you could say which is super fun and we've met amazing people like Dustin as well who's one of the kind of oldest and like most I think relevant researchers that we have but yeah we'll learn more about that we'll gas him up later I think latent diffusion and just generally like VAEs as well

and then for us in space uh is this general idea even our logo is uh actually a vae and it's this idea that you can translate things into spaces and then explore things that are adjacent to what exists today um and then you can sort of retranslate back yeah and i thought that it was like such a cool insight for creativity and generation that i never really understood until i studied stable diffusion like the launch of stable diffusion changed my life i started in space because of stable diffusion i I quit my job because of Stable Diffusion. Oh wow. Because I was like, okay, everything I know about AI is wrong because I can run this on my laptop. And if I was wrong about this, what else am I wrong about? So there's that. And then there's another relationship with Terry Pratchett, which I don't know if people have... Writer, right? Writer. He has this concept of L-space, which is library space, where he has Orangutan that's like the librarian in Unseen University, which is Wizard University. And it basically is a library with all possible books. So if you have a library of all possible books, you can just theoretically go in there and find a book that is right for you.

And that is Latent Space. So I originally called it L-Space. One of my readers donated Latent Space and then became Latent Space. Amazing. Dustin, what was your journey into diffusion and AI in general? Yeah, so I came from a little bit of a different background than Andy and Robin and team. So I'm not from the German side, but I've still had a fantastic time working with these guys. And honestly, it's funny you mentioned that you quit your job when Stable Diffusion came out because the original Stable Diffusion, I was not working with these guys at. And it was actually a cause for me to quit my job at the time. So I am... You caused mass unemployment. I'm actually curious, like, the numbers of people that just, like, changed their life direction when Stable Diffusion dropped. I would actually be very curious to know kind of the branching effects from that. But I don't know. We can theorize on it. That's actually super interesting because here in SF, in the Bay, it had this huge impact. Whereas in Germany, back in the day, people didn't really notice it because the people were so far away from tech.

There are a couple of people, obviously, that have been involved in the open source community also in Germany. But the majority of people didn't really notice that drop in summer 2022, which was really interesting. And then you see what happens here. Like, yeah, but there was, to us, that was really kind of super interesting. Yeah. But anyway, so I guess to give a little background on me, so I, at BFL, I'm one of the founding researchers that joined in with the, we had a bit of a larger team, and I'm sure we'll get into the history of kind of how that all came to be. But my background is not actually traditionally in, you know, the ML space whatsoever. I was traditionally a software researcher, or a software developer for, you know, around 10 years or so. So early in, you know, around the 2020s, I became very, very interested in just, like, open source AI. I saw, like, the kind of the early work in GANs and other, you know, in generative media. And I was tracking with it. I was experimenting with it a little on the side. It got me quite excited. I mean, and I remember following originally the latent diffusion work.

I was actually, I was following that and quite excited about it. And then the one, you know, one of the things that kind of tipped me off to go, okay, maybe this is something a little bit, you know, a little bit more serious. This was actually a shout-out to MidJourney V1. Wow. Back when that came out, and I remember thinking, like, okay, this is actually going in a direction where I can see this being quite impactful. And then the one big moment, the big drop that really changed everything for me, as we talked about, was the Stable Diffusion release. And at the time, I was working at a software studio down in Los Angeles. Stable Diffusion came out, and basically that day, I turned in my resignation and said, okay. What was the plan? The plan? So the way that I saw it, the way that I envisioned it at the time, I saw this thing that felt like, it literally felt like a magical thing that came out, and I viewed, okay, the people that are working on this must be like arcane wizards, and I need to do whatever I can to understand what magic or what runes or what things that they did to imbue in this to make this happen and let this go out to the world and have this impact.

It wasn't like I was just totally disconnected. At the time, I was in the Sable Diffusion Discord. I actually had friends in there. I was kind of interacting with the community. So was it viewed as basically open source mid-journey, or do people not make it? I mean, I was in both. I remember there were a lot of kind of discussions around this, like, oh, Stable Diffusion was having such a, you know, there was a lot of discussion, and I remember the mid-journey discord and the Stable Diffusion discord around that time, around the impacts it was having. I mean, it's been a long time, so I can't remember the exact nature of it. I don't want to speak on the, you know. Mid-journey's going to be fine. I mean, they're doing great things. But for me, it was more just like I was very interested in the fundamental technology and then the fact that I can now run this at home. And then really the thing I saw was then just this community of developers actually like taking it home, developing on it. I was hacking away at it. I was meeting a bunch of other really, really smart, great people, some of which are actually at BFL now that have come along for the journey. That moment of just community interaction and learning and developing and building together was really an impactful moment for me.

luckily enough for me, I was able to find a job opportunity to get myself into Stability at the time where this was starting. And I didn't actually start working with these guys right away. It took me a while to kind of work my way up to that, and we can go into that history later if we want to. Stability had a bunch of different teams, right? It was Stable LM, Stable Music, Harmonai, the audio team. It was like I don't know, the way I would kind of describe it is like if you grabbed a bunch of Discord was kind of the space that a lot of this happened at the time. I don't know why the whole AM. It was like if a Discord had a physical manifestation. It was a melting pot of very different people that came together, but just like interested in this emerging new technology that came, obviously, and that's also how we are. This is one of the most exciting things about our team. Our founding team is it comes from very different backgrounds. Like we have this hardcore academia research branch, and then we have people who were from the open source community like Dustin who actually got a very

different kind of trajectory into this space but I think this is what makes our team so strong. We have these diverse backgrounds and there's rarely a team I think that has this history, also this kind of joint history together. This is just amazing. We can go into most of our other co-founders first. I was also in that space literally, I was also very influenced stable diffusion, I did the wrong thing. I basically went down the path of Comfy UI, where I built UIs for stable diffusion. You seem to have done the right thing, where you went to research. It's funny you mentioned Comfy, because I used to work directly with, I mean, Comfy joined. He moved here. Yeah, yeah, he's like my neighbor. He's a researcher. How do you become a researcher? I'm someone that's like, when I get interested in something, I am obsessive to the point of I cannot put it down until I pull it apart and figure out every aspect of it. And so to me, I'd already been experimenting with pulling the models apart, talking to a lot of

different people in that open source space and not just like developers, but other researchers. I mean, remember, talk about another discord, the, you know, the Eleuther discord back then. At that time, I met a lot of people who I was learning from and I was reading a lot of papers and just trying to uptake and understand for myself. Joining stability was also another major point for me. Originally, I didn't join as a researcher. I joined as an engineer. But all of my free time I spent with essentially any researchers I was able to make friends with, you know, shout out to Dr. Scott Hawley there who worked on the audio team who mentored me quite a lot there. And at some point, I was able to sneak my way onto the cluster there and got myself access to, you know, some idle compute. And I really just like I treated it like I treated how I learned most things, which is just open source hacking, trying things, reading papers, trying to implement different pieces of work and just seeing how, I don't know, how models trained, how they learned. I mean, that was really it. There was no magic to it other than just I was excited. I just wanted to learn about it. I wanted to tear it apart until I understood how it works.

I mean, this is open source. It is best. Yeah, yeah. I think that's generally how you learn best, just trying out things. And he was also Luther. Yeah, yeah. Affiliated, but not on the image inside. But back then, everything was very hackable, right? The delta between open models was pretty drastic between what was closed and good, right? So people on language model side were doing post-training. That's where a lot of the news guys started. In this, you can look at early stable diffusion. This took so fast. And then where we get to today is pretty crazy. But you both had pretty impactful roles. You did SDXL. You did latent diffusion. We'll get through all the history, but for people that haven't super followed the space, where are we today? Like what's the state of things? I know Flux 3 Video is out. Flux 3 Action, we're now like 7B doing robotics. What's the latest in this space with you guys? We started as an image generation lab when we started Black Forest Labs, right? As a kind of legacy of stable diffusion and the likes.

And that has been, I think, our thing for the first year. But then we realized, okay, you actually, image generation is nice, but this technology can unlock so much more. and that's when we slowly went into what we call visual intelligence which is basically trying to get our intuitive intelligence that we people have when we catch a ball and just understand physics without actually knowing how to mathematically describe it or something or when we grab a bottle and just know exactly how to do this because we've done it 100,000, maybe not 100,000 times, I don't know, but a lot of times in our childhood, right? We want to be transforming that knowledge and that form of intelligence, which is very different from language model intelligence, into models. And learning based on pixels, which means not only images, but also videos, and maybe also audio, say not pixels, but natural signals is maybe a better term. That really unlocks this. And that is what we realized.

And we realized, hey, we have the algorithm to actually do this, which is diffusion or flow matching. and that is when we just said, okay, we need to extend from image generation to basically all the natural representations, which led to Flux 3 basically, which is one model that learns multimodal from images, videos, audios, also actions. We can talk about how this comes together later. To really get this, to keep it with Daniel Kahneman, I don't know if you read The Geek Pass and Slows. It's a system one intelligence. It's not system two intelligence. Oh my God, everyone needs system one models now. Maybe language modeling is system two, but we're focused for now on system one, which is extremely important for everything that is intuitive and that requires real-world interaction. That's why, on a high level, it's also so good for robotics. So that's been the journey, and now we're really focusing on that. And in the end, obviously, it's about combining both of these systems into this very, very broad way of intelligence that the humans also have.

I want to go into sort of history and all these things, but just to also recap for people at the front, you're already making significant money as a generative media company, right? I don't know what the numbers are, and maybe you can sort of recap there. You raised a ridiculous amount of money. Yeah, we raised, I think, all in all, $450 million in funding so far. Yeah, so there's way more than stability. any revenue announcements that you've announced? Currently the last that was published in December was 100 million revenue, cross that since. Clearly across that. Yeah, exactly. Anyway, I'm just saying like, you did not have to pivot. And to begin, we're not pivoting, we're just extending. This model is really straight on everything. It's always been the plan. And obviously to create artworks, you also need this visual kind of intelligence to create a video that we see here, you need to know that how physics play out.

I need to know that the model needs to understand, this is actually interesting, but the model needs to understand that the arm of this, say, this capybara or what it is, is not just falling away, that it has to follow some law that is described by physics. It's just not describing the law itself, it's just describing the output. In a similar way that we humans don't understand physics, but we understand that if I let a ball fall, it just falls down, right? Yeah. Well, I think there's different concepts of a world model, and obviously that's something that we can get into, where can you learn, this is sort of the bitter lesson version where you just learn on a bunch of input sources, or you have to actually physically structure the 3D model and kind of learn from that. No 3D model at all. Just learning based on natural data. We also have a bit of triangulation from our eyes, right? We have two eyes, and that gives us a kind of slight understanding of 3D perception. But effectively, to a very large extent, even if you would close one eye, we could still kind of learn a lot about the world.

That is what these models are basically learning. You just learn from natural data, 2D video data. That obviously has a lot of 3D understanding, because if you want to generate a pan around an object, you need to be hallucinating how it looks from behind. And if I look at you, I have a pretty good understanding of how your back looks, basically. Yeah. Because I've just seen it before, right? So this gives you an implicit 3D understanding, but it's really trained on kind of 2D videos. Yeah. I'm trying to steel man the counter argument, the opposite argument, right? Just like the Gary Marcus, like, oh, let's don't learn anything. Or I forget who are the famous critics of this kind of stuff. But basically, it's like I have never lived. I, as a model, I only see on inputs. I don't have personal experience. You can model my back because you are also a human, and you can model other humans. but maybe this analogy isn't exactly correct because obviously I'm sympathetic to your side of the story. But I think that is how people criticize this approach to world modeling, which is that, yes, you can approximate closer and closer and closer, but you'll never quite get something core to the existence of the thing

because you never modeled the thing. You modeled the most likely constraints around the thing. I think to some degree you could argue that we operate under the same principle, right? Like, you can say, like, that we are, you know, okay, we have this embodiment, and we're constantly, like, taking in data, and, like, obviously we regularly update. But in some principle, like, we're still just taking in these, like, filtered signals from the world. I mean, there's all sorts of wacky things with how our eyes process, and we have to, you know, part of our brain kind of has to hallucinate some of the info because we have the blind spot in the center and other aspects of this. But fundamentally, like, I don't think there's that much of a difference between how we're out here taking in this natural data and building this model of the world that we're able to act on versus these models also learning from... Obviously, the signals are different that they're turned into pixels or into waveforms and processed through this other mean, but the information is still there for the model to take in and build this representation of the world that then you can essentially operate with.

I was actually arguing this with Lucas Beyer. He's also with Meta now, I think, formerly Google and OpenAI. He's in Zurich, which is pretty close to Fiberg, and he's coming over from time to time. We were debating this on human learning. Like, he was saying, okay, we actually don't need interaction, which would be kind of the, I think, what I would say is important for learning, like, higher forms of intelligence. It's like, first, it's observation. That's also how we humans learn when we're babies, right? We just lie around first and we observe things. And from that alone, we learn so much. This is where you can actually exactly map to what, how we're currently training these models. This is observe a lot of 2D videos, audio, and then they learn from it. And I think next step will also be getting, closing the loop to actually interaction with the world because that also I think that that very important in how we learn basically around the world We can then tap and we see oh there a correlation between I knock on wood and there a sound attached to it right I can just like manipulate the world then to understand it better

That's how I think how humans are mainly learning, at least how I personally learned. And he was actually saying he thinks the majority is observation, which I found very interesting, because if that would be true, then we just need to do what we're currently doing more, and that could basically unlock an insane amount of bandwidth. I wonder how much of this is like emergence with good priors, right? We've seen adding new modalities to say language models. You don't get the full benefit of doubling your performance since you've added vision to a language model. But if you have good priors, is there light benefit? Is there big benefit? It's interesting to see. I think there's a lot of technical stuff that will be interesting to walk through, like caution splatting was a big thing. World Labs did a lot of 3D data. Where are the trade-offs between video generation output versus what else? But I think something fun for listeners of a podcast would be the history. You worked on, you know, very early stable diffusion, SDXL, latent diffusion even before that. Video diffusion as well.

Video diffusion. Basically, there's a lot of buzzwords here. You know, LCM, LoRa, flow matching, old prompting. Can you teach us everything? Yeah. I mean, what's interesting? Like, how big are these? I've actually built, like, a tech tree. Like, I wanted to build this game where it's, like, civilization, but the tech tree is AI. Like, key AI paper. So to spend research points and get it. And so, like, these are the tech trees, right? Like, yeah, first it's GANs. Then, you know, dot, dot, dot, like, lane diffusion happened. But, you know, like, there's a lot of core points beyond that. I'd play that game. Yeah, I'd play that game. Okay, anyway, just to pitch the game, it's like you role-play as a Neolab and you have to spend GPUs or researchers. And then you have to collect data and there's all these little things you can research. Come on, whip it up. We're in a bi-toting role. I want to play it tomorrow. I think game balance is really hard because I had AI's played.

I built it. Oh, fantastic. But game balance is really hard. It's actually an AGI hard problem, I feel like, Because, you know, once you find a way to win, you just always win. And it's not interesting as a game. You need multiple ways to win, and you need high-risk ways to, you probably lose, but maybe you win. Multiple Fs, classic problem in Gans, by the way. So I think one thing that people are interested in, I'm interested in teaching our audience. Like, what are the key points that, like, if you've heard a lot of terms, like, what are the key things that have really impacted the field? And, you know, a lot of you have, you've lived through a lot of that. I'd also say this is a great way for people that want to get into the space, good interview questions if you're hiring as well. You should know most of these things. I think there's some points where there's spicy, different paths that's popped up, so autoregressive image gen versus diffusion. Two backbones or over glass diffusion. All these things. What are the key things people should know? I'll leave it pretty blank. I mean, first would be SDSL maybe. Yeah. Talking about SDSL a bit.

I was going to say, what were the things we did in SDSL? The refiner. No, please, not the refiner. Why that reaction? So SDXL was a big piece of work in trying to make the models feel good to use for the first time, if I was to kind of, like, lay anything out. Not to say, like, Stable Diffusion 1, 2 weren't, you know, I mean, they were very exciting, they were very hard to use, but to anyone that was, like, in the space of the time between the period of SD, you know, SD1 to SDXL, we saw a ton of work out in the open source community working on how do we fine-tune these models, how do we make these models, you know, more aesthetic, How do we make these models into, like, you know, multi, you know, if in our members, like, the original stable diffusions, we're only able to do square images. And so SDXL was the first work of trying to actually release, like, the base foundation model kind of with a lot of these general improvements implicit in the model. So SDXL involved, you know, the first time we did large-scale multi-aspect training. There was a, you know, at the time, this was before packing.

there was like an efficient way we could do essentially bucket style training. So training, you know, on all sorts of different resolutions of images, you know, at large scale. This was the first time we did multiple actual training stages. So taking it from pre-training, actually developing like post-training sets. So there was a lot of data work that was done in SDXL. I'm trying to think like all the other aspects. And then I'll give you kind of a little bit of a backstory on the refiner and maybe some things that didn't quite make it in. And so here was the whole process of the final release model is essentially SDXL fundamentally works similar to the original stable diffusions. It was a latent diffusion model, took in text. You had a noisy latent, took the context of the text, actually generated the imagery in latent space, and then a VA decoded it. And then from there, the refiner was actually another full diffusion model that the whole point of it was to take essentially what was not a rough image, But SDXL wasn't the best at what we would call high-frequency detail,

which is this, like, the fine detail, like skin texture, texture, grass. Yeah, upscaled. It wasn't an upscaler because it didn't actually change the resolution, but it, yeah, upscaled in terms of the, refined it. The reason I thought it was janky is because I worked, not just me, the whole team that worked on this was fantastic. I don't want to take all the credit at all because there were a lot of fantastic people that worked on this. We did a lot of work to try to push the detail quality of the model while trying to keep this, like, generality, and we just didn't quite get there. And so then there was this, like, side track of work of, like, okay, what are other methods to do this? And this is where the refiner came into being with actually. The refiner was actually a model that was, it was what was supposed to, so this is a little bit of the secret lore, is there was a version of SDXL that was never released. It was a different architecture. It didn't quite work as well, but it was trained pretty far along, even to, like, the post-training stages. The refiner was that model. It is a full diffusion model. You can't even forget about that. Yeah, yeah.

Like, the refiner was a full diffusion model. And the one thing about that model is it had a little bit of better performance on these high-frequency details. It just had very poor structure. And so this whole idea came of, like, what if we essentially, like, different model, different job. Different model, different job. We took it. We did a lot of work to really push the work on how well it could actually refine this high-frequency detail, and then we made this essentially this pipeline. Yeah, which is the X-File part. Which then became, okay, this together is like a more performative model, or a more performative system altogether. And then that was what we ended up releasing here. Which is actually interesting. It was one of the early pioneering works of like actually that looked into, okay, what kind of properties learned in which part of the trajectory of this diffusion kind of trajectory, right? because you have these structural things that are learned closer to the noise, if you become closer to the noise. And then you have the fine, high-resolution, high-frequency stuff that comes further down and is closer to kind of the clean image, basically.

And today there's a bunch of different works that actually analyze this, but you do actually, you kind of, you generate from low-frequency to high-frequency if you go from, like, noise to clean image, right, along this trajectory. But that was not really known. Like, it was informally maybe a bit known by then, and this was one of the first works to basically exploit this and say, okay, I actually dissect this kind of trajectory and I have two kind of experts, one for the structure and the next for the texture, basically. And right now we have modulation, which is like conditioned on the time step to actually account for that as well and stuff like that in the architecture. But that was really one of the first works to really make this distinction. And as you see, it worked, right? Yeah, it did work quite well. I guess the reason I still hold frustration because at the time I wanted this one model to be so good at everything, and I felt maybe it was like a little bit of a cop-out. Nowadays I take a bit of a different view because, I mean, we see systems and models do incredible work together, but at the time I remember feeling, and I'd still just have this like old frustration,

just from back then. Well, it's also very fair, because you should always go with the simplest solution that works, and if you then have a kind of a... At which point do I need a machine learning refiner? approach. Yeah, it's just like, when it comes to scaling, whatever the simplest solution is that works, you should go with kind of a bit of Occam's razor as well in statistics. Yeah. So, you know, Runway was also kind of doing their Gen 1 and Gen 2, and Anastasis on this part also admitted that Gen 2 was pipeline mix. No, it works. They just didn't admit it, whereas yours was public, so you had to admit it. I think if you go back to At the time, too. People coming into the space today haven't seen 200, 500 steps of diffusion, right? As you say, you have a background, and then minutes later, you have little fine details coming in. It was also very tinkerable, right? This came out as what, like a 3B, works on 8 gigs of RAM. So as much as you could, bit or less, and do the general approach,

if you're not a lab and you're just doing this at home, you have one or two GPUs, you're going to do pipeline stuff, right? It's like that's how you can see the differences right there. Yeah, yeah. So video. I've been working on video for quite a while. Actually, already before stable diffusion, I was working on video generation for NVIDIA. Had a paper there, Align Your Latence, where we basically mapped the – it was actually in parallel to stable diffusion. We basically mapped this latent diffusion algorithm to video, which was the natural next thing to do. It was very exciting to kind of then advance the things moving. and you can't imagine the kind of compression reasoning behind why stable diffusion was so efficient compared to other models in the time also holds for video where you actually have this temporal dimension that you can also compress slide. So that made a ton of sense worked on this for NVIDIA and then when we joined stability we basically after I think after SDSL we worked on this stable video diffusion which basically

first apply this latent diffusion approach to video as said, but importantly this work was a lot about investigating the effects of data filtering, which as we felt, or as I personally felt back in the day, were like in research, and they still are, very much underappreciated. Data is just like when we think about what these models are doing, effectively we're estimating a probability distribution and the data set is what gives rise to this probability distribution, right? And if you have the perfect algorithm, the best algorithm in the world that just learns anything you want, but your data is very bad and you have, like, on average, blurry videos or images and, like, not a lot of motion in the videos, your model will learn that, no matter how good your algorithm is. So what comes out of the model will actually be very bad and, like, not nice. So the data is one of the most important levers that you have as one who wants to train these models. And in the entire research field, there was never much talked about

data because a lot of people from the research community, they just worked on benchmarks and tweaked the algorithms. But if you talk about frontier models, obviously we know that by now, that data is extremely important, right? And that is what we wanted to be really sharing more context on. So we investigated how much data... And then you have this trade-off, I want to be training on a lot of data. Quantity versus quality. Exactly. Where's the sweet spot? That was what we investigated it there. And then also, a very interesting card we should pull up. We can actually take it. The most exciting experience that we did, it was not the data one, but actually if you go down a bit, and it comes back to what we discussed earlier, we then showed that just pre-training on plain 2D videos allows you to fine-tune an explicit 3D model extremely fast from it. So if you go to the last paragraph here of the paper, here we go. These are these 3D views here of the model. So what we actually did is we took this video model

that we trained, and then we had, I think, 1K fine-tuning steps on 3D data, and we got a 3D model out of it. And that comes back to the question, okay, what do you actually need to kind of understand the world in 3D, right? Only 1K steps of fine-tuning that shows that this video model, although only trained on plain video, actually had this insane amount of 3D understanding, and you just took a bit of a a focus to get a 3D model out of it. And that is really like, that was one of the early experiments where we saw, okay, this is actually kind of, you can get world understanding through training on plain video. And like from a research perspective, this was like, I think, the most interesting experiment in this paper. Also pretty short paper. There's like a section here where you even explain the pipeline, right? There's like three, four stages, pre-training, some post-training. Yeah, stage one, image pre-training, video pre-training, video fine-tuning, and then 1,000 steps, you can also make it 3D, right? Mm-hmm. Crazy.

At the time, I don't remember what I did, but there was basically this temporal consistency problem where if you just stretch out diffusion and try to stitch frames, it wouldn't work. So a very old hackathon project where I forget the method we even used, but we fixed this temporal consistency to scale it out to 15, 20 seconds. It took forever, but it was a very hacky solution because you could do this in an afternoon, right? But clearly you need this kind of learning over the temporal dimension. And if you just sit together kind of a very fast use model, you have this flickering. Even if you fix the feed and you do a shit ton of hacks, but you realize you need to. Today everyone is using Transformers, so in the Transformers week, you need the attention over the temporal dimension as well to kind of learn from that. and then understand what is actually, how does consistent motion look like, basically. And as far as I understand, we asked this of the McGraw-Imagine part as well. There's no compression in the video side that is useful, right? You just have frame, frame, frame, frame, frame.

Exactly. It's the simplest version. But it's, I mean, well, if you look at the MP4 codecs, you have very smart compressions, but that is lossy, obviously. And you need to connect the frames in a way and extract how consistent motion looks like. I don't think any video model exposes this, but how far you look back in the temporal attention. Because sometimes you do need longer, or it needs to be more adaptive maybe. For example, if something gets occluded and then it comes out again, you want them to be the same. But how long is it occluded? I mean, principally that depends on the problem you want to be solving, right? If you want to be creating a kind of a long memory model that can create your virtual worlds, then you obviously need to find a way to look back very, very, very long. That's super memory-hungry. And that is just, I think, also one of the most exciting research problems to work on these days, long contacts. It's just extremely important, right? I don't know if there's more there before we cut over to Black Forest Labs.

Anything else in the stable world? I mean, there's ST3. That was the last paper that you did before. Yeah, SD3 was a, I felt like a pretty big paper. That was a culmination of my work of stability, I would say. Yeah, absolutely. Your graduation degree. If you want, yeah. This paper was really about understanding more in which part of this trajectory from data to noise does the model actually learn what properties and where do we have to spend most of our computation that we put in training to maximize the performance of the model. And obviously that was also the transition from classic diffusion to flow matching with a slightly different paradigm, basically. We call it rectified flow. Exactly. This was also the first time I used diffusion transformers, right? Yeah. And then a bit further, yeah. So lots of changes, like going from a convolutional architecture to its former architecture, changing the probabilistic model from

a diffusion to basically flow matching, which is way simpler mathematically, which is super, super nice. I think you should actually define some of those terms for people that haven't heard. Play in the vision, you know, flow matching. How did you come up with the concept of these things, right? Well, he didn't come up with flow matching. Exactly. We didn't come up with flow matching. We just saw this algorithm as something that is in a way more general because it can, like, for diffusion, you always need one of your two distributions between which you're basically translating needs to be a standard normal. That does not hold for flow matching anymore. And instead of approaching this kind of problem to transform one probability distribution into the other, seeing it as a solution of a stochastic differential equation, you're actually seeing it as a solution of an ordinary differential equation which simplifies everything completely, like mathematically completely it makes also fast sampling much more feasible

you could say because what we always do obviously is when we have trained the model we want to get making as many steps as little steps as possible to basically sample as fast as possible right some people would say it's actually a special case of diffusion flow matching others might say okay it's actually a generalization of flow matching of diffusion the flow matching algorithm because you go from you can translate between two distributions no matter how they are shaped, and you don't need this kind of standard normal, basically, to be one of them. But, yeah, in general, it can be seen as, again, a model that learns, basically, a vector field that translates between two distributions. And once the model has learned that vector field, you can just integrate it numerically, which is then as easy as solving an ODE by means of an Euler forward step, basically. So it's very similar in many respects, but it was just like, I would say, mathematically easier, more general from the kind of probability distributions that it can translate between.

That's why we basically went with it. And maybe as a third part of that, it's also easier to derive a fast model from those flow matching models, basically. It was much simpler, more scalable. It was much, much easier to work with and then had this nice performance benefit as well. So, like, I don't know, the diffusion space got quite complicated. Are we satisfied that there's no loss? Is it lossy in acceptable ways? Lossy? You mean... Lossy compared to full diffusion. No. Yeah, okay. So that is good. It's a strict improvement. Exactly. It's great. It is an improvement, definitely. Yeah, I mean, it's effectively just drawing a line between, like, you know, noise to data, really. It kind of took this, like, there was this complex space on, like, how do you map between these two, you know, like these two distributions, as Andy was saying, and it simplified it down to, like, the simplest of the line. Because you're trying to do a shortcut, like, what if it's a bumpy distribution? Yeah. And, like, the most important part of the paper, or one of the most important, like, on the probabilistic modeling side,

was actually looking at the trajectory from noise to image and which paths really matter and which don't. Because if you just like, we train this in kind of steps, right? We always, on this part from data to noise, we only always train on one kind of step and then we approximate this direction in flow matching with the model. And if you just sample those steps in between data and noise uniformly, you might actually be spending a lot of time optimizing for things that actually don't matter. So one central part of the paper was where should we spend our compute, which parts of the trajectory from data to noise carry the information that's most important, because this is where you should spend the bulk of your compute at training, right? And oversampling these parts during training, and then undersampling other parts that are less important. It's effectively like a much further, better exploration of part of what we talked about with SDXL, which was like the refiner. We spent time training the refiner on the end of this trajectory to learn these high-frequency details,

and this was more of an exploration of the entire space of where you should spend time to make the most general good model to learn the data. Amazing. One thing I did pick up with SD3 was text in images, which you guys showed off a lot. There was a time when images could just straight up not do text, and then they did more and more text, and then now it's like, you know, here's our nice typography. And I feel like, can we attribute that to the diffusion transformer? I don't know if I could say it was the diffusion transformer. I mean, I think it really was just better modeling and better data. Like, it was fundamentally, like, as much as I want to, like, as much as, like, I don't know, you always want to attribute some fancy thing, it really was just that it could model the world better. It's the data thing. It's also, like, you always need, like, for everything that you train on, you need a caption, right? If your caption doesn't describe the text that is visualized somewhere, your model will not learn it. So that's also something that is often, sometimes we spend a lot of time talking about the nitty-gritty details of some algorithm.

In the end, it's again the bitter lesson. Simple, scalable is almost always the most effective. I would say always the most effective. But you can obviously scale in a smart way. And still, if you spend tons of compute, you should be spending it as efficiently as possible and only where it matters. And that is also at the heart of this paper, basically, right? I think prompting also got a lot easier at this time. We used to have the 4K. 6K. 4K. RCM. Ultra. 8K. Yes, yes, yes. Greg. Who was the Greg guy? The poor artist. Yeah. And now you can hear him. Oh, Greg Rakowski. Yes. I wonder how he's doing. I hope he's doing well. I hope he's doing well. Yeah. And also, you know, for context, this is when Dolly 3, Mid Journey, Ideogram V1, nothing was open. Yeah. So anything that you would see of this quality is either old stable diffusion or pretty much a closed. Then you had to make the big break to Black Forest. What was that whole period like, right? Like basically you had to leave and start a lab from scratch.

I think it was probably one of the most exciting times we all had. Because when you're in stealth, you're just like, you're a small team. We were 13 people, I think, when we started. We first went to France, had a fun little week in the mountains in France in a chalet. Spent the entire week just hacking, hanging out. Dustin figured out some really cool stuff to make our models way more consistent. I remember that. That was so fun. It was a magical week. That really was our kind of founding off-site. That is when we started training Flux1, which three months later resulted in this nice company launch, basically with a model that I think blew pretty much everyone away, at least it blew us also away. Commercially, what this directly unlocked, like only having kind of incorporated the company for three months and then releasing this model, and it was just like what's received very, very well, right? That was amazing. So that was a lot of fun.

But did you know that when you started this, I mean, you had to get all the clusters together, like all the data, you had to start from scratch, scratch. we had start and stretch yes but we knew what to do basically we had a lot of experience but like did you have you just did SD3 did you have a hypothesis that you were exploring what you would do differently we've obviously gathered a lot of knowledge in these entire times even before stability as well and that all culminated and then we also had a team that just like was in one place basically that was very focused, which had never been the case before, because if you're working from very different parts of one company, also as Dustin said, Dustin was actually not in our team. And then we basically had a super nice team together, spent a lot of time together discussing how we would actually design the kind of learning process of this model which was at this off and then we just kicked off and that resulted Was it like discussions of what is Flux 1 or was it broader where do things go in a few years

How much of this vision was like working? Well, I guess if I can jump in and say, I mean, one of the reasons the company was started, I will say there were some frustrations of time at stability. You know, we saw from our side, we saw the impact that our models have made out in the world. We saw not just the impact from like the open source world, but also we saw companies that were started and run and quite successfully used our models out in the world. Like we knew there was a lot of draw and a lot of potential in what our models could do. And unfortunately, I think we also felt that at Stability that we couldn't essentially leverage the value of what the models could be as well. So not only did we have the belief in, like, our team's ability to actually do the research and perform and build these models and get them out of the world, but this would be our shot to kind of correct those inefficiencies or mistakes of not being able to actually capture the value of the models themselves. And so that was also a big part of the belief of why we thought we could, you know, go forward.

Because, I mean, we have this record of doing this and making this impact, and we figured that could continue on, and we can actually make this successful company out of it, which now, I mean, I think it's gone pretty soon. I think it was a good thesis. Yeah, it was a good thesis. As I said before. We've been talking about how we can generate images and video and improve the perceptual understanding of models, but what about voice? And that's the subject of today's ad. We've been working with DeepGram for a while on the AIE side, and they recently acquired PipeCat, which has also been a big sponsor of AIE. So DeepGram is becoming one of the critical pieces of voice infrastructure for all agents. So this episode is sponsored by DeepGram. If you're building a voice agent, you need it to keep up with how people actually talk. That means understanding words in context and keeping latency low enough for a conversation to flow. That's what DeepGram's speech models and real-time infrastructure are built for. One in 5,000 organizations uses speech-to-text, text-to-speech, and voice agent APIs in production. The focus is accuracy, low latency, and cost efficiency, so you can build and run voice experiences at scale.

You can use the cloud APIs or deploy it yourself, including on-premises. And there's an offer on Flux text-to-speech. Spend a dollar, get a dollar in credits, up to $500. To sign up for free and find out more, go to dgram.com slash context is everything. dgram, context is everything. Thanks to dgram for bringing us this episode, and now we'll go back to the interview. And, like, what use cases were you seeing? I think a lot of people went straight to, like, Hollywood. Like, you know, that's our goal. Or, like, Ahmad used to say, we will make Game of Thrones Season 8 again. I mean, I'm trying to remember this. I don't know if I'll be able to remember the name of this company. This might have even been before SDXL. But it was a company that for like a week or two, it went viral where anyone could take like a picture of themselves. And then it created the older version or so. The older version. And it used like the old Dreamboost style fine tuning, which was extremely inefficient. And then, but it like blew up. Like everyone was using this. Everyone was excited about it. The models could make so many things, and we saw that there was a lot of potential in how everyone wanted to use it,

and we saw where the models were going. At the time, they couldn't do typography. We knew, obviously, okay, typography is going to be important, and we eventually got there with SD3 and so on. There wasn't just one use case. We saw that this was like a general technology that had a lot of potential and that was going to continue to grow and have potential in the future, which now is expanding into the physical world a lot more. but yeah this was actually already the potential we saw like yeah yeah absolutely BFL was not founded to be an image generation lab it was more of a kind of a matter of the the flux moment being so good that we then capitalized on it as well because it gave us a kickstart but when we pitched the company to investors for the C-Series it was clear we want to be working on video because we see that as a I think we shared this on a podcast actually back in 2023 already when we still had stability we were saying okay this like video modeling is actually the central means to understanding how our world basically works. Because, again, what I said earlier, right, essentially we humans are also first observing the world, then starting to interact with it,

and by that we learn so much. So that was always a thesis. And then obviously we had to kind of take this detour, which was very successful, I would say, through image generation. And then you always face the kind of issue of, okay, where do I spend my time on? That's unfortunately a thing we cannot get across, that the day only has 24 hours, unfortunately. But eventually, I think, Flux 3 is really where we saw this going for a long time and what we wanted to be doing for a long time. And that's why we're so excited about having the model out there, being able to create videos, being able to create images, being able to create audio, being able to act as a robot brain to actually kind of work in real manufacturing workflows. And there's a lot of stuff coming this year still. that we have to announce on the physical AI side. But really, as Dustin said, our goal was always to create the most general technology, and we see this happening currently, which is why we're so excited, basically.

Yeah, I mean, I think you guys also took the right path to stay focused, right? If I'm not mistaken, Flux1 launched with XAI. It was their mid-gen model, good partnerships. That's right. Before they had their own, but that's a good way to solve, do we build distribution ourselves? Even to the latest, Flux 3 Max is post-trained with FAL. They do a lot of entrance distillation. They're good at that, right? So it's interesting. But it seems like you're expanding now towards you have, what's the latest one with robotics? Flux 3 Action? Flux 3 Action. Before we get there, there's one more one to cover, which is image editing in context. Sure. We have a key. Also, really nice history right here on the web. Yeah. It's weird. We're now at the stage of company that things just happen, and I'm like, oh, we have this. But I was kind of sad Flux 1 wasn't on there to view, but it's like the original one. Yeah, but it's fine. I think even just the visuals here, like maybe someone will clip it off. Original Stable Diffusion and Excel.

You see the progression. But one thing that I do want to cover at some point, I no longer personally see the progression. You show me state-of-the-art image video models. A lot of Chinese labs are putting them out. It's hard to eval. I know people care. I've talked to Sumit at BFL about, okay, no, actually people in the industry, aggressively A-B test industry, there's a lot that people care about in evals, but there's also a level of it all looks very realistic to me. The rest is prompting skill issues. I think there's a lot of little nitpicks. Like there were hands, there's still issues in tech sometimes. There's still hands issues? No, there were hands issues. There's issues in perception and layout. like I'm redesigning my living room. It's horrible. Controllability is the thing. I think controllability is the thing we need to solve, like, large scale. There's so many ways you can control these models. And maybe that is a good stack into a flux context as well, because that was a pain point that we saw for a lot of time. People were training so many LORAs to get this character consistency into models, right?

I tried to replicate back then with, you know, all the LORAs. And then we had this amazing pitch of Dustin. it was also in France there was another cabin a lot of things happen in cabins we should go to a cabin you should do more and then we had this pitch of Dustin who actually saw this potential I need to get this context I need to be able to prompt not only the text but also images for image editing I did this pre-hype as well I've heard the story that Flux context was like you and one or two other people Yeah, so for a long time it was basically my side project, and then I stole one of our data guys a little bit into it and said, come where we're building this thing. So for me, like, so yeah, as Andy was saying, it originally served as a pitch of mine at some cabin in France. I did essentially like a, I like to do these presentations every, you know, every off-site or every time we get together, where I try to look at, like, the state of the field

and where I think the fields are going and kind of where, like, What is something we're not doing or something where these models have a potential to do something novel and useful that, you know, maybe we're not looking at right now or maybe isn't solved? And so I actually did a presentation on, I know this is like the most overloaded word in the world at this point, but I was like, oh, it's the Omni, the Omni model presentation at that point. Because there was a lot of research work out there at the time on, you know, other editing models. If we remember back to, I think there was the old prompt to prompt, and then there's a, why can I not remember the name of this? The pitch was essentially these models are clearly learning, like, these deep relationships about the world, and this is going to keep coming up. It's about these relationships. And to me, you know, like, the relationship of, like, okay, this Coke can is sitting on top of a table. This bottle is sitting here. And you prompt the model, and you go, okay, I want a photo of a dog wearing a hat. And it needs to know, okay, the hat is on top of the dog. It needs to have this general understanding. And my kind of theory in mind at the time was that since the model already has all of this, like, spatial knowledge,

if we essentially build the right data and train the model, like, train an already pre-trained model the right way, we should be able to evoke essentially, like, general image editing with very, very little training, capturing these already, like, pre-built relationships of when things are there, when they're not there, and how these, like, relationships, these spatial relationships inside of an image work. And then essentially what it became is like a three to four. I don't remember the timeline of me here over in our SF office with the one data guy stole building out essentially just early test training data sets. And I remember the moment I saw that this was like clearly working quite well when we made this effectively tiny data set of just very like high quality labeled. It was essentially describing like simple relationships in an image. And that was really hit. like it wasn't a way that you would prompt a modern like edit instruct model it was just literally describing these relationships and i trained the model for like i mean it was maybe like six hours of training four hours it was it was very very short training run and all of a

sudden like i saw what felt like a clear generality coming out of it because it was i think it was like a photography data set it was just like photos of stuff but even just from that i was seeing like i can put a logo on a shirt i can put like i can do and it wasn't good it was just like The model was clearly learning to attend between these two concepts and use these pre-trained relationships. And then I took that to our next off-site, showed results, and then it became this essentially more company-wide project for about a couple months where we really put effort in, brought people in, really improved essentially. It was mostly a data set project. I'll say it was mostly a data set project. And then the other bit of installation. And then there was a lot of very nice distillation work by Axel that was done to speed it up. then we essentially were able to put this model out in the world that I was quite proud of. That was the first model to really do this character-consistent editing, right, in high speed that made it, again, accessible to so many people. For most people, it's very common to generate something, chat with it, change it.

Before this, image generation took quite a while, and you needed to recreate the image. You couldn't change the parent. You just get the same character. It wouldn't be the same. You need to mask something that you don't like. I wanted to get away from the slot machine, essentially. Everyone had to do the slot machine, prompt and prompt and prompt and prompt, and it was very clear that iterative editing was much more effective. So my Minecraft VCon VUI was that. It just swapped out like 10 different versions. My favorite part, I still swap out. Like I'm generating 4, generating 6, redoing something. Just make 10 iterations. And eventually that one's good enough. My favorite is that you pray to our Jesus. random number. Basically, to me, this concept of what is a foundation model is something like this, where it's like, okay, you know all these little workflows that you guys do to do that, and do that, and do that, and do that, and do all these different tools? We're going to replace that with one model. That's actually what we're building towards, right? That was where Flux... This was a very clear aha moment for me.

I don't know the... I never tried prompt-to-prompt, but I think, to me, the closest that has come to Flux Context is Quint image, which the Quint folks are doing really good. They had a Photoshop layer model. Yeah, no, they're doing pretty cool stuff over there. It's interesting how much of this just comes back. There's a lot of good data. Yeah, I mean, it was targeted good data for was it a good concept? I don't know how to even describe this here, but yeah. I think one thing that I'm surprised by is, so, you know, comment on stability one more time. Stable Japan, right? Japan had this whole law of like, we will protect you if you train on copyright. I'll open anything up as a country. And I was like, ooh, it's stable Japan. It's interesting. Nothing happened. I always wanted to visit that office. I'm sad I never did, actually. I think on your Omni point as well, it was an interesting time. There were the two others. Gemini is obviously the latest Omni. ImageBind, which I didn't realize is 2023 from Meta. They threw in one in my head. HeatMaps. That's a big one.

Which, you know, I'm like, oh, very early. But like, it's cool. I think this work has progressed. There's definitely newer work on it, but like IMU. You know, this is just another color, right? It's RGBA, whatever that is. Yeah, yeah, yeah. Right? So how many colors are there? Oh, the fun thing, this was also all in, all out. So you can go text to heat map if you want. You can go IMU to depth. Yeah, big market for that. But no video out, which is, I think, the kind of the central thing that we need to understand, basically the world better at. This is, I think, what I said, what has always been at the core of what we're doing. Yeah, okay. Well, before we leave the sort of data or IP discussion, is there a particular kind of data that everyone in the industry knows is valuable? If people have a data vendor listening, how do they sell to you? How do they approach you? It depends heavily on what type of problem you want to be solving. That's just super important. And if I train a robot, I obviously need a lot of eccentric data.

I think that's not a secret, right? Because that is just how this thing will be. Egocentric? Yeah. I mean, there's also a benefit for you guys, right? On the language model side, there's a lot of people doing RL environments. Is there a parallel that people should come to the team? I'm trying to get you vendors. Oh, we have so many data vendors actually that are both. Too many? Yeah. It's a lot. Oh, okay. What is good versus bad? it again depends on which stage of the training you are and what your effective goal is I think there's a trade off between we have this already right scale of the data versus the quality and that is what you have to for every training run like last scale pre-training you do again you have to kind of find that sweet spot out because depending on the architecture you might be using that might look very different again and that is one of the central pre-training experiments or pre-scaling experiments that you need to do. You need to do a letter basically to understand at scale how much to predict at scale how much data do I actually need and how should the quality look like. So where's

the sweet spot between these two dimensions of data quantity versus data quality? Any other major jumps that we want to call out? So Flux 1 context was a big one. Flux 2, Flux Tools, Action, we talked a bit about. Any other? I mean, I think the three sizes, that's very important in terms of like the dev models and the Schnell. Schnell? Schnell, yeah. What is Schnell? Schnell means fast in German. I think you probably covered that in the book first. But yeah, I think that you have open source, you have the closed source, and you have a range of performance. I think that makes sense. Yeah, absolutely. I guess if anything to track on is like, okay, the one thing that made me quite sad that we didn't quite get in with Flux Context is that it was only single image edit. Because, I mean, And it was very clear that there was a huge need for multi-image. People want to be able to put in many, many different images and references. References is a better word here, yeah. Many different references. Which I think Grok is king right now. I don't know this thing. I know Mid Journey does quite a bit on style books where you can build up references.

They've always been very good at the style reference, but this is more just general. Flex context would only take one image, and so you could only iterate for that. See, a bad photo of one person. But you could have three different photos of the same person. But it was actually general enough. I remember it was actually quite funny. People were, like, doing workflows where they were stitching a bunch of, like, images into one image to, like, emulate it. But then with Flux 2, we finally were able to, like, spend the time to kind of build out the data and actually do this larger scale training for handling, I think it was 10 reference images at the time. So essentially building up much more context to work within. Yeah. Which is, again, also a lot of a, to a large extent, a data problem that you just need the data. Yeah. As many references. Yeah. Is there a practical limit? So, you know, let's say the industry standard is like 10 characters or 18 characters. That's it, right? Yeah. I don't need 100. Obviously, you can show crazy numbers on how many characters or like images you can get into one image or also video. But there are like that is agent work then to get me 100 references. That's the easy thing, right?

It's agent work. But then I need to check it. Like I have to review 100 images if that is actually what I – it's actually also a hard creativity problem. Like how should I – where should I put these kind of – Composite these things. 100 images so I think without like over indexing on the number everything that goes up like over 10 references is also hard to follow for a human I think like if you say okay we want to be creating a video right I obviously can put in 50 people there but if you look at the most important movies how many central characters do we have like 5 6 maximum unless you're Avengers exactly then also at some point approaching the kind of limit of what we humans can actually kind of process in a two-minute clip or something. This also gets way, like, coming back to the, I know you mentioned the evals earlier, building evals for larger, like, contextual models becomes a much bigger process. Like, even for, we ended up releasing an open eval set for, I think it was ContextBench, which was built off the internal bench that we used to evaluate context.

And then as you scale up how many references... Oh, I haven't seen this. Yeah, yeah, please pull up context. This was a lot of work, but it was actually quite fun to put this together. It makes a cat very fat. Yeah, I mean, I remember sitting in the back room of our SF office with, like, a few of the other guys. Like, me and Tim were laughing. Tim Dockhorn is another one of the great guys that we're doing. I feel like, so, like, you should take some edibles and just go, like. I mean, you're just sitting there late at night, literally just typing in the, like, we're just pulling it. It isn't typing in the stupidest things. I mean, we're. Yeah, that, like, but it's, like, real. Like, I don't know. Because in the end, you always need to kind of make one step to make it a kind of. What is ground truth? You need to make sure that you have a ground truth somewhere. No, this is just the benchmark, basically. It's just inputs. Right, but where's the output? Where's the e-mail? I mean, that is the interesting thing, right? You just ask people. This is a public bench to give everyone like... This is the e-mail. Yeah, this is the e-mail, and then you have two models that you want to be carried, right? Yeah, exactly.

So it's not a validation set in that sense? It's just the e-mail. This is just Dustin sitting here. This is, it, it, it, genuinely, like, 80% of this is that, like, it, it's, um, also heavily influenced by our, by our, uh, I think by our Italy offside, uh, Why do you do, like, a Mad Libs, like, like, a combinatorial explosion type thing that is more structured than just random expression? So, so, so, I mean, now we're doing, we definitely do more stuff like this, I will say. So, one of the things is there were, I mean, there were certain things we had to, like, filter out for the public set on here, because there were some very dumb things in the internal set. But I don't know. I spend a lot of time. I like to see how people actually use our models. And I'm someone that actually really likes to use our models. I think it's very, very important that you test and play. Like, really just play. I'll say play with your models. Like memes. Like memes. Yeah, use it for what people actually want to use. Like I said, I came from Discord open source communities. You see people doing the stupidest things. But it's like people are having fun with it. And so a lot of the set, you know, besides trying to cover a bunch of actual use cases, like production use cases, which were very important, I also want to make sure it's just like the models are fun to use.

Yeah. And so. Good vibe-y valve, I've been. Exactly. Yeah. What I think is like, I underappreciated three years ago, which was like, yes, this is memes. Yes, this is like a mid-journey, whatever, subscription. But like, it actually becomes very real work. Yeah, exactly. Absolutely. Because they've done it for fun. Then they're like, oh, I can do this. And there can be also like when you automatically eval these models, like there's a bunch of obviously human evals that everyone is doing in the field, but every evaluation will have their blind spots. And you should be combining as many independently as possible to kind of picture as good as possible of like how the model is performing. But that doesn't like remove the burden from you to kind of, or it's not a burden. It's actually super fun to then sample the model yourself and, like, so I understand where are the strengths, where are the weaknesses. Like, this is the core part of developing a model is actually responsibly sampling when developing it and checking, is it where I would expect it to be, right? At some level, it's, like, what you guys care about too, right? Like, these are your use cases. It's a good way for, if you want something to get better,

like, make a little benchmark and see how things perform. No, absolutely. I think it's very important that the people building the models actually, like, want to use the model. Like, I think this is very important. There's a big distinction in this and, like, the microphone brand trying to, you know, use it for product photos. They're a very niche use case that they can min-max on compared to remove. Very general. Yeah. Yeah, I think one thing I do doubt or question is that it's a very well-known mid-journey loop, which is it's Discord. Everyone sees, like, other people prompting. You get the four options. You choose the four. That is feedback for your next model. How come you didn't do any of that? We didn't do it publicly. scale. So every one of them... But isn't it valuable to do? It can be valuable, but I think only up to a certain extent that if you then, like, cross certain limits, the signal that you get is pretty, pretty... Like, early days, we saw this insane potential, right, of this tuning, say, of majority models, like V5 or something crazy. But if your base model just reaches a certain kind

of level of strength, it's actually you don't get these super good examples anymore where you can clearly see the model improving from this, and this is also a lot of noise. So there are a lot of other ways how you can actually... I had this debate as well with someone where back then it was very useful because prompt adherence wasn't great, right? It's pretty judgmental on which one follows the prompt better when neither of them do it great, right? So you have a lot of preference tuning towards there's very high signal and okay, the people prefer this. Now back to my point a lot earlier, like if I take the top five models and tell them to remove a bed, it's all going to look the same to me. The preference is not that much signal, right? They're all filling it in pretty good because there's like, they understand this task very well, so that side of the eval for me is kind of gone. It all looks fine, but yeah. I was going to jump and say that was actually one of my favorite parts of moving into context is finally it wasn't just like, I don't know, it was finally just like, okay, did it actually just do the task? And I really enjoyed getting to these like very kind of clear, concrete tasks.

One more part of the tech tree, and then we want to go into robotics in action. Autoregressive. It seems like a big trend with both Gemini and ChatGPC doing it. Is that as big an impact in your mind as it is in ours? You know what I mean? It seemed like a very big change because then you could conversationally edit. It is the next obvious step from Fox Context where instead of editing with an image and text wrong, you conversationally working with a model That I would say is not a feature of autoregressive models only It just based on the data you train on it And like you can get language capabilities into a diffused model as well right Which we know is working. And as I said, long term, this is where we believe this will also go into and where we're working towards to also get more modalities, including this language model, right? It's not a question of the algorithm there. it's just a question of like what you focus on basically

I think both approaches are extremely exciting and have a lot of strengths and a lot of like shortcomings I would say autoregressive models are way more data efficient you can train them way better in data because every like for diffusion models you have this trajectory from data to noise right and every training example you're training on is effectively because you have this continuous path from data to noise in between it's giving you rise to infinitely many training examples because you can pick every possible spot to denoise from this trajectory between data and noise for one example. Obviously, the model would overfit if you would only train on this one example. Theoretically, it's just like one training example is giving, or one data point is giving rise to infinitely many training examples. Whereas, autoregressive models can just train on all the tokens in parallel, basically, and get the maximum gradient from each training sample. That's the training side. So there I would say autoregressive models are more data efficient.

However, at sampling time, autoregressive models don't get across that. They have to autoregressively create one token after the other because they're iterative in this data direction, right? And diffusion models, you can just go down and distill 50 steps down to four, and by that you can speed that up so much, which is on the autoregressive side just not possible because you have to autoregressively generate token after token after token. So these are just two... Wait, you make it sound like it's just worse then. So autoregressive generation is in that respect. It has a limitation. But on the other side, if fusion has in the terms of data efficiency and training, there's also a limitation. So we actually... I think it's... I like both approaches. We've also been working on autoregressive models like long, long, long back and both are super exciting, I would say. So on data efficiency, autoregressive wins out, though? Yeah. Okay. So scale. I guess one of the other big things that you see, having talked to people that worked on both, like, TechDBT ImageNet, is how you can add in the loop URL, like, agentically.

So models can do agentically, you know, generate something, reason on it, edit the image, and continue that workflow. They can do that. But this is separate than pipelining in diffusion generation, right? It's all the one, say, model doing it, where as you train that in on a data level, you can scale that. I guess, is there a counter to this in diffusion? There's no inherent limitation that you couldn't have a model that is diffusion-based to the exact same process that can do all of these aspects. I had the wrong, basically, nano-banana happened, and I was like, holy shit, autoregressive is a thing, but you're saying it's not. Am I wrong? I thought nano-banana was diffusion-based. I think it is diffusion-based. I don't have insider knowledge here, but I'm pretty sure it has the language component to do autoregressive, but I think it's what we'd call a transfusion model, so it can essentially do both. But there's no inherent limitation that says you have to do it one way or another, and it's not like we're...

They're exploration. What I'm getting is that it's not settled consensus, and I thought it was that there was a wave of people shifting from diffusion to autoregression for MS2. I think there's a lot of exciting research on that end right now going on. I will say there's some very impressive work out there. Obviously, we've seen models work extremely well on that front. The same is going on for flow matching is also improving. I would say the debate is not settled yet. There's still significant research on both fronts going on right now. And I think it's important to watch both and try both, be excited about the... It's probably the same is happening for robotics as well. we have we have basically which is super super interesting as well yeah the last other thing I guess is there's also the diffusion for text right so there's also some of that's which there's benefits there's of course speed right we've seen that already like with text we see it clearly right we've got diffusion that people can play will they ever reach parity is it just a matter of investment in money and data or like you know

I would say a lot of it is data if it doesn't work why doesn't it work my simple answer is that we reason left to right. We don't reason and diffuse into vision. At the same time, with auto-aggressive models, I think a lot of people like to focus on the step-by-step of the token, but at the same time, it also can parallelize. We can't just take in a whole book at the same time as well, which those models can. And you can take in latency. I totally get it. That is the valid question. How come diffusion protection never took off? Now it's slightly better, but I think it's just a matter of which approach was earlier and has been becoming for language from the scanner. I think, as your point, you said if it's more data efficient and the world has been doing a lot of scaling, there's a side there. That doesn't mean that there aren't pockets where there's use cases that you can get really good things at a certain size. The example is like, yes, Diffusion Gemma is pretty good and it meets a bar and can run really fast, right?

If you want that, that's great. Yeah, I mean, on the autoregressive language front, I think it really comes down to it was simple, it was scalable, it was there early. It was like a very, I mean, it's clearly been a very successful trajectory. A lot of time and money and research has gone in to make it not work. There's right now a lot of very exciting research on the diffusion front, you know, and I'd like to separate the notion there's, because there's, diffusion language is really two camps. And I think a lot of people tend to not realize this because there are discrete diffusion language models, which I almost don't, I don't like to call them diffusion models really. And this is where like the diffusion Gemma and like the Mercury code, like Mercury models live, which is more of this like masking style model where it jumps from a mask to a token, but it's still in the discrete space. versus now there's a big wave of research right now going on continuous diffusion language models, which those are actually operating as like, I don't want to say flow matching models, but they're operating in this essentially denoising space.

So more like... Is it used for audio, if I'm not mistaken? Audio generation? Continuous diffusion? Continuous diffusion? I mean, it's what we use for... It's for audio modeling, audio modeling. My last one question there is, do we have a good way to do like GRPL style RL on pure diffusion text models? Yeah, we're doing this, actually. Okay. So for Flux 3, we did this, and it has had tremendous gains that it brought us. So Flux Action, this is coming to very, very recent launches and work that you're doing. How do you approach this? What is the feedback and impact so far? Yeah, it's actually how we approach it. Basically, you pre-train on, again, videos, images, audio, everything together, and then you have this very general model that understands the world, and then actually you can take a bit of action prediction data. Basically, you add kind of predictions in the state space of this robot, which is basically a kind of a parameterization of here is this robot arm, which every joint will be described by a translation and a rotation,

six degrees of freedom, basically. And then you predict that together with a video. And since the model can already reason about the video and what it needs to create or to fulfill a certain task in video space, It's actually very simple to then add this new dimensionality or this new dimension, this state space, how we call it, and then basically predict that as well. Given a chunk of earlier video, condition it on that, enter an instruction. Basically, the model creates the next chunk of actions to basically fulfill this action. And this works extremely good. We can do it in the real world, like for real world robots. We can do it in a simulation, like computer use. There's also kind of a mapping of that to the digital world, and we can make the model play games, right, and stuff like that. So this is all enabled by this single flux-free model. And then you see also that it's extremely capable because we have this very general pre-training, which is not only limited to, say, classic robot data, but actually all kinds of data that gives the model just this inherent world understanding that you also see that in performance.

And being a diffusion model is also pretty speedy, as we see here with these distilled versions, basically, right? Would you call this a world model or world action? Do you have a term you like to use? This one is a world action model. I think the term world model I don't like to use. It's still loaded. I'm the same, but it feels like a hominid at this point. Let's call this a visually intelligent model because it can do all these basic things, right? I guess the distinction that I see in these models is you're not just predicting video or world. you're also prediction action states, right? So one is what is happening in the world, what is the state, but also what is the next action to predict. So it's like sort of two outputs, right? Is there a target frame rate that is industry standard? Because you don't have to predict out that far, is my idea. You mean in terms of the real-time factor? It's like 10 FPS, I think, what the model currently runs. And you're projecting out Like 1 FPS or 5 FPS

Sorry, 1 frame, 5 frames I would have to look it up It's a small chunk of frames I would say like in between 9 and 21 frames That it then predicts again And then the model is conditioned on the previous Generations again And when it runs on the robot What the camera actually perceives right And then you basically predict this further So it's a bit of a chunk It's not running autoregressively So to say, in time basically but it's a chart prediction. What I'm driving at is this video prediction side, that is very much for predicting the results of what you do. But probably you need some kind of long-range planner existing outside of that model that will coordinate the actions that... I mean, this is why we're discussing the language side earlier. Yeah, yeah. As it runs right now, what we see is that there's no high-level planner involved or something. This is just a model running on a... On a loop. Yeah. Some of the interesting... Here's what we did with Dyna Robotics. Yeah, Dyna has one, Runway has one. I guess talk about the fine-tuning side.

So if I want to take your general model, fine-tune it for a new robot with a new task, not something super crazy, but, you know, something that's out of distribution, it hasn't seen, say it's my own home robotics, I can program in how to use a humanoid arm or whatever. What is that process like? How long does it take? How important is that in fine-tunability? Also, again, like, since this model is so general and the preaching was so nice, it takes like compared basically you need a bunch of hours I think we showed one example where we compared it with I think it was Polyo 5 which was also open source right it took 8 times less data to basically train on it that was for an industry application it was like 4 hours of data that we needed basically to get to a success rate that was like good enough for this basic application and is that a thing you kind of want to have time on right now or is that like we go back to the context thought we shouldn't be doing fine tuning what we are targeting and what Bill working on right now extremely carefully and with a lot of focus

in context learning for us you should be showing a robot an action one time and then it should be able to do this and this would be truly a super human intelligence like if you look at how long it takes us as children to learn walking, to learn using our thumb correctly to grab this can for instance, right, this is like weeks and even months of learning that it takes us to do this. Our vision is that an autonomous robot should be able to observe any action once and then be able to do it. That is what would be a clearly superhuman kind of visual intelligence and that is what we want to be reaching. What is the standard now? How many demonstrations do we need? It's around like four hours of fine-tuning you need at least. Even four hours is pretty good. I would say 2027 will be the year where we will track the show. Mid-tuning is an interesting one in here too, right? So is this like I want to go from manufacturing to maybe medical stuff, like filling pill capsules, specifically different niches, for example?

Is that where? I mean, the model is trained quite generally. So even the mid-train, like before fine-tuning, it might be able to handle many of the tasks. Like even this version might be able to handle many of the tasks you throw at it. The fine-tuning is really just nailing. In the robotics world, you want a certain level of accuracy you expect out of it. Exactly. This is just a standard large scale mid-training and then we move on to this fine-tuning stage where we add the high-quality robotics data in and then we get this further bump. But you see that already to a somewhat good extent on mid-training the model can actually do these tasks. I guess the other interesting thing is it's a 7B, right? Is that mostly just constraint from these things are deployed? Exactly. You want to be having a model that is as small as possible because you need to be able to act first on device if you want to be adding this to a manufacturing basically and then also in real time. And this is on the backbone of Flux 3? Yes. So it's like the image and audio, all that

stuff? This is derived from this one model basically. We have this multi-model model. That is really important to understand. This has been trained on a huge corpus of images, videos, audio and from that we can derive everything that we basically released this summer, which was Flux 3 Video, Flux 3 Video Edit, Flux 3 Action, Flux 3 Image, Flux 3 Image Client. We also derived an upscaler from it. We have a super nice 4K video upscaler, which is, I think, super fast also. Can I recommend? It's pretty good. And this is all powered by this one model. And this shows how training these very general models actually compounds then over time because you can derive all these applications on them, right? Is there a difference between a video upscaler and an image upscaler? A video upscaler, we talked about the temporal consistency. Obviously, if you would have the same upscaler, you would have no temporal understanding, then you would have this flickering again, especially in the high frequencies. You don't see this too much when you look at two images side by side. You don't see that much, but if you switch between the images,

that's one of the basic tasks that everyone who's looking at a lot of images does, just opening two images in two tabs and then switching left, right, You see every single difference. And that would just amount to kind of flickering in temporal space if you wouldn't have this temporal component in the video offscaler. And that's what makes the difference from an image offscaler. Any notes on real robot evals and how to scale this? Like, I see, you know, sponge in bowl three times. And at some level, like, if you look at our Dynapod, there are literally people just one-off testing robots because you have to test in real scale. but like, come on. Two out of three runs on Spongebob, just restart, do it again. You can be three out of three, right? It's a hard run. I think one thing that's constantly being worked on is just closing the sim-truel gap, and this will get better and better and better over time. But to be honest, EVAs, no matter where you do it, it's a painful thing. It's also painful when you look at the contact creation EVAs. Like, ask three different groups of voters

what a good image looks like, and you will get three different answers. You'll have wars over it. We have more We have more But that's EVAs are just regenerative Modeling EVAs are just hard Yeah Is this already like how developed is the commercial side? We will have more of a story I cannot tell too much but there will be soon EFL VFL powered robots in productions live And is 2027 will be the year of robotics Oh absolutely Okay Say more on predictions Where do we go from this in 2027? How fast do things pick up? Single-shot learning. Yeah. In context, it's a big thing we'll see first. We'll see first commercial applications of these things in real manufacturers. That's why it's actually good that we have this German base because there's a ton of industry data and obviously lots of potential for partnerships. People are really hungry for robotics, I would also say. So there's a good bunch of synergies

that we want to be using to actually make 2027 the year of robotics. I'm very excited about it. The research on this is also moving very quickly, so I suspect that robotics will improve very quickly. What about the side of robotics is very physical manufacturing constraints. You can scale up tech, you can scale up models, but it's pretty hard to affect, right? So predictions, 2027, where are we realistically on robotics? Can I buy something off the shelf that can single-shot learn? I suspect, like, later 2027, like, people will be able to get some quite interesting robotics in their house. I mean, whether it's, you know, is it going to be able to do, like, fine motor tasks and all this? I don't know. But I suspect, like, and, again, this is just entirely predictive. I have no baseline for this. But I suspect next year there might be some interesting, like, you know, at least niche for, like, the tech side of people like us, not like, you know, the average person would be picking this up. but some interesting potential useful robotics in your house.

And for manufacturing, I think 2027 will be a year. Because manufacturing is more restricted, right? It isn't that long tail as, say, autonomous driving and stuff like that, right? Or consumer robotics where you would ideally like to do this thing to do anything you want. But for manufacturing, I think there's a very good chance that we will see a lot of applications actually being unlocked by robotics in 2027. Very much so. And that's what we are also working towards, basically. Very much so. awesome I mean exciting it seems like a very fun place you guys have robot labs image video model you're also you're just missing the flying car you mention Germany a lot but you also have a pretty big SF office I actually didn't know there was a big European there's a substantial SF presence inviting everyone to come over to our place 410 Townsend Street super super nice we're not a European company we're company with roots both in Europe and the US and we're acting globally

basically. That's fine, we got you on because you're visiting in town and we're like, okay, how can people help you? Looking for... I think just supporting us in marketing, talking about our models, helping us. Also here, I think, in the US, getting our name out in the community we're well known. I think for in the broader society, not that much. as the big players, right? That's also what we need to change in 2027, really kind of getting our name out because we have extremely strong technology. I think that's what we saw. We have a long history of having built the core pillars of what everyone is using, basically. It's now also time to get our name out to the kind of broader audience. Yeah. The tech community, I think we're well-known, but apart from that... Tell your mom about Flux. It's very interesting that the overall... Obviously, I'm a marketing guy and I help companies in marketing all the time. You guys are very different because generative media in general has a weird mixed reputation in the broader population, right?

Like normally I would suggest, oh, have a TikTok or Instagram marketing campaign. But pretty much you'll be killed just because they hate AI so much, right? And so it's very interesting to find a niche where this isn't regarded as AI. It's just normal, a thing you can do with tech. But it's not AI. Because I think the moment you say it's AI, the population will just have pitchforks. I don't know if you have any responsive thoughts or explorations about that. I think it's important to just always also emphasize that we take this responsibly. We're like, our models are not creating like avalanches of flop or something, right? They're actually live in real use cases, lots of like professional workflows, and they just help making the lives of like professional people who work with this way easier and just increasing productivity so much. It's Photoshop. Exactly. It's so crazy how much more productive you are then. And that's really the goal of this on the content side.

It's not about putting people out of work, not at all. It's just making everyone's life so much easier. Oh, yeah. I wouldn't say that. I'm just like, if I worked on your team, this is my GTM problem right now, which is how do we get people to basically go viral in the same way that maybe Whisperflow goes a little bit viral because they're like, oh, yeah, I understand how to use this. It's not AI, it's just Whisperflow. It's Granolah, same thing. I think that the general media hasn't had that crossover, except for Nano Banana, and then whenever Chachi Biti had the Studio Ghibli moment. Those kinds of, like, that is the crossover, where it's like, oh, this is a fun trend that I want to hop on. There were moments that hit well, like your point of the see yourself as you're older, like that had a huge moment. People would look at themselves as they're 80, send it to their family. There have been moments where nothing is super... I think one of the difficult pieces is actually very hard to predict one of these viral moments. Like, Nano Banana was their internal name, and then they just kept it because people just liked the name and it stuck.

For context, actually, the one big use case that we saw a ton of usage on that we didn't necessarily expect was image restoration as well. So there was a ton of, like, I saw amazing, amazing examples of people, you know, taking old, degraded family photos and then restoring them, colorizing them, fixing them up, and it was able to keep, like, a very, very high level of similarity to what the original would have been. And then, I don't know, that was actually one of my favorite moments that we've had, you know, people showing, like, their family these kind of restored photos and the reactions from it. And I really, really enjoyed that, like, positive moment that it brought. But like I said, so often we can't predict what the thing is going to be. Even with our old, even back in the stable diffusion days, it was always surprising what the trend was online. They just kind of emerge. You kind of plan them. I think once you have them, you need to kind of exploit them, obviously. And you should be always asking, what is the next big thing,

have next capability that has not been able, or like that models have not been able to do, what would that be? And then trying to bake this into your model. Making bets, some of them will work, some others will not work, but this is just extremely important to kind of always have this mindset of, hey, what is the next thing we should be working on that will just bring a lot of joy or productivity increases to people, right? Well, that is the domain of research. You guys are doing an amazing job, So thank you for sharing your story. I hope more people come to know you as a team because they've seen a lot of the model buds. But I feel like this is also the first time I've seen, I've heard the full story of you now. Yeah, thank you. Thank you so much for having us. Thank you so much. I mean, this was a, it's fun to finally have gotten to do this. Yeah, yeah. I mean, it's a year in the making. But also, you know, I bought a bunch of micro ducks and, you know, I'm looking forward to them running around. Do you have, like, I want to get my hands off. I feel like I missed the window of these being available.

You have connections. I'm sure I can. Whoever out there can get me a mic or a duck, give me a mic or a duck. I'll get you a mic or a duck. Okay, perfect. I bought a bunch because I know it's going to be a hot gift for Christmas. Yeah, yeah, yeah. It's like... But, yeah, you know, Robotics, 3027, that's, you know, one year away. Like, you know, we'll see where this all learns. But I'm very optimistic about this approach, and I think you guys have explained it very well. Fantastic. Yeah. Amazing. Thank you. Cool. Thanks, man. Thank you. Thank you.

番組の概要欄(原文)

What can a model learn about the world by watching video and listening to it? Black Forest Labs trained FLUX 3 on images, video and audio together. Andreas Blattmann and Dustin Podell join swyx to explain why. A moving camera reveals the far side of an...

出典・この記事について

番組
Latent Space: The AI Engineer Podcast(Latent.Space)
元の回
Why FLUX 3 Trains on Images, Video and Sound — Andreas Blattmann & Dustin Podell, Black Forest Labs
配信日
要約作成日
公式サイト
www.latent.space/podcast

この要約・解説記事は、元の音声を文字起こしした内容をもとに AI(Claude)が日本語で作成したものです。 固有名詞や数値に誤りを含む場合があるため、正確な内容は元の音声でご確認ください。制作方法について

X でシェアSpotify で聴くApple Podcasts で聴く

関連エピソード