NVIDIAのフィジカルAI：ワールドモデルとロボット開発の最前線

In recent years, the main impact of AI has been on the digital world, and most notably software engineering. However, its longer-term effect on the physical world may prove even more profound, reshaping how cars drive themselves, how factories run, and how robots move through homes and warehouses. World models are at the center of the movement to bring AI into the physical space. These models take in observations of the current state of the world and predict what comes next, which lets a robot plan its next move or car anticipate the road ahead. NVIDIA has been building a family of these models called Cosmos and sits at the frontier of physical AI and robotics. Amit Goel has been at NVIDIA for 15 years, and he is currently the head of robotics ecosystem and edge AI product. In this episode, Amit joins Matt Merrill to discuss what separates physical AI from digital AI, how world models plan and predict, the two-tower architecture that lets a robot reason and act at once,

why training on video demanded a new data pipeline, and how any developer can start building in this space. Matt Merrill is a software engineering leader with over 20 years of experience building and scaling software teams across enterprise and product-focused organizations. His background is in back-end development, cloud architecture, and distributed systems design. He currently architects and delivers software products and leads a team of engineers at Dept Agency. You can learn more about his work at code.theothermattm.com. Hello, everybody. My name is Matt Merrill, and I am here today with Amit Goel from NVIDIA, who is head of robotics and edge AI. And I'm really excited to be here to talk to you about physical AI. So why don't I turn it over to you, Amit, to introduce yourself?

Hey, Matt. Great to be here. I'm Amit Goel, head of our robotics ecosystem at NVIDIA, which means my team gets to work with all the developers who are doing some amazing work with AI and robotics. And I also lead our product team that builds the computers for running the AI on the device on the edge. It's called the Jetson family of products. I've been at NVIDIA for 15 years. We're excited to talk about physical AI today. All right. So I am super excited to talk about this because I think that a lot of folks listening here are probably at this point pretty familiar with the software side of Gen AI and LLMs. And what I kind of want to do today is just start from zero, right? Like, and that's actually, I'm not playing a role here. I purposely didn't dive into this deep so that I could kind of be fresh. So when NVIDIA or anybody says physical AI, what do you mean? What makes it different from LLMs and coding agents that most of our listeners are using every day?

Yeah, the big difference between LLMs or internal digital AI and physical AI is that you're working with atoms. And what that means is you are dealing with sensors. You're looking at spatial information, 3D information. You're looking at videos. You're understanding dynamics of the word. So physical AI essentially is a model that can take as inputs the information of the world around you and predict what's going to happen in that world. So everybody is probably familiar right now with self-driving cars. If you live in San Francisco Bay Area, you will see the Waymos driving around. So that's probably the closest way to think about physical AI in the real world, right? You have a machine that is taking the information from the cameras, the LIDARs, and the actuators in the car,

and making a decision on what it should do next based on what the driver wants out of the car, right? So it's taking the data from the observations from the physical world and determining actions that have to be executed in the physical world. And it's been around for a while. The simplest form is probably your nest thermostat that's controlling the temperature of things in your house based on your activity. And even like a small traffic intersection that's observing what's the activity that's happening and controlling that particular traffic light. So physical AI has been around for a while, but it's been very narrowly defined for one particular use case. so now what's happening with the large really large models and general models is that you can have a single model that can understand all of this and similar to what happens with LLM right you don't have a sentiment analysis model you don't have a translation model you don't have a summarization model you have one big

model that understands everything in the textual world and you can program it to do what you want it to do we are getting to the point where we can build this general physical intelligence models that you can use for a lot of applications in the physical world. Got it. Yeah, that was one of my big questions. I never really thought about that. Like a self-driving car or even a Nest thermostat is technically physical AI. But when I was prepping for this, seeing some of the stuff that is currently happening is so much more advanced than that. And it makes sense that in the past year, the explosion and what these models can do enables a lot of this. So let's dive in and try to give folks like a mental model of how these things break down. And so one thing that I was reading about prepping for this was a concept of a world model. Can you tell folks about that and like how that fits in here and how it might drive one of these devices? Yeah. So the way to think about the world model is in the context of what are the inputs and what are the outputs, right?

So for a world model, the inputs are observations of the current state of the world. It could be a sensor coming data from a motor of a car, of a robot. It could be a signal that is audio command that has been given as an input. It could be a textual prompt. And, of course, it has to have the cameras and the visual representation of the world that's coming into it. The next question is, what does the world model predict, right? And inherently, the world model, because it's dealing with the physics of the world, is a physical AI model. It is multimodal because the real world has multiple modalities, right? We are hearing, we are listening, we are processing information. So it's multimodal, many inputs and also many outputs. The typical output of a world model is a video, which given a prompt, given a current state, if the world model understands what happens in the world,

it can predict based on what you told me, this is what's likely to happen. In the case of a car, you can say, I'm going at this particular speed. This is what I see is happening around me. Tell me what is likely going to be the state of the world in the next five seconds. Based on that, I can then control the car to what action it should take. Same analogy applies to a robot. right? You asked me to pick this thing and put it over there. This is what I see is my current state of everything around me. This is the object that I see. How should I move? Should I go left? Should I go right? Should I go backwards? Do I need to bend? It can predict what is the next few things it needs to do. So the world models can predict the world around you. It can do a 3D representation of the world, it can also understand, hey, I want to be there. This is my objective. How do I get there? What we think about in our head, it's called planning, right? So the world models can predict the plan to go from the current state to the desired state. So this is essentially

the planning function of the world model. And the other thing is you can say, hey, this is my current state, I'm planning to take this action. Can you tell me what is going to happen if I do this? And that is super important for robotics, right? Like if you can say, I'm going to do this and tell me what's going to happen, you can actually decide whether that's the right thing to do or not. So rendering the entire world around you, given an action, predicting what the world is going to look like after you take that action, or given the current action and the desired state, predicting the action I should take to get to that desired state. These are all of the outputs of a world model today. That sounds like it could be, especially for something like a self-driving car, an absolutely enormous model or set of models. Yeah. It just seems like a lot, like a video, potentially audio. I don't know. Just a lot. I mean, I'm almost having a hard time wrapping my mind around how much stuff is going into that. No, you're right. I mean, the amount of data that gets fed into a world model is orders of magnitude more than what you would feed into a text LLM, right?

Because just a video of 15 minutes is several gigabytes of data. So, of course, the amount of data that gets processed in a world model is a lot. But at the end of the day, I think the size of the model we see varies depending on what you want the model to do. You were asking about physical AI and difference between LLMs. And one of the things is that you have to run some of these models on the device itself, on that particular robot or on that particular car. You do not have the luxury to say, it's okay if it takes one minute to get back to it with an answer. I'm going to throw all the compute available at it. So you need to think about how you're going to make distilled version of these board models that can run on device. but for something like evaluation, which can happen offline, right? If you're training a model, you collected some amount of data, but you want to train what it is going to do if it was in a particular street of New York.

You can just ask the world model to give it a particular image and say, can you create a world around me and I want to drive this thing in New York, right? That part is super heavy, but that part can also run offline. You can throw a lot of compute at it. So at NVIDIA, we have been working on the Cosmos models, and a variety of those exist. And we have many different sizes of the models, starting from things that can run onto the edge all the way to really, we have nanos and super models. They can be used for creating data. They can be used for evaluating things offline. So you're 100% right. These models are processing a lot of data. You anticipated my next thought comment, which was just the speed. These things can't wait. It's not like me waiting for Fable to analyze my next presentation or something. Yeah, okay. You cannot just leave it around and come back and see what happens. Especially if you're deploying it on the device itself. Yeah, that's amazing. So I'd love to walk through, you know, you get these interview questions of,

I remember back in the day, it used to be like, what happens when you press enter on Google, right? So I kind of want to give you this for one of these situations. So we can pick a type of robot or whatever. What happens in 100 milliseconds between that thing capturing, let's say, an image or some sort of multiple sensor inputs, and then a motor command going out of that thing to perform an action? Yeah. So typically, if you think about, let's say, a robot, the moment you are capturing the information from the robot's camera, typically you need more information than just the camera. You need to also understand what is the state of each of its actuators. You need to understand some of the other sensors like IMU, which is an inertial measurement unit that tells you the pose of the different parts of the robot. And once you capture the information, first you need to synchronize all of this. What that means is my camera is observing at this particular time.

At the same time, what was the state of the motors? What was the state of the IMUs? and you need to fuse them together. The next thing you have to do is, if you're given a task, let's assume we're running a vision language action model. You need to tokenize this data, all of this data, the data from the current frame, but also the data from the past frames because your next action is dependent on not just what you're currently seeing, but also a little bit of history. So you have to take the data, you have to tokenize it, and then you feed it into this model, which is now going to do two things. First thing it has to do is it has to do a planning, which is the task of taking my current observation, my past observations, and my objective function. What should I do, right? If I was asked to pick up that particular cup, do I need to move left? Do I need to move right? Do I need to go up or down? How far do I need to reach?

So this is the planning task, and today this is something that is done by an autoregressive model that will predict the sequence of tasks that you need to do based on the objective function and current input. Now that's your reasoning and planning brain that kicks in. And the planner tells you, okay, based on the task that's given to you, this is the next step you need to do. Now comes in the second piece of the model, which is generating all the actions. And that part is very complex because depending on the robot type you could be controlling 30 to 40 motors and you need to generate actions of all of that synchronously You cannot see really generate that right You need to control the output. And one thing that's really unique about physical world is that a lot of things are continuous. You have to control the current, and you cannot send it for like this much current. It has to be continuously generated. So the kind of models that you need is different than what you need to predict a discrete output.

A sequence of text versus the current for 50 motors that you have to send it at every millisecond. So the frequency at which the output must be generated is very different for a robotic and a physical AI model. So you have to make these two models work together. A reasoning model that is planning can exist at a low frequency. The inputs of that goes to another model that actually generates the motor control commands. And that one is running really, really fast because it has to control all that at high frequency. And once you have that output, typically that's happening in the brain. But typically at the end of that, it goes to a small computer that is attached to the motors, which does your PID control based on the input that it got from the model. I'm not familiar with PID. What's that? So PID is the closed loop control, that low frequency control loop. That is, you have a target position, you get the current position,

and you control the current. So that is running right onto the motors. It does not need to know what's happening on the other 30 motors, right, because its objective is to control that one motor. And that runs on a very small computer or microcontroller. I'm thinking of this in terms of like a BIOS or something like that. Yeah. Yeah, it's a very simple firmware that's running on Twitter. It's just taking commands. I'm listening to this and I just can't help but think like this is the world's most insane state machine ever. That's just so much to keep track of. Yeah. And so you talked about this in a way that almost seems like there's two different models happening. There's a reasoning and a planning, but yet they're all sharing the same context, right? That's right. How does that work? How does it share that? Yeah, so that's the interesting part. There are many ways these models are being architected. And one of the ways that we are looking at this is having a shared latent space

where the attention from one model and there's cross-attention between one model and the other model, the embeddings are shared so that you have a really high bandwidth communication between these two brains. If you look at the latest Cosmos, three family of models that NVIDIA released, we have these two heads, right? A reasoning head, and on top of that is the, we have the reasoning tower and the action tower on the output side, and they talk to each other. They have shared context in which they exchange information in the Latin space. If you were running and distributed, you may want to take the output of one and decode it and send it to the other one. and again, Encore did, but that slows down the bandwidth, right? As I was telling you, for a robotic system, they need to operate really in a closed loop. So that's why we build Cosmos with that shared context so that both these can coexist and exchange information with very high bandwidth. I'm almost picturing this just like, I don't know,

I'm trying to think of like a CPU analogy, but it's logical. It's almost like a logical division inside the same hardware. Is that correct or do I have that wrong? You're absolutely right. Think about like in a physical space, we have a shared memory, right, where you have two different processes. You don't want to send the data over because it's basically bandwidth. The same thing we're doing at the model level now, right, where in the latent space, we can share the information and the context between these two models. So they can operate, although they're operating in different frequencies, they can share the same context. Got it. That's really interesting. I was going to ask about that two tower design because that was something I came across. It didn't make sense to me, but as you explained that reasoning step, that does make more sense. Yeah, and another way to think of it is an architect, which is planning everything that you need to build. And then there's an executioner, which is, given the plan, is doing the execution. They both need access to the same information, but what they do with that information is slightly different.

This episode of Software Engineering Daily is brought to you by VariaCode. If you're an engineering or IT leader, you know how painful this is. An open rec sits unfilled for three months. Your tech lead is burning out, covering the gaps, and your product roadmap gets pushed back again. VariaCode solves that. For over 10 years, they've been embedding senior Latin American engineers directly into U.S. engineering teams in under 10 days at up to 40% less than local hiring. We're not talking about generic contractor pools. These are pre-vetted, high-quality engineers with real on-call and SLA experience. They work directly in your Slack, JIRA, and daily stand-ups in your exact U.S. time zone with zero friction. If you're ready to stop losing sprints to open recs, head over to variacode.com slash S-E-D right now and get a custom shortlist of senior candidates in 48 hours. That's V-A-R-I-A-C-O-D-E dot com slash S-E-D. Every visitor on your platform looks the same at first.

Can you trust them or are they a threat? Fingerprint lets you answer this question right away. With one API call, no model to maintain, and implementation in minutes, Fingerprint recognizes bots, AI agents, VPNs, tampering, and over 100 more unique signals to give you durable risk scoring and threat detection in real time. Even as human and agentic activity blends together, you can identify suspicious traffic faster and build safer and more secure sites and apps. Over 6,000 companies rely on it every day. Join them now at Fingerprint.com. Think about your mobile app source code. Once it hits the app store, it's out in the wild. And without the right protection, decompiling is easy for malicious actors looking to steal your IP or tamper with your software. That's where GuardSquare comes in. GuardSquare provides the highest level of mobile app security for Android and iOS applications and SDKs. Their advanced tools integrate seamlessly into your CICD pipeline.

We're talking polymorphic, multi-layered code hardening techniques and automated runtime application self-protection, paired with mobile application security testing and real-time threat monitoring, to deliver the highest level of mobile app security without compromise. Don't leave your hard work exposed. Secure your mobile applications today. Go to guardsquare.com to learn more. so one of the things here that's like i mean we talk in textual models about using gpus because they're really good at this task but i can't help but think like this requires just so much more horsepower so to speak than what those need so like if you can simplify it like what is the difference in the hardware that's powering these things there are a few things that is different And as I mentioned, when you're thinking about the world model, the number of modalities that you're dealing with is quite different. What that does to your architecture is when you're training a model, when you're dealing with the text inputs, the bandwidth that you're processing for data itself is not that high.

When you are working with the world models and you're training a world model, you need to retrieve the images. You need to retrieve the actions or the motor control commands. You need to retrieve the text annotation for all of that in parallel and bring it into your model. That creates pretty unique challenges from the hardware perspective as to what do you keep in the storage? How do you manage the data? we in fact had to go ahead and build out a completely different stack in order to process this data. Because GPUs resources are super scarce and precious. And a lot of this pre-processing typically was done on a CPU. And so you would have your model waiting for the data to do the training, but it's just not possible to do that with the existing pipelines that were built for text-based model training. So we created something called Curator,

which was the ability to now take large amounts of videos, process it, store it in a way that now you can keep the training data coming in. So that's on the data side of things. On the hardware side of things, when it comes to training these models, as you know with LLMs also, there are multiple phases, right? There's pre-training, there's mid-training, and then there's post-training. You see a lot of post-training happens with using the reinforcement learning. You're using different tools. The same is true for world models. But when you're training a world model, let's say it's a robotics world model, how do you verify it? How do you do the reinforcement learning? For a coding model, reinforcement learning is you go, run it, compile the code, verify it, works or not. For a world model, its job is to simulate the word. you actually need to go to a physics simulator as a tool, right? And that physics simulator is best run on another kind of GPU,

which has the ray tracing cores that can do the sensor simulation. So it becomes a heterogeneous computer architecture. On the one side, you need something that can process all of this data. Then, of course, you have your transformer model that needs to get trained. And then after that, you have to continuously call a simulator where you go and do the reinforcement learning and testing whether if it's working or not. So it becomes a pretty complex architecture. And inherently, it has become the heterogeneous computing problem, right? That's one other thing that developers who are used to LLMs get access to a whole bunch of computers, all, you know, latest Blackwell clusters. They can go and just train the models, test the models, evaluate it, serve it all on that cluster. but physical AI requires actually three computers. You need a computer for training the model. You need a computer where you do simulation either to create data or to do reinforcement learning post-training.

And then last piece of it, deployment. You cannot deploy it on the same infrastructure that you trained it on. You need to deploy it on a small computer that's sitting inside the brain of a robot. So you need to go do some more work on the model quantization, distillation, compression to go deploy it. So it requires a very different way to think about the compute hardware that you need if you're building a physically high model. I'm joking here, but I'm imagining you being at like a cocktail party talking with a software developer about textual LLMs and going, that's cute. Yeah. Just kidding. Yeah, I think, you know, Jim Phan, who leads our research team for a physically high and humanoid robotics, He often jokes like, I'm always envious of my friends in the LLM world. Their problem is I don't have more GPUs. And my problem is like, okay, first of all, the data that I got, like half of the sensor data is missing. The synchronization is not there. And then I'd have to, I get the GPUs, I train it,

but next I have to find a simulation environment to go test it, deploy it onto the real robot, the cables are breaking. So it's a very different world. Yeah. That is something. The simulation piece of it never even, I mean, yeah, the horsepower even just to do that simulation is mind boggling. Yeah. So switching topics a little bit, like when these things are deployed, they need to react to somewhat unpredictable things, right? Like they effectively need to learn. And I read a little bit about imitation learning and watching humans do tasks. And can you say a little bit more about how that happens, at least in current research and in the real world, and how that operates? Yeah. So two things that's being done right now to make these models more generalizable. First is to collect data in diverse environments, right? So this is if you are supposed to assemble, let's pack a bag of shoes, right?

Often the size of the bag of the shoes is going to be different. The size of the box is going to be different. So one way to do this is you have a lot of people collecting data in different environments, in different factories, bringing the data of doing the tasks in different environments and training the model so that it now has seen enough distribution so that the real world falls somewhere within that distribution, right? So that's why when you see imitation learning, that's what is happening. We are collecting the data, increasing the distribution in which it's been collected. But as it turns out, however much data you collect in the real world, the real world has an infinite distribution. It's continuous. So you can collect it in 10 different lights, but there's going to be something that's always going to be out of distribution. that's where the second part comes in where you need to think about having a reasoning model

paired with a model that has seen enough distribution they go together right You cannot just have a reasoning model but not have enough real world data because the reasoning model can tell you this is closest to this particular real world distribution And then you can go act on that. Right. So the reasoning model is learning from data in the wild. It's using all the text data that's available. It's using the video data that's collected across different domains. Right. and learning from that. So combining this pre-training with data in the wild, textual, video data, along with real data, action data, which is people doing the task, robots doing the task, combining these two, the world models are now trying to solve this problem of how do I know enough that I'm physically capable of doing the task? I know how to pick up an object. I know how to slide something. I know how to bend over and perform a particular task.

And how do I do it in a different environment and in a different context, right? So both of these things are being trained together. The first one, how do I do a certain particular task, is being done a combination of either imitation learning and reinforcement learning. And then how do I do this in any environment? This is where the data and the wire is playing a role. Okay. So the reasoning models, are these various tasks specific? The reasoning model, this is where the generalization is coming from, right? So you're trying to collect diversity of information. Your robot might be designed to work in a factory, but you're putting in all the data from homes, from outdoors, from indoors, because that's when you build a reasoning capability that can generalize and you can draw parallels. I see. Yeah, that makes sense to me that you need to give it tons of different inputs to be able to react to different things. So, okay. Yeah, because if you were just looking at factory, you would probably not see a cat or a cow come in.

And so, you know, when you're doing the reasoning model, you expose it to all elements. This is a little random digression. I was on Reddit the other day, and somebody posted. I live near Boston, so it was on the Boston subreddit. Somebody posted a picture of this guy with his wacky-looking backpack on. It was like a crazy backpack. And people were making jokes like, oh, he's the rocketeer or whatever. and somebody was like, no, I think that they're gathering world data for AI models. I see you nodding your head that that's probably what that guy was doing. Yeah, one of the things that is happening is simulation is important, right, as I said, for physical AI because that's where you can safely test your models. That is where you can do reinforcement learning faster than real-time speeds. but the challenge with simulation models is the cost of creating that simulated world is quite high. So what you would probably like to saw was somebody, you know,

carrying a backpack with some cameras and LIDARs and what they are doing is capturing the real world and passing it into a model that can create a 3D representation of that world. What we like to call real to sim, right? You take the real data, run it through a model, and create a simulated version of it. Now, once you have the simulated version of it, you can drive in that simulated version and test how your car is going to behave. You can use that and do augmentations, right? What if this building's colors were different? What if the lighting was different? What if the road was narrower? So you are closer to the actual distribution. and this data is often used for post-training. This is where, you know, I want my car to operate really well in this particular block. Okay, I'm going to collect the data and let my general model post-training in this world. Now it becomes really good at that environment.

That's really interesting. That's actually the opposite of what I might have thought. So thank you for clarifying. Cool. One thing that just came to mind here is, you know, like hallucinations in text models or, you know, you just kind of chuckle at them. But hallucinations in a physical space sound a little scary. I would imagine that there's better guardrails for things like that, and I'm curious how, you know, you handle that in the physical AI space. So one of the things that is important for physical AI is, as you mentioned, that if you're out of distribution, if you're hallucinating, the impact is pretty significant. So you have to do two things for that. Either you can make your model really conservative and says, okay, this is out of distribution. I see something that I've not seen before, or this is not part of my learning curriculum. I'm not going to act on it. The other thing you can do is, well, I'm going to act on it.

And in fact, the way I'm going to make sure that I know how to handle these out-of-distribution scenarios is I'm going to actually give those and stimulate in that environment. I'm going to evaluate my policies or my models in a lot of different environments, which some of them may not even be real, and then score them on their performance. So one of the ways to make sure that your robot policies are not hallucinating or if they are out-of-distribution, they're still safe is to expose them to those adverse environments in simulation and see how they perform. That's one of the ways to kind of make sure your policies and the models that you're training are within the guardrails. The second thing also is, you remember we talked about a planning brain and an action brain, right? The planning brain is sort of like a VLM or an LLM where you can do the guardrails into that so that it does not plan something that is not acceptable.

If it does not plan, then the action head cannot, you know, it is operating on the plans that are coming out of it. How do you put the guardrails around your planning? It's very similar to how do you put guardrails around an LLM to, you know, not give you an answer or information that is not suitable, right? So you can constrain things at the planning stage. On the action stage, again, you know, it is mostly learning in domain. You can go to simulation and test it. And the third thing that we are also thinking about is when it comes to safety and actions is you have to have some level of redundancy, some level of external element that is observing and verifying. So because that kind of separates the concerns, right? The person that is acting and the person that is monitoring, if there's another AI that can observe what is happening and then, you know, have a way to control the robot or the car, that's another way to manage safety, right?

So I think it's a combination of all of these things that are coming together to make sure that as we are building these models, as they go into the real world, we can make sure that, you know, we're not being extremely conservative and not being able to do half of the tasks. We don't want that, right? But we want them to be able to do that safely. So we are bringing this outside-in element of safety where you're monitoring, you're looking at it, and making a decision. You make sure that your reasoning brain, planning brain has the guardrails, and then extensively test in simulation. That's one thing I wanted to dig in. When you say it evaluates those plans, it makes sense in training that it does that. But I want to clarify, is it also doing this in the real world too? Like if it is unsure about its action, is it actually going to run simulations in real time to evaluate which one may be the best? That's the biggest value of the world models is they can do counterfactuals.

They can simulate. If you look at the work that we published recently with our self-driving car team called Alpamayo, it's a reasoning model. And within that, it can reason, it can predict 10 different futures and choose one of them. Because it is a world model. it is not limited to only the right thing it's not just imitation learning this is the right way to do it it can explore 10 different directions 10 different outcomes score them and act on the safest and the one that gets you the most to your objective function so that's the beauty of the world models it is doing that at evaluation time and it's important to have these models open for that reason as well Because what is acceptable in one part of the world versus what's acceptable in another part of the world. Thinking about driving rules, right? What is an acceptable driving behavior in Boston is likely different from the acceptable driving behavior in the Bay Area.

And if you go to a different country, of course, it changes. So taking these word models that can predict all of the different counterfactual scenarios and then post-training them to choose, you know, what is acceptable. It's important. When I heard you talk about the security, I can't help but think like this is security in layers, security in the planning stage, security in the execution stage, evaluating options. So that's really good. And that is not something that, well, I don't know. I was about to say that's not something you hear too much about in the LLMs, but that's probably not true. We're starting to do things like that as well. That's right. So you mentioned the open models, which is really exciting and also a little bit scary because, you know, one of the things where, you know, my catastrophizing brain goes is like, what happens if this tech gets into the wrong hands? Right. Like and I know you're not a soothsayer. We all at least, you know, our listeners have the best of intentions. But like, is there anything being done in the research community to help get it, get ahead of this in terms of ethics, legalities and just, you know, human safety in general?

Yeah, I think fundamentally to think about it, if these models are open, right, as long as we have more people who have access to them with the right intention, they will outnumber the people who will have access to this with a bad intention. Hopefully that's the word we live in. So in that case, I think being open and accessible actually gives us more tools for everybody to build systems that can monitor them. As I said, you know, the thing with safety is you have to assume that what if it happens? How am I going to counteract? I think we saw what happened a couple of weeks ago with the agent getting attacking the hugging face. and they were able to use an open model to actually go and triage that and fix it. So I think that's why we believe that having a pretty strong, open ecosystem is vital to make sure that there are checks and balances.

It's vital for people to create models that are better aligned to their philosophy. and it's especially true for like physical word. I was off kidding with somebody that me and my wife can't agree on how we load a dishwasher right like that is the right way so we need an open model where you know I can she can reinforce train it to do it the way she likes it or I can do it so if it is a you know somebody decides for us how the dishwasher is going to be loaded there's no way that's going to work. I mean do you have a dishwasher loading robot like is that are you doing that at home like because i'm totally jealous like is that okay not yet but i'm looking forward to it i think it'll happen soon give me a prototype yeah that's really interesting and tell me if i'm completely off here but like i almost think of this in terms of like human morals right like you the deeper the moral is set in someone the harder

it is to compromise it and i think what you're saying is that these models are open and they are deeply ingrained in ethical things. It's very hard to unprogram that. Is that kind of what you're saying? I think if these models are open, you can introspect them and you can steer them. If they're not, then you can put monitors, you can put all kinds of things, but you cannot, you know, you're limited in what you can do. And then you also introduce a single point of failure as well. Yeah. Right. So I think diversity and choice and, you know, one model checking the other one is the best way for making sure that you have a better answer to the security problem. I can see what you mean by the transparency being huge, because, you know, if there's a bad guy, you don't know what he's thinking. But if you can see what he's thinking, that's a little bit of a different story. It's really interesting. I want to talk a little bit about, so you released Cosmos 3 Edge.

You mentioned this Cosmos model. And for somebody who's not familiar with this, right, like I don't really know what that, I think you said it's a model, but I don't really know what that means, Edge. So can you talk a little bit about what that is? As I was telling you earlier, right, the world models can do many different functions. They can create, predict random worlds in which you can, you know, evaluate your policies, They can take real-world data and augment it to change the environments. But some of these things are pretty compute-intensive, right? But one of the functions of a world model is also what we were talking about earlier is that they can be the planning brain for your robot because they can predict a lot of different scenarios and pick one that is most appropriate that gets you to their objective function. That model needs to run on the robot, right? That part of the world model. This model is not really generating, you know, pretty videos, 10-second long videos.

It doesn't have to do that. This model does not have to, you know, you can distill it down to certain particular domains that you care about, but it should be able to, you know, do the task of predicting the future and controlling the robot. So we took our Cosmos 3 model, which comes in different sizes, and we created a 4 billion parameter version of it. And the main purpose of the Cosmos 3 Edge is that it's a world model that can take the inputs as text or actions or videos and generate, you know, the next action that the robot needs to do. Right. And by making it a four billion parameter model, we can now deploy it onto the robot. So it can be used as the policy of the robot itself. And, you know, we optimize it. It can run real-time on a Jetson Tor, which is, you know, a computer which is the size of a passport that has 2,000 tera-ops of compute. Did you say the size of a passport?

Yeah, it's the size of a passport. It's your passport. Wow. Yeah. That's what is, you know, inside a lot of robots. If you look at the 1X robot, the Boston Dynamics humanoid, they are all powered by this small computer, but it has 2,000 tera-ops of compute. It can run 10 billion, 4 billion parameter model on it. So that way is, you know, it can do the reasoning. You were asking about counterfactuals. It can do all of that on-device. That is wild. That is really cool. So I can see why that's a big deal. It's a mini model that can run on robots. On the robot. It's so funny because when I think Edge, I think of, you know, like, oh, CDN distributed geographically. But when you guys talk about Edge, it's like, nah, it's in a factory. or it's, yeah. Yeah, yeah. It's just battery powered, you know, running on device, running all the time. Yeah. Yeah. And the power consumption of those things has to be reasonable too. Not only is it small, but the power consumption. Wow. Yeah. That's crazy.

Okay, so I'm a normal developer. I want to start playing around with this next weekend. How do I get my hands on one of these passport-sized computers? They're only like 20 bucks, right? I'm just kidding. How accessible is this? Like, I would love to play with something like this. Like, how does a normal dev get into this? Yeah, I think it has become much easier in the last, I would say, 12 to 18 months. You know, in the past, the hardware was very expensive. It was because it was designed, like, for roboticists, right? And one of the things we started working with the team at Hacking Face was to target and enable all the software developers who are used to, you know, writing programs to start programming robots. And to do that, you need to have a robot, right? You have to have, you're building physical AI. Of course, you can do a lot in simulation, right? That's the first step. You want to get started with this, you know, you use your existing computer,

you download a simulator, and you download an existing model, connect it together and see how it works, right? And this is all just running on your laptop or computer or whatever. You can run on a decent laptop today, which has a 4090 GPU. You can get a DGX Spark, run it on that. So you can run the simulator. You can train the models, test it. But the interesting thing about physical AI is that experience of touching a real thing and dealing with the connection and the motors and the variations there. So now, for a couple of hundred dollars, You can get a robotic arm that you can train, you can teleoperate it, collect data, format it, publish it to Hugging Face, train a policy, convert it, deploy it onto the small Jetson, and let the robot do its thing. And all of that, I would say, for you to get started, if you have a good computer, $1,000 budget, take a couple of days, and you can have your first physical AI model doing things.

Wow. What are the arms? Is that something NVIDIA produces, or is that something that other third parties produce? It's just third party. There are lots of projects. If you're into 3D printing, you can just order the material, 3D print these things. get it together. That's really cool. Pretty big open source ecosystem. If you look at SO101 ARM, that's the most popular with the makers right now. You can 3D print it or lots of vendors do the kitting of all together and they ship it to you, but you still get the joy of connecting it and building it all together. So that's there. And then you can start from there and go to a humanoid and get one of these humanoids for $15,000, $10,000 these days. that you can make it do whatever you want, dance or, you know, or team your home. Wow. When somebody's, like, looking into the, like, those, I don't know what we call them, like, classes of product, like, what would I Google for or what would I ask Claude for in terms of trying to find that? That's something that, to me, if I were to step away, I'd be like, okay,

I guess I could search, like, open source robotic arm, but I feel like there's probably something more specific that people could search for. Yeah, I think if you're interested in using this new age of robotics, which is end-to-end models, right, that is taking pixels in, controls out, you want to look for models that are supported by, they support NVIDIAs like CRUDE or Physical Intelligence or these popular AI models, right? So those robot hardware that has been tested with these models. The way to search for it is look for different robot policies, AI models that exist, and say, I'm looking for a hardware that works with this particular model, right? We don't want to have to customize the model for a different hardware, right? Yeah, that's a bridge too far for somebody trying to get into it. Exactly. That's why the hugging face checkpoints are really good

because they already have checkpoints for this SO101 arm. If you're looking at action-changing transformer model, you are looking for a Groot model, you're looking for any of the popular AI models for robotics, you can find a checkpoint for specific hardwares, and that's one way to go find it because then you can just cobble it together and have your first rollout. And then you can say, okay, I want to go customize it. Yeah. Oh, that sounds fun. Okay. Yeah. Cool. I hope that inspires some people. All right, let's close it out. I got a couple of open-ended what if questions. So what sounds like science fiction right now that you actually think might come true in five years with this? If you think about robots today, what comes to your mind? Probably that Boston Dynamics thing that can like, you know, jump upstairs and jump up on platforms and things like that. Yeah. Yeah. I think for most people, when I think about robots, It's the embodiment that comes to mind, right?

As you said, that humanoid or the big arm that they saw in an automotive plant that was carrying the entire car and turning it upside down. But I think in five years, we will see definitely robots that are more around us, more ubiquitous, that are multifunctional and can do many things. Robots today are purpose-built. You think about robot, you think about either that humanoid or you think about a Roomba or you think about, you know, that industrial robot. You don't think about a brain that can do 10 things for you. You think about the body, not the brain. I think in five years from now, I expect that we will have robotic embodiments that are going to be, you know, capable of doing multiple things. So when you think about robot, you will not think about the body, but you will think about, oh, that thing that can do all of these tasks for me.

The embodiment is just, you know, an artifact. So that would change. You would not think about a robot for A, but you would think about a robot that can do a class of things for you. The other thing that's going to change is for the robotics people is the task of programming was very intimidating. It's like you need to be an expert. You need to be either, you know, a technician or a PhD to be able to program the robots. That is going to look very different. You know, with these foundation models, you would be able to likely show a task. Do it one time. Do it with the robot two times. And teaching the robot with data is what is going to be magical. It seems impossible if you tell a robotics person today that, hey, you know, programming a robot is just like, go show it. hey, this is the task I want you to do. Here is a video. Can you go do it? Right? That seems like magic today. But that's going to be how we're going to tell robots how to do a task.

We're going to either demonstrate it, we're going to opt-in operate it just one time, two times, and the robot will go do it. I mean, if we didn't have this inflection point with software-based LLMs in the past year, I probably would not believe you, but seeing what happens in the past year, I believe it. Yeah. Yeah. It's crazy. So I think those are the two things that I, you know, I want to think what happens. Yeah. Multifunction robots that can be just taught by telling them what to do, showing them what to do. Right. You make such a good point about, I mean, for the history of robots, it's always been the thing with the arms and the face and it talks. Yeah. It doesn't have to be like that, right? So, yeah, I can't wait until I have a disembodied arm robot that can do whatever I show it to do in my house, like load dishwashers. So, so AI is reshaping a lot of software engineering jobs, right? I mean, it's a little scary and also enthralling at the same time. Some people are wondering where their careers are going to go.

And, you know, maybe you're you've inspired some people to look into this. So if I'm a senior engineer today, I have no robotics background, but I have a heck of a lot of software background. what would you start learning besides the basics that we just talked about playing around like what what's the best way to really get into this the best way if you want to you know get into robotics it's just starting i believe with simulation because that is where you know it still feels like you're working with software but you get to experience the physical challenges where you don't think about, you get exposed to gravity, you get exposed to friction, you get exposed to all the real world challenges that you will encounter when you get to the hardware. So I think that's the first thing, is get out, start working with the simulator, get exposed to those terminologies

and the data that you're not used to when you're working with code and software. And then once you get comfortable with the simulation, then deploying it on hardware, as I mentioned, there is a lot of different hardware modalities that are available now where you can deploy the models that you've trained. You can get one of those and start deploying them. You have to get physical when you're working with physical AI. So don't be intimidated by it. Embrace it. And I think the more and more software developers and hardware developers work together, I think we'll have better systems. Today, still, I feel that a lot of times the hardware developers who are building the mechanical team is a little bit operating differently, decoupled from the software team. But I think when they start understanding each other's languages, which is what would happen if, you know,

as software developers get into physical AI, they need to understand, you know, all the mechanical constraints. That would be pretty exciting. and I think we'll have much more capable systems. Before we talked, I was wondering if you were going to say, like, oh, you need to go get a PhD in math or something. I'm joking. But, like, I think what I'm hearing is, like, no, that's starting to change. Just like in software, when you wanted to mess around with, let's not even talk about LLMs, but machine learning, it used to be you've got to know a lot of math, you've got to really get in there with the concepts, and now it's becoming a lot more user-friendly. Sounds the same thing. Yeah, I think the open-source ecosystem around robotics is pretty strong. You have everything that you need from tools to assets to models. It's all available. So you can really go ahead then and get started. All right. Well, Amit, this was fantastic. Super, super interesting. I really thank you for being here and explaining this to us. This was great. And I really enjoyed being here.

And hope you have a good rest of your week. You too. Send me that dishwashing robot when you get it. Yeah, I will.