How AI Data Platforms Are Shaping the Future of Enterprise Storage - Ep. 281

Hello, and welcome to the NVIDIA AI podcast. I'm your host, Noah Kravitz. Quick note before we get started. If you're enjoying the pod, please take a moment to follow us wherever you get your podcasts. It'll only take a second. It helps us deliver a better show to you, and it helps you make sure you don't miss an episode because they'll all show up in your feed. That being said, let's get to it because we've got Jacob Lieberman back on the podcast, and I can't wait to get into this. Jacob is the director of product management for NVIDIA's Enterprise Product Group. He was on the show recently. It feels like just yesterday, but so much has happened since then. We were talking about agentic AI, and at the time it was kind of a new thing, and Jacob was explaining what it's all about, but really getting into the potential, sort of the promise of human and AI agent collaboration in the workplace and the enterprise in particular. So we've got Jacob back, and I can't wait to dive into it. But first, man, welcome. How are you? Thank you, Noah. I'm very grateful to be here, and I'm very happy to be back. So we're going to talk agents. Also, we're going to talk, you're going to talk, maybe I'll ask some questions, about something called the AI Data Platform, which is a new class of GPU accelerated storage, as I understand it, that I don't want to put words in your mouth, but it sounds like it could be game changer, at least like another big step along the path to this promise of human-AI agent collaboration that we're fulfilling. I use agents all the time, but I'm excited to get into it. So why don't you dive in and tell us how it's going with agentic AI adoption in the enterprise? Yeah, it's interesting to hear that you said you're using it yourself because I think many of us are. Many of us are now starting to use AI agents in our daily lives as consumers. But the needs of a consumer are very different than the needs of an enterprise. And so since last time we talked, AI agents, the technology has come a long way. The open models of today are just as powerful as the commercial models were a little while ago. But it's still challenging for enterprises to put agents into production. So it sounds like the technology's there or, you know, there enough because it's always advancing, right? Yes, yes. What are the challenges then that enterprises are facing to actually deploy and gain adoption of agents? That's a great question. And so almost every enterprise and every CEO and CIO, CTO, they are increasing their spending on AI. They want to deploy AI to production, but it's difficult to move from a POC state to production. So the enterprise adoption is happening, but it's still challenging. There are still challenges. Can you dive into what some of those challenges are? Yes. So many of the challenges revolve around access to data. Mm-hmm. There's a few challenges. You know, you have to remember that AI agents are being deployed into a context, a context of human workers, a context of existing technology systems. And these systems really weren't built for AI agents. And so regardless of how you're using AI, whether it's an agent, whether you're training a model, whether you're fine-tuning a model, or even if you're retrieving data to feed it to an LLMS context, all AI still relies on secure access to accurate recent data. Right. So, you know, you mentioned the human factor and that's separate episodes, separate podcast in itself, right? Because that's, I mean, for all of us change, you know. Yeah, we have to bring lawyers and HR folks for that one. Right, exactly. When it comes to the systems and, you know, the legacy infrastructure and that kind of thing, is it a situation where like you kind of have to burn it down and start from scratch? Or can you kind of, you know, update and transform existing data to be AI ready? That's a fantastic question. So there is this term that's emerging called AI-ready data. AI-ready data has gone through a number of transformations to make it efficiently and securely useful by AI systems like retrieval augmented generation or data flywheels or AI agents. And making enterprise data AI-ready is challenging. It's challenging because the vast majority of enterprise data is unstructured. It has no structure to it. It's things like PowerPoint presentations and PDFs and audio files and videos and all kinds of things that you can't just shove into a database and query with a structured query language. Even though people like me try to do that. Yes. I mean, you can try, but it's a difficult problem. Yeah. So what do you do with all the unstructured data? How do you make it AI ready? Well, in general, you have to establish a pipeline that first finds the data, gathers the data, and then does a series of transformations to make the data AI ready, depending on how you're going to use it. The most typical use case for AI inference is retrieval augmented generation. Right. And to make your unstructured data ready for RAG, you have to collect all the data, extract all the text, the semantic knowledge within the data, chunk it up into homogenous sizes, enrich it with metadata, then embed it, which means transform it into numeric representations that can be efficiently stored and searched. And at the very end, you index it into a vector database for retrieval. And you're doing this at enterprise scale, so I'm guessing it's maybe a little slow and expensive? Well, it can be. It depends on the amount of data that you have. Sure, of course. This is one of the challenges is that enterprises generate a lot of data, and they have a lot of data that changes every day. And when you put these two things together, we call it data velocity. It's the rate at which the data grows plus the rate at which the existing data changes. Right. And you have to continuously perform these operations and these transformations to make sure that your data is recent. Got it. So this isn't a one-time operation or even something like you do, you know, twice a year or something like that. Because as you said, not only is there new data being created, but the existing data is changing all the time, right? As, you know, again, I'll use myself as the example. I write a draft and then I change it and I change it again. And then my manager reads it and says, no, no, this isn't it. And I start over. So you've got both. You've got all this new data, all this changing data. So does this have to be just an ongoing, continuous kind of process? Yes. It's easier to think of the data as kind of a river that's flowing than think it as a kind of a static thing. And the big challenge to enterprises is from a governance standpoint, they often don't know which data has changed. Sure. And if you don't know which data has changed, you have to re-index all of it. And so that reminds me, you know, when I'm at home and let's say I put a dish into the dishwasher and it's dirty and I don't know that the dishes have already been washed. And then, you know, my wife Tracy says to me, hey, which dish did you put in there? And I can't remember. So we end up having to re-wash everything. Right. And that's what enterprises are doing right now to make their data ready. Yeah. They're re-washing all their dishes every time they have one dirty dish. Right. So there's got to be a better way. Is this where the data platform comes in? Well, it is. And also, you know, but just to touch on one more thing, another challenge is data security. You mentioned, yeah. Yes. So data security is really a less obvious problem than making your data AI ready. But it's potentially the bigger problem because when you put your data into a pipeline to make it AI ready, that generally involves copying the data. You don't transform the source of truth documents, you make a copy and you chunk that, extract text, embed that. And every time you make a copy, you're increasing the attack surface on your data. And it becomes even more threatening if you have to move that data out of your storage because as soon as you copying the data somewhere else it becomes disconnected from the source of truth documents So that means if the contents change like we just said, we call that data drift, the content change might not be reflected in your AI representations. But what's even worse is what if the permissions change? What if Noah should no longer have access to this document? His permissions are removed, but he can still access all the copies. And we're seeing that in a typical enterprise, as chatbots and agents proliferate, we're seeing multiple copies of the same data set. Sure. 713 copies appearing all around the data center. Right, right. Disconnected from the source of truth. So how do you secure it? Well, this is really where the AI data platform that we're working on comes in. Got it. So AI data platform is a reference design that we're building, and we're sharing with traditional enterprise storage partners that shows them how to use GPUs to make their data AI-ready in place without copying the data, without moving the data, and continuously. And by doing this, you can make your data AI-ready as a background operation that doesn't require these complex pipelines. Okay, so there's a lot in there. I'm going to ask you to unpack it, right? Because it sounds so good, it's got to be complex. Well, you know, the best technology is so complex that it seems like magic. Seems like magic, exactly. But yes, it is complex and we're not quite there yet. But which part would you like me to dive into? I'm curious, first off, what the GPU is doing and sort of how, I mean, my biggest question is, how can you do this without copying and moving the data? because the way you set it up was great. And it made me realize, you know, the security issues with that. And I could see those immediately in all the files I upload to the different chatbots and different systems. You know, exactly, right? So I don't know, dive in from whatever angle I think makes the most sense to kind of explain how the reference design works and how it, you know, improves upon the legacy system, not just by not having to copy the data, but, you know, how the continuous monitoring is enabled. Sure. Yeah. Yeah, I can go into great depth here. But the most important thing to understand is that your data has gravity. Data gravity means that it's big, it's growing, it's hard to move, it's expensive to move it. If you push your data to the cloud, there's egress fees, there's ingress fees, and there's latencies associated. So naturally, enterprises prefer to keep their data in place. So far, in order to do AI, you've had to send your data out to some kind of AI factory with a GPU, do all your processing and copy it back. Right. So the data has gravity. And it turns out that instead of sending all your data to the GPU, you can actually send your GPU to the data. And what that looks like is actually putting a GPU into your traditional storage system on that same storage network and letting it operate on the data in place where it lives without copying it out. And the advantage of generating these AI representations with the source of truth data is that if the source of truth changes, you can immediately propagate those changes to the representations. And if the permissions change, well, guess what? Those get propagated too because they're governed in the same system. Are there potential drawbacks? Or not drawbacks, I guess I'm thinking more as you're working out the reference design. And we'll get into in a minute, I want to ask you about, you know, how the reference design is being used right now internally or even with partners if they are. What are, maybe to phrase it this way, potential drawbacks you see that are challenges that you're solving as, you know, this moves towards production? Well, drawbacks are challenges that we're solving. Well, one of the challenges to implementing an AI data platform, you know, this is really funny, Noah, because like I said, all this stuff occurs in a human context. Right. Human, all this innovation, it's occurring with people. Yeah. And when I talk to people about AI and storage, they look at me like I'm crazy. So, and a lot of things we do at NVIDIA, people look at us like we're crazy. But this one in particular, you know, I might say to someone, AI is changing the world. They nod their head. AI will be everywhere. Yes, yes. AI will be in your refrigerator. It will be in your car. Yes, yes, yes. It will be in your storage. No way. That will never happen. Storage hasn't changed in 30 years. So within the context of an enterprise, they draw a very bright line between holding the data and working on the data. Interesting. Yeah. Yeah. So there's this kind of natural pushback. But I do believe that we will have AI everywhere, especially close to the data where it's the most secure and efficient to run the AI operations. Okay. So I'm an old school, I don't even know what the right word is, right? But I'm in charge of data for my organization. Yes. And you convinced me that this is the future and I've got to let go of my old ways. Yes, I'm very persuasive. Embrace the future of, you know, of bringing the GPU to my data, right? Yes. Okay, I'm on board with you. What's the GPU actually going to do there? Yes. So the GPU will be applied to managing the data and preparing it for AI continuously. and as a background operation. So that complex pipeline I mentioned, finding the data, the data's there. It's found. It's in the storage. Gathering the data, extracting the text, chunking it up, doing the embedding, doing the vector indexing, and even doing the search and retrieval of the data, the semantic search, which means you're searching based on the meaning of the documents, not keywords in the documents. The GPU can be used for all of those things. And it can just hum in the background. And then when I'm ready to work with the data, the data will be ready. Right. And if you look at an enterprise right now, data scientists are spending an inordinate amount of time, maybe some estimate up to 80% of their time, wrangling the data and preparing it for AI. But if you can free them of that toil and the data is being prepared for AI continuously as a background operation, not only is it more secure and more efficient because you're not copying data around unnecessarily, but it also frees up your precious data science resources to actually do data science. And that's the whole idea. That's the whole idea. You want to get insight from the data. You don't want to spend all your time messing around with the data. I'm speaking with Jacob Lieberman. Jacob is director of product management for NVIDIA's Enterprise Product Group. And we've been talking, well, we've been kind of following up on our previous conversation that we mentioned, which is really at the beginning of what's really just a once-in-a-lifetime transformation, you know, to call it that, and you can correct me if I'm wrong, of AI and specifically agentic AI just changing the way that all kinds of work gets done. And now we're talking about the AI data platform, getting your data ready in the background. Jacob, you mentioned that it's a reference design. Yes. So what does that mean and how's the design being used at this point? So those that are familiar with NVIDIA and a lot more people are familiar with NVIDIA now that when I started here, know that we always go to market with partners. We don't sell anything direct. We always sell with and through, never to. And this is no different. So we equip our partners with reference designs based on our AI blueprints, which are reference architectures implemented in code. We talked about them last time I was here and how they show with our embedded best practices how to use our GPUs for some business outcome. So we give the partners a hardware reference and we give them software references based on the blueprints and then they go to work. And what's really interesting, though, and what really surprised me is that the storage industry is eager to transform. They see how transformational the potential that all of this is for them and for their industry And so they are taking these designs and they are going far beyond what we giving them They're differentiating and innovating in many ways that we didn't initially expect. Fantastic. And it's very cool to see. Yeah, I bet. Any examples you can share? So, yes, I don't want to highlight or talk about any particular partner specifically. If you go to our web pages and look up AIDP, you can see plenty of case studies and ways these are implemented. But I will talk about them in the general sense. The most exciting thing for me has been that now that the GPU is running in the storage, doing these background operations to make your data AI ready. Well, the GPUs are powerful, and some of their spare compute capability is being used to run agents in the storage. Hang on a second. I'm wrapping my mind around this, right? So you're bringing the GPU to the storage so you can get all the data ready. But then almost as like a bonus benefit, you can actually, and maybe this isn't the right way to say it, but you can actually have the GPU like start doing the work on the AI-ready data that it just made ready. Yes. And so it's just, it's all right there. Yes. So that's a great way to describe. Actually, I might start saying that. Streamline it a little bit and take it. No, that's a great way to describe it. So if you have the GPU there, this is what our partners are starting to do. They're running agents in the storage that are operating on the data in place. One example would be that one big concern around data governance is making sure you don't share classified documents. It's possible to look at all your data and find the word classified or do not share. But what if your document should be classified, but it's not marked that way? Okay. You could have an AI agent running in the storage that's looking at all the documents and saying, oh, you know, this one really should be classified. Right. And letting you know before that data gets out. No, that's great. It's so simple, right? Because it's already getting the semantic information out. Yes. Not just gets deeper than that. Yeah. And there's many, many ways we're seeing other partners who have an agent running in the storage system that monitor the telemetry and the usage of the storage system and then make recommendations to the administrators about how to optimize themselves. So that sounds just like efficiency taken to another level. Yeah, so that's a good way to put it. I like to think of it as letting your AI agents work from home. So, you know, trying to stir some controversy. Well, I don't, you know, I mean, a lot all over the all over the world, people are returning to the offices. But it's been shown that workers are generally happier when they work from home. And why is that? Maybe I mean, you've worked from home. What did you like about it? I mean, the first thing is, you know, instead of spending the time commuting, you're spending the time getting things done. Right. So that if you think about it for many types of work, that commute is unnecessary data movement. You're moving the data in your head. Your data in your head doesn't need to be in the office to get your job done, right? And it's also true for an AI agent. It's more efficient to send the compute to the data than to send the data to the compute. So I think just like human workers, AI agents are probably happier working from home than they are, you know, working in some remote office. In the right situation, right? Does the metaphor extend? There are situations where, you know, it's better to be in the office with the other people for, you know, work reasons, social reasons, all that good stuff. Absolutely. So when you want to communicate with other agents, when you want to build multi-agent systems, when you want to use your agent to trigger an event and drive another process, that's when you go out. Right. You know, but the metaphor also works in the other direction. A lot of people like working from home because they can control their environment. They don't have that noisy neighbor. They don't have that. They have the lighting how they want it. They can work hours that they want. They might have a dog. They might have a dog. I don't know if AI agents have virtual dogs yet. But the storage vendors putting AI agents in the storage know the environment they're going to be in. They know the operating system. They know the capabilities, the APIs. So it's a very controlled environment where they can work very securely and efficiently, just like you do when you're at home. Yep. So NVIDIA isn't known as a storage company, which is kind of irrelevant these days, right? Because, you know, for a long time, people didn't think of NVIDIA as a software company, but full stack, right? Cuda, hardware, software, the whole thing. Well, people don't think of NVIDIA as an infrastructure company. Well, you know, there's a lot of cool stuff going on with NVLink and, you know, all the interconnectivity stuff. How does NVIDIA approach, how do you approach working with storage vendors? And, you know, you mentioned that the storage industry is just eager to transform. What's that all like, you know, in your experience working with these folks? And then kind of NVIDIA's take or how people might start thinking about NVIDIA in the context of storage? So that's a, wow, that's a two-part question. So number one, working with the storage partners, I found it to be fantastic. Yeah. And the main reason is that it's really a better together story. The storage vendors are experts around data protection, data governance, encryption, auth, RBAC, ACLs, all those things that you need to do. The things you need, yeah. Yeah, all those things enterprises expect to be there. when they move something from a POC to production. Well, guess what? We're the acceleration company. We are not the company that knows those things inside and out. And yes, while we can probably do it, it'll take us a long time to build up the expertise that the storage partners have. Similarly, they could all learn how to keep pace with the innovation velocity of AI. But is that really where they want to spend their time? So now by partnering, we can both bring to the table what we're uniquely good at, our complementary skill set, and bring something to market that's even better than what we could do by ourselves. Now, in terms of NVIDIA and storage, because that was the second part of your question. So NVIDIA technology has been used in storage for many, many years, particularly in high-performance computing and scientific computing, where you're having to move around huge data sets very quickly. And we've taken those learnings and we've applied many of them into our networking business and into how we manage compute. And I could rattle off a dozen technologies that are related to storage, but we don't have a storage product. Now, I expect that we will take all of these learnings at some point and really build out a complete reference design that goes very low level down to the disk all the way up to the top of the stack. Because that's what we do. That's what we did for AI infrastructure with DGX, the best AI training system in the world. That's what we do for networking. So I would expect that at some point at storage. But the moment for the innovation is right now. And the storage vendors are jumping all over. So, Jacob, I've got this picture in my head now that you've painted of, and I don't know, the metaphor that comes to mind, right, is like a library, right? And it's this big building full of all kinds of knowledge. And, you know, it's actually sort of structured to some extent. Thank you, Dewey Decimal System. But you know what I mean, right? It's all in there and it's vast. And then I'm thinking of the librarians kind of as like the agents, right? Because they're in there. And particularly if, you know, like the GPU was brought to the data, as you were talking about, right? Like the librarians know the library. They know where the different sections are. and, you know, the different, all the different, like, idiosyncrasies of their particular library and how to work, right? So I've got this vision in my head. I don't know how accurate that is, but it sounds like the agents are here. They're going to keep improving. Obviously, technology keeps improving, but they're here to the extent that, like, they're ready to do real work, right? Yeah, they're ready to go. And so now we've got the AI data platform that's working in the background, getting all the data ready for the agents to work with it. Yep. So when we get out of my library metaphor and get into the enterprise the real world what next Are they ready to go Is it still some steps to getting the AI data platform kind of to production ready Is it there yet? You tell me, what happens now in the enterprise adoption story? Well, Noah, I'm a sucker for a good metaphor. So I'd like to stay on the librarian theme for a minute. I love that. But, you know, a really good librarian will, when you go in there and you look for a book, they'll ask you what you're looking for. And maybe they'll bring you the book or article you asked for. But a really good one might bring you some other things. Exactly. And say, oh, look at this. Oh, have you thought of that? And these are things you never would have found on your own because you didn't ask for the title. But they're semantically related. and maybe even more relevant than what you initially asked for. So that's the value of a good librarian. But all of that is contingent on the librarian having recent, accurate access and understanding of all the data. And, you know, security, I guess, also is there because sometimes people would take the books home and not bring them back. I think all of us have been guilty of that. Or maybe not returning them on time. On times. Yeah, on times better. But yeah, but I love that librarian metaphor because that's kind of what we're building here. In terms of the readiness, we do have partners who have AI data platforms in market. Yeah. They are out there with Lighthouse customers and we're getting feedback and we're improving and making them better. And I imagine a future where all storage will be GPU accelerated and there will be no AI factory that does not have an AI data platform. There's too many benefits to it. So it sounds like this is a pretty radical shift, the idea of bringing AI into the storage, like a big sea change for the storage vendors. In some ways it is, but in some ways it's a very natural evolution of what our storage partners are already doing. So I remember a time when you connected storage through a cable to your server and you use the CPU on the server to set up the RAID and protect the data and maybe share that volume via NFS or another protocol. But then at a certain point, the storage systems themselves got CPUs and network cards. And you could network attach the storage and it could manage itself. The CPU was part of the storage infrastructure. And as CPUs got more powerful, all of a sudden, instead of just serving the data and protecting the data, they were doing things with the data like compression, decompression, encryption, and keyword search. So there's some level of data manipulation already happening. Right, right. So if we extend that now with the GPU, which is much more powerful than a CPU, instead of encryption, compression, and keyword search, we have embedding and indexing and semantic search. So in some ways, it's totally different. But in other ways, it's a perfectly logical evolution from where we were yesterday. Yeah, you make it sound very much like a natural evolution. And I think it is. I think it is. I think the divisions we've put up around data management for AI and regular enterprise data management are somewhat artificial. And really, it's around the GPU. Yeah. It's around the GPU. But when the GPU is in the storage, that division goes away. I really want to say tear down that wall. Say it. I just did. Yeah, tear it down. So you mentioned, you know, the vision of an AI data platform in every AI factory. And to be honest with you, it took me a second to get past, well, wait, aren't they already there? Oh, no, wait, they're operating on, you know, the way we've been doing storage. So dive into that before I let you go. Kind of unpack what bringing the AI data platform into the AI factory is really going to mean. Yes. So what is an AI factory? I'm sure you've covered this. No, but please. Okay. So an AI factory is infrastructure for data center scale, AI training and inference. That's what it is. And we work with many partners to make sure that all of their software runs great on our GPUs. And so the AI factory is where you actually run the majority of your agents that are delivering business value. So if you already have those GPUs, why do you need an AI data platform? It's a great question. Well, there's all of those things we talked about in terms of efficiency and security. Those are still true. Whether you're sending your data to the cloud or an endpoint to be prepared or sending it to a factory, it's more efficient and secure to do it in the storage and in place. Yeah. On top of that, there's other advantages. If you decouple the data from the compute, they can be scaled independently, which is more efficient and cost effective. Okay. So in AI factory, you want to add GPUs and you want to add memory to increase your token output. And as you add chatbots and as you add AI agents, the AI data platform grows when your data grows. You don't necessarily need to add more of your compute servers because your data is growing. And conversely, you don't need to deploy more storage just because your factory is growing. So this lets you decouple their scaling. It also lets you decouple their lifecycle, the lifecycle of the software. So in an AI factory, Gen AI is evolving so quickly, you want to be able to absorb all of that new innovation. Of course, yeah. Yeah, day zero, you have a NIM. You put the new NIM in there. It runs great on our GPUs. Right. But your data is a different story. You want to protect your data. You don't necessarily want to change that all that quickly. So when you connect an AI data platform to an AI factory, you can innovate quickly on the compute side, but be more conservative in the data path. Got it. It's amazing stuff. For listeners who want to learn more, maybe want to find out about building an AI data platform in their organization, their AI factory, maybe they just want to dig into some of the technical stuff behind all the innovations. Where's a good place for people to go to start their journey to learning more about the AI data platform? So we do have a landing page. If you search NVIDIA AI data platform. Easy enough. That will take you to an estate on our webpage that actually has links to all of our partners' implementations. Perfect. Yes, because the partners are really the star of the show here. Right. They have taken our design and turned it into, you know, 10, 15 different, very innovative things. And so it's a really exciting space to watch. Very cool. Jacob Lieberman, again, this has been really fascinating stuff to think about. And, you know, the idea of just whenever we can talk about what all this amazing technology is doing on the ground in the real world, solving business problems, solving real world problems. You know, those are always kind of my favorite episodes. But this is great because the innovation, you know, it's an extension of all these things. Extension of the agents, GPU in the storage, kind of an extension of what CPUs are doing in the storage. And it's really cool just to see, to hear you talk about the evolution. So thank you for taking the time to come on the podcast. Yeah. And thank you for having me. I'm very happy to be here and I'm always grateful to spend time with you guys. ¶¶ Miss bulgey Luna