Nebius CTOが語る、ハイパースケーラーが目覚める前にAIクラウド市場で先行する戦略

If NVIDIA could have bought or could have produced like 10x the chips, this whole market would have been shaped differently. Because there is not enough capacity for their scale, this is a relatively small business for them. For the hyperscalers, it's not just a matter of like sales. It's also a matter of changing the infrastructure that was built at scale for other kinds of workloads. This is a massive investment on their part that needs to be justified by the size of a potential business. And it's not that there yet. The emergence of Nebius and the other hyperscalar competitors these past years has been a fascinating part of the AI story. It's something I've really been interested in, and so it was a lot of fun to have Nebius' CTO, Neela, on the podcast, because we were able to dig into a bunch of interesting things about the business and how the space has unfolded. We talked about Nebius' really interesting origin story and the original relationship with the Yandex and how the company came to be. We talked about what it's like to be inside one of these companies, how Nebius works with NVIDIA, what it looks like to get their next generation of chips live.

We talked about why the opportunity actually existed and why AWS, GCP, Azure did just capture this entire market. And we also talked about Daniel's view on a bunch of new technologies like DISAG, inference, among other things. All in all, just awesome to get to talk with someone who's at the epicenter of the AI infrastructure market about his thoughts on everything about where the space is headed. I think folks will really enjoy it. Without further ado, here he is. I guess I'd love to talk just maybe about the broader, like AI cloud market, you know, today. And I think that there's a few maybe larger and prominent public, you know, I guess termed NeoClouds. And then, you know, it feels like there's a new one getting funded every day. I know you don't like that term. But maybe just talk a little bit about how you think about what differentiates. Like some of our listeners are probably thinking like, you know, all that really matters is just access to GPUs and like, you know, wherever you can get them, you go. Maybe use that as a straw man and tell us why that's not the case. Well, I'm obviously biased because, first of all, I'm a software person, and our product and business was launched on the idea that software is the mode, right?

So your ability to serve the customers that consume, I don't know, tens of thousands of GPUs in one interconnected cluster with the same platform as you serve a self-service customer that rents one GPU for half an hour. was our value prop from the very beginning. So we believe that this is extremely important, being able to build large scale, but have the possibility almost like sell it in retail, in a sense. This is what we think definitely differentiates us from the market, from everyone on the market. Like a small remark, Neoclouds hate that term with a passion, because it started as a distinction as in these are kind of cloud companies that are doing this weird, strange new thing, but then everyone wants it in that market. So when we started, there were like five, seven companies that were in this area that were called NeoClouds.

Now I think it's closer to like 300 to 500, depending on what you are ready to call NeoCloud. the scale and we really don't like this stretched umbrella term anymore we are the ai cloud we don't care like let others other guys beat neoclods we are the ai cloud and this is our mental model we are not competing with the neocloud markets we kind of believe we are past that our next competition point is taking on AWS and Google Cloud. Yes, we are way smaller. Yes, our revenue is like 1% of AWS nowadays. But we are running fast. We serve a very particular market. We have a bet that this is the market that is going to grow, that is going to explode. So people are concerned that these investments are huge.

people put a lot into the infrastructure, etc., etc., etc. My view, if I compare what we see in the market with what was there two years ago, we are barely scratching the surface. I think the overall demand in the next three years will increase tenfold, hundredfold maybe. So we are at the verge of another explosive growth. right now we see these huge facilities right but a lot of those are privately owned for internal needs of certain companies right so Elon Musk is talking about his Colossus cluster and it's like it's a man's scale like really it's like real solid large scale thing but it's not for the broader market right it's for their internal needs maybe for some partners that they decide yeah and traffic can open AI, amassing the capacity, but they do it for their own product. And this is like

a wholly separate discussion. But these companies will essentially want to be everything. They will want to be the internet and contain everything and every product and every capacity angle within themselves. There is a question of Google, Meta and Microsoft who have a lot of capacity by constantly juggling and struggling between giving it out to the market to rent or giving it out to internal teams because all of these companies are extremely machine learning, AI enabled. They need a ton of capacity for internal users, for runtime and for research. So they have to constantly juggle this. So our bet is that there is a niche for independent as in not affiliated with a large company with a huge internal consumption player that will serve the markets of the mid-tier players who want to be the large companies. I thought it would just be interesting for our listeners

to talk a little bit about the history of Nebius. I feel like it's a really interesting path that you had to end up here. And so maybe just talk a little bit about how the business came to be. I'll start with the moment I've joined, right? So I am CTO of the company, but I've been with the company from the very beginning, like since before it was called Nibus. So the new company is established, I think, during 2024 also. The deal was finalized. We were finally announced our separation from Yandex. No connections to the Russian assets anymore, Clean Slate, et cetera, et cetera, et cetera. And that was it. You know, people have come to these AI cloud businesses in different ways. and obviously, you know, I feel like the Yandex team was, like, legendarily really, really strong, you know, on the engineering side. You know, a lot of folks building iClouds came from, like, crypto or energy or, like, not necessarily the same, you know, maybe technical heritage. To what extent, I guess, was the Yandex experience really relevant, and to what extent was it like, you know,

it turns out no one knows how to build a GPU cloud because it's all kind of new and, you know, there's everyone's learning on the fly. First, we had and still have the core team that supports the hardware side of business that came from Yandex, that had experience operating hardware infrastructure at scale, like at real scale. Hundreds of thousands, millions of servers span over multiple facilities, etc., etc., etc. These people were the core of our build-outs, initial build-outs and expansion, and they're still with the company. So this definitely helps. I think, like, we would be way less successful if not for that core team. On the software side, it's interesting because it's a two-fold. Because, like, with hardware, it's very straightforward. We needed people with experience. We had people with experience. They started building out from the get-go. On the software side, it was a weird one. First of all, we had to rewrite everything from scratch.

So our previous experience was kind of, like, garbled. Second, we had to grow so fast that by, for example, by the end of 2024, when the new platform launched, a significant portion of our engineering team, software engineering team, was not from Yandex anymore because we had to hire during this period extensively. As a team, we were working on this cloud platform from the initial idea, right? So we kind of just went with it and doubled down on software. We are not doing a managed bare metal. We are not doing a patchwork of third-party services. We want to build a holistic cloud experience, a holistic cloud platform. And we kind of doubled down on that even with the whole rewriting of the platform, from doing it from scratch, etc., etc., etc. And, of course, having a ton of talented engineers on the team obviously helped. Just for the scale of it, right?

You've mentioned people who came from crypto or came from other avenues of life. They are mostly concentrated on hardware. Their idea is you need to buy GPUs, you need them to run, and then you'll find the buyer. in our case for example in 2024 when our hardware team that actually like procured gpus did the build outs uh we're working on our own servers and chassis writing firmware and stuff building the network that whole team was roughly like 40 to 50 people the software organization was already 200. Right? So we kind of think of ourselves as a software-first company. We are building the cloud on the solid foundation of hardware. Yeah. We're not a hardware-first company. Makes total sense. I guess I'd love to take our listeners inside Nebius, and I feel like there's so many interesting parts of the business, but maybe one place to start would just be, I imagine as you're getting

new NVIDIA chips that come in, there must be a bunch of stuff you have to do within the business to get this all up and running. So, like, what does that look like from, you know, I don't know, you're talking to NVIDIA folks and they're, you know, giving you early insight into the next generation of chips to, like, you know, actually getting these live for a set of customers. That is a mess. No, I mean, like, in a good way, you know, like, this whole market, this whole field is running at such a pace that, like, everyone is, like, trying to catch up, right? And NVIDIA, despite being, like, in the lead of this, essentially leading and showing the direction for this market, is not the exception. So, first, NVIDIA folks, like, way before, show you the prototypes of what they want to build in the next, like, year, right? They kind of show you the schematics, like, how it will be, like, optimized internally, where is your, like, PCI connections, how would that look like, what is the future platform would look like.

Then sometime later they come to you and say, oh, we've optimized things. It's now going to be different. You're like, all right. Then they come to you and like, oh, we have the first lab samples. Do we want to have a look? Yes, of course we do want to have a look. Oh, by the way, it doesn't look like the stuff you showed us two months ago. Oh, we've changed stuff because they are so compressed in time, same as everyone else they are constantly iterating while these chips are being produced right so it's not your like you know there is like an ocean in the consumer market Apple iPhone that you see today was actually planned three years ago and final design more or less was settled like a year ago and they take the final year to polish it to build this product like AI market is running at such a pace like that whatever they announce is, it's indicative. And then teams are constantly iterating. The workflows are changing, right? So I'm sure they're iterating.

Yes, they're iterating, going further and further and further. NVIDIA is a behemoth now, right? Whatever they do influences the whole market. Supply chains, everything. So they have to... So at some point you get a sample and it's more or less like what you'll get in production. then you start building like you are testing everything you like incorporate support for the new hardware for the new restrictions or capabilities into the software platform you figure out a way to isolate workloads from the we are a cloud, we are not providing bare metal hosts, right? so we need to split, we need to make sure that our virtualization works, that our storage works that our networking works sometimes we are on the bleeding edge of this so we for some of the generations we were probably the first team, like maybe one of the first teams that actually built a cluster of a certain size

so we've hit some issues with the software that no one has seen before so we had to report that back to NVIDIA like propose a patch get there, fix from it make sure that it runs, etc. etc etc so like we kind of feel like we are at the forefront of it but all in all it's a pretty normalized process it may sound messy right there are like multiple iterations you go back and forth you're in constant uh contacts with like nvidia hardware folks with the nvidia software folks uh you're trying to solve the issues on your end at the same time incorporating into a more complex platform thinking how can you expose that to customers? Because, for example, working with, like, your usual black hole chief, like, you know, the B300s, and then transferring that experience to GB300s, which I call different architecture in terms of isolation. You are talking racks rather than servers.

You need to find a way how to do maintenance on the rack instead of the server, which you just switch off and you move the workloads to another one. Despite all of that, this kind of works all the time. And every day you, like, push it a little bit further. And then in the end, yes, there is a product and the customers use it. And that's it. One thing I think would be so interesting to get your thoughts on is just the differences in, you know, kind of what end customers end up wanting. Like, obviously, it can be easy to think, okay, everyone just wants to, you know, come to you and get similar things. but I imagine you segment your customers or think about differences in what is important to different customers How do you think about the key differences in these customer desires today Hi I Anne Maris and I the producer of Unsupervised Burning Thank you so much for listening to the show. And if you've enjoyed the conversation, please subscribe. Subscribing helps us grow, which helps us continue to lock down some amazing guests. Thank you so much for your

listenership and see you next episode. How do you think about the key differences in these customer desires today? It's not that different, to be honest, because like every customer is like its own unique flower, right? But at the same time, workloads in principle are not that different. It's either training and you kind of know how training looks like or inference and you kind of know how inference looks like. So coming back a couple of years ago, for example, we kind of underestimated the storage component. We thought GPUs are the most important thing, and they are. But alongside the whole training angle, you see a massive demand for storage. And it's not for the amount of AGGs or VMEs that you can provide to the customer. It's about throughput. It's about latency. In my prior life, pre-AI workloads and infrastructure build-outs, I was talking about gigabytes per second if we were talking about network-attached storage, right?

It was like, it's willing. Right now, we have customers routinely kind of like hitting a common share, like one volume, right, connected to a number of GPU servers with like, you know, five, six, seven hundred gigabytes per second of throughput. this is extreme like two years ago I didn't even think about that and now this is more or less like something we expect at the same time if you are targeting these numbers for the highest demand like the most demanding customers your baseline is so high you kind of can't serve anyone right so if you hit this like use case where a customer and needs 700 gigabytes per second from, like, you know, three petabytes of shared file storage connected to 1,000 of servers or, like, 8,000 GPUs, you kind of know how to serve a massive cohort.

So I'd say in terms of, like, performance or features per se, the customers are not that different. Then you meet another subset of, like, what the customers really want because there are definitely different customers in terms of not their scale, not their size, not their workloads, but rather how mature they are, or what are their obligations for their clients, right? Because we can talk about, like, a Neolabs. Like, Neolabs routinely have the seed rounds in the billions nowadays, right? So most of that budget goes into commute. So you can have, like, a contract for a billion dollars with the newly founded Neolab, they don't care that much about anything but GPU storage performance availability. This is it. Oftentimes, they don't even care about the location of those GPUs. They don't care if that's US, Europe, France or Iceland.

Why bother? Internet connection is good. Storage is there. GPUs are working. and we are talking about a billion dollar contract, right? At the same time, we now see and start onboarding enterprise customers. These are companies whose contract with us will be smaller, sometimes significantly smaller, but they are more established. They have customers and they have applications for that customers. They have procedures. They need to make sure they're compliant to a number of regulations in like multiple countries. So now we are getting into that whole angle of being the real cloud, you know, like certifications, compliance, security posture checks. Let us look at your source code. How do you solve problem X? How do you like protect your customers from attack vector Y, et cetera, et cetera, et cetera.

And I'd say if you are talking about differentiated customers, this is the most differentiating thing as opposed to what you would expect from the size or the performance. You know, obviously, I think a lot of people are thinking through this trade-off of like throughput latency or a trade-off of like latency and kind of cost. And I imagine you have some customers that have workloads that are super latency sensitive and they'll, you know, do small batches or whatever is required to get to that latency. And then you have other customers that are much less latency sensitive, maybe background agents or something. is that all just, you know, determined on the software side, I guess, if you have those different customers of, like, basically how, you know, the batch size that you're running these workloads in? Or do you actually construct the clusters in any different ways based on those workloads? No, no, no, no, no. This was the premise from the very beginning. We are building the universal cloud platform, and we need to make sure that our baseline is high enough to fit most of the workloads. Yeah. Then there is the SLA play, right?

So customers come, like, for example, for the managed inference part of the business, customers come with the different demands. And then you are in this kind of a benchmarking game, right? They want this particular footprint based on this particular traffic pattern to fit these SLAs in terms of latency. Or latency is time to first token. Or the whole request latency. and you start optimizing for that but this is never done on the hardware level because hardware is good enough this is rarely done on the lower infrastructure level because it's universal and again, good enough no compromise for performance, etc. and you're going further away to the orchestration level to the model level way further up the stack all the way to how exactly does the request from the customer, like the endpoint, gets to the influencers and what are your performance hits along the way.

Yeah, that makes sense. So this is like 100% software. I guess a lot of people have talked to me, it feels like this disag, splitting pre-fill and decode is very in the air these days and a lot of people talking about that. Is that something you guys think about now or at least how do you think about that kind of in the future? I think we're in the stage where everyone's talking about that and no one's doing that properly. Because it is a complex problem to solve. And particularly in inference, you have a lot of other problems, like way lower hanging fruits that you can fix address and hit those SLAs. So we are quite cautious about that. We don't have it, for example, as a go-to solution for every customer, for every workload. Because first, it doesn't make sense. Second, it's really hard to operate and work with, like provide proper SLAs around that is like proving quite difficult as of today. We have some promising results in our like prototype services, but not as of today.

At the same time, we have some like disaggregated setups for the most demanding customers in production right now. It's just more, you know, wide glove manual approach. But especially for this newer architecture Like those gray black walls Whole node that has really fast interconnect A lot of challenges that you've had before With disaggregated steps are kind of soft If you're able to fit in the footprint of a rack Because the throughput is so much faster Like interconnect is so much faster You kind of stop caring And that this is a way of just balancing the perfil and decode nodes to match the particular pattern that you see from the customer. What's your mental model for like the workloads that do fit in a rack? I don't know. It's very case by case. Well, these are definitely significant workloads. Like we are talking like thousands of requests per second, right? We are talking like billions of tokens that flow through.

But at the same time, because we are the cloud, we can, for example, like share the rack between two workloads and not be that bothered about that, right? So, for example, on the cloud side, for this particular product, we only allow to consume it at rack level. You cannot order two grace blackwells nodes. It doesn't make sense. Right? You have to allocate the whole rack and then structure workloads within it. But, for example, when you're a client of such a solution, as we do with managed inference, for example. You can collocate workloads, and you can find a way to do that. So, in a sense, you don't need to fit anything in the rack. It's just, is there a workload that will benefit from this disbalance, like intentional disbalance of pre-filling decode nodes? Yeah. And can you fit more than one into a single rack, and how do you solve the packing problems?

Yeah, that makes sense. You kind of mentioned there's, like, lower-hanging fruit on the inference side than doing that maybe. What are, like, some of the lower-hanging fruits today that you still think we're not, as an industry, optimized on? Well, first of all, I think as an industry we didn't optimize shit. So, like, pardon my French, but in a sense, I strongly believe that running inference, particularly in the, like, variety of open-weight, open-source models that are available today, is not a solved problem. Like, if you take something, just run with it, as it's published, like, in the initial support in some of the open source inference servers, run with the novel models, your results will probably be underwhelming. It's not that fast, it's not that cheap, especially with, like, current prices on the infrastructure. It's easier to go and use, like, I don't know, Fable 5.1, Astra 6, et cetera, et cetera, et cetera. But because of the open architecture,

because of everything that we know as of today, you can optimize the way you process those requests. You can optimize the way the influencer is configured. You put a lot of effort in kernel optimizations, sometimes kernel over-optimizations based on the particular pattern that you see in the traffic. There is a lot of underexplored potential in model weights optimization. And I'm not just talking about stuff like speculative decoding, for example. I'm talking about model pruning, selective pruning, compression. Because I believe that, and we see that within our customer base, 95% of our customers do not run vanilla open source models. These are modified, fine-tuned, et cetera, et cetera, et cetera. it means that these are the models that start to address a particular use case that the customer has.

Particular use case usually means limited domain, scoped domain. Scoped domain usually means that you need to retain frontier intelligence and reasoning for a particular subset of tasks, and you don't care about the rest. and now we are getting somewhere in terms of like maybe you don't need such a large model anymore maybe you need just like a model that is selectively compressed in a way where it works for a particular task and it doesn't work as good for the rest of it now of course that doesn't work with like open-ended you know deep research or broad coding tasks anymore But I strongly believe that for real AI adoption, when we are talking about serious enterprise business, when we are talking about practical AI that actually solves problems, you are getting more and more into this specialized agents, limited domains, where you even stop caring about the costs.

You care more about predictability of the outcome, and you care about time as in, like, real seconds and minutes till you get that outcome. Yeah. And I think there is a lot of potential there, and we still will see a couple of nice breakthroughs there. There's a ton of work around inference optimization, and a lot of that gets done in open source, right, or, like, you know, contributions that share throughout the community. There's obviously a bunch of work on smaller purpose-built models and compaction and a bunch of these things. Across all these problems, which do impact how enterprises consume inference, which ones like Nebius should have a solution or offer a product around and which ones are like, hey, it's great that that exists and we'll benefit from that, but we don't need to be at the forefront. We want to be at the forefront of everything. in a sense these are additive and even maybe multiplicative right? if you have a right subset of things

optimizations that you introduce for a particular use case it means that you can reach the targets easier, better, faster in a more predictable fashion so we invest in all avenues like one of our acquisitions back in like at the beginning of summer was targeted specifically that. We basically bought the company for the talent in model optimization, model compression, speculative decoding, etc., etc., combined it with existing talent. So we have a team that is 2x the size. And we have serious manpowers, as in tens and tens of people who are working on these problems. We have specialized groups for kernel optimization, specialized groups for model compression problems. Because this is, like, I feel if you want to talk about, like, running inference as a service at a serious level, this is the only way you can do it nowadays. And I guess, like, you know, maybe along those lines, you guys have done, you know, a few acquisitions, and one of them that

caught a few people's eyes was, you know, Tavoli in the kind of, like, web search space. Did you talk through a little bit, like, you know, the rationale for that, how you think about that fitting into the broader platform? So first we were very impressed by the product and their go market motion It was almost a year ago now right But we felt like we knew how to help them run even faster with our expertise and technology and low-level stuff, because actually running a web search engine is not a very simple task, neither from the software development perspective, nor from... A lot of you used to run one. And some of us dabbled a little bit in our previous lives, yes. Right? So this was the premise. Like, we have a great team that is running real fast in terms of, like, market penetration and, in general, go-to-market motion.

And we know how to make it complementary with our expertise on the low-level stuff and running large-scale infrastructure and, for example, large-scale web index. It's not exactly embedded into the rest of the offerings that we have now today. We are exploring and we have something cooking for the managed inference combined with agentic tools. Because, for example, we also develop sandboxes nowadays, right? So this is another building blocks for one of the building blocks that people use to build their agents. You have several other products that are in the earliest stages of development in that regard, too. Yeah, I mean, it seems like there's obviously a set of tools that folks are going to use to build agents and build on open source. And it seems like you want to have a bunch of those tools to be able to offer to your end customers. And WebSearch kind of being a key primitive for any long-running agent will consume a decent amount of tokens. I'm just pulling information in. Yeah. I guess I'd love to switch gears to just like the broader, you know, industry build out and kind of the, you know,

I think there's lots of questions folks are asking about that. I figured one place to start that actually might just be interesting for our listeners is, to what extent are, like, the data centers that you work with similar? And, you know, like, there's a reference design, and they all kind of, like, you know, I don't know, to you it doesn't really matter which one's which. And, like, what are some of the differences in them that actually impact the business of, you know, running an AI cloud on top of them? Well, to be frank with you, I have no idea what the walls of the facility are made of. so they're probably different but they don't really help but in all seriousness what's within the walls is something that we fully control we do not rent data centers when someone like have built the network on their own or brought some GPUs or servers etc. if we don't own the facility and we rent it for example We rent, well, the building, electricity, but that's it.

All the internal build-out is controlled by us. It's done according to our standards. So from the software side of things, there is literally no difference which data center we're running. Is it our own facility? Is it something that is built within a rented facility? or for example we've recently announced this so-called asset light model when a third party brings capital in order to set up the facility and GPUs and we run our platform on top of that it's going to be the same these data centers are built according to our like fitted out according to our spec hardware is something that is familiar to us because like unknown hardware is quite hard to adopt and make sure that you provide proper surveys. And that's it. So we don't compromise there. We never rent something from people and present it as their own cloud.

If you launch a virtual machine in Nibius cloud, you can be sure that it runs on our stack top to bottom. Yeah, makes sense. I mean, I guess there's a lot of dialogue around the bottlenecks to the AI build-out today, and I think people talk about a bunch of different ones, whether it's power or labor to actually make data centers, There's pooling, networking, you know, HBM, GPUs, I guess, as overall. Like, how do you think about what the key bottlenecks are today? Or if you were to kind of rank across all of them, the few that feel most poignant right now? I'm sorry to disappoint. There is one single bottleneck, and that's your ability to execute. Because, yes, all of those are issues. Like, there are GPU shortages in the market, components shortage in the market. nowadays it's CPUs shorted from the market like NVMe is shorted from the market there are like not enough facilities, not enough electricity, not enough qualified workforce to

help you build it out not enough small components like network transceivers for example, what else is not in abundance oh, you know like those large scale I'm not sure I like I know the proper English terminology but like you know those machines that actually convert electricity in place like transformators transformers yeah like do you know the lead times on those these days? Ooh years right so like everything is a problem and I think the single bottleneck and at the same time the single like mode that you can impose on that is like your ability to execute around all these problems and concerns, your ability to source, your ability to cut corners where appropriate, your ability to execute in general, your ability and flexibility to move stuff between facilities. Okay, we're going to move this one from here to there,

and then we're going to move it back to the mainland. Yes, we will postpone this one, but this means we'll be in time with this one. This is important, et cetera, et cetera, et cetera. So I think that this particular field, Like an experienced project manager is probably in highest demand I've ever seen throughout my career. Yeah. That's, I guess, the bottleneck is really good project managers. Yeah, exactly. Like your overall ability to, like, have this large context of, like, problems everywhere in mind and navigate that becomes the mode. Yeah. It's interesting because you obviously have the hardware itself changing. You have a bunch of workloads changing. Obviously, we have much more long-running agents. We have more post-training going on, people running open-source models. How does that actually impact the way that the overall build-out of the clusters themselves? To what extent is it like, hey, workload's a workload, and we can run it?

No, the workload's a workload. exactly like you said it is slightly different but in a sense you need a high enough baseline to not be like constrained by that and then the rest is just your like optimization problem the way like you pack things the way you split things right like you don't need whole nodes for some of the workloads so you need to collocate workloads you need to have the right primitives you need to expose those through the platform, etc., etc., etc. But in a sense, like, differentiated workloads almost never, like, impose any changes on how you build on the lowest level. Yes, of course, like, nowadays with, like, these new architectures, like, what is it called? NVIDIA LPX, like, the outcome of the Grok deal, right?

this, like, adds to it. But in a sense, if you have experience running other generations, this is just another, like, 0.5 generation, right? Yes, this will be a little bit novel, and it's connected in a weird, like, compared to the previous one's way. And it kind of, like, there are some restrictions when you can run and not run on that. but efficiency gains. This is just a niche local addition to the platform to what you offer to the broader market. But in general, no. There is way less custom solutions that you would expect from that. The workload is a workload. Why didn't the hyperscalers move into this market faster or win this market more convincingly early on in this first chapter, at least? First of all, sometimes there is an ocean that they didn't even try.

There is a solid amount of business that hyperscalers get from out of this market. Huge deals, they are very competitive. Sometimes people say, oh, have you seen AWS prices? Oh, these are public prices. I assure you, for proper deals, they are doing proper prices and are very competitive. They're very aware of the market. I think in a sense, like, there are two angles to that. First, like, what I've mentioned before, a lot of these companies need to juggle, like, their internal consumption and what they can put for, like, for the broad market. And there is, like, we are in chip shortage world, right? So, like, if NVIDIA could have bought, like, could have produced, like, 10x the chips, probably this this whole market would be shaped differently because they were like Microsoft's and Google's of the world would both buy capacity

for themselves and for the broader market to operate this is part, like simply not enough capacity for their scale right, because there is not enough capacity for their scale, this is a relatively small business for them AWS revenue is what, like 100 billion or something I mean like a business for like 2, 3, 4 billion it's not insignificant pay margin but compared to what they conservative large scale established businesses it's not that easy either but your point is basically like they get a chip allocation that is like not as large as they'd want they have to use some of that a large amount of that for training themselves there's only so big that that inference business or, you know, that the cloud business can be given all that. And, yeah, it's interesting to figure out how to prioritize. Do you go after, like, the few folks with large contracts or do you try and get this, like, mid-market enterprise, you know,

up-and-coming segment? But I guess you're kind of doomed either way if you only have a certain number of chips. Yes, exactly. And then at the same time, this becomes a relatively small, like, part of your revenue portfolio and you're like, do I even need to allocate resources to that? And that is especially a pressing question because for the hyperscalers, it's not just a matter of sales. It's also a matter of changing the infrastructure that was built at scale for other kinds of workloads. The networking, the throughput performance, the storage performance, by the way, right, that I've mentioned earlier. So this requires, if you want to have it universally implemented through the whole stack, this is a massive investment on their part that needs to be justified by the size of a potential business. And it's not that there yet.

I think if we start getting out of this cheap shortage in, I don't know, two, three, five years maybe, we will definitely see this shift and they will pay more attention because this will become like increasing a large chunk of business and they will start investing they will like reorganize and redesign their like internals to be more accommodating to these demands etc etc etc and then with the whole sales power they'll go and try to win this market and the question is whether the novel players like us, for example, will be fast enough to grow and represent significant competition when they get there. I'm 100% sure that they will pay more attention in the future. Our bet is that we'll be fast enough to be in the league with them and fight as equals

rather than a small player trying to run in their trails. Makes sense. I mean, what do you think happens to, like, you mentioned there's, like, these, like, you know, 300 to 500, like, you know, neoclods, everyone trying to get into this market. Like, as you kind of, you know, you mentioned, you're kind of thinking the ship shortage comes to an end at some point. Like, as you zoom five years, ten years in the future, like, how many of these businesses are still around? Or, like, what is the end state of this market? Listen, I think the whole, like, AI consumption is going to grow, and it will be, like, universally used everywhere. Like, AI is everywhere. Like, right now, like, web services are everywhere. AI services will be everywhere. So, in a sense, like, you have AWS, right? But you also have Hetzner. You also have DigitalOcean. You also have Linode. You also have, like, tons and tons and tons of local, like, hosting providers that are, like, giving you virtual machines,

and I'm experimental, et cetera, et cetera, et cetera. So this market is going to be vast. Will all of the current, like, 300 to 500 companies survive the next five years? I'm not sure. But I am pretty sure that there will be a place for the very long tail of providers down the line. like some of this the current players will consolidate some of them will die off new players will be launched etc etc etc it's not a bad market right now, it's not a bad business to be in so it will constantly attract capital and attention and investments etc etc etc and I trust that in 5 years time there are still going to be hundreds of players in this market who found the investment and several hundred million dollars, and they can offer something to someone in the market. Same as it is with the small-scale clouds like DigitalOcean these days, right?

DigitalOcean did not die because AWS is growing. They're obviously doing a lot of things in-house, like sandboxes and web search and helping people with their models over time And you know I think one question a lot of people are asking is you know what parts of the AI infrastructure stack end up just being subsumed into these future AI hyperscalers and, like, live, you know, in just one combined offering? And then what are, like, the standalone pieces of AI infrastructure that will persist as companies? In a sense, I think it will, like, it will kind of mimic what's happened with, like, the legacy generation of clouds. previous generation of clouds, right? When your growth kind of dictates that you need to start addressing new sectors, new markets, etc. Because otherwise your growth is like bounded and you can't show your projections, you can't justify your valuation, etc., etc., etc. so there will be a natural push for the larger companies

to start like increasing their portfolio trying to get into new products into new services it doesn't mean that there will be a one size fits all solution because in a sense if you're concerned by that you will like we just need to stop doing whatever we are doing and just wait for for like OpenAI to rule the world and Tropic to rule the world. Because these companies will eventually, you do realize that OpenAI has models, runs inference, they have their own search that they don't exactly expose it, but if you're building the agent on their platform, it has access to search. They have all kinds of like sandboxing, local agents, cloud agents, etc., etc., etc. they both OpenAI and Entropic are now investing in forward deployed organizations and are gonna go

and do the work that was traditionally like I don't know Deloitte's or like Accenture's piece of pie right so they will be everywhere and I think this is a natural motion for the large company to start getting into a broader market but it doesn't mean this is they are going to be the only ones. So even nowadays, there is a place for independent object storage players all of a sudden, right? Or independent compute players or independent small-scale inference players, for that matter. Independent database players, right? Like, one runs, like, assume like Google has, like, BigQuery and whatever, all kinds of data and stuff. But then Snowflake kind of, like, appears from out of nowhere. AWS has, like, all kinds of, like, managed databases, large-scale databases, like Aurora, Redshift, etc., etc., etc.

But then MongoDB appears, and it's pretty successful. I can't say it's, like, super large-scale, and probably by nowadays standards will never be, because nowadays standards for large companies are, like, trillion-plus, right? But, like, two years ago, it was a very solid and very successful company. Or someone like ClickHouse appears for their cloud offering, right? Also, a massive managed database, an independent company that finds its niche and works it real well. So, I think, like, larger company, larger clouds will grow. They will have, like, products in every sector and will be pretty successful at selling those because of bundling, because of their sales power, et cetera, et cetera, et cetera. But there will always be a place for niche players. And because the market is huge, niche doesn't mean small. Niche doesn't mean small scale or low revenue or low valuation. This can be solid companies. Yeah. You mentioned OpenAI Anthropic having this kind of broad suite of offerings too.

I mean, in many ways, they're obviously building their own AI clouds. And I thought one thing that was interesting recently was, you know, So, obviously, a limitation of those AI clouds versus others is having just one model family to be able to call from. But I thought it was really interesting that OpenAI announced that partnership with Base 10. I'm curious what your take was on that. I mean, is that them just trying to get into this general hyperscalar game across models? In a sense, we see a lot of push and concerns about people claiming that OpenAI models or entropic models are too expensive. everyone is puzzled or surprised by the build they get at the end of the month so this might be like an answer to that well if you're like go go try open open models at the same time open models are still behind the closed source and closed models right however good those models are they're still again behind and giving people an option to try like lower scale with the ability to switch back

to the top frontier model might also be a good incentive like anyway I think like OpenAI is doing a very interesting move with that partnership and I like it really much because like this kind of shows flexibility right they're not set and so like they're exploring different options different things that they can or cannot do, like looking for different options to, like, but I think in a sense this all plays into one single thing. Like OpenAI wants, same as Entropic, they want to be one single, like, touch point. Like, they want to be one single thing for everything. Even that is, like, okay, but sometimes I want to use open source models within my codex-controlled environment, by the way. right? So, like, well, good for them. But if the company, the person, the user is still within your

ecosystem, you still have a lot of options to steer them, to upsell them, to introduce them to new services, et cetera, et cetera, et cetera. So if that, like, alleviates one of the concerns, so be it. Why not? Well, I always like to end these interviews with some, like, overly broad questions that I step into the very end. And so if you'll humor me, I guess first question is just, what's one thing that you've changed your mind on in AI in the last year? In the last year? A lot has happened in the last year. No, like the obvious answer would be software engineering. Like a lot of people forget, but a year ago, we lived in a totally different world. Yeah. It's like dog years in the AI world. way less capable in terms of the model capabilities and in terms of how good the modern harnesses and tools are it's like the AI coding because like a year ago it was total bullshit

right now it's a solid part of like your everyday tooling but for example if you're not talking about the industry trends or like talking about more like my personal thing, I've started relying more on the, like, AI-enabled, like, local agents and harnesses to afford a part of my cognitive load, to be honest. I mean, obviously, you run a really large technical organization, and I think everyone's trying to figure out, you know, I mean, obviously, the models only recently got really good on the coding side, and so everyone's trying to figure out what it means to, like, build teams in the, you know, given the new model capabilities and run an organization. I'm curious where you've settled and how you thought about that. Cautiously, conservatively. I'm very concerned that modern models and the attractive user experience that they provide, like model agents, modern models, et cetera, et cetera, et cetera,

is something that will lead us to issues. You see a slop being revealed in chats, in merge requests, all over. And I try to pace that and I try to be very observant and vigilant about that. So, for example, we don't restrict talking usage. People can use whatever they want. Our budget is quite significant. We don't impose per-person limitations or per-department limitations. But at the same point, we try to look at the usage, figure out what it is used for, look at the most ineffective ways people use it. we constructed a set of policies that doubles down on the idea that whatever you've done with the agent

it's still your result it's your responsibility it's your pull request it's your code it's you who's on call it's you who's going to wake up in the middle of the night if something goes awry etc. etc. etc. and if something was created by a completely autonomous agent It is marked like that. Completely autonomous agents will not touch, like, the core services for a while. Because, like, this is, like, out of bounds. Like, sometimes people are surprised that, like, I work in a company that kind of participates in this whole AI revolution. And I'm very conservative. But at the same time, like, I've seen how this sausage is made. And I, like, I'll be cautious. What's the biggest mistake you feel like you've made in building Nebius so far? Obviously, a lot's gone right, but I'm curious, you know, something that in retrospect you wish you guys had done differently. It might sound funny these days, but we underestimated the scale.

Like, and it's not about the growth of the company. We were pretty optimistic from the very beginning. But I personally, that was like my strong bias for the first significant part. I underestimated the scale that a single client operates at nowadays. Because like five years ago, if you're talking about a 30 megawatt data center, it was a serious facility. It hosted like hundreds of thousands of servers, like run gloriously large workloads. By now, 30 megawatts sometimes is not enough for a single client, which I'm still getting used to. the storage I've mentioned, right? Like four years ago, when you were talking about network-attached storage, you were talking about like hundreds of megabytes, maybe gigabytes per second of throughput. Right now, some of our clients routinely fetch, I don't know, seven hundreds of gigabytes per second

from a single shared file system. I'm still, I underestimated it in the beginning, and I'm still getting used to this scale. Like, I'm an infrastructure person. I've worked and I've seen these numbers throughout my career multiple times. And all of a sudden, you add a couple of zeros to the end and call it a new norm. Yeah. Still puzzling. That's crazy. And I guess lastly, like, there's obviously a lot of, you know, unsettled questions in the AI field today that, you know, lots of people are trying to figure out where the world goes. I'm curious what you're paying attention to and I guess with a lens toward if it goes one way or another it would really change the strategy for Nebius as a whole. What are you most paying attention to on these unsettled questions of where the world's going? Well, I stopped caring about the new models being larger or smarter.

I just accepted that it's a given. I'm kind of trying to look out for new novel architectures and approaches but I don't see any as of today I hope somewhere deep down in a stealth lab something new is cooking but for now I frankly don't see anything that is a thing that I would be would place even a small bet on but I'm more and more interested in the whole outcome optimization part on the agent side because we are used to talking about models and size of models and families of models etc. but now that we've shifted to this multi-turn, long-running agents that we are talking not about requests but rather outcomes companies are talking about bidding per outcome right I'm very interested in how that part will shape up and we have like a small research group

internally within EBIOS that's working on this problem actually how is like how do you increase the chances of getting to that outcome not by increasing the model or like changing the model that is of course part of the discussion but I believe there is a lot of optimization on the part where you introduce more sophisticated or maybe more simplified tools you introduce the harnesses that start to evolve according to a particular task. And this kind of gets us back to significant CPU consumption because a lot of modern AI workloads actually go back to becoming code. It's codified, it's very formal. You see quality gates, you see sophisticated tools, et cetera, et cetera, et cetera. And sometimes your agent works for minutes and minutes minutes without even issuing a back-end request to an LLM because the workloads start to sit on this part. And this is the part that really interests me because I think a lot of future AI

adoption lies in how reliable your agentic scenarios are and how reliably you reach those outcomes. And I think the whole optimization loop outside the model is an important part of that. 100%. Well, this has been an awesome conversation. Really appreciate you coming on the show. And yeah, thanks so much for coming on. Thank you for having me. I'm Jacob Efron, and this has been Unsupervised Learning, a podcast where I get to talk to the smartest people in AI and ask them tons of questions about what's happening with models and what it means for businesses in the world. As I hope is clear, I have a ton of fun doing this. It's a nights and weekends project, in addition to my day job as an investor at Redpoint. But our Our ability to get these incredible guests on really comes from folks like you subscribing to the podcast, sharing it with friends. It's really what ultimately makes this whole thing work. And so please consider doing that. And thank you so much for your support and listening. We'll see you next episode.