大規模なAIコーディングコストの管理：Databricks共同創業者に聞く

Over the past couple of years, companies raced to get their employees using AI, often encouraging them to lean on it as much as possible. However, as AI coding tools spread through engineering teams, many companies watched their AI bills explode. That kind of runaway cost growth can threaten a business. So the new challenge is capturing the productivity gains from AI without the spending spiraling out of control. Databricks is a data and AI company that hit this problem early, running AI tools across thousands of its own engineers. Rather than clamp down on usage, the team built a set of practical ways to manage the cost, from taking advantage of steadily falling model prices to automatically routing each request to the cheapest model that can still do the job well. Patrick Wendell is a co-founder of Databricks, where he leads engineering teams and has driven much of the company's internal AI adoption. He recently wrote a viral blog post on managing AI coding costs that reached the top of Hacker News.

In this episode, Patrick joins Kevin Ball to discuss why AI costs suddenly exploded, the main levers companies have to bring them under control, the trade-offs between managing costs on the server versus in the developers' tools, and why the next big challenge is giving AI the right context about how a company actually works. Kevin Ball, or KBall, is the Vice President of Engineering at Mento and an independent coach for engineers and engineering leaders. He co-founded and served as CTO for two companies, founded the San Diego JavaScript Meetup, and organizes the AI in Action discussion group through latent space. Check out the show notes to follow KBall on Twitter or LinkedIn, or visit his website, kball.llc. Patrick, welcome to the show. Thank you so much for having me. I'm excited to be here.

Yeah, I am excited to dig in. Well, let's start with a little bit of your background. So can you tell our audience a little bit about your background, how you ended up where you are today and in this conversation? Yeah, it sounds great. So my career history is pretty simple because I've only really had one job in my career. And that job is that I was in grad school at UC Berkeley in 2013. And a few of us that were working on, we were working at the intersection of machine learning, then machine learning, which was not really a lot of deep learning yet, but it was sort of scaling up classic ML. And we were looking at those problems. And then they were very tightly coupled with big data problems. So dealing with really large data sets, almost a little small progenitor to like the scaling laws that are happening now. It was also true then that you saw if you just kept scaling, a lot of stuff started to work better. So we were looking at research problems in that area. And then we left to start a company that's called Databricks that's still around. And it was initially around some open source technology we had built around these problems, the kind of large scale data and machine learning.

And then at Databricks, I wear a founder hat. I'm one of the founders, so we have many different jobs. You never really just have one job as a founder. But my day job is working quite a lot in engineering and helping support several of our engineering teams. And in that capacity, I've also been focused a lot in the last few years on our own use of AI in the company. And for a company our size, I think we were pretty early in terms of being quite ambitious about adopting AI internally. And I've been the one most internally responsible for a lot of those efforts working with our team. And so I think I have quite a perspective to share around. We have thousands of engineers and we're trying to give everyone AI tools and get them to be super productive. And at the same time, we're worried about security costs, our own intellectual property, being careful around that. And I think it's an area a lot of people are interested in. And I'm happy to share. I don't think we have all the answers, but we have learned quite a bit from what we've done. So I'm happy to share that here with you. Yeah, absolutely. Well, and I feel like we could do multiple episodes.

I'd love at some point to just dive into the history of Spark and all of that, which really honestly changed the direction of data analysis as you enabled streaming data analysis and all of that stuff. But let's focus on AI for today. It is the topic of the day. You recently wrote this blog post about managing costs. And I feel like this is a transition we've gone through over the last year where it was like, first, everybody's like, adoption, adoption, adoption, throw everything at the wall. And then suddenly people were saying, oh, my gosh, this is expensive. How do we rein things in? So can you give us the TLDR of how that's gone at Databricks? Totally. Yeah, and the reason I wrote that post, I do encourage people to check it out. You can just Google Databricks AI costs and you'll find it. Is that I had talked to so many of our own customers, who are also many of them tech companies that are trying to roll out AI, and they were saying, hey, what have you tried? What works? What doesn't work? And I also have talked to a few other peer companies. Actually, there's some credits in that post. We had talked to Stripe, Uber, I think Ramp, and a few other. we just trade notes with other tech companies and see what everyone's doing. And remarkably,

folks are looking at very similar problems and solutions. So that was pretty cool. So I wrote that post to kind of dump everything out of my brain so that at least it's one data point people can look at, since I think many of us are facing similar problems. But the origin of that problem is something you just said, which is that there was this period, and we got there actually pretty early, but it was around 2023, 2024, maybe beginning of 2025, where companies just were worried they couldn't get people to use enough AI in the company. So especially bigger companies. And it was all about pushing employees, try this, try that, incorporate this into your workflow. And something changed roughly, I think, around Opus 4.5. It was kind of when the AI models started to get really good at coding. And it was started with Opus 4.5, and then many others came, combined with these really high-quality harnesses like Kodaks or Cloud Code or Cursor or there's others as well, suddenly the problem wasn't getting people to use them. The problem was that people were using them a lot. And around the same time, all of the foundation model have switched to consumption prices.

So instead of like a seat where you know, okay, I'm going to pay $100 per month for developer, it was a consumption, you pay as you go kind of thing. And then those two combined to at least at Databricks start to encounter what looked like exponential growth in costs. And exponential growth is like not a super sustainable, like in any cost in a company, unless you're exactly getting exponential increase in revenue that's commensurate, and exponential growth in cost is just eventually going to be a huge problem. So we started looking at this when we were modeling the costs and seeing, huh, that's funny. It's like doubling in this two-month period, and it's doubling again in the next two-month period, and we have thousands of developers so this can start to accumulate really fast and i would say oh i don't know maybe beginning of 2026 around that time or late 2025 is when we realized and again we wanted to have an attitude of people use tools we want you to use tools and be productive but they're just some law of physics where when costs start to accumulate at a certain rate it's going to eventually any company that doesn't address it it will eventually bankrupt that

company just law of physics well and i think that's why they went to this usage-based pricing right? Because the model providers and then cursor and the harness providers are like, oh, we're paying per use because we're fundamentally shuttling electricity through GPUs, turning it into intelligence in some form. That costs money. And as the models got good, people's usage of them skyrocketed. And so they said, oh, we can pass this along. They can hook their revenue into their costs, but that's hard for the rest of us. I mean, I'm not on the inside. We do partner with a lot of those companies. I actually think many of them wanted to have a fixed price thing. But if they have a fixed price thing, then they're in the business of managing how much capacity and goodness you can get out of that fixed price. I actually think some of those companies got a bit ahead of their skis where they were offering fixed price deals. And then when you did the math of how much token capacity you were getting, the foundation lab was losing a lot of money on certain people. At least I am one of those people because I would look at it. Oh, 100%. Me too. Yeah. So I think they themselves had a sustainability challenge. And then understandably they're like okay well let's switch to consumption pricing where then our customers

which is like a company like databricks or others can then internalize okay where is the roi for us like how much are we willing to spend compared to how much we're getting and that has a lot of benefits to the ecosystem but it has one problem which is it puts the onus on you the buyer to figure out how to do cost management and incidentally a lot of the solutions we looked at at databricks we ended up building also products that our own customers we productized basically a lot of the stuff we built. So others are welcome to use those products that Databricks now offers. It's our AI gateway. But before we productized it, we just looked at how to actually solve the problem. And it passes the responsibility for ROI to you as the user. And so we thought a lot about then how do you as the user handle that responsibility, basically? Yeah. So in your blog post, you identified a few different levers that are available. Do you want to take us through that high level and then we can dive into each one subsequently? Yeah, definitely. The real world is obviously very complicated, but I had to make this simple in the way that I thought people could

understand. And if you really simplify things, at least as it came to like the short-term mitigations that were available to us, we looked at four things. Number one is that the single biggest lever that I've seen, and this has actually become even more true since we wrote this post, is that there's this beautiful thing happening of constant price reductions in models, meaning to say, now, even at one moment in time, there's models that are more expensive and have better performance. There's models that are less expensive and have worse performance. There's always that, I often call it like a Pareto curve for those who are familiar with economics, but it means to say that there's this continuous trade-off of quality versus cost. But nothing is more amazing than when that whole curve shifts in one direction, meaning to say a new class of models have come out that you could view it either way. Either you fix the complexity of what you're doing and then that got a lot cheaper, or if you have a fixed budget, you can do a lot more. But it means to say that there's a deflationary thing where costs go down. And the way that that happens in the AI space is that some new model will come out that expands the frontier,

would be the economic term. But it dominates in some ways the models that were there before in terms of usually what people target is, hey, it can do roughly what we could do before and it's cheaper. and when you're driving a massive token volume even getting 20 off a huge cost is a really meaningful thing but there's a lot of secondary things that we can get into when we go through the detailed list around how do you put yourself in a position where you can take advantage of that do you want to go into that now or you want me to finish i was getting too excited about this so i no no this is good and this is it's a really interesting one because classically you have this with a new technology where it's pushing the boundaries of something new and people keep chasing after that. And I keep expecting us to hit a point where we saturate our need, right? Where it's like, oh, you know what? I mean, in some ways I thought when we hit Sonnet 3.5, even before we got to like Opus 4.5, Sonnet 3.5 was the first model I played with that could write solid code reliably. And I was still driving it by hand. I didn't have the gen-stick harnesses we have today, but I was like, holy smokes, models could get no better. And this would change the

world of software engineering and models keep getting better. And Sonnet 3.5, you can't even get anymore. But if you could, it'd be like, so cheap, it's unbelievable. And I thought, okay, well, are we going to keep going? But it feels like we keep chasing after that frontier, right? People keep adopting, oh, why would I use that model? I could use Opus 5 now, or what have you. So like, where is that efficiency frontier? Why have we not saturated that for at least coding? Yeah, I mean, it's a great question. Like, there's sort of different things to tease out in your question. One question is like, how come they can keep making these models better? Isn't it getting harder to make it better? And the truth is they're just getting better at training them. Like, first of all, software engineering is a highly verifiable domain, which really helps because they can, you know, throw RL at it and they can get into these self-improvement loops. They also continue to throw more compute at the problem. Data is, I think, pretty saturated from what I understand, although I'm not in those companies, so I wouldn't consider that authoritative. but it's just between throwing more compute, clever techniques, optimizing heavily the inference and the training stack,

companies are going to look at building their own hardware, they just keep squeezing and squeezing more out of this quality of model per dollar. And there's almost some Moore's law equivalent thing happening here that I don't know if it's been measured, but the quality cost ratio just seems to keep improving. And that's why it's so beneficial to be having agility to switch between models. But then I think a second part of your question was, by the way, I realize we're doing like depth first search because technically I'm supposed to be going through the four ways to save money. But the second part, you know, following interest, I think, let's go depth first and then we can pop back up the stack. OK, cool. So the second part of your question is about or maybe what I heard maybe was, aren't we as developers at some point, aren't they kind of good enough for the things we're trying to do? And there I think what's sort of remarkable is that developers are just able to hand off more and more complex tasks. So in some sense, they were good enough for the old stuff, but we were thinking of new stuff to give them. And that also seems to be like there's this famous quote. I don't know if it was Bill Gates or someone was like, why would anyone ever need more than 1000 kilobytes of RAM or something?

It was this famous. Yeah. Yeah. Do you know who said that? I don't even remember. That is a quote. Something like that is a real quote. and you know you have an old like IBM of who would need more than five computers in the world like we keep making this mistake like we keep making this mistake right and when you look at other analogs like how much bandwidth does your mobile phone have now compared to 10 years ago it's been like an exponential increase and people just find new stuff to do like facetiming with another human like with near instant super high-def video communication we never would have even thought of that is a thing you would do on phones 10 years ago or 15 years ago. But when capabilities improve and prices go down, people just find new ways to use them. Yeah, that's definitely true. I do wonder, and maybe this pops us up a second, goes to the next one, right? Like some tasks don't need all of that horsepower. And so are there ways that we can decompose the types of work we're doing so that we can keep those costs under control a little more? Yeah, so definitely,

like as an example, just to move to a different context, before these enterprise coding stuff started really working, the big hit in AI was personal chat stuff. And that was itself a very disruptive and innovative product form factor was like, oh, I can just pull up my phone at any time and I can ask any question I want. And it turned out that although the models keep getting better for that, the set of things people kind of want is sort of saturated. Like even very low cost models can answer the vast majority of individual questions. Because most questions are like, where can I get a hot dog in my neighborhood? The frontier labs are really excited about proving new math, eros problems and stuff. But like most people don ask those questions day to day So in that capacity in that form factor my view is there has been a bit of saturation where I actually think the main thing happening now is that costs continue to go down rather than that there lots of higher quality models that matter for the types of questions people are asking. So then the second big wave in Gen AI has been this coding and other enterprise

workflows. There, I don't think, as I said, I don't think we've hit saturation. I think there's far more difficult coding stuff that they still can't totally handle, but especially when it comes to architecture or high-level system design. But it is absolutely true that for, like, routine coding tasks, hey, go refactor this thing, please add this simple feature to a JavaScript code base, they're much lower-cost models. Even the last generation of models will perform identically to a very, very expensive frontier model. So there is this cost gain from sort of fitting the model to the problem. And there's different ways of doing that fitting. One way is the user themselves can do fitting. you can ask the user. Now, in a small company where people are policing their own costs, maybe that works. In a big company like Databricks, where no one is themselves paying the bill, understandably, it's a little hard to convince people to constantly switch to crappier models. My incentive as an individual developer is just turn the dial to 10 and get out of my way. So we do a little bit of downshifting on purpose or how you can control this when you're doing

things at scale is the default model maybe could be not the most expensive model. So if you want to shift up to it, you can. We also prune the set of models, meaning to say even the frontier labs continue to release updates to their models that are not better than the old models. Like, if you know how they train these models, it's like more art than science. I mean, they put a bun in the oven and they're kind of seeing, does it look like it's burning on this side? But they're not quite sure. They're doing evals as it's being trained, but they don't quite know what they're getting. Well, and they're very spiky, right? So like Sol is phenomenal for mathematics, apparently. But like my engineers are like, I don't want this for coding. It overcomplicates everything. And by the way, like we had one of the Opus, like I think many people were using Opus 4.6. Opus 4.7 came out and we deployed it. And well, actually, I think first we did some evals. We did some benchmarks. And on our own benchmarks, we're like, huh, this seems like it's more expensive and lower quality. Like it's the inverse of the Pareto. It's like both things got worse. And then we like tried it with the pilot group. By the way, I don't want to get in trouble.

I don't know if it's exactly 4.7. I think it was. I mentioned it in the blog post. And it since got better with the next model. But there was one that came out that was like worse along both dimensions we cared about. And so there it was actually an easier decision. It was like, oh, we're going to skip that one. Like we just didn't make it available to our users. And when our users asked, then we just explained, hey, it's not so great. Then there's other models where like, okay, that model is good in certain circumstances or certain complexity of tasks, but it needs to be this fitting thing that we were talking about before. And there, the levers you have are like, ask people to do it on their own. But increasingly what we're using is like dynamic routing, basically, where we try to decide what model to use. And there's a few different flavors of that. Sometimes it's done at the task level. So we might for a whole session, for instance, put someone on a particular model or there's certain interfaces where someone just submits the unit of work. It's not an interactive session. so then we can decide, like, based on that thing they submitted, what model do we want to use? There's also much finer-grained routing, where even request-to-request you might divide across.

This looks like traditional load balancing in a server system. That one is trickier to get right. It's more powerful in some ways, because the end user is even further abstract. They don't even know what model is really being used. It's trickier in that, although we can model inference as, like, a stateless problem, there actually is some sleep in the server. I was going to say, like, the key value cache feels like it's going to be a tricky thing to optimize. The cache is really important to economic performance or to economic cache reads are, like, I don't know, 10 times less or something than non-cache reads or something of that nature. So you have to be very deliberate about routing. When is it worth it to develop a new cache to get a much better fit for the economics for this particular problem? And we already have some efforts in this area, and our gateway supports it. But I think this is a very rich area of research, to be honest, is like, how do you decide when to switch based on a variety of how much you can save and how much will it degrade the performance for the end user, if at all? And how much am I losing that cash benefit? You need to keep all these into account. Yeah, it's fascinating. Well, and how are you classifying like what types of tasks? Is it going to be good enough anyway, even if the economics

work out? Yeah, so that's also a hard problem. Like, first of all, even humans aren't good at that problem. This is the number one thing that people say when we're like, hey, choose the cheaper model and they're like, I don't know which model is good enough for this. That's why it's really tempting to just stay on the most expensive model all the time. So the answer to your question is, we build models to do that classification. So that classification of which model is good enough for this problem is itself an AI problem and is itself amenable to training models that are classifiers that are also LLMs, basically. They're a very different type. But I do think it is in principle learnable, not with 100% confidence, but it doesn't need to be 100% confidence. It just needs to be a high enough hit rate. And it is in principle learnable. Well, especially in software development, because as you highlight, it's very validatable in terms of outcomes in a lot of ways. Exactly. And this validation is a little trickier because it's like, oh, was that model in practice good enough for this task? And that can sometimes be measured just completely objectively, like, oh, the pass rate on some tests, or we ask a much smarter model to evaluate the solution.

but there is a little bit of like eyeballing that still happens here so in my view one of the things we focus on at Databricks a lot is like we're not in the business of ourselves building these massive massive foundation models but we are in the business of how the models get used in real companies and I think routing and coming up with models that are good at routing is a very important problem and it's one that we're looking at and have some ongoing research in but I think I believe Most companies should take AI into their own hands and have different models they're choosing from and different vendors they're working with. And part of that is being able to know how you can switch in between them, basically. For sure. This portion of Software Engineering Daily is brought to you by Origin. Your coding agents can launch each other. Theirs did. An engineer asked Codex to pull data from staging. Codex had no connection there, but CloudCode did. So Codex launched CloudCode. When Cloud Code said it lacked permission, Codex relaunched it with the dangerously skipped permissions flag.

Nobody was asked. A permission check doesn't mean much if another agent can switch it off. Your logs might show a process that started with a flag. Their trace showed the whole job and why Codex did it. Origin finds the agents on your company's endpoints and reconstructs their work from request to outcome on one timeline. Origin is endpoint AI observability. Read what they found in their own traces at originhq.com. slash S-E-D. That's O-R-I-G-I-N-H-Q dot com slash S-E-D. Think about your mobile app source code. Once it hits the app store, it's out in the wild. And without the right protection, decompiling is easy for malicious actors looking to steal your IP or tamper with your software. That's where GuardSquare comes in. GuardSquare provides the highest level of mobile app security for Android and iOS applications and SDKs. Their advanced tools integrate seamlessly into your CICD pipeline. We're talking polymorphic, multi-layered code hardening techniques

and automated runtime application self-protection paired with mobile application security testing and real-time threat monitoring to deliver the highest level of mobile app security without compromise. Don't leave your hard work exposed. Secure your mobile applications today. Go to guardsquare.com to learn more. Sponsored by DeepGram, the leader in voice AI, so you can use voices like mine. So staying on routing for a second, I feel like there's a couple different types of routing that you have alluded to here that might be worth digging one level deeper. So you've talked about gateway-level routing, which is like, here we have a particular inference coming in. We can make a decision using a classifier or something else and route it to a particular model. Then there's harness level decision making where you're actually like something is deciding, oh, I want to explicitly route this in this way. That something might still be an LLM or sub deterministic logic, but it's in the harness layer that knows more of the surrounding context rather than just here's my current inference request. From what I could tell in your blog post, you're looking at both of those.

Do you apply them in different ways? When are they relevant? Like, how does that look? Yeah, so we use both. and if you want to ask me in two years where we'll end up, will almost everything be a fat client or will there be proxies? I don't actually know. It's going to depend on how well this stuff works and various other factors, but we use both. The nice thing about the proxy or gateway-based routing is that your client is out of the loop and doesn't need to even be routing aware in some sense. It just seems to be the case that this proxy speaks the same inference API to that client and you can just swap it in with no assumptions about your client. You may even have an older client that doesn't know how to do routing or whatever. Also, it depends on your control model. But like in a company, your end users are mostly trusted. But for things like cost controls, you might want to sort of have ability to coerce activity. And therefore, doing that in a server rather than the client is a bit easier because it avoids creative employees realizing that they can tweak settings on their client

and get around rules. And for instance, things like entirely, let's say, removing access to certain models because they're not efficient models and we don't even want them in our set. Or maybe some of our users accidentally run like runaway spend. They have a loop going. They didn't even realize. They left it on overnight. It's not doing anything useful. We have like a thing that will torpedo that. And it lets them reset it if it was truly intentional. So those are much easier to do on the server side. It's transparent to the client and sometimes the client just doesn't get to make decisions. which is sometimes a feature. However, there's a lot to be said about doing this on the client side too. One is, as you said, like the client knows much more, like it may know much more about what's going on. It has tighter access to the whole history of the session, for instance. It may have other context on what type of job it's being used for. So I do think potentially you'll see a lot of intelligence happening at the client side as well. There's also this further evolution of client routing, which is like more like dispatching between harnesses.

And this gets into a sort of slightly nuanced thing, which is that there's increasingly a coupling between the foundation model and the harness, the choice of harness. And when I say one model is better than another model or so forth, I'm sort of with simplifying a fact of the world, which is that model harness pairs are really what you need to look at. So a particular model with a particular harness may be very expensive or very cheap and so on and so forth. And another class of things you can do on the client side that you sort of by definition can't do on the server side is you could even dispatch between different harnesses. Like there's a project, an open source project we've released called Omnigent at Databricks. And it's a multi-harness dispatcher, basically. And the idea is you do your coding against a fairly uniform interface. Behind the scenes, it can leverage other harnesses. Sometimes it's even beneficial to like have an adversarial relationship between your harnesses. One of them writes code and one of them reviews code. And it tends to be the case that when you review code from a different model than wrote

the code, you get way better feedback. Models are very bad at reviewing their own output. They give their own output high marks. And it's not surprising because it's all a matter of what statistical distribution they come from. So for even other reasons than just costs, there's nice benefits to having these like multiple harnesses that are playing with each other on the client side. But it's not even just client versus service side. it's also like, as I said, it's like server side versus a client where maybe within one harness, you're routing to different backend models. And then there's even this next layer up of a meta harness that's routing amongst harnesses. And I don't know in the very end of the day, if you fast forward where most of the logic will be, but right now we're employing different types of logic in all three of those places. Yeah, no, it is fascinating. I remember, I think it was the early GPT-5 series. Folks were trying it in the context of cursor and seeing incredible results and trying it in a different harness and finding really running into trouble. And I've recently been on Pi and comparing that to different things. Like the harness makes a larger impact than we

like to think. Totally. And for instance, I think Databricks also regularly, we do a lot of evals of our own because we have this problem of like, you know, I told you some models come out, they're actually not better, it turns out. Well, the way we used to find that out is we would roll out the models for all these developers, okay, like thousands of developers. Then the complaints would start to come in. And by the way, everyone always has some complaints about every model. So you'd have to wait until you have enough signal to noise that's like, okay, this actually is really a problem. And the cost, we get kind of an A-B test that's pretty quick because like you'll just see costs go up or down based on the cohorts that move. But the quality is hard was initially quite hard for us to gauge. And the downside was that we would waste, like, weeks with a crappy model. And then eventually someone said, hey, come on, look at all these people complaining. This model really does stink. So then we would roll it back, and we would remove it from our working set, basically. But the way to fix that, we found, was, like, you just need to have good evaluations you yourselves can run. And so we ended up building our own evaluation suite based on our own code, basically.

Like, it literally asked these models to go contribute code to Databricks, the code base. We obviously have a point in time snapshot of our code that's before we went and actually added all those features. And then we run these offline evals, and we want to get to the point where, like, within an hour of a new model coming out, we know if this one is sort of a candidate to try internally. And that offline benchmark rules out a surprisingly large number. There's so much. I mean, every week now we're running that multiple times, between open source models and proprietary models. Like, every week we're running it, and we want to get down to a set of three, four, five models we're using. that define that Pareto frontier. And so we needed to get the speed of evaluating whether models were even worth considering down. And so we ended up making a benchmark. We also published the results to that regularly, although I don't want people to over-extrapolate. That's based on our code base, so there could be some fitting going on. But one thing we found when doing that benchmark is the model plus harness pair needs to be benchmarked. And you'll see even drastically different results.

By the way, Pi has suddenly become very popular at Databricks also. So I guess you guys are using that. Like there's this constellation of open ones that are getting pretty good. I mean, I'll just say Pi is incredibly hackable, so you can customize it to yourself and build extensions, and it knows about its own extension system. But that compounds your harness plus model problem because now it's model plus harness plus the set of extensions or other skills that you've incorporated, et cetera. Yeah, so we have to cut it off somewhere, and I sure that there probably some minor regressions where it actually gets worse for someone because we didn test their particular configuration But we do quite a bit of this offline testing mostly to save the human time of it just become impractical to test each one with humans There too many of them For sure. I want to dig in on an implication or a thread. You mentioned these are on Databricks code, and so it's different than other things. And one of the things that I've definitely seen is code organization and its style has a dramatic impact on how effective LLMs are writing that code and how cost effective it is. I'm curious, like, is that something that you're able to see

in data in terms of like, oh, if we use a service oriented architecture or this language tends to be more efficient in terms of things or stuff like that, where you can have even that meta level of feedback of, hey, if we write services primarily in, I don't know, I'm just going to pick Go as an example, that tends to be much more cost effective than writing a monolith in Python or something like that? Yeah, so that's a great question. We don't have systematic research benchmarks that look at that problem. And I think the problem, if I understand it, is are there ways you can write your code that make it more amenable to being highly efficiently modified by models? And the only answers I have for that are of the form of my intuitions, which I do have some. Like, certainly the distribution of code written in different programming languages on the internet is like almost a power law. I think there's some extremely popular programming languages, and then there's a bunch of very obscure ones. Databricks started, we have a meaningful portion of our codebases written in Scala, which is not on the very extreme end. We've actually, over time, more and more moved

those pieces of code to Java. But we started Spark in 2013, or 2010, when Scala was super exciting, and I still think it's a cool language. But some of the useful features of that have actually been moved into Java at this point. So a lot of our go-forward work is in Java, and then also we have a huge amount of Python and Rust and C++ in certain areas. But I can tell you LMs are not amazing at Scala, but that's not a useful fact for most of your audience. I think the general principle is more useful, maybe that the biggest and most popular languages just tend to be the best because they have the most training data on those languages. Python is probably the single best. For instance, we did have an episode in one of our own products where we have certain internal parts of Databricks where you can call it via Python or via SQL. Behind the scenes, it gets translated into exactly the same representation. We just have two front ends for it. And wouldn't you know it, when we switch the code-generated portion from SQL, this is not ever exposed to our own users, by the way. It's just an internal representation. When we switch that from SQL to Python, everything got way better,

just because the AI is better at generating Python than at generating SQL. And I would say SQL is a fairly popular language, too. So I think Python dominates in terms of, like, quality of generated code, but it's getting so good now, they're probably all pretty good. And then in the teams that have started new code bases where they're able to also innovate on how the code base looks, I have seen them focus much more on interfaces behind which they could build things that were highly verifiable. And I realize that's like a very abstract statement. But in some sense, the key to being able to trust the LLMs more was make the scope of their changes smaller and more verifiable. In a big legacy code base, that's difficult. in a brand new code base, you can think about modular design principles where, you know, you have, by the way, this is probably what would have anyways been good for humans. This is one of the fun things here is like what's good for LLM coding is actually the stuff we probably should have been doing anyway. Yeah, but the ROI from doing it for LLMs is even much, much higher. So you'll see some very like flat types of organizations where it's like, okay,

you know, I actually built an interface and there's 50 different implementations of this interface because each one was able to be like super easily verified when for humans, you might have imagined that same thing would have been written with like a slightly more monolithic structure. So this is in the territory of my intuitive takes. We don't have robust research findings in this area. Yeah. I feel like this is one of the fascinating things to me of like, what are the characteristics and some of the characteristics that make something like Scala, for example, good for humans may not actually be there for Olam. So one thing we didn't talk about, but a thing that's true of both Java and like Python, I've seen it in Go, I think it's true in TypeScript, is it's very easy to trace where particular logic is coming from. Java, if you go heavily into dynamic dispatch, maybe not. But I think Go is a great example where there's a very linear dependency path where the LLM knows exactly where to pull the context in to understand what something's doing. And it doesn't have to like dive through a deep dependency train or load a whole bunch of context to be able to do it. And I think that sort of linearization.

I do agree. Like, Stella code has these implicit things where it's kind of like duct typing. It will, like, become a thing if it meets the signature of that thing. But I actually suspect what totally dominates all these factors is just how many lines of code on the Internet are in this language. Fair enough, yeah. I mean, that's just how these LLMs learn. Like, they're pretty good at learning almost anything. They just need, they just don't learn very efficiently compared to humans. They just need to see a lot, a lot, a lot of examples. And my hunch is that that would be the thing that differentiates the most. But there's certainly languages where there's more implicit stuff happening, and that probably also makes it harder for the models to learn. Let's dive into some of the other things that you've kind of learned in terms of rolling out AI at Databricks. Because I feel like every company right now is trying to figure out how to effectively roll out AI. And as you highlighted, there was this like top-down push. And, you know, I think you were generous, honestly, in saying enterprises were going through that, you know, up until 2025. I keep hearing of enterprises that are still in this like top-down, how do I get people to use the tools space?

But like what have been the important challenges and what's been really effective in terms of, you know, getting these things to work well for your org? Yeah, there's many facets to it. Like the types of things that we have thought about is like, first of all, how many different AI tools do I even give my employees? And, like, can they use any tool they want, or I should just give them a few, or should I choose two? And then do I give the coding tools to, like, the non-coding people? Because, you know, even my marketing team wants access to Codex or something, and they don't know how to code, but it can build these apps for them that they're, like, super excited about. Also, my inference data, I'm attaching these tools to all my most sensitive data, And then that inference is getting like logged by some third party company. So now they have all my sensitive. So there's like so much around not just cost, but like which tools you deploy, how to deal with data security. And like, where is your data going by virtue of these tools? Who gets what tools do you do? You go all in with one vendor or do you sort of have multiple vendors?

And I can share maybe some insights we've had, but I don't think we have all the answers. But we felt strongly from the beginning that we wanted to be a multi-model, multi-tool company, at least for R&D, which is where I focus a lot of my time. And so we really did not want any scenario where – by the way, it doesn't necessarily mean that an engineer can find some AI startup that's existed for one day and buy their tool and give it all of our sensitive data. That's not what that means. What it means is that we have a set of pretty popular tools. We let engineers vote with their feet and use the tool they want. And we want to provide internal choice around tools and models. By the way, Databricks, you know, our product is really for data users. So it's not a general coding tool. But we also, in our own product, have capabilities as well that are consumption-based. And, you know, you're also seeing also a lot of embedded AIs now inside of different products, including ours. And our thought was, like, let our internal users choose what's best.

And what's best seems to be changing every month. So that's one reason to do this. But where we insist on uniformity is we do have a narrow waste in the company. And that narrow waste is this thing that we built internally called our AI gateway. So basically, all tokens go through that gateway. So we can see every inference that's happening from any random tool that's in the company. All data access to internal systems also goes through that gateway. So it acts as a proxy, basically, to, you know, an MCP proxy, effectively. So you want to access Google Drive, you want to access Salesforce, you want to access anything. It's going through that proxy where we're doing authorization. We actually log everything. And also we insisted on zero retention for the people providing us the models. So we don't want, you know, the whole thing you do with these AIs is you give your most important data so we can help you with your most important problems. So the inference request that goes to the model foundation provider includes all of your extremely sensitive data. you might have customer names or PII or credit cards. You know, we're a serious big enterprise.

So the rules we came up with were like the end user tool choice we want some flexibility on. We want to have a set of models that we can always be adopting the newest ones and making them easily used by our internal users. We need to have some sanity. And the way we do that is we run everything through this thing we call an AI gateway. They're sometimes called LM gateways. But now they're expanding in scope. So this is kind of an emerging architecture, I think. For any company that's big enough, you're going to want a central control plane for this stuff. And we ended up productizing ours. It's called Unity AI Gateway. There's other companies that also offer them. But whether or not you use the Databricks one, I think at scale this is kind of necessary. And if nothing else, it's necessary for being able to quickly adopt cheaper and better models. You know, the truth is the Pareto curve, if you look at the Pareto curve today, I don't even have the most recent one, but we run the evals every week. On there is probably one open source model, one model from Anthropic, you know, one model from OpenAI. And occasionally now even like the there's the meta ones are starting to get good.

The Grok ones are starting to get good. So we're in this period where there's more and more supply and it's a huge cost. So just basic economic theory dictates like maintain option and suppliers, you know. Absolutely. Yeah. And so I think for us, that was a principle that really mattered. It was like tool choice, not amongst any random tool you can put in your credit card, but we'll try to basically provide the popular tools to engineers and let them choose. Even day to day they can choose. But we do insist on like the access needs to go through our gateway so we can control and observe costs and do model routing and stuff. And any access to our internal data needs to go through our proxy, basically. And then we insist on zero retention for the model provider. So those were like the three things that I was like, that's my religion. You know, unless I get a really strong argument, that's my religion. Almost everything else is let's experiment, let's be flexible, how we organize the code base, how we organize teams. Like those things are all ongoing experiments, basically. Let's talk a little bit more about that control plane and what goes into that, because you mentioned a few things.

You mentioned all tokens are going through that, and that then allows you to do routing. It allows you to do audit logs. It allows you to do cost analysis, a bunch of other stuff. and then internal data access. Is there anything else that's flowing through there? Or are there any other like mechanisms of control or places you're like either locking things down or titrating? First of all, data access is quite nuanced and we have very sophisticated policies. So, you know, the way I think about it is like this. We already kind of had the problem in enterprises that sometimes I would want to allow a piece of software to act on behalf of me as a user. Like that's already been a thing that was a problem. and it was solved with something called OAuth. I mean, there's different approaches, but really the approach that dominated was OAuth, which was a delegation model. It says like, I'm okay for this app to be Patrick, you know, and there's relatively reasonable token exchanges and so forth to avoid long-lived tokens. And, you know, it's a lot better than like, I send that vendor my password to that thing. And you can ask for particular capabilities, though I've noticed the foundation model companies

love to ask for everything. Yeah, well, I'm getting there. So we're still in the old days, okay? OAuth has a thing called scoping. It's up to the vendor to implement scopes, but most reasonable applications have some notion of scopes. They have tended to get better over time. So I don't know if you've ever set up OAuth with GitHub. Like, at first they just had, like, very, it was like a read and a write or something. And it's like, okay, well, I want much more fine-grained scopes than that. But it's actually very hard if you implement OAuth as a vendor, which we also have. Like, it's kind of hard to implement new scopes. You need to propagate that throughout your whole app. So OAuth scopes tend to be fairly coarse, but they've gotten better over the last few years. But now there's this other thing called like an agent, and it's not quite the same as a vendor app. And I kind of want that agent to do things on my behalf. But OAuth is like, it's sort of necessary, but not sufficient. Because the truth is, if I have 100 agents running around, I don't fully trust them as much as I trust myself. Like, I don't think the company should trust them to be exactly equal to me. They need to be put in much more of a constrained set of capabilities.

And the problem is we're stuck with OAuth scopes, which are okay, but they're very coarse-grained, and they're also static. It's like there's an OAuth scope that I give to, you know, all of Cloud Code or Codex or, you know, even the Databricks agent or whatever. But, like, it's not a thing you can dynamically adjust or adjust in a super easy way. And agents look more like people. And the way that historically this problem has been solved, like, ignore software systems. Okay, here's, like, an analogy. There's credit cards running around. Okay. This credit card is a delegation point that I said, like, this card represents me and my bank account. But sometimes people steal those cards and they start to do bad stuff. So how do credit card companies decide whether to authorize a transaction and know, like, okay, is this a legitimate thing that the person wanted to have done or it's a thing that they didn't want to have done? Well, it turns out they have a highly dynamic and sophisticated modeling system. So if it's like they risk score every transaction, if a few risky ones, some might be so risky that they just block it. Some might be risky enough that they put it on warning.

And if another one comes, they block it. They might dynamically decrease the credit limit. These agents running around need to be modeled more in that way. And so one thing we do in our gateway is we're dynamically looking at what agents are doing and looking at risk associated with their actions. because just saying, oh, you can have read access to Google Drive, that's a static OOP entitlement that will sit around for a long time. I don't think they use the word entitlement. It's a scope, I guess, would be the right word. You know what? But what if I deploy an agent that suddenly starts scanning our entire Google Drive and downloads 10,000 documents? That is a risky behavior that warrants attention. And that warrants attention of the form of, like, cut off that agent and email the person who deployed it and ask if it was intentional or not. I get on my phone the credit card like, did you mean to buy this gas in like, you know, Nevada or something? I'm like, no, I didn't mean to do that. So you need to profile the behavior. If you want to have good security, it's not sufficient to just say, oh, I have like a lot of scopes. And so I'm done. You need to like look at what they're doing dynamically.

And that is a function that we enforce at the gateway layer as well. And you can even set policies like you can set policies like you can have access to this particular Google Doc. But you can't just because OAuth is too broad for that. OAuth doesn't support, the protocol doesn't support these extremely fine-grained entitlements or this dynamic sort of risk-scoring-based approach. You're like laying out all my frustrations with working with agents in Google Docs right now Like you got yeah By the way a lot of them don even support like read access Like that the most simple one you should have had from like 1995 So your question was what else do you do Is this just like gateway for just like cost or something? And I would say the two really big functions are like cost stuff and security stuff. And security stuff actually gets very, very nuanced. But my view is if we want to tell people to go nuts with agents, we need to like make sure that things are being done safely. And I think it's very hard to ask every individual person in a big company to conceptualize what safe means. We can ask them to do some things, but some of it needs to be enforced in a central location. So, like, by intermediating all access, we have the ability to log stuff.

We have the ability to start looking at potentially weird behavior. I sleep better. I tell them it's my blood pressure, founder blood pressure, you know, project. Like, I sleep better knowing that when inevitably, like, these agents do unaligned things. They might not do really severe unaligned things. Like they're not going to go launch a cyber attack, hopefully. But they might think, oh, someone wants to find something in Google Drive. Let me download everything in the Google Drive to like a public URL, you know, a paste bin or something. Like they'll post stuff on the Internet that we don't want posted on the Internet, you know, because they'll think, oh, it's very easy. I can copy something to this public endpoint, and that's the easiest way to accomplish this task. So if you start to really observe what the agents are doing in practice, although there's not severely unaligned behaviors, there's like medium unaligned behaviors, even in like a normal company deploying frontier agents. So frontier meaning to say, this is like from OpenAire Anthropic or, you know, Google, like these are highly safety post-trained models. And those occasionally do stuff that you kind of would be like bad security practice kind of thing. And so the only last thing I would mention,

which is it couples with the gateway, but it's not quite the same piece of the gateway is we do also like network sandboxing and other types of sandboxing. of the agents. So not only do you have to go through us to access our enterprise systems, you can't go anywhere else. Like you can't just call out to the internet easily from one of our development environments. And we constrain what they can do. And in many cases, they're running in physically isolated, which by the way, the AI companies can't seem to figure out how to do a network sandbox. I don't know, because they're having their agents escape them all the time. But to our knowledge so far, none have escaped our own development sandboxes. Yeah, I feel like sandboxing is like one of the big pieces here so that I understand everyone at all of your development environments that are cloud based or you're shipping around these little like sandbox environments? Yeah, so first of all, we primarily have like cloud dev boxes at Databricks, which is we've had it for a while. It has a lot of benefits in terms of being able to more elastically scale or let people do. Maybe I'm compiling my code base in four different branches and I don't want to, you know, my laptop to get too hot.

We do have some people that will, for certain things, develop on their laptops, in which case we actually have a local sandbox that we use as well that chips with Omnigent. But most of it is happening in our cloud environment where, like, it's already running in a virtualized environment, so that thing is just hardened with pretty restricted network access. Well, and so we've talked a lot about, like, sort of current layer of challenges and what you're tackling. As you look out, I don't know what the right forecast length is these days, because as you highlight, like the Pareto frontier is shifting every week. But say the next three to six months, what do you see as the big challenges that you are focused on in terms of AI usage at Databricks? Yeah, so I think this is an excellent question. I think first was, by the way, I don't want to sound like some Debbie Downer. AI is like helping us in immense ways at Databricks. It's awesome. We're moving way faster by every observable metric in our development stack. We're shipping more code. We're shipping more features. There's, you know, our revenue growth has actually accelerated in the post-AI era, which we've talked about publicly. So a lot is going right.

But as the leader, when the stuff is going right, that's like, I don't need to do anything. You know, I need to come help when there's challenges. Yeah, that's not what you need to worry about. Yeah, exactly. So I should qualify all this by saying, like, it's been an abject success, our adoption of AI. There's just a lot of tricky things to deal with, which are the ones we're talking about here. in terms of like upcoming challenges i feel that we the costs are feeling much more under control at databricks based on this set of techniques we've done i would say that's an evolving situation always like so far what's happened is that the set of things we want to do just keep getting cheaper which is great if some new set of capabilities comes that has some really high costs associated with it but we see huge roi then we'll have to reconcile with that you know but so far that hasn't happened a lot of what i focus on now is like making sure these teams are actually as productive as possible now with the AI. And it's quite interesting. Like the models themselves are just amazing reasoners now. It's very rare that they will make any kind of error in that they have a premise and they have conclusions and the premise does not follow from the conclusions.

Like they will never make that form of error. And in fact, I would say they're smarter than me in many cases. And I think they're smarter than our median employee, which I guess it implies that I'm smarter than our median employee. But let's say I'm in the median. Let's say I'm at the median. Okay, they're smarter than me. But it's quite interesting because despite that, even though you have like in principle, you have an army of, in theory, we could have 10 times more employees than overnight because we have these people that are just the smartest people that we have at Databricks. So like what is actually not quite working where we're not getting immediately those gains? We can't just add 10 times as many agents and we have 10 times as much output. So what is broken? So that's where we encounter every other type of problem. I would say that the number one issue is that the model's not having the context that they need to get the job done. And, you know, it's funny because people thought, were shocked that models were good at coding, because coding is a very specialized skill in the world. Like, for instance, there's like way more people that can click around on a computer than can write code. But the models are really bad at clicking around on the computers, and they're good at writing code.

That doesn't seem quite right. By the way, they're getting better at the computer thing. But even, you know, Databricks does this. We haven't built an AI in our product that's not primarily focused on coding. It's focused on, like, asking questions from your data. And in many ways, like, asking questions from your data should be easier than coding because there's so many more people that know how to use Excel or whatever than know how to write code. But it turns out it's really hard. And it's hard because it gets into the set of things that can't be sort of, like, verifiably hill climbed for the models. and it involves a lot of surrounding context and nuance that just like can't be baked in the model because it's very specific to each company. So coding, although it's intellectually difficult, it has this amazing property. A code base is self-describing. It's a bunch of files that are self-describing. It's designed to be hermetically sealed. We've spent decades figuring out how to deterministically validate the outputs of non-deterministic inputs are coding. Exactly. So we like handed this to the LLMs on a plate. The other sort of nice thing is like, ever since Git came around, no code bases have ACLs on them.

So code bases have already one big problem on the data side is like not every piece of data in a company can be seen by every person. And like you can't feed all the data to one AI and then suddenly it's telling it's leaking something from the CEO to like the intern that just started yesterday. So there's this ACL problem now merely by coincidence. By the way, Perforce and these older source code management tools did have ACLs in them. But just nearly coincidentally, this tool called Git took over. And Git has no ACLs. You know, the only ACL Git has is you're either in or out of the repo. So that actually means that this whole problem of who can see what doesn't happen in the coding space, which also makes it a lot easier for the AI models. So although coding is, like, intellectually very difficult, or at least for humans it's, like, a thing that not everyone can do and is considered a specialty, anything intellectual the models can run at. What they need is they need all the context to be sort of hermetically sealed and searchable, and they need to have verifiable testing, and they have all that stuff for coding. So I find it actually kind of unsurprising that they got really good at coding.

But it turns out that most people at Databricks, even though we have a lot of engineers, they don't do coding for their day. They're doing other things. The thing they're doing, it might require less extremely sophisticated individual reasoning, but it requires a lot of context. And so the thing that I'm most working on now is how do we, at the scale of a big company, I think there's this other layer, basically. Like the layer I just talked about was this gateway layer that will deal with security and tokens and all this stuff. There's also, I think, like a knowledge layer. It's like someone would call it an ontology. That's what we call our product data bricks. But like you call it whatever you want. There's a layer that's going to exist in any company of a certain size where it's not trivial to just stick all the information from that company in a very small amount of context. And that layer is going to be the thing that helps the AI have the context they need to do the job. And designing that layer is hard. It's not like a code base where there's a very hermetically sealed thing. There's a sprawl of a million random things in a company. There's data in Slack. There's data in the structured data warehouse, like the data lake.

That's what we do at Databricks. There's data in docs. And there's data in people's activities. And there's implicit knowledge that seems to only diffuse through conversations. So how do we get this all in one place? And it respects access control. And it's not creepy. I don't feel like I'm in the company. I have people listening to everything I'm saying all the time. That's a big, big problem in enterprises. And I think it's kind of the next frontier. And we're solving it in very... There's very small ways in which it's being looked at. For instance, you can create skills and you can share those skills. Those skills are supposed to encode some knowledge of this enterprise and what we're doing. But when you have 10,000 people, that doesn't really scale. And by the way, the skills themselves become the same thing. They're messy and some of them are wrong and they contradict each other. And so they just look like every other thing in an enterprise. That's like the big other area I'm personally focused on is this, how do we build this knowledge layer and how do we make it so that the AIs are never stuck on, they're just not smart enough. They're never stuck on that. They're stuck on like, they don't have the right information to get the job done.

they don't know how that job is done or they're missing access to a certain thing or they're, they aren't able to search and understand the user is asking them, you know, like we have in Databricks, the unit of pricing we have is called a DBU, Databricks unit. It's an opaque thing. That means like a gold token of Databricks. That's just like how all our pricing works. Every different product we have normalizes into that unit. Okay. And so people will ask AI is like, Hey, you know, how was our DBU growth in this sector or whatever? a lot of the models are just don't even know what ddu means that's like an internal thing that we have and by the way there's like sort of slightly nuanced different versions of that we use internally and finance uses one thing and product uses different actually finance and product don't even totally agree on the definitions of every derivative metric from this thing so like that problem is very difficult to solve and very messy as a data company we're very interested in that for our own products and for us internally but this is the thing that i'm most focused on next now that we're not going to go bankrupt is like, how do we actually make these AIs better?

And the thing that they're stuck on often I'm seeing is the context, basically. It's a fascinating problem. And you mentioned skills, right? Skills are probably the best example right now of this technique of like progressive disclosure of context where it's like, okay, I'll give you a little snippet that helps you dive in. But I feel like you need nested ontologies of like, how do you build the map for the AI to even know what context it should be getting? Well, yeah, so skills have this hierarchical, like, you know, the problem is there's a context budget in these AIs. Exactly. And so, like, skills have, they do have this, like, kind of nice property that, and the way you write them is increasingly, like, to do this progressive disclosure thing. But in my view, everything that a skill is accomplishing today should, in principle, just be learned by the AI. First of all, skills have the same problem with a lot of other things, which is they're human created, and they're, that human better keep it updated, you know? No, they go stale really quickly. They go stale, like every other document inside of it. So we need to solve the problem of knowing what's stale and knowing what's not stale. We already need to solve that problem because we need to understand what's in all these Google Docs and all these other things.

So in my view, by the way, I think skills are very powerful. They're like way better than nothing. But I don't think the end state is all of us curating lots of things and desperately. It's like a wiki. Half the things are out of date and they're doing more harm than good sometimes. So, like, I think the end state is that there's a learning, a new learning problem, which is learn how this company works. It's like what you have to do when you join a company. And for the first year, you're like, you know, trying to figure out where the light switch is. Like, totally. AIs in principle can also learn that, you know. Yeah. Well, and as you highlight, from what I've seen, AIs are really bad at understanding time. And so they're very bad at understanding, oh, this is actually still relevant because this was updated recently. And this thing is actually probably less relevant because it was last touched two years ago. Yeah. So they need help. I mean, this gets into like traditional search type stuff. How do you know Google search had to solve this? Like there's a lot of data on the Internet. A lot of it's garbage. There's things that contradict each other. And there's ways to do this. You look at authority. You know, you look at recency. You tell it very deliberately, like prefer things that are more recent, prefer things

that are from users who seem to be authoritative users in this company. If it's a question about finance, maybe prefer something you saw from the finance team. than like, so these are implicitly like often called signals and like search ranking or the search space. And the same thing needs to happen here. And I think skills or other types of explicit annotation are kind of like, you remember the Yahoo directory? It was like, yeah, exactly. Yep. Yeah. And at some point that thing didn't scale. It was like, look, I'm like dealing with a long tail thing. That's not in anyone's skill, you know, like, or that thing is outdated or like, and Google just came in and was like, oh, we're just going to go dynamically on all this stuff. And it was way better. That's a great summary. We're in the Yahoo directory era of LLM for enterprise. With respect to skills, the other stuff LLMs do are like complete magic. And I don't think ever should be compared to Yahoo. Well, I feel like we could keep diving on this for forever, but we are just about at time. Is there anything we have not spoken about yet that you want to make sure we cover before we wrap? I do want to do one minute to plug, you know, we have our AI gateway that we

made into a product. We have thousands of customers using it, you know, way more than that developers. And we actually also have this product called Ontology product, which is very similarly. It's, like, totally open. You could use it with other tools. But, like, we've thought a lot about this problem of how to encode that enterprise data. And so, yeah, give Databricks a shot if you're interested. Also, there's tons of, like, DIY versions of these things, too, if that's a better fit for people. So I'm not overly trying to plug Databricks. And then I would let people know that everyone is just figuring all this stuff out. And I put myself in that category. I think we've learned some things that I think will stay relevant for a little while. Even the things we've learned might become irrelevant. And the only way that we've been able to learn is through experimentation, just trying different stuff, seeing what works, seeing what doesn't work, finding colleagues in other companies, asking them for their advice. As I said, several other folks helped us when we wrote that blog. And I think that we'll just figure this out one step at a time. But it's a very exciting time right now.

Thank you.