1つのAIコストを削ると2つ生えてくる：Prosusが語る社内AI運用の総保有コスト

It also causes there to be shared state between the time and the server. So certain features might be the same. Okay. Where the server wants to... I'm not going to be too hung because I'm watching this keynote. But recently, I caught up with Yones. We talked about the total cost of ownership in AI on a bike. Enjoy. Beautiful. Beautiful. One, two, three. Whoa, why are you going that way? Way harder than I thought it was going to be. I'll see you on the other side. You're good at it, man. We're going to talk about your million dollar decisions. Are you ready for that? You're going to be able to tell me about this million dollar decision that you did? What was your whole revelation?

There are a couple of things, right? So you get lots of peace of mind when you go and hit an endpoint from GCP, OpenAI and Cloud. It's just that it doesn't scale. It's too expensive. It's too expensive. And you're talking doesn't scale to 1,000, 10,000 person companies? Yeah, it doesn't scale to, I think, in the end it doesn't matter to 5, 10,000. It can be one person company. It's on the other side, right? Who consumes your product? The moment that you go into a real consumption, especially if you have harness, it's as greedy as it gets in terms of tokens, right? Because actually you give your harness a whole computer. Yeah, with the sandbox. So it's like, oh, hey, we got the sandbox that we're now paying extra for, and it can do whatever it wants. Yeah, so the sandbox itself is a cost. It's not very expensive. It's just the storage that you think about.

But the agent now can store lots of things in the store. And then you can connect, you can do 10 times more integration than it used to be in the past. Because now it can run its own skills and it can run its own integration. So it's 5 times smarter, the models are smarter. And then suddenly you need to pull out information from Slack, email, Google Doc, Granola, Notion. and check whether what we have discussed today has been properly translated into a PR or into a linear, and that can happen a hundred times. And then the cool thing is that it feels it's free. But actually it's not free. Because it's free in terms of human consumption. It's free in terms of human consumption, and it's free in terms of... We've been consuming one way or another this type of technology, highly subsidized. You're on the highly subsidized narrative?

Anything in that? I think we are still on the high subsidized. If you put down the numbers from the unit economics, I think we are still not yet. And then what's happening is, okay, you build this product, you give it to the people, and then it's new. So everyone can try to do all the crazy things that they can think of. So they build apps, they create documents, they create presentations. They schedule things that probably they don't need to take place every five minutes. So there's also the misuse. And admittedly, let's say, first you release the technology and then you fix the experience. we are in this kind of idea this moment because it's also very important to validate the technology and it's also changing very fast. So yes sometimes you release it and then you don't make it clear, you know, you don't make clear problems. Do you really need this to run every week? Do you really need this to run every five minutes?

Do you really need to run every Saturday? Why do I run every Saturday? But it's the same in the data world, right? You didn't need data to be fresh a lot of times and a lot of ways that you could save money was... 100% just by... Exactly. Reduce the frequency. I agree. It's just much more expensive. Yeah. And easier to set up. In the data world, there was only the data folks that were doing that. Exactly. And they knew. And that was the responsibility. And someone was looking at them, right? So now you have given this powerful tool to everyone. Which, by the way, a lot of them, they still put down a lot of effort into... into what can I do? Yeah. How can I help me? And they can make the same up 10 times. Because the first time they didn't know, so the second time they tried. And then there's a cool thing. They try to do very cool things that you wouldn't even think about. So you're pushing that down, and that's cool. But then you're also getting this repetitive nature, because maybe the first time it doesn't come out as right.

And then they'll throw away the whole code base. And then you do it again. And then you do it again. And then again. So there's this type of... And that gets expensive. And that can get expensive very quickly because the context is getting expensive. And then you need, of course, to go into context management. You need to go, how do you do semantics search of your skills? So how do you fit it? In the end, what works best is the simplest. So give the agent as less instruction as it requires. because then you are not overcrowded with lots of context. And it works better even. So we went into that path, and we went into that path first with cloud and OpenAI. And then we find out very quickly that the costs are growing exponentially day-to-day. And this is, according to you, with the subsidies. So you're expecting that eventually the subspace will run out and it's going to get even more expensive.

Exactly. You recognize that's not feasible. Yes, that is not feasible because a lot of applications during this transformation are actually becoming viable. In the end it's a unit economy, right? So if it costs me $10 to summarize my emails, I can live without, right? But if it costs me 0.0001, then that can become a big pipeline of 10 things that in the end, it can make my life 10 times faster and better. So then you need to build a unit economics in place. How are you deciding what use cases are worth doing and which ones aren't? Is that where the hard work comes in? Is that why it's so hard? Yes, it's very hard. Because at the end of the day you can't have a judge looking at every use case and saying this one is worthy, this one is not. No, there are two ways. The one, the traditional one, you just keep the human in the loop. So you made someone responsible for the budget.

CFO, or somebody around there. Someone needs to pay the deal. And that person you tell him you have hundreds of something of tokens, right? And then he's the one responsible to distribute it accordingly. That's one way. And then there you can make everyone's life easier. You can explain to the users how much something costs. you can make a little bit of education, there's a lot of underground, you know, you can become more informative, for example, hey, you know, you don't have to rebuild this, someone else has already built it. For that in mind, we are building now in this live marketplace, that you can select different apps that others have built, different agents that others have built, different skills that others have built. I've always had a lot of hate against that, and I'll tell you why. Nobody actually wants to comb through a marketplace to see if they potentially can find their use case and then tweak a little bit of it.

Maybe they send their agent out to do that and that can happen. But if it is so easy to just reproduce it on my end, why wouldn't I just have my agent reproduce it in the exact way that I want it? Yeah, I agree. So I think the first objective of marketplace is educational. So it shows what is possible. Okay, yeah. So lots of people don't know that. So that's my first objective. The second, on the product point of view, you need to be able to surface this work in the chat. And not the user will... So the user shouldn't go to the marketplace and start searching. If I ask something, build me a to-do list to track the velocity of my team on the PRs and the productivity Dora metrics, right? So then it can immediately say, hey, someone else has built it. Boom, you get it ready. That would possibly work. That would possibly work, right? And because then sometimes, you know, judging all these hundred things per dashboard,

it also takes effort, right? So in the past you had the person doing that and only that. Now I need to, and then I have the space and Dora and then I need to explain to the agent how I, you know, and all these things. You know what man? If someone has done it, I don't want to invest creativity on there. I prefer to invest in creativity somewhere else. So that serves that goal on the market. So we are also going that to avoid rebuilding a lot of things. But the biggest change was we built very good avals. We have talked about this a year ago. I think the first discussion that we had about having this discussion was on avals. I remember we scheduled this call and you never show up. Me not show up to a partner. And you know who you're talking to. And we were going to talk all about evals back in 2024 and then again in 2025 now we in 2026 We had so many evals And then it evals evals evals

In the end, what you want is that you want to find a way to make your engineering work more professional. Yeah. And good evals is the way to make it more professional. If you build good evals, you can run and benchmark all the models per different tasks, right? You cluster your evals. And then you can say, for this task I need this model, for this task and this model. And this is where we moved now, right? You're really getting granular, though, on A, the task, and B, the evals with all the different models. Sounds like a headache. Yes, that's what we go about. It's hard. Yeah. So you get lots of things for free if you rely on just the API, just hit the API, the most expensive model. And ignorance is bliss. And ignorance is bliss. And by the way, their SLAs are working quite good because any other NeoCloud today,

they are still trying to figure out and optimize and it's not GCP. So if there is one thing for sure, it's not GCP, right? We had time to figure it out. Exactly, they had time to figure this out. Should we go on tandem? Yeah. Let's go ride some bikes. Do we start? Yeah, we're starting. We're starting. Dude, you're much more maneuverable than Dex. Yeah, man. Because with him, you have to commit fully. He was trying to lead, bro. If he has decided the path is right, then you can go left. I couldn't choose left. Even with the steering wheel up here, I could not go. So, but anyway, you know what I was going to talk to you about? You know what I think what you're saying reminds me of is this idea of total cost of ownership. Have you heard that term thrown around? Yep, yep. A couple of years ago. Yeah, exactly. And now it feels like what you're saying is we thought AI had this total cost of ownership,

which was we would just ping an API and figure it out, which in a way it does kind of have that. But then you start to realize we're not prepared to pay that cost. We looked at the total cost and we said, I don't want that. Yes, and also I think there are lots of applications for intelligence that we are still scratching, right? Yeah. Oops, that wasn't me. Whee, nice, on the fly. What do you mean by that? Like we're still scratching the surface on certain things that... We are still in the... the ROI has not been maximized yet. And you said this to me before. ROI is a funny one because I've heard that the finance folks of public companies don't like to talk about ROI as much these days since they have to go and then say that on public earnings calls.

So they avoid that term at all costs. and when you say ROI has not been maximized or found, you're just kind of saying like, we don't know if what we're doing with AI is actually providing value. Is that what I'm understanding? No, no, no. For the same cost, if you have completely optimized your way of working, you can extract much more value. Yeah. So, think it like that. Think it like... Even if it stops tomorrow, the development of the models, there's still been, for next two years, with the same level of intelligence, we can do optimizations. That's what I mean. There's a lot of stuff that you can squeeze out of it. Yeah, there's lots of stuff that we can squeeze out of it. are a lot of processes that you can remove and there are lots of processes that you cannot.

Right. There are lots of things that you can do on user experience to use waste and lots of engineering optimization. Right. So we can we want to go a little bit left here. Yeah. All right. There's some glasses. Oh, no. I understood left. All right. Don't worry. Just going with the flow. You don't have an option, bro. I'm controlling this. Right, yeah. So that's the value then comes when you really... The easiest optimization is actually controlling the context. Yeah. And then the models, right? So how you can find out which model do you need for which tasks. Yeah, basic router. Basic router. not the smartest router in the world just the basic a basic router and that'll get you pretty far and that's and that's where we are at the moment right so I mean with this approach we have reduced our cost 10x and we are just starting

with this right so there's going to be much more optimization going forward but then at the same time we are also building 10 new features every month So the pace is out running the optimization. Yeah, I think we are in this phase, right? So I think we are in the phase where you cut to one and then you see two popping out, right? Yeah. Isn't that your Greek history? Yes. Isn't that what happened? Exactly. Exactly. So we need Hercules. Yeah. Are you that Hercules we're looking for? I'm not Hercules. Yes, not yet. Not yet. And then what is cool is you can start training your own model as we can also start doing with LCM where then you start plugging into specific tasks.

And that's all for this whole idea of ROI? Yes, don't stop puddling or you stop puddling on purpose. On purpose because, yeah, exactly. Should I go down? I think so. I'll see you on the other side. Yeah. We don't have to get any... Oh, I almost hit that car. Wait. Run, jump on. No, no, no, no, you don't want that. Okay. All right. Ready? The three. One. One. Two. Three. This is nice, full trust. You just got to go with me, huh? Yeah. So, and then you start fine-tuning smaller models because lots of experience, it also requires a much smaller model also in the sense of latency, right? Yeah. But you are then having to go and recognize

which use cases are going to be done repetitively. Yes. And how can we optimize that so that it becomes a workflow and we can use the smallest model to get us the end state that we're looking for. Exactly. And a lot of these use cases are one-offs. Yes. Yes. And also if you, if you take an organization that has logistics, then it would probably pay off to start doing some post training or fine-tuning on some of these logistic traces, right? Okay. And then you expect to get similar performance, maybe a little bit better at 20, 30x the cost, at 10x the latency or 5x the latency. It's simply because the model is much smaller, right? So there's no... You have smaller, faster, cheaper. Yeah, Yeah, exactly. And then you can't do this for everything, but for example, you can probably

train a model, and that's something that we are discussing, for doing specific data analytics, right? Because this is, I mean, yes, it's never, the question is probably never the same, but everything around is exactly the same. I think we're clear. Where are we going? Clear, we're going straight. And then we're going left. You're saying the question isn't the same but the process is the same yeah the process is the same the data stays the same data data databases right so you can you can have probably a track of hundred thousand questions that they've been already asked but does it need it that's the question I have like does the new we're going right here right yeah yeah all right all right we did it and then we're gonna do another another one eventually because are you not just doing hard

engineering problems for the sake of hard engineering at that point you know what I mean? Like, does the, is the juice worth the squeeze? Or is it just a whole lot of squeezing? Do you understand what I'm saying? Yes, I understand. Training a model or RLing a model, going back to that total cost of ownership, there's a whole lot of things that you need to do to make that successful. Yeah. I think the answer lies on the... It's always related to scale, right? Yeah. So if you do it for five people, then no, it doesn't. If you do it for 5 data analysts or 5 engineers that they hit this every day than it does Now can you potentially run this exercise to the cost of I don know the engineers

spend also three hours doing this instead of... No. But you can make some good numbers and then you can see, okay, if we go that way, with that type of extrapolation then it will pay off right that's the first thing and then you're also building the muscle right so you are training your team and also there are lots of byproducts of this for example the building a model router right yeah so mill the building a model rather is one thing and you can buy it let's say yeah there's a ton of them out there yes but you need also for a model router you need also to have an amazing eval suite right and then you you go into the by-products sometimes yeah the steps removed of what the main thing you think you need and then you realize wow if we're going to have success for this thing then you have to look a little bit Exactly, exactly. But then, yes, it is engineering and I don't think so that Gen AI has been removed the hardcore engineering.

I think contrary, because now you can build so fast, you need the basis to be very solid. So you need good infra, you need good observability. Where do we go? Right or left? Left, left, left. We're gonna cross here. All right. See if we do our right crossing. How the hell? Ready for this? And then we go right or left? Left, left, left. Into the park. I'm trying to go into the park. Get a little... but there's stairs there. So we go right here. We go right there. No, there are stairs there. I know, that's not gonna work. Look at the lion though. Lion in a bear park. It doesn't look like a lion. But I understand. No, that's totally a lion. No, that's definitely not a lion. Oh my god, you're just trolling me now. Except if you just had a haircut. Can we stop pedaling?

Alright, toast. Okay, so, going back to it, the muscles you build are valuable, And it's much more beyond like the basic hey 100% and then what it happens there is a byproduct is that these muscles and these capabilities can actually be then Lance of scale to the whole organization, right? So the moment that you find out how to do good evals Yeah, then you do them for everything then you do it for everything because every agent will require some evals one way or another right so then you can package it and then and then and then enable everyone to do it right and then that that depends on the it's always on the scale in a scale of 30,000 employees like process ecosystem has at the moment this is incredible it can you can

completely change how we do it I'm sure I'm drinking there, huh? On that turn. You got a little scared. I heard it in your voice. Oh, we're going left. And so, okay, there's the cost-benefit analysis that you're doing, too, to decide, does this use case warrant us to go and try and do our thing on our own or try and create our own version of this internally, right? But also there's another thing when the technology change so fast you want to build some technology in-house oh He said no keep paddling No bike So how the Dutch person knows right? He has done it

keeping right yeah I know this would have been a lot easier so so anyway yeah yeah I remember so I was telling that because technology is changing so fast you also don't want to lock yourself in in something that you don't control mm-hmm right so there's also a value of optionality right and those are the muscles that you're building. Yeah. So that's one of the advantages if you build. Imagine now you're committing to a... there is no cling cling cling cling. There is no horn. I don't want to do that to the kids on their shoulders to scare them. So the... Imagine now if you commit to OpenAI or Claude and then you're locked in because you built everything on it. Yeah, exactly. It's not so easily swappable. And I mean OpenAI is probably the safest bet. But there are also lots of open source libraries out there. Imagine

if you have come here to Open Claw in December. Probably around March, you have to scratch down everything and then follow Hermes. And now Open Code or Pi Code. And then these projects can also divert so then you always need to think okay is it then do I convert back right so there's the open source there are there are other small companies that they can pivot any time also right yeah so the fact that you're a customer doesn't mean that they don't decide tomorrow to do something else and then what do you do with the whole stack right well yeah even Amazon or GCP can do that too yeah exactly yeah yeah yeah I mean there's so many products that they've released that eventually just died. Yeah, I agree. So in that sense, in that sense, having, let's say, being a, not being the first that you adopt

something, it might also be if you want to scale, sometimes good option, right? Yeah. Or you build it yourself because then you can quickly pivot. So if you think about account, we start building a first chat GPT assistant on Slack. In Slack, yeah. And it's just a way to plug in chat GPT to Slack. Yes, as a way to talk to GPT3. In 2022 or 2021? One month before chat GPT. Yeah. And then we pivot from there into, let's say, a web application where you had to select different roles. And then you went from this to an agent where the agent could select the tools. And now you went into a harness. And this is only realistically possible if you control at least a big part of the stack.

Yeah. Because otherwise, what? You're just at the liberty of OpenAI doing that? Yeah, or if you use another startup or any other, let's say, open source library, there is always the risk that there's the other thing around, which is better, faster, cooler, and then you either need to commit to what you have and make a pull request and then pray that all the forces will align to get accepted, or you need to make changes to your stack, but then you cannot ensure easily backward compatibility. Well, I guess what I'm thinking as you're talking about this is there's a certain amount of use cases that you're trying to figure out where to invest your time in to optimize.

And you want them to be at a scale, but you also want them to almost be like generalizable in a way. So you were talking about the data analyst, and it's not that you're learning a one data analyst workflow. You're not optimizing that one workflow. You're trying to optimize as much as you can in the data analyst's complete workflow. Yeah, the suite. It's not just that one thing that the data analyst does, right? And so there it's more generalizable than it is use case by use case. How do you go through and you look at these and you recognize, all right, this one feels like somewhere where we should be investing our time and effort? Yeah. Good question. There are some operations that are critical for business, right? And those are probably those that have been measured over time. There are data, metadata. Do we step out here? Yes.

Yeah, right. Stepping out. Don't fall. All right. One, two, three. Send it. So the metadata. Yeah. So there are already some operations that... Actually, there are operations that you're investing already effort before AI. Yeah. Right? So customer support, data analytics, account management, optimizing your marketing budget, optimizing your logistics. Right? So you know, right? So the business didn't change. Right? So then you know what are the use cases that you need to invest and make them more scalable. And then at the same time what you need to do, which is also our strategy, is that you want to give, you want to unlock all the other new things that are now possible right And the way to do it is by giving a tool like to count where you can build apps and then we everybody can build up slow yeah and then you can see people that before they need to have an engineering team now they can they can literally just go and build something

themselves and then you can see probably over time how many other people are using it yes and then you are start getting new insights okay now this is how we'll see and so that's that's the more bottom-up approach right of course then the top-down which is okay where the company wants to go right so where the company wants to invest and where do you want to optimize and innovate well Nope, that wasn't me. That was me, my bad. It's crazy. I just do a little bit left and then it feels like you go completely left. We're going right here. Did I answer your question? Yeah, so the other piece to this that I think is fascinating that we've talked about before, and I don't know if you really have a clear answer on yet, is who are you hiring?

And what kind of skills are folks expected to have? Because AI engineer means so many things now. Yeah. Right? Like, what does an AI engineer even mean? So you don't hire for, for sure you don't hire for titles. You hire for specialization in a specific area of your product. Okay, so what do I mean by that? It's probably extremely hard to find a person who has STRE experience so he he knows his infra and platform and AI and evals, right? So the whole and the whole stuff, right? Yeah, so what you're doing optimizing serving and GPUs But there are some things that you cannot discount and this is for me good software engineering design thinking around this right so how you can design good lean scalable solutions because the coding now more or less has

been abstracted but how do you design your solution has not been right and how are you testing for that with the traditional old design system solution So you do a whiteboard design system and maybe you do more than one you might need to do two. Yeah just to really get a feel. Yeah where are we going? We're gonna go left we'll go up here and then go left. Which one? There? Yeah so the crosswalk. All right see if we can you tell me. See if we can survive this intersection. Oh boy. Now we're on white relaxation mode. All right. One, two, three. Let's fucking go. Sorry. Did I go too hard on the pedal? No. Okay, so you've got the traditional whiteboard interviews. Yes, that needs to happen, especially now because coding itself doesn't say anything.

You need to understand if you understand how to build things that they scale, things that they can, you know, they are not over-engineering. Now over-engineering is quite simple because you just ask. But then maintaining is hard, if you like it or not. It's smelling the over-engineered code that Fable tries to give you. Exactly. And then if you ask what else I can do to improve, it just keep on coming and keep on coming, right? So it never says, I think I've done enough. Let me go back. Yeah. Congratulations. Yeah, exactly. And then, that's the first one. Then second, he needs to know AI. And for me, it's a very broad thing. What do you mean by AI? Yeah, exactly. So, he doesn't necessarily need to know how to train a cat boost, or not even a neural network, I would say. not for what we do. There's another team in process that they are responsible more for training the models.

So that's not the thing. But they need to have a good intuition and a good knowledge about how to set up an agent. What are the existing limitations? And when you say agent, what do you mean? You can pick up Hermès, for example, right? Okay. So can I pick up something around why, for example, Hermes has been a preferred agent over OpenClaw, right? Okay. Have you tried and have you tested and what have you found out, right? So why OpenClaw stay, right? So what is the reason for it? How do you build Evals, right? So there is a little bit of intuition. Understanding all of that is knowing AI. Exactly. That's my kind of health. Mushies. And then he doesn't necessarily need to go build evals, but he needs to know going forward the more successful products in terms of engineering and scalability is going to be those that have a very strong foundation, but then there's lots of work that's being done from the agents.

So he doesn't need to engineer things if the agent can take care of it. So you need to understand whether this person can realize the trade-offs. Do I build this how I used to build it five years ago? Or do I completely delegate this to be a decision of the agent? So do I build, for example, a static dashboard with front-end to show some numbers? Or do I allow the agent to develop a custom canvas style, dust board thing that can change over time? Right? Because the whole software is going much more into personalized software. So if you build a very hard tightened product, it's going to be very hard then to decouple it. Recognizing that product sense. Yeah. Almost expecting someone to understand that just as much as they understand the software side of things.

Yes, and I think the hardest part in hiring is to find people that they know both. Because people that are very, very good at software engineering, they are already having some years of experience. and if you think about it AI was there before so then the question is okay where I find the person who is good at software engineering but then at the same time he switch me that has experience in software engineering but then at the same time he's in love with these new technology that you want to try everything right this is but I'm sure that there's there are easier by the day yeah because everyone has 10 we do the one two three yeah one two three all right I'm gonna give you some force you gave me the little push yeah accelerate acceleration there's your car nice there's your car that's my car we've made it in one piece I didn't expect that I would need to say sometime hey

Nina my wife my wife I love you I'm not gonna see me again where are you going I think we can park it here all right all right all right okay we stop stopping stopping I don't see you stop you want to finish that thought yes and then we can finish the policy it's getting easier and easier but I don't know if it's getting easier to find someone now. I think it's harder to find someone with AI engineering and then train him how to be a software engineer because that requires a lot of grinding. With coding agents now it's much harder than finding software engineers that they are very open minded and then you plug them into a very heavy AI team and then they can start smoking AI 24-7. Smoking out. What I've found is it feels like it is a lot of noise in what the title is or

what someone who says, yeah, I understand AI. Does that mean they understand how to code with AI? They understand how to build agents, they're using Hermes. And they understand the context window, they understand the serving side, understand the intro side what does that even mean and so the lack of understanding and what or the shared concepts I think is the lack of shared content steps makes it much more there are no job titles your job description really needs to be these are the five things that we mean yeah we say yeah AI engineer fine our turn yes exactly let's go back okay AI engineer doesn't mean that you build an agent using Landcraft. Yeah. Right? And you know how to use cloud codes. Yeah. Right? That's what I mean. OK? And then sometimes this is an interview, right? Yeah. So we spend, we invest lots of time in talking to lots of candidates to find out who can be happy, right?

Because it's a different role than probably what they're used to, right? Yeah. Or if they've done it, then they know. That's also the case. So you can also hire people that are building similar products one way or another. They know, yeah. Yeah, then they know. Then you need to check other things. The evals, dude. The evals are what will get you. Like, are they ready to grind out all these evals and that hard part? Man, it's been awesome. I'll let you go. I'm glad we finally did that. It was a long time coming. Thank you. Let's do it again. Let's talk about evals. We can always talk about evals. As long as you're ready to get back on this tricycle, this tandem, weren't we? Do it. Nice. Nice.