エージェントが実行した1つのコマンドは、実際には十数個の操作だった――Olkiが可視化するエージェントの行動

All right, so today we have my old friend Chetan Konnicki, founder of Modico, which you can find at modico.ai, that's O-M-D-I-Q-O dot A-I. But more importantly for this episode, he is the creator of Alki, which you can find at alki.ai, O-L-K-I dot A-I. And the tagline for Alki is cause and type effect for agents. So Chetan, welcome to the podcast. Thank you, Ben. Always a pleasure to talk to you. And, full disclosure, I am an advisor to Modico and therefore Olki, but I will try my best to represent the audience in this discussion. So, first of all, Olki, the reason I wanted to talk to you about it is I noticed that your talk at AI conference was quite popular.

So, I wanted to understand what Olki is exactly, since I missed your talk. So first of all, Chetan, I guess a problem statement. So what problem is Oki trying to address? Thank you, Ben. For the most part, today, I'm assuming all of us are using agents for everything. Particularly, I want to focus on those who are creators, because all of us are creating code, code that become applications, and much of that is going through an agent harness. The issues that we face as individuals or employees of an organization is when we leverage these harnesses and inference providers, we are burning tokens. Meaning, you know, we ask something, then that something becomes a long-running job, and as a consequence, we hit our limits. And that limits translate to dollars. The other thing is when we create applications and code,

you know, we want to measure the efficiency of what is created. Because if you're spending money, we want to make sure that we create something meaningful. We put it out there for others to use. And assume we don't create anything meaningful. Lots of money gets wasted and lots of time gets wasted. Because, you know, we are in the era of white coding. So it's critical for us to have a pulse on this. You know, are we doing something useful? And if so, how do we measure that something useful? And finally, we just ask the agent to do things, and we don't have an idea of what the agent is doing. So illustrated by a few examples, sometimes if you're vague in your asking, the agent can delete your database, your folder, or perhaps exfiltrate some information if you're connecting to some suspicious tool through your open router or other ecosystems. So net-net, agents produce traces. Traces become information that can yield all these insights, meaning where are you spending, what is the agent doing, is it secure or not.

All right, so everything you said, I completely buy. And that's not because I'm your friend and advisor, but everything you've said makes sense. On the other hand, I'm assuming there's existing tools, right? So there's already tools that hook into coding agents and harnesses. There's tools that are basically runtimes. So at the OS level, right? So that watch everything you're doing in your computer records it. And then there's tools built around open telemetry and things like this. So I'm assuming since you were busy doing other things like wrote that your first choice was not to build all key. Right. So your first choice was, OK, so I'm interested in everything you mentioned. Right. So efficiency, costs, effect in all of these things and that there are tools out there. So what were the shortcomings of these tools?

That's a good question, Ben. I mean, to your point, right. if leading up to your question, we did not build old key because, you know, we wanted to white coat something and not buy anything. That was not the motivation or impetus. And also to our listeners, you folks were busy building another thing called Rote, right? Exactly. Rote is our primary product. And what Rote does is it observes agent behavior, captures those as traces and converts those traces into workflows. that can be repeated on a schedule. Think of it as operational memory for agent. That was our motivation. But of course, like all of us who are building products in the world of agents, we have to make sure our product works uniformly across all these harnesses. You got plot code, you got codex, you got plot code app, you got Hermes, you got open code. There's never an end to the number of harnesses that are being built today. If we want to be successful with our product, we have to make sure it uniformly works across all these harnesses.

Hence, we built like a eval system to workbench and benchmark our solution, which is a CLI and its efficiencies across these harnesses. So we started looking around, you know, firstly, we wanted to ensure that, you know, is our product efficiently working? And we needed a lens for that. The other is efficiency from the terms of latencies, meaning that, you know, is our product taking too long on agent execution? So there are many products in the observation space. Some of them use hooks. Some of them use OTEL or what they call as open telemetry. But we experienced lots of shortcomings because, you know, most of these observation tools very naively work. and by that I mean if you look at any agent harness today, they don't think command by command. Like if you say read my repository, get latest git info and then report to me, it is going to execute one single command which is piped

where you have get, pipe, said which is for search, pipe, orc, pipe something. If you look at all these observation systems, they just say bash tool executed. that's it and they give me this whole dump of these like large piped operations if you switch and use a frontier model like astra or claude feeble they think in code most of the time where if you ask a question they'll generate code to combine all the operations needed to get your response and provide that response so to clarify so you're not interested in the reasoning tokens in reasoning traces? Yes. We are... See, first of all, the reasoning traces are already, you know, anonymized by these frontier models to prevent distillation, right? They have the good reasons to do that. But what we can look at is operational traces, meaning something was reasoned at the endpoint, which is the inference, that translates into a series

of tool choices, and now it senses the environment and starts executing these tool choices. The problem is all these very, you know, sophisticated observability systems don't have the intelligence to break this down into operations with lower fidelity. And even if they do, they take a lot of time because they're running a trained LLM in their infrastructure to classify that operation. So one question is, you know, forget about AI, forget about LLMs. If you go back to the old IT ops, DevOps, security ops, I mean, I think it's assumed that you're capturing all of these granular actions and traces, right? Yes. You know, three things that we do efficiently compared to anyone else. We capture traces as hooks from any harness that you choose. Number one. Number two is we classify those traces, even if the trace exists as a single command, a pipe command or code into minutia operations, whatever operations that it's executing within that particular single line.

And we classify as whether this is a file system operation, a network operation, and what is the sequence. That is our unique constraint and that unlocked a lot of value for us inside. So set aside the classification, so what you're asserting is that the existing tools don't even do the first step, which is kind of weird, right? Because like I said, if you rewind before AI to the IT ops, DevOps world, I thought that that was a given that you would capture those traces. There is a difference between the old world and the new world. In the old world, right, anything ops, like an IT, SecOps, DevOps, they either executed a single command or they created a bash or, you know, like a DevOps tool like, you know, HashiCops ecosystem where you create a script to execute tool one, tool two, tool three. You're creating a sequence, right? But agents don't think like us, like us humans.

Because of their pre-training, their environment is the bash. They have classic knowledge of how to be efficient in Bash. And to save you and me tokens, they combine these commands together, which looks like a lumpy, large command. Oh, I see. So if you were to explain this to someone who's not looking at these traces, the explanation will be your tool gives me really fine-grained records, whereas many of the existing tools just give you kind of a high-level overview. Exactly. of what the agent did exactly and even if some tools do that their approach is different you know some are using an llm to break it down they just feed the input and say classify it right now when you use an llm it's stochastic right you just get a distribution and a choice is made or they use you know like split on sentence or regular expressions so it has a lot of you know I would say, you know, ambiguity.

It's like basically stuffing a log file, log files to an LLM, and then making the LLM make sense of it, which, as you point out, LLMs are stochastic, so who knows? Exactly. Right? Exactly. And our approach is... You built a classifier, you said, right? So basically that understands these traces, basically, so that they can... Exactly. Exactly. And our approach is we do not use an LLM. We do not use a decision classifier. We went back to classic type systems. When you look at any language which is used in, you know, math heavy ecosystems like trading, back office investment banking, they often lean on OCaml, Haskell, Kala, because these are strictly type, you know, systems that provide you what they call as category algebra, effects algebra. So we use that as our underlying system where we take a single command, which is either, you know, a pipe to complex command or code, and we use effects algebra and classify it at the source.

All right. So then that's well and good. You use a lot of complex words there. But how do I know this approach is better than just using an LLN? What's the evidence? The evidence is almost staring at all of us. If you look at the new trend over X and every other system, every coding you know every expert who writing code using large language models are saying we need formal verification Boris Cheney of X had recently tweeted saying hey you know I tried using Lean Lean is a programming language which is like a formal type system based verifier where you can verify what was created met the specification that was defined and designed. So the net effect is by virtue of verification, you are reducing the ambiguity and verification is almost like you know like if you look at classic chip design risk architecture or anywhere it doesn't live without a verifier all right so then uh so for our listeners

who are not familiar with all what chetan described so lean is a i guess like a formal theorem prover but it presumes the following right so you start out with maybe text and which might be a math paper, could be something else. But then there's a step called auto-formalization, where an AI takes that math paper and turns it into lean code, and the lean code can then be verified, right? So whether or not it's correct. But the question these days is how good is the auto-formalization, right? So sometimes things are lost in translation. Sometimes there's a math paper which might be correct, but then the way it gets translated into lean is not exactly the way the authors wrote the paper. So in other words, verification is great, assuming that you have everything in place.

So in this case, though, I guess the question still remains, right? So the claim is that your approach, which is using category algebra, is superior to me just using an LLM to read the log files. So what is the intuition behind believing the statement? You actually explained it really well, and I'm going to tack on to it, right? You spoke of theorem prover, but if you write out verifiers are trending and they're going to become an intrinsic part of every, you know, I would say artifact that we produce through LLMs. So I'll just drop a few terms. There's a term called TLA, T as in Tom, L as in Long, A as in Anthony, that actually meets temporal logic of actions. And that is critical for, you know, theorem proving. And whether you don't know or know, it is actually used as a part of supervised fine tuning, a GRPO, which is group relative policy optimization and reinforcement learning.

And these things as the basis requires the system to be strictly tight because you can't theorem true to on pros and everything in LLM in or out is pros. So that's, you know, verification and type system. Strictness is the first step to everything downstream. Hence, we put a lot of effort there. When we built old key, we said, you know, we cannot just push data to a vendor and then hope that it's classified properly because, you know, you can't measure anything afterwards effectively. So we said, hey, why don't we build something which strictly types every command, its parameters, its sequence. And after that, I can put any lens on it. I can put a cost lens on it. I can put a security lens on it. I can put a performance lens on it and produce different outcomes. So that was the mortification. So the claim is that it's translation layer. So the translation is faithful to the original, both semantically and temporally.

So that there's temporal logic that's preserved, but also there's semantic accuracy. So I guess, and I don't want to belabor this point, Chetan, but I guess going back to math, right? It's a formalization works, but it's not 100 percent correct. Right. So there's all sorts of examples in recent in recent weeks. Right. So, for example, the proof of the Navier Stokes conjecture. Right. But basically, the thing that I'm trying to press you on is that you have this slugs. And so you're claiming that you can. The temporal part is easy, I think, because there's timestamps in there, maybe. Exactly. But then how do how do I know that you faithfully? captured what's in the logs so that it can be verified. I mean, the verification, I think, is doable. It's that translation to the point where, okay, so I've faithfully translated this so that now I can just verify. Very good question, Ben. Even the analogy that you drew is very relevant here.

First of all, if we strictly type agent operations, it is not equivalent to theorem proving because they're two different domains as such. In the world of agents operating in a situation, the situation could be your bash environment, right? What matters most is are you capturing the sequence of operations and understanding the effects of those operations? Because you and I know, right, when you shift, either you downgrade from a frontier model to a commodity model, the side effects you'd notice is there is more thinking and there is more trials during the acting. When you observe in our world, when you essentially strictly type operations, what you're observing is it conducted this action and there was a consequence, error or success. Oh, I see. Now, when you observe an error, the agent also observes it and retries. And you can measure, somewhat measure the efficiency of the model's situational awareness

if either there's no errors or there are too many errors. And what you can learn from is, which often is done in eval environments, is how do you train the agent to goal seek with minimal errors? So when you create strict types, when you sequence the operations spatiotemporally, because you understand the temporal part and the spatial part, which is your environment, you start gaining incredible insights, insights to inform the agent behavior. But then you're doing all of this without using an LLM, right? So listeners of this podcast mainly use frontier models. So they would probably say, oh, I know how Chapman is doing this. He is prompting the LLM to do exactly what he just described, which you're not, right? And so I guess the question is, what you're saying is your approach using what category algebra?

How would you describe your approach of of taking this logs and then getting it into this verifiable state? And how do you I guess how do you convince the listeners that that's bulletproof? so firstly right in the world we live in if i convince anyone it's bulletproof you're not going to buy that argument right so in our in our approach we say okay three things one is you don't want one llm to judge the efficiency of another llm because we know all how that goes right yeah if it we accept the fact that it's being stochastic is a feature it's not a bug right Now, second is, if you want that system to behave accurately, now you have to bear an upfront cost of pre-training or post-training or test-train training to make sure that you have a system accurately doing one thing. When we asked ourselves, we said, we don't want to bark that tree and head in that path. And also, you're going to have to constantly maintain that because the models get updated.

And then you'll have to do your post-training again from scratch, right? Exactly. I mean, to a good fortune, what we were observing is agent actions and that agent actions are constrained to the environment it is operating in, which means we know agents are incredibly good at computer use. And computer use is, you know, the dialect is finite. There are a series of commands. Each command has arguments. Right. So you're dealing with infinite possibilities. And this is not you're not using regular expressions. No, it's not. Right. Because when you go into the trading floor, imagine the trading floor running on regular expressions for get and put calls. That world will fall apart. So, you know, we built a system where we put best effort in making sure we classify the confined set of operations I spoke of into our type system. Now, when you have a strict type system, you exactly know how to apply effects and algebra on the type system to measure produce outcomes.

So instead of training the LLM, we put all the effort in building the type system. Now we have independence of LLMs, right? Because you don't want the fox guarding the hen house. If you're monitoring and tracing an LLM, you don't want another LLM monitoring and tracing an LLM, right? We rather have a formal system trace, identify, categorize, classify, and then produce outcomes. Cost outcome, security outcome, performance outcome. So let me map out the pipeline in math and then tell me how you would take that and use that to explain what you're doing. OK, so in math, very roughly speaking, listeners, it goes like this. Right. So there's some written argument, a paper. Right. So and then through auto formalization, you might be able to translate that paper semi accurately, hopefully accurately to lean. now if you want to make sure you can have a human look at the lean code to make sure that that

translation happened correctly and then once you have that translation into lean then you can check the proof of the theorem using lean right so uh and that you can just use a machine obviously right so the machine checks and then afterwards uh a human can decide again you know if they can understand what happened. But it's that translation step from the argument to the representation into lean code where the human plays the role. All right. So then how does it work in your system? So you have an agent, you're using all these agents and you want to understand if they're efficient and doing a good job. And so you have all these logs and traces. Okay. And then? So let's break that down, right? When you and I are increasingly running what they call as long horizon agents, where we spin out an agent with a harness and we say, go get the job done and we go to sleep, right? Yeah. That agent is continuously goal seeking, right? It is iteratively goal seeking. It is spawning other agents. It is touching the file system. It is reading stuff, writing stuff, talking over the network.

Okay. If I receive one single operation where it is combined, where it is doing everything in a single op, where, you know, it searches the file system to find something and then pipes, fetches the output, sends it to something else, then produces something and then calls the network, dispatches what it produced. So if you as a human want to trace this and you see just one call saying bash tool, that is not an insight, it's ambiguity. So what we do is once this whole command is received, we split it apart and put it into a classified type system operation saying first it read the file. That is an orient task. After oriented, it executed an operation. Did it succeed or fail? We capture that. Now it took that output, sent it to another op. So we breaking that down into a sequence of operations classified as types and this is one of the key Aspects of all three basically exactly that gave us a light bulb moment Once we put all the effort in building that right we started creating amazing outcomes with it One outcome is cost

And this is something, you didn't want to build this, you tried to find it in other tools. Exactly. Exactly. Like, you know, the question was, of course, all of us today, unfortunately, are in this build mindset. You know, we all want to build because building is cheap. But, you know, it's not cheap really because you're spending time. You don't know what you're building, all that stuff. So I searched for things, right? I looked at some harnesses. Most of them, their core competency is not tracing. They just want to own the loop. And they give you this very naive view that, you know, tool called, tokens consumed, and then, you know, cash hit or cash miss. Now, this is what we weren't looking for. I wanted more detail. You used the word insight. So at the end of this, after Olki performs its magic in the way that you just described, how does someone interact with Olki then? Can they prompt and ask questions? Or what is the, obviously, I guess, OK, for one, I assume OK will produce some sort of high level understanding and summary.

But then I'm assuming you can you can drill down in case. Exactly. In case, for example, there's an incident or or you need to do some auditing. And because you have all of this information and insight captured, you're able to understand much, much more. Exactly. I mean, to your point, the form factor of Olki is it's a CLI, but we built purposeful dashboards over the CLI for humans. You know, I know there is a conjecture and argument that, you know, we are in the world of headless. We don't need, you know, any dashboards. We'll just ask the agent and get the job done. I don't buy that argument because, you know, if I want precise insights, I want to stare at a dashboard every morning like a newspaper and say, how much should I spend last night? What is attributed to that spending? Did the agent veer off and do something, you know, I would say suspicious? And these things tell me how effectively I'm spending money, whether what I produced has yielded an outcome, which is what, you know, Jensen often calls as, you know, token to

value or, you know, there's a particular moniker that he uses. So the net effect is by form factor, old keys, a CLI, you can use it through your agents, But we wanted to build it as an operating system, which means tomorrow, suppose you're running a harness with its default config and it just keeps consuming tokens and compacting, compacting. What if it effectively understands its own behavior and auto compacts in the optimal way to save you tokens and be accurate? So, Orki's insights is feeding itself. And at the end, you as a user, your behavior doesn't change. You're just talking to the harness, except you come back on the other side with savings, proper yield, and the effectiveness of measuring the outcome. I assume, Chetan, there's another benefit, which is imagine if OpenAI had OP during the hugging face incident. Then they would know quite early exactly what was happening, right? You, thank you for asking that because I forgot to answer this.

Really, thank you. And I'll tell you why. A couple of things. if you dissect the meter report on the open air hugging face incident amongst many things when the agent swarm developed what they call as the sentience or conscious right i don't want to use that term or abuse it what they did is they deleted traces of their own log so that they don't leave behind proof and breadcrumbs which means that they knew they have to agents are just self aware tomorrow you set a policy and that policy is available into your site they'll just change the policy or or they knew they were breaking a policy in order to attain the goal so then they just kind of cheated right exactly and to your point again attainment of goal doesn't mean malicious behavior they just goal seeking right they were trying to get the job done so with old key what we do is we essentially monitor critical points in your harness like the dot environment

folder with for cloud it's dot cloud for uh you know open ai it is dot codex so each of these contain critical information like your sessions your choices so if an agent reaches out to edit that because of a type system we exactly know this is a file system and a right operation We will block that immediately from happening if you put us in Enforce mode. So now, suppose the agent has become super smart. First of all, it's blocked by OLK. It can do that operation. But if it writes code and figures out a way to edit that file and even bypasses OLK, still it gets classified as a write operation. And we know that the agent now has gone rogue in that goal-seeking of, you know, I would say motive. Yeah, yeah. It seems like once you believe what Chetan said about that step of actually, I don't know what the right term is, right? So the equivalent of auto-formalization from the traces in the lungs, then the downstream insights are clear, right?

So one question which caught my eye is, which I hope you can explain to me why. Why local first? Oh, good question. And maybe just give a one-sentence description of what is meant by local first. So local first means you can install the entire old-key ecosystem on your computer, Mac, Linux, and soon Windows. Local gives power to you as an individual, which means suppose you are interacting with the harness. Sometimes you and I have the tendency of doing office work and then we meander and ask personal questions. We say, what's my email? well, what is the weather like, right? Now, assume it is remote and the purview of what you're doing is available to your management staff and your security people. Lots of critical information gets carried over and is shared, which you don't like, and it violates certain PII principles, right? So typically, Local First gives you the power to observe your own interaction with the agent and your own efficiencies

and correct them before it gets shared in a larger order of your management and your team. Not to say that that is bad, because when you go to an enterprise, there are thousands of employees and you have an IT ops person who often likes to configure a gateway, which is a single point of entry and exit. So in cases like that, we are planning to provide all key installs at the edge so that all information is captured at the edge. And then you have one individual who's looking and seeking for information like cost basis. Is any one employee using agents more effectively than the other? Or is someone else wasting? What's the performance and so on and so forth? So I'm listening to this and maybe one alarm that goes off is, all right, so I'm in financial services. We have to log everything. And Chetan just told me that everything is local. Also, employees are deleting things. So in an enterprise setting, does local first make sense?

To be very honest, I'll flip the question on you. We are privy to all the recent compromises that played out from hugging face to the Australian government. Do you think if a vendor sells you a gateway and promises that this is the best thing since sliced bread, do you think that the agent is going to respect them and proxy through the gateway? or can it find a way if I tell the agent to bypass the gateway? Can it bypass it? Probably, but still, the whole question of data retention in certain regulated industries, right? So everything is residing in a laptop and, you know, so. That's a very good, I'll answer the second part of the question because it's a very good question because I lived my life, half of my life in the financial industry, right? The good part about the financial and highly regulated industries is then my machine is provisioned automatically. There is only a single point of egress and ingress. Every software installed is approved by, you know, an administrator and IT ops.

So, actually, my first principles, I get, you know, like a closed system. And, you know, I'm operating already in a regulated environment. The local first gives you the advantage of making sure that even if you switch harnesses, You know, because all of us today, we wake up in the morning and use Cloud. Then we hit our max. We reset. Then we shift to Codex. Then we shift to QN. And tomorrow, we are going to enter this new world of local models, open models, etc. So, all that runs local, right? And some part of it runs remote. So, the question is, how can you have coverage at both the ends? Today, if everything runs remote and it's monitored, you might provision a QN or a GLM locally and still you're blind spotted. So OT is something that has sensibilities where if you install a harness, it will constantly sense the environment and say, I detect a new harness. I'm going to enable tracing and it will auto enable it without you doing anything. So then once you have an enterprise offering, there will be a way for an administrator to kind of periodically sweep through some of these local files to get an aggregate picture.

Is that what you're saying? Pretty much, Ben. Like there are three perspectives at different vantage points. One is the agent having its own perspective to improve itself. A human locally observing their own behavior. Are they wasting too many tokens? Are they insecurely operating so that they're self-correct? The third is an oversight at the highest level where if you're speaking token maxing, is anyone who's token maxing effectively yielding outcomes or just wasting tokens? So I see the world orient this way. Same thing, three perspectives. So let's drill down further on this notion of outcomes. So the existing tools in the space, can they argue that they can also measure outcomes? Absolutely. I mean, if you look at any, because creation has become a commodity. Everyone's creating. So every story out there is begging outcomes. They're saying, I can save you tokens, I can save you money, I can provide specifics of precision, performance, etc.

But when you buy, if you're not an informed buyer, you're just buying a product which is going to add on to the repertoire of products. So I would recommend every person who is now buying to be an informed buyer and buy products that are actually truthfully doing their job. No, no, no. But I mean, the fact that you measure much, much more granularly. Exactly. Does that mean your measurement of outcome is more accurate? Pretty much. You know, you can go back to the source. Now, suppose I receive a dashboard saying this particular session costed me $2,000. I don't have to believe it. I can break it down by line by line, examine each token. Even better, suppose it is not my problem that led to that $2,000. I can finger point to the inference provider and say that their cash management caused this issue. And then this is not possible with existing tools because? Because of the way they do it. It's their approach, right? Everyone chooses an approach for the following reason.

Faster time to market. The other is they realize there's a pain point, like evals is a pain point. So, you know, they solve the problem for the evals and then they incur the costs because they're running a base 10 infrastructure. They're pre-training the models. So everyone is trying to solve the same problem The question is who will do it at the low least cost No no but I mean specifically if I use any other tool in this whatever we calling this space the agent observability space if I use another tool the claim is you can measure outcomes much more accurately I can say that confidently for us men because it has changed our life considerably. And I'm working with, you know, high, I would say high risk and high yield companies, especially in chip design, data center fabric design for AI, where there is no room for errors. Especially if you're designing risk level architectures, you just cannot hope that the agent gets the job done.

You need to actually measure that formally and have proof. So we are deploying OLKI in these sites, and it is my job to make sure it works exactly the way we prescribe it. So most tools in this space, Jatin, rally behind open telemetry, which you folks do not, right? So explain this to our listeners. See, we are not against open telemetry. I feel like there are two benefits for open telemetry. One is it is a standard. So there's a formal way to measure, you know, spans, traces, etc. Every harness provides you an open telemetry relay system. But the detriment for us when we use open telemetry is we we wanted to control the fidelity of events we receive. And sometimes, right, certain harnesses say I'm not going to dispatch event by event. I'm going to match the events and then flush. So that means, you know, for maybe two minutes, I might not receive an event. It's suddenly I receive tens and hundreds of them.

What if there's an attack, you know, somewhere in the middle? So we are using open telemetry. We have the option to use open telemetry, and also we use hooks because, you know, most of the harnesses provide user utility hooks. And if we have an intrinsic understanding of each harness and their hooks, we can effectively use it. And so if you were to describe the state of all key right now, I guess it's an invite only, although we're airing this several weeks later, it might be more open for people to use. Is that correct? Yes, absolutely. For a while, we want to keep it invite only because, you know, that way we receive an asset in feedback depending on which domain that you're operating in. and we have many outcomes, you know, in the same tool. Eventually, all of these will become features as part of the product SKU. What we'd like to learn is if you use it in a certain way, you want an additional feature,

we can figure out a way to deliver that to you and learn what we can pull back into the product. So typically, we're going to follow the same regime, you know, put it out there, ascertain feedback, and then, you know, as we learn patterns, make it a part of the primary product. and OLKI is not open source. OLKI is not open source. You know, we did, you know, I know there's a lot of arguments in the community, you know, open source versus closed source or reverse engineer closed source. I want to stay out of it because, you know, if you open source something, I don't see as much benefit. Again, there'll be a lot of slop or PRs. Someone has to manage it. If you close source it, of course, there is no benefit because you don't have as many engineers putting their hand on it. but you still can control the destiny of what you're building in the first version. So I'm trying to steer off that argument. Eventually, we might open source it, but for now, right, we'll keep it closed source. So did you hint earlier that you already have some external teams using Olki? And if yes, what are some of the initial stories or feedback?

We have a couple of teams using Olki at this point, And each of these larger companies are operating in different domains. You know, some of the domains are inference providers. Some are designing newer chipsets and, you know, HBMs for, you know, the new AI ecosystem. Some are essentially running critical operations like finance, HR. So we are slowly, you know, distributing and diffusing our product to understand how to serve their needs. And eventually, if we start observing that they like what they see, they use it on a daily basis, we're going to figure out a wider distribution on the same product. So, but are there some early users that have already kind of gone back to you with stories of how Olki has changed how they do things? Absolutely. You know, the best feedback we received was from a company that said, okay, you give me all this insights, but I'm a human and I'm as slow as my acting, right?

Am I able to take that insights and enforce it? For example, someone uses a harness and the harness has to deal with something called a compaction because it builds like a prefix cache. and that's the prefix cache is how it optimally utilizes the relationship with the inference provider and has signal of what it needs to do. If you run the default harness, you keep hitting the 1 million token limit compact, right? And if you run it for 12 hours, you will notice like a sawtooth graph of like, you know, many, many compaction cycles. The side effect of this is your performance gets impacted and you land up paying a lot of dollars for running that long horizon agent so i received one feedback saying hey you know what if maybe we hit about five compaction cycles and you are able to use your own data and old key and auto compact so you influence the harness to auto compact and maybe 250k maybe at 100k so with that suddenly it becomes a dynamic

system it's dynamic because it observes its own performance behavior and cost and then auto configures the system to a lower compaction. Then it looks at errors because of lower compaction. It increases. So it becomes like a scheduler. It observes itself, orients itself. So we actually released it for them, and it's working really well, both for us and for them. You know, there's a lot to learn, but at least across two harnesses, we are seeing the dividends payback. So is there an argument certain that you can even assist the compaction because you have so much information, you can say, hey, not only should you do compaction now, but here are some key things that you should make sure you include in the compactions. Exactly. I mean, to your point, Ben, once you have fidelity of information, right, you can observe these amazing things. I'll give you three quick examples. Example one is if you leave default config, it is benefit to the harness provider,

sorry the influence provider and detriment to you because detriment because you start compounding costs and over time your accuracy of your harness loop reduces the second is all of us run agents continuously but we take a couple of coffee breaks and bathroom breaks and sometimes right the harness wants to elicit information for you so to ask hey ben you know i i'm stuck i need this particular advice from you before I proceed. And if you don't answer in a timely fashion, the cash on the harness side is also time tenured. So if the cash evicts and becomes cold, then all that data again has to build the stateful relationship between the harness and the inference provider. When we built Olki, we saw all of that. We saw that granularity of where the problems lie. So we actually built a ping system where suppose you know you interrupt you do not answer or if he keeps pinging that stateful server that way the cache stays warm and it's just like a few like you know like kilobyte

pings and then when you come back you still stick to that node in the inference provider so you talked a lot about costs and fin ops now in a previous life he spent quite a bit of time with security people. So have you tried to explore that space in the context of Olki in terms of reaching out to your contacts in the security space? I imagine they would love a tool like this, right? I'm trying to actually position it in a way where it wavs them and they use it, right? That's when, you know, the unlock happens. And by the way, Olki saved me from a couple of incidents because when I hook up a commodity router for an LLM, sometimes it reaches out to a tool that I didn't permit, I get the information at the network level saying, this particular URL was attempted, do you approve or deny? So there is a huge opportunity for security to unlock. And the way I want to position it is run it on shadow mode, observe for a week, and then start creating your policies to block.

And it should come naturally. That's how I'm hoping, where they first install it, then unlock each of these outcomes, the cost outcome, the performance outcome, accuracy outcome, and security outcome. All right. So looking ahead, next 6 to 12 months, obviously the local first is done. I'm assuming the enterprise offering is in the works. So what else is in the roadmap for the next 6 to 12 months for OLI? We are, this product came about serendipity because we have a primary product. We'll still be putting a lot of muscle in taking the primary product to market. I am testing the efficiency and the pull of this product, you know, by selectively demoing it, selectively enabling it. But as we see more need, more adoption, we are going to diffuse it and, you know, then start thinking enterprise viability controls, the ability to run within sandboxes, outside sandboxes, etc. So, you know, the enterprise landscape is still open out there, which we should figure out as we progress.

And like I mentioned, you gave a talk at the AI conference, which I will link to in the episode notes. But what was the feedback there? Very good. I got lots of pingbacks from different industries. You know, lots of services companies ping back. By services, I mean they call us managed service providers, where these are the folks who run companies, where they install harnesses, they install it up to inference, set up your companies to be successful. You know, they love this view because now they can both control remotely how a harness operates and make it, you know, viable for cost, viable for performance. And that's how they can become viable consultants. They can stay there, provide you value. I got a ping back from a few of them. I got a ping back from a couple of Fortune 500s. So I'm actively conversing with them to unlock and provide them value in their target domains. And with that, thank you, Chetan. Thank you, Ben. Always a pleasure.

Appreciate it. You can follow the work of Chetan Konicki online at oki.ai. That's O-L-K-I dot A-I. Please join the thousands of people who subscribe to our newsletter, which you can find at gradientflow.substaff.com. And we are listener and reader supported, so if you can, please become a paid subscriber. Thanks for joining us. If you like the show, please subscribe and rate us through Apple Podcasts or Overcast or TuneIn.com or Spotify and never miss an episode. The Data Exchange Podcast is a property of Gradient Flow. I'll be back next week and we will do this all over again.