ドメインエキスパートにエージェントを作らせる：Lyftのエージェント構築プラットフォーム

All right, so today we have Akshay Sharma, machine learning engineer, and Nick Ung, staff applied scientist of Lyft. So guys, welcome to the podcast. Thank you, Ben. Thank you for having us. Yeah, yeah. So the reason I have you here is that you have this new agent platform. As I understand it, in the past, when you had to build agents, it was quite an involved process, which involved a bunch of technical teams. And so the goal of this platform was to make that easier, streamline it, and more importantly, to allow some or if not most of the agents to be built by domain experts who are not technical. Is that correct? Am I correct in my high-level description? Yes, it's different. All right. So then I guess let's start there.

So building a platform for non-technical people. And we'll get to the technical stuff later on in the conversation. But building a platform for non-technical people, I guess, is both positive and negative, right? So the positive thing is they have the domain expertise. The negative thing is they may not understand data or data science, right? So I guess let's start there. So what are some of the wins as far as this platform is concerned for the personas that you targeted? I think one of the best wins is reducing the development time for our agents. That's very visible and much more prominent than other wins. So initially, as you said, we used to take a lot of time to build an agent around a product use case. Our team especially focuses on solving customer problems and customers for Lyft are drivers, riders, people who are applying to become drivers. So we had this vision in mind that we want to solve their problems, but we want to solve them through AI.

And as customer support is like one of the most prominent cases for agent tech stuff, we started building agents to solve these problems a year ago. and then we realized that we did have a lot of issues to cover we did have a lot of problems to solve but the time we took in building the agentic orchestration around every sub-agent or every specialized agent was a lot and I think initially it was a lot more because we were setting up some stuff right we were building the orchestration platform we were doing the evals we were we were just learning how to build these things and how to how to go about it so initially it was a lot more then it kind of gradually started to reduce but even then there was a lot of expertise which was involved from ml engineers software engineers who were kind of bridging the gap between the domain knowledge which the experts have who decided how the agent should behave because they had that understanding from dealing with human agents and solving customer problems before we

even started building AI agents right so we were kind of bridging the gap there we were trying to translate the human agent behavior and policies and response guides and knowledge bases to the AI agents and then there was a lot of tooling around it so there was this orchestration part and there was this tooling and there was this prompt which kind of instructed AI agent on how to behave so and there was a lot involved in tying all these components with each other and then And there was part on how to make sure that everything is working correctly. So even if we shipped like one agent to solve one or two special issues, it involved a lot of elaborate process and orchestration and the development lifecycle. So Nick, since you're a data scientist, right? So now you're having people who may not know that much about data or stats or ML. They're building agents. So what are some of the pitfalls and some of the things that you had to make sure was available in this platform to make it easy for them?

Because these are people who may not know EVAP. Yeah. Right? So I think you're right. I think a lot of time when even this is true for like product managers as well, right? people that are not from the data domain. Their mental model, when they go into building an agent, is that they know they're, you know, for specific customer issues, they know, oh, these are the five golden paths. And then there are maybe one or two edge cases they want to cover. And they, you know, as part of, you know, our internal testing framework, we have like simulator app that allows users to test on these flow, right? So their approach to eval is, hey, I know these are the golden paths. I'm going to write a prompt and I'm just going to test on the app to see if it works. So the goal is this is really some sort of self-serve platform in the sense that they're doing that themselves. They are doing that, but we have a lot of, we put some guardrail in place to not just let them do that, you know, without any approval.

You know, we are going over all the prompts before we, you know, let them, let these prompts hit staging or even productions, right? So there's usually an engineer-in-the-loop process. We tell them, hey, these are all the MCP tools that this agent has access to. Do they even know what MCP is? Obviously, we don't tell them what MCP is. We just tell them, hey, the agent can look out. If the agent needs to understand if this write was canceled, we just tell them, hey, if the agent has access to this, you can just reference that tool name in your prompt, and then the agent will look it up. And we don't, obviously, we don't tell them what MCP is over-ended. You know, there's no point explaining to them what that is as well. But kind of building that, like, share language of, you know, these are the list of tools, these are the list of actions the agent can look up, and how do you plug that in the prompt, right? And, you know, if given in a specific user journey, if we hit a point where the AI agent was not supposed to handle that specific issues,

Maybe that was safety-related. Maybe that was fraud or deactivation-related issues for driver. AI agent is not supposed to handle that. How do you then instruct the agent to escalate to a human agent's right? So we kind of built these small building blocks in a way where the domain expert can understand. So they can go and build these initial versions of the prompt right? Or then for an engineer to review, make sure they don't make any mistake. And I think one of the quality projects that we are doing right now is like a prompt lender. We have these deterministic rules to like check whether or not, hey, did I make a mistake in referencing the wrong tool name? Did I make any other kind of like quality mistake right in the prompt? So I think there is something that we can try to catch with a prompt lender, but we still have an engineer in the loop process to help like double check if the prompt is actually written in the way that we intend it to be. So a couple of questions. So the first is give us a sense of the level of complexity of agents that domain experts are building.

I mean, I'm assuming they're pretty simple agents whose whose workflow is something they can understand. Right. So it's not some long horizon agent with complicated branching and things like that. Right. So what is the is there like an ideal sort of agent for these people or are these mostly internal? or customer-facing agents? These are actually customer-facing agents. Thinking about the type of customer support issue that Liff handles, right, there's generally a range of, like, low complexity to high complexity, and we're pretty opinionated that, you know, like, for more long horizon, like, I mean, none of this is going to be as, like, long horizon as, like, coding agents. To be fair, like, an agent is not going to go to work for, like, two hours and then come back to you with the results. But there is, like, more complex customer issue, For example, if the driver submitted a damage claim, they're saying that the rider damaged their car and they need to submit photo and the AI agent needs to analyze the photo.

So the domain experts who could be product people, so they're building kind of customer-facing agents. So they're not necessarily building internal agents. I mean, because let's face it, even product people might have some complicated tasks internally, right? So they're not building agents to automate their internal tasks. They're building agents that interact with customers. Yeah, they're building agents to interact with customers. So far, we've asked them to start with a lower complexity issue. Like one of the less complex ones. Sorry to interrupt you. So the logic here is that you have them build it because they understand it better than the engineers, right? Yeah, exactly. These are people that they either handle customer issues themselves or they are the ones that are literally looking at our tickets every day. They're manually reviewing the tickets. They know the customer issues very well. They know how customers want their issue to be resolved, right?

And this is where we kind of take ourselves out as the bottleneck, right? Like we, and the engineers, we don't want to be looking at tickets all day. But one question is, how do these people know Nick and Asha? How do they know when the agent that they've developed using this platform is ready to deploy to production? How do they know if it's good enough that they've done enough evals that there's, there's i'm assuming you guys are able to test for edge cases and things like that so how do these non-experts know that as nick said we like autonomy comes with with some guardrails right so we we are giving them freedom to to write prompts and to also define the behavior the ai agent should should follow but we do have guardrails and guidelines both for prompts and also the agent itself so when we are writing prompts as as nick said uh we have something called prompt

linter and it has both the components it has deterministic checks to validate the syntax the structure the tool references and stuff like that and then we also have an llm which is checking the prompt for inconsistent behavior in different phases uh every phase should have like an exit condition and and things like that so all the things which are kind of non-deterministic and behavioral llms are going to check that but once the prompt is ready an ml engineer data scientist or software engineer is part of the review process yeah so we we have that review process in place and let's say once once the prompt is ready to be deployed and all the tools and all the orchestration is already in place to make it function in production we have these evals running so we have offline evals and we have online evals for offline evals we this is something which we do before launching an agent into production actually i guess my question is so you have the evals but the non-experts are not looking at the results of the evals it's the

the team that supports them the data scientists and engineers is that right yeah yeah so we we do have like data scientists involved in publishing or deploying an agent to production so we so like I think some of the people domain experts they can look at the metrics and they understand that okay some things which are tied to business some things which are tied to like escalations and I guess the the reason I ask is obviously if let's say I'm a product person but in earlier in my career i was a data scientist so it's not like i'm not aware of some of the basics right so do you make a distinction between people who have a little more background than the others as far as their ability to grow faster yeah i think i can i can take this question so i think ben i think there are like a few different way of doing evals here like i think there is definitely the data driven approach where you get a sample data set you know maybe the initial verbatim and then you

run it through you have deterministic checker you have lm judge yeah historical traces historical traces right and then we score them and then understand you know whether a change is going to impact uh certain metrics that we care about right but typically for domain expert what we strictly refer as domain expert here then you know non-technical they don't have data science background they might not even have like technical background right so usually what we ask them to do is mostly we deploy the problem to staging. We ask them to, you know, use our app to test a couple of flow that they, like the golden path, the couple edge cases. And then we have, you know, data science sort of in the loop, right, to help. A second check, which is the data-driven eval approach, right? So I would say we have like multiple gates that we, you know, that an agent needs to agent deployment needs to go through before you can actually reach productions right And I think you know each person different persona like do different things well i think domain expert they do like these

manual manual check manual testing really well and analysts and data scientists will come in then and do like more data-driven eval we run our offline eval to check if everything looks good and if everything looks promising then we can try try to deploy to production then and depend on the size sort of change. We might run an online A-B test. We might not. It's definitely a little bit more of a case-by-case basis. So the types of agents, it sounds like that we're talking about here, are agents that interact with customers, but they're basically replacing or automating certain existing customer-facing offerings, right? So in other words, it's something where, as Nick describes you have a golden path so it's something that the non-technical people have seen a lot of previous customer interactions of so then they can go ahead and do that but as far as inventing something new customer facing that's not happening yet so i think it's it's both it is also the first

part is like the the interactions coming from human agents or human escalations to reduce the cost and we are moving them towards ai agents so that we can we can automate things we can uh response and resolve things quicker with much less cost and much less resolution time using ai agents but there are also some things then which which is very difficult for human agents to do like let's say if we have like uh nick was talking about damage so i can i can take that example uh drivers are submitting photos as proofs that someone has damaged their car right so i think for human agents to go into their history on on checking like how many past claims they had what is their driving history are the images which are uploaded fraudulent or not uh do they have like the metadata which we think uh which is like good enough proof that these are legit and also some things like uh when was the photo uploaded what's the driving history was the photo uploaded even before the ride happened and things like that so it becomes like very complicated when we ask

human agents to do that and before this it used to take a lot of time and there were a lot of gaps which we couldn't like fill in so now with the ai i think those become simple so the innovation is handling issues much better and having these actions and components into picture which can reduce fraud and which can automate and make a lot of things quicker which we weren't even able to do with the help of human agents. So it's interesting. So the non-technical people who are authoring these agents mainly represent the agent. So the main thing they have to come up with are prompts. Is that correct? That's the main thing, yes. Yeah, yeah, yeah. Interesting. So there's no visual tool where you go, okay, so, you know, like a workflow where you sketch out the thing and so it's all just text, right? So you're describing something. And then in the text, you are describing in as much detail what the agent is going to do,

the sequence of things that might happen. What about, I don't know if you have to worry about this, but things like LTL, linear temporal logic, right? So things like, by that I mean always do X before Y or never do Z once A has happened. I mean, I guess someone can write a prompt and you can just automatically kind of discern the temporal logic that needs to be enforced. Right. So then you, based on the prompt and the tasks of the agent, you kind of extract the linear temporal logic that must be enforced in order for this to be correct. right yes i think i'll also let nick speak but i think this is a good good issue and we do have these checks where we can deterministically check that if if let's say our tool is like a prerequisite for another tool to call let's say we we want to have like a ride id and ride information to get

sorry ride id to get ride information and things like driver id to get driver information so these are some things which we resolve deterministically and the temporal logic which you which you are talking about i think it is enforced by deterministic checks i know you we can we can that example you said right so maybe i have to look up a certain database first get that information before i proceed right right so i mean like like you said in the prompts we can write like you have to get the right id first and then look up right information and then like now now the non-technical expert actually the non-technical person won't be able to do that right do that yes so in what happens in things like this is we have tools like let's say we we need to get ride information and that tool itself has a doc string which tells it that like ride id is a prerequisite for this tool as an example so what it does is is let's say there's like an action which gets which shows

the user a ride selector who selects the ride and then we can get the ride id from the agentic state so that that tool will again call the llm node and it will instruct that we don't have ride id for this can you please ask ride id to the user so it will go and show the ride selector to the user and collect ride id and then call the tool again but the point the point is this is all automatically discerned from the prompt yes sometimes like when things are very dynamic and some things which are not captured by the prompt behavior, we have to do these kind of things. But mostly for the responses or for the workflows, which are very straightforward, I think we can put most of the things in the prompt. And then for the hallucinations, I think we can have the guardrails and deterministic checks. Is there a role for, you know, one of the fashionable things right now is formal verification, right? So is there a role for formal verification in some of these agents? Can you elaborate a little bit more on that, not too familiar with that body of work?

Oh, so basically, if you can break down what an agent is doing in a way that's verifiable, you can formally verify it using, you know, some people use Lean, for example, you know, the TR approver. Maybe that's too advanced. I'll skip it. All right. So, okay. So one of the things that you folks do, of course, is you have a router, right? So actually broader means obviously you're using multiple models. So I don't know how comfortable you folks are if you're allowed to talk about the suite of models you use. So I'm assuming there's commercial proprietary models. But do you guys use open rates models? So we are in the process of exploring that. Not yet. Not yet. this is happening i think like probably all the companies who are using lms that they have realized that the cost is going up a lot and we are trying to identify places where it's kind of an overkill

to use uh frontier lms and we can replace them with open weight models now that they'll become a lot more capable within last six to eight months so we do explore these situations where we do not have a lot of complicated decision making so we can use open weight models and we are also looking to fine-tune some of the open-weight models for our very specific use cases which which contains a lot of policy learning which is which means like the case is narrow and niche and we don't want like a generic frontier LLM for for that but we can probably tune fine-tune like a smaller open-weight model to handle those kind of scenarios. So right now the the routing you're doing is just deciding between what tools so the routing routing right now decides so it basically it's like a filter where which understands what the issue is uh for the customer so which agent to point it to which agent to point it to and if let's say if we don't have an ai agent to

handle that kind of problem we either it can be anything it can be like someone is asking a question about like a tutorial or how to like follow some steps in the app they they can be asking about like like a special use case like a refund ride refund and things like that they can also be asking about something which cannot be handled by ai right now and we identify all these things and based on what the problem is what the issue is we kind of divert that to either a specialized ai agent or an educational response from our knowledge base using rag or escalate than to human agents if you can't handle it. So, Nick, I'm assuming you paid attention to all the hullabaloo around JEV. A little bit, yeah. So I don't know what to call it. So I guess I'm calling those things the decider, right? So it's trying to decide using probability or some score. I mean, it'll give you a probability or a score, and then you can decide what to do next. So do you see a role for that in what you folks do based on what Akshay just described here?

I think there is a place, definitely a place for that as well. I think that is, like, Jeff is essentially a universal router, right? Whether you're trying to, like, point a certain issue to a certain agent or, like, who... But the point is, it's not a chat bot. It's not going to generate text. Right? so it's simply just helping you decide what to do next that is kind of the reasoning piece of it one of the things that I which by the way is basically just going back to old school old school classifiers which is interesting because we talk about this Asian router that we have to decide which Asian to point to In that decision point itself, we also have to determine, oh, did the user give us enough information to make that routing decisions? So our current flow is actually both a kind of a classifier as well as a chatbot.

If a user is just saying they're frustrated, they say something nonsensical, we don't know which agent to route to, we have to prompt them to say, oh, can you tell me more about that? right so it's it's a little bit of it the concept is like both right like you have to like know oh like i don't have enough information to route i need to like stand like i need to go go through a generations model to like generate a response to the end user right and then at a certain point that you know oh i have enough information to route then you can you can route to the downstream agent but i think what jeff kind of comes in is oh like you can make the decision point of like oh i don't have enough information i need to call a generation model or i have enough information i can route to a multi-agent model but i think it's sort of like we we are now just using a punty l and to like do both right i think there's like more cost efficiency or like lower lower latency that we can achieve by like replacing some of those steps with jeff and i think like another thing where we can use dev is for llm judges like instead of llm judges for evaluations

we can use dev because it has like very structured output like a like a boolean or like a score which is very useful when we are trying to evaluate a metric from the responses and because it's very fast and a lot cheaper than frontier llms it becomes like a very good model to use for evaluations because evaluations doing evaluations with llms is very expensive so everyone wants to do them but the cost and like how much to sample and like what frequency we do we need to do that becomes a bottleneck. But I think Jeff can solve those issues. By the way, there's already a ton of open source versions of Jeff. So if you go down that path, you might as well check out the open source versions. That way you control your data, right? Why send it to another API provider? On another topic, so I use one of your agents, right? And as Nick alluded to, so let's say the prompt from the customer is actually something that an agent can address.

But then maybe the agent has, and I'm just speculating here, are there scenarios where the agent actually has sub-agents, where the agent decides, hey, I myself, I'm not, I can partially do this task, but to complete this task, let me call this other agent. Yeah, we do have these kind of cases. So I think like sub-agents, we have like two different types of sub-agents when we call sub-agents. And by the way, so when you get to the sub, you got an agent calling a sub-agent. It's just still in the realm of an application that the non-technical person created? Or is this already something more complicated that an ML or a data scientist should do Yeah we do have those flows in like more complicated agents The agents which are built by domain experts are right now they are like simple react loops So we have like an LLM and we have tools and we have like a loop between them based on what the situation is.

We are thinking of expanding those horizons by making expawning dynamic agents with different contexts if the issue becomes complicated. But yeah, right now we don't. Right now we have simpler agents for the configurable or for the agent builder platform. But we have big and complex agents which have different phases, which can spawn sub-agents based on a situation with limited scope and context and then solve those issues is reserved for the more complicated ones. So Nick, what about safety? So I guess I don't know if this ever happens at Lyft, but maybe the customer sends a malicious prompt to try to extract information. I don't know. I mean, I'm just making it up. So what about sex for safety and safety related issues? Yeah, I think you're probably more referring to like malicious, you know, malicious actor, right? Who are trying to do like thumb injections or some other type of.

Or someone, I mean, you can imagine a customer opening a complaint and then I don't know how that turns into an attack or maybe they just start threatening the chat bot. Yeah, I think they can also ask like exposing internal policy. Yeah, it can be anything. Yeah, sure. Yeah, I think we have, this is what we generally refer to as like guardrail. we have like these malicious intent check which is essentially Ellen is a judge, right? Which we, you know, calibrate the prompt on, you know, we have basically following like traditional ML best practices, right? Having a set of data, trend, validation, test set, calibrate this malicious detector or malicious classifier, right? That basically runs on every user input message. Well, so this is just part of the platform? Yeah, this is built in. This is not even something that we ask the domain experts to do, right?

All the different gut rail checks, malicious detector, hallucination detector, these are just ML-provided capability that comes with the platform, like the domain experts don't need to worry about it. So, Akshay alluded to RAG, so grounding the agent. So, I guess, as best I can tell, there's three ways, right? So there's the context in the prompt, there's memory, and then there's actual knowledge, enterprise knowledge, which could be, you know, a database, could be a knowledge graph, right? So I guess the types of agents you have must have some memory, right? At least in the conversation, right? Exactly, exactly. So like all our customer support agents are like multi-term conversations, right? So we definitely need memory. we need to maintain the agent state so that based on what the situation is it can make decisions and we also have and and this is this is like the checkpoint uh if you don't mind me asking so what's

the what's your memory what's the what what's the do you have your own solution or you you we have our own solution so we for checkpointing which is like stopping and resuming conversations we use like our internal DynamoDB. We have like adapters on top of it, but we use DynamoDB for it because it needs to be fast and efficient. For maintaining long-term memories and facts and these kinds of things, we have different databases like S3 and like SQL tables internally. For the enterprise knowledge, which will include RAG, do you use knowledge graphs or just... We use vector databases for that. So we... Which one? Which one? so so we are we are using uh like something from aws but we are even that is like in process of improvement so we are looking at our chunking strategies if we can create like uh semantic embeddings on top of it so that we are able to retrieve for some reason i thought you guys use

ray and lance db and no you don't you use ray no i think we've like our platform team probably uses the ray for uh for like the fine-tuning pipeline but yeah we we don't use the ray for for the rag all right let's do a deep dive on eval so nick started to allude to this so so walk me through the different kind of stages of eval i think eval is interesting right i think this is something that i've noticed a lot in people that are building agents today right and And especially engineers are taking the lead on building agents today. A lot of time people don't really follow ML best practices. Well, they're engineers, man. Yeah, they're engineers. So, you know, that's kind of like, I think the principle that I will always preach in AI Eval is that like, it's not that different from ML engineering or not ML engineering, but traditional and best ML best practices, right? Or data science best practices.

you wouldn't ship a neural network blind without any offline validations or offline tests, right? But I think that the challenges with, especially with these customer-facing use case and product is that, you know, oftentimes the offline eval is not that straightforward, right? It's not like a regression task or like a classification task where you gather a set of inputs And then you have labels that you can just run offline eval over and over again as you tune the model. And with these customer-facing use cases, we sort of then have to understand, how do we recreate that production-like environment when you're running each offline eval task, where you're actually talking to a user and finishing the entire trajectory and traces, and then which thing you will be able to score. right so do you have a don't just aren't you starting to collect like an eval data set of some sort for each of these agents and i guess second question is uh do you use synthetic data

simulations yeah so i think the way that we're thinking about this is like there is a task specifications right like oh like thinking about each eval as a task specifications or a configuration file, you have the initial user pervading by how that conversation is being started. And then you have like some characters stick around, you know, what the issues are, what tool is available, what are some of the data points that are, you know, true for this particular user, right? You have to kind of recreate that whole scenario of this customer support environment. And then you, what we do, what we do initially is like, obviously we kind of curate this scenario from some production data and some of it are through like syntactic data right that we kind of just generated with other element model it's sort of kind of a mix of both like we we do need to like and it's not a static thing because you obviously these agents are interacting with real world customers so you'll keep improving that data set right yeah we'll curate you know

add more data points right those are we things are like representative of what good look like challenging tasks, right? So then we built this offline environment where, you know, given each of these task specifications, how do we replay that full environment, right? And interactions between our LN agents as well as with a syntactic user that kind of speaks somewhat like a Lyft user, right? Like if you just kind of use a frontier LN model to like role play your Lyft user, it's going to set up something like really polite and your eval would be very, very, very easy, right? Like in real life customer support setting, people are not usually that easy. But also these agents, Nick, might be doing things, right? So might be looking up things in a database. And so there's, in other words, there's a whole harness, right? So there might be using RAG or something. So you almost have to have all these, I mean, production traces almost, right? Yeah, so there's a task specification about what the task is

and what are some of the tool calls, mock output for some of the tool calls. And then there's that harness of, how do you hook out all these different other components so that you can reproduce and replay an entire run. So there's a lot of scaffolding to be able to set this eval environment out properly, which is also why we also think, most people kind of skip this step and go to productions and just try to understand what that is. And because you have these historical traces, right, there are certain interactions where you have the desired outcome and resolution, but it turns out that the agent was very inefficient. In other words, maybe the agent asked too many questions, checked the wrong databases, had to go back to the customer. And then so how do you discern that from the production traces to improve the agent so that the next time the situation comes up, boom, done in one shot. This is when, you know, judging is important.

How do you score traces? Your evaluator then becomes a lot important. Like if you can clearly separate the signal of, you know, like, I mean, conceptually, we all know that if user LM agent is asking too many questions, look up the wrong database. it's not going to lead to like a good resolution. But how do you actually judge that? Sometimes it will, you know. You know, when you interact with a customer service person, sometimes it takes them a while, right? So it's the same thing with an agent, right? So an agent could be, you know, the first thing the agent looks up is wrong and then the second thing is right, right? So I'm not sure if you have mechanisms to discern, you know, just because the outcome was achieved, maybe this this was not the most efficient path right so yeah i think like only by judging the output so it shouldn't be like an ideal way to to evaluate so we do have like trajectories which should have been followed for by an agent if we if it's trying to solve a customer issue and

if it if there are like wrong tool calls in between or like wrong database read in between we have specifically for those two calls we have these checks okay this was this tool called correctly or was it with incorrect parameters and then went back to the llm nodes and then was called again so we we do have these sort of metrics in when we when we are trying to judge agents so and when when you talk about identifying patterns right identifying signals from production traces i think it's a difficult task like we do have to have some some manual intervention in between to do that right so like data scientists can help in in those aspects and where where we can identify some patterns okay it's like how we do eda in in ml like doing like some sort of error analysis of what what are the patterns what are the common issues we observe when we look at the production traces so i think evals should start with that first step you will only know your actual issues once you look at the production traces so there is some like data science involvement for

understanding and identifying the patterns and then from there we try to convert them into like metrics or like evaluation prompts which we want to actually evaluate in production for these for these agents and then we see like how these are performing and what we need to improve based on these metrics scores uh which we see in production yeah it's it's like the production traces and production trajectories become a form of training data exactly yeah traces are the the the most important thing i think for agent stuff so they are the source of truth for any kind of data if you want to do fine tuning continual improvement judging anything everything anything and everything comes from like traces so i think those those are the most important things so what what about the harness that's also internal that is internal yes you're not using codex or prod code no so we We have used LangGraph, which is like an open source framework by LangChain, but we build our own harness.

So we use nodes, we use like graph and subgraph patterns. We have like our own state machine. We have like our own state schema. So everything around it is like internal. The harness is built by us. So the eval right now, do you consider eval as a piece of infrastructure, shared infrastructure, or is it still mostly kind of ad hoc? I think it's definitely a shared infrastructure and it's becoming more and more important because as agents are becoming more and more and more complex, it's becoming difficult to identify issues and then solve them. And anyone who's working on multiple agents or like a multi-agent architecture, even within a company, multiple teams are working on different agents. Evaluation platform can definitely benefit all of them. So it can be like a platform layer which can be tied to like a data curation thing with your own traces from production And then it can be tied to annotations It can be tied to offline evals

It can be tied to LLM judges and even like if you want to do any fine tuning later on. So, Nick, has there been a noticeable improvement in terms of this eval infrastructure in the following sense? In the early days of this agent platform, you would deploy agents and then withdraw them in two days because for some reason they weren't hardened enough. But now, because the eval infrastructure is much better, the agents tend to be much better off the bat. Yeah, I think that is sort of the evolution of our eval journey as well, right? As these ideas of trajectory, eval harnesses, and task specification really solidify, right? We also built this shared infrastructures where different teams can build upon these infrastructure layer events. Some of our test engineers are able to contribute to our test eval use cases, right?

Getting a shared understanding of how we, where and how we run these evals, certainly I think it helps streamline the end-to-end agent development lifecycle, right? Whereas before, you know, when we first started, it was like we kind of do some testing and staging. We kind of feel good about it. We wrote it out to productions and a bunch of people will be, you know, all hands on back looking at traces that are coming in to see if it's doing any good. But I think the beautiful part about eval is that, you know, we get this first-hand experience of, you know, what a good eval is. And then we slowly automate these different parts of it to make our life easier, like building these automated observability layers, like improvement layers to help, you know, the next agent get better. All right. So we're winding down. Let's do this quick round of tips for other teams. building either agents or tools for an agent platform just like yours right so let's do two tips each but one at a time so you don't have to stick with your area so actually doesn't have

to only stick with infrastructure and and nick doesn't have to only stick with ml so actually you start tip number one i think tip number one i would say that prompt quality probably is like more important than you think now that and like as agents have evolved in multiple generations right initially we had like single shot agents then we had this complex multi-turn agents with our own orchestration now people are moving this orchestration piece to the llm so now everything is kind of controlled by these frontier llms uh what decision to make what sub agent to pawn or like which tool to call even creating like multiple sub-agents in parallel to work on different things and and stuff like that so i think now that a lot of these things are very abstracted from us like even the low level orchestration it it becomes more and more important to focus on how you write the prompt and the difference between like a good engineer or a bad engineer if we have like access to all the same tools is is how you use your own brain

and reviewing things and how you write verbatims or write the prompt to make or to make use of that ai is i think is the difference does something like dspy help i haven't used it personally but yeah probably yeah nick you tip number two tip number two um i think you know as i as i said again i think you eval is your competitive advantage right i think a lot of teams are maybe initially looking for that product market fit and you know eval wasn't so much of a focus at first which is sort of totally understandable there are a bunch of you know series a and series b are trying to make it right but i think as you really think about okay what next ray like how do you continue to improve your product like without eval it is just create create a blind spot right so i think eval is non-negotiable it's a little bit of a art to get it right in the first place But really thinking about deeply investing in your eval infrastructure is how you go a long way.

And what are some resources if people want to learn how to do this? Do you happen to know any good resources? I think especially for product team, I highly recommend Hemel, Hemel Hussain and Shreya Shatka. I think they've been doing this for a while. they have a course well I'm not marketing for them but I think they share a lot of good insights and blog posts about how how to approach you know eval especially as you're building like a zero to one product in the beginning both have been on the podcast both have been on the podcast yeah a big fan of them I think the blog posts are great and yeah you know that has also done and the folks at Lan Chen have a really great philosophy on eval as well highly recommend checking out there uh actually tip number three uh tip number three i think i would say let's not blindly follow what other companies like this ai space is so dynamic and it's changing every day new models are coming

in new products are coming in people are building application layers on top of llms so every day you hear things like drag is dead evals is dead and things like that but i think it's just not like blindly go with the flow understand your own problem because different things have like different teams and different companies have different use cases with different criticality and i think it's important to to stick to the basics and then have a vision how is your infrastructure or how is your agentic platform is going to look like one year from now two years from now if we cannot envision that and we cannot build like a scalable platform to support that i think that's the gap we need to we need to fill nick tip number four i think one thing that we learned early on in our journey is you know be as flexible as you can with your with your tech stack and platform and don't introduce you know lock in too early you know one of the reasons why we use like overall using langraph lanshan is that you know it's overall model agnostic you know bring

your own model type of framework, right? And, you know, there was a lot of other agent SDK early on that were very popular, right? OpenAI, SDK, Cloud sort of have their own as well. I think the reality is that, you know, agent engineering is simpler than it looks. So you don't need to go with a solution that abstracts too much of the detail, right? Like owning that low-level layer is actually also part of your competitive advantage, right? Like if you want to do like model customizations in the future, keeping your tech stack open and off-stack is sort of a way to go. All right. So I'm going to close with my own two tips and have you guys either agree with me or tell me I'm crazy. All right. So the first one is, I guess, assume that your agent will make mistakes. And that means you have to design for some sort of recovery from mistakes. That could be checkpoints. that could be some other form so what do you think about that yeah definitely agree with you ben so

doesn't matter how good the the llms will become or llm models will become there's definitely always a scope of error and it can be because of the prompt is not well written or it can be because of the tool doc string is not it and it can be like because of a million reasons so definitely assume that they're going to be mistakes it cannot be perfect so build your guardrails around it and focus on evaluation so that you can build a robust system if the use case is critical especially tip number two is to is related uh they're related but there's actually two things in here one is give the agent only as much autonomy as it needs right so you don't want in other words keep them I'm a little bit more boxed in. I think this is, I'm not comfortable with OpenClaw running on my laptop. And then also, don't confuse your agents with too many tools, right? So, in other words, if you have multiple search engines,

multiple ways of doing the same thing, the agent will just get confused. So, what do you think about that, Nick? I mean, actually, it's probably got a lot to say about this, but this is why we have progressive disclosure, right? This is why we only give the agent as much information as it needs at the given point of time. And I think there is also some, I think actually there's some study out there, you know, the model performance is great when you have like a certain amount of tool, right? When like the context rod is definitely real. I think your first point about like only give the agent as much autonomy as it needs, I think I'm more of a proponent of like giving more autonomy and, you know, box it into agents in. And I think you'll be surprised by... Your agents may end up exploring hugging face, right? Well, I mean, I think obviously we don't want it to do that and we don't want it to delete records from our database. And I think that's where, you know, we will limit that. But I think what I'm trying to say is I think early on, you know, especially in the early day of like LN product,

you know, development, you know, product managers sort of coming with these like strict workflow and Figma rule, right? And it's not an AI agent, it's just agentic workflow. There's a very specific, you know, logic branch in how VM1s the products work. And then I think that sort of, like, limits the ability of the agents to, like, handle the non-golden path, right? And I think it creates a lot of performance issues or, like, quality issues in production. So I just don't want people to think, like, when you design an AI agent, it needs to, like, strictly follow if A, then B, and then C, right? one more thing which is I guess when it comes to multi-agents keep it small because I think right now most teams are not yet at the point where I think multi-agents to me is a bit of an advanced thing a lot of things can go wrong for one thing there's the communication between the agents right so and then talk about temporal logic right so the sequence of events so I'm not sure how much multi-agents you folks use but yeah i'm a little wary of i think learn to walk before you

run i guess is my tip when it comes to multi-agents what do you think actually i think that's a that's a good tip ben so our team does a lot of multi-agent architecture and i think in current industry there are ways to efficiently do the communication i just i just don't want people to like overdo it so that like there's like a bloating in your check pointing table or like a context bloat for agents who hallucinate or confuse about so i think you can build multi-agent architectures for use cases like customer support but beware of like all the consequences and make sure that you have all the checks in place like you are doing context summarization or truncating to to prevent bloating you have progressive disclosures for tools you have some skill so that's why we have skills right so skills will tell you like which tools to use and not introduce like 100 tools to the to the agent in the beginning make sure like what which part of the state you want to share between agents and which part of the state you shouldn't share so things like that i

think there's like a lot of design principles which come into picture but yeah if you if you make sure that those are in line with your expectations and your robust design, I think then it becomes easier to build a multi-agent architecture. And in closing, I like what Nick said about eval. The one thing I will add is I think a lot of people confuse eval with evaling the model, but it's really the entire harness, right? So it's everything. Basically, your agent might be retrieving things from a database. be doing all sorts of things. It's not just the model, it's the entire harness. Yeah, the teams that get really good at eval I think are the teams that will ultimately win. Couldn't agree more. Yeah. And with that, thank you Akshay and Nick. Thank you Ben for having us. Thank you so much Ben. You can follow the work of Akshay Sharma and Nick Ung online at lyft.com.

Please join the thousands of people who subscribe to our newsletter, which you can find at gradientflow.substack.com. And we are listener and reader supported, so if you can, please become a paid subscriber. Thanks for joining us. If you like the show, please subscribe and rate us through Apple Podcasts or Overcast or TuneIn.com or Spotify and never miss an episode. The Data Exchange Podcast is the property of Gradient Flow. And I'll be back next week and we will do this all over again.