正解データが変わり続ける中でRAGをどう評価するか

Hello and welcome to the AI Engineering Podcast, your guide to the fast-moving world of building scalable and maintainable AI systems. Your host is Tobias Macy, and today I'm interviewing Ofer Mandelovic about how to evaluate your RAG systems when you have incomplete or changing data to grade against. So, Ofer, can you start by introducing yourself? Hi, thanks, Tobias. Great to be here. I was a machine learning and kind of recently deep learning leader here in the Bay Area for the last maybe two decades. I actually started my career in machine learning. I had a fun story to tell about this, which is when I started, Naive Base was a leading algorithm at the time. I don't know if anybody in the audience knows what that is, but it's probably the first machine learning algorithm ever created that I've known of. and nobody uses it today, of course, because it's so old.

But, yeah, so I've been around for a while from the time that we called it machine learning to data science to now we call everything AI. In addition to that, I was sort of in engineering too, so part of the engineering stack. I was a week of engineering in a couple of startups in Silicon Valley, so I've seen a lot of the ups and downs of the Valley over the last two decades, yeah. More recently, I've been involved in some efforts in also developer relations, And for a company called Bend and other companies. And wrote a book actually with O'Reilly recently called Hands-On RAG for Production, which I think we're going to talk a little bit about today too. It kind of tries to summarize all of my experiences in bringing AI systems, RAG, and today agents into production and what that means and all the challenges that are part of that. So you mentioned that you've been in the space for a long time. I'm wondering if you can start by giving a bit of an overview of how you first got started in the overall field of what used to be machine learning and has now been subsumed by AI and what it is that has kept you here.

Yeah, so I actually started with machine learning in my first startup, which was founded in 2000, more or less. And as I mentioned earlier, we did a very elaborate system. It's funny to think about it these days that did document classification. It was called Quiver. The company later was acquired by another company called Infamy. But at the time, it was all new. People did machine learning with only PhDs. That's the only way you could do it. You had to have this very specialized expertise. But that's how I got started. I started a company, and we required this classification engine to do our product, to implement our product. And I just learned on the job, essentially. Then this, of course, was very interesting to me. So I kept in this field. Then things evolved into decision trees and what we know now as XGBoost and all these other algorithms that evolved over the years.

And around 2018, 2019, I actually had my first meeting with the LM space. So a friend of mine who used to work at OpenAI in the past told me to take a look at this, and I was very fortunate that I had this friend. And this was the time when GPT was GPT-1, and GPT-2 just came out early 2019. For those of you who haven't seen that, there's a whole set of articles published by OpenAI. I know those years, it's really funny because they were funny and interesting. They released GPT-2 in three phases. They had the small, the medium, and the large. and they had extra large. They decided not to release. Now, I'm saying it's funny because the extra large, which was deemed very dangerous, was about 1.5 billion parameters, which is kind of laughable in today's terms. But I think they had the right idea of thinking about alignment and thinking about protecting because nobody knew what this would do.

Anyway, I used this for another startup that I started at the time. In healthcare, we used GPT-2. We trained it on healthcare data and fine-tuned it. Yeah, and that's kind of been my journey from the early days until today. In the process, of course, I've learned a lot about how to build machine learning models, how to release them to production, how to test them in production to ensure that they don't regress in their performance and in different fields. So it's been in financial services, in travel, in healthcare, and other industries. Now, digging into the topic at hand, as you mentioned, you wrote a book on the topic of RAG and bringing it into production. And RAG as a concept was one of the first ones that really gained a lot of attention as LLMs were exploding onto the scene because it was something that was fairly straightforward and easy to engineer because we didn't have all the frameworks. We didn't have all of this concept of agentic capabilities and the models themselves weren't really strong enough to do much with that.

But we understood how to do some of that few-shot learning where you do the in-context parameterization of providing some examples so that the LLM could then synthesize that. And I'm wondering if you can just talk to what you see now that we're a few years in, the role that RAG is playing in some of the actual systems that people are running. and now that we've moved beyond it being a toy and just an easy way to get a quick demo running to something that people are relying on for actual production applications? Sure, yes. So about two, two and a half years ago, you're absolutely right. RAG was the hottest thing, right? And let me clarify a little bit the reason for that because we started by having LLMs and they were answering questions really well. And the main challenge or the main opportunity, I guess, also with RAG was I want to connect it to my enterprise data. I want to connect it to data that's documents on Google Drive or on S3 or in Notion or in anywhere like that.

And the LM doesn't know about your private data. So the main reason to do RAG was to ingest the data from these data sources, prepare them for retrieval so that you can use retrieval as part of RAG. That's the R in RAG. It's retrieval augmented generation, right? So the R is the prepare the data for semantic search, and then retrieve it at query time so you could use it inside of RAC. And the RAC was a simple idea. It was a prompt that said, you gave the LN a prompt that said, here's my question, and then here's some relevant documents or chunks of documents that might have relevant data. And the end of the prompt was, please answer this question given these documents that are relevant. And the LM will process everything, will summarize those documents, and sort of give you the right answer. And you're right, too, Tobias, it was quite magical. A lot of people said, like, there was a time we were like, oh, look at me.

Talking to my PDF was like a magical sort of simple use case that everybody really liked. And it was easy to do. I mean, you had a vector database. It was like, at some point, like, I think 50 of those available, both commercial and open source. It was an explosion of vector databases, and you just did some chunking of the documents, parsing the documents, checking the documents, put them in, even in memory, because it was a small demo, and it created a new capability that people really liked. Instead of reading the documents for 100 pages, they could really, like, ask a question, which was really nice. Now, that is very different than doing this in a real production environment, right? and I think that's where companies that are that build right systems, like I worked at one company called Victara, which is where I wrote this book as well, has done a lot to create that retrieval and query pipeline to be much stronger and much more useful. Let me talk a little bit about where some of the challenges are in that.

So first, I mentioned the ingestion. So you ingest the data into a semantic search engine, like a vector database. And just in itself, when you think about it, at scale, is much more difficult. Let me highlight a couple example reasons, right? There's big companies who have PDFs that are, and I've seen one of those, 15,000 pages of a PDF, right? So thinking about that scale, like you run something in Python, let's say, and it could take, I don't know, hours to just process the PDF, and maybe in the last two pages, it runs with an error and it fails and you have to try again. So you have to think about sort of normal engineering best practices, right, of how do you tackle really large files so it doesn't go out of memory and you do it in small batches or something like that. The other thing is just the number of files. I mean, it could be a million files, right, or 20 million files. It doesn't have to be files, by the way. It could be Notion documents. It could be Jira tickets.

It could be, you know, stuff from SharePoint or whenever. But again, if you have a lot of these documents, you have to build a production system that not only ingests all those documents for the first time, which again could take days or weeks and you have to parallelize it maybe and build retries in case some documents run into errors. And you also have to build things for refresh, right? If I have, let's say, a Google Drive source of data and I ingest all the files there, new files come in every day or files get changed. How do you refresh without re-ingesting all the files, right? And then the last thing I want to mention on Ingest, which is really critical to understand, is multimodal, right? So a lot of real-world use cases do have multimodal data and quite a bit of it. So, for example, tables, images, diagrams, you know, flowcharts, things like that. And a lot of times the value of answering a question in RAG or in agents is really embedded in that image, right?

You might have a graph that shows the revenues of a company, and it's not in any of the numbers. It's just in a graph. So you have to really understand that image to understand what it is. You can even have videos. A lot of companies have training videos, for example, a lot of them for staff, for sales, for customers, et cetera. So you want to be able to understand video and audio and transcription and so forth and so on. And so that is a key interesting set of capabilities that you have to develop beyond what's involved in ingesting text. So that's the in-depth part. Now, another set of challenges is I kind of laughed it out a little bit that everybody built is with a vector database. But there's a lot more to retrieval than that. So today, the sort of state-of-the-art retrieval engine does include a vector database, but also what's called hybrid search.

So there's also a lot of use cases where you do want to have the old-school, I'll call it that, search capabilities like with TF-IDF or BM25 type of lexical or keyword searching. And hybrid search is the ability to merge semantic search or vector search with that together to create a way to identify both keyword search use cases and semantic search use cases. And then people use re-rankers quite extensively too because vector search and embedding model, what's known as embedding models, which are the models that you embed, you create vectors from text, are pretty good for initial selection, but actually they're not as accurate in terms of the semantic ranking. And so you use re-rankers, and there's a lot of work in the re-ranking space as well. So those are a couple of examples from the querying part, right? And then, I mean, if you do RAG, which is sort of a single shot,

so I think you have to now think about the generation part. How do you craft the prompt properly so it's accurately taking all the chunks together and creating the best, most accurate result? and then that leads into the topic you mentioned earlier about evaluation. How do you, in production, evaluate that the results are accurate, that the retrieval is accurate, that there's no hallucinations, it's a big topic, and so on. So there's a whole area of evaluation that is very, very important. And on that note of evaluation, that I think is probably the hardest part of RAG, despite all of the other complexities that really come into it, because changing any one piece of the system can have a completely unpredictable result on the output where you might change a piece of data in the system and then that might actually have been the keystone that allowed the LLM to be able to provide the accuracy that you wanted or maybe changing the model because of either different thinking parameters

or different size of the scale of the model or maybe you're changing the retrieval or re-ranking system. Any one of those things can have an outsized impact on the overall. And so there have been numerous frameworks and approaches developed to be able to do some sort of at least heuristic testing of the outcome. A lot of the at least earlier iterations were reliant on LLM as a judge where you just had another language model gauge some sampling of the outputs to say, yes, this looks good or this looks bad. And as long as everything is about average, then you ship it. But also there's the offline versus online testing where with offline you say, here is my set of examples and here are my expected outputs. As long as everything looks about right at the end of the day, then we're fine. But really, the changes in these systems come about when you actually put them in front of people, because the way that somebody asks the question or the way that the prompt is structured can also have all these different variables involved.

and so I'm wondering if you can talk to some of the current set of approaches that people are using for these production or enterprise-grade RAG systems that they rely on to actually build and maintain confidence in the accuracy and reliability of the overall stack. Yeah, 100%. You're right, it's a super important topic in RAG, and I want to mention also one that that people don't necessarily think about upfront. They just build first and then they realize, oh, we gotta build something like this. So let me try to give the full picture here. So first, yes, you're right, people use LM as a judge. What is that? That like just to give us some eval prompt and you try to evaluate the accuracy of the results But I want to separate between retrieval accuracy and generation accuracy right Two different things. So retrieval accuracy, we just described, you know, vector search and hybrid search and all that.

So the end of that phase of RAG is a set of, they're called chunks, they're parts of documents, let's say a couple of paragraphs sometimes or something like that, that are deemed to be relevant for this particular query. And so the first question is, did my retrieval pipeline retrieve the right set of chunks, the most relevant set of chunks? Now, you have to think about, do I have the relevant set of chunks? The first problem is, like, sometimes the question doesn't have an answer in the documents that I ingested. And then maybe I need to figure out where those documents are and bring them in. It's not obvious in a big enterprise that you have the right baseline ground truth to answer every question that a user might ask. But assuming you do have them, your retrieval pipeline, like I said, might have shifted. You may have implemented a new re-ranker. You might have increased the number of results, reduced the number of results. You may have re-done how you do chunking. There's so many levers that you could have changed,

and every one of those could have influenced your retrieval pipeline. And so on retrieval, people tend to do LLM as a judge or they try to use the regular techniques that we have, like precision recall, F1 from machine learning. The problem with that is those techniques, those metrics that are pretty simple or pretty proven require social truth. So precision, for example, is how many of the retrieved chunks actually should be in, let's say, I have top 10. How many of the top ten that I actually retrieved should be in the top ten are actually relevant, right? And you have to – somebody has to sit down on all the chunks in your, say, 20 million document organization document set and look at all the chunks and say, are those relevant for this query and that other query? And I said 20 million to, you know, emphasize it's completely impossible to do it in practice, right? It's completely infeasible.

Nobody can do that. And so in theory, those techniques work, but in practice, it's not working. In the answer side, so the generation, let's say that I don't do this on the retrieval side, but I go to the generation side, people have done in practices, they said, I'm going to come up with a thousand questions that I think are relevant. Or sometimes, if they're smarter or they have the system in production already, they'll look at the most common questions or take a random sample of questions that users actually ask. and then they'll have somebody sit down and generate a thousand answers. It's a manual process, but it's doable. You can sit for a week or a couple people and generate that. And then they'll have some metrics to measure, is the generated answer similar enough to the ground truth answer? And I've seen a lot of people try that. there's two problems with that approach which also leads to lm as a judge

the first one of course as i said like the questions themselves could be hard to generate sometimes remember you have to generate not just questions that people ask but also questions that people might ask later and so it keeps changing the tendencies to ask questions that have obvious answers but you want to ask also tough questions that don't have answers necessarily and things like that. But the answer is, that's where, again, I see a scalability issue in practice. Sometimes these organizations are so complicated. They've existed for 30 years and nobody knows the answer to some questions. It's like in so many systems. And so you can't even find the engineer or the product manager or whatever the function is, the production manager that knows the answer to that question to validate it, to write that golden answer, right? And again, it keeps changing. Like you mentioned earlier, some of the answers might change over time because I have a new version of a product or I have a new set of capabilities in my API.

So it's very hard to keep track of in practice. So let me talk about two approaches that are using LM as a judge that I also write up in the book that are really cool to solve both retrieval and generation. The first one is called Umbrella. and oh by the way before I go there I want to mention this has been really interesting research that we did at Vectara along with the Jimmy Lin's lab in University of Waterloo I want to give all credits to those guys they did a lot of the research and we just collaborated to implement this in something called OpenRackEval that we created later but Umbrella is really cool it's a very simple algorithm if you think about it first and this is to identify chunks that are relevant so you take every chunk in your top 10 and you run an LM as a judge that gives a score between 0 and 3 0 means the chunk is completely irrelevant for the query 1 means it's somewhat relevant

2 is like it's pretty good relevance and 3 is like it's super relevant it might even solve the whole problem and you know for those who are with LM as a judge would ask me like okay, great, I can generate this. What's so researchy about this? Why is it so special? The reason it's so special is, that's the algorithm. You take this and you rank it like this. But the really cool part about it is that the lab for Jimmy Lin, the research they did is they tested this approach with a particular prompting language, also with human beings. They looked at the chunks. they had a bunch of data sets and they compared the results of human judgment with this over time a couple iterations of crafting the particular prompt sentence and they found that the correlation is very high, I think it was like something like maybe 80% if I remember correctly. So very high

correlation between what this particular prompting LLM would do compared to what a human would rank it as. And so you have confidence that if you use it In this way, you get the zero, the three actually mean what they do. And that's a general problem of LLM as a judge. It's a fantastically easy algorithm to implement for a lot of different use cases. The problem is, do you trust it, right? It's got its own biases. It's got all kinds of problems. The LLM might change, and what you get might change too, and stuff like that. Yeah, with the LLM as a judge, you then have two non-deterministic systems that you have to have some sort of evaluation for, and then you're in the catch-22 of which one do you tune when and how do you decide which one regressed versus improved. Exactly, exactly. So with Umbrella, you kind of want to keep it the way it's implemented first. It's been proven. It's shown the sort of correlation, and that's why it's powerful.

I'll just mention briefly the other one, and, you know, it's called AutoNuggetizer, and it's a little bit more complex approach, but it's similar in concept. Again, an LLM is a judge approach that takes those chunks and ranks them relevant to the generated answer. This is for the answer relevancy. And what you do is you then see a test for a different thing, which is, is this answer that was generated by the LLM in RAG, does it use all the relevant chunks properly? So if one of the chunks was rated three, like the three, like the highest rated thing, you wanted to make sure that that piece of information was moved over into the answer. And again, it's not deterministic, so you don't really know. So that's the goal of that algorithm. Again, something actually called nuggetizer, which is deterministic, and then autonuggetizer is this generation of nuggets using LLMs

and then sort of evaluating this in the context of the generated answer. So those are two LLM as a judge approaches that I wanted to highlight. And as you mentioned, one of the standard ways of doing the evaluation is that you have that set of labeled question and answer pairs of this is the question that I have, this is the answer that I expect. And that can be a very time consuming and potentially manual process, which can also be expensive and also requires that you have some concept of what data you have, what types of questions or inquiries are being performed. And that requires some amount of bootstrapping that can again become a catch 22. and I'm wondering what are some of the ways that people are tackling that start from zero position of, I have the LLM, I have a way of generating the embeddings, producing the context, building out that whole machinery, but I don't have the flywheel in action yet.

So I need to be able to either generate synthetic data to start with or hypothesize a certain set of questions and answers, or maybe I just do it where I have a live feedback loop in production and I'm doing online evaluation so that the system can evolve and improve as it's operating rather than having to get a running head start first? Yeah, so first of all, yeah, synthetic generation is actually a great technique. You can point kind of an LM at a good sample of your data that you just ingested and actually generate some really good, with a good prompt, you can generate a diverse set of questions based on this data. It's not perfect because usually we'll do it one document at a time, so it can generate questions that necessarily require information for multiple documents. That's a little more tricky, but people do that all the time. I also highly recommend looking at your logs. If your system is already even a little bit of time in production,

what types of questions people actually ask that they use the system, and ask your subject matter experts, you know, for suggestions for different questions that might be useful. So that's a good way to kick this off. And sort of from like a release perspective, right, you want to build that system to be sort of part of your regression suite in a way, right? So you want to be, every time I do something, we kind of mentioned it earlier, If you change a component, you change something, you fix the bug that changes your chunking strategy or something. Everything impacts the system. So you want every time it's part of your testing, part of your regression testing or something, you want to be able to run this complete set. And it's got to be versioned and controlled and sort of so you know that you at least got the same results as before and hopefully better over time as you improve the system. Now let's talk about online testing a little bit. So it's a very interesting approach, but it's more difficult, right?

And it's more difficult for two reasons. One is you don't want to increase latency, right? You don't want to have yet another call to an LM to test before you produce the answer. It also, of course, costs a lot more, right? It may be, so you have another LM call. It's more expensive. People worry about cost of LMs in 2026 much more than previous years. I think it's becoming more and more of an issue. So what I've seen in my general recommendation is to do online maybe as a phase two, like if you want. Or you can do it in phase one if you're up to it. But, like, it's okay to do it later on after you do the offline sort of release. And also do it as a sampling, right? You don't have to do it on every one of the queries that come in through your system. So the way this works, right, is your user query comes in. You go through your RAG pipeline. And then you test. and you don't have to test on all the queries. There may be a lot, right? Some people have like a million queries a month, right?

It will be a lot of cost to run another LLM call or maybe multiple LLM calls on each of these. And so you can say I want to test on a random 10% of the queries or 20% or 5% or something like this to control your budget and also do it out of band. So I don't recommend doing it like waiting and holding off the user until it's finished. Think of it as diagnosis for your system. You did it offline. You store the results, and then you have some analysis later on and see what you can learn from it. You can learn that, hey, this query, the result wasn't that great, et cetera, and then you go and fix it back. It's something that goes out of band of your release cycle but gives you more immediate responses. And one of the other evolutions of RAG that came about as the idea of agents came into the picture more strongly and as the models became more capable is agentic RAG. And I'm wondering if you can talk to some of the ways that that is distinct from the first generation of RAG stacks and some of the ways that it fits into the current ecosystem and maybe how it complicates the picture of how to actually evaluate the effectiveness of it.

Yeah, so the way I've seen this kind of evolve over time is, as we said in the beginning, you know, RAG was the kind of hot thing and everything happened. But at some point, people saw some of the challenges of this one-shot flow that RAG has. Query, retrieval, generated answer. And so it evolved into what's called a genetic RAG. And a genetic RAG is essentially a simple form of the agents we have today. A very simple form. It allows you to do maybe a couple of iterations, right? So the agent takes the user query. The first thing it can do is actually it can rephrase the query in a way that's better, maybe appropriate for the retrieval part. So it can rewrite the query in a different way. It can actually rewrite it a couple of times and maybe apply it to the retrieval pipeline a few times. to get a couple of different results back.

And that became possible as our LLMs became possible too. So it didn't just happen in a vacuum. We had longer context windows. So you can not just put five results and that's all space you had to have. You could do 10 or 20 or 50 or 100 because the context windows became longer and you can put more results in there. And so Argentic RAG, the primary value I see is the ability to do multiple iterations and also rewrite the queries. So a lot of times if you run a query the way the user wrote it, it may not be the correct way that is best suited for accurate retrieval. The user may have an acronym there, for example, that is unknown to the system, and the LM might actually translate it using some glossary that you give it or something like that. So that been actually really helpful to just improve accuracy of retrieval in general And then this evolved into what we know today as AI agents right So agents are sort of an evolution of that where we have what we call the harness

which is the evolution of the agentic rag loop, the simple loop that we had at the beginning. And now it has access to tools. So if you think about it in that sense, A genetic rag was an AI agent with one tool for retrieval using the rag kind of retrieval pipeline without the generation. And now we can have access to tools anywhere from retrieval tools like with the rag, but also tools that go directly to target systems. Like there's tools to query databases with text to SQL, for example. There's tools to go to the native systems, like, you know, search capabilities, like in different places in Notion and other places, they all provide these kind of API-type capabilities, all through MCP. And that's what the harnesses we have today for AI agents building. And that idea of having the tools be the retrieval engine has also maybe diminished some of the initial supremacy of vector databases

as the end-all be-all for that retrieval step and changes some of the equations as far as what you need to do for the seeding of the data, where you don't necessarily have to do the whole pipeline of collect the data, chunk it, stick it into a vector database full of embeddings in order to be able to get anything out of it. But it is still a necessary element. And I'm curious how you're seeing that change the ways that people are thinking about their overall retrieval and context system, particularly for these more agentic use cases, as well as some of the reasons why somebody might want to stick with a more straightforward RAG system in the kind of first generation definition. Yeah, excellent. Excellent point. So, you know, dragging all the data over to ingest it into the sort of the RAG pipeline was hard. You had to connect to the source system. You had to essentially replicate all the data.

So you had two duplicates. Those were the downsides, right? And you had, like we talked earlier, we had to refresh it every once in a while. So you had to sort of create like a hot copy of the data, only in a way that's sort of in a vector database or in a lexical database for ready to be queried. But you did have the advantage of semantic search and hybrid search and all these things we talked about that created better results. And so today, you're right, as things evolve, more and more vendors created tools for their system to be able to query that system. But the downside of that still, and it keeps improving, is that you are dependent on the quality of retrieval of that tool. So if I call, and I want to say Notion, but I think actually Notion search is not that bad. I'm not kind of telling anything about them just because it's a common thing. If you go to, if you call a tool of Notion instead of bringing it back to search, you depend on whether Notion search is good or not.

Does it do multimodal search properly? Does it do the tech search properly? All these things we talked about become important. And if they're not, then it doesn't help your agent. It's not going to get the right information. Similarly with Google Drive, similarly with SharePoint, similarly with Jira or anything else that you have. So that's the trade-off you make. And I think as an industry, we're kind of in the middle of that. So over the last two years, a lot of these sort of API search functionality has improved, but not all these companies have, you know, search has kind of its own little sets of capabilities and sort of know-how, and I think it's not super easy for all of them to sort of improve the search to a level that it can be as good as you might need it to be. The other challenge is a lot of companies are now moving to kind of what's called, there's a word for it, it's called AI sovereignty, right, but kind of on-premise, right? When you go to on-premise, and we haven't touched this yet,

but just on this point, they want everything in control. So they're calling a local LLM, maybe it's OpenAI's open source model, or maybe it's the NVIDIA one or thinking machines or whatever it may be. Nothing goes outside, nothing goes outside. And so, again, if you have to do that, a lot of times you might want to ingest the data earlier into that local environment so they can be part of what can be queried and you don't have to make any external API calls in sort of these air-gapped environments. One of the other interesting pieces of bringing agents into the mix is in particular on the side of data pipelines where the first generation of RAG required a lot of upfront engineering effort to build that pipeline of doing the chunking and embedding, et cetera, keeping that up to date. Now agents are doing a lot of the work of actual code generation. They're being brought into data pipelines. And so that also potentially acts as an input to

the RAG system, which can be yet another source of errors or improvements. And I'm wondering, what are some of the ways that you're seeing that come full circle where we're bringing LLMs and agents fully into the process of managing the end-to-end system for the data seeding as well as doing the retrieval and as well as doing the evaluation and just some of the ways that that maybe unlocks some new patterns that you're seeing? Yeah, great question. So, for example, there's a lot of different places. I see, for example, an ingesting of data, right? It used to be a data engineering pipeline that did sort of, let's say, take a PDF file. You have to parse the PDFs, split it into pages, see what tables and images are, do the chunking, all that stuff. Now people do it with agents, right? You have an agent that just does all that process for you. That's one area. Generally, the other interesting thing is all these systems used to be with APIs, and they had very elaborate documentation about the APIs.

And you had to, as an engineer, as a developer, have to read all of that and figure out how to implement it and APIs change and everything. Today, you know, I run Cloud Code or Codex or Copilot or whatever, and I just point it at the APIs, and that's it. It just does, it produces a really good system for querying for my actual application that uses the API. It produces a good thing for indexing the data or adjusting the data. So like everywhere else in engineering, the coding agents are fantastically powerful to help us build systems much, much faster. and honestly with ultimately with less mistakes. We still make mistakes as humans. I know we all say like, oh, is the coding agent really doing a good job? And in some cases it does not and we can talk about that too. I have some experience with that too. One of the things I like to do is to write something called code and then have codecs review it and have this adversarial between them because I found that a lot of times that really helps.

You show something to code code and the S codex to view it and it says, oh, I found three bugs that code code said everything was okay. Or the other way around. It's not against one or the other. But anyway, getting to your earlier point, yeah, I think I've seen this in ingestion. I've seen this in sort of just writing other parts of the system. I think in the infrastructure of the right pipelines, like where the embedding happens and things like this on the retrieval itself, I haven't seen a lot of people use it yet. and also an evaluation I haven't seen it yet, but that's just maybe me. I haven't bumped into people who did it. I could expect that this could be helpful in any part of this. Another term that has been percolating through the ecosystem is context engineering, which if you squint is the same thing as RAG, but people like to come up with new terms for the same thing. And I'm wondering what are some of the ways that you think about the overlap and some of the differentiation between the ideas of RAG and the idea of context engineering?

So, yeah, context engineering is, let me sort of first state what I, kind of my understanding of it, just so we align on, because the terms can be very confusing, right? So context engineering is this idea that you as the engineer developing your system control what fits into the context of the LLM before you send a request, right? So, by the way, when I started using GPT-2, for those who don't know in the audience, It had 512 tokens, context in it. That's it. We think about now we have a million or 10 million. It sounds ridiculous. But that was it. And then it became 1,000, 1024. It was like, oh, my God, this is double. So context has grown a lot, which is amazing. So context engineering now is this practice of when you create your system, of deciding what's in your system from. What do you tell the agent to be like its guiding principles? And what sort of intermediate data you bring into the same context that you put in front of the LLM?

In terms of when you think about RAG, it could be chunks that come in. If it's a more advanced agent, it could be sort of the tools that it may be allowed to call and their definitions. It could be historical information about the user. People now talk about memory. So there's a lot of things you can pull into the context. But it's the same thing. It's how do you take all this stuff and then you send it to the LM and ask it for help to do something. Either call one of the tools, call many of the tools, whatever it needs to do, either answer the question or perform a task for you. Yeah, so that's kind of the evolution. that. One thing I should mention is one way I think about RAG in a way is a little bit, I haven't heard it anywhere else, but I thought it was an interesting way to think about it, is RAG could be the kind of attention mechanism for context engineering. And what do I mean by that? People know about attention, you know, the famous word in deep learning, right? Attention is a

mechanism in deep learning itself where it sort of looks at the history of the conversation so far and picks the right words or the right tokens in the history that are more important to generate the next token, right? That's what the attention mechanism is in general. And so sort of taking that idea and saying, I have a question for the user now and I have all my corporate, all my enterprise data out there. And RAG sort of picks the right sets of information out of there that needs to be pulled in to the context of the LLM in that sense. And so I think of it as, again, a selection, a selector operator that helps you pick what's important into the context engineering sort of landscape. And as you have been working in this space and particularly in your capacity doing developer relations, working with people out in the community, what are some of the most interesting or innovative or unexpected ways that you've seen teams deal with the production requirements of RAG systems?

The first thing I wanted to mention there is not necessarily unexpected, but actually expected, but important, is how do you deal with roles and permissions? right so this is an important problem in rag and in agents right so i let's say i have uh all my google drive ingested into my right pipeline maybe the salaries of the company there's excel sheet that's all also there only the ceo and hr have access to it but if i ingest it without any protections then everybody can can pull permission out of that right so you want to make sure that you have this role-based access control the same way the company implements it be applied in your retrieval pipeline. That's a really important thing to do. And I guess the innovative way people implemented this is just by implementing some form of filtering. So if you say attach to every document that you ingest into your

pipeline, you attach some notion of permission. And again, permission is a whole world of complexity. And a lot of companies have multiple permissioning systems. And, you know, it's a mess. But if you can normalize your permission, your role into something that you can sort of add as, say, metadata to the document, and then when a query comes in, you can sort of say, if the CEO is asking the question, then I'm going to allow them to see everything. If it's somebody on another team that's not supposed to see the salaries, then you would then filter the documents that have the permission block in the metadata and not include that in the chunks, even if it's relevant. So they supersede relevance with filtering beforehand. And that's been really helpful. That's one thing. I think multimodal has been an area that I've seen a lot of innovation. so I mentioned this a little bit in the

beginning on the indexing or the ingestion side but think about multimodal also on the UI side right your application needs to show the response but if you have an image you kind of want to integrate it into the response so again a lot of people don't think about it if you have a video you want to embed the video in there to show where the response was retrieved from and maybe even point the video to the right second right so you kind of want to have some innovation on the UI side to make the user experience like really helpful. Ultimately, like everywhere else, every other application we've ever built in tech, people are lazy. They want to get the answers right away and they want it to be appropriate for them. So you kind of work really hard to make this, work in this way in multimodal data. I think the other thing is, as I mentioned, I think people are working really hard to make it be viable, both RAG and agentic. in the air-gapped environments. That's a really important use case for a lot of big companies.

Think about people working in the government industry where there's different agencies and things like this that data can never leave, or companies in the defense industry or semiconductors where there's a lot of competition. Those are areas where we see even financial services, again, a lot of different things. And so if you unpack that, you have to do a lot of innovative stuff to do this. One of the surprising things that I bumped into was that, well, somewhat surprising, I would say, is people are afraid to use the Chinese models, right? Chinese models, the open source models are very, very good in terms of performance, right? They are on par with the commercial models that are API-based. almost there I think like Minimax 3 Kimi 2 you know 3 like there a lot of recent models QN like DeepSeek All of them are by the benchmarks they really good And they can be hosted locally. And there's not too many models that compete that are sort of by U.S. companies.

But I see a lot of resistance of using these models in at least U.S. and European-based companies. And I can see why that would be concerning to some extent. And so I really like, for example, that thinking machines recently came with a very good US-based model, and I see there's a lot of movement there. So that's something that initially surprised me because open source weights are just weights. They can't like – I don't see, as a technical person, I don't see a way how this – they would send something or the Chinese could control them. Maybe I haven't thought this through enough. I'm not an expert on security and relationships like this. Maybe it's about the fact that people might get used to them, and then once there's a need for additional code, they'll forget about it or something like this. I don't know. I'm really not sure about this. But I've seen a lot of resistance to that, for sure, in the sort of sovereign AI space. And in your experience of working in this area of building and evaluating RAG systems,

What are some of the most interesting or unexpected or challenging lessons that you learned personally? Well, first of all, it's something I mentioned earlier, that teams don't think about the challenges up front. They just rush. I mean, they rush to build a system. And like we said earlier, it used to be a lot of engineering. Now it's a little bit less, but it's a lot of work to build a scalable, robust sort of like system with all the bits and pieces. and a lot of teams, I was surprised that people don't apply the same rigor that we do to any other system that we ever built in microservices or whatever other technology in the past to this system. I think part of it is because there's just been a lot, it's happened so fast, and there's been a lot of push, like, hey, we've got to go into AI, we've got to AI-fire systems or whatever it may be, right? So I think that's something that I've seen, like the rush and not thinking through it and then being surprised that it's more complicated than we thought when you go to a production scale.

And the second thing is really, and we talked about it too, is like just evaluation, how little people think about it. And they're then surprised that results are not good. And then they put it, slap it on the system after the fact. And then realize, okay, how difficult it is. It's not just creating questions and answers. That becomes complicated. So people are used to regular applications, right, where the metrics you have. So people understand metrics, right? You have to have measure latency, measure downtime, you know, all these things that we're accustomed to from the DevOps world, right? But these new metrics that are ML-based and sort of LLM-based are new and more difficult, and people, like, don't think about them up front. What I've seen is if you have an ML team internally that is powerful and strong, then they might tell you about this and you might think about it up front, but many companies don't and then they don't think about it in a sense. And as you continue to explore and work in this space and keep tabs on the ways that the industry is evolving, what are some of the predictions or hopes that you see for the future of RAG systems and context engineering and just the overall space of data-driven LLM and agentic applications?

so yeah first of all I think retrieval will continue to evolve probably without much the G some people still use naive rag in the old way and it's available for them but I think more and more will certainly go into agentic and we'll just use the retrieval part where it fits and continue to improve that I still see new embedding models being created that are better, new ranking models that are better so there's still work to do People use glossaries, for example, and build models that are specific, they're fine-tuned specifically to that particular industry, which helps with retrieval. So there's a lot more to do in retrieval and search. I definitely think we will see the retrieval of other systems that you connect to directly using an MCP tool or something like this become better and create better results. I still see a big gap in the multimodality in that respect,

and I hope those vendors will keep improving that as well. So that's one thing. Context engineering, again, can used to be a buzzword that's very popular. It kind of goes into what people think of as memory systems now. So people talk about, like, the immediate memory, right? In a particular session in a chatbot, how do we talk, you know, pack and forth me in the chat bot and so if I say something that we talked about before I want to remember that that's the simplest form but then there's things like oh 20 days ago we talked about something related to I don't know maybe different banks in my area and I told you the agent that I like this bank so don't ask me that again so like historical contacts of earlier conversations and I think both of these are really interesting and people attack them in kind of different ways. But I think, again, that will probably commoditize and be like certain two or three approaches

that people do and they'll be really easy to use. In terms of agents, I think there's a lot of interesting things coming up. So one, and let me talk about regular agents and then coding agents because I like to talk about coding agents separately too. On regular agents, you know, So the LMs do really well with calling tools and deciding what to do with agents. There's no doubt about it. If you compare it to a year ago, it's incredibly better. But we've seen the agents move to what's called long-term horizon sort of things. And I think they'll still suffer from accuracy when the task takes a lot of steps. Because what happens is the context becomes sort of like poison a little bit. like what's called context rot a little bit. And they'll start doing the wrong things a little bit or ask questions again. So, for example, when I use a coding agent like Cloud Code or Codex, a lot of times when I use the same session for a long time, after about 200,000 tokens or something,

I start seeing it not act as well. And I sometimes benefit from killing it starting a new session. So this happens in regular agents too. And I think we have to, as an industry, I'm looking forward to seeing what solutions people come up for that. On coding agents, I think we're seeing this interesting evolution from one coding agent to a swarm of agents. And this is something we're working on at BEND about creating this, using BEND, which is agent-agent communication, to create this idea of multiple coding agents that talk to each other to solve a problem, that collaborate together. Just like we humans in a team can collaborate together, right? So I think this idea of coding agents that specialize in a particular area, so you can have a coding agent to write a code, coding agents to plan, coding agents to review the code or to review the plan, coding agents for compliance, coding agents for machine learning, for design, for different roles.

And they have a very precise context. They don't have context of everything, so they can work more precisely on each of these. and then they collaborate and hand off work to each other as needed when they need to develop code for us and do that. So I think in my mind, this will create a much better outcome for everybody and especially if you use multiple agents so you can take advantage of CloudCode and Codex and Copilot and I don't know, Kuro or everybody else. There's going to be so many of these coding agents produced from the anti-gravity. Because every one of those has a little bit of a different harness, so it behaves a little bit differently. It has a different LLM that drives it. So you will keep seeing them. I think we all will keep seeing them sort of have different personalities or different strengths and weaknesses. And as we get to complex code generation, long horizon tasks, benefiting from sort of adversarial review

and seeing multiple points of view would actually create higher quality outcomes. Are there any other aspects of the work that you're doing in this space or just the overall challenge of building and operating and maintaining these RAG-based systems that we didn't discuss yet that you'd like to cover before we close out the show? Well, I want to mention one more area, which is kind of SQL-based data that we haven't talked about, and that's interesting. So there is a lot of work in text-to-SQL, and again, think of it as a tool that agents might call. The reason it's interesting is very obvious, right? Like most mission-critical data is still in databases, in Snowflake or Databricks or Microsoft, Oracle, whatever it is, right? And text-to-SQL is going through, first of all, it's really good, right? It is pretty good now. And it's going to get better over time. There's models being developed specifically to solve this problem. but I think one of the interesting things that are happening

is it's related to this world of semantic modeling there was Looker that started this and there's a new version of this called Malloy which is open source and there's other companies that do this kind of thing to build semantic models to enable enterprise to sort of define what something means in SQL because the LM again, like with reg, doesn't have context of what things mean. So if you ask it, tell me about the churn in our recent customer base. Churn can be defined as somebody that dropped off for 15 days or 30 days or 100 days or whatever. So there's no definition the LN knows is the absolute truth. And so the ability to define this is really important. And it was important before. We had this with data analysts and things like this, but now we have to define it for agents because there's going to be so many more agents asking so many more questions out of these stacks of SQL pipelines that we need to enable, make sure that if agents

are going to do this at scale and we can trust it, it has to be enabled. So it's just one thing I wanted to mention that we didn't cover. Yeah, it's definitely still an interesting space. And I see the word semantic layer and semantic modeling probably at least once a day, definitely multiple times a week these days. All right. Well, for anybody who wants to get in touch with you and follow along with the work that you're doing, I'll have you add your preferred contact information to the show notes. And as the final question, I'd like to get your perspective on what you see as being the biggest gaps in the tooling technology or human training that's available for AI systems today. The thing that, first of all, let me start with human training. I still see a huge gap in terms of people adopting AI. Now, I think you and I, and maybe others in the audience, some of us live in the tech world, and we try our best, actually, to track these things every day. It's so difficult. There's like 50 things happening every week,

and it's hard to keep track of everything. But AI is so powerful and so useful in a lot of different ways, and I'd love to see more. I see a lot, but I'd love to see more people and companies help with education of everybody of how to take advantage of AI This started from people don't even know how to use ChatGPT. Now I think more people know how to use that, but how to use it effectively, how to connect it to your own systems and things like this. Very, very important. Also, a little bit in a narrower context, but related to that, I think the education system itself, the education industry, teaching kids, teaching adults that do later education, I think has to change. We have to integrate, in my mind, how to use AI effectively in every part of education. And it's a massive shift that has to happen because learning, memorizing things and kind of answering questions based on memory is just not going to work anymore.

I'm very well connected to the education ecosystem here in Silicon Valley, and I've seen this a lot in the last couple of years. where teachers are looking for help too. How do I teach Python? How do I test? Because the test is usually solve this problem, and you give it to the coding agent, and the second later you have the answer. So I think people need help, not just learning how to use AI, but how to teach AI. So I'm really looking forward to seeing how that. And I know a lot of people are working on it. I'm not saying it's not solved, but I'd like to see people help with that. Yeah, and generally, I'm really excited to see what some of the commercial labs come up with. I know there's a lot of people that kind of came out of the labs now or starting a lot of new companies like Ilya's company, SSI, and Thinking Machines, and Core Automation, and other companies like that that are trying to solve the next generation of LMs.

and also very excited about image and video generation. I think I've been tracking that space a little bit too. And kind of the way I like to characterize it, I feel like it's in a GPT-2, GPT-3 moment of its evolution. It's very powerful. We use it all the time. But I think the potential will be unlocked in maybe a couple of years where we're going to see this amazing innovation and much more direct use cases that are more immediate in that sense. Absolutely. Well, thank you very much for taking the time today to join me and share the work that you've been doing and some of your insights on how to evaluate RAG systems and get them running well in production. It's definitely a very interesting and continually complex problem and one that keeps shifting. So I appreciate you taking the time to share your insights on that and I hope you enjoy the rest of your day. Thanks, Tobias. Glad to be here. Thanks for having me.

Email hosts at AIengineeringpodcast.com with your story.