エージェント検索はページを探すのではなく、答えを組み立てる

Hey, your open finance prototype works. Now, make it production ready with MasterCard APIs. Access permission financial data and scale on infrastructure designed to meet the security and reliability standards that financial services demand. Head to developer.mastercard.com. Hello, everyone, and welcome to the Stack Overflow podcast, a place to talk all things software and technology. I'm your host, Ryan Donovan, and today we're talking about what search engines look like from both the user and the engineer perspective when the primary user is agents. My guest for that is Jeff Wang, who is the co-founder of Exa. So welcome to the show, Jeff. Thanks for having me. Nice to meet you, Ryan. Yeah. So before we get into today's topic, we'd like to get to know our guests a little bit. Tell us how you got into software and technology. Yeah, sure. I was from a young age,

always interested in computers. I remember sneaking downstairs at 6 a.m. in the morning when I was 7 or 8 so I could use my dad's computer before he wakes up so that he wouldn't realize that I was using it so much. And yeah, I kind of developed into a love for programming and led me to tech. So, you know, we've heard a lot of information, data on how the primary traffic is moving from humans to agents. And I think before we talk about what happens when agents are the primary users, I sort of wanted to take a step back and talk about what are the sort of engineering concerns for search overall. I will give a little context about what EXA is to preface that. Well, EXA is a full search engine built from scratch for agents. There's all sorts of reasons why search for agents is different from search for humans. But yeah, a lot of things do hold the same, right? Search over the internet is a large-scale problem. The internet is very, very big, petabytes of data, trillions of pages. And the goal is to take that data,

take a search query, and figure out what the most relevant information is. And so this involves, mandatorily, what one is you need to go discover the internet, you need to go run a crazy number of crawlers to go find the data, scrape the data, then you need to store that, and then you need to build some type of search algorithm over it, over some representation of it. And for that reason, the core problem in its essence from a technical architecture perspective is not entirely different. AI makes a lot of components simpler, but yeah, it's still the same thing as it was a few decades ago in many ways. Yeah, it seems like if you reduce it down, the primary thing it is is like a database, right? For the search part of it, how does it differ from a database? Good question. How does it differ from a database, just a regular database like a SQL database? Yeah, I mean, I know for a lot of old search, you'd use keyword algorithms like BM25 that process the text and had things like that. But for a regular database, you search on a sort of unique index ID, right?

Yeah, that's right. I would say that a search engine is a database with, at least in the modern day, a few components that may be different from like a generic database. Well, one is search involves a vector database in the modern age because you can build really powerful search engines now using vector search as opposed to keyword search methods. And so mandatory you have this vector component. Another component of the database would be just like some form of inverted index where you can take words and figure out documents that contain those words. And that's another mandatory component of the system. I guess other components of the system are like, I don't know, there's a lot of write operations. There's like very large scale of live streaming writes to all the systems. And then also, you know, because agents are calling search so much, there's a very, very large volume of reads. Yeah, yeah, that gets into the agent part of it where they are creating a lot of reads, a lot of writes. Is the difference between human usage and agentic usage just one of scale or is there something else going on?

Yes, good question. Agents use search engines in very different ways and have very different needs. One would definitely be scale. You know, one thing I'd say is like there's there's eight billion humans in the world. Collectively, they're making about a few trillion searches per year, mostly on Google. But very soon there's going to be trillions of AI agents. Right. And they're actually each they can actually each search more than humans. Right. And so you kind of can imagine this like scale that's almost incomprehensible. Like literally the world is being terraformed to build data centers to have more agents, right? To have more large language models. And so, yeah, one is definitely scale. Two would be, I would say, like there's different patterns that agents can use to search. For example, nowadays, most agentic harnesses support using many agents at the same time. In the pre-AI world, you as a human, if you're, say, researching really cool podcasts about technology, you might google that and you might get some you know web pages that are known as listicles like oh here's like like top seven tech podcasts you definitely have to pay attention to and and you

as a human will go and like read that page but agents can actually search in a much more comprehensive and complex way than that they might actually want to get a list of a thousand podcasts and then for each of those podcasts the agents can then go do a bunch more searches about each podcast to get even more information so you need to build basically a search engine that can support this type of massive scale operation. Right. Because you're not just getting a set answer from a web page. You are creating a set answer from a thousand web pages. Yeah, that's exactly right. So I did an episode a while back on bot traffic and what the numbers were. One of the things that was interesting is that a lot of the AI labs are the big drivers of this traffic, but Google was not because a lot of their data is pre-cached, right? They don't have to go out to get it. And I wonder for agents, is there a way to bring that pre-cached benefit to individual agents? That's a good question. I would say that, at least as far as my understanding goes,

my knowledge goes, most search engines for agents, whether it's OpenAI's internal implementation, for example, or Google or whatever else, does depend on a cache of the internet. But one way you can think about it is in order to build the search index in the first place, you need to know what those web pages are and what the contents of those web pages are to search over it But I imagine that one factor might just be people are used to Google scraping their website already it like this identified pre scraper and then you bring in like all sorts of other crawling systems that all different types of ai platforms are using and people are like not used to it yeah wait i mean like google already has that internet corpus like downloaded somewhere i imagine for you know individual users even in you know large-scale users you don't want tens of petabytes of data somewhere that you have to access all the time Is there a way to give somebody the benefits of that cached petabyte system without actually dropping the petabytes on their system? Oh, yeah, absolutely. I mean, I guess one way to think about that is should every AI company, whether it's a startup or an AI lab, be building their own search engine?

Well, you can, but in our experience, it's not that easy. It does require scraping and indexing just like this gigantic corpus of the web, storing it, building the search over it. Like, it's a huge operation, and it's not for the faint of heart. And it's a continual operation, too. Yeah, that's exactly right. It's a constantly updating system, like, live, which is actually different than the large language models of today, which are a fixed set of weights once you finish training them. Yeah, so that may be another difference between, like, a database and a search, is that the data is sort of constantly changing, right? There's a lot of databases out there that also have the property that they're constantly changing. But, yeah, search has its own special unique complexities with respect to how much everything needs to be written, updated, read, et cetera. So what would you say those complexities are? You know, to be honest, I'm not. Like the complexities that we face today as a company on the technical side, I'm not even most looped in on, right? It's like the search engine has become its own behemoth and so complex and so many of its different components that even I can't wrap my head around all of it.

But, I mean, a few components are, again, like mass-scale web crawling, mass-scale processing of that data, and that involves running embeddings models that we train at Expo over that data. And then finally serving it at really high scale. So that means building like a really, really performant vector database that can return at really low latencies. Yeah. Does that mean you built your own vector database for that? That's correct. I know there's a lot of vector database providers out there, but, you know, what was the performance edge that building your own vector database got you? And what do you do to make a vector database more performant? So for some context, we're kind of – we're like an OG AI company. Like we're almost – like we're like over five years old, which is a dinosaur company. Like ancient AI. Yeah, like founded in 2021, right? And so when we were starting, there just weren't, like, today you're like, oh, my God, you can use TurboBuffer, you can use Milvus, like, whatever, right?

That didn't exist in, like, 2021 or 2022. And so, like, sometimes people ask us, why do you build all this stuff in-house? It's like, well, in the beginning, there was no choice. But today, I mean, like, the major answer is that the vector database is one of the most important parts of our entire system. Right. And so it's just a little bit like nonsensical to outsource it because there's so much we want to do with respect to improving its performance and modifying all sorts of ways. So it's mostly about the control. That's the central piece of your operation, right? You want to have a lot of control over that. And I think the other piece or another important piece, like you said, is the embeddings model. Did you also build your own for that too? Yes, that's correct. The embeddings model is the heart and soul of the search algorithm. And if we were to use a default off-the-shelf one, that would mean that we'd have default off-the-shelf performance. We consider ourselves like an AI research lab for search, essentially. And so we have like a team of researchers that explores how to train these embeddings models to do better at web search.

There's a specific embeddings operation for the search, right? What changes with an embedding model for search or what's different? Yeah, that's a good question. Well, the task is just pretty specific. So the task is to match search queries, which are like these like relatively short information seeking strings to the contents of URLs. And so that's just like a fundamentally very different task than like often generic embedding models, which are which might be optimized for embedding sentences and comparing different sentences or paragraphs or even words in the case of like the earliest models here like Word2Beck. And so this is a very specific task. And by specific, you're matching a whole document. You know, sometimes you have a massive document there. Do you think about it in terms of, like, parts of documents providing an answer? Or like you talked about, the search that may gather up a thousand documents to produce an answer? Yeah, that's a good question. We do, you know, often consider very large documents to be kind of almost like many different documents,

just simply from a technical perspective. but I would say the other like really interesting component of search for agents is that you know at the end of the day agents don't actually care about whether the information is coming from 10 web pages or three web pages I mean they do I think typically we've seen that agents care about having source diversity but what the agents really want is like just the right information for their need no more no less high trust right and so because of that we actually have a second step after we do our traditional normal pre-AI web search, which is we actually look at a big list of results. We look at the contents for all of them and we try to de-duplicate that content. And so, for example, we might have 10 web pages, but then because there's so much redundant information, we really have the equivalent of only one web page worth of text that we return to the agent. Yeah, that is interesting, you know, especially in this day and age with a lot of people sort chasing the same AEO results and generating the same AI generated content, right? That deduplication

is going to matter because I think everybody is writing a lot of the same stuff out there. There's a lot of AI slop out there. And so we try our hardest to make the very best index that we can. To de-slop the results. How do you dedupe documents, which will probably not have the exact same information on there in the exact same format we train models internally call them like extraction models that are transformer based models so you know they operate in this in this token world we actually like go through every document and basically token by token select which tokens are relevant to a search query and we don do that just for every single document We do it across a set of documents And so in that way, we're actually deduplicating information across a whole set of web pages, every single search. I think everybody, almost everybody, is including agents in their standard software process and standard software.

How much of sort of the agent itself is becoming part of somebody's software infrastructure? Are agents in general just becoming part of the software stack? Yeah. I mean, absolutely. In the early days of our company, almost all of our customers were like these like AI native startups, like really small teams, their product says AI in the hero section of the landing page, like .ai domain. Yeah. But now we're seeing that almost every company is using AI inside of their product in some way. Like a simple one is like Intercom. Like Intercom used to be this like software that's like this like chat support agent where you're connected to a human or like get answers to a predefined list of questions inside of any type of support system. And then now it's like a full AI system and like essentially like a full AI product. And so, yeah, I think we're obviously, yeah, we're seeing a huge change to AI across all tech. Yeah, and that AI agent obviously has to hook into the existing knowledge corpus to create that customer service experience, right?

So did you find when you went to those companies and were like, here's the search I'm offering, did it play nice with their existing search or were they like, this is better and they had to replace it? I would say most of our customers are typically building a net new use case. And so most of our customers, it's maybe their first time putting a web search into an agent because before it just didn't exist or maybe the product didn't exist. Right. I think a lot of people still, when they think of accessing documents with agents, they sort of think of the RAG paradigm, retrieval augmented generation. Is there a sort of conceptual leap you had to get people to do to think of agentic search? That's a good question. I would say for those listening that were in the, that were paying attention to AI in like 2023, rag is this kind of weird word because, you know, rag, first of all, it's kind of like a ugly sounding word. I've never really liked the word rag.

It's like, it's like a rag, the dirty rag. That's right. That helps you mop up. Yeah. Like rag in like 2023, 2024 was kind of this term for, it was basically all this problem of like, hey, like these language models have a context window of like 4k tokens and so if you're going to do any type of you know question answer over your internal corpuses using these very limited models you need to like break up the documents figure out which parts of those documents are the right ones to put into that tiny context window and then split out answers right and so rag became synonymous with this weird chunking weird gnarly engineering problem of trying to hack around these context windows that's not really what the term meant though if you think about the term like retrieval augmented generation what it really means is just okay what are you doing you're retrieving right you're getting you're doing search you're getting information and you're generating some response based on that and so so so i guess my answer is like over time i don't think the that rag has changed too much the strategy has changed like in the beginning

in 2023 and you do this weird like chunking like look at tiny parts of documents get half baked answer kind of approach. Now these context windows are gigantic, right? You have million token context windows across almost all the models. And so you could just stuff documents in there. Now, does token efficiency matter? Of course. Token efficiency means you save a lot of money. It means your model's smarter because it doesn't have this polluted context window, but it's still ragged at the end of the day, whether it's 2023 or 2026. With those massive context windows, I've heard of people running to, you know, needle in the haystack problems or sort of the results petering off the farther you go into a document. Is that still true? I believe that's still true, but the models are so good now that, at least in my personal experience, I don't really observe this. I haven't seen results of those types of evals for a while, I suspect, because the models are just so good at it now. Although, you know, when we run these types of evals, we obviously do see like performance differences between models. Yeah, yeah, for sure. You mentioned the increasing power, the increasing

ability of these models. I've heard people talk about the models getting so good that they're eating part of the agentic harnesses, right? That it's like you don't need as much framework around it because the model can just do that part. Do you ever worry that search will become part of the harness that the model eats? Yeah, yeah. I would say that, and I have some data to back this, like search is the most common tool call that the largest language models make. Think about it, The language models are just like you and me. Like they're, they're like reasonably smart and they can do a lot of things. But if you were in a black room without access to your computer or phone, you can't do very much. Right. You need access to internet. You need information. Right. And so it's very core. And so what we find is that most models are trained to have, trained to do search in some way with some specific shape. And so in that respect, we're seeing models have more opinions about what search is, and that makes this distinction between model and harness and who's responsible for a little bit subtle.

Yeah, but I imagine having a sort of dedicated search – or I wonder, having a dedicated search, is that a little gentler on the people who are providing the websites? because I've heard of some people being like, my site is getting hammered by agents coming in looking for info. I see. It's a question. I mean, I think whether a search call is being made by an agent as it's associated with the model itself and inference provider or with a harness, it's still going to hammer the page. At the end of the day, you need to use a search engine to do this work. Like, you need to use a search engine, whether the model is using the search engine or the harnesses. and somehow making this search engine that conceptually is different from how you think about how the model is tied with the search engine and it's going to be it's going to be like it might be hitting the public web as part of that process although caching is a is a nice approach because you get speed let talk a little bit about the exe engine itself What do you think would surprise people who are familiar with any sort of traditional search

Good question. I would say that one surprising thing is just how much agents search. Like, I think it's really hard to think about how agents are different from humans. Like, humans are so different. Like, humans are so lazy that I think the average search query is, like, three to five words or something like that. Yeah. Like, humans type these, like, small search queries. Nowadays, people just, like, a lot of people just read the Google AI overviews. But before, you know, before that, it was, like, you type in, like, three to five words to a search engine, and then you'd click, like, one of the first links that you saw on the search engine results page, and you'd just be like, okay, cool. Like, that's going to be what I'm going to read and, like, what I believe is true. and the agents are just so different like agents can output a search query of arbitrary length really really fast now and then after they do that they can read all the results they don't

need to read one two results they can read a thousand results and so yeah it's just the agents are totally different beasts yeah it's that's interesting that because most of us humans that go to the web search, we're sort of trained on the keyword search, right? We're like, what are the important things? Are we at a place where, you know, the sort of semantic, agentic search gets better results the more words you put in? That's correct, yeah. I think that in the long term and even now, the more information an agent is able to provide about what it wants, the better the results that it can find can be. That's interesting. And sort of on the other end of the surprise side, Are there any sort of implementations or use cases of XO that surprised you? Yeah, absolutely. XO is used in just so many different ways because search is just such a horizontal problem that applies to every single domain. I remember one of our earliest customers was a very early social network for agents.

And it was a platform where there were, like, literally thousands of, like, unique agents all participating in, like, a Twitter-style forum. And the way that EXA was being used is, well, these little agents, and this was in 2023, and so it was, like, literally GPT 3.5 and models like that. They need, like, new things to talk about. And so they would use EXA to find information. For example, you might have like a cowboy agent and the cowboy agent is interested in like new things that are happening in Texas or something. And it like do searches with Exa to find information like that and talk about it. The world's data is so rich and can be served in so many different ways. There's all these obviously really cool business-facing use cases that XSERV, like we power Cursor and Cognition and like tons of other AI native companies and their agents, but also just these like random side projects.

The AI industry is moving super fast all the time. What is the future of agentic search hold in your opinion? Yeah, AI ecosystem is so crazy. Like I always tell our team this, but basically AI changes, I used to say monthly basis. Now I tell people it's changing on a weekly basis. Like literally every week there's something, maybe many things that are happening that change how everyone should think about the ecosystem in a pretty serious way. I think about agents in three eras. So the first era was single step agents. So like, hey, agents can like, can do like one thing. You ask them to do one thing. Maybe they can go get a web page or something and then summarize it. They can do one thing. Then we kind of moved into the multi-step agents with models like Sonic 3.5, where it's like, oh, hey, all of a sudden, inside of Cursor, you can read a few files and then edit, like, two of them, and that just works. And that was, like, mind-blowing, right? It was like, oh, my God, this is crazy. Now my work is being automated.

So that was like a second era of agents. And finally, I would say, since November of last year, after Opus 4.5, we've had a true era of general agents, Agents that can kind of take whatever you give them and just execute. And I think that that has been amazing. We've already seen crazy trends in what types of products have been built on top of search. For example, there's a lot of personal agents. I don't know if I'm actually allowed to name any of them, but a lot of the personal agent ecosystem is being built off of Exa. And it's extremely interesting because those agents have different queries. because now we're all of a sudden moving a little bit towards the consumer world, where there's all these consumer things that people are searching for restaurants to book and make reservations at, et cetera. And finally, I think the next step of agent world might be agent swarms. So things like what we're reading in the news about OpenAI spinning off 10,000 agents to solve a math problem, things like that. And so I think that might be the next era.

Well, it's that time of the show. again, where we shout out somebody who came on to Stack Overflow and dropped some knowledge, shared some curiosity, and earned themselves a badge. So they were shouting out the winner of a populist badge, somebody who dropped an answer that was so good that outscored the accepted answer. So congrats to Santupa J.O.Wanda, Donna, apologies if I butchered the name, for answering Key Cloak client versus user. If you're curious about that, we'll have the answer for you in the show notes. I am Ryan Donovan. I host the podcast, edit the blog here at Stack Overflow. If you want to reach out to me with comments, questions, topics, you can email me at podcast at stackoverflow.com. And if you want to reach out to me directly, you can find me on LinkedIn. Thanks for having me, Ryan. Again, my name is Jeff. I'm the co-founder of Exa. And for any listeners out there, if you're interested in getting in touch with me or asking any questions, my email is jeffjff at exa.ai. And my agents will triage your email.

Well, thank you for listening, everyone, and we'll talk to you next time.