ZeroEntropy 共同創業者 Ghita Houir Alami 氏に聞く：リランカー訓練、校正された関連度スコア、エージェント時代の検索モデル

Rita, welcome to the WeVA podcast. I'm so excited to welcome you. I'm just such a fan of your work and what you were all able to accomplish at Zero Entropy. I'm so excited to dive into these things with you. Thank you so much for the invite. Happy to be here. Awesome. Well, I'd love to kick it off with the sort of founding story of Zero Entropy. It's just watching from afar and seeing the content you all were putting out and testing your models and just is such an inspiring story of just the hustle and getting up to the level of coherent voyage and just competing in that search model space, I'd love to know just, I guess, even from the beginning, just this journey of building this. Yeah, absolutely. So when I met my co-founder in 2024, we were both interested in building more on the infrastructure layer to unlock a lot of AI applications. And at the time, we were thinking a lot about context. So it became more popular afterwards with the whole context engineering buzzwords.

But fundamentally, it's a really important problem to solve. And that's how we got into search and started going down the rabbit hole. Initially, we were building the whole end-to-end infrastructure as a search API. And then we realized that the models powering these search APIs are incredibly important. And we saw an opportunity to do something better. And we were also very excited about, you know, training models. So we started doing that. And that's how we released our first Reranker model, which was incredibly successful. And so we just kept going in that direction, training more models, Rerankers, embeddings, and more. And that's how, you know, we were able to get, you know, hundreds of thousands of or millions of downloads of our models and processing trillions of tokens every month.

And, yeah, that was basically how we got to build all of these cool models at Zero Entropy. Yeah, I guess maybe just a quick question on trading models is, did you start with the Rerankers or the embeddings? and what motivated that decision? Yeah, we started with the Rerankers. The idea there was, you know, we were a very small startup and it felt to us that for multiple reasons, Rerankers were better suited as a first model. First, convincing model to switch their embeddings is always pretty difficult and it requires re-indexing their entire corpus, the reliance on our company would have been much, much higher. And it's not something that companies do so often. Swapping a re-ranker as the last step of your system is a lot more compelling. And it's easier to tell also the story of, you know,

you can keep your existing search system and you can just swap out the last step with a simple stateless API call and you'll instantly see better results. And, yeah, that framing was very successful, and that's actually how our customers were using our Reranker initially. Yeah, so I heard that, like, same argument as we talked about maybe building models that we've made and that kind of the ease of adding on the Reranker. It's such a cool – I remember, like, a lot of Coher's early communications around that Reranker was kind of just stack this on top of whatever database you have. And I think there's some interesting work now with switching embedding models that maybe you can have this, like, either algorithms that can propagate the new embeddings throughout the rest of your database or maybe, like, lighter weight translation mappings. No one's interested to, but maybe before diving into the technical for that, I'd love to just stay on the founding story. And then this acquisition with Notion, I'd love to know just maybe even starting just from sort of, like, emotionally what that was like into maybe now leading the model research team at Notion

and just what that whole phase has been like. Yeah, absolutely. We first got to know the Notion team through a business partnership. They were actually using our Reranker in production to power their enterprise search products. And so we really got to work pretty closely with them, understanding very precisely how their system was architected and the tradeoffs that they had to make and the kind of like production traffic that they had. and really got to know the team there. And then after the re-ranker, we also started exploring other models that we wanted to train with them, namely models that were able to take an initial query and rewrite it in a very fast way in many different, you know, wordings. So you could run parallel searches and do that extremely well and extremely fast.

So that was something that they were interested in. So we, you know, started working more and more closely with the team. And that's what eventually led to the acquisition. It seemed pretty obvious that there were clear synergies between the two companies where they had, you know, incredibly large distribution, data, compute. And, you know, we had the interest and the ability to train specialized models. and that was clearly something that was a need on their end as well. And yeah, that's how we basically made the decision to join something bigger. And yeah, so far we've been able to completely integrate the company inside Motion, grow the team internally there so more and more people are involved with model training, which is super exciting. and we already got to work on our first models there. So more to come soon.

Yeah, so many excited ideas in that I got to see the Notion office in New York City in like 2023 and it was such a cool office. I remember being really impressed with like the cafe. They had all these sparkling waters and it was just such a cool company and super serious about their search architecture. And yeah, I guess that idea, Notion is such an interesting company, how they combine, like they have like Notion databases. We use Notion a lot for managing our communications at WeVA. And I guess that vision of training models at Notion, that's just super captivating to think about, I guess, the expanding scope of like who trains models, which, you know, like more companies training models. Yeah, totally. And I think it's been super interesting to see. more and more companies are leaning in and starting to train their own models for various reasons. It can be, you know, having something that's more specialized. And by being more specialized, you can, you know, basically go up in the Pareto curve of, you know, accuracy, latency and cost.

So that's extremely compelling for companies that, you know, resell tokens in a way. and you know having that ability in-house to train models and to go from like an open weight models and make something better out of it is I think extremely critical today so it's been very interesting to see a lot of companies not only notion invest pretty heavily in this and you know We might talk about this later, but models like JEV are also, for me, an example of this where having something that's task-specific can become incredibly valuable because you can gain a lot in terms of the cost and the latency because it is specialized and task-specific. and there are a lot of such use cases inside companies and a lot of applied or vertical companies are realizing

that you can go from a general open weight model and just fine-tune and post-train on specific tasks, which I think is super interesting. And yeah, seeing it firsthand inside of Notion, there are a lot of such use cases and it's been great to hell climb on the Pareto curve. Yeah, just continuing on that, I've been super excited to pick your brain about Jev, someone who's just led the space in training cross encoders. I think you're just the perfect person to analyze this kind of new wave of decision models and Jev. And maybe I guess just to start off, maybe we can start off with the topic of Jev versus cross encoders and Jev as a re-ranker and what you think about re-ranking with decision models. Yeah, totally. In fact, it's pretty interesting to see because when we were working on our class encoders, we had already seen some use cases that looked roughly like JEV, where some of our customers were mostly interested in something that looked more like a classifier rather than having like an exact relevance score.

The relevance score is useful. And in fact, we also had something that looked like a confidence score. those are all useful but sometimes what you really want is just to prune and classify even like starting from like binary classification and you can do that with cross encoders pretty easily in in fact z rank 2 was included in one of the dev benchmarks and scored pretty pretty well knowing that it wasn't trained specifically on this, but more on, like, relevance. Basically, the only thing that you need to do is a little bit of data science to figure out what the threshold value needs to be. But once you have that threshold value, you can basically transform your cross-encoder into a classifier and just have, like, those, you know, positive or negative labels coming out of that. So I think that's a pretty interesting direction. And in fact, we've been, you know, considering whether to, you know, do another training run on top of ZRank for use cases like this.

In our training set, we were not focused on classification per se, more so on like calibrated relevance. But that could be interesting. And I think that's a pretty promising direction. And, yeah, again, for me, this is just the tell that, like, smaller specialized models are going to be an important layer of the stack. Yeah, I think the smaller specialized model is such a powerful idea. It's so cool seeing all these lighter weight decisions. Just the whole space seems to have exploded from type safe and JEV to open AI decisions, maybe battling it out at the top. And then you have all the liquid, just so many different decision models. I guess maybe you want to start on that calibrated relevance score. I was always kind of curious about cross encoders. They query candidate document output a score like 0.86. And I've always been kind of curious where that score comes from,

like how you sort of think about that score during training. Is it just kind of something that comes out of the training data, of the kind of like triplet loss where you have the positive and negative and you just, just, it just sort of, like, how do you think about that threshold value? Yeah, that's, that's super interesting. Yes, usually if you, if you train with triplet loss, you're not necessarily going to see something that's calibrated. In fact, you can run this experiment where you can take an ensemble of, of LLMs and with very specific rubrics in terms of how to score the relevance for a query document pair. And basically look at the spread and compare that to the spread that you might have from using a traditional re-ranker training on triplet loss. And you're going to see that for existing re-rankers from most of the companies training them, the scores are typically between 0.4 and 0.6.

and re-rankers are typically not optimized for, you know, looking at the score as an absolute score, but really just to be able to rank the documents without too much attention to the scores themselves. So you can plot the scores, I'll put it from like a Voyage re-ranker or a Coheria re-ranker, And you're really going to see that it's not going to have like a linear correlation or relationship with what like an ensemble of LMs might output. Or even if you like look at manual annotations, you're going to see the same thing. And in fact, we have plotted these graphs at zero entropy. So maybe I can link to the blog that has them. But, yeah, I think it's becoming more and more important to have calibrated scores. a calibrated score, especially if you start thinking about having like a dynamic K when retrieving results from a search query.

You can look at it as if you have a calibrated score and you do a little bit of data science, as you always should, you can basically now have a dynamic K and not return the same number of results every single time but have something that is more dynamic and maybe return like the top three results or the top 10 or 20 post re-ranking based on the actual relevance of the document. So that's something that we were super interested in at zero entropy. And yeah, that definitely unlocks using cross encoders as like classifiers if you're able to do that. That being said, even if the score is not necessarily calibrated, you can calibrate the threshold in a way that still will work. Yeah, there's so many amazing ideas. Maybe if I could just build off that last one of this concept of searching with a dynamic K.

I think it's just so powerful. We saw a lot of, there was like the sort of early days of agentic rag. A big thing was, say my query is show me all the houses in Boston that are between $2 million and $3 million or some query like that where it's like I don't want to just see the top five. I need to see the entire list of results. And then there's this world of like AI SQL where they particularly have like AI filters, one of these operators. And that kind of has this idea of you're going to apply this AI over the entirety of the database. And, yeah, I guess that idea of a dynamic K, it sounds like how search can kind of bridge that gap. And, yeah, it sounds like it's just a super powerful thing. And maybe if I can add one more thing. We did this thing called auto-cut with our embedding distances where we were looking at, like, the steepest slope descent. So if you'd have, like, 0.91, 0.90, 0.89, then all of a sudden, like, 0.86, maybe to make it steeper, like, 0.82.

And then you say, like, 0.09. I'm trying to explain this and think of the numbers. And so that would be how you'd cut it compared to, like, these cross encoders. yeah so I guess just maybe that dynamic K kind of what you're seeing with that yeah and yeah totally totally agree you mentioned like exhaustive searches yeah that's definitely something that could help with yeah cool yeah so that topic is amazing that dynamic K and how you can do that with re-ranking and I guess maybe two things I want to pop off the stack there too talking about like like cross encoders are like point wise encoders where they have the query in the candidate document. And I'm really curious what you think about pair wise or set wise or list wise comparisons in search. I always thought a cool idea with cross encoders could be instead of just the query in each candidate document, maybe it also takes as input the top ranked result from hybrid search, like from the first stage retrieval as well as maybe the score it had. And it maybe uses that as sort of like an anchor for evaluating this new

candidate document. And I guess generally just this space of like, yeah, going beyond point wise comparisons. Yeah, that's super interesting. In fact, at ZeroMTV, we did have a pair wise re-ranker because of our whole training strategy where we first had a pair wise re-ranker that then we kind of like distilled into a point wise re-ranker. And we thought about releasing that pair wise for a very long time. In fact, maybe we should just do that now since all of our models are fully open source. I think it's a very interesting idea, you know, from like an academic perspective. When it comes to pairwise, yeah, basically the issue is the compute related to making that happen in like a real life production system. for many use cases, even having a re-ranker is like a pretty big investment

in terms of both like token cost and added latency. So even for a point Y, sometimes the, you know, the case is hard to make for some real-time applications. So if you have something that's even more compute hungry, hungry that might be difficult um to have in like a production system although i think that from you know like an academic perspective it's incredibly um interesting and definitely when training our point wise our goal was to match the accuracy of our pairwise but we found that through distillation um we were able to like pretty much map uh match um the accuracy of the pairwise. So distillation worked pretty well. And so it seemed to us like the pointwise was, you know, the most optimal solution, because we were able to approach the pairwise

option while being a lot more efficient. So that's kind of like how we were thinking about it. But yeah, pairwise was definitely slightly better. And I think interesting for like the community and we actually might just, like, release that. Yeah, it's so cool. And I think it transitions. I guess if I can add a couple comments on that, I think this concept of the computational efficiency is just so fascinating. We have these, like, late interaction models are really popular where, like, you're delaying that sort of point-wise encoding. And a paper I really like is called MICE. It's got the – you can have the mouse emoji sharing this paper. And it's about like exploring how much modeling capacity is needed in the cross encoder and ablating like two layers up to like four layers, eight layers. And I really love that one too. And then, yeah, like the thinking about pair wise, adding this extra computation. I'm curious if you could have a late interaction pair wise, set wise, list wise, like where it has flies over vector embeddings and just,

I guess the computational space of that is so interesting, but I think probably it comes into the, the greatest topic. Z E low. this is just one of my favorite papers it's so cool maybe you might take the mic and explain what is the ZELO method yeah so ZELO was one of the big unlocks and breakthroughs that we had at zero entropy and the idea is quite simple when training a re-ranker instead of relying on binary classes of positive examples, negative examples, we thought that using a pairwise comparison could be a much stronger signal. Pairwise comparisons are just so powerful. Even if you talk to a human, it's just so much easier to choose between two things than giving a score to something absolute or ranking things in the context of your brain.

So pairwise comparisons were really the basis of everything. And one of the reasons why we were interested in this was when we started training our Reranker model, our Pointwise Reranker model, and started looking at, you know, data sets with queries and annotated positive and negative samples, basically what you want to do to train a model is to have for each query a negative example that is hard enough so that your model learns anything. And so when we were trying to annotate data, if you find a negative example that's just too easy, you're not going to learn anything. And so you're trying to find a negative example that almost like looks like it's positive, but it's actually not. So it's hard negative mining. And the more you do that, the more you actually land on like negative examples that are just positives that were like never annotated.

And your model actually like learns less and less, which is so interesting. And it's, you know, just like a symptom of contrastive learning and the available data sets. And so we thought, okay, one solution to this is just doing pairwise comparisons. If you do pairwise comparisons, the signal is very clean. So, yeah, you can train a pairwise re-ranker in that fashion. But then the question becomes, how do you transform pairwise preferences into an absolute pointwise score? And that's how we were reminded of ELO and the Thurston model, which is basically this statistical model that allows you to turn a lot of these pairwise comparisons into one single score. So it's very similar to, you know, how like Chess ELO works. And what's interesting with the ZELO method is that you might initially think that it's going to be incredibly compute intensive to have like all of these pairwise battles.

But you can prove that you can have a pretty like sparse sampling of all of these pairwise bottles and still get enough signal to basically fit your model and find your ELO score. and so yeah we did a bunch of research in in that to figure out what was the most effective sampling strategy and all of the requirements such that we can basically choose which pairwise battles to make happen because they're of course expensive and yeah from all of these pairwise battles turned into an ELO score that's basically how we can create a point wise model from all of these pairwise battles. And that was roughly the training strategy for Z-Rank 1 and then Z-Rank 2. Yeah, that's awesome, right? I mean, ELO ratings are so cool, whatever the context is.

And, yeah, this idea of having, I guess, just starting from that insight of hard negative mining and that it's a totally intractable problem for humans to really go through all of the candidate documents. Exactly. and um we consider it for for some time like what you need to understand is like we came up with this by tour like trial and error like how do we how like we had never trained a re-ranker so we were trying to figure out how how do other people do this and what are the most efficient ways to do to do this and so um to have like extremely high quality training data um we we initially considered just like going the normal route and maybe even hiring humans to annotate the data we even like annotated data points ourselves to like understand how this all works um but yeah you need to understand that for search in particular it's just it's just completely impossible because for each question you have a pretty large corpus and searches only becomes hard for large

corpuses. If the corpus is too small, the task is too easy. And so you need a way for humans to have a question and then go annotate exhaustively a giant corpus, which is pretty much impossible and way too expensive. So, yeah, it was just like the only way to have high quality, high signal data for training. Yeah, super cool. I guess like maybe staying on the training data aspect of it, I'm curious about like synthetic hard negative generation or like generating synthetic documents and maybe that those could serve as the hard negatives. I don't think that idea is that. I think the use of synthetic queries is extremely popular, but I don't think like synthetic hard negatives is popular. Have you ever considered that idea? I know it's a little random. Yes, we considered that idea. We did a lot of research on synthetic data.

We did the obvious thing, as you said, of synthetic queries. So going from, like, one document to generating a pretty hard query for that specific snippet. We also did what you mentioned in terms of, like, writing a hard negative sample. um we experimentally um the like lms that we were using were not good enough to produce signal that we could clearly see the the model was like learning from um and so yeah falling back to to our original strategy was the most efficient way yeah awesome and uh sorry to distract with I thought I want to pop back up to ZELO. And I'm curious about extending beyond pairwise comparisons to maybe having, like, sets of three or four in the comparisons and how that might change the sort of ELO model and whether that would be more efficient or maybe more effective.

Oh, that's an interesting idea. That could be an interesting idea because obviously the – I mean, it will probably make it a lot more compute intensive as well. But one of the problems with the pairwise comparisons is like we assuming that there some form of like transitivity and adding like more comparisons at once may help mitigate that We haven't really explored this, but could be very interesting for Zelo 2.0. Yeah, the transitivity point is so interesting how you have this connected graph where it's like Connor versus Rita, Rita versus, I don't know, Bob, and then therefore I've compared against Bob through the link, and that's a super cool part of it.

Yeah. Yeah, amazing. I guess so maybe kind of popping back up the stack of talking about kind of JEV as a re-ranker, just broadly this topic of re-rankers and these models. And this is a bit of a pivot in topic, but I'm curious about this idea of the sort of shared pre-fill cache idea where you can, I think my understanding with the JEV models is that you have the pre-fill cache of, like, the state, and then all the decisions are in parallel. And I may be curious if that relates to serving re-rankers and if there's a similar design there of caching maybe the query and then all the candidate documents and how important that design is to the systems of, like, serving these re-rankers. Yeah, super interesting. we thought a lot about caching at zero entropy for the use cases where Reranker was used there was not a clear way to make the most of like

caching the queries but yes definitely for decision models like Jeb when you have a clear query that you're always trying to basically replay on like various inputs, definitely caching will provide a lot of efficiencies. And our rebanker does not support that. But if we were to actually retrain it for a decision model type of architecture, we would definitely investigate that. Yeah, I guess the bridge is maybe like the query would have to be maybe like 500. it would be like a really long query before that you'd really see any benefit from setting up that kind of way exactly exactly so like our maker did support like custom instructions but they were never it was never really um worth it um but for something like a decision model um it definitely

starts making more sense yeah super cool i guess i just that topic of this kind of particularly what's happening under the hood of inference with these decision models and re-rankers. Yeah. I just think that's a super interesting one. And maybe like one more question in this sort of headspace of like the infrastructure around re-rankers, decision models, and zero entropy. I'm curious, like how you think about sort of like GPU resources and what that, what kind of, I guess the GPU infrastructure looks like. Yeah. In fact, a lot of customers, like, switched to using ZRank because of the latency improvements that they saw from using our model. We basically, like, we're handling the inference ourselves. and so for the use case of Rerankers where you have, I guess the format is pretty specific

compared to a regular LLM in the sense that you have so many input tokens at once and you're trying to respond as quickly as possible so the batching logic can be optimized for this type of use case and so yeah we did a lot of work in like you know even quantized aware training and the way that we were serving and like batching the requests and you know spreading them across GPUs that we had like all over the U.S. to make sure that we were responding quickly enough and that was a clear differentiation for zero entropy compared to other models just because of how much we invested in this and the latency that we were able to provide. Yeah, that like multi-region latency is super interesting. I just got back from our company trip in Barcelona, and we were about 30 minutes outside of Barcelona, and I was kind of becoming more aware of,

oh, I'm sending these requests over to US East 1, being reminded of that. But that's super cool, that kind of GPUs. And how about just like getting access to GPUs? like is that a bottleneck or is like how is the pricing on GPUs when you're kind of building like Reranker as a service? Yeah, great question. We were getting GPUs from various sources. We had like a minimum capacity that was, you know, like monthly reserved instances that we directly rented from like data centers. the issue with that was that we didn't have like the largest clusters that's let's put it that way and so you know our neighbors were like these giant companies and so we would like regularly see like networking issues and like drops which was like a pretty big problem for us

we were also using some of the instance providers, you know, as what we call it like a slow mode, meaning that there was like a cold start time. And this was really for like peak usage or like very bursty, spiky type of traffic that we would like sometimes see. And so that's basically how we architected it. So with like, you know, our baseline capacity that was monthly reserved. And we were able to get like pretty good pricing because of like the fact that it was like reserved capacity and that we were like renting it without like any middlemen. middleman um and then like on top of this um yeah we were basically paying for like the additional uh capacity needed um on on top of of this um it was like yeah pretty pretty interesting to navigate this and look at like you know all of the shortages and of course like priority given

to these giant labs and companies. So, yeah, for inference, having a pretty small cluster compared to these other companies did lead to some inefficiencies. And, yeah, as I said, a lot of networking issues and random drops because of that. But we made it work. Yeah, it's so interesting. My understanding is kind of as basic as you have this sort of model where you have, like, a pool of always-on, always-ready GPUs, and then you maybe use something like modal or something. I feel like we have a good relation to modal at Weave8, and they have this great cold start where if you want to get 50 GPUs and maybe take, like, a minute or so to, like, spin it up. And that's as deep as I work. We're also using modal at 0NGP. So, yeah, we have, like, a list of, like, fallbacks just in case on top as well. Yeah, awesome. Yeah, it's super interesting.

Yes, I'd love to maybe pivot sort of out of the infrastructure and back to just sort of talking about topics around re-ranking. One topic I'm curious about, there is this paper called Drowning in Documents that kind of suggests that there is, like, a limit where re-ranking just can't really scale further. And I'm curious about, like, what you're seeing with re-ranking stages, like maybe following cross encoders with list wise re-ranking. And I guess generally like that sort of school of thought around, uh, first having like initial retrieval, then cross encoding and then maybe list wise re-ranking. And just if you, if you're have a positive opinion of building search systems that way or negative. Um, I think I have a positive way of like, this is probably like the right, um, the right system and the right architecture. Um, there was like you know a lot of debate around like you know like greb just works and like all

all of that um i don't know if this is like what what you're referring to um but for like pretty large corpuses of like completely unstructured data um like what makes more sense and what just shows to have like the highest like recall precision um clearly seems to be like these like multiple steps um you know for various reasons like i think you do need a mechanism like on top of like your initial um first pass um search embeddings are not going to be enough for like the whole like dimensionality problem um where if you're trying to encode information into like and embedding with like a certain number of dimensions, you can like prove like there were some papers about this as well. You can prove that, you know, you're never going to be able, even for like a small K, to retrieve all of the documents in the right order.

And so having like a step after this is definitely necessary. I think that point-wise re-rankers are simply like the most efficient way. they can almost match the accuracy of pair-wise or list-wise. Of course, they're not as calibrated and they're not going to be as accurate. But I think that they're still the most efficient way to reorder results from the initial search path and solve some of these problems. And so, yeah, I'm pretty bullish on this architecture. and yeah, I think you can like increase your top K in terms of like how many documents you're like re-ranking and that's also a trade-off. Like you could just like re-rank your entire corpus if it was like small enough and that would be great. But yeah, finding that trade-off

on top of like an initial like clustering phase is, I think, like a pretty solid architecture. Yeah, I think like that idea of re-ranking the whole corpus is so interesting because it totally changes the shape of the underlying like vector search problem. If you're going to be returning like 2,000 documents compared to like typically like it's 10 or 100. And I guess like just the nugget of being able to distill the pairwise model into Z-Rank, I think that one's really powerful. We did a study at WeVA where it's been sitting in archive for two months. I don't know if you're referencing it, but hopefully they decide soon. But where we did find positive results, adding this list-wise stage on top of the top 100 from ZRank, ranking from the top 2,000 from hybrid retrieval. And then we found that the list-wise ranker didn't scale very well. Like it worked well with the top 100 documents, but then that top 500,

it just that was too overwhelming for it yeah that's that's the issue that's the issue is like like intuitively you can get why like that would be hard to hard to scale um and like even like lead to like hallucinations of like documents that like are not even there like i imagine like some weird stuff happens after a certain limit um yeah i think yeah maybe like an even like idealized architecture like even better would be like you do your first pass and then like point wise re-ranker and then like if you really need it just like list wise re-rank the top I guess like a smaller n um in like in practice we found that in like most use cases you don't necessarily need like that much precision in like your top 10 re-ranks and you're mostly trying to like prune like the like 90% of like your,

your initial search, search step. So there might be use cases where like re-ranking the top 10 is like really crucial. But I don't think they're as frequent. Yeah. Yeah. I just think that's so interesting. and it maybe makes you think about like these long context LLMs and how they play with search and maybe that relationship isn't as adversarial as it's often like advertised as. Yeah, so I guess I know that we're kind of coming up on 40 minutes in the podcast and I have a few topics. I think this topic deserves more depth, but there's maybe like four more I want to get to them. So I think the next topic, maybe also talking about re-ranking is agentic search. And yeah, we mentioned earlier kind of the success of GREP with agentic search, BM25, and the argument is kind of like the agent is doing this iterative learning about the corpus while it's searching, and it gets this great feedback from GREP for BM25 because it has like this no matches found And there a pretty cool paper that called the questions gambit and they like sitting on this chest is the elo the questions gambit

and it's exploring like using re-ranking at like the first step of agentic search like really spending compute on making sure that first search is like really accurate and i guess i'm just curious what you think of the role of re-ranking in uh agentic search yeah great question i mean I think for like, I think the majority of search is going to be agentic search. So I just like don't even like discriminate between the two that much anymore. The way that I think about it is like, what you care about um is like what is the fastest way and like most token efficient way for your agent to get the context that it needs um i think like token cost is also like a topic that's pretty much like top of mind and so minimizing like the number of steps or like

just generally the number of tokens that your agent needs to take to gather the context that it needs is like pretty important. And so I think it's definitely worth spending the token budget on like a lightweight re-ranking step to make sure that your agent gets what it needs like in like the fewest number of turns possible, rather than like having an approximate search where your agent is going to keep going and going and going at it. Even if it's like a bit faster, I think overall it might just cost you more very expensive tokens from the LLM. And so I think that's also shown by, I remember Cursor like a few months ago showed that with Semantic, search, they were actually able to prove that Composer and their agents

were using fewer tokens on average, and that there were fewer undoes also, which means that it was also higher quality. And I fully believe that, and I think providing the best tools to your agent, that's what they deserve, and that's also what allows them to find information faster. And so, yeah, I think the Remaker stuff is definitely worth it. Yeah, I think, like, I guess for me, I'm kind of, these days I've been thinking that agentic search is only useful if you have multi-hop queries. And then I even think that it might not even be needed then. So I'm pretty bullish on the kind of, you still have the learning to search component, which I think is what people love about agentic search because you, you like RL, the LLM and the search loop. And then it like writes really great search queries and maybe has like developed a domain understanding and therefore it writes these great search

queries. But you still can like have that without the iterative looping, which is like a hundred percent of how I think of the agentic prefix in this. But I feel like you can still learn to search with a query writer and then you get a massive pool and hit it with some, with the re-ranking. And so I kind of, I guess I'm not really sold on the agentic search personally. I think it depends also on like which layer you're thinking about. I might be biased from like being at Notion for the past few months where I like mostly look at the agentic harness that they've built. and I do see that, you know, for example, if we're comparing different models and the way that they use this harness, some of the models are lazier and will, you know, do fewer searches or fewer queries and that does lead to overall a slight drop

in like the overall quality and like task completion for certain queries. And so I do perceive that for the models who are like searching more and gathering more context, overall, the quality of their answer is going to be slightly higher. At least that's what we see like for our harness. but it might be the case that like for other applications, if it's like a human asking a question and just wants like a quick answer, definitely in like the AI overviews and quick answers of the notion product, it's like not a harness and it's just like a simple one, one step search that powers that. So I guess it depends on like the product layer. Yeah. Super interesting. I think this even transitions to the next topic I wanted to ask you about, these instruction following re-rankers, because I guess the connection is, I guess something that is really cool about agentic search,

but I kind of will say that a query writing inference that you put at the top could also write several queries, so you can still kind of get this notion of these results came from this particular query, and that maybe is an interesting bit of information to add. but like these instruction following re-rankers are pretty interesting where you can sort of like change the notion of relevance I guess the best example is I'm looking for a similar query instead of I'm looking for a relevant document like if you're searching yeah yeah yeah yeah it's it's super interesting we in fact we discovered the use case for instructions following um through one of our customer who was building memory systems um and they had this issue where basically um the question of the user was why should i cook for dinner tonight and in the list of like possible responses there was like i'm highly allergic to

nuts like in a traditional sense i'm highly allergic to nuts is like it's not answering the question or it's like not typically what you would call relevant but in the context of like a memory system it's like probably the one of the most relevant things that you would want to give your system to help it give like a grounded and useful answer um and none of the you know queries that we were training on, like, looked like this. And so, like, clearly the Remaker was failing on this and giving, like, a pretty low score to that snippet. And so, yeah, I think it's exactly like the use case for instruction following is slightly modifying what you mean by relevance. And, you know, given enough training data that, like, has various definitions of what relevance means and additional instruction in the context of the Reranker. We clearly saw a drastic improvement in queries like that.

And, yeah, that was, like, super interesting to see and interesting to see that the notion of relevance itself is changing in the context of the types of systems that we're building. and, like, a memory system is pretty new and not something that you would have imagined, like, searching across, like, before this whole, like, era. Yeah, amazing. And I guess, like, my sort of selfish question is someone who is building search systems and are a WeVS query agent but not quite at the modeling layer. I'm curious if you think that you could encode the ranking instructions, like, with the query, like, say I just, like, append it to the end of the query, or if it will be like the interface to the re-ranker APIs will be that I have to pass in the re-ranking instructions separately? Sorry, if that's too ambiguous. Yeah, could you repeat the question? Sorry. Sorry. Like if I could just send the re-ranking APIs, the query plus the kind of ranking instructions,

I guess you could just parse it out with a string. Like, I think that's what Voyage does. They have, like, some delimiter where you just parse it, like, hashtag, hashtag, hashtag. Oh, yeah. What we did at Zero Entropy is basically we trained with, like, various formats where you can, like, use, like, XML or markdown or, like, just, like, hyphens or however you want to do it, like, bullet points. And we just trained on, like, all of these formats. So there was, like, not even, like, a need to, like, parse it in a specific way. And just like having a clear delimitation between like where you put your instructions and like where you put your queries was enough for the model to figure it out. Yeah, sorry. That was a question that as soon as I said it and heard it said, I realized it was a bad question. So my next question, I think that transitions from this nicely is how do you think about metadata with re-ranking? re-ranking because I know like with deep learning we kind of have this setup where we have chunks of

documents and we're training with that but then in the real world there's this interesting thing where you bring in like price or if it's in stock or the color and I guess I was always really curious how the how re-rankers hand like how you have like maybe like json format like just I guess and how that goes into the training data and just yeah I guess that topic of metadata and re-ranking Yeah, that's interesting. We thought about it a little bit, but it seemed like for, you know, categorical features like this, maybe it's easier to have like a separate model learn these patterns and use the Reranker more as like an input to that model. and that would just provide like the relevance of the snippet. But, you know, taking that into account when like re-ranking, like maybe with like an XJ Boost type model or something like this, we saw a few of our customers doing something similar

and we had not like investigated too much time in like adding arbitrary metadata because it seemed like this like roughly worked. So, yeah, that's what I think about it. yeah super cool at weva what we've done is we have a boost api where it does like reciprocal rank fusion between the results of a sort with the search i think that's pretty cool and then with our query agent we can like write the particular boosting filter and yeah that makes sense yeah i guess the xg boost thing is kind of a bit of overhead like you need to maybe train it a little bit which i yeah yeah i think yeah yeah what i mean is like yeah you have like all the ways of like including it in your ranking without necessarily including it in your re-ranker score directly which i think is like harder and more like fuzzy to do when you can do it like we did does it or like with a trained xg boost maybe that's so cool uh rita this is just so awesome going through all these different re-ranking and search topics with you um a question i love to

anchor the weva podcast with it can be about search or whatever is interesting you um as someone who's working in AI, leader in the space, what directions for AI excite you the most? I'm very excited. We talked about this a little bit, but very excited about task-specific models, you know, taking open-way models and then, like, postering them for, like, very specific use cases that are common, repeatable inside a company. I think that has, like a lot of promise and it's clearly like more optimal than using like a very general model. And we've been doing this at Notion and that showed like great results. We're seeing other companies doing it as well. And it's so exciting to use AI to train AI models. And that's becoming easier and easier. You see like, you know, other versions of Jeff got

released like literally two weeks after the initial launch um and i think seeing that you know iteration process becoming like shorter and shorter is super exciting um and yeah empowering companies to train their own and post train and use open way is like super cool it's kind of like an interesting subset of like the recursive self-improvement idea that it's not it's like that they're training smaller models and like making the systems more efficient, which is sort of like in turn improving like the agentic harness. Yeah, totally. I mean, you could argue like the small models like Luna, like maybe they are like an initial use case of like RSI. It's like the big model training, the small one in a way. And yeah, I think more of that inside other companies than the labs is very cool. yeah so interesting like what particular shape does rsi take on and like been looking into this and is it like with um the framing around like discovery and it's like auto research and you

just kind of think of it like it's discovering new knowledge or it's like this vision of like it's training these small specialized models i also like this machine studying vision where it's like it's studying the corpus and it's maybe training specialized search models is like a part of it studying. Yeah. Yeah. It's such a cool space to all that. Amita, thank you so much for joining the WeVA podcast, what you and your team accomplished at Zero Entropy. I'm sure you do amazing things at Notion now. It's just such an inspiration to everyone working in this space and just so grateful for your time and for joining. Thanks again. Yeah, thank you so much. Bye.