ランキングモデルは「間違ったもの」を最適化している――Zoe Weil（Sequen）が語るセッション内シグナルと購買最適化

TikTok is known for getting you to buy things you never knew you wanted. Online retailers would love that kind of impact, but they don't have TikTok's data. So they try to show the shopper what they're looking for and hope that sales will follow. Optimizing for what you can measure is not unique to e-commerce. AI teams often settle twice. They lean on signals that are easy to collect, like a click or a thumbs up, and they optimize for metrics that are easiest to evaluate, like relevance or benchmark accuracy. Then they hope it all adds up to what they actually care about. If you're building a rag system, a co-pilot, or an agent, you may be making the same bet. Zoe Weil's answer, at least in retail, is to look between the clicks. The company she co-founded and leads, Sequin, is built on signals most systems ignore. How long a shopper hesitates, how fast they scroll, and how many times they go back and forth between two products before adding one to the cart. and it judges the results not by what people click but by what they buy. I pushed back on two things, whether this is just age-old upsell and cross-sell with better data

and where it starts to look like surveillance. But she made a point about the playbook most teams follow that I hadn't heard put this bluntly. The biggest innovators in the space are social media companies and they publish a lot in this, right? but they have a wealth of data that none of our customers have so if you're a scientist at wayfair or at priceline you're going to read these papers and you're going to try to implement it and you know what oftentimes you're successful at implementing it except guess what it doesn't drive the same outcomes for you as it does for them because you don't have the data that they have and Seguin's big unlock is that we're finally able to get you that unlock without you having that data. I'm Sam Charrington, and this is the TwiML AI Podcast. For over a decade, I've been exploring the ideas and innovations shaping the future of AI through conversations like this one that help you understand what's real, what's next, and what matters.

Let's jump in. so anything that you see on the internet is ranked right whether it's a pair of pants that you're looking for at a retailer hotels that you're looking for for your next travel content media even the ads you see online are powered by an auction and these seem maybe to most people to be unrelated, but every dollar of online spend and consumption goes through a ranking model somewhere. And we power those ranking models for the world's largest consumer companies. Walk me through kind of the highlights of what you all are doing, what makes your approach unique and, you know, with an eye towards or ear towards the kind of technical innovations?

Unsurprisingly, when people think about ranking, they think about information retrieval. In fact, I went to graduate school for information retrieval, which is literally the study of search systems, right? How do I retrieve the most relevant items or documents for a given query, whether that's in a search bar or some kind of relevance module somewhere out there. And historically, and again, quite I would say intuitively, the wisdom here has been that utility should be measured by whether or not I'm able to retrieve documents that are most semantically relevant to a user's query, right? That's a very technical side of it. And that works incredibly well. And I would say, by and large, maybe it's not a solved area of research, but there have been so many advances on it. It's been pretty remarkable. I mean, the frontier labs have it, you know, pat down or down pat.

I forget what the idiom is, but they're pretty good at it. It's not really on its own an interesting, I would say, area of research. however in a commercial context you know modern recommender systems if you think about tiktok instagram and the like they're not really recommending content anymore and they're definitely not trying to help people find things instead they're bending the end user's will in subtle ways over time to make them want things that they otherwise wouldn't want right so that destination vacation, you know, the latest fashion that you might not need, you know, whatever it is that you can think of that people end up just, you know, buying on Instagram or TikTok. They do that not because Instagram and TikTok are, you know, really good at helping you find what you're looking for. No, they're very, very good at influencing your behavior. And the reason that they're able to do that is that billions of people use their apps and are, frankly, addicted to those apps.

They open their apps, you know, 10 to 21 times a day. And when billions of people do that, you have an insane amount of data on them. And so you're able to create these incredibly immersive, I would even say addictive experiences that are hyper-personalized and dynamic. And, in fact, the apps react in real time to the user's feedback, right? So even, you know, use, if you, what you see on the site or the app in the day, throughout the day, will vary based on your interactions with the app. But the truth is that consumer companies, regardless of how large they are, you know, regardless of how many customers they have, Wayfair is a good example. Priceline is another really good example. These are very large companies that have incredibly large user bases, do tens of billions of dollars of online transactions. But again, people don't use Wayfair the same way that they use Instagram or TikTok.

And so regardless of how big you are as a consumer business, you know, you are going to be data starved. And so in an environment where you don't have as much data on the user, where people are not addicted to your app, how do you influence their behavior in a moment of high intent to increase their basket size, increase their average order value, to increase their engagement on your site? again these aren't like big changes so on Wayfair for example if if you're going to buy a hundred dollar couch it's not like Sequin can come in and get you to buy a five hundred dollar couch and also a thousand dollar coffee table the point is I can get people to buy a hundred two dollars but I can do it consistently and at scale and when you do that that translates to forexing their target profit for the year in less than a month.

Upsale, cross-sell, like we've been, you know, doing this from a data science perspective, pre-data science, you know, we've been doing this for a long, long time, trying to get people to buy more. What's different about the approach? The TLDR, as I was saying earlier, and in fact, for the entire history of this field of research, people have been trying, the wisdom has been, if I can make it easier for people to help, to find things, then by extension, I can get them to convert better. That has been the prevailing wisdom forever. Every company that's ever existed in this domain has been trying to solve search and relevance from the perspective of search and relevance, make it easier for people to find things faster. That is not what Sequin does at all. We are not at all trying to help people find things. We're trying to influence their behavior in a moment of time when they have high intent.

How do we do that? How do we go about doing that? Which is, I think, the meat of the matter. So we started the company with a black box solution that was powered by what we call large event models. so you can think of it as first of all it's not a large language model we actually don't use large language models anywhere that's not our business whatsoever and we don't use generative ai either and in any case so we we started with large event models and the idea behind large event models is that the same way that large language models are able to generalize uh to text including you know text that they haven't seen before ever. Large event models are able to generalize to human behavior and streams of events all in real time. And creating something like that is incredibly difficult to do for a number of reasons. One, you need to be able to create extremely ultra-low latency systems that are able to

keep track of data at a very, very fine-grained level, right? So I'm not only talking about clicks and conversions and the traditional good stuff, although we use that as well, but I'm talking about hovers, scrolls. I'm actually talking even deeper than that. The speed of a scroll, right? The dwell time on a hover. How many times a user goes back and forth. Creating systems that are able to digest data in, frankly, nanoseconds, transform this data also in nanoseconds, put it through a model, several layers of ranking models in milliseconds, and serve dynamic, re-ranked items to a user as they're browsing a site all in under 50 milliseconds for thousands of requests per second is incredibly difficult to do. In fact, we created what we refer to internally as sequin ranking clusters to be able to even

make this feasible at all. And so there was a lot of work done there, not only from an algo perspective, but also from a data engineering perspective, engineering perspective, specifically for ranking and a ton of MLOps work and infrastructure work that was done. It's hard because you need to do three things that are very, very difficult to do on their own. You need to have high fidelity leakage-free data pipelines, which is a really difficult problem on its own. You need to have, obviously, models that are personalized to every user and continue to improve on those models. And then you need to be able to serve them fast enough such that rankings are able to react to live user feedback and signals. So building this system at all requires you to do these three things really, really, really, really well. And there are very few people on the planet that can do this for the explicit objective of generating outcomes for

businesses right and then so that's one part of the problem that sequence solves two even if you manage to build that system at all improving anything within that system is really really difficult because it's gated not only by how fast your scientists can do experiments and do research but the mlf component right the business component of it right like how much am I trading off diversity or freshness of products or, you know, bookings or media content for direct impacts to my bottom line, right? And so the ability to have this unified ranking interface that is focused and brings all these disparate parts of the organization together is incredibly helpful on its own.

But the real secret sauce and where we started the business was with a collection of proprietary ranking algorithms and harnesses that are designed specifically again not to help people find things And I know I keep repeating this Like I aware that I keep repeating this but that is really the golden thread Because once you realize it and you see hey even though you operating in the search space you not trying to help people find things you can unsee it You said that earlier and I think I had this like And I maybe expressed a little bit a degree of dissonance And that like that doesn't sound new to me And certainly you've lived this space more than I have But you know we've been talking about kind of next best offer And propensity to buy and like basket size optimization And all these things have been around forever

And so is it like old habits die hard and, like, you know, they approach – folks approaching this in a traditional way approach it like a search problem? And really the key is to, you know, be like maniacally focused on business metrics, which are kind of harder to do than, you know, focus on like traditional search metrics or what? Yeah, yeah, yeah. So actually the first part of it is true that, you know, there are really old habits very hard and they continue to see it as a search problem and try to solve it from a search perspective. Let me help people find things more easily. That is definitely a part of it. The other part of it is that, like, the biggest innovators in the space are social media companies. And they publish a lot in this, right? Amazon used to publish quite a bit about this, too. But they have a wealth of data that none of our customers have.

so if you're a scientist at Wayfair or at Priceline or at any of the companies that we work with you're going to read these papers and you're going to try to implement it and you know what oftentimes you're successful at implementing it except guess what it doesn't drive the same outcomes for you as it does for them because you don't have the data that they have and Seguin's big unlock is that we're finally able to get you that unlock without you having that data. And is that kind of aggregating customers across? We don't. We don't. We do not share any customer data between our customers. And that is what makes us truly, truly innovative, is that by relying so much on true in-session live signaling. I get it. So within the same customer interaction, you're extracting 10x, 100x more data than the typical.

And so we've made ourselves very, very independent of that. How do you respond to folks who kind of see that as like, you know, going deeper into this like surveillance capitalism kind of rabbit hole and like extracting more data about folks, you know, as they're trying to do pretty pedestrian things like, you know, that whole, I'm sure you've encountered that. Oh, yeah, I hate it. I hate it. I'm very, I'm one of the people that's probably the most vocal opponent. of that kind of, I love what you called it, surveillance capitalism. No, I absolutely hate it. We don't do that. And for several reasons. One, and this is what distinguishes us from social media companies, we're not trying to create addictive experiences whatsoever. We're not trying to get people addicted to anything. We don't store people's information. It's all in a live transaction. And I cannot tell you the number of times even I've been online

and I'm just trying to find well now I have a kid and I'm trying to find toys for my kid but before I try to find toys for my kid my my friend's kids and it's insanely difficult to do that right and I can actually I'm good at verbalizing what I'm looking for and it's still difficult to find that that thing that inspires you and so what what makes us different is that you know we're not surveilling people or not like chasing them from site to site to site for not doing any of that you already have intent which is why you're on wayfair.com you're on it already because you want to do something uh would just make it easier for you um to actually get to the thing that maybe you can't even verbalize to create moments of inspiration for you spontaneous inspiration for you. And so it does, yes. Priceline ends up making a lot of money. Wafer ends up making a lot of money. But the customers end up walking away happy. We're not doing anything like price gouging or

weird stuff. If anything, the difference is maybe cents on the dollar per transaction that we're able to increase. A sense is like you're using the data that's available through that interaction like a salesperson might in a store. I noticed you were looking at that cell phone a little bit longer. I'm going to kind of steer you to things related to that. And maybe you actually, you know, you're looking at the cell phone. What about this case also, you know? But we actually don't care about the identity of the user. We don't care about your name. We don't care about your last name. We don't care about your location. We don't really care about, like, you know, we really care about that interaction, the live interactive component. You mentioned MLOps several times. When you think about the kind of evolution of ML there and MLOps, what are some of the big things that you and your team have learned in trying to put this platform or environment in place?

I think we got lucky in a sense because maybe like six, seven years ago, Rust didn't have a ton of support in like open source AI libraries like Hugging Face. That began to change rapidly in the past, I would say, three to four years. And, you know, this is why I always say whenever you make a successful startup, like there's quite a bit of luck there as well. Timing is everything. Yeah, timing is everything. We got very, very lucky because we started building this company at a time when support for Rust and many, many open source frameworks, AI and all frameworks. Do you think you could not have done it in Python? No, I don't. I genuinely don't. I don't think that we could have because we, you know, ultra, ultra low latency, future transformations serving are so core to what we do.

I do not think we could have done it in Python. Certainly, we do a lot of R&D in Python, but the core of our back end is all in Rust. So, no, I can definitively say we could not have done this in Python at all. And we are very, very lucky to benefit from a lot of the open source contributors that have, you know, for free, devoted their time and efforts to bringing support for languages like Rust. I'm imagining that another timing tailwind is the availability of coding agents now. You may have needed to build a lot of reporting infrastructure before because it just wasn't available for what you're trying to do. But now you can just ask an agent to build that for you. For sure. I mean, a thousand percent. You mentioned that Wayfair was an early adopter of Transformers. Are your models Transformer-based? We have some that are Transformer-based. We have, I would say, all of our models certainly use self-attention, the self-attention mechanism.

I can say that without a doubt. All of them do. We have many mixture of experts models now, but they all rely on self-attention. I couldn't say that they use the transformer architecture across the board because, I mean, Now we have so many proprietary models that we have built that kind of, it's kind of like an orchestra of many, many, many, many models. It's like a mixture of experts, but for ranking. Is the idea at a system level that you've got all this data coming in and you are applying a bunch of different models, is it routing or it's like you're simultaneously passing the same data through a bunch of different models to kind of come up with new features? Actually both, right. So there's definitely a mounting component to it because if you're like in the advertising space, things are going to be different for you than, you know,

if you're just like in traditional commerce. But if you're operating within an ads module, let's just say Wayfair, there are things that the more specialized advertising models that we would use to power the bidding platforms might be helpful there. So in that case, there's some bit of routing involved, but also there's a simultaneous kind of information sharing and multitasking that we also do. The thing that I was trying to get at is a theme that comes up on the podcast quite a bit, and it's this tension between, like, we're going to take a bunch of data and train some big Mongo model end to end, and it's going to like take in everything and spit out the one answer versus we've got a bunch of data. We're going to apply a bunch of models to it that, you know, in some ways are, you know, creating features. And then those features are going to be used by some other model, you know, to or in, you know,

or some set of models or independently, it's like this kind of, you know, one big end-to-end thing versus, you know, we kind of know a lot about this problem. Like we're going to model, you know, individual aspects of the problem the way we might think about it as data scientists. And, you know, that has, you know, some advantages around explainability, model size, et cetera, et cetera, like. We're closer to the second because we do have these mega models for like, advertising domain or, you know, different domains, you know, but they tend to be on the whole much smaller than, you know, what you would see as large, ginormous models in the industry because the mammoth models nowadays are so, so big. Our largest models are a tiny, tiny, tiny fraction of that. What are some examples of, like, you know, a model input, output transformation? Like, I'm trying to get a sense of the granularity.

I can give you a couple of examples here. So, you know, creating, even if you look at, like, creating embeddings, right, for queries, right? Very, very simple text. Queries tend to be smaller size. queries in different contexts in travel and in travel hospitality and media and in retail can mean completely different things and so using a kind of generalized language these giant giant language models to be able to create these holistic representations are not as useful to us in our context because again keep in mind we're a ranking business we do one thing and we do it incredibly well and that's re-ranking so that's an example of where a lot of people would just use you know one kind of embedding model a really really great embedding model for text that is able to contain the most amount of latent knowledge about intent etc etc but like queries are like

what one two words maybe there is it like there is a lot it's very information starved so so our ability to uh narrow that to specific contexts and see just like two or three token um queries in a very specific dimension within the user's overall journey because the user is the first class citizen in sequence. It's not, and by the way, when I say user, I don't mean the user identity. I mean the user is a collection of events in a session. When I think about a user think about it as a collection of events in a session I don care if you John Smith or Adam whatever uh to to seek when you just a collection of events and a live session and how I'm able to best represent you in in in that context is what sets me aside from every other provider out there and um so we take these very very raw streams of events but where that gets

ratted to in other words how I end up representing that information to what then goes on to the final re-ranker is a big part of uh our secret sauce on what makes us really really special Is there a re-ranker or are there re-rankers for, I'm envisioning like, you know, we've talked about Wayfair. You go to Wayfair, you search for Blue Sofa, you've got your primary list and that needs to be re-ranked. But then there's like, you may also be interested in this. There's like a bunch of other stuff on the page that. We power all of it, by the way. Okay. And they're different. They're very different. So I'll give you an example. Even a very tangible example that I think you'll be able to, that any listener technical or less technical will be able to follow. Think about like complementary products, right? So let's say you bought a tennis racket and I'm not going to show you other tennis rackets.

You just added it to your cart, right? In fact, that will most likely dry down conversion. Well, you would think, but then we also have the classic, I just bought a refrigerator on Amazon. and you're just telling me refrigerators. Like, why is that? And that's why you would use sequin, because you would never do that. That's exactly – no, I'm serious. But anyway, you go on the Add to Cart page, and we show you complimentary items. So we don't show you other tennis rackets. We show you, hey, what about this, like, sweatband or this tennis ball? But which tennis band and which – sorry, which sweatband or ball we show you is going to vary from user to user. But the point I'm trying to get at is that that complementary module exists throughout many of these sites. So it'll exist on a product display page, but it also exists on an app to cart page. And how you rank items on a product display page, even though it's still a complementary module, will vary greatly from how you rank items on a complementary module on an app to cart page.

And that's because by the time the user gets to the At-to-Cart page, they've already committed to the tennis racket, right? Whereas I need to be able to show greater diversity on the product display page because they haven't necessarily committed to that tennis racket yet. Does that make sense? Yes and no. Greater diversity in – That's just one example. But the point is that they're both complementary models. Right, but I'm saying like if I search for a tennis racket, I mostly want to see tennis rackets. I don't want diversity in that list. You don't. Right. But once you click on it, you click on a specific tennis racket. Right. Then you don't know what I'm going to need next. Exactly. Well, we're betting that we have an idea before you do, or we can get you to click on purchase. Well, so this is an interesting thread. Like how, so I'm, you know, I'm in this journey. I search for tennis racket. A bunch of tennis rackets have come up. I choose a tennis racket I add it to my cart

and then you show me a bunch of other stuff how does any of the information you've collected about dwell time and all these other things while I'm looking for tennis rackets apply to which headband, which set of balls, which anything else. A ton of things oh my god, you give the site so much information you might not even realize so like how did you end up deciding between the blue tennis racket, let's say, and the pink tennis racket. How long does it take you to decide that? And how does that compare to every other sequence of events that every other user has ever had? Like, is it classic, like, collaborative filtering, you know, folks who... No, because in this case, keep in mind, the identity of the user actually doesn't matter, right? And it's all in real time, right? So I look at the sequence of events that I've collected in real time. when people usually, let's just say, hover for X amount of time, go back and forth X number of times before settling on this time.

It almost sounds like an implicit personality modeling. Like this is an overly analytical, anally retentive person. He keeps flipping back and forth between these two things. He's scrolling over these lists. kind of, you know, he's like scrolling over the specs or something like that. Yeah, yeah. So that he probably wants, you know, something different from the person who like is flipping through all the pictures. No, it's the difficult thing to do, again, more than anything else, is like keep in mind the data, the environments you operate in are incredibly data starved, right? So with as little information as possible, what can I do? And most of the reliance comes from streaming in session events and getting all of that to a model. First of all, the data comes raw in a live session. So you have to clean up all that data in, like, literally nanoseconds. And it has to go into a model, which has to then spit it out in milliseconds.

So it's a lot when you think about the entire journey that the raw data has to go through. But, yeah, the idea is based on how you got to finally adding that pink or blue or whatever color tennis racket to your cart, I'll show you just the tennis ball or sweatband or bag that you're going to buy. And in fact, one of the things that we are very, very, very good at is increasing basket size, like regularly. the other thing that we're really really good at is typically historically you know ad revenue and organic revenue are considered competing metrics which makes sense typically usually when ad revenue goes up it goes at the expense of organic because you at the end of the day have a list of items and either organic goes to the top or ads or whatever and what we've been able to

consistently show is that we're actually able to drive both up together so that one doesn't kind levelized the other, which allows us to compound these massive, massive gains on incredibly mature, relevant stacks. Elaborate on that for me. When you talk about ad revenue, are you talking about ad spend that's getting the user to the site, or are you talking about like at an Amazon where there's promoted items versus second. Okay. Right. And I guess I think about that from an Amazon perspective because it's, you know, very, you know, it's surface is, you know, more or less transparent, you know, a little bit more than less, you know, whereas I don't know that when I think about Wayfair, I think about sponsored items versus non-sponsored items. But, you know, a side of that scale is probably doing that whether or not I know about it is kind of the. I think they're fairly transparent about it.

I just think that you see it more on Amazon because Amazon has a much, much larger ads business. And they're a huge, huge company. And so you're saying that, you know, typically when an organization is kind of, they optimize ad revenue as a... As the primary objective, conversion tends to go down, yeah. Yeah, okay. Because people will, because they're optimizing for people clicking on stuff. as opposed to buying things as opposed to buying things and what Sequin has proven that we can do time and time again is we can get people to click significantly more and get them to also buy after the fact why is that is the same thing we're talking about just more data like is it that you're individually optimizing these things and so they both go up or is it some cross no they both go up we're just creating like much more we're showing people things that they actually want to see based on what we call micro feedback that we get.

Things like scroll speed and stuff. Yeah, it's funny you say that because I was thinking just a bit ago about how, you know, typically the big kind of aha moment for relevance is if you can find, you know, some feedback, like any feedback. And usually it's like, or, you know, historically it's been in, I'm thinking about like RAG systems, for example, like, you know, did the user click? Like, you know, that's a, did the user click on the thing that we showed them? Or did the user thumb up or thumb down? Like, you know, there's, the teams have kind of been starved for like, you know, any kind of feedback and like they, you know, have very few levers for this explicit feedback. And now like you're just creating way more, way more feedback. Implicit feedback.

Yeah. Yeah, no, we rely, I would say overwhelmingly on implicit feedback data. Overwhelmingly. It's crazy. I would say I was talking to, I was talking to, and I'm not going to name because, you know, I don't want to embarrass them, but I was talking to C level executives from like a very, very major bank. so they're like you know you'd be like you know work with us we're this giant bank and we have all this transaction data and blah blah blah i was like actually that's like that's the least interesting data you could give me because you're only showing me a sliver of human behavior that transaction data is useless because 99% of people don't actually convert i'm interested in the 99% of data you actually don't have. And that's what I have, and you don't have that. We didn't get very far in stack. Like we mentioned, Russ, like the things that come to mind as you describe the system where questions are like analytical databases, NoSQL, you know, queues, confluence kind of

thing. Like what are some of the infrastructural, the data infrastructure elements that allow you to do this? Yeah, so we definitely use a lot of NoSQL for sure. Distributed databases, we definitely use REST. We rely heavily on, we rely very, very heavily on open source technology. We use a lot of Hugging Face, huge fans of Hugging Face here. I can tell you also what we don't use much of. We almost do not use LLens at all. um this is not relevant to our business uh we're not trying to do generative anything we're not a lab uh we're not trying to reinvent the wheel there um uh so yeah honestly it's just a it's a lot of ranking models it's actually like the thing is that getting these models to work

requires you to have an innate understanding of what features drive value and that is not written in any research paper. And so what we have done is we've created a harnessed library with proprietary institutional knowledge that we've codified, what we call residual knowledge that Meta has never published about, that Google has never published about, that Pinterest has never published about, their biggest moneymakers. even the papers you see them publish like they're not the scientists that work on them have to wait for months to get legal approval and the version of those papers that actually ends up getting published is a very very watered down version of those of that line of work and so what we've done is that we have created a harness library and a knowledge base of some of the most effective proprietary methodologies to drive outcomes in ranking And some of it has come from hiring folks from these institutions and you know gathering residual knowledge that not published anywhere

And then second of all, we have four deployed engineers embedded in all the customers we work with. So we actually don't need feedback data from the customers, but residual information that we gather about what works, what doesn't work is all codified. You mentioned or you referred to, is the harness and recursive like the same concept or is the harness a subconcept? Harness is one component of the entirety of our recursive stack. So you can think of the recursive stack as like being this engine with many, many different components. We have a proprietary knowledge store. We have a harness library. We have what we call frontier ranking models, all of which work together to power the recursive stack. And so our core IP is the recursive stack itself that produces these insane ranking models that are able to, you know, 4, 5, 10x Fortune 500 companies revenue in just a matter of weeks.

and that's why we are able to allow them to then own the ranking models, right? Because the real meat of the matter is the ability to continuously build ranking models, better and better and better and better ranking models that continue to drive revenue. And so when you talk about this harness, what makes it a harness? Is it just a library and like harness is a cool word or is there something harness-y about it in the same way that, you know, codecs or quadcode is a harness? Yeah, so it's actually a collection of harnesses that we have. Definitely not a cool word at the time that we started building it in. We just needed a way to specifically have what we call eval harnesses. to find what we call the commercial ground truth of interactions.

And so that's why we started to build these harnesses. And we quickly realized in short order that even, you know, in the space of evaluating these ranking models, how we're able to do so offline in, let's say, the travel space, which is considered obviously higher ticket items has a ton of overlap with the furniture space for example which also has higher ticket items but guess what it doesn't have a lot of overlap with with the ad space because the user is just clicking on things and that translates directly to revenue but it's not coming at a cost to the user so how you evaluate these different kinds of ranking models requires you to have a very, very rigorous approach. And so we came up with a collection of these evaluation harnesses, and that was the start of our harness library.

You call the tool or the platform recursive, like what's recursive about it? Recursive Ranking Intelligence is kind of the first name. Is recursive just a name or is recursion a part of what you're doing? Actually, if I were to be very, very specific about it, it's most similar to like tail recursion, meaning that you're able to, instead of having to rebuild the entire ranking, iterate on the entire ranking algorithm and rebuild it from scratch, which I would consider iterative and not recursive. You don't have to rebuild the entire ranking stack every time you're learning. So you're able to learn continuously and make minor modifications here and there. And in doing so, you're able to compound gains at a much faster rate at a significantly less fraction of the compute. So that is, if I were to get very technical, that is why we call it a progression,

is that we don't have to rebuild ranking models. previously when we started the business we actually did so we would train the ranking models a daily sometimes hourly but it would be like from scratch and it took us quite a bit of time actually to figure out a way how can I have how can I quote-unquote iterate on this without having to rebuild it from scratch every single either day or hour or whatever cadence and insert that recursive element in there. We talked about this a little bit earlier, but kind of with the context of the full conversation thus far, what is large event models? Is that just an instance of one of your ranking models or is that a particular architecture or? It's one kind. It started as the thing that we did. Let me be very clear. That in itself was insanely difficult for us to do.

To create these models that were able to generalize to stream of events. It sounds extremely intuitive and it is. But getting it to generalize across customers and domains is insanely difficult. Like intuitively it makes a lot of sense. but to get it to work in travel and media and retail at different cost points, different geographies without relying on any end user identity is really, really, really difficult to do. So that was one of the hardest things that we began to work on. What's the input? Is it a stream of events? It's just a stream of events. And each event kind of has a label and a value, for example? Yeah, so it would be like hover combinations, right? So it would be the query, the embedding for the query, and that query could either be literally the source item or it could be an actual query of words that users search for or whatever. That was the first, very, very first thing that we worked on, and it was the black box model on top of which we built rankers.

then we built what we what we called uh large mesh models which were like large event models but they are much more relationally focused so now i'm looking not only at streams but i'm looking at many streams simultaneously i'm looking at the relationship between those streams together and trying to create generalized representations out of that right um then we started creating different classes of what we called frontier ranking models, which were independent of large event models and large mesh models, but which large mesh models and large event models contributed to, right? And so we built these collection of many, many, many now in the hundreds of thousands of frontier ranking models that we have that are proprietary, that are not published anywhere, a lot of them with very novel architectures, and that then became the first building block of the recursive stack.

Over time, you started to develop harnesses and knowledge store, et cetera, et cetera. To what degree does the idea of time series models or time series modeling play into what you're doing? Do you think of these events as time series, or do you just kind of capture a frame of events and kind of process it independent of, you know, longer time sequences? Earlier, I will tell you that earlier it played a much, much bigger role. Earlier in our journey, now I would say it's a small part of a growing stack. And I'll tell you why. I think I can give an example that is perhaps intuitive for people to catch on to. You know, time series and sequence-based modeling works really, really well for languages and generalized reasoning because you can show A follows. Sorry, B follows from A.

And there is a general order to things. And even though we're in the ordering business and our whole business is ranking, I'll take you back to the fundamental concept that the first class citizen at Sequin is the user. And the user does not behave as cleanly or logically as you would expect. There's a lot of randomness in people's behavior. People aren't like languages. language, despite however creative that you can get, has a certain repetition to it that human beings simply do not have, right? You could be in the middle of buying a couch, right? And your dog just comes barking in and completely ruins your train of thought. And then you suddenly realize, oh, I'm going to buy a dog bed too. You know, there is something inherently unpredictable about human behavior. To the degree that we've talked about like a big unlock being like capturing more information from the user.

Capturing the events of the browser is certainly complex but doable. But then you talked about like the infrastructure to manage those events at scale. It's a lot. And then transform all of those in real time. And then know what to transform in it. Yes. Which, by the way, we have dedicated models to do that. So you'll literally have streaming data. that's going in, not to one model, but it's going into like hundreds of models, which all have to then output that data, reconstruct it, put it into the final ranker, and have it all be served to the user in sub-20, sub-50, sub-100 milliseconds. It's an enormous infrastructure and modeling undertaking. With that in mind, when you think about kind of the technical horizon or the innovation opportunities, like what are the big opportunities to go after that will help you kind of push your system and approach further?

There are several routes that we're looking at right now without revealing too much. I think, you know, one of the biggest debates we've had internally is how much do we invest in traditional representation learning of, like, images or texts and the like versus actually continuing to innovate the ranking models themselves and the nuances there. and I was talking about how our eval harnesses and the library there is absolutely critical to our success and so I would say right now at this point in time anyway we're very very focused I would say on the latter rather than focusing on things like that we think are going to be very commoditized if not already you know in terms of having multimodal representations of

like hero images on company websites or whatnot. So I would say that is probably where we're most focused on, are on things that actually make the core ranking model itself better rather than the auxiliary components that involve more traditional representation learning. We're going to leave that to the big image or video or other frontier labs to do. and open source projects and will continue to benefit from that research as it comes out. Well, Zoe, it's been great chatting with you and learning a bit about, you know, what you're up to and how you're approaching ranking. It's been super interesting. Thank you so much for having me. It's been a huge pleasure, and I'm a huge fan of your podcast. Awesome. Thank you. It's an honor to be here. Thank you.