忘れることで発見をスケールさせる：Shangyin Tan 氏と語る Weaviate Podcast #149

Chang'an, welcome to the WeVA podcast. Thank you so much for joining. Yeah, yeah. Thank you. Thank you. Yeah, I'm very glad to be here. It's my first time on a podcast. Awesome. Yeah, I'm super grateful to be hosting you for your first podcast appearance. And yeah, I've loved reading your work on Scaling Discovery Requires Forgetting. Obviously, seeing a lot of great works from you in the DSPi community, works like DSPi Assertions. And yeah, just really happy to finally be, you know, brainstorming and talking about all these things together. Super awesome. So I'd love to kick off the podcast by, so your paper, Scaling Discovery Requires Forgetting, I think, just maybe this first term, discovery, and just maybe what is discovery? Yeah, yeah, yeah. So this is a very good question. I think a lot of people would have very different definition of discovery. And our point of view of discovery, it comes from one of our older work called JEPA. So JEPA, in short, is a prompt optimizer that leveraged LOM as a proposer

or as a optimizer to optimize your prompts. And then when we were doing JEPA, we kind of like started to realize that JEPA is a great framework that not only works for prompt, but also works for other texts. So we kind of like started this live software where we're tuning these kind of like auto-research algorithm that leverage LOM or even agents as a proposer or optimizer to optimize your textual artifact. So this can be applied to things like code, kernels. A lot of people have done this for different kernels on GPUs and even agent harnesses. So you can literally optimize anything with your text. So what we mean by discovery is that through this optimization process, you kind of like form this very natural optimization tree that, you know, each candidate will be depending on its parents that, you know, have some benefits over the other candidates.

And then you have this tree, and then this really resembles the discovery structure of the textual artifact. Yeah, there's just so many interesting ideas. and also the JEPA versus GEPA. I thought we were going to call it GEPA, but I hear it back and forth. And I just came back from our WeV8 company trip where we were studying the JEPA model from Jan Lekun. Oh, yeah, yeah, yeah. It happens to be the same name. Yeah, a lot of people are, yeah. We're starting with the G, not the J. But that JEPA is also very good work. Yeah, so I guess just maybe if we could first, I love the evolutionary structure of GEPA and the kind of how that creates diversity, the tree of prompt proposals that definitely we're going to dive into that throughout the hour podcast. But I think just this topic of, you know, I think like optimize anything and this extension beyond optimizing prompts into optimizing code as well. I guess I'd just love to maybe start on that kind of idea of prompt versus code optimization and like what's similar, what's different.

Yeah, yeah, yeah. This is also a great question. So after JEPA, we published another paper called Optimize Anything. So this is literally like the name of the paper. And in that paper, we discussed the difference of optimizing different kind of artifacts. And then we proposed like three different set of questions. So prompt optimization is kind of like a generalization test. If you think about it, it's similar to harness optimization where you wanted to optimize something that's general enough for a large variety of inputs. Because you don't want your prompt to just work on a single user career. You actually want your prompt to work on a lot of different things. And then in your training, you actually, or in the optimization process, as we call it, you only have access to a subset of the whole data distribution. So when you're optimizing prompts or harnesses, you cannot, there's no feasible way to know every single input that this prompt will be run on.

And then the program optimization is a little bit different. A lot of times when people are doing the program optimization, they only care about this particular program. So think about an example like kernel. If you're optimizing a kernel, oftentimes people are just optimizing this specific kernel, like a MatMod kernel or whatever, on a specific GPU. So you don't really care about whether this code actually generates on any other things. So the optimization process is something that you can actually trust. So if your score goes to like 10x in your optimization process, you know this will go like 10x in your production because you're directly optimizing towards that target. So we call this like a single problem optimization in the paper. And then there's the third type, which is, you know, in between of those two. You're not directly optimizing a single artifact. You actually wanted to optimize a group of artifacts. Let's say you wanted to optimize, like, 10 kernels together or, like, 100 kernels together.

So this optimization process, you cannot, like, rewrite every kernel at a single time. So maybe you'll have, like, something in the middle where, you know, you're still writing a prompt. You're still optimizing a prompt that generates a kernel, but you don't care about generalization outside of your training data. So you don't have to, you know, be too careful with validation and stuff like that. So we kind of like separate problems between these three. Yeah, that kind of generalization, like how generalization changes when you're training maybe a model instead of a prompt compared to writing code. I think that's a super powerful nugget. And I also kind of think of it like the evaluation is maybe more expensive. Like we just had a podcast with Mert, Shovem, Shu, and Alexander from the… Oh, yeah, yeah. We're friends. We're very good friends, yeah. and it's like about this particularly I really was interested in this idea of you know say using Cloud Code to write an entire database from scratch and just I guess trying to expand my scope of like what's possible

with Cloud Code and I guess for me the key difference between that and like training models or like evaluating models is like particularly that kind of evaluation is really expensive because you have to like load up the database with a ton of data and it just takes a while kind of and I guess Right, right, right. Yeah. Yeah. So this is, I think we can get into a lot more details later, but I think this is also one of my recent thoughts on this space. So I think, you know, I always felt like there are two, like, parallel timelines in the world of, like, doing auto research. So one is, like, this JetPod-style stuff where you're, you know, using a framework to do your optimization. And then the other parallel timeline, which is also great, is like capacities auto-research styles, where, you know, all you have is one coding agent plus a prompt. And then in this prompt, you tell the coding agents, you know, what to optimize, how to optimize, and then how to evaluate. And then, like, maybe, like, there are, you know, pros and cons of those approaches,

but if you think about, you know, since, you know, the auto-research starts to get more popular, and a lot of people are actually using that in their actual, you know, work, day-to-day work. So the auto-research one, I think one of the biggest problems, as you said, is that the evaluation might take a lot of time. And then because all you have is a single coding agent, all the control is in that single coding agent. If your evaluation takes a very, very long time, the agents sort of like have to wait there until it gets signals in order to proceed. So I think maybe like Shu, they introduced their recent work, Dual Discovery, that's basically fighting the problem of what we should do when the evaluation takes a long time. And then the forgetting work, maybe we can talk a little bit more detail, is that when I started that work, it also came to me that sometimes evaluation takes a long time.

So what we need to do about that. Awesome. So, yeah, I think that would transition nicely into, I guess, like, yeah, talking about scaling discovery requires forgetting. And maybe just if we could start, we've kind of started on the former end of what is discovery, what does discovery mean? Now maybe if we could talk about the role of forgetting, what is forgetting in this particular context. Yes, yes. So when we were doing a lot of the experiments for optimizing anything, One thing that's pretty shocking to us is whatever method we're doing, because when we were doing Optimize Anything, one particular thing that we have in mind when developing that framework is that we want Optimize Anything to be the universal front end for all these kind of optimizations, and then you can switch different backends. So we actually implemented a bunch of different backends for Optimize Anything, including JEPA, auto-research, like the agentic one, and also another framework called meta-harness.

And then when we were doing experiments with that, we realized that all these frameworks, all these different backends, they all hit the same problem when doing this long trajectory optimization where after a while, the progress seems to be stuck there. So you see very fast-climbing trajectories of these coding agents or optimizers, they're extremely good at turning a single or a very simple baseline into a more sophisticated one. But as you go to maybe two hours into this optimization, things start to flatten out. You don't see a lot of progress made through the whole process. So this basically made us thinking, what's blocking the progress there? So we did a super, super simple trick in that experiment where we just take the best, whatever we have so far, the best candidate that we found from this process, and then we put that into another optimizer.

And at the time, we were thinking maybe like Jirpa got stuck. Maybe like all the research can like unblock this process. So we take the candidate from Jepot and put it into auto-research. And then it starts to go very well. This other optimizer starts to optimize this. And then after a while, I actually realized that putting it into another optimizer is not the key to the success. Actually, the key to the success is to forget about anything else. Just to have this best candidate so far is good enough for unblocking the process. So what forgetting means in our context is that, you know, basically we don't want any context for the optimization once you've got stuck. We only want the single best candidate in this particular case. Yeah, I guess it first makes me think about the Pareto frontier idea in GEPA, and I just love that Pareto Frontier idea.

It's so cool how you have, like, the performing well on some of these samples. And I guess that's kind of one idea of, like, having the history. But then it makes me think a lot about just how I'm using Cloud Code personally to do auto-research style optimization where I think about forgetting as, you know, getting rid of that massive history as it's going through its iteration. Yeah, I think anyone have used coding agents extensively will have this experience where you started – the coding agents or the model started to drift a little bit, right? So it starts to do something that you do not want it to do. And then you tell it – you tell the coding agent that don't do this. And then the coding agent will probably follow instruction for a few turns. And then after a while, it will start to do this thing again because this thing is in the context. It's not, especially the older generation of models, it's like so, they're trained to attend to a lot of its context.

So this is something that, you know, unfortunate for all these coding agents, and then it also showed up in this, you know, long trajectory discovery tasks. Yeah, I think there's definitely a lot of research still to be done on long context, than just even if it can process it, is it optimal to process it? I mean, I think there's obviously context compaction and, like, all that. But, yeah, I guess I'm still just kind of – so it then kind of motivates a lot. I guess thinking about diversity, you think about, like, maybe agent teams. I guess a lot of people are thinking about, hey, I've got Claude Code running on one terminal. I want to have eight going at the same time. and I think this is a really interesting topic for people is just how do you think about designing, you know, discovery agent teams? Yes, yes, yes. This is also a great question. I think, yeah, one reason that, you know, one big reason that forgetting will work, like even for single agents,

is that the single agent will, you know, lose, I think, lose two things with longer session. So first, they will lose diversity, as you said. that as the session goes longer, it will be very stubborn on a particular direction. Like it will try different versions of this direction because they have seen this good signal from that direction. Let say algorithm A right So the coding agent might have already seen a very good results from you know implementing algorithm A so that the coding agents are like very you know motivated to like sort of like tune this algorithm A to make it better This is like similar to, like, people's, you know, when people are training models, they find a good architecture and then they were, like, two different hyperparameter to train the model better. So this is kind of understandable. And then the other thing about this single agent one is that it's like really when you're having a longer section, it will also lose motivation.

Like the one thing that this is a little bit sorry, this is a little bit maybe let's get back to the diversity. I'll talk about that later. So about the diversity idea, when you're having multiple agents, you're of of course, increase the diversity, right? So if you're sampling the same trajectory multiple times, you're sort of like having this diversity by like repeated sampling or parallel sampling. So one would argue that, you know, one way to like increase diversity in this kind of like discovery task is just like stack agents. So you can stack basically like, you know, eight parallel agents. If you're at OpenAI, you can stack maybe 10,000 other agents. And then this will give you some sort of diversity. But the thing about these kind of auto-research tasks is it's not like all the tasks are long trajectory,

which means that for the current set of coding agents, no agents can one-shot this problem. They will all try to implement something first, and then they'll try to modify it. And then when you are doing this trial and error and then doing this iterative improvement, you can't go away from having a long context in your optimization process. And this is why even if you just wanted to do parallel sampling, even if you just wanted to do this most simple way of scaling up the discovery, you still have to do this kind of like forgetting. So in the paper, we actually showed that, you know, forgetting not only helps for the team setting. So the team setting is where we actually allow like collaborative communication. It also works for like single agents and parallel skating. So it works for all of them. Yeah, I guess I think like one way, I think there's kind of like two schools of thoughts with designing the agent teams,

one of which is maybe less relevant to this, but that's where you have like different roles. Like you're like the planner, you're the implementer, you're the designer, you're the educator. And they're all kind of working on these different roles. And I see that as a bit maybe adjacent to this idea. Whereas this idea, I think maybe you would start off by, by giving your initial specs a cloud code. And you'd say, uh, what maybe propose like 10 different ways of solving this problem. And then they're all working on those 10 things in parallel. and then you have some sort of way of synchronizing it and how you prune it and say what's the best to keep going? Yeah, so the way that we implemented the teams is actually, I would say a lot of people would not do what we're doing. So we had the simplest way of implementing a team. You could never imagine. So basically the way that our teams work is that we'll just open up everyone's workspace so that other agents can view,

so all the agents can view each other's code. And there's no direct communication. So the main reason that I do that is that direct communication actually costs a lot of tokens. And a lot of times it's wasting the time or the token of the agents. and because the task, I think this also depends on what kind of task you are solving. If you are solving a task where it requires a more extensive collaboration where everyone works on different parts of those bigger projects, then I would say communication is definitely, point-to-point communication would definitely be a very good thing to do. But here, if you think about it, every single agent in this team is working towards the same goal. They wanted to make this kernel run faster. So it doesn't make sense for, like, this coding agent that's, you know, having a slower kernel to, like, talk to, you know, that agent that already had a faster kernel about what's going on, right?

So the thing that actually makes sense is that this weaker agent or this agent with a weaker kernel, well, actually, like, this agent can actually, you know, check what have the other agents done. and then it will try to implement others' idea into their own code base. And this actually works very effective for our use case. And we sort of got this idea when we were launching this parallel... It was, I think, a few months back when we were launching this parallel experiment and we were not sandboxing the agents very properly. And the agents will start to actually look at each other unintentionally. And then I was getting this very good score, and I was like, yeah, why? And then I found this. So this works actually very, very well for us. The only caveat in this implementation is that naively the agents will, you know, try to copy each other's code.

So we made this, like, and then you kind of sort of, like, lose diversity, right? So if everyone will be on the same code, then you are starting to – even if you are having agent teams, they will start to behave similar to parallel sampling because everyone is on the same code base. So we implemented this small, small trick where basically you don't allow the agent to read others' code. You only allow them to read others' commits. So this is a super dumb trick, but I think the way that it works is that agents are not able to see the full file. It can only see the diff of the other agents. And a lot of times a diff of the agents in this setting actually corresponds to an idea or an algorithm. So the other agent, like the weaker agent, will actually try to incorporate this diff into its own code base

instead of copying the whole thing. But this is like, I would say, maybe we'll publish actually another blog post to talk about this. Like, this is a very different view of doing communication. Yeah, I think I'm about five years into my software career and, you know, this is absolutely the best way to see some code changes compared to like reading it a whole entire. Yeah, yeah, yeah. Yeah, exactly. Yeah. Yeah. So this kind of idea of Git for multi-agent communication just sounds so powerful. I think maybe the other solution a lot of people probably use is you'd have a markdown file, and maybe you have an orchestrator agent that is sort of checking on each of these parallel agents, like checking if they've submitted their sort of markdown to give their current progress update, and that's maybe one other way to sort of do this. Yeah, I think another very common way that a lot of people do is to have this collaborative scratch pad where every agent can basically write to and that every agent can read from. So I guess as the research community, we're still super, super early on understanding what kind of method works best for different kinds of problems.

From my perspective, I feel like these communication methods are all very problem-specific, where for different problems, you kind of need different communication patterns. and yeah we picked one that works for us but I think there's a lot more opportunity for research there. Yeah I guess it reminds me of coming back to GEPA and how you have this Pareto frontier and I guess you kind of have this idea that when you find a little prompt tweak that works on one of those outlier samples it just sort of goes back into the current, it just kind of goes back into the evolutionary pool, and there isn't maybe such a structured communication of, I found this update on this particular sample, and definitely just motivating this whole, like, natural languages, optimizers, how structured it could be. Yeah, yeah. So I wanted to maybe talk a little bit about,

So basically, I think our thoughts for JEPA is the reason why we invented this Pareto frontier pool in the first place is to actually increase diversity. So if you think about it, before JEPA, a lot of what people would do for auto research is that they'll just always keep the candidate with the best score. and then you actually start to lose diversity because a big problem in prompt optimization is that let's say you have like 10 different problems you wanted to solve and then maybe prompt A can like solve problem 1, 2, 3 and then prompt B can solve, you know, problem 4, 5, 6 so both of them solve like 3 questions but they work on completely like disjoint set of problems and that means that both of the prompts will actually be valuable to be the base of your evolution. So if you remove, you know, either prompt, you're kind of like are losing important information.

And this is the way that we're, you know, we're like sort of like keeping this diversity in JEPA. So I think, like, JEPA is a great algorithm that not only works for, you know, that not only works in our particular framework, you can actually implement this idea, this, you know, Pareto pool idea into a lot of other frameworks. And for guiding lines of work or optimizing lines of work, a lot of the problem setup is a little bit different, that we don't care too much about generalization. So I guess the way that you wanted to, because you don't care about generation, it's harder to form this Pareto frontier pool. So the way that we're keeping diversity kind of like starts to evolve, right? So we started to think more about what we can do to improve the diversity of generation. And then we thought about, you know, doing parallel sampling.

That's definitely one way. And then this forgetting, like, it's kind of like natural and extension of, you know, still keeping more diversity. I think, yeah, diversity is really important in discovery. Yeah, so I might be maybe popping back to our earlier conversation topic on discovery of prompts versus code. And I'm just super fascinating in this topic. And I guess, like, earlier we discussed, like, generalization may be totally different in this context. now I'm thinking that maybe if it's if the code is trying to optimize as a database system and then you think of the Pareto frontier as different kinds of workloads you could have maybe one workload where for some reason for sure for sure yes yes that's a great way to think about it yeah if you do have such use cases yeah this would be great like if you think about it harness is also code right but it needs to work on different kind of inputs so you don't want your coding agents to to only work on, like, code, you know, database applications. You want your coding agents to be general.

So, like, if you wanted to optimize your coding agent harness, the JEPA idea still applies very well here because you wanted to – you could keep a Pareto frontier for a harness. And I think we've done some earlier experiments on, like, the Arc AGI harness, and it seems to work very well. So, but yeah, we haven't tried a lot of experiments on like coding engines, but I think that would be a very good direction as well. Yeah, I love the idea of maybe you have this Pareto frontier of harnesses that you then route to when you have a new input. Like I had this work that I called Retrieve DSPy, where I was looking to search for different DSPy programs for information retrieval programs. And like say you have like the Bright benchmark, which are like these long stack overflow posts. in that particular case you might want to harness that really decomposes the query into subqueries compared to something like beer or latte where you have these short queries and in that case you might want to just fetch like crazy to then re-rank

just that idea of different inputs have different harnesses yeah so I also spent some of the time at Databricks as an intern and a lot of things that I've seen there is kind of similar where if you're a lot of times people are afraid of overfitting and I think people will call this kind of like routing like you know route to a specific harness like overfitting because you're you know you sort of like look at the data first and then you decide what code to load or what harness to use but I think we should not we should not be afraid of overfitting because when you're doing things actually in production, if you overfitting to the whole production distribution then you not overfitting right So this is all the things that you have So yeah I think this is the routing like the prompt routing We also tried this in Jepa

It works pretty well as well. Like, you know, you don't just have one prompt. You have like multiple prompts. But, yeah, I think this is a really good direction. And also, like, I feel like it's a little bit under-hyped. I don't see a lot of people, you know, talk about this all the time. Yeah, that's like the classic deep learning. No need for generalization. Just make the training set cover the whole. Yeah, exactly. Exactly. That's the real bitter lesson. Yeah. I mean, this idea that your optimization process should produce more than one solution. I think that's quite a powerful idea, especially like for prompts is pretty intuitive. For harnesses, I think it's pretty intuitive. and then maybe for like database systems it's a little less intuitive because of just the operational overhead of doing that. Yeah, it's a little bit less intuitive I think still because of the evaluation is about an act. Like how would you define a good matrix?

Like how would you define a good matrix? Let's say you have 10 different tasks and then all these 10 different tasks contributes equally to the good performance of the database And then they might show different behavior on different kind of code. I think this actually happened. I remember Shu told me that they had some workloads where different inputs, you kind of need different codes for different inputs. It's also true for kernel optimization. If you're familiar with that, like a lot of times when you're doing kernel optimization, you actually generate specialized code for different size of the input. And then I can actually imagine you apply the Pareto, the JPA idea to that, where you keep the Pareto based on different size of the inputs.

And each one will calculate a different score, and then you can keep the Pareto of the candidate on that. We haven't tried that, but I think it's a very interesting direction to do as well. I'm sorry. I have, like, no experience with kernel optimizations. I hope you don't mind if I bring it back to databases. Yeah, sure, sure. I think, like, I don't know if I quite agree with the vision of just-in-time databases. I think that's a little intense. I think I agree more with sort of, I think, the vision that we're having where you might create, like, 20 different databases and we have more of these kind of separation of storage kind of architectures where you initially ingest some data, initially ingest some workloads in like a pretty generic database system. And then similar to these kind of like branching database ideas, you have this ability to go and get the data into this specialized system. But I don't know if it'll be just-in-time compiled because I think you... That's a very valid point. But if you actually think about it, like what does just-in-time mean here?

So it can actually mean a lot of different things. One way to think about it is that I will generate the whole engine based on the inputs. And I think that's very scary to everyone. But another way to think about it is that it can be, I don't know what's a good analogy to this. It's kind of like a little bit off policy. So you're not directly generating a good database for your immediate input. But if you're observing a trend in your input or if you're observing a certain pattern in your input, you can start another process of generating another candidate of the engine you're running. And then while in the foreground you're still using your old engine, and then when your auto-research agent decides that the generated one is mature enough and will be more performant based on the prediction of the curves, you can do the swap.

So it will not be like, yeah, I also agree with you. It won't be something like, I don't even have the code for my database. I just have a SPAC. And then once the inputs come in, I'll start to generate. That sounds very scary, but I think there are definitely like a slightly less just in time, but it still can generate stuff on the fly. Yeah, I could also see it has the particular workload, and it could just open it as a new branch, deploy that for that customer, and then open it as a PR that then gets resolved in the background. Yeah. It's definitely a spectrum, right? So I wouldn't say, like, it depends on the power of coding agents. I remember when it's like two, three years back, when it's like GPT 3.5 or 4, there was this very dumb project in Python where you can basically write a,

you don't have to write any code, you just write a comment, and then it will call like a language model, and then it will implement that function for you at runtime. And then at that time, it was kind of like people thought that as the meme because, you know, the coding agents are, like, pretty – or it's not even coding agents. Like, the language model are not that good. They can, like, generate, like, simple stuff. But actually, I think, you know, that thing might actually, you know, become something pretty real in the near future where you could have, you know, a specific section of your code. You don't even have any coding in that section. and then it will be generated at runtime. But I don't think anyone has, you know, done that, at least to my knowledge. I've never seen that, you know, in very serious places. But I feel like from what I read, like, you know, those people who are doing Out of Infra,

they're also thinking about those kind of approaches. Definitely in a more principled way, but like it's still like yeah you don't you don't even need all the code and then this kind of like auto research can do the a lot of the heavy lifting of the code yeah i'm about to be eating my words and earlier saying oh i've got five years of that they kind of like yeah uh compile at run time the sort of like i never i don't i don't know if i really ever got too deep into understanding the difference is there. I know that compiling can take a while. I don't really know too much about that topic. Yeah, yeah. One thing that I think a lot of people like to say is that I forgot who said it, but I think maybe Lina said it as well. Will you try to understand the output of your compiler? So no, right? Compile is also a tool that compiles your code into a low-level language,

and then people will just trust that tool. And then maybe when language model is good enough, when the auto-research system, when the whole system is good enough, people will start to trust that, and then people will not look at the result. And then people will just trust that, you know, a function header that asks to implement a specific optimization will work as intended. Yeah, I definitely think that the trust, a lot of it comes from all the tests it runs, and then it's got this prior where it's, yeah, I guess you just trust it. It's a great point, because it's also a big problem in our work in doing auto research and discovery, right? So one big problem I feel like in a lot of this auto-research discovery, like the gap between research and the actual production is when we're doing research, we always find problems where a matrix is very easy to define or where a matrix is very easy to run, where we can run, evaluate under 10 minutes, and then we get a good score for that particular candidate.

And then in production, what I was saying is that a lot of times, people really want to optimize something in their whole pipeline. But either it is very hard to run the evaluation of the change of the delta, or you don't even have that metric. Like a lot of times, the metric is like some senior engineer's approval, right? So that's a matrix that's almost impossible to even simulate. So I hope more and more and more people will work on this direction of research about basically how to define matrix. And we all know that current LOMs, current coding agents, are extremely good at hill climbing. So if we can turn a lot of the stuff into matrix, then we can basically optimize more things, and then we'll be more productive with the coding agents. Yeah, I think there's like, yeah, I agree.

I think that's such a powerful idea with GEPA and the natural language feedback for metrics. And then I guess there's like two ideas here. There's like one end, I think, mostly what we've been talking about where it's like running the evaluation just takes a day or costs $100,000 or something that really puts you off of doing it. And then there's the other side where the metric, where the bottleneck isn't necessarily like cost or time, but it's like human opinion. And I think there's works like Persimmon from Human's Hand. And so there's some cool works there that maybe will help with that human bottleneck. But you still, that problem of like testing this thing will cost a million dollars. I think that's pretty, yeah. Yeah, yeah, yeah. That happens a lot in real work, right? So a lot of times when people are doing RSI, people are, like, you know, training models or, I don't know, doing some other crazy stuff. So it's not possible, like, to spin up a coding agent and then ask it to, like, put everyone on, like, a lot of GPUs.

And then that will burn a lot of money. Yeah. Yeah, super cool. I guess I maybe want to even come back to this topic of forgetting and just maybe abstractly understanding, like this is a big topic with agent memory where you might want to forget some parts in the, of the, what the agent memory was. It also seems to be a big topic in like there, I think there was machine unlearning. I don't think it's quite the same topic as forgetting where there is kind of like, maybe you can better optimize neural networks by explicitly forgetting in their training. and maybe just this meta-topic around just, like, forgetting an intelligence and the role that it plays. Yes, yes, yes. This is great. I think my view of this is more of a context management problem. So we always think about forgetting as a context management trick. And the reason why that this forgetting particularly helps,

I think earlier I said I'll get more into detail, and then maybe now is the time, is that, like, one concrete example in our traces is where we ask DeepSeq v4.1 flash, which is a relatively good open source model, to optimize a particular, like, algorithm problem. And then in the middle of the run, I would say around, I don't know, 300K, 400K tokens total in the context window, DeepSeek starts to say something like, I've reached the practical limit of this problem. So DeepSeek basically said, yeah, I can no longer optimize this. This is like beyond my capability. And then when we're doing the experiments, we have this rough loop where, you know, if we're not exhausted on the budget, we'll just keep sending, please continue, please, you know, keep optimizing.

And then after we send this for a few iterations, the model starts to say, okay, yeah, like, if he's asking me to do, maybe I'll try to do it. And then it's thought about an idea, and this idea actually works. So the idea actually pushed the scores higher by a few points. And then after that idea worked, the model starts to get back to the old thought about, you know, I've reached the limit again. Like, there's no way that I can, you know, push this further. So this happened a few times, not only once. Like, it happened a few times where the model said, I've reached the limit, and then later, like, somehow it, you know, starts to work again. and then it said that it reached the limit again. So it's quite obvious that if you can actually find a better solution, you've not reached the limit. Like, you know, what you have said previously is wrong. But because of all these things in the context,

the model really starts to have this trend of giving up. So this is what I really want to say. When you're having these longer sessions of optimizing something, The model will basically lose a lot of motivation of keep trying it. And then if you think about it in this particular case, if you remove all these messages about I've reached the limit, the model will, of course, try to do something else. So this is one problem with this long context and where forgetting helps And another more quantitative way to put this is that we found that basically as you getting longer as your sessions is getting longer, the model is basically using less and less thinking tokens per term. And correspondingly, the lines edited by the model start to shrinking very, very fast.

So the model will try to edit 100 lines at first, and then maybe after 30 turns, 40 turns, it will only try to edit five, six lines. And another very shocking fact that we found is that a lot of the lines of those five, six lines are changes in numbers only. So the model will start to tweak those minor stuff, like hyperparameter, instead of trying bigger ideas. And this is, if you think about it, like for algorithm tasks, for like kernel tasks, the biggest gain will all obviously come from a different algorithm or a slight modification of your algorithm. It will not come from, you know, tweaking numbers. And then tweaking numbers will, you know, work by pushing your score by a few point percentage, but not by a lot. so the model starts to show this behavior well it's very aggressive at first and then it will slightly

be very conservative at the end and then the other thing that sort of like discovered the model is that because you have a lot of headroom at first no matter what you do will increase the score by a lot and as you are exhausting this kind of like headroom later in the optimization most of your trials are failures. And we also saw this in models, the chain of thought, you know, I've failed too many times. Doing this is, implementing this idea is risky. So this is like super bizarre to me. I don't understand what, what risky, like what's even risky about implementing an idea. You're not like, you're not doing anything, right? So this is crazy. And then after we remove all these contacts, after we forgot about all these contacts, the model starts to think again. Like, the model starts to, you know, be aggressive again about the alternation. And then it starts to shrink, you know,

from time to time, and then we have to do this, forget, multiple times during the alternation. And if you take a look at the paper, the first part, I think it's quite obvious that, you know, it starts to, the curve starts to flatten, and then after the forget, it starts to climb very fast again. And I think this is a very good empirical evidence about why this forgetting is a great trick in this. That was such an awesome explanation. I wrote down so many ideas while you're talking. I guess just the nugget about the thinking tokens per turn in the agent loop and observing that go down, that's such an interesting empirical observation about this. And I think generally we've already circled around it many times with this kind of like global versus local optima and how that plays at this. But really, I'm just super interested in this kind of context compilation. I think I've maybe been overlooking it personally, and I haven't paid enough attention to this topic.

But after listening to that, I'm just like, you need to learn more. Like, instead of forgetting, what if you just do one of these, like, summarize the context steps? yeah that I think so from my so another I guess another motivation we don't have a lot of like experiments to support this idea but a lot of the but one intuition that we had is that when you have more things when you have more available things in our context the model will follow the old trajectory so if you have if the model have already seen algorithm ABC in the context, the model will, you know, try to, like, go along these directions. So I actually wanted to recommend another concurrent work called Fork and Flush. So they proposed an idea to explain this called idea basing.

So basically what it said is that when you have the context for a very long time, a lot of the code that's generated by the coding agents, by the proposer, is similar in a sense that it will not jump to another idea, basically. So when you have all these code or ideas that's in the same basin, you're only getting a very small, you know, improvement. I think this is sort of, like, complementary to our analysis on, like, syncing tokens and lines of code change. So, yeah, to get back to your question, when you're doing compaction, you're still remembering a lot of stuff. And then this may not help your exploration. And another intuition that why we think that only keeping the best candidate will work here is that if you think about it,

there's two ways to think about it. One is that the model had access to all the other candidates and all the other contacts. And yet, the coding agents still get stuck on this plateau of performance. So maybe all those things are not that important, right? So this is one way to think about it. And then another way to think about it is that the good candidate, the best candidate, actually contains a lot of useful information in all the previous iterations. So oftentimes, as we said, this best candidate does not come from, like, the first shot generation of the coding agent. It actually comes from multiple iterations of, you know, keep building on each other. So the best candidate might come from an idea from iteration 11 plus an idea from iteration, you know, 23, and plus another iteration. So it actually contains a lot of user information about the previous row. So you're not losing as much information as you saw.

Like a lot of times people talk to me, people say, yeah, you're too aggressive of removing everything. So I would say it's actually not that case, right? So if you think about it, this best candidate actually contains a lot of information. Like in a way by being the best candidate, it sort of captures the history in a way? Yeah, that's a very good way to think about that as well, yes. I guess it also just really inspires me thinking about like how conditioned, like, you know, if it's just basic probability and it's like P of Y conditioned on A and B. Exactly, exactly. So the best candidate is actually conditioned on all the previous stuff, right? So you don't, it's not a single candidate. So, yeah. And this is really interesting thinking about these evolutionary trees, because a lot of, if we have these parallel sub-agents, there's still like evolutionary, like each loop of the auto research is still like, you could abstract it like an evolutionary tree, but like the idea that giving a language model,

like two, like a subset of two ideas to propose new ideas, if those ideas are like A and B compared to A and C, you might get different potential new ideas. Yeah, yeah, yeah. This is, yeah, this is, you know, this relates to the team setting, right? we were doing teams, it's actually, we saw a lot of cases where it's actually A plus C. Because the coding agent that produced A currently had a lower score than the coding agent that produced C. So the agent will take a look at C and then find the idea of C and then try to implement that. And then you have A plus C. So this, we probably need to do a little bit more analysis sound like where this helped but empirically like Teams it's really great. So the reason I think the reason why that you know forgetting works very well with Teams is that if you think about

it both ways are so basically Teams makes you hill climb faster and then forgets makes you to like not plateau. So if you have something that climbs faster and do not plateau then you have like a perfect auto research. hardest. Yeah. Yeah. No, it's super interesting. And I guess I'm, I'm just still a little stuck on thinking about this kind of idea of diversity in LLM outputs. And just, I was, I was always curious if vector embeddings could have a role in this where you kind of have the vector space and maybe you, in this particular case, you could condition on two points that are kind of like in an outlier cluster and like maybe puts the LLM into like continue thinking in this sort of abstract space with these other ideas and just how much that conditioning biases, what kind of ideas it produces. Yeah, this is definitely very interesting. I feel like one future direction that we're going to pursue is to think more about, so right now we are sort of like having a coding agent as the optimizer.

So everything will happen in this coding agents. the way that we're doing this framework is that we sort of like want to separate, we sort of like want to have like a meta control of the coding agent so that we know exactly where it produced a candidate and then exactly like where we can, you know, continue or like resume, stuff like that. But we have not separated the idea of, you know, forming an idea versus executing that idea, right? So if we do have that control from the framework level where the framework actually knows that this coding agent has formed idea ABC, then we can implement your idea about we can do more things about those ideas because we have an abstract representation of those ideas in the framework instead of everything being in the context of the coding agents. So that sounds like a very good direction to think about, yeah. Yeah, I can maybe tie that back to the optimize anything and earlier talking about

optimizing prompts versus code. I think that idea that forming an idea and executing it are totally different. That's really intuitive for code, but maybe less so with prompts where it might say, hey, I have a high level idea of how to improve this prompt, but I've been trying these ideas and it's not particularly getting there. But I think that still there's some phrasing of the prompt towards this direction. Does that make sense? Yeah. If you think about it, every action made by encoding or an LLM corresponds to one idea. The word idea is too abstract. Maybe one way to think about it is that the chain of thought before the immediate action is the idea. That's what the language model wants to do. That's a thought process. That's like the abstract idea. So I would say, yeah, even for prompt authorization, the language model will form an idea.

Like this idea could be something like based on certain data, based on certain examples that we've seen, I wanted to modify this prompt in a certain way. so yeah so I would say it exists for all the problems it's just like we haven't figured out a very good way to represent that in a you know in a good representation so that we can directly use use this term idea yeah yeah well just somebody else in my view is it's been so awesome discussing this work on skilling discovery requires forgetting It's a super awesome one. A question I'd love to anchor the WeVA podcast with, in addition to your current work, but more broadly just as someone working in the field of AI, what directions for AI excite you the most? Yeah, so the direction that excites me the most is definitely based on one fact that I truly believe,

which is the frontier models are incredibly smart right now. and people are not using them to the maximum capability of those models. So the direction that interests me the most is that how we build systems, harnesses around these models to make the best use of these models. And this is, yeah, so basically all my research are focused on this direction, and I think this is a great direction to think about. Yeah, the mismanaged genius. Right, right, right. I wanted to say that, yeah. Yeah, it's a super cool one. And yeah, I guess there's an interesting way of thinking about it, that the models are already as capable as they need to be, and it's just about engineering from here on. Yeah, if you think about it, even this forgetting work, right? So if the model is not capable enough, no matter how many times you make it forget, it will still not produce a better candidate. But the model is smart enough.

the thing is that your harness that remembers everything kind of like stops the model from making that particular progress that's why we have to like implement this forgetting idea, that's why like it has to be, we have to build better auto-user harness yeah awesome, yeah I think that's the mic drop at the bottom point in the episode on Shengen, thank you so much for joining us and so awesome sharing all these things in depth with you and just, thanks again yeah thank you thank you so much