正解を知っているAI科学者向け評価環境 - Ian Quigley（Leash Bio）

Hey there, welcome to Lane Space Science. This is Brandon and RJ, joined with Ian Quigley from Leash Bio. We recorded an episode earlier this summer with Ian, where he talked about the cool things Leash was doing with DNA encoded libraries, and how they could screen millions of compounds against thousands of proteins in a high-pickup way. Around this time, Ian realized that they could do fun things with this data. It essentially allows them to create like a virtual sandbox where agents can explore and make decisions in real time as if they were medicinal chemists and get instant feedback, but with like a huge amount of exploratory rollouts. Since that episode, Ian has unleashed, have developed this idea much more fully. And since then, they have been there now releasing a paper and a data set and a blog post along with the episode that we just released. We wanted to sit down with Ian, talk a bit more about what he's done, and hear about the updates.

Anyway, welcome to the podcast. Thank you, Ian. Yeah, so last time we talked to Ian, you were saying that you had some agents exploring a sandbox, and you were like, oh, this is just a fun idea. We don't know what we can do with it, but it's an interesting idea. I think you've since fleshed this out into a full environment for agents to explore, maybe do RL on, you know, scientific reasoning. Can you talk a bit more about that? Yes. So the thing that I talked about last time was this idea that we had these large collections of screens and that they were big enough environments and counterfactually dense enough that you could take agents and drop them in there and then give them tasks and they could go do those tasks on a limited budget in this landscape that we had measured completely. And so the flagship problem that we spent a lot of time over the past couple of months on is a structure activity relationship problem. And the idea there is like, okay, I've got some molecules and I know that they bind, maybe they bind to this protein that I want them to.

How do I change them to make them better? This is a very common task for medicinal chemists to do. And SAR, as it's sort of abbreviated to, also includes other stuff like certain drug-like properties. You know, does it get metabolized, all these things. We're not doing those right now. We're just going, can we find molecules that bind better to the thing that we're interested in? And so what we do is we take an agent, like a language model, and then we say, here's a catalog that you could look at. You have a budget of 100 molecules. We're not going to tell you anything except the target, and go find the best binder you can. And so different agents try different stuff. They have different search policies. They have different ways of wanting to spend their budgets. and what they'll do is they'll put out in readable language a SAR hypothesis about why they think these molecules are working or not working and then they'll go in and with this limited experimental budget, they'll test those hypotheses. And because it's all pre-measured, you can roll out a lot of these. We're up to about 90,000. So you started doing these on foundation models.

I think you have like a paper or a little preprint or something at this point that you can share. Do you want to share that or talk about it? Yeah, so the preprint will probably be out at the same time that this episode comes out. It is pretty ready. And what we show in there is that we create separation of agents across model families and active levels, that kind of stuff. But another thing that's really interesting is because we have not just the results that you could use as an eval for how the agents do in the job. We also know the results of the entire measured landscape or world. And so then you can start to interrogate how good is the agent at making scientific decisions? because maybe it only got to like the thing that's half as good as the best thing in the entire environment. And so there's all kinds of interesting behaviors like agents that give up early, buy molecules that don't agree with the rest of their hypotheses, ones that keep retesting the same molecules over and over despite having a limited budget And also when it good to explore versus exploit And it turns out that more performant models are much better at abandoning

kind of unproductive neighborhoods. And that was actually a really interesting surprise for us. Yeah. And so when SIPA comes out, we're going to release a bunch of these environments as hardware packages and then many thousands of transcripts for people to go play with and go see how their size is. That's super cool. So do you have a feel for why the smarter models are able to figure out that they're in a bad landscape and move? Or a bad part of the landscape and move? We don't know that at all yet. All we know is that they're abandoning these crappy neighborhoods more quickly. And we definitely see a progression of, like, a more willingness to jump around overall for more performant models. But they especially abandon the lousy places fast. And then the really poor ones stick around. It's not just that they jump more. it's that actually they're figuring something out and saying, hey, this is not good. Let's get out of here. Exactly. Exactly. Yeah. And what happens with the dumber models, they'll just keep running little subtle changes on

molecules that aren't ever going to go anywhere. Right? Like, oh, these two, they have a low score, but I feel good about them. And so I'm going to keep trying stuff. And then the smarter models will abandon the entire process. We're going to go somewhere else. Have you looked at the rollouts to see if it's thinking about whatever? Yeah. think about all kinds of stuff you know when we release all these transcripts like everyone should look at one of my favorites is just take large numbers of transcripts and just dump it to LLMs and just have a look at the different failure modes and stuff like that but you'll see yeah I think that this ring is good and I think that this particular nitrogen is painful or you know whatever and they'll say these things and then they're very plausible you know medicinal chemistry narratives so it's pretty fun yeah that's amazing So, I'm curious, have you given humans these same questions? I have, to have them look and evaluate, like, not a ton, but because, you know, we have a large number of them, but when we get medicinal chemists to look at it, they're like, yeah, no, those are very plausible thoughts about, you know, how to conduct this particular part of the SBR campaign.

Maybe I'm more thinking, could you make a game out of this? This is sort of a game, right? And then medicinal chemistry, the video game, and then you can put people in these environments and give them... A medicinal chemistry capture. Yeah, you didn't have a business model. I like this plan. We did do a game a little bit like this for Name That Chemist. We would have to probably boil it down to have a smaller number of building blocks. Because right now, to have a chemist go in there, you can imagine that if you're a medicinal chemist, you might have this suite of things that you might want to try over time, like put a metaphor here, like move this around. And when we have these pre-method environments, And we are going to be restricted to the things that we did actually measure. And so then we have to be asking the medicinal chemists, oh, well, you can't do whatever you want. You can just do a lot of stuff. And they have to know what stuff. They have to know what the stuff is, yeah. So we'd have to figure out a way to surface it to them in a meaningful way. But I kind of like where you're going with this. Yeah. It's like we could definitely shrink it down to a smaller number of options.

So. What model is best? Oh, the French-year models. What are the best ones? Let's see. I was looking at the plot a minute ago. It's kind of embarrassing that I forgot already. Right now, the champion is after six. Nice, nice. Not a big shock. Not a big shock. Sonic 5.5 is actually pretty good, too. Okay. Yeah. And Sol 6.1 is also pretty good. And everything 5 is just a racing pilot cap, right? Yeah, Sonic 5 is both very expensive and not very performance. Yeah, yeah. But yeah, I think 5 will be long considered a big mistake by Entropic. 5.5s are pretty good. Yeah. Do you have Opus 5.5? We don't yet. No, that's one thing that has actually been a little infuriating, as you guys can appreciate. So when we talked last two months ago we started these rollouts to see what happened And then there been a new Frontier Model drop what like once every 11 days 11 days yeah Yeah so like every time oh we going to ship this we need to drop this public stuff and like some other new stupid model shows up

I'm like, ah. Yeah, they ship faster than Anthropic. Yeah, yeah. Right, good luck, Ian. So, yes, so we are excited to put in more in here. And we also have run a large number of the open source ones as well. So we've thrown a bunch of smaller quins in there and GLMs and Kitty and Deep Seeks and that kind of stuff. So there's a lot of these. How do they do? Deep Seek V4 Flash is pretty good. Not that bad. Yeah, maybe I should just send you guys the full chart of how things are going. Yeah, we can just pull this up on the screen. Yeah, we can put it in. Pick it up there. I can say it kind of looks a little bit like, you know, like the Arc AGI 2 plots where you've got some crappy models. Right, right. It does kind of look like that, and that's kind of why we made it like that. And then I think one thing I mentioned earlier that has been very interesting to us is about this time last summer, we had started an internal project to do supervised fine-tuning on some of these language models to get them to understand smiles

because none of the existing frontier models were any good at it. So, like, none of the anthropic or open AI models at the time could even convert smiles to non-canonical smiles very effectively. So, just for non-chemists, smiles is a language that you can use, a string language that you can use to describe accurately a chemical, right? Yes, yes, that's right. I think we mentioned it a little earlier when we recorded last, but it's a string, and molecules can be forked or circles or both put together, right, these different things. And so a SMILES is basically a graph traversal across the molecule written down into a one-dimensional string. And so you can imagine starting the graph traversal at a different point in the molecule. And so there's multiples, SMILES, ways of representing the same molecule. And so it's nice to be fluent in transferring from one to the other. It's pretty easy. there's a couple open source packages people use like RDKit to do this. But we wanted the language models to do it just because it would go faster

and they weren't as good at using tools on their own back then. And nobody could do it. And then around February, like a lot of other things, it became clear that they could start to actually work with Tempistry in a meaningful way, which is a reason we started going in this direction. That's really interesting because that means that it's internalizing some understanding of the physical nature of molecules. I think a little bit. Yep. Yeah. Like, these are the tasks that we're having these things do right now. Like, they are iterative tasks where they're looking at molecules and they are learning something about those molecules to make choices to make better molecules next. Yeah. And we remove the information about the molecules. Like, these decisions really suffer. So they're learning something. It was very interesting. Or at least they're using it. I guess this is better. We're not changing weights here, but they're definitely using the information as to make future choices. You mentioned all these models, right, that you've tried. So I wanted to ask, have you just used one of these

Bayesian hyperparameter algorithms as a deterministic baseline for your problem? We think it's too spiky of a landscape and it's very high dimensional. We don't do that, but we mostly think it's not going to work. I think it's a party line. You plan on releasing all this? So I guess we'll put the... Wait, what are you releasing? You said you're going to have a pre-print blog and then some data rollouts and everything so we can put all this in the show notes. Is there anything else? Yeah, that's it. Yeah, I think that's the big thing. And that's probably something closer to like... I guess what I would probably say is we have built this thing where if you've got scientific agents and you want to know how they behaving what mistakes they might make how they might learn from physical feedback You know if a new training recipe improves scientific decision we got the capabilities to measure these things at scale. Yeah, that's really cool.

Do you have plans for other similar problems for this same kind of thing? We do, yes. We have, broadly, the way that we think about this is it is a lab in the loop with amortized latency. So it's like, go out, you record the thing once, and then you can have a huge number of roll-outs through that thing, effectively infinite roll-outs. And we are thinking hard about other types of biological or scientific questions that you could transmute into these kinds of environments. Because if you're doing conventional lab in the loop, it does take a long time. And because those things themselves have limited budgets, it's also not counterfactually dense. so like you got to keep running it and do different choices with every rollout to make the things to make the hs learn right but with your your multiplex strategy then that then you can just amortize you can just amortize so it's it's it's not quite accurate to say it's a lab in the loop

of zero latency but it's pretty close like this is the same thing that lilo's doing i mean it's not the same thing but they also do experiments and then they use that in their rollout, but they're not doing them live. Right, right. Doing it in the rollout one step at a time offers a lot more freedom because you can choose almost anything, and here you're restricted to the number of choices that we made when we pre-measured the environment. That's definitely true. But it's still a pretty big number of choices. And then we have right now, we've screened about 1,000 proteins against a couple of these chemical libraries since we started Leash. And we're going to start to grow that. I think we're going to try to get to maybe 20 of these new environments a day. That's a plan for us to try to get to the beginning of maybe a 26 or 27. We're doing two a day right now. Yeah, I think we could probably get to 20 a day without too much lifting. So we're pushing. We'll see if we can do it. Wow. And so you're going to have this huge RL environment.

Yeah. It seemed like the thing to do. I mean, kind of informally, it seemed like where a lot of these agents are making the best and the fastest progress is where there's instant feedback that's very scalable. Yeah. It's really hard to do that in the physical world, but we think that there's probably a handful of biological questions that you could rebuild in this way and go after them. any than any. Yeah. And then we don't yet know if we're able to take reinforcement learning progress from these kinds of environments to move on to new biology yet. But, you know, like we're trying, we're going to find out. That's the obvious goal. Like how does this transfer to other biology? You can imagine if you did it for some sort of covering set, right, that you might be able to get things in between though, that you could transfer to pretty well. Yep. Yep. And then also one thing that I need to do is talk to some more decision people, because a lot of this is about how do you spend a limited experimental budget,

and then what can you do to do better at that? Yeah. Yeah, beyond just medicinal chemistry, right? Just like, yeah, when do I stay? When do I go? How do I get better at that? Interesting. Yeah, you can imagine it if it knows how to connect a decision at that point to a knowledge that it has, then it could learn that skill for other problems. Yes, yes. So we are incorporating more and more structural tool calls and things like that with this as well. So it's a pretty fun time right now. Yeah, we'll be sure to link to all that cool stuff you have coming out. And, yeah, thanks for taking the time to chat. Cool, cool. Yeah. Or thanks for taking the time for a follow-up chat. Yeah, yeah. I think this will actually appear back-to-back with the rest of the episode. Okay, this was great. Well, thanks, guys. Always fun. Yeah, bye. See you later. .