私のAIエージェントが車を買ってくれた：Hermes Agentと自己改善の最前線

In SF and New York, they drive around these little three-wheel ticketing cars. They started to do some research and found out they're sold on government auction. In May, a listing came up that I bought. You have one? Yes, I drive it. You took the bit too far? Yeah. Really? I have one. I run loops on my own hardware of, like, I have a Quinn model running as Hermes that it designs an RL environment, kicks it off, downloads the weights, and then hot swaps into this. You're literally Neo learning company. Yeah. They call it trace to environment. And the idea is that you take a bunch of agent traces and synthesize RL environments based on those traces and then train on them. And this could work in a continual learning loop like every night, every task that you've had your agent do while you're sleeping, it goes and creates RL environments, trains itself on them, and updates. We've only trained models to be like this because we expect that they sort of make a wrong mistake and then correct themselves. I actually think they could reason less and still get to the right answer.

We're back. And this time I got an old friend with me, Sam, who's working at Noose Research. which you all probably know that company because of Hermes, the agent that has taken the world by storm. Sam's working on RL and it's been great to catch up with him. And I just want to like talk through what you're most excited about, some use cases you have for Hermes, RL in general, like what you were doing a year ago versus now, how it's changed. And then maybe we can touch on the hottest topic in the world, the continual learning and potential recursive uh what is it self-improvement i think it recursive self-improvement yes the rsi dude it is so hot right now so i laid it all out for you where do you want to begin uh thanks for having me here first of all super super stoked to be sitting down and chatting um i'll start off with kind of my personal uses of hermes because that's a good intro to those viewers that might not know you know what hermes is or the full power of what you

can do with it um and i'll just give a little bit of the context and the background about how hermes started and if you know what hermes is or if you're using hermes tell us in the chat because i think that is a good gauge of oh yeah you know like who's using it how they're using it yeah um so hermes was started internally as sort of a proactivity tool it was an idea to be like we have open claw and it kind of had had its moment sort of about a year ago and changed maybe you know nine months ago it was really really starting to to take off and some people internally at news were like wanted something different out of it and they wanted to design their own version of kind of a long-running agent harness that would just improve productivity productivity internally like there was no ever semblance of like oh we're gonna make this a product and people are gonna use it and so we did and they kind of started like telling us like hey you should you should start testing out hermes and i remember like telling some of my friends like oh yeah we're like testing out this new thing internally and i didn't talk about it much um i was still using cloud code at

the time for all my work uh and i really didn't like have anything i was jumping out at me to use hermes until one day uh i kind of either was bored or i think i was on i was having a conversation with a friend on twitter uh who was posting all these pictures of like ancient art pieces like these old ceramic pieces and like native american art and they were saying like how do i find clothes that match this art nice and that was the first idea clicked in my head of like you know what i could probably task hermes with doing this and just see how far it could i knew the pipeline in my head of like pull a bunch of images from men's clothing site you know do some sort of image embedding similarity to the target image of whatever art you're looking for and then rank them by like price or by you know whatever there's there's a million ways to filter it once you have this like similarity distance so tasked hermes with doing it and an hour later i came back to him with a program that you know it was like codas on github in an hour no and it could find any clothes currently listed that matched either clothing that you could send it or a picture of

something random like yeah yeah so i still use that a random item just like pottery yeah that And it's like, what vibes with this? I really am inspired by this pottery. I need that. All right. I'm going to go look up that. Yeah, yeah, yeah. Do it. There's a fun use, and I still use it when I want to find clothes. That's great. The other thing is in SF and New York, I guess other cities, they drive around these little three-wheel ticketing cars. They're these tiny triangular cars. I have seen one in my neighborhood, one or two, and I started to do some research and found out they're sold on government auction. So in February, the first live demo of Hermes Agent that we did on Discord, I got on and showed everyone the end-to-end process of me tasking Hermes to write a script to search government auction sites every day and buy one of these cars. In May, a listing came up that I bought. You have one? Yes, I drive it. You've hit the bit too far. I have one, and I drive it around San Francisco all the time.

people probably are like no don't wait i'm coming back i paid for my parking well it's spray painted it's interesting it's all it's painted so like it's got all this like graffiti on it and stuff because the guy who had had it previously was an artist so it's like people think oh i'm a ticket person and then they realize it's like not um i'd love to repaint it it's kind of blue and and black right now like it almost is like harmy's colors yeah um it'd be cool to have some some yellow those freaks in there or like i i'm jokingly calling it the hermese mobile yeah to co-workers i mean it got you the car it did it did and that's why i tell people often like if you're looking for an apartment that fits the specifications or you're looking for a car and like instead of checking craigslist every day you know that's a great use of hermese it's like life admin tasks that like would require you to kind of be on your computer a lot or like be and follow up yeah and you know outside of that we have seen wild uses of hermes tell me about that yeah that's what i want there's a lot of people use it for creative um creative pipelines

like our launch videos on twitter super hype they're beautiful the music behind them people love all done through hermes through this hermes workflow that some of our designers have laid out that you basically end to end get this beautiful ASCII art launch video where it composes the music itself. And the whole thing is just this cohesive creative pipeline. And so our designers use it a lot for that. We've seen a lot of uses of people using Hermes for smart home automation. So like having Hermes do analytics or control on like your lights or like one person, they had like a dog door that was attached to their smart home And, like, they could pull the numbers of how many times their dog had been in and out, you know, within a week and, like, start tracking the behavior that way. And it has a beautiful data visualization. Yeah, yeah, yeah. It makes this pretty, yeah. I mean, anything's possible, man. And, like, of course, a lot of people use Hermes for things like, you know, automated crypto trading workflows and market analytics.

You know, I think like we created an internal and sort of external benchmark to evaluate model performance inside of Hermes. And there's already a ton of coding agent benchmarks. Like everything out there is targets like the majority of what people use that for, which kind of comes down to writing and executing code. But we wanted to target the things that were outside of that. So a lot of it was life admin stuff. A lot of it was, you know, managing your utility bills, having Hermes go through your email, compile all your utility bills and like even draft a check to send out like you're the last step in the approval workflow for that. And that's what that's probably what I love is how far it can get without me. Oh, yeah. And because you don't necessarily need to. And also, you don't need to sit there and explain it step by step. It's like, hey, here's this outcome that I'm looking for. You go figure it out. Once it's figured it out, then you can optimize on it and you can decide like, okay, now that

I have this workflow, let's codify it and let's make it a little bit more deterministic where we can. You don need the biggest model to go and do this every time that we have it because we already explored the space We found what the optimal way of doing it is and now we can optimize and that the cool part about Hermes too I mean you teeing me up right there it's like when we designed it we started to add in the ability for it to write its own skills and normally with the agentic workflow stuff as you were starting to do more you would tell your agent like oh right this is a skill Hermes just like if you have it do something especially multiple times it's all converging to the same skill i think that was a big unlock man that was the thing that made like that magical experience where you're like oh wow it just does it and it it does it like automatically i don't need to think about it i don't need to tell it to do it and it knows how to do it yeah it's it's pretty wild um so yeah we've had a lot of fun and growth with hermies and i think this whole landscape is a lot of learning about you know what the right

paradigms are to to use and how to we talk a lot in news about you know time to magic we want to understand like from someone's first uses of hermes how long does it take them to really click and understand that this you know this agent grows with you and has a task yeah like me as that user how long till i can offload a task and like trust that wow that can get done and i don't have to think about it should we talk rl yeah yeah this is perfect segue into the rest of what i wanted to talk about so you asked what i was doing a year ago and how the landscape has changed um a year ago i was working on we had our own internal rl framework i mean it was public it was open source but like we used it for all of our rl training internally and thinking machines had just released their tinker library which was basically cloud rl you could offload your training loops to their compute and you could pay them per use but for a while in their dev you know beta phase it was free

so we basically were getting like this super great deal on like running rl experiments um take advantage of it while you can't that only after a while started to cost us low money but still worth it um and i i had was an intern at news for about a year and then i joined full-time in October of last year. So I just passed your anniversary there. Congrats. Thanks. Um, so my first project was to integrate our RL training library into their online RL, you know, their cloud RL thing. So I built out this whole library. Um, it was a lot of the RL stuff that we do now is very different than what it was back then because the whole idea of like a genic RL, like was just starting to form like a lot of times we were like the stuff i was proof proofing out for this was like rling on like certain benchmarks like math benchmarks and like different types of problems like it wasn't until about march when i was using this framework to build out um

it was a project called endless terminals and it was basically procedurally generated terminal use tasks that you could just keep generating them with varying levels of difficulty and then training the model on it, and it would just get better and better at, like, this complex terminal management. You're talking about Hill Club. I know, which is, like, ground zero for agent performance. Like, you optimize that, your agent gets so much better out of the box. That's awesome. So that was a really cool one. And what I'll say about this is that the main contribution I made to Hermes, which we ended up moving off of our RL framework, but in the early days this was the biggest pieces of code that I ported into Hermes Agent. we built that idea of using our rl framework with this on you know this cloud compute into the earliest version of hermes and the idea being that if you're running hermes with a local model and you have an ability within hermes to design an rl environment kick it off on some cloud compute and just let it and then download those weights and replace them with whatever's running

in hermes it's like hot swap yes it never oh it can teach itself a skill like if you task hermes with like i want you to yes create an oral environment to learn how to use the terminal better yeah right i'm having a horrible time with this erp system yeah like here's a bunch of data of me using it go make it easier for you to use it yeah and so that was the earliest like embers of what we would see rsi becoming and there's there's a lot of other ways that this has all kind of come into the fold um but that was a big contribution i made and i felt you know i i've run loops on my own hardware of like i have a quen model running as hermes that it designs an rl environment kicks it off downloads the weights and then hot swaps into this literally neo learning comes yeah i know kung fu in the moment and you're there like oh sweet my model is now much better

yeah and it can do that but then it changed yeah so we ended up like our training framework was good but a lot of priorities in the company have shifted since we did all that namely hermy's agent took off way more than any of us expected so we shifted a lot of resources towards that um model training paradigms got a lot harder since then like the the deep seek rl explosion was sometime you know yeah like it was recently enough that people are just now figuring out how to really harness like not only rl on specific tasks or specific task types but this like long context long horizon rl and like really hammering it for agentic loops and like agentic you know models and And I think we've noticed this a lot in, like, the latest, you know, anthropic series of models. I think all of the open-weight Chinese models that have come out have since, you know, really benefited from focusing on this, like, longer context stuff. The first one that I'm forever will show this model because I personally love it.

So Xiaomi, the telecommunications company. And CarMaker. Yes, and CarMaker, right? Yeah, the massive. Incredible CarMaker. yeah um i'm really thinking about getting one because i live in germany and oh yeah you can ship them over yeah yeah you should i know see them a lot they seem really good anyway i digress what do they do they were some of the earliest to jump on um this agentic rl stuff and they had also picked up focusing on training specifically for hermese agent pretty early on and they're doing it in the open like writing papers about it or yeah yeah really and so i haven't seen this at all xiaomi is like i was a fan of their model it performs really well in hermes for its like weight class or whatnot um and so you know we were really inspired with what they did to sort of we've been we've been amassing all of these different learnings and basically saying like okay if we can take you know all this agentic open training and stuff and and we're just kind

of reshaping the post-training team to be able to do this kind of thing well like that's i think our goal as the rl post-training team that news this we got our start training models yeah and we have a really strong talent density in terms of the you know there's eight of us and we can train a really good model given the right time and resources that is so wild so we've been sort of biding our time and making sure we can do it right yeah um and so definitely more to come on that soon but how does it go to the continual learning like explain that now where you're at with all of it i'm glad you asked all right i'm glad you asked so xiaomi the latest model is mimo v2.6 pro in that paper they talk about this brilliant thing that has come out lately and it's sort of obvious when you think more about it but still the idea struck me um they call it trace to environment and the idea is that you take a bunch of agent traces and synthesize rl environments based on those traces and then train on them and this could work in a continual learning loop like

every night every task that you've had your agent do while you're sleeping it goes and creates rl environments trains itself on them and updates um or you could do you know amassing tens of thousands of different traces on data sources that you have like why not just spin up Yeah, if you have a lot of data of Hermes agent uses, what we do first is what's called failure analysis. So we go through and find the exact points where the agent makes a mistake. Then we cluster those by the mistake type. And once you have a bunch of different piles of, oh, the agent, you know, entered the wrong directory here, or oh the agent you know copied the wrong file here like you just have a bunch of different types Then you can sort of build an environment that captures those mistakes or build a bunch of are you human labeling those or is that all all automated pipeline also awesome yes and again that helps with the recursiveness yeah huge okay that was one of the last things i've built that news was like our internal it's we call it the insights and observations pipeline it's actually based on

some research, Anthropic Publish, where they do the same thing. And they didn't publish the code for it, obviously, but they did publish their methodology. So we sort of co-opted that between Claude itself and their Claude code variant of doing it and built our own internal pipeline that does this insights and observations, like failure analysis, clustering, and then we have someone working on lately, which I'm going to start collaborating on too, environment synthesis, and then we can just loop that. What's the environment synthesis? That's the generation of, like, you take a bunch of failed traces, and you have a bunch of examples of, like, here's where the model messed up, and then you build some kind of, like, fake environment that's supposed to match pretty closely to, like, those, like, the world that those environments were in, like, the state of that computer at the time and the files, but it's not the real thing, obviously. It's just synthetic, and then you can add sort of levels of difficulty based on how well models do.

How often are you trying to train on like gigantic cloud? So not train on the cloud, but like train for cloud environments. Because if I've got a bunch of different AWS services I'm using, and maybe I've got some Google Cloud services I'm using, and I want to recreate a little bit of that, can you bring that into the environment? Yeah, we've done a fair bit of like Google Cloud, you know simulated like cloud environment training too and and i think that that's a lot of this is very green like there's so many areas of exploring ways in which you can put an agent in the simulated scenario that we like probably haven't touched yet and i think exploring like cloud deployments and managing cloud infrastructure is one of them because a lot of times the traces that we see or the traces that we get from ourselves are all hermes running on our own local hardware local machines. That's what I was thinking. But if you're, oh, maybe you want to start serving a bunch of traffic and you want those traces too. And you want to see like, oh, now the agent

is trying to fix something on production traffic or simulated production traffic, right? But it's messing up and we've got a step one incident. And then we can see that, cool, it's messing up here. Let's spin up that environment, kick off another training run and have that whole loop happen again. Yeah. So I'm very excited about that. That is one of the things I'm the most excited about, and I think it's a very healthy place that the RSI conversation has turned to, is less of just having the agent decide to teach itself some new skill. because we kind of do this already with like the tiers of memory in Hermes and this like automated skill writing like the agent can kind of already bolt on things that it's learned to itself in context but learning from its own failures you can only do so much without sort of this this idea of like like just collecting all the failures and building some learning environment for it and you know that might honestly be what's left in post training like everything else has kind of

been really yeah i mean there's a lot there's a lot to do but i think that that's like the last like yeah i i mean only thing we can do right now is improve models for agentic use because that is by far in a way like what we care about now it's doing how long they can act in a harness how long they can go without making mistakes yeah improving token efficiency is one like that's one i explored early on in my research tell me about that you want a model to be able to finish a task as efficiently as possible and then there's this like exploration versus exploitation balance of like you want to produce yeah you want it to spend some tokens exploring other options but if it knows that it's going down a wrong path you don't want to waste a lot of turns or tokens doing that yeah you know what messes you up there though is like the models will a lot of times lean on their pre-training yeah and so you say like you want something done and it goes oh yeah i know how versell works meanwhile versell came out with a new version and guess what you don't know how

versell works so it doesn't go and like read documentation or try and figure out how it works because it says i know how it works and then later you're like ah it doesn't work like that does it so you get these i don't know how to explain it but there's those moments where you the model is magical because it knows things but it also is almost like it's achilles heel yeah 100 that and And that used to happen a lot worse with knowledge cutoff dates. And now less so, but I think a lot of labs are trying to train models to act with the, oh, I should look this up. Instead of, you know, oh, let me pull from my memory. But then it's not as token efficient, right? Exactly, because then it has to read through the documentation. Yeah, that's what I was thinking. Like, if you're doing research on token efficiency, like, you want that, but you don't. it's a huge trade-off. And for me, token efficiency comes more in, like, long reasoning traces. Like, I think there's a lot of cases in which models sort of over-reason

because they've been sort of trained into this basin where, like, this longer reasoning answer involves a lot of backtracking and second-guessing. I think a lot of that is, I mean, this is, like, my personal hot take because I think that, like, a lot of that is somewhat unnecessary in that, like, we've only trained models to be like this because we expect that they sort of make a wrong mistake and then correct themselves and like most of these models are good enough like i actually think they they could reason less and still get to the right you know right answer yeah we also just kind of assume that the longer it reasons the more correct the answer will be or the more accuracy we're going to have yeah at the end of the day and that's true that potentially like a lot of that reasoning is wasted reasoning yeah It's just tokens, like, and, you know, on the contrary, like, I did do research on this. I published a research blog about this. It did turn out that the longer you have a model reason, the better it does in a generally constrained setting.

Intuition is right, but. But I also think that, like, a lot of people liken this to a Turing tape analogy. Like, if you're computing a program on a Turing tape, like, the longer the Turing tape, the more, you know, you're going to approximate the program you're trying to compute on it. like this classical old computer science analogy and that's roughly the case here um i wonder if there's a point of diminishing returns like i was looking at you know up to maybe 2 000 reasoning tokens but like when you're reasoning for 10 000 tokens or more like there might be a diminishing returns point and yeah i think it's worth looking into more um but that's also one that comes into training like a lot of training regimens these days involve like reasoning length or you know token efficiency and reasoning like sometimes you want the model to reason less it's usually there's less room for mistakes the newest quen series has three reasoning settings and default is extra high but you could do you know medium and low reasoning if there's some cases where you might want that dude i'll tell you what i never know how to for whatever task i'm giving it's always high and then

And every once in a while, it's like, ah, I think this could be medium or low. We'll see. Well, first of all, like, I'm pretty sure the OpenAI models, they don't even have low. No. Starts at BIA, yeah, yeah. They don't. So, like, and then it just goes medium, high, extra high, ultra high. Ultra high. Like, I don't understand when I should be using these. I have no idea. It's so complicated. And please, can it just figure it out for me? Yeah. Like that is the fact that we have to decide what we want is a little bit for me. It's probably like an artifact of today. Yeah. But I get the feeling it's not going to. No, I also I mean, I never think about it and I don't change the options. I'm like, I'm always using the highest reasoning because I don't think about it. That's it. But then you end up wasting a lot. Yeah, it's I think it's just like a hypothesis that I'll put out there is that that's likely the case is that there's maybe more of a dynamic way. you could control reasoning length and then you would also ask like things I excited about and right now the areas that I most excited about with all of this actually or even outside of the RL post regime inference time improvements

You know, I think between there's a lot of work that comes out in steering and, like, using these external vectors and activation math to, like, guide a model in some behavioral direction. Yeah. and then also sampling like the order you pick the tokens my earliest research for the token efficiency thing was in sampling and latest research that i've published um was in steering and it was specifically focused on like refusal steering like safety related work but i am more interested in steering towards behaviors that could make an agentic model better especially smaller models yes so like an example is like if you're solving a math problem if you had some way to steer the model towards being more pragmatic. Like if you could multiply some of the activations at inference time and be like, hey, you're solving a math problem. I want you to go hard. Yeah, be more pragmatic. Or if it's doing creative writing, I want you to do more. At inference time. Yeah. It's almost like you have latent lauras or something.

Yeah, in essence, it's kind of the same thing. It's like a trained adapter that you can basically choose to add in at inference time. Oh, man. And LORAs are an extension or sort of a parallel, you know, adjacent thing to that, that you can train specific LORAs and hot swap them. And Apple was one of the first people to publish research. Yeah, I remember. That's why it was like, oh, yeah, I think somebody was doing that. Yeah. And you can have that come in, especially when it's on device. It's very useful. Yeah, super. But then you're talking about it a little bit different and you're saying not necessarily these LORAs that are hot swapping in. it's just inside of the actual transformer? It's like at the end, like the mechanism that would be multiplying the vectors in is external to the transformer, but there's no really swapping. It's sort of just multiplying a vector into the last layer of the model or whatever as it's sampling to change the behavior. And you would create that when you're RLing the model?

We would create that as sort of like an external thing that builds into the inference engine. So that's like my vision is that if we are able to serve inference. So into like VLM? Yes. Yeah. It's like a VLM, you know, sidebar that basically does this like dynamic. Whoa. And that I think is like a really underexplored area of improving small models. Because right now we're kind of trying to RL them to death to get performance gains. Or just steal the shit out of it. Yes. Make them super small so you can have them on edge. And then it's a different story when you're saying you can still get the value of the small model or of a larger model, but with that smaller model, maybe you need it on device. Yeah. And I think, I don't know, I just think, again, there's some areas of this that are really underexplored. And so this is what I'm personally very excited about. I think most people don't want to touch it. It ties a lot into, like, the interpretability field and understanding. Oh, really? Yeah, because if you can control a model's behavior, likely you're getting closer to understanding the components that you're touching, where those behavioral things come from.

It works a little bit in the other way where if you do understand a behavior, then you can easily control it. And sometimes when we do the steering, we control something and think that we can explain backwards why it actually exists in the model. Normally not the case. um but that's why they're like the this venn diagram overlap of like understanding the model interpretability steering like controlling behavior and like the middle is is when you hit both on the head and you understand both the components that exist and how to you know maneuver them i'm normally focused on the latter like i only care kind of about steering behavior at least in this agenic context like yeah i'd love to understand how a model works but for us it would be really beneficial to be able to just kind of guide it in these ways um yeah okay because you can get such a better outcome and it's just two sides of the coin it's like one you're training this behavior into it and sort of designing these rl environments and these post-training environments to like converge into the behavioral basin that you want and the other you're sort of

sprinkling it in at inference time and choosing to like like you know and i i like that analogy Yeah, I have a hunch that there's these ways that you can sort of do it a little bit more deliberately and a little bit more fine-grained at inference time. In my head, I almost envision that animation that you see when you look inside of how a key clicks all of the different pieces of a lock before you can open it. it's like you need certain things to be lit up and so then you get that key and it sticks in and then boom we now have this model it's the same model but because we put the key in it this way it hits all of these different pieces and then you can open it and it gives you that yeah yeah and and every lock is a different you know you insert the key for one behavior and all of a sudden now you yeah that's a great analogy i think or or like swapping you know how like gears kind sometimes yeah so you have like oh the gears fold together like this that kind of thing i'm

seeing it very visually because it is so it is pretty uh abstract in a way so i'm like only visualizing it and it's it's a little bit harder to figure out the um the things on it but dude okay so the last thing i wanted to mention on this was well i mean there's so much there's so much and i just got word that we got one minute left um so i'm going to try and like close it up But you kind of blew my mind on a few of these pieces so that, damn, as I went into the visual analogy, I started just thinking about all different kinds. But what I wanted to say was my friend Linus, one of my favorite talks that I've ever seen was from Linus Lee three years ago when LMS first came out. And he was doing a bunch of interpretability stuff. and he was talking about how we can visualize the inner workings of how the model is coming to its conclusion and hopefully we can get different ways of interacting with it so in this talk he

talked about how when we interact with computers sometimes we will use histograms or sometimes when you're like with music you can have a high pass or a low pass and you see the ways that you're interacting with the um the generation or the computer generation in a different way than just like oh we're looking at output like just a bunch of text logs and whatever and it feels to me like some of this stuff that you're talking about it would be so cool to see a histogram of the inner workings of how it's going and can we steer it differently not necessarily steering it in how we want to like type and say it needs to go this way or that way but can we like have this rsi loop training and we are deciding how we want to steer things because of a histogram i know it's way out there and i may have no i mean you're touching on like all of this does connect if you can understand

components of it and the visualizing as a whole like everyone's trying to figure out the best way to visualize this kind of stuff but if you can understand different components of a model and sort of you know shuttle rsi learning updates into them like eventually we will get to what we envision is like true continual learning like a model that can just update itself constantly like we humans learn like that's the goal is we want to be able to understand like how to update one track in the model's mind and not do a full wave update like every little neuron gets changed and we don't know why like we want to know why and like if i encode some factual information about agent con in san jose 2026 like i want the one pathway in the model's you know neurons to encode that information um so i think you're that talk sounds interesting you should i'll send it to you all right so we're gonna shoot over to the next talk speaking of which and let us know in the chat i'm gonna go look at it now what you think if you're using hermes uh give a shout out to sam thank you for

doing this dude thanks this was awesome man and i'm like i'm really glad we got to nerd out on it Yeah, me too. This is great. We'll see you all after this next talk.