エージェント活用の科学とエンジニアリング：Show Us Your Agent Skills 第2回

What I love about working with agents, are you all familiar with Richard Hamming's writings on science and engineering? So this is like a classic and really essentially the TLDR is he says, you know, in science, if you know what you're doing, stop. And in engineering, if you don't know what you're doing, stop. And that always stuck in my head because certainly working in data science, working as a software engineer, we use the same tools, but we use them in very different ways. there's a different amount of experimentation. And this is a long way of saying that I feel like this moment where using agents and software development, using it to build all sorts of things, is bringing the science back into the engineering and the engineering back into the science, like what Eric was showing us just earlier, in a way that is really cool. That was Hilary Mason, the CEO of Hidden Door, on how agents are changing her relationship with experimentation, whether she's writing software or exploring a creative idea.

In this episode of Show Us Your Agent Skills, Thomas Vicki and I were joined by Hilary, Eric Ma of Moderna, and Brian Bischoff and Tomasz Tungus of Theory Ventures. Each brings the skills, workflows, and setups they use to work with agents and walks us through them live. In the full episode, Eric shows human-led data science in reactive notebooks. Hilary talks about creative workflows built around taste, variation, and evals. Brian shows what charting looks like when agents are treated as users. And Tomáš gets into parallel workflows, local models, latency, and memory. One quick note before we dive in. This was a live show and tell, so there's a fair bit of screen sharing. The guests were great at narrating what they were doing on screen, so you'll get nearly everything from the audio alone. If you want to watch over their shoulders, the video link is in the show notes. We also live stream many of these conversations and run free workshops and would love you to join. Subscribe to the Vanishing Gradients sub-stack newsletter and YouTube channel, and links are in the show notes. We're also putting all the skills and workflows our guests share in a GitHub repo so you can grab them, fork them, and try them yourself.

The link's in the show notes. If you enjoy the show, the best way to support it is to share the episode with a friend or colleague who you think would get something out of it. And if you want to set up agents to take on work for projects of your own, Eleanor Berger and I are teaching Build Your Agentic Software Factory starting this week. We'll help you specify what you want to build and check the results when you return. We would love you to join. There's a link and discount in the show notes. I'm Hugo Bowne-Anderson, and welcome to Vanishing Gradients. Well, welcome everybody to episode two. Clearly need to edit the outro of that video of Show What's your agent skills? So excited to be here with the one, the only, my fabulous co-host, Thomas Beeky from High MC Labs, coming at you straight out of San Francisco. I'm in Sydney here with Eric J. Ma, who is currently at Moderna Therapeutics, and we'll talk a bit more about everything Eric's been up to and up to previously, coming straight

out of Boston area. What's up, Eric? Yeah, great. The ceiling, Hugo. The ceiling is always up. I have to do that every single time you do that for me, man. Well, as I told you, my dad always says the sky's up. That's great. So I absolutely, absolutely love it. I would love to welcome everyone joining now to episode two. As we said, today Eric's going to show us a lot of his agent skills. And he really is a token maxim, maybe not quite as much as Wes McKinney from last week. But I remember late last year when you messaged me some of your, what was happening with your cursor budget. My jaw hit the proverbial floor. For those joining, please do come and chat with us in Discord. We have the Q&A there so we can continue the conversation afterwards. Also, if you enjoy this type of thing, the obligatory, please like and subscribe, share with friends. We've also got links to upcoming events in the description, our Luma calendar. Check out our Substack where we have a lot more of these types of things also. And this was motivated by a course, Thomas and I are teaching soon, on agentic data science.

So something we'll be talking about with Eric as well and seeing how he thinks about it. But if you want to learn more about how to use agents and agentic engineering principles to build data science workflows, check out our course and feel free to ask any questions about that. Without further ado, perhaps we want to get into it. Do you want to take it away, Thomas? Yeah. Oh, hey. We've got Hilary. She got joined as well. What's up, Hilary? Hello, hello. Great to have you here. And Hilary is going to show us her agent skills on the half hour. And Hilary's coming straight out. Are you in New York at the moment? I am in New York. Yes, I was at the Artist and the Machine Conference today. Amazing. And something I'm really excited about what you're going to show us is how you use agents for creative work.

as well, and perhaps tell us a bit about all the wonderful things you're up to at Hidden Door as well. You bet. Awesome. Good. Do you want to jump in, Thomas? Super excited to welcome everybody. Welcome my dear friend Eric Ma, who has been a Bayesian since before it was cool. So very early day, contributed to PyMC and core member of the PyMC development team, and then, of course, brought a lot of that in data science to Novartis Neber and now is at Moderna leading data science team. And then what I always thought was so fascinating, you always seem to be just on the next thing. So that is Bazin now, also like a genetic data science. So you have a talk at Sci-Fi and just a lot of really interesting LinkedIn posts that I love to read and a very active blog post with a lot of great resources that inspired me many times to look into new technologies.

So thank you so much for coming on the show. It's great to have you here. Yeah, my pleasure. My pleasure. Yeah, I guess I'll just get into it. How about that? Well, before that, we have three questions we'd like to ask. Two questions, yeah. Okay. Yeah, exactly. Just to warm us up. So the first one is, what do you love the most about working with AI agents? A lot of the boring stuff doesn't have to be done. That's the big one. That's the big one for me. Boring work has taken away. And I think it's an amplifier. It's like a megaphone. It's amplified the stuff that I know, and it's also amplified to myself where I don't know stuff. So it's been really cool. And I've been using AI tools and exploring and experimenting with ways of using AI tools to learn new things. So and and where I don't know a domain, it's what it's coming back with.

If I am careful enough with it, it actually can help me amplify where I don't know what I don't know. Right. And and in that way, then it's like it's a it becomes a useful tool and not just so much as what they call that word word, a psychophant. The tendency to just agree with you. Right. Like, yeah, there are ways to use AI in that way as like Socrates instead of a psychophant. Yeah. You know, that's that's a challenge. Right. So I'm not sure we get into it. but how to actually get AI to also disagree with you is an ongoing challenge. But I actually feel like the more recent models have been a bit more challenging, which I appreciate. Opus 4.7 is actually quite cheeky and can be quite rude, I found, as well. I'll be like, give me your feedback on this. And it'll be like, that was a pretty boring meeting and shouldn't have happened, Hugo. You probably shouldn't have called it. I'm like, far out. Okay. Say it how it is, Claude. I will link.

I'm going to put a link to a live stream Eric and I did called Building Tools for Thinking with AI, where Eric took us through a workflow he does of essentially having tree-like conversations where you can kind of dance around the different parts of the conversation you're having with any LLM to have as a thinking partner on a canvas. Canvas chat. Yeah, that's right. That's right. thing I vibe coded as flop and then had to use AI to undo that vibe coded flop as well. I remember that experience too. It was pretty fun. I'm already anxious what Opus 47 will think of our live stream here. Oh man. It'll be tough. Um, so what is the, uh, the thing that you find frustrating about AI? Uh, when it, uh, tries to, I think it's the lower intelligence models keeps me frustrated because I know, And right now I think it's the pricing and economics that's a little bit something that we have to keep dancing around. I just don't like that.

I would love to see intelligence for cheap for real. And that's actually motivated a lot of experimenting with alternative models to Opus and stuff. So, like, I actually do use Kimi, and I have a GLM coding plan as well. So that gives me access to Opus 4.6 or 4.5 quality models at about an eighth the price. And I also verify that the servers are hosted in non-mainland China. So they're like hosted in Singapore or in – they're outside of mainland China. And it's for personal use only. So I'm cool with that. Or I'll use like Open Router to try to access Kimi K2.5 or Olamas hosted Kimi K2.5. And those are pretty powerful too. so yeah I've been like I think the frustrating part has been the economics so like just been searching for intelligence on the cheap really yeah Have you tried DeepSeek before? I found that actually to be like pretty nice and then you can start like mix and matching between

different models so for like the high level stuff Opus have you experimented around with that? Yeah a little bit of that as well and I have multiple harnesses as well there's like Cursor which I'll be using today primarily, but then I also have CMUX running multiple terminals, each with open code. On each open code session, slightly different, different model and stuff. I've also tried hosting an LLM directly on my laptop. So the Quen 3 model series, it fits generously nice on my MacBook. So there's enough RAM for it. So it kind of works. Though I haven't really put it to the paces yet. So I haven't really put it to try to do what I did here. Quen 3.6 is really good offline on the plane when you have no Wi-Fi, which is mind-blowing. So I'm curious to try that when I get a chance. Very cool. Yeah. And then the last fun question is, if all your agent conversations somehow got leaked,

what would you be most concerned about yours becoming public? the fact that I do commits and pushes by agent commands I don't do git commit on the terminal anymore I don't even git commits I couldn't even when Thomas asked you what's your favorite thing about agents and you were like they can do the boring stuff, the first thing I thought of was git, like it has made git so much more straightforward 97% of the time, or 90% and then the other 2% to 3%, like, you end up in rebase hell that you get brutalized, I find. Right. Yeah, the fact that, and you and I, you know, have had, you know, in our repositories, you know, GitWars over the past, so allowing our agents to have those battles instead, I think. Yeah, yeah, yeah, yeah. I did learn some from a colleague who used to be an ex-Amazonian. He told me you have to rebase every time. Just don't do anything but a rebase.

But I didn't understand how to rebase. So I actually just nowadays just tell the agent, help me resolve merge conflicts on the PR such that I can rebase to merge later, right? Like little embarrassing things like that because I don't want to think about rebases. It actually, yeah, that's probably the most embarrassing thing that I've said to AI, I think at this point. Well, that's not embarrassing at all. So, better yours will be leaked than mine. I'm more embarrassed about my embarrassments now, hearing how lovely it was. Okay. But cool. I mean, I'm super excited then to learn about your workflow. So, please take it away. Yeah. Okay, cool. I'm going to share screen. I'm going to share the cursor agent. So, let me just switch over to that. Y'all can see cursor agent, right?

New cursor agent view. Is the font size big enough? Yeah, that's great now. Is this better? Okay, cool. So one of the things that I've been really fascinated with is just in doing interactive data work. And just to set a bit of context for what I'm going to show, I'm going to show Marimo pair. But at the same time, before I knew about Marimo Pair, which was February. February, Marimo Pair, I learned about it in March. In February, I did a few experiments that were about how do I do data, interactive data analysis with coding agents. And the best tools that I had at that point were if I have a CSV file, which is very common in laboratory science, I will need to write a UV runnable script that is TEP 723 style inline script metadata, UV run a script to produce a plot, and then write some

text into a journal.md file with my coding agent, right? Like that system was built around if I need to look at a plot, because Jupyter Notebooks, there's no good easy way to interact with the kernel from a coding agent. So therefore I just need to like make my plots in a script and then write markdown files and include the plots in the markdown files and that sort of thing. And that was like the best that I knew. A month later, Trevor Manz from the Marimo team reached out and was like, hey, I've got something really cool to show you. I want to see you get your reactions. So then he showed me Marimo pair. And ever since then, I have not, I have like, Like, yeah, at work, I took the time to teach people that workflow. And I'm like, guys, I need to take back everything I taught you because there's a better way to do this one month later. And now it's May. So two months later, actually last in April, I think it was in April, was when ODSD happened. And because of all of the developments with Marimo pair, when I first submitted my abstract to ODSC to do the Agente Data Science workshop,

I was going to teach people based on, like, scripts and the same paradigm. Scripts, reproducible scripts with UV run, make your plots, write journal.md with your agents. I had to throw that all out at the last minute and just showed the Marimo pair. So I'm really excited about that. So I'm going to show a data analysis using the Marimo pair skill, which is, I think, really, really cool to see. And let me just get that started. All right. So to start, I need to create a new Marimo session. So UVX Marimo edit sandbox no token. And let's call this Hugo Thomas podcast demo.py. And I'm going to create a new Marimo notebook. For those who don't know, Marimo is like a new notebook system. Its premise is basically reactive notebooks.

And so what I'm going to do is I'm going to open up the notebook directly in the cursor embedded window. So it's a reactive notebook. I'm not going to go too much into it, but if you know the troubles with Jupyter notebooks, A plus B, right? Like, if your notebooks can, your cells can go stale in Jupyter notebooks, but they will never go stale in RIMO notebooks. and I've been burned by stale cells before, and I've seen colleagues been burned by redefined variables as well, and so I know that it's a dangerous thing, and, like, reactive notebooks really solve a lot of those problems. But we're not here to manually type. I'm going to get started by going. Let's actually define a model to use. I'll use 4.6 today. Connect to my Marimo session. let it work. The way Marimo pair works is actually an agent skill that has both a series of markdown

files, plus it also has a bash script that is used by the coding agent to reach directly into the Python runtime, into the Python kernel directly. And so that gets really, really cool that you can do really cool things by interactively, right? Like it becomes more like a canvas that you actually directly manipulate. So let's try a few things. So first thing I'm going to do is I'm going to say, we're going to do a data analysis of a protein engineering campaign that I did when I was at Novartis. So, and it was published, we published a paper and I have access to the supplementary data, so I'm going to do that. I'd like to do, actually, let's just talk to it. I'd like to do a data analysis of the data that's in our data folder. It is a protein engineering data set. I would like you to load the two CSV files that are inside there. The one

labeled 002.csv is about enzyme activity, and the one labeled 003.csv is about enzyme chirality. And I'll explain those terms in a moment so everybody can understand what's going on. What we did was the data that we collected looked kind of like this. So I've got data that every rho is a mutation, and then I did a Bayesian estimate of what's its mean activity over here in this column over here from replicate laboratory data. And so, you know, like, you'll see strings that look like this, A111C, which means at position A, you know, at position 111, the original wild type letter was A, but we mutated it to C, right? And you might see like another, let me just like sort this. Some of them are single point mutants, in which case like this protein only has one mutation at position five, but some are double, some are quadruple.

Like these have four positions that are mutated, and that's kind of the notation that protein engineers and bioinformaticians use to denote protein activity. Okay, protein mutations. All All right, so we have this protein data analysis notebook that is being created. You'll actually notice something cool. The markdown cells that document your notebook are automatically being created, and that is amazeballer because I have read and written too many notebooks that are just code after code after code after code after code after code with some comment here and there and then more code, And there's zero markdown cells explaining what the hell is going on inside that notebook. And oh, my gosh, to have an agent just go, all right, I'm just going to give you lightweight headers so it's easy to even navigate is already amazing. And that was part of what I included in this particular code repo. I'll send you the link, Hugo. And Thomas, I'll send you the link in a moment.

But I included some extra rules on top of the Marimo pair skill on how I want my notebooks to look. I want to have markdown cells interlaced between code cells. We have a much more literate programming paradigm, but also we have, what do you call that? Like it's easier for a human to read and navigate in the future, right? So that's the kind of thing that I want. And so, you know, the coding agent here is obviously respecting that. So when I look at this, you know, like the first thing that I'm really thinking about is I'm going, well, I know we have some single-point mutations, and I know we have some double-point mutations, like higher-order combinatorial mutants. The first and most obvious thing for me to do would be to look at, like, you know, to visualize a heat map, perhaps, of how the mutation's profile looks like, right?

So, like, on the x-axis, you can imagine we take our protein positions and we label it from 1 to N. It's unfortunately not like Python's zero index. So it's from 1 to N, and on the y-axis, you have the letters, the 20 letters of the amino acid alphabet, and they represent the mutant character. So we can probably ask for something like, I would like to do a heat map. So let's try that. And you can imagine, like, the mean activity or the mean chiral or, you know, the RN anti-americ access is plotted over there. And actually, maybe before I go on, I'll just very, because I really love this data set because I was involved in designing all the experiments to make this data set come true. The enzyme catalyzes the conversion of a substrate into its products,

and one of the products that it makes can take a D or L form, or R and L form. I can't remember the exact organic chemistry terminology. but the amount of the percentage conversion is what we're showing on the activity table, and then the excess amount of L or D form, right-handed or left-handed form, is what we're showing on the bottom table, right? So let's do a heat map real quick. I would like you to create for me a Plotly heat map that lets me visualize the single point mutations for both activity and chirality. Let's use Marimo's ability to put dropdowns and interactive dropdowns so I can select whether I want chirality or select if I want activity. And so the x-axis is position, the y-axis is the mutant letter. and to select the single point mutations,

you just have to filter on the column mutation. They will, all the single point mutants do not have any semicolons inside there. And then the value to plot is either mean from the activity data frame or the R in antimeric excess from the chirality data frame. So we'll let that transcribe. Give it a moment. While that's happening, I'll just add, I mean, there are so many wonderful things about this workflow. One, perhaps obvious thing to state, though, the ability to be able to speak to the agent, which is seemingly our preferred way of working with such things, and then get visual feedback almost immediately or on the order of seconds or whatever, is actually really powerful. Yep, yep, exactly. And so just the ability to talk with your machine is already great for typing. And I think one other thing that will become evident by the end of this session is that I don't go into the analysis

wanting with a vague question and just ask the agent to do it all for me. Like that, I think, is irresponsible as a data scientist. Instead, what I'm doing is I'm going in and I'm taking control of the direction that I want to take the analysis in. So the human is very much in the loop. The human is there to take responsibility for the actual analysis that's happening and the interpretation that is happening, right? And so I think that's like... And particularly with doing, like, exploratory data analysis as well, right? Like, when building machine learning models or, like, doing ensemble methods and, like, grid search or whatever, random grid search, whatever it is, maybe then you can automate a bit more of that and be out of the loop, but especially for EDA. Yeah, exactly, exactly. So data science activities usually get split up into one of two big buckets. One is the load the data context into my head kind of activity, and then the other is I just got this routine optimization thing that I need a machine to automate.

And the latter is where you go Karpathy mode or, you know, like auto research mode, where you just go like, hey, go optimize this for me in a loop and let it run for 14 hours. But that's a mild, that's an extremely small fraction of what needs to be done, right, as a data scientist. A lot of it is like looking, staring, and understanding, and making sure we know exactly what is going on inside here. So I'm looking at this, and I'm going like, okay, let me, actually, just give me one moment. Let me resize this a little bit. The heat map, the colors don't really make sense for this data. Like, activity goes on a zero to one scale, but chirality, it does make sense. Like, chirality is okay to have a divergent scale because the values go from negative one to plus one. But activity is on a zero to one scale. So, like, I really think the heat maps need to have a different color so that it's much easier to interpret.

Make sure you are using the Viridis color map for activity. and the current divergent color map is okay for chirality given the data range that we have. And I want you to document this as a note in a markdown cell just above the heat map cell itself. Okay, cool. So I'm going to leave some notes for myself here. While that's happening, Eric, can I just also ask you, I saw in your local file system an agent's markdown, and I presume you might have a few instructions there for how you like to work with your agent and rules. Yep, that's right. Which is super important because it will go and do stuff that's totally wild and that wouldn't be like use UV or whatever is what you'd want it to do. That's right, that's right. And you'll notice interleave explanatory markdown with code so that the notebook is readable and presentation ready is one rule. And the other rule that I have is give every cell a unique descriptive cell name so cells are easy to reference during collaborations and demos, right?

Like that's the other one. Okay, so that's kind of cool. I'm going to do two more quick ones over here. This next one is I want to make sure that I can check the correlation between chirality and activity. So give me a plotly scatter plot as well. And then while we're also at that, I'm also noticing that... Sorry, give me a plotly scatter plot on the join of the single-point mutants that are shared because not everything was measured together. But while we're also at that, I'm noticing, like, there are certain positions that seem amenable to mutation. So I do want to have, like, the mean and the mean mutational effect at a position across all mutations, as well as the maximum mutational effect. And so I'd like to have another, like, line plot below where I can toggle between mean or maximum for both enzyme catalysis as well as the conversion and the chirality data.

So help me with that. So while that goes, I'm just going to go a little bit meta. Again, I'm gating my analysis one plot at a time, right? Like, that's the big one. I know I'm thinking about, like, what is the next most logical question that I want to answer, and I have a plot, and what is the plot that I can use to answer that question. I think I've noticed for, you know, new folks who want to come into data science but are coming from programs where research training isn't really that big of a thing, they don't always fully understand, you know, you have to have artifacts that back the claims that you have. And so if you're going to make a claim, you need to make a plot. you obviously need to make a plot to like you always need to have this artifact I don't know how better to say it but like you always need to have an artifact otherwise it's not going to work so I'm looking at this I've got the correlation

I've got this like scatter plot and I'm noticing there's not a lot of correlation so I'm just going to make a quick note manually over here noting that there's not a big correlation between R and NTO, between chirality and activity. What's the note to self? NTS, what's the correlation? All right, so I'm just leaving some notes for myself. This is treating the coding agent really as a pair programmer, not as a thing that just does the whole thing for me, right? And I think this is like the cool thing. um okay cool so it's got the it's got the positional activity for mean as well as maximum and if i remember oh yeah okay i remember this guy i remember this mutant this position 220 was huge in my life back then um we did a lot we did a lot we chose that one and we chose one

other mutant i think it was 154 uh one of the mutations there and we just said like we need to go out and milk those mutations for those who are the best. What I love about this workflow, before we wrap up, is it seems really to be a senior data scientist working with a junior data scientist or maybe even the scientist working with those data scientists, right? Yeah. And, like, the voice motor thing really enhances that. So, yeah, it just feels actually pretty familiar to work like this. Yeah, exactly, exactly. All right, here's the best one. Here's the best part, I think. A protein is not a linear structure. It's actually a three-dimensional fold of a polypeptide. And so we actually went and crystallized the protein itself. So I'm going to ask the coding agent to build me a viewer for the protein structure, and we'll see what we can visualize on there. I'd like to be able to build a viewer for the protein structure that we crystallized. The PDB file lives inside the data directory.

It's got some waters inside there represented, so I'd like to not visualize the waters, but the other head atoms, which correspond to the substrate that we co-crystallized with the protein, I would like to visualize the substrate as balls and sticks, and I would like to visualize the protein, both chains, A and B, as a ribbon. So help me make an AnyWidget viewer that uses 3DMoldJS to visualize that particular protein, and we'll do one more thing with that structure after you make it work. So one of the cool things about Marimo is it does support AnyWidget, and AnyWidget is a system that allows us to basically turn anything into a, you know, Jupyter widget style kind of interactive tool.

And when I first saw AnyWidget about two years ago or a year ago, I can't remember now, I was blown away by, like, the potential, but it also involves the use of JavaScript and ESM, which isn't the easiest. So it took a little bit of time for me to learn how to wrangle it. But now with coding agents, because coding agents are better at TypeScript than I am or ESM than I am, then it makes it a lot easier for me to go and try to build these custom AnyWidget viewers for data that lives on my disk, right? And so here's one that I'm creating. The one that I'm creating right now is a protein structure viewer. And so you'll notice, oh, it didn't get it right the first time around. It's got balls and sticks for everything. Ribbon for, okay, that's not true.

So I need to give a little bit of feedback. Opus made a mistake on this one. right now I see the balls and sticks for everything I don't see balls and sticks I don't see ribbon for the backbone chain of chain A and B please help me make that happen so we'll let that go through Hillary my apologies I hope you're okay with me eating a little bit into your time this one is just too cool I really want to show everyone about this the ability to make this happen. I'm so enjoying watching this. It's really cool. It's a lot. Okay, so we'll let Opus figure it out. And if this fails the second time around, I will actually just do the cooking show thing and show a previous notebook that I had built. Oh, there we go. Oh, my gosh, that's right. That is it. So we have the protein over here, all right, and it's loaded. And you can see it's got, like, it's got a dimer structure.

it actually looks a little bit like two hands interlocked with one another And that thing that I going to zoom in into over here that is the substrate that we trying to catalyze with I think it cofactors inside there as well. So like those, those are all there. Now, one of the things that, like I was mentioning, this coloring scheme isn't the most informative right now because it's just a rainbow color thing, but it would be way more effective if I was, for example, able to plot the color of the maximum mutational effect or the mean for activity or chirality directly on the protein structure. So let's try that. I would like to be able to visualize using the color scheme that we have already established above, I would like you to color the ribbon according to the mean or max activity or chirality based on whatever

drop down we've selected above so that we can actually visualize what's happening on the protein structure with respect to mutations. So let that cook for a moment and see whether it solves it right the first time around or not. And this one's important because, like, are the best-performing mutations happening near where the substrates are, or are they happening outside? That's an important protein engineering decision that we might want to be able to make. And so, you know, like, you can imagine with the linear chain, we're not, with the linear left-to-right view, it's very hard to see. But if you go and actually try to put the color directly on top of this, then that makes it really, really cool. so let's see how that is going. It's going, it's thinking, parsing and aggregating the data. Okay, it's doing its thing. We'll let it do its thing.

All right, okay. I'm going to reveal the answer a little bit while we're here, by the way. We did find that most of the best-performing mutations were actually outside of the active site, and that one other really cool experiment that was motivated by a question I've been having since my undergrad. Oh, my gosh. Since my undergrad days. Yes, there it is. There it is. Was where if I were to mutate two positions and just add them up, do their effects also add up or not? And they only add up if they are not inside or nearby the active site. So that's, like, a really cool thing. So, like, let's take a look at the activity. and let's look at the maximum. And you can see over here, we have the protein structure colored according to where the, according to the maximum mutational effect at a given position. And you'll notice the best performing mutation, this guy, these two, these are outside of the,

they're very far away from where the substrate goes into the protein, which is like, I don't know, I don't know how to convey this, but I am a bio nerd and this is so cool for me. This is like, oh yes. Okay. We know what to do now with this molecule. Right. So, um, right. But this is all driven by an agent skill. And this is a, one of many skills that I've used, you know, I have other skills that I use for like writing my blog, uh, helping with writing and editing the blog and whatever. But like this one has been mind blowing for me in most, uh, in the most recent time. And so I was really, really, yeah, I just wanted to share this one with everyone. I'm going to end it off with this. I'm going to put down some of my, I'm going to end it off with this final one. I think I have a good sense now of where the mutations are. They generally occur at the interface between the two enzymes. I would like you to note that as part of a markdown cell at the bottom. And finally,

ensure that we have the correlation between chirality and activity calculated and interpolated into the markdown report final cell as well so that we can report on the lack, the quantitative lack of correlation between the two. Let Opus do that. And while that's running, I was just going to say, I've put in the Discord the link to the show repository, and we'll link out to everything Eric's doing and include some of the workflows and skills, if possible, as well. And, Eric, I'm actually wondering, as an artifact, if you can share this notebook with us as well. Yeah, totally. It would be incredibly cool. This is virtuosic, man, and seeing you as a performer, commentator, and cheerleader all in one is absolutely wild with your agentic companion. Thanks, Hugo. And this really does mirror the kind of new, at least at Moderna, one of the things I want to do with the data scientists is, like, pull them into this kind of excitement of doing science with, whether it's their LLM evals or their, like, molecular biology.

I want them to be doing data science in this new way and to continue to, you know, experiment with, like, ways of making that happen. It'll be really, really cool. I'll look at that. Man, yeah. Marimo lets you interpolate stuff. It's just really cool. So, all right. Cool stuff. Amazing. Well, thank you so much. And we will share all of the skills and references for people to get up and running with. So I do, just before we jump into having Hillary share her skills with us, I did want to share, we make every guest a little video, and I just wanted to, we have an agent skill to do that. And let me, well, you know, the demo gods shine. Oh, yeah. So what Eric did with agents felt definitely more collaborative than that.

But I actually have an agent skill in the repository where everyone can go and build their own such videos. So thank you once again, Eric, for coming and showing us your agent skills. Yeah, thanks, Hugo. Thanks, David. So, and please do feel free to stick around as long as you want. I know you've got dinner and got to put the kids to bed, but you're more than welcome to stick around. It is with great pleasure that I get to introduce Hilary Mason, who is working on amazing generative AI stuff at Hidden Door, which I'm excited to hear about, kind of has been doing generative AI stuff well before our chat GPT moment as well, and was one of the few people who kind of put me onto really seeing the power of language models early on. I first met Hilary when she was running Fast Forward Labs and in fact, fun fact the first podcast I ever put out

was with Hilary really? yeah, back in the day and it wasn't the first one I recorded I think it was the third because I was a bit nervous so I thought I'd I mean, you're wonderful but I was like I definitely want the one with Hilary to be a good one so and that was That was in 2018 or something. You were so good. I had no idea it was among the first. I appreciate it very much. I've got to tell you, though, definitely you get to the 20th or 30th and you realize you learn so much and you don't want to go back and listen to any of the first ones. Hillary has done so much from being a data science in residence, Excel partners, to being an advisor to the Data and Society Research Institute. and she's done a lot of data stuff in New York City and community data stuff in New York City. And so I did want to show the video we created for Hillary up front

because it is New York Focus. There you go, Hilary. Golden hour in Brooklyn looking out over New York City. That was amazing. And there's no Times Square in Brooklyn, but I'll let it go. That was awesome. I think it was over the river. Oh, okay. I could be mistaken. But the other thing about Hilary that I just want to make clear is Hilary loves data and cheeseburgers and always has. Yes, that is one constant. Cool. So should we dive in and see some of your agent skills? Oh, actually, first, our questions. What do you love about working with agents? What I love about working with agents, I'm going to give you two quick answers to that.

One of them, I'm going to say, are you all familiar with Richard Hamming's writings on science and engineering? No. Okay. So this is like a classic and really essentially the TLDR is he says, you know, in science, if you know what you're doing, stop. And in engineering, if you don't know what you're doing, stop. And that always stuck in my head because certainly working in data science, working as a software engineer, we use the same tools, but we use them in very different ways. There's a different amount of experimentation. And this is a long way of saying that I feel like this moment where using agents and software development, using it to build all sorts of things, is bringing the science back into the engineering and the engineering back into the science, like what Eric was showing us just earlier, in a way that is really cool. Like it changes at least my relationship with writing tests or doing some of the more, you know, boring parts of software engineering.

It also changes the way I think about doing product experimentation or being able to like make something to touch and develop an opinion about where previously the making would have taken so long that it would never be worth it to do something you might not use or build on. all of it with the goal of like having a more robust understanding of taste of what grade is for whatever you're trying to do. That's one thing I'm excited about. The second thing I'll give you is just purely personal. I'm a CEO. I have a million bad ideas. My schedule is often going this way and that way and going over there and over here. And now I have the tools to set something moving and then have it, I can go do something else and I can come back and I can remember where I was and what I was doing without losing the whole thread or the whole idea. And that may just be the way my brain happens to work and my schedule happens to work.

But even a few years ago when I was doing engineering work for the company, I would have to set aside and block out like two and a half days a week and be like, I need at least four hours to get into this problem enough to make enough progress. so that I don't lose it all when I have to go to a meeting or go do something else. And now I can sit my engineering work in around everything else in a way that has never been true before. So that's the other thing I love just personally for my very scatterbrained schedule. Yeah, I can relate to both of those. And I do think the organizational aspects and being able to have something which keeps everything in context and serving you the right information, I feel like the systems can do that. they're not always consistent still, but it's amazing how far we've come. What do you find most frustrating about working with agents? I think it's the same thing I find frustrating about LLMs and most of these models to begin with,

which is that they're very mid. Like, they are super biased. They're very themesy, and this is a problem we deal with a lot at Hidden Door, where we're trying to, like, let's say, make a character, And if you ask it for like a doctor, it's always going to give you like a man and it's going to have, you know, maybe an Indian name because it's racist. And, you know, you're going to see a lot of repeats and it's just not necessarily great. And so in order to get to great, you have to bring a lot of context. You have to be very sharp in what you actually want out of it. I'm a huge fan of like the superpowers plugin. I think everyone you've had on here has been because it brings that rigorous process and sort of verification into things. So I think it's just that they're aspirationally very mid. Totally mid. And I love that you use that term. I've actually been thinking a lot about recently what the Internet, future of the Internet and the agentic Internet.

and I use the analogy of, I'm clearly on a 90s tip at the moment as well, but there's an episode of The Simpsons where you see a close-up of Homer Simpson's, this is biological people, sperm, and it's all these little sperm with Homer's head on it. Oh, I do remember this. And they're going around just bumping heads into each other. And I think that's what a lot of the agentic internet, that's best case scenario for what a lot of the agentic internet will be, a lot of mid-agents going around trying to sort stuff out, and some will try to context inject. I think a lot will try to steal reasoning tokens from each other, like get other agents to do work, right? You know how you can go to, I don't know, what was the most recent example? Chipotle, and get it to build a React app for you, or something. And it'll just be agents trying to do that with each other. I mean, I had an experience recently where I took something like a sampling of a thousand player inputs in our story system outputs from our game and I put it in a google

seat and then Gemini pops up with like can I help you and I'm like actually yeah for the first time I have a thing I want you to go through and classify repetition and it said no I can't do that that's qualitative a part you need to do that I can only do quantitative things it's like I don't be an LLM to do quantitative things. Like, I can remember that formula myself. Yeah. So, yeah, it is also the pushing that labor back to us. Totally. No, thank you. And they do push labor back. And so our final question is, if all agent conversations got leaked and released, what would you be most concerned about becoming public? So I heard you ask Eric this, and I was like, you know, okay. I'm not actually that embarrassed by anything so I went to Claude and I was like go read all our transcripts and tell me what the embarrassing thing is and so what it came back with was that sometimes I ask what were we doing? Why are we here?

What did I tell you to get this outfit? So there you go it's that context switching again but yeah I don't know what are you embarrassed by? I feel like I'm missing like a good story here. I think the thing that I am, and I mentioned this last time, when I get frustrated, I dictate a lot. I can use language that I would never use with a human colleague. Like when I've probably been, you know, working too late and, you know, we're both making mistakes and I just want to get the final thing done and it comes out with absolute trash after bamboozling for 11 and a half minutes or whatever it's doing, when you get really dumb-clawed and not smart-clawed. Oh, yes. I can use words that maybe are a bit more acceptable in Australia and not in the United States, but stuff that I definitely wouldn't use on a YouTube live stream. Fair enough. I don't think that's incredibly embarrassing either, but, yeah.

You haven't heard the words. Okay. Half-joking. Let's love to see what you're up to with agents. Yeah. What's your agent for us, Hillary? I was hoping I could show you some of what we're doing at Hidden Door first because it does involve agents and then sort of go into the, you know, how do we make it all work as a useful context. So if you will bear with me, I am going to first share a web browser and then we will get into probably Zed and Warp over here. So let me pull that up. And or maybe we can do it all at the same time. So share. Oh, there we go. Okay. We are going to share, not the tab. I want to share the whole window and hopefully not our own faces. All right. So I have a few things here. My current company is called Hidden Door.

We make a platform for people who like to build worlds to come and world build. So mostly people who write books, but also people who make movies or just run D&D campaigns, all that stuff. I'm going to show you a little bit of what playing a game looks like just so you get the idea. This is a world I made. It's a startup parody. And it is sort of a, like, you know, extreme startup parody with the investor being an actual robot. But we have a mechanism where you can throw in, like, modifiers. So we can make this, let's say, a Western. And now it gets, you know, written as a Western. And so we let anyone who's a fan just sort of come in and riff. I'm going to jump into that. I'm not going to spend a lot of time on product stuff, but I do want to show you the context because it's pretty important to understand the other side of it. I have to make a character. So I'm going to say badass, coffee, drinking, coding, CEO, lesbian lady.

And now we're going to get a character back. All right. Oh, I left the Western on, too. So we're Western startups here. We are going to coding queen of the Wild West. We're going to go with us. That's going to be fun. Because Startup Land does feel kind of like the Wild West in a lot of ways, right? And there are, you know, frontier models, I suppose. And that's one way to say it, right? I hadn't put the frontier connection together. Yes, these are ways that our players can come in and be like, oh, I want this one to be in a coffee shop or I want it to be a romance. This is a pretty, you know, gameplay. We're here in a startup co-working space. All the, you know, sort of characters are cards. That is a good name, Shirley Locust. Eager Beaver, she's the intern. Okay. And that's it. So, like, in this game, you can, we always give people examples.

Because if we don't, if we just give you, like, a text box, people get writer's box and they don't know what to do. So, but if, you know, you can type anything. Let's see how this goes. And then I'm going to show you one more thing. Okay, so it's an easy dice roll for this cowgirl agent. And off we go. So this is what we do at Hidden Door. That's the whole thing. We are role-playing. Y'all, I'm live. All right, we're going to go with that. uh i'm going to show you one well yeah i'm going to show you one more thing which is a new product we haven't released it yet but if anyone wants to play with it uh you're all welcome coming out next week uh this is something where i've taken everything i've learned about uh creating evals for creative works in agentic software systems and brought it into a product experience and so

I'm going to show you this as the output and we're going to go back around and see the behind the scenes sort of like how does this actually work so we're going to start from an idea to build a world I'm going to go through this really quickly um so it's not going to be like a a good idea as I said I'm like full of bad ideas I'm going to say uh you know or if you have have one Like, Hugo, Thomas, you look, maybe you have a world idea in you. Oh, it's science fiction. Okay. A live streamed science fiction show where every participant must solve a puzzle or, you know, die horribly. Let's make it horror. Horror sci-fi. Okay, we'll see what happens. And can I just ask quickly, firstly, Hilary has been working on Hidden Door since 2020 or even before.

Yep, 2020. And you've burnt into all our traumatized memories. The other thing I'm interested in is, Die Horribly is something which a lot of models won't do for you at the moment. And we've talked about, I think the last podcast we did last year or the year before, talking about how frustrating guardrails can be when you're trying to be creative. Well, and we had a whole year or so where we had to solve the problem of, like, how do we get our characters murdered? So your off-the-shelf LLM will not do it. Like, we have a world you can play in the world of the crow. So if you remember this 90s movie, ours is based on the graphic novel, But the idea is that you are murdered in the first scene of the story with your lover and you come back for revenge. And this is your like inciting incident. It's the motivation. But in the very first version of this we made, the LMs were sort of like, oh, they put the gun close to you.

And the next turn is closer now. And they would just keep going. But never. It was like, you know, Zeno's, you know, armory, I guess. and so yeah solving this problem has been pretty tricky but uh but it is now possible um with our own guardrails and content standards and taking certain things out of the hands of the um so candidate on this live stream uh and must be entertaining or they lose great uh as I said I'm going through this super fast I want to point out a couple things though um one of which is that as I type answers you'll notice on the right side are sort of like sharp editorial, what are we making here? A vision is updating. Our inspirations are updating. The tone and the sort of like TLDR is updating. And one of the things I wanted to highlight on

to this crowd is that this is really about bringing an interface to this kind of like agentic looping around an idea. And you can do this for software and we often do it with the whole like, here's my goal, you know, let's make some tests and evals and then let's like, let the models go and try and make the software until the test pass. And then we say like, cool, we did it. Then you look at it and you're like, okay, maybe you did it. Maybe you didn't do it exactly the way I wanted, but like, that's the loop we find ourselves in. This is trying to do the same thing for helping a person have a better idea of a creative thing they want to make. and so our goal with this is that it is never really mid it is either something that is like yes that's what I meant or you hate it you should not feel empathetic um and I just want to skip through these next few things you'll see we do give you uh we give you options you can just sort of take and edit sure we're gonna go with this um and then you have a whole world and we move to

reviews. That's our world. It'll be playable and you can even, starting next week, publish it and have fans and we do Repture and it's a lot of fun. But that's the product side. I do want to show you some of the really bonkers fun stuff we're building. And I guess I will now switch over to the terminal to do that. Okay. Quick word from our sponsor which is, well, me. Vanishing Gradients is a podcast, workshop series, blog, and newsletter focused on what you can build with AI right now. Over 70 episodes, hundreds of hours of free hands-on workshops, all independent, all free. If you want to help but keep going, you can become a paid subscriber on Substack, starting at $8 a month, or share this episode with someone who'd get something out of it. Links are in the show notes. All right, let's get back into it.

Well, I cannot wait to next week go and start building games and sharing them with people. Yeah, yeah. Let me find that. Correct, Cameron. Oh, no, now I just found the one that's how embarrassing is everything. While you're finding that, just want to say what's up and fish off. Hello. Hello, hello. Great to see you, man. Likewise. Okay. spring up there we go I'm going to share also I just want to say Brian I've never seen that evals baseball cap before that's so cool that you are you are next level evals dude appreciate you so much man I'll give you one hint as to where I got this hat one guess one guess Hamill producing them?

One's part of the generative process in these, but yes, these are from Hamill. Sorry, Hillary, I don't mean to interrupt. Go ahead. No, no, I love it. I've been watching on YouTube, Hillary, so I'm here for the punchline. All right. So I'm not going to show you pure software stuff, but I'm going to show you the, this is the idea behind our adaptation of sort of agentic software workflows to creative workflows. And we have a bunch of places we use the philosophy, but it's essentially first interview the person doing the work. What do they actually want? And this is not trust them when they say something. It is ask questions to come at it from five different directions. If we have time, I can actually show you. We have a skill for this. And how are we going to test this? Of course. Then we go to the, what is our current state?

So we have a bunch of test fixtures. In our case, it's like frozen game states from our game engine. But it could be anything from like a story, any form of output. What is the current state? So we sort of run against currency, save it. Then we look at variations, and this is where I think there is actually an idea worth talking about. If you ask an LLM to say, like, give me a better version of this prompt, say we're working on tuning a prompt task or something like that, or a better version of a skill or a better version of a post, it will give you one thing. If you ask it again separately, give me a better version. I want to get it here. It will give you something very seemsy. But if you ask it at once to give you three very different versions that have different magnitudes of change and risk in the changes, you can actually get multiple variations that are somewhat more creative than if you ask for just one.

And I've experimented here with this number going up to 10. uh there are very few problems in which like 10 stabs into the potential space you're exploring are actually useful so like three seems to be the happiest number five sometimes is good if you're doing something a little bit more out there but like you need three different variations um in the same prompt so that's my one like if anyone is like taking one thing away for creative stuff multiple variations and then you have to have a really robust set of evals so that you can actually test the variations and know if they work for the goal so um in our case we define evals as sort of like a you know here is a set of criteria that we are looking for editorially compare each output against the set of criteria score it run multiple variations and therefore we get to something that is not, it's not going to give you an independent sense of like, this is

great, but it will give you something you can compare to that eval we did at the beginning of the whole process so that you know you have made something better or you have made something worse. Well, this is super exciting stuff. So we use the Jest file here at Hidden Door like endlessly and write a bunch of scripts that get called from here. So like this little bit is what I'm going to show you. It is our test story task. That is some of the LLM tasks that we use to monitor stories in the background. This is actually just a little Python script. I don't know. There's nothing super fancy in here. It calls this skill. You said there's a repo we can throw examples in. And I also had Claude sort of pull it out of our contest so anyone could use this for prompt refinement. Though it lost a lot of its personality, so I'll have to go back and edit it again.

This is what I'm going to show you. So we're going to switch one more time. Actually, I have three things to show, but probably that is not actually going to happen today. because this is taking too long and it's more fun anyway. So here we go. Share something else. Let's go find. I'm curious, like, how many people that you have on this show are actually using an IDE? because the last time I asked a bunch of my, like, super agent friends what IDE they were all using these days, I got roasted for, like, using an IDE at all. Like, why would you even look at your code anymore? I was at the AI council yesterday, and Wes McKinney was giving a talk, and one of the questions was like, so which IDE are you using today, Wes?

and he's like, IDE, editor, what does that even mean? So not that much. Well, I do feel Wes is an outlier. I saw Brian's hand up, so I'd be interested in his thoughts. But, I mean, I know a lot of people who still use VS Code with all types of plugins or use it to look at code every now and then while using Claude Code from the terminal. Cursor, a lot of people use as well. people, and I do think Jeremiah had VS Code open, I think, and I'm not sure what Randy did, but yeah. Brian, how about you? Or you had your hand up. Yeah, I'm still using IDEs. I use a couple for different things. I use Cloud Agents. I'll show some of this during my session, but yeah, I still think IDEs are valuable, and I use them a lot. Okay, that's very reassuring. I don't want to get roasted for actually using an IDE, though I was a Vim user for so long.

VS Code was good, but very heavy. All right. Oh, come on. We are going to pick a task here. So we're going to work on a prompt refinement task. I'm going to say our scene notes task because that is one that looks at what's going on in a scene and sort of gives an editorial take. uh, for role-playing. So if everyone remembers back, what was it, like 15 minutes? Um, and here it's starting that, uh, that interview. So what do we want to say? Like, um, you know, I played a scene, I don't know, I'm sort of role-playing an idiot here. Uh, but like, and it didn't have specific details of the world for immersion. This is an example of something we've played with a bunch, and we can see how it goes. And while that's running, and this is relevant because we chatted about it last week, I think.

Did you use a slash command to invoke the skill? I did. So this comes back to agents being kind of mid as well, because I don't always use slash commands, but I have some skills that I know it's just going to be easier. Like the agent doesn't always know what I mean when I say, you know, I want to do this, right? With the 8-bit video generator, it's fine. Like I say, I want to generate an 8-bit video. It won't try to do that any other way. But some people... Exactly. And some people must say it's a skills issue, as in a Hugo skills issue in terms of the name and description, tried a lot of different things, doesn't really work. So slash commands can be pretty powerful there. Absolutely. It is running against some of our existing test fixtures. It's going to do that audit. It's going to come up with some, like, hypotheses. I don't know that we want to wait for the whole thing. We still sort of in like slow claw mode over here but I did run one earlier so i can show you the output and then i want to show you um my pet gremlins and then i will let brian jump in with some fun

fun stuff um so here i cheated i did the test kitchen thing and i ran this earlier today now we're going back to the ide's sorry sorry yeah the other thing no i mean and this is we experienced this last week as well, demos with agents can be tough. And Thomas and I are thinking about this a lot because we're teaching our course on agentic data science next month and just figuring out how much to do live, how much to have canned, because the latency is non-deterministic as well, right? Yeah. No, when I ran this earlier this morning, I just got back from Europe, so I'm suitably jet-lagged, and I was running it at like 5 a.m. Eastern or something, and it was lightning fast. the best time. So this is sort of the example output. What did we do? We ran a baseline. We ran some variations. We did a bunch of diffs. And then we have our eval criteria, which I can also show

you. And it's getting a report. This is a summary of the report. It's not usually presented this way. But looking through quality, does it meet our programmatic metrics for successful output? Does it meet our qualitative metrics for successful output? And that's basically the whole idea. I think it's super similar to the processes that we use for making software, but it's for, in this case, a prompt. We use the same process for making the content that is part of our system. Sometimes, if it's not just easier to write it by hand. But you can say, like, write me a really good heist and then test that against the editorial criteria we've written separately with independent sub-agents. Super cool. It does remind me, and I said this to you, but Randy Olson presented how he trains AI agents to build Tuftian-style

visualizations. Oh, I love that. Yeah, using this generator-evaluator pattern, which is one of the five patterns that Anthropic published in their building effective AI agents. Is it? They're going to find five different workflows at the end of 2024, like chaining, routing, parallelization, and the fifth one was generator-evaluator or generator-optimizer, where you get this iterative loop, which for creative purposes can be incredibly useful. Absolutely. And I just want to say, I've probably said this three times, so I'll keep saying it. The point of all of this is to get a really sharp creative take from the person doing the writing and to really understand that if you do it, if I do it, if any one of us does it, we're going to get a different output. That's not because one is better or worse. It's because we each have a different idea of what, you know, the creative work should be or like what makes a really good, I don't know, output for, you know, a startup story. And that's a,

that's a feature, like that variability is something that we certainly bring. I'm going to show you quickly, because I forgot I opened this tab, but I like it. So I'm going to show it to you all. This is one Python package I made, and I was going to share it as an example of the kind of thing you can do in between a bunch of meetings. This was a typo I had of deep copy, the Python function for actually copying a data structure. And I thought, oh, hey, that's actually kind of cool. What if I could hear my data when I copied it? And it turns out it works. So this is a Python library that just does deep copy, but it also makes a music file of your data when it's copying. and then you can, it ships with a few different renderers. You can plug in your own renderers. You can get like a little techno EDM version of your data structure. And before you laugh at me too much, I found a bug in our like game state representation

because of this. Cause I heard this like repeated refrain. It's like, we shouldn't have that. So that is the kind of, you know, bullshit you can get up to when you're, you've got 30 minutes between two meetings and a bad idea. I love that. And I was giggling with you rather than... No, I know. I am the champion of, like, use the text to explore the space of what is possibly interesting. And if you only, like, if you show me what's on your roadmap and it's, like, only good, obvious ideas, it is very boring. So it is, like, push those a little bit in a creative way. And to that end, the last thing I will show you is my gremlins. So I actually did this a couple of weeks before the whole OpenAI gremlins thing. And I wonder if I had like subliminally picked up on the gremlins, like being in there. But I love them.

So I was inspired by Night Shift, which I don't know if you know it, but it's a library that checks if you have any extra tokens in your subscription budget at the end of the week. And then it does all the things you're supposed to do. It's like a really good software engineer. Like it'll go through and run your LinkedIn every so often. And it tracks when it last did that. And it'll look for tests that aren't relevant anymore. Like all of those really boring tasks nobody really wants to do. We've automated a bunch of those anyway. So not useful for us here, but like, it's a great idea. And I looked at it and I was like, oh, I am, I hate this because I love the idea of like, okay, it's the end of the week I have some tokens um but what I want is to work on the weird stuff like all the bad ideas all the stuff that like nobody has the time for because it's way too out there on the distribution of things that could be productive okay Claude finally figured his stuff out um so it took that long if that uh if that helps six minutes 23 seconds to run our little eval here

um so that said uh what the gremlins do is they each have there are three of them they have personalities and we have at hidden door a slack channel called bad ideas and people put their like they're not bad ideas they're usually the ones that are just shower thoughts or like maybe someday or sometimes a bad idea or two and then it's sort of like we collect those up into a text file it goes in the repo and then every Sunday night I have a like sort of procedure kind of job or whatever the claw and crumb job thing is and it runs the gremlins and then we you know sort of wake up every Monday morning and we have like this is our real gremlins from this week um and these are all ideas uh that each one has a personality so this is me um there is a perfectionist, which is like make our code base and our product amazing, but like perfect. And then there's the sort of player, the user perspective of like, what would a player want?

I sort of pull off this bad ideas list. They make their own spin on it based on the other goals they've been given. And then they propose, they pitch to each other, they critique each other, and then they write a design doc. And so this is just our version. And then if we like any of it, then we can say like, okay, you know, go, go actually build this. But I also extracted a hopefully working version that anyone can run against their code base. So maybe we can share that. And I would love, love, love if people give it a try and hopefully it gets more of our bad ideas out there into the world. I can't wait to try it and we'll share it. I have so many bad ideas that... I believe you. I mean, it's obvious, right? no no I mean like this whole podcast is the kind of bad idea I mean where I'm sure it started as a what if and then a why not and now like here we are now and uh it's super fun so this is uh

this is the kind of chaos we need um thank you so much for coming and sharing your agent skills with us Hillary and we there were other things like a lot of your personal agentics oh yeah We'll stay back for next time. Exactly. We'll have to have you back. And I do, if you're able to stick around just for a couple of minutes as I introduce Brian, it will become obvious why. Okay. If you have to run, of course. But firstly, Brian did send us a demo. No, that's not Brian's. That's Hillary again. And let's, whoa. Oh, this is actually interesting. I did let's see yeah it is across the river in one point but it was at Times Square in another so that's wonderful AI slop there yes but that's why we love it exactly

So that was a demo video Brian sent us last week of what he wants to show today. And the reason I wanted you to stick around, Hilary, is because I actually vibe-coded something else for Brian in the spirit of what we're doing, which Brian doesn't know about. And it's very silly. But here we have a game called Brian Quest, Cosmic Run, which I'm playing right now. Turn down the audio. Yeah. It's so loud. Oh, shit. I know how to... Oh, I just died. Oh, no. I can turn down the audio. Oh, no. No, I'll platform. No, I'll put the lamp in. Get it cut down at all. No. That's fine. But it's fine. Just go ahead. Okay. And show us how it moves. Avoid that guy. Yeah. Okay. I avoid him, and then we've got some power-ups here, and I can jump on him or not.

And if I jump on him, we actually go to the Cosmicron. Ooh, wild. Yeah. And if Brian, because his... I don't know what happens when... Ooh, okay. Great. So, I will stop that now, but I just wanted to share all the bad ideas one can possibly have. I love it. and the silliness of gaming as well and showing what Brian's stack really looks like. Yeah, that's my day-to-day, so I appreciate you revealing that here today, Ego. Yeah. Amazing. Awesome. I will drop off. Thank you again. That's been awesome. Yeah, great. Speak soon. So I'd love to introduce Brian. Brian is, like, I love Brian's work. I always learn from Brian in every conversation we have, one of the most interesting people in the space. Brian leads AI at Theory Ventures. He's worked all across data, ML, and AI. Most recently, head of AI at Hex,

where led the team of engineers building magic, data science, and analytics co-pilot. Started the team at Blue Bottle Coffee. Led several projects at Stitch Fix and built the data teams at Weights and Biases. Co-authored O'Reilly Book on Building Production, recommender systems and is teaching data science and analytics at Rutgers as well. We also have the wonderful shared background of not only doing graduate work in pure math, but in non-commutative algebraic geometry, which we won't dive into too much today, but category theory for life, my man. And let's jump in. So just quickly, our questions. First question. What do you love most about working with agents? I feel like it raises my level of ambition. There are things that before I'd be like, I don't know, there's no way I'm going to have time to do that. Or like, oh, this is the way I'd do it if I really had time to get it right.

And now sometimes I'm like, YOLO, let's see what happens. And then one thing leads to another. And, yeah, I actually do much more ambitious things than I would have done before. So, yeah, I think it's really upped the ambition, which has been super fun. Amazing. What's the most frustrating thing for you about working with agents? Yeah, and you'll see this a little bit in my demo, but sort of like consistency or any sort of like awareness of where you've been before. I get really frustrated when I like tell an agent something and then like two turns later, it's completely forgotten what we talked about. And it drives me nuts. it makes me want to not use the agent at all and it makes me very sad. I can relate to that. And it's when that happens that, you know, I start using words that one wouldn't use in other situations. If all agent conversations and interactions got leaked, what would you be most embarrassed or concerned about becoming public? I think people would be like,

wow, he's really bad at typing. Like, wow, he can't stop anything. Awesome. Give it a name with an I. Not today, but some days. Brian, show us your agent skills. Okay. So I've been working on something for a little bit because I have a problem, and I wanted to try to solve that problem. So I like charts. I just really, really like looking at charts. I like making charts of data. I like taking data and making those into beautiful data visualizations. I'm one of those people that used to, like, look at every D3 visualizations that dropped on the Internet and kind of, like, obsessed over them. I learned D3. I got really into, like, you know, trying to make beautiful data visualizations. But these days I have a lot of charts I want to make, and I want to make them with agents. Now, I assume both of you have tried this.

I assume that both of you have made charts with agents. I would also expect that you've hit some annoying roadblocks. One thing that you've probably seen is that agents are really good at map plotlib. That's true. Another thing that you've probably seen is that agents constantly forget what the hell you're doing. So you will make a chart, and it will be kind of close, and then you'll ask for one change, and it will somehow manage to completely break the chart. and then you'll get it back, and you'll say, like, oh, the legend's in the wrong place, and then suddenly there's no legend at all, and you're like, okay, that's not exactly what I was thinking. It's very hard to iterate. It's hard to say, here's the vision I have for the chart. And then once you realize that you want to get into, like, some other variant on that chart, it's very difficult to get the agent to do that. Also, charts are not very portable the way that we currently build them. I can dump a ton of Matplotlib on somebody else, or on another agent, and all the context is lost.

The entire conversation just goes out the wall. And so I've been thinking sort of like, you know, as a grammar of graphics kind of guy, I've been kind of thinking like, well, what would be the grammar of graphics for like the agent age? And so very humbly, I decided to create a package called BBplot instead of GGplot. and BB Plot, I'm BB, but BB Plot has like a couple core foci, and that is what if all of your charts that you want to make, these are static charts, these are not like sort of like reactive or animated, what if all the charts you wanted to make you were making through a human agent conversation? What would you do differently? What would that stack of sort of stack of fundamental ideas from grammar and graphics look like in the age of era? And so I started thinking about it. And I've been working on a project, which I will be open sourcing.

But I will be – this is the first time I've shown this to anyone. So we're going to give it hell. And again – Amazing. Well, thank you for sharing it here for the first time. And I love that you're bringing BB Plot, and Hillary just showed us Beat Copy as well. So it's the days of the BBBs. Yeah, a lot of Bs. So let me see what I can do here. So like all great movements in technology or politics, I started with a manifesto. And so here's the BB Plot manifesto. a chart spec should enumerate how to render the chart and what context has been provided by the human pay attention to that second part we'll come back to it every choice that's been made explicit by a human should be codified in the spec what are my core beliefs my core beliefs are that the context should be packaged with the content this is the concept that I've been working on with my colleagues Adam

that when you are trying to build something with agents, you are providing lots of context about what you want built. And you either often lose that context or you package it as a separate thing that kind of gets lost or forgotten. It's not packaged with the core asset or artifacts. And so you have no hope that it'll stick around. What if you just force it to be in the same place? BB Flat does that. next ambiguity yields variance there is a ai app that i think a lot of us have played with but it may have been a while for some of us and that's mid-journey and mid-journey always did this really amazing thing you'd ask it for an image and it would give you four and those four would be subtly different and often you would find one of those four that felt really aligned with your vision, and maybe one that was like, okay, and then two that were way off. I think that's a great design pattern, because when you are trying to build things, you very

frequently have some things that you know, and some things that you're trying to discover. So my claim is that you should be constantly generating variants, and that the agent, yeah. I was just going to add, I love that, and this is actually a core tenet of parts of recommender systems and what we want in retrieval now. Like, sometimes you're looking for something very specific and you want the solution. Other times, you actually want to have several different options, right? Exactly. Exactly. And you're an old Rex's head. Exactly. Exactly. So, yeah, I'm top K. K does not need to equal one. You're allowed to let K equal five. Well, there also is a design principle, and we don't want to go down this rabbit hole too much, where, you know, internet search, Google search, maybe didn't, for all searches, need to be rank ordered, one through whatever. Perhaps some searchers want a cloud of different options to navigate. Totally. I used to work on diversity in recommendation systems,

which was precisely this idea that, like, actually, not getting the top K best, but getting the top K that spans the sort of distribution is even better. But I could ramble about this for too long, So I will move on. The render explains itself. So this emits a scene graph. Some of you may have heard of scene graph before. It's an old idea from computer vision. But a scene graph is a thing that tells you what is on the screen and how they relate to one another. And so this, you know, object detection we're very familiar with. It's like what's in the picture. But what's in the picture and where? And how do they relate? One of my old colleagues actually introduced me to scene graph. And, you know, it's a really great design principle for understanding visual type things. When you're making a chart, yes, you care about what's on the chart, but you also care about where they are and how they relate to one another. Let me give you an example. If you tell me there's an annotation for these five points on the chart, that's interesting.

but it would be more interesting if I knew that they were on top of one another because five annotations on top of one another is not super useful. And this is sort of like has a double feature. The second feature is when I say, no, don't make the legend overlap the data, the scene graph knows what that means. Agents really struggle to understand those kind of things. even, you know, VLMs or like multimodal language models struggle to do image understanding in a way that can easily defrag where the legend position is. This helps. This is a... We just had that last week with Randy Orson, who also complained about the same thing. They just like to put things on top of each other. So, he's solving that problem. That's like probably one of the biggest in the database right now. Precisely. Next is that all features should be demonstrated. We live in a world where agents are going to be the main consumers of these things.

Remember, BB Plot is not for humans. BB Plot is for humans plus agents. And so what's the best way to document a chart builder? By demonstrating the features. So every single feature must be demonstrated as part of the gallery with the spec and the image. And then finally, the DSL should be optimized for agents. Now, this one is the most trite, but also the most important. Because as a human, I don't actually know what agents want. I can make some guesses. I can make some claims. But you should not believe me. If someone tells you, I know what agents want, you should be immediately suspicious. And there's only one way that we could ever hope to determine what agents want. And it's on my hat. And so that brings me to the second project, which is called BBplot eval.

It is important that this is a separate project. In fact, this is a separate repo. I'll pause for a second before I explain BBplot eval. a dramatic pause or a pause for a question if you've got a question yet if not I'll move on I'm here to see yeah and I love the design choice total separation of concerns here and I presume one thing there is that so agents can't game it as well oh yes and we will get to that right now BB plot eval is a project that determines what is possible with BB Plot. It has a set of target charts that we hope BB Plot can make. But more importantly, we hope that agents can make with BB Plot. Not just can be made with BB Plot in a superhuman oracle charter, but can a random agent, can a new fresh agent with little context.

That's the goal. BB Plot is pinned and external. It is separated. It is, you are not, in bbplot eval, you are not allowed to look at bbplot code. You are only allowed to look at the documented examples. What documented examples? The one that I required must exist. And also, for people who are doing this, like, I don't want my agent to look at code, different repo, do not remove it from the git history. They'll look at the git tree. They'll go anywhere they absolutely can. And I'm sure it can't read other directories in your local file system as well, if you've Git Exactly. That's why it's pinned. You'll see the last point here, how I help make this easier. The chart targets are prompts and requirements. I recently introduced this definition of an eval card. All of the examples are eval cards. Those have prompts. Those have failure funnels.

Those have predictions on how the agent could screw up. That is really, really essential because the goal is not to score high on a benchmark. The goal is to generate information about gaps. Feature requests to BBplot are developed by generalizing failures on specific evals. The feature requests are not allowed to mention specific evals, But everything that is built in BBplot is not generated by me, a human. It is generated by failures at the evals. It's like a totally new development workflow, right? I never heard of this before. Exactly. And so BBplot eval has a builder agent, and the builder agent is trying desperately to use BBplot to pass its evals. And when it fails, it has to write up a failure report, which I'm going to show you. And that is what I hand off to the builder agent in bbplot, which then is trying to satisfy its requests.

bbplot versions are evaluated. So bbplot must be released as versions, and that is what we sort of, like, evaluate. But we are going to use a ratchet. And so the idea here is that if you regress on any evals, that is caught by bbplot eval and is filtered back to bbplot, and there is a sort of like patch version in BBplot to address the regressions. We're not allowed to move forward until we have continually ratcheted. We're not always getting better, but we're never getting worse. And finally, BBplot eval is an orchestrator. The goal of BBplot eval is to dispatch prompts and requests to other agents and see if they can pass the evals. And as they fail, the builder agent says, hmm, I know how to get this right. Let me see what maybe I need to pass along to BB plot.

And so these are the two repos and the two projects. And remember what I told you, I don't know what agents want. I'm just trying to converge to what agents want. So let me show you a little bit what kind of stuff we've got. This is the gallery. You can see, these are relatively basic charts. No big deal. You can see here that there's like a spec. A spec demonstrates sort of like how I'm generating this thing. And yes, it's YAML. Deal with it. But these charts demonstrate all of the functionality of bbplot. I don't know how good you two are at matplotlib. I'm not good enough at Matplotlib to make this chart in Matplotlib I promise you, like I'm not capable of this, but BDplot is an agent with BDplot it doesn't look like Matplotlib to me and Matplotlib usually looks like Matplotlib having said that, Seaborn

which is built on top of Matplotlib customizes it in beautiful ways as well so it is possible but it does not look like Matplotlib to me and there are themes, there are six themes built in, and that was intentional, because I want to make beautiful charts. I love charts. That was one of the first things I said. I love charts. I want to make beautiful charts. I don't want to suffer in, you know, sort of the bad place of charting. This takes arbitrary functions. This polynomial line, I need a fit. BBplot can make that. Agents know how to make it in BBplot. You can see, quite a lot of functionality. Now, I'm not going to show you every possible thing, but what I do want to show you is just one quick thing. I have this features matrix. I told you everything must be demonstrated. Features are linted for. If a new feature is introduced and that is not in this feature matrix with a one in one of these rows, it fails linting.

It can't get pushed. Here's the patterns. This is a separate concept I won't get into. So this is how we communicate to bbplot eval what has changed. So it communicates the updates via feature release notes. And now let me show you the evals. These are some of the most important plots in history. Let's take a look at them. This one is sort of like a technical one. Let's look at this one. the O-ring temperature risk review. On the left, you see the target. That's the chart that sort of, like, tells the story of the Challenger explosion. On the right is what a fresh agent with no cheating can make. That's what's possible with BB Plot out of the box. Wow. You can see stuff like Datasaurus. You can see things like our buddy Dubois.

Now, Dubois made some really beautiful art in addition to the chart. I'm a little bit more focused on just the chart. But remember, I didn't make these charts. Some random agent with very little context made them. So this is the idea. And this is how I review. And you might say, okay, well, this one looks bad. Like, what's with this giant header? Why does it look so janky? Well, that's great. Because all I do is come over to feature requests and I can see, you know, this is it going through and saying that there's issues. I can look at specific release versions and I can figure out what's wrong and I can copy his markdown. This allows me to paste it right into the other builder agent. So that workflow is very fast. And so this is the basic idea. So how do I develop it? I develop it like this. I primarily use Cursor.

I have, you know, bbplot eval on the left and bbplot on the right. Usually these are on two different monitors, but I'm currently on just my laptop. And I will talk to the agent here. I will have it browse the sort of like in-app browser, and it will sort of like present to me examples in the in-app browser. And then I will sort of iterate with it. I will make it run its evals, and we will see sort of how it does. Let me show you, you know, I know this is a skills demo, and so let me show you. I have a lot of agent skills because I have found that otherwise the agent does not behave super well. So you can see here, this is the bbplot eval, just to give you a sense. Like, run it from an editable install instead of a local checkout. Like, you never are allowed to do these things. I'm very clearly communicating. This is how it's checking it out. This is where it sort of needs to understand where to get the information.

This is it just doing some boring things, how to run the server, blah, blah, blah. You can see there's quite a lot of agent instructions, and I have some other stuff too. But this is all trying very, very hard to keep the agents honest. Because the goal is I want to be able to, like, publish all the traces. I do have a local DuckDB instance where it's publishing all the traces and sort of all of the iterations as we go. And sort of this is how I'm building out the eval data. And then similarly, if I were to come over to this repo, you can see I also have agents, and I also have agents reference for different things. And so you can see, like, quite a lot of instructions here on very specific things. I also have, like, you can see, like, I have a workflow built. I have multiple workflows, actually. And these are things that have been written by the agent because I've complained.

I said, no, you screwed up again. Make yourself a note. And then I review the note. So this is the way that I'm sort of like trying to converge to good functionality and good behavior in addition to just the output. The last thing I want to – Go on. I was just going to say something we chatted about last week is a lot of my skills at the end, I say after you run this and output is produced, ask me or the human if you want to update the skill at all in order to have it kind of continually iterative. The other thing, and people who do this will have experienced this a lot, but when you get agents to write about workflows, they will over-index on specifics and not generalities. So you need to be very mindful. Like, literally pull out file names from the current workflow so it won't generalize, for example. Yes, 100% agree. And I wish I could say, like, I wrote down these agent instructions, and they worked, and everything was happy, but that is not the case. In fact, it required an incredible number of iterations, and there's been a lot of slop.

Recently, I complained a bit on Twitter about, like, Codex being bad. This was a project that I was trying very hard to build purely in Codex, and what I learned is Codex is just, like, incapable of using a local file system and constantly breaks with image rendering. And so I completely gave up and went back to Cursor. But that's a story for another day. here you can see I'm using Opus for my daily driver now so anyway that's BB plot amazing I mean even more than like the functionality I think is like the development workflow right with the software like sort of spheres itself to like epithelial effects it's brilliant yeah thanks yeah it's very much inspired by this idea of like sort of like letting it evolve where it needs to go to pass and sort of like dropping the fundamental assumption that I think a lot of people still hold on to that they know what is good Like I don really know what good What I do know is what charts would be great to be able to produce and I can be a very very late backstop on this chart looks bad

And so those are things I do know. And so I let the rest be up to the agents. I am wondering, two questions. When do you plan on open sourcing it? There's still a fair bit to build, but I would say like if it's not done by the end of this month, I'm going to be annoyed. It's mostly a nights and weekends project. But it's also in conjunction with some things that we've been like sort of learning here at Theory around like how to work with agents in a really, really authentic way. Myself and one of my colleagues, Adam are currently working on this like sort of like blog posts and philosophy that like you can package the context with the content to make agents much more reliable. And so as we've been evolving that thesis, this is a demonstration. And so you will see that package as part of the blog post, along with two other examples, not just charts, but two other examples of working with agents that this makes them much more effective. And so I'm really excited about this blog post. Amazing. Can't wait to check it out.

And I do appreciate where over time, and Tong is going to show us his agent skills. But I am wondering, we had thought about doing this if, while Tom's presenting, you want to run the eval on a couple of figures I give you, and then you can come back and show us what BBEX says about them. And just as a little teaser, I'll share some of the images I'll share with you. Let me just make sure. so firstly Robert De Niro's Rotten Tomatoes scores I don't approve of this figure I think it's a wonderful data I don't agree with it with the scores anyway the joke is that at age 59 Robert De Niro decided I'm not going to do serious movies anymore and we had Meet the Fockers Analyze that Dirty Grandpa if anyone tells me Meet the Parents and Meet the Fockers aren't two of the great comedies of the early 21st century you know, you can get out of town.

So I'm going to give this, having said that, wow, the deer hunter and raging bull, I'm going to give that to Brian. I'm also going to give him Leonardo DiCaprio refuses to date a woman over 25. This is a dated figure, okay, because I think he is now, but it's Leo's age, and then it's actually a weird visualization to try to grok in some ways, but you get the picture. So I'm going to share those with Brian while we bring Tom on to show us his agent skills. And you'll come back and show us what BBE Val tells us. Sounds good. Sounds good. I'll be here in the background. Fantastic. So, without further ado, I would love to introduce Tomas. Before that, I do want to share... Let's try this. oh I need to stop sharing my screen and share so Tom I don't know if you're aware but we make all our guests

a short video using an agent skill I have oh no way okay cool So, the reason I did that is last time we chatted and did a podcast together, what you were doing with agents blew my mind so much. It seemed like wizardry to me, and I was really excited to have you on. Just to introduce you, Tomas is a venture capitalist who has worked with eight unicorns, predominantly in data and data infrastructure. Companies, you know, a lot of our viewers in the community know, Looker, Monte Carlo, Dreamio, Hex, Omni, Mother Duck. Wonderful blog at TomTunguj.com. and before becoming an investor, Tom Hanks was a product manager at Google,

managing a billion dollar business unit on the AdSense team. Does a lot of wonderful work in thinking about the future of technology and agents and AI and how we use it in his workflows and in everything you do at Theory with Brian as well. So great to have you on the show, Tom. And is there anything I missed out there? No, you covered it all extensively. And yeah, thrilled to be here and have the privilege of working with Brian on a daily basis. Absolutely. Well, we ask each guest three questions just to kick off it. The first is, what do you love most about working with Brian, not with agents? Well, I think the answer is true for both. I think I'm dealing with a brilliant mind, and one is a cellular mind, and the other one is a silicon mind. And I think what I really love about working with agents is there's, I think, the great power of agents is parallelization.

And in order to take advantage of that, you really have to think differently about how do you equip parallel processes with enough context and enough planning in the workflows that they actually save you time. I love that so much. And I do, because I have thought about this quite a bit and what we can learn from distributed computing challenges over several decades. And this may seem like a strange way to put it, but what can we learn from the Amish as well? That I don't know. Did they parallelize farming? Well, no, but when they build houses, there are actual wonderful videos that I used to use to tell people about how to think about distributed compute. and they're like super cuts of Amish people, like dozens of them in parallel putting up beams and they've got coordination down. Now the technology, arguable, but they've got being able to coordinate and distribute their efforts down without a doubt. Boy, that's awesome.

So the orchestration they have down, I wonder, I would love to see them pack a box. Totally. They solve the bin packing problem. Let me just... I don't mean to interrupt, but we had Amish come and lay our foundation, and we had been working on the foundation for almost a year, like, stacking these blocks, and we had made, like, three rows of progress, and we were just way behind schedule. And so we hired Amish to come do this, and in a single weekend, he did the rest, which was about 15 times as much as we had done in a year, and the rate at which they did this terrified us, and the way that they did it was very much like, you know, like assembly line, like person throwing block to next person putting it in place correctly instantly. Their mastery of working together was astounding. So this isn't quite the one I was looking for, because it doesn't show the entire thing going up, but we get a sense of how distributed efforts and parallelization, how they're doing everything as a swarm, right?

Yeah, you're right. That's a great analogy. That is really impressive. it's truly wild stuff and i mean yeah it felt ridiculous to invoke the amish when talking about ai as well but you know some things some things work um second question what's what do you find most frustrating about working with brian and agents i think the most frustrating thing i think the most frustrating thing which is now being solved by skills and its memory it's a really effective memory systems because they demonstrate such a level of intelligence and then they forget. And it's hard to remember, one, what they have in their brain. So let's say you're switching between five different tabs. I don't know. I've forgotten which tab. What is this agent doing? What is that agent doing? So there's my memory associated with it. And then you have this expectation of like, no, no, you should have remembered this. And so there I don't have a

great solution for memory and context. I've been playing around a lot with QMD, and there I'm sort of at a, I mean, I'm kind of at a loss to understand what do I put in here that is not within a skill? Like, that, you know, plug-in skills and then QMD, they don't live cleanly. They don't have clean demarcated lines of what goes in which. It makes a lot of sense, and I do. memory is such a challenge I think people who have their open claws or Hermes does it in a different way, the reason I invoke these are systems which a bunch of us have used for personal and professional stuff but we've pushed up against, they work for some time but then you really need to go in and actively manage memory and context yourself and even after a couple of months they can fall apart in a variety of ways Yeah, that's right. I mean, you were talking about too much specificity. That's a really big challenge. When you ask systems to update, they might not update everything.

So they might leave relics of previous memories in the past. You have notions of different levels of granularity of memory that become really important. And they're sort of obvious to a person and to many people within a room. But the right encoding of a memory, that is not obvious to me. It's like what, you know, it's hard to, I mean, we're doing work in understanding what's actually happening underneath these models, but we still don't have it that deep of a sense. And so I don't have an intuition for what is the right way of structuring a memory of any kind for an agent in the most effective way. Or, like, will there be a model that will ultimately encode memories in a different memory space that is unique and better for information retrieval for these systems? Yeah, well, I think if the last year is at any sign we do see harnesses, everything that that means, what we build to turn LLMs into doing things, being sucked up into models. I presume, I don't know, I presume it's like the reinforcement learning

with verifiable rewards on traces or whatever turns chain of thought prompting into quote-unquote reasoning models, than our harnesses into stuff that can code really well with 4.5 and GPT 5.2 or whatever it is last year. So it is interesting to see when we build, actually get robust memory systems, whether the models a month later will just absorb it, right? Yeah, it is also interesting to think about the amount of code that we need to develop around the model to get it to work, even with Pi, right? Even with the simplest version of a harness, you just find yourself, or I find myself just adding more and more around. And maybe that's because I'm asking more of it and I want more of my mind in it. But yeah, I hope the models can do this better over time. I love that you say that as well, because I do encourage people to think of agent harnesses as not something you take from an agent lab and use, but something that you're actively involved in the process of building and reshaping constantly.

whether it's through skills or connecting to MCP servers or your agents.md, whatever it may be. Yeah, that's right. I was at an event and a CIO asked me, at what point do you think people will bring their own agents to work? Right? Before mobile phones, I mean, mobile phones came and then all of a sudden you brought your iPhone to work and then an enterprise, what are we doing with this? Are we allowing you on the enterprise network? Are we not? Are we securing the device? Do we provide it? Do we provision it for you? And now we kind of take it for granted that, okay, whatever. You just throw on enterprise email and it works. But when the mobile phone started, BYOD, bring your own device, was a big security risk. And so this CI was asking, well, if I hire another graduate who's just recently graduated and they have spent four years embedding literally all of their education within an agent, do I allow that on my network? because the graduate will say,

well, if you want me to be as productive as I represent, I need that. And then you're going to say, whoa, whoa, whoa, what is this thing doing? And so there's an interesting tension to resolve there. And do you have a different, you know, some people have different cell phones for work and personal. Do you have a different agent for these two different things? I think you'll need to. I think it's so difficult to secure that you will probably have to somehow bootstrap an agent inside of it? And then does that become the intellectual property of the enterprise? So while you're working at Theory and you work here for four years, does your agent actually become and is it owned by Theory or do you have intellectual property rights to it? Do you license? I mean, all kinds of interesting questions to explore. I do see Brian's hand up. I will say one thing. First, I do think the adoption of like mobile cell phones, you know, there was some corporate branding that helped with that as well. I think, you know, the introduction of the BlackBerry for example, is something which brought cell phones en masse to certain verticals.

And, of course, then the iPhone was something very different. But it's that type of, I think, situation which can help with adoption. Brian, please. I was just going to add, yeah, like leaving a job and going to the next one and being like, I have to leave all of my agent contacts behind feels already starting to feel frightening. I'm like, man, I don't know, it bakes so much in. I don't know. Also, because it is your second brain in a variety of ways, right? Like, what do I... I mean, there's a reason that there are people rotating around Frontier Labs. There are reasons why certain levels of software developers went from Apple to Google to Meta to Apple to Google to Meta. In order to get raises, I think, was the actual reason. But that's dispersion or diffusion of knowledge. So being able to take something, I think, is super important. Okay, final third question before we dive in. If all agent conversations got leaked publicly, what would you be most concerned or embarrassed about becoming public?

Oh, that's a good question. I think I would be most embarrassed by the very basic programming problems I ask my agents to solve for me, like the level of either laziness or lack of knowledge. I think you can project a lot of sophistication, and you can talk about different models and harnesses. And then I was wrestling with – I was vibe coding a Super Whisper competitor, a Whisper Flow competitor today. And it took me 45 minutes to figure out the accessibility permissions so that it could copy and paste whatever had written into the – it's like 20 bucks on codex. Oh, my gosh. I mean, you know, if I knew any kind of Mac OS program, I could have solved that in 30 seconds because all you have to do is just sign the app properly. But, like, okay, I don't know what that is. Right? So I think that would be, like, how base my level of programming has become.

I love it. I love it. And Brian's just written in the chat, PhD required to understand OS X settings menus. It's so true. Well, Tom would love for you to show us what you're up to with agents at the moment. Sure. So I am going to demo a skill that I use to analyze publicly traded companies. And it does a couple of different things. Okay. Can you guys see that? Oh, it's small. I'll see if I can. Yeah, that's great. Does that work? Okay, so I live in Pi, and I run local models. And I love QAN 35BA3B. And I am going to run a skill in Pi, which is public company analysis for Figma. And Figma just announced earnings today, and they crushed earnings. Way to go, Dylan and team.

and what this is going to do is it is going to pull down the csvs from a data source it's already been done which is why i said that put it into a folder it is going to find the earnings transcript using exa and then it is going to use a series of our libraries to generate charts not nearly as pretty as brian's and then dynamically assemble a presentation to architect a narrative about what has happened in Figma's business. So that instead of reading a whole bunch of different things and understanding the financial performance of a business, I can have a single presentation that walks me through exactly what's happened in Figma. That's the idea. And then it generates an HTML. So while we were talking before, I had the oven on, and I will show you what that looks like. Also wonderfully modern of you to be generating HTML, Tom.

And I actually mean that, right? Like, currently, we've been generating Markdown for a long time, and I've been getting a bunch more HTMLs recently, and then suddenly, of course, Tariq at Anthropic posts about it, a few other people do, and we've kind of rediscovered HTML as a great way for agents to present information to us, right? I agree. It's really good. It's really hard to share privately, which is kind of a bummer. So you have to generate a PDF. And there, again, going back to my base programming skills, I have not figured out a way of generating a presentation that renders well as a PDF because it generates all the, well, anyway, whatever the margins are not properly managed. But this is an example. So, you know, I think within two and a half minutes running on the local model, I can get a sense of exactly what happened in the business, right? So revenue grows up 41%. Cash is where net dollar retention is an exceptional metric. And you can see they crushed. there are quotes at the bottom from the quarterly earnings reports. And so you can just kind of go through exactly what's happened in the business.

I think the, you know, you have a quote here from Dylan talking about how AI is really important for the business. There are some bugs, clearly. You can see here, like, you know, there are two charts and this is not rendering properly. And so I need to go and use Hugo what you were talking about, which is a skill to continuously improve. But in terms of understanding, you know, overall guidance or liquidity, this works really well. And then what I can do is run this on a launch team on my computer. So every morning it checks, okay, which publicly traded companies have announced earnings. And so, you know, I can get a quick briefing on exactly what's happened in that business in a really consumable way. that's incredible and how can i ask i mean this clearly is something that's been iterated on how how many how long did it take you to get it to to be able to do do something this sophisticated it it took about well so the r charting was something i had done like um created style

sheets and lots of examples of other analysis in the past but then pulling together the different data sets, it takes maybe three or four hours to get it into a place like this. But to give it an example of a CSS that it really likes, just to make sure to choose the right JavaScript library, and then connect all the different data sources. I'm also interested in how dogmatic are you about what it uses, giving it up-to-date sources on the packages it's using, and yeah, all of these. That's a really good question. And so as it was installing NPM packages earlier today, I panicked because I haven't pinned my NPM packages. And the amount of supply chain attacks is pretty significant. And so I need to, you know, I don't like this would be a great question of where does that knowledge live within an agent? Like, is that should that live within a skill? Does that live within a plug in? Does that live within QMD? this liveinagents.md. But like, you know, a really reasonable proxy

for secure NPM packages is don't install anything that's newer than 14 days. And great heuristic. I don't know where... Anyway, so I was a little nervous about it. I think I tested a bunch of different JavaScript libraries to get to a place where the formatting was really great. D3 actually is amazing at this. So these little models can program D3, particularly for workflow charts or any kinds of diagrams. And then that is very simple to generate into a PDF or a PNG if you need it. But right now, I think this is present.js, and this is the best library I've found for customization of software. Awesome. And we saw you were using Pi. I may have missed what model you're using. I'm using Quint 3.635 billion. Okay, cool. Have you had success with, I think last time we spoke, I don't know if we talked about this on the podcast, but we're talking about smaller Gemma models as well. I don't know. Do you use like 7B or even 2B for anything?

Yes. So I've coded a Super Whisper alternative that uses Parakeet locally running on Swift LLM. And then it uses an actually Gemma for E4B. and you can, if you put that on the Swift LLM, you can reduce your latency from 3,000 milliseconds to 300 milliseconds running locally, so a 90% reduction for both ASR and then post-processing with a prompt. And so, yeah, I mean, I think the local models are broadly, like QN3635 can do some, like, I mean, the coding that, But anyway, it can do some pretty sophisticated coding within a particular file, and it can do almost any single tool calling that you want. I've played around with a lot of the Gemma models. And candidly, I have a really hard time getting the Gemma models to follow long-term tool following, which has been tough.

And the initial release, a lot of the tool calling parsers were broken in many of the different inference engines. Like MLX, it was broken. Lama CPP, it was broken. And so I need to go back and test it again because supposedly, like the Gemma 4, the sparse one, is supposed to be excellent. It's just kind of – I really like Quen. It works really well. Do you run these models locally or remotely? Yeah. You can get – I mean, like Quen 3.5, like the 35 billion parameter model, you can get – I'm on a Mac M5. I can get 120 to 140 tokens per second. with a 256k context window. And just to give you a sense of what that looks like, oh, let me change my window. It's faster than cloud. And so then you start using cloud code, and you realize, okay, well, let's see, let me share something else. You realize how much you're waiting for,

you realize how much you're waiting for so you know here's pi you know tell me the weather it's really fast right or like here summarize the news on Hacker News what are the top five articles do you do that for security reasons I do it for convenience I mean, I was on a plane, and I was getting on a plane going to Atlanta, and they said, there's no internet. What are you going to do? And so, like, I had to download local models as quickly as possible. You know, and there are other benefits of using local models. Like, if you're dealing with secrets, you have no fear of pacing in a secret because it's not going anywhere. The other reasons I really like local models, I mean, like, yeah, latency is a really big one. And then the tool calling accuracy is really good. and the UX within the terminal is awesome

because it's really thin. So the only reason I go to cloud models are large-scale coding challenges, particularly multi-files or re-architectures or bugs that simply can't be solved. Or, you know what's really interesting is Kimi K26 is the best creative writer by a country mile. And so some of them have particular nuances that are different. and so I hit a cloud model for that. I think you're the first guest to come on that actually uses local models for the majority of the workflow. It's really interesting. I think this is where it's going. It's just like, I mean, and you can run five of them at once because now the inference servers are really quite strong, and you're seeing, you know, a lot of the reasons that local models are slow is because the middleware is primarily built in Python, but now LLAM is primarily written in Go, So you've seen a 40% to a 50% speed up.

You have TurboQuant where it's like dumping the disk. And then even like draft models make it a whole lot faster. You're starting to see acceptance rates between the really small models and the medium-sized models go really fast. There's one more meaningful innovation. Maybe it's Delta attention. Anyway, you can see like even with like a 4-bit or 8-bit quantization, you can see really tremendous tokens per second. where you're generating 60 to 120 tokens per second. You're not waiting. If you're launching these to a Lama, you mentioned. Yeah, so that's the backend server. This may seem like an almost naive or silly question compared to all of the sophistication of these wonderful techniques and open models. I am interested, when you ask it to tell the weather, what is in the model or the harness or your agents.md that got it to just curl the weather today? The agents.md is really thin.

Let's see. Show me a summarized version of my agents.md for Pi. That's it, right? So it knows about some skills. It knows about a couple of workflows. There's one thing about using the Gmail API. We have Brian and his team have built an MCP for us. And then there's a QMD lookup. Interesting. So it's really thin. The other thing that's really important locally is making sure that whatever harness you're using isn't stuffing the context. So there are certain harnesses where, I mean, you can see, like, this context window is 256,000 tokens. But certain harnesses, and we're only at 6.2%. certain harnesses right out of the gate you'll have 25 or 40,000 tokens as part of the system prompts and there you really have a hard time getting the tokens per second to a place where

it feels interactive and so that's one of the reasons I like Pi you can literally strip almost to the bone and it'll still work yeah I mean Pi's system prompt I'm actually trying to find it I'll I'll share the link in Discord afterwards. And Mario, Zekno, who created Pi, has actually got a blog post, and I think Armand does around system prompts and harnesses. And it has a wonderfully small, concise system prompt. Yeah, I think it's 2,000 or 3,000 tokens. Yeah. Yeah. And fun fact, you can actually, I don't suggest you do it, but you can build pretty robust agents without even having a system prompt at all, if you have good tool definitions and names. The other thing that I love about Pi is it passively saves your sessions. And so then you can ask this model, as I did earlier today, I was working on this particular thing. Can you run ripgrap over the Pi sessions for the last six hours and find where we were and then pick it up?

And that's awesome. Fantastic. Yeah. The other thing I really love about it is it's minimal. It's got four tools, read, write, edit, bash, and it will, the whole philosophy is if it needs something, it celebrates the idea of creating it in code itself and it writes code to do it. It will write its own tools and hot reload them so you can use them in the same session. And I actually, I linked in Discord to a workshop I did with Ivan Leo who built at Manus, built agents at Manus now. He's at DeepMind, but it's a workshop in which we built like a minimal version of Pi in Python and then added, because it can type script right, built it in Python and then added some OpenClaw-like functionality to it. So sandboxing and putting it on Telegram, like being able to chat with it on Telegram and putting it up on modal as well. That's awesome. So super fun stuff. I think it's probably time to wrap up, and I'm wondering if there's anything else you wanted to share or chat about, Tom. No, no, that's it. We covered my materials. Thank you. Super cool. I would love, Brian, maybe,

if you want to come back and show us what BBEvals said about the database. All right, here we go. So, yeah, I think, you know, we'll start by just showing you sort of the results. So this is where it got on its first attempt. You can see it says candidate V1. This has not been promoted because it doesn't think that it was good enough. and so the evaluation criteria were not satisfied, so it doesn't promote this. You can see that it has picked up some things that it thinks are important and it has communicated that through the chart and it has gotten some of the key ideas. You can also see that there's some interesting things going on here. The data looks a little bit different. Also, you know, obviously there's some stylistic changes.

But if you look here, you know, on the age axis, it goes out a little bit further now. And if you look at the age axis over here, it kind of ends at 70. So it really should be truncating and so on. But let's just draw attention to a few things. On the right is my chat. And so I'll just show you what I did. the first prompt was I sent it the image I have a new chart I'd like you to add to our benchmark evals I'll first ask you to find the data and then let's see how well we can recreate it with bbplot that's it and I can scroll and you can keep me honest I'll stop when I get to be prompting again yes I'm ready to see the attempt and if I keep scrolling you can see that it generated the first attempt and it automatically sort of like, you know, no shaded backgrounds, no curved arrow annotations, the hand-drawn aesthetic, and the label crowding.

It thinks that the labels are too crowded. It knows because it's getting warnings from SceneGraph. And then I said, can you show it to me in the built-in browser? Because I wanted to be able to demonstrate a little bit easier. And you can see it's getting ready to do that. And then show me the comparison page in particular, because, again, I don't want to click anything. And that's it. and then I wanted to start giving it some feedback and I said, hey, isn't there an example in the gallery that shows the shaded regions? And it says, yeah, it turns out that's possible. And so now it's going to go ahead and try to fix that. And so there you go. Yeah, I'd never seen this chart before. I mean, I think I saw it like a long time ago, but I had never shown it to BB Plot or BB Plot Eval and so that's what it's like to build BB Plot. Incredibly cool. Well, thank you so much, Brian, and for humoring me with a plot about Robert De Niro's rotten tomato scores as well. It looks like maybe he's recovering. You know, I know you were worried, but, like, if you look over here, it looks grim.

But if you look over here, you see some, yeah, I think he might be coming back. Yeah, that must be his, like, Scorsese collaborations, again, with The Irishman and Killers of the Flower Moon. exactly you can see you like the last my concern was his movie choices my concern was that Rotten Tomatoes just doesn't appreciate Meet the Parents right understood great well I think it's time to wrap up I'd love to thank you for joining Tom for joining of course Eric and Hilary earlier and my co-host with the most, Thomas, as well. And thank you all for joining and watching and all the great questions in Discord. We've got our European special next week where Nico from AMP is going to join. The creators of Spacey and Explosion AI are going to join. Eleanor Berger from Elite AI Assisted Coding,

Vincent Varmadam, Alan Nickel from Raza and perhaps a couple of other special guests. so it might be an epic session once again, check out our Luma calendar, hit like and subscribe, share with a friend, all of those things, and cannot wait to see you all in the next episode. Thanks once again, everyone. Thanks for tuning in, everybody, and thanks for sticking around until the end of the episode. I would honestly love to hear from you about what resonates with you in the show, what doesn't, and anybody you'd like to hear me speak with, along with topics you'd like to hear more about. The best way to let me know is currently on LinkedIn and I'll put my profile in the show notes. Thanks once again and see you in the next episode.

Thank you.