スタンフォードが教えるAIネイティブ・エンジニアリング — Mihail Eric（Monaco）

Okay, this is a pretty special episode because we're celebrating 1 million subscribers on the AI Engineer YouTube channel. So go over there and subscribe if you haven't heard about it. The guest is Mihail Eric, who is not only teaching the monosophage developer at Stanford, which is going viral right now for basically covering all the AI native development best practices, but also he is head of AI in Monaco. So he's not only a teacher, but he's a practitioner. And those of you coming to AIE Code Summit next week, it's going to be the celebration of all the extreme progress that you've seen in coding in the last year. I'm excited about all the themes of fighting slop, of system one models, and of increasing your ambition for major rewrites of basically everything that you've seen in code. There's never been a more exciting time to be an AI engineer. And with one million AI engineers, I think it's just getting started. So let's get into the interview. Okay, we're in the studio with Michael, Eric, a professor at Stanford and head of AI at Monaco. Welcome. Thank you. I'm glad to be here. You caught our attention because of your course

that you are refreshing for the third time? Second time. Second time. The CS146S, the modern software developer at Stanford. This particular generation of junior developers, of junior engineers, the people that are now entering the workforce will I think be the first kind of generation of that new shift. And I looked at the syllabus and I was like, this is everything we cover. So every now and then I like to invite people like yourself who like you cover a broad range of topics. And then we can sort of go topic by topic and give people the state of the art on each of these things, which I assume you're doing with all your guests anyway. Tell us the history of the course. What made you want to spring out CS146? Yes, this is the second time it's being taught. And I taught it for the first time last fall. So fall 2025. This was pre like Opus 4.5, 4.6 days. But even before fall 2025, the course had been about like a year in the making. The genesis of the class was probably November of 2024. And I had sort of had this idea that we're seeing all this stuff happening in coding agents. And, yes, they were an earlier version of the coding agents.

But already it felt like there was something drastically changing in terms of how developers were doing their work, how quickly we were getting work done, all that fun stuff. And so at the time, it was all just scattered blog posts, scattered, like, things you could find online if you were trying to really get into the methodology of how to do these things. And so I had this idea, like, hey, well, is there a way to kind of structure a curriculum around this that's a little bit more formal? I had the idea, and, you know, my alma mater is Stanford, and I always had this dream to go back and teach something and kind of pay it forward. I've always been very deep into education. I talked to Chip, who also taught a class. Chip, you did. Yeah, Chip, exactly. I spoke to her. She had taught her, I guess, designing machine learning systems. And so I asked her what that process was like and sort of took a lot of inspiration from how she designed the curriculum and thought about getting something through the school. Put it together, spent about nine to ten months. This was now fall of 2024 through to mid to late of 2025. Got it through the committee, made a case for the class to exist, had to put together a proposal and everything.

And that took me to fall 2025 when I had it for the first time, which was very well received. Students were really interested in it. There was even, I guess, like a Business Insider article that was written about the class last year. And then, yeah, this year a lot of things changed. I would say really coding agents, autonomous coding agents, really picked up in a way that we had not seen before. So it felt like it was time to do this again, refresh things, and then here we are. Meanwhile, you're also head of AI at Monaco, which is also a very buzzy startup. Yes, yes, we are. We're in the airwaves right now and having a fun time doing it. Yeah. So I do like that you're also, I guess, a practitioner. You guys are obviously implementing all this stuff at Monaco. Any, I guess, principles or observations about just what you have changed from last year to this year at a high level? Yeah, yeah. I mean, a ton. I think that when I reflect on what last year's version of the class was, I mean, coding agents had been around, but we were still in kind of that phase where companies were emerging that were tackling every single part of the AIS DLC.

There was a coding agent for CodeReview. There was a coding agent for spec design. There was a coding agent for SRE. And, of course, some of those companies still exist. But it felt like last year it was a lot more about a broad tour of here's everywhere you can use coding agents. And in a way, a lot of the curriculum last year was geared toward exposure. and this kind of survey, this kind of like odyssey of sorts through, hey, everything that you knew about the ASDLC is now changing because of AI. This year, I think there's some principles that are solidifying, right? The idea of like very careful context engineering, the idea of like really thinking through sort of like design and architecture as a way of handling really complex tasks. And so the nature of my class is it's not intended for the like beginner programmer. It is not a replacement for like traditional computer science in the sense of like algorithms. And, in fact, a lot of those classes are prereqs to what I'm doing here. But I do know that if you have a good foundation, going now toward applying that and sort of using those principles in production environments in the real world, that's something that I think is now at a place where, you know,

at Monaco we deal with massive code bases. You know, I help a lot of companies also with thinking through their massive code base designs and things like that. And so really it's more of a practitioner's course for once the students have graduated or now entering the workforce, how to be the most productive members of the workforce. You said you're passionate about education. I feel like students in that phase of being in college are in a pretty interesting spot. Any advice and any take? Should people study computer science right now? Is cloud code and codex all we need? I want to hear your thoughts. Yeah, there's a lot of anxiety from students, as you can probably imagine. And I think a lot of what I'm trying to communicate this time around is that the anxiety, a lot of it still is here. And there was anxiety last year as well. But I want to make the case that a well-trained AI native engineer, and I know that's like a buzzy word, but we'll dive into maybe what assets that are most interesting. A well-trained and native engineer can be some of the most useful assets like any company around.

And so this is the first week of class, actually. And week one, first lecture, hey, I really believe that every company needs you, but the floor is much higher. So you're entering as your first job out of school, but it's not sufficient to just be an L3 and a Google term if you're going to use that kind of leveling system there. So you have to generally be better than almost at the senior level going into this. So, like, what can we, how do we get there more quickly, right? And so I'm trying to say that there is still a core bit of, like, foundational software engineering that is important to have, some of these core values of sort of how software systems are built and the trade-offs of those systems. But then beyond that, we go and, you know, that's where I think you as a human should still be valuable rather than, and the rest of the implementation details you can kind of defer more often. Yeah, this is something I've said before, which is almost like the controversy. it's like the kind of thing you say to be spicy but I almost I do genuinely kind of believe it which is most students are cooked, some of them are going to be super valuable

I don't think it's a very popular thing Stanford's kids are going to be fine right, they're from Stanford but like what happens to like the rest of America that is an open question it's a weird situation because I feel like a lot of education is slow to move I'm even curious in how how do you determine where do you put the end date to curriculum in a course oh my god you're teaching a stanford modern software like that's changing everything you cheat you just get the people from the the coding agents to come and speak right oh yeah like yeah i have to be there yeah that's a funny well yeah i mean you know that was uh okay so so on on the guest lectures point i think that was um i mean that was also last year um that was also the case so that was by design because I came in, right, and I'm saying I'm a practitioner, as you pointed out, I'm a practitioner, I do this thing on my day-to-day, I design these systems, we build these things in Monaco and have seen that many times before. And so that being said, I don't want to claim that I have all the answers today. And that would be a lot of hubris to make

that claim. So the structure of the class was always, here's one practitioner's sort of aggregated set of insights from working in the field, from talking to a lot of people in the field. But then there's people that are literally on the front here, seeing things even beyond that, and invite them to sort of put their proposals forward, their view of the world forward. And that's why the guest lecture is there. I think it's really impactful for students to see big names, people actually doing it, hear from them directly, be able to ask them. Of course, they want to intern. They want to get jobs. It's good to have guest lectures. No, I think what he's saying, so, okay, I'll sort of rephrase it. So, okay, we've also interviewed Ange, who teaches the Coachella course, right? Which is like, here's all the names, it's only the names, right? Whereas he's actually also producing, here's cloud code and 200 lines of code, right? Which is, no one's telling you to do that, no one's going to come see you to do that. But yeah, when you want to teach fundamentals in a framework that you can use in a working understanding,

this is the kind of thing you do. That's right, yeah, exactly. And I think there's a balance for this kind of material, to your point, is that there is a, you can't, it's not 100% solidified. I mean, by design, it's a feature and maybe a bug of some kind, but a big portion of the curriculum did change from last year, right? And so there's like a dynamism to the whole development of the curriculum. But I'm trying to at least put forward that there are some things that I believe are kind of sticking around. And, yes, prompting coding agents, they've gotten smarter. What does that mean in terms of the new ways of dealing with them? but then still Boris is also going to know something that I don't know. So let's have him also come to the table and share his learnings. One of the fun things I noticed this time around is a good bit of the portion is on open source contributions. What's the thinking behind that? You have people. Yeah, yeah. This is one of the things that I'm honestly kind of most proud of and most excited about. I mean, it's about to be week two of the class. Yeah, so this is the very gracious open source partners that I spent the greater part of this summer just kind of talking to and getting them kind of excited about and interested in sort of this offering.

Basically, the idea was, so last year when I taught the class, we had like nine assignments, and then we had like a capstone open-ended project, right? The nine assignments were basically me sort of like coming up with these pedagogical exercises, like, oh, here's a repo, like use an AI code review tool to, you know, post a code review and then see why is it good, why is it not good. So relatively pedagogical, honestly, pretty easy. I think for a lot of folks, they could have just one-shot it, and I thought, okay, this is not that helpful. At this point, agents are round. I mean, they have every student that I surveyed on Tuesday. Not a single person really types by hand anymore. They're like, everyone's using coding agents. And this is all sophomores, juniors at Stanford. Do you survey them with the market share? What tools do they use? Okay, I haven't yet, but that's a good thought. You should do a lot. Yeah, yeah. I will do that. spicy, Stanford prefers. No, no. Well, I think through the university, I think they have some, like most of the providers have an educational program, right? Yeah, they're very generous on plans for students. Right, right.

So I think by default, even before the class, there's like, you know, a CLAW license, I imagine cursor license. There's like probably a cognition license as well. I don't know, you might know that better than I do. But what I was able to also just accrue on top of that was, hey, like, if I don't want people to be borrowing, like by like the $20 a month subscription, Like, I want you guys to go, if you want to, like, token max something just to see what's possible, then you should. And so, and I'll come back to the partners, but the additional thing I was able to accrue through instead of talking to folks was, like, additional credits for students. So, like, factories also are getting involved. The cognition folks are getting involved. OpenAI as well. But anyway, so for the open source partners, I sort of, like, was like, okay, one of the things I'm most concerned about when it comes to educate the next kind of generation of students is that if everyone is saying, kind of, to this point, like, they're cooked. Like, you know, what's going to happen to them? So there's almost like this sense of them being left behind. And I think that mentorship is a really important kind of piece of just up-leveling everyone and making sure that there isn't this massive gap that occurs between the senior people that have already been in the industry before coding agents and know what's up,

and they adopt coding agents and they become amazing, and then the juniors that are just like, okay, now what do we do? So in a sense, one of the kind of goals was, hey, coding agents are really good. I want to put you guys in situations where you're dealing with complex code bases, large code bases at scale. We're going to work with some of what I think are among the best AI native repos that all do AI-focused things, or either coding agents themselves, or building tools in the service of agents in the modern world. I want you guys to get involved with these projects. I want part of your assignments to be actually contributing to these projects. In the course of being at all those projects, you will actually be talking to the maintainers. They will be reviewing your code. They will be helping you build mental models of good software. In that way, there's effectively this passing on the guard of true good software design in a way that I don't think most students get the opportunity to do. And then there's the benefit, of course, that I'm sure some of them will get poached and picked up as part of doing well in those assignments. Yeah, one of the interesting things, for what it's worth, I'm going to get a bunch of hate if I don't say this. I think you say cooked versus cracked for dramatic effect.

But, you know, it's just nice to facilitate extremes. Sure, sure. So maybe we should go into the sort of detail of the curriculum. or is there anything else you want to explore at the top level? I mean, that's very helpful. I have high-level questions about students using tools, best practices. Are you pro it? Individual tutors. I'm assuming this will also be picked up by students. So before we dive straight into the course, just anything you have guidance on there for? Like in terms of what? Yeah, you said you're passionate about education, right? Like, there's also a stigma of one side is how do teachers, professors use it? And then how can students best use these tools? That's not just throw your assignments in. It can kind of one shot it. How do you still learn? Like, if you have anything there. Yeah, yeah. And I think that there is, I still have some assignments thrown in there, which I think are not intended to be the hardest things in the world, but are meant to be more valid concepts, like more conceptual things. Like, you know, what does a skill mean?

sort of like maybe more of the tactical of some of these tools, skills, like agent-ready code bases, like agents MDs, things like this. So there's a certain amount of just, hey, if you haven't heard these terms, kind of get up to speed on them. I do think most of the learning, though, in terms of the kind of putting the tools to the test, will happen both in the OSS and then also in the final project, where it's very open-ended, and they'll have to just kind of explore something interesting. Cool. Yeah. So maybe, yeah, I just want to talk about, well, let's go through one, two, three, four, five, two, ten. and then we'll dive in deeper. So like breath first, basically. Sure, sure, yeah. So we start with internals of coding agents. You teach people how to write their own cloud code clone. It's very helpful that it's not open source. And then? Yeah, yeah. So week one, it's all internals of harnesses, you know, kind of build a coding agent from scratch. Week two is more, okay, now assume that there's, how do we start thinking about the context engineering at scale? So beyond just vanilla prompting, and then I think coupled with that is just

in doing that, thinking about tool design, MCP, people, I think for a while MCP was getting a lot of hate. I'm still pro-MCP. I see it all the time and so I think that there's, at this point it's a standard that's existed and I think it will continue to exist. So talking about that, effective tool design, that's week two. Week three I would say is more about just like skills and what does it mean to extend beyond simple, single purpose tools to more like procedures and that's where we'll talk about skills and that kind of starts leading into the automation of agents. Week four is a lot of just more of the foundational pieces of how can you customize an agent. This is things like agent config files, a la agentsnb, quadmd, hooks, kind of sub-agents. There's a lot of mechanisms that exist in modern harnesses, which you can use for different purposes. I want to expose students to that. Agent-ready code bases, this is probably one of my favorite topics. It's like, and this is especially relevant for folks that are going into industry and thinking about code at scale, is essentially what does it mean to have a repo that's ready to be ingested, consumed effectively by an agent.

And so thinking a lot about the tradeoffs of designing of kind of service level boundaries, but also docs and verifiability and things of that nature. So that's week five. Yeah, week six, sort of a genetic code review. and here I think what is interesting is thinking also kind of extending from week five around code bases, like agent-ready code bases. Going to week six is more about like, okay, in order to, like code review is now becoming a bottleneck, which I don't know that students, unless they've, you know, have done a lot of, some of them are master students, they've been in the industry, but a lot of them have not seen this. But for most people that are using code agents today, we're all experiencing something of a new bottleneck in code review. So how do we unblock that or sort of unbottle the bottleneck a little bit? what mechanisms are there there, and so verifiability and things of that nature and how we can extend that into the code review process. Week seven is security. I mean, obviously, I think there was a point. So this was in last year's class as well, and this I wanted to reinforce. I mean, with any autonomous system taking a lot of actions, we're seeing a lot of, this year especially, a lot of things getting hacked,

not least of which are just some of the core pieces of software from Vercel to gateways, LM gateways entirely. So I want to spend some time just, like, talking about what new TAC vectors exist in security and, you know, how we can mitigate some of them. Week eight is background agents. So this is what I want to start touching on here is, you know, how do you, like, basically run agents in many different, like, product service areas, not just in what was traditionally an IDE, but also having things that are, like, based on triggers in your code base and in your repo, during CICD, things that get triggered even when, you know, based on events, things like SRE-style bots. How do you even build a system? I think this past year was when a lot of companies started releasing their posts, like Ramp Inspect. Spotify has their own version of this. I think Stripe Minions, all these folks were doing really cool write-ups of their internal infra they were building to do this. I want to touch on that, get people exposed to what that is.

yeah this is um yeah this one is uh more on like what happens when you're in in a team in in the modern world and sort of like the core pieces of of team level infrastructure so everything from uh LM gateways to you know NCP portals which is like NCP at scale and then just also how to think about the evolving roles I would say so this is maybe a little bit like a soft thing it's like what what what is now you know and the the vanilla may be like not super nuanced thing to say it's like everyone's a builder, but I want to go a little bit deeper beyond that and sort of like what thing is actually getting subsumed? What roles are actually either evolving or merging that didn't once exist? What should students think about in terms of defensible skill set that maybe wasn't true before? And so a lot more like team dynamic type considerations. And then, yeah, this is, it all kind of builds up toward the factory, which I think is top of mind for now, for a lot of folks today. Once we're building factories, where do humans still exist and all that? What are the new choke points that are intentionally there around human involvement?

Things like intent, things like setting up a verifiable systems, and what happens in the limit when maybe 89% of code is fully driven by just agents, where we kind of fill into all that. Nice solid 10 weeks. I was going to say, some of these are more established patterns than others. and I think people need to know that not all of this is settled science. Sure. But it is everything, I'll endorse everything as things that people talk about and you should know. We know the open problems, we may not know the answers. That's absolutely right. Yeah. I mean, you probably have your own version of this. I'd be curious to hear what you think is still not as defensible or not defensible but not as a set in stone. But even like the software factory, are we talking about dark factories or are we talking about these sort of half-lights-on factories? I think even there's a spectrum there. But, yeah, in there, you know, there's the people that are on the very frontier, like the warps and the factories that are, like, building factories and selling them services. And, you know, even they will only claim, like, 30-some percent,

if even that, of, like, fully, you know, gentically driven contributions. So what does that mean in the limit? It is hard to say. And there's people who went very, very hard-dunk factory and then came out saying, I was wrong, guys. Yeah. Right? So, like, we covered that on AIE. Just be aware that, you know, not everyone, And they probably believe the things they're saying at the time, but they have the right to change their mind, and they do. And even Boris recently tweeted that, yeah, I am often wrong. And everyone is like, because he created a clock code, it's like treating his word as gospel. And I'm like, no, well, sometimes he disagrees with himself. Yeah, yeah. And that's okay. I mean, this isn't politics, right? This is where we're still scientists. We're engineers. We're subject to, I don't claim to be dogmatic, but I just want people to experiment. I think that's like a principle as well. you as a principal earlier, just like experimentation. I mean, like you just have to be ready to, and oftentimes when I sort of like give sort of these lectures or talk to you about this stuff, they're like, how do I, there's so much going on. How do I stay on top of this? And it's like, you know, there's something simple to just like experiment for 10 minutes a day. Like just spend 10 minutes doing something

that you didn't do before. If that's a traditional way that you were solving a problem, there's now a new tool that you want to try or a new like framework or something. And just spending, maybe even doing the exact same problem just with that new kind of thing is, alone will cover most of the gap. Yeah. I will call out, so some of your, for example, you have Cognition on the week that you're talking about code review. Yeah. I actually did work with Silas on Devon review. Yeah, yeah. Pretty good, I would say. It's hard to benchmark these things. Sure. There is a review bench. No one uses it. But qualitatively, Devon review is pretty good. Well, we use you guys. That's also, yeah, so Monica does use Devon. Oh, and that's, I mean, Silas actually spoke last year, and so Silas is great. It's just super, super smart. But, yeah, we use you guys. And so I was like, okay, among the providers of kind of code review tools that we – and we sampled a lot of them. We actually did kind of converge on Devon for a lot of these cases. Silo is actually more model-side these days, and then actually it's Walden that you want. Yeah, yeah, yeah. I was talking to Walden as well. I will also shout out Eno from Factory because, you know, talking about agent readiness, they've been all about that.

They build signals, tooling on that. Do you guys use that in your opinions? Yeah, so we do use agent readiness of our own kind. I mean, Eno is awesome as well. I think they were super early on this train. And I think when he codified that in his talk, I guess, at AIE, what was it, like last, I forget when it was. But one of his talks. Yeah, Europe. Just fantastic talk. And I think it was kind of way ahead of its time. And so I was like, there's only one person I could see being like the most appropriate person to talk about this. Where I really wanted to give him the spotlight. And they're super excited. Yeah, it was funny. I was at college. We were talking about this stuff. And we were like, okay, we need to form some opinions about it. And then, like, factually, like, shipped a full product on it that week that we were talking about it. I was like, okay, you won. Yeah, yeah. I mean, like, everyone kind of agrees on the same thing. Sure, sure, sure. And it's just a question of, like, do you want to turn it into a dashboard? Yeah, yeah, yeah, yeah, yeah. Because it's a dashboard. Right, right, right, right. Right, right. But, you know, there's value. And, like, I think they did well from a marketing perspective jumping on, like, the writing report. And I've definitely spoken about it. You know, I will regularly kind of pitch that to other people and say, like, yeah, you should run writing reports of your own kind. And, yeah, of course, some people have open source versions of it.

But I think the principles are super important. Yeah. Do you want to go piece by piece, or do you have any other things? I can go piece by piece. Yeah, okay. So the first one, right? You focus on cloud code, but then you also apply it, the open source sponsor. Yeah. What's up? Okay, so this one is maybe a little bit more marketing-y, but okay. The one thing I do want to highlight, and I've made that clear last year, but I'm not endorsing any tool, okay? Like, I don't, I think that is the wrong way to approach something like this. There are tools that come and go, and when, you know, transparently, when the classes taught last year, Cloud Code was not as popular. It wasn't as readily accepted. Boris was only at 4,000 Twitter followers, and now he's at whatever million. I don't even know if he is, but he's at a lot of followers. So Cloud Code has really emerged on the scene. I think we also underestimated Cloud Code when we had Boris and Kat on the podcast. I was like two months old. I used it once. A query cost me 50 cents. I was like, who's going to use this script? It was still in beta for a while,

but it's good on them for how they built it before the model was ready. Yes. Like you hear so much build for where the model will go. Totally. Super ambitious. Like models were not that good. You couldn't feel it. And then, ah, we see that. Yeah, it's funny because when he gave his talk last year, I remember it was like a very high level sort of set of slides, but I remember one of the slides was literally built for where the model is going to be six months from now. And this was like October of last year. And I can see why he would have made that point if he knew in six months where like Opus was going to be because that was also what I think was a huge unlock where I would argue was one of the reasons why the curriculum changed as much as it did, right, where there was this huge arms race around true autonomy in these agents, like long-running agentic coding agent tasks. But, yeah, so back to your point, six. I'm not endorsing CloudCode. I mean, CloudCode, of course, I use it and use other things as well, but here I was more about what does a modern coding harness look like. I wrote a blog post that actually, yeah, this one, where this was earlier this year, where I sort of broke down some similar principles.

The TLDR was that, you know, you could sort of, through, like, pretty much introductory algorithms-level understanding, you could just find, like, essentially an agentic loop that does the bulk of what you would call a coding agent. And, of course, there's, like, bells and whistles. There's no sub-agents. There's an MTP and none of that stuff. But if you want to see something that you talk to and then it literally will edit your file and you can do pretty sophisticated things with it, it doesn't require more than 200 lines. So, you know, a lot of what I've tried to do in education, to try to peel away the stuff that's like the secret sauce and show that really at its core, I mean, these are still students, and you want to teach conceptual principles. There's nothing more to this than, in Stanford, there's a class called 106B. It's like the introductory algorithms. You don't need more than 106B to build something like this. So there you go. There it is. Given that you already taught this class, any pushback, any questions, any weirdness that you observed from the students? What do they not get? What do they ask? It seems to cluster questions around. You mean like now that this particular concept or just in general? This concept.

Of like building you from scratch and then like doing what you did. Because this is basically what you needed for Pi. Right, right, right. Exactly right, yeah, yeah, yeah, yeah. Right. So, yeah, it's not really. I think they're, I haven't received any like pushback on like this idea at all. I think they're actually more just, most of them are consumers, right? Most of the students do just like end up using the tools. I don't think that they get a lot of exposure, if any, certainly in the coursework around peeling that onion back, right? And I think, so this is the week of the harness. So Tuesday, first lecture, you know, logistics, blah, blah, blah, all that stuff. But then first half is, you know, the first true, like, bit of curriculum is the 200 lines of, like, a harness. And then yesterday when I had sort of the follow-up to that is, like, we actually took Cloud Code, and then, you know, you have these, like, proxy things, like, proxy tools where you can intercept the request. Am I in proxy? Yes, exactly. Exactly right. So use that. tool, intercept the requests and responses, look under the hood, see what's happening there, and then it's like, okay, I did say you have quad code in 200 lines, but really, okay,

here's actual quad code. And so what I literally did was I took Opus 5.5, so this model of two days old, and was like, this is I intercepted those requests, here's exactly what's going on, here's the prompt, here's the system prompt, here's the plan prompt, here's the compaction prompt, and then we just went through all of that and sort of showed it to the students. So now they really, really have a state-of-the-art harness that they can understand. Yeah. It's nice to get those three levels. Students are very much consumers. Here's the baseline, agent in a loop, and then the real one. I like how you have a video walkthrough of your blog post as well. Do you see students are on extremes of some people are power users. They consume a lot. For some people, you could assume that this is kind of very straightforward if you know how agents work. How do you kind of balance that in the classroom setting? like do you tailor towards the average yeah i think it's a bit self-selecting because the curriculum is all here right so the curriculum has been available since i mean you know left least like a few weeks and and i guess i announced it a few weeks ago but i'd kind of been in the

works for like probably just one to two months so if you looked for you could probably find it um and i feel like the reality is if students are not interested in this then they won't like they'll they just won't show up and they will choose not to show up what i'm finding that's interesting is there are still many students that show up, but then there's also just a lot of folks from, you know, broader Stanford, postdocs at other departments. There's a lot of, like, interdisciplinary sort of interests where folks that are in, you know, somewhere in pathology or, like, you know, like medicinal sort of fields or even, like, random other engineering fields that aren't pure CS that are coming in and also interested in kind of getting exposed to this. Also a lot of GSB students, MBAs, they're also wanting, I guess, learn how to not have a technical co-founder or whatever the motivation there is. Yeah, so I think, by the way, if anyone hasn't done the MITM proxy exercise, you should do it. Yes. You'll be shocked. CloudCode has the longest system problems of all the coding agents, something like 20,000 tokens. Yeah yeah although yeah that is true That is true Yeah yeah yeah Although they also reduced the size They cut it by 80 Yeah I know but then they grow again and then they cut and grow again It a seesaw Yeah yeah I think it interesting

You'll be surprised what goes in there. Actually, sometimes you don't use. Sometimes it actually hints at the internal workings of some of the tools. Anything that you found there that could be cool to talk about? Yeah. No, I mean, in fact, yesterday when I was talking about this, it was a little bit of this, like, one of the things that I thought was most interesting about Cloud Code back in the day, and I think part of why it really blew up in terms of popularity, was their focus on simplicity, right? So in the early days, they're among the first to really lean into this, let fewer tools do more, right? Lean into SQL, lean into Bash, lean into the things that are powerful, like Swiss Army Knife tools, and have that be a lot of the driver of capability. I think they did a fantastic job leaning to that, and I think that's why they were as popular as they did. And I think that when I presented something like the system prompt, it was like, well, there was a different system prompt about two months ago. It is now, they have reduced in complexity, and it turns out that now what was once part of the system prompt, things like how to use even Git, right, or how to, like, set up tasks or things of that nature are no longer part of the system prompt. It's actually expected to be more kind of captured in the models, like the model weights itself.

So they've kind of baked in a lot of those learnings more at the model level. And that just raises interesting questions around, you know, the model versus the harness, right, and how do you kind of think through that tradeoff? What do companies invest in? What should you invest in to really push the frontier of your own capabilities? And so I think we still, even as a community, we see saw between the two a lot. I think you don't want to fault them for growing a system prompt and then cutting it. At some level, it's also how the model was, right? Weaker, older models, you need to give them examples of this is good, this is bad. And then obviously you cut that away at some point, right? And that's just part of learning what has been done, right? You can apply the same towards if you're using a local coding agent with not the strongest model, you might need to add these examples, right? That's right. Is that correct? Yeah. Yeah. And then, of course, there's interesting AI learnings to follow out of that, which is why, you know, as the models get better, when are you over constraining it by adding those examples, right? And then how do you pull away the over constraining? And that's, I think, part of the learning as well.

Okay. Week two, you're going to talk about context engineering. Repit, I haven't seen, but there's many versions of Spectrum Development. What would you highlight to a broader audience about what you're putting in here that people should know? Yeah. This is like the, you know, one-hour version of the course. Yeah, exactly. That's fair. Yeah, that's fair. It's also a trailer, right? It's like, well, if you want more, go to the store. If you're signing up next year or next fall for courses, you know, this is your one-hour. This is almost like the quibby of the class. It's too long to be a trailer. It's too short to be the class. Is this like what I guess people would double click? And isn't the best part? If it's a trailer, by the next time you teach it, it will change again. That's true. That's true. That's true. Yeah. Well, yeah. So it's okay. So a good call on that. It's funny. I think that the main thing that I just want to reinforce here is, and I see this even when I'm interviewing folks, very few people in industry even progress beyond just vanilla prompting.

Right? If there's a form like, hey, do me the thing, right? Like, do the thing. And that, certainly as the models get good, that can carry you a certain distance. But inevitably, you know, any folks that I speak to that come out of, like, a Google or someplace where there's a massive, like, code base, the reality is you have to be a lot more thoughtful about how you're, like, ingesting context, right? And so thinking through what are some of those principles and those techniques around how to, like, point, you know, and I said, like, you know, the rapid framework that I'm sort of describing here is functionally similar to, like, Dex's RPI, right, or something of that nature, where you have this, like, research kind of proposal, sort of spec, and then you do, like, your implementation, and then you kind of tie it together with some testing and whatnot. So what I just wanted to reinforce here is that, like, you should have this step, especially when you're entering a familiar codebases, when you're trying to sort of infuse into the context something that an agent needs to understand about a very large codebase, there's value in something that is a research phase. And then, in the case of this particular sort of framework, that I'm putting forward, it's like also then suggest some proposal

where there's a few alternative ways of solving a problem once you've done the research. And so this is where I sort of also invite the human in the loop to sort of be the decision maker on one approach versus the other. And then you enter the plan, which is kind of the spec. All to say that the TLDR is like there is a way to do this that actually does scale out to code bases that are millions of lines long. And it's not just the vanilla prompt. And I think that's what I want to highlight. If you're going to go work on these big code bases, you should use something a little bit more involved. Anything to add on context engineering? I think that probably the weirdest thing about how I think about context right now is when you split out to a subagent, because subagents can benefit from inheriting your entire context of the main thread, or not. Sometimes there's a regular disagreement on that, and then sometimes you want to condense what the subagent does and researchers put it back onto the main thread. And it's very hacky.

I don't like anything I'm doing. So I don't have any answers. All I know is when you want to fan out work, it is not just about you have a big spec and you do RPI, you do R-Repit, whatever. It is not one agent anymore. It is about fanning out work and making sure they all have the right context and they all have the right guidance. If you have a review agent reviewing all the other works, that guy have everything. And you always go through this phase of expanding contracts. Like, oh my God, these guys don't have enough context. I'm going to give them more context. Okay, great. And then they're going to over-tune into some random one-off comment that you made a while ago and then make some really stupid mistakes because you didn't mean it that way. But they took it as the law. Something like that. So I don't know what the word is I'm saying. It's just like multi-agent management for context. Yeah, yeah, yeah. And that's absolutely right. there's like some aspect of this is maybe the part that's more maybe among the most art than science right it's like some combination I think to your point yes the fan out

for the exploration the fan out for like the subtasks and then the consolidation compaction whatever you know you might want to call it I think being thoughtful about refreshing of the context understanding when things are getting a little bit too bloated I think you know decks have this concept of like the dumb zone and sort of like being a little bit more thoughtful about the dumb zone he's good at these equating things oh yeah yeah the little diagrams yeah oh yeah I didn't have him on But, yeah, well, I mean, he definitely informed a lot of the curriculum here, and so he's just a fantastic thinker in all this. No, I think the diagrams are huge. That's useful. I would add breadcrumbs for people that are trying to look into this. It's good to see best context engineering preferences, like on your sub-agent inheriting, cloud-managed agents has a version where sessions are separate and it's not inherited. This used to be a big topic before reasoning models when agents didn't run as long. We had shorter context windows. Everyone was very, very managing that precious context window, right? LinkedIn used to write a lot of blogs about this in good context engineering.

All this stuff is good for people to know. I think in the student context as well, you can expect the average student to take your class because it's a great class and they'll know all these things. So you should know them, know them better as well. this is a fun space for research as well one of the papers research papers presented at AIE they looked at the delta in the cloud code system prompt what got changed and they plotted a whole bunch of changes in performance they tried to reverse engineer it it's just good research that you can do there's another one that also looked at context window management and how it affects performance downstream they weren't the best papers but it's quite accessible and it puts out good blog posts. So people that are trying to work on it, I just recommend it. So it's a good area you can easily poke into. Yeah. Good shout out. Agent Skills, CLI. Yeah. Yeah. I think all this is a little bit of an extension from a single kind of function tool calling, like MCP, interacting with third party external services, and then extending

into what you would call procedures. How do you automate some set of actions over and over again that you then formalize in the form of like a scale md or something like that so i think just making that a little bit more uh giving people exercise in in codifying some of those procedures i think what i'll also sort of like include here is is a few few kind of fun skills that that i think are good to use like i think matt matt pacox like grill me is like a good one uh and then i there's ones that i also like from more from the educational perspective like jeffrey litz uh he has like explained diff which i think becomes especially interesting once you start talking about code review because you know the whole understand the code understand the code exactly yeah you got to understand the code and and i don't know if you guys have used that but effectively like you run you know you have your your kind of the the diff you've put in your code and then you run the skill and at the end of it you have this not only like kind of a visual representation of what's happened so you can you know kind of skim it and understand what's what's actually at the core of it but then what i really liked was this almost like quiz at the end of uh at the end of it where you go through and sort of like really reinforce whether you understand what's happening.

And I think that level of interactive sort of back and forth, I think is quite nice. Yeah, there's a couple blog posts in this domain that I really like to bring up. There's one, I think, recently, which is like kind of, there's no situation in which turning your brain off is a good idea. And I think these skills actually force you to turn your brain on. And when everyone wants to use coding, you just turn your brain off. They're just like, make me an app and read my mind and get everything correct without me telling you about it. That's right. But a lot of these skills are mostly in the genre of like, no, no, ask me more questions, please. I think there's a lot more output than people are used to. I can type in a one-line prompt and get tens of thousands of lines of code and a whole structure. It's like, look at my productivity, guys. But more effectively, you should know what you're working on. You can really be put out of touch, out of the loop. You should spend more time thinking about what you're actually doing. Doing the work and getting the code written is much easier than it was.

Yeah, yeah, the understanding there's no excuse for, for sure. What is a web skill? Yeah, so, I mean, effectively, like, skills that interact with, like, the web itself. Computer use. Computer use, yeah, things like this. Anything else? I mean, mainly things that involve, like, pulling information from the web. Are there best practices on skills that will interact with the web versus? I think this is more of like a bit of a survey or kind of a demo of stuff that I found that was kind of interesting. I think it might have been out of, was it like a Vercel? I'm trying to remember now. I'd have to look back at the full reading materials. But there was like one site that I found where I think that maybe it was, it might have been browser-based, actually. They might have done. Do you remember this? Well, it would be done. They'd been quite on this thing. It's just that I was like, did I miss something? This was like a big part of the skills community. He's good. He's being educated. Yeah, I think they made some sort of, because I guess they were, yeah, I mean, browsers obviously like. They like browsers. They like browsers. But yeah, they had a good, I think, website where they showed some of these interesting capabilities. Seems legit.

CLI is a fun one. Do you have a lot of students where this is their first time in a terminal in a CLI, or is it kind of the norm now? No, I don't think so. I think that given that most of them, again, last year I would say that some of them had probably still remembered the days of finger-on-keyboard typing. This is most, I think, students have, like, interacted with Cloud Code at this point. I'm curious the split of people that are now just using the apps, right? Cloud Code also has a desktop app. So does Codex. Codex is primarily an app now. You're kind of not in the CLI, not in the terminal. it's a thing that puts off a lot of average people but just a random note I would say I think there's also another thing about skills that are bundles of skills where they sort of cross each other and then there's also I guess the art of writing a skill which is a thing that I've also focused on for AIE I think a lot of these are

focusing on skill consuming and you have all these influencers like Matt Folkog and what have you that are like, this skill will save you. It will solve all your problems. But no, you should write your skills yourself. And actually, when you try to write them naively, they're usually bad. And actually, it takes a little bit of skill to write a skill. That's absolutely right. That's absolutely right. What are Swix's best tips on writing skills? I have a public skills repo. I went through definitely a phase of psychosis of skill everything. No, no, no. Don't give us skills. How do you write good skills? Yeah, exactly. So I have a skill for cutting skills. So every Cloud Code and Codex, they all have skill creation skills, right? So then there's, I have the skill cutter skill that cuts it down because they're all it just documents every way in which those default skills that are created are bad. So you always run this after any skill is created to cut it down. I like this. Yeah, it's really annoying because sometimes there is slop, but there's good in the slop.

And then you have to sort of whittle down what they're... Right, right. Oh, interesting. As you're saying, even the slop, it's still in the language of the agent. So the slop means something. It's like a... Yeah, okay, got it, got it. It's fun. Everyone has their own version. First, it's very easy to vibe check a skill, right? You just call it, run it a few times, and then I distill these preferences into a skill that's like, this is what I like, this is what he doesn't like. And it adjusts over time, and then, you know, it can automate. So the problem is that, yeah, you're calling it, But the actual, the worst part is when a skill triggers when you're not expecting it to. Like you wrote it one time to fix a thing. And then like now you're working on a completely different thing and it triggers. Why? Because it's badly written. Because the trigger says like, oh, you call me whenever blah, blah, blah happens. And you're like, okay, it's good. And then you do a different project and it triggers and you're like, what the hell is happening? What the hell is happening? There's like a few people that are just like, okay, I have a bunch of skills. I keep adding on sometimes. And I know some people like Eugene Yan. they are pretty particular

about how many skills they have they prune them if not used they clean them every few weeks yeah yes so yeah this is like a huge problem if you're sort of in that skills world I think this is actually why I suspect they're good they're very smart people at Open the Hand and Topic that want skills to die which is not a popular statement they won't say this publicly but it just feels sloppy this feels sloppy I can see my own skills before being sloppy. It doesn't feel AI-pilled. Do you know what I mean? Yeah. It's not what AGI was supposed to look like. It's like everyone having a giant bunch of markdown files. I mean, I think it is. There's a gardening problem, yeah. It makes some sense. A lot of people actually don't look at their markdown files, right? Like the UX of it, of just chatting what you want, creating a skill in chat, using it, seeing output. But there's a lot of value you can get without ever looking at your markdown.

Don't overcomplicate it at some level for yourself. You can get a lot of value. If you do stuff repeatedly, just be like, look at the last five times we did this, make me a skill. And you can iterate it with chat. There's a good side of it as well, right? There's another thing. There's another way to look at it as well. Because there's the use of the skill as just procedure that the agent kind of runs whenever to do like a code review or whatever. But then in the case of something like Devin, right, where we've codified skills as a way of sort of representing our best practices, right, and then exposing that to the agent or the code review system to then institute the APIs the way that we like to define them, institute kind of the service level boundaries the way we define them. That's like an alternative way of sort of representing like org-level knowledge and maybe not so much this like arbitrary thing you want the agent to do so you don't have to write the prompt each time. so I like that I feel like that positioning might that purpose might exist longer than just you know these kind of random procedures

I just googled Monaco AI skills and it's giving me the country maybe the country one day we're going to be the AEO that's for sure you just buy the country on this we're about to no we're not about to buy the country but we did get into a little bit of flack when we bought the monica.com domain. I will say that. We did get sued by them. What? Yeah, there's like a whole public thing about this. Yeah, so they didn't like that, but here we are. And we won. We won. Okay, we're keeping it in. I was about to say this is spicy, but you won. Yeah, yeah, no, no, no. Is that why we see the flags flying around? Because it's like a victory. It's a victory. It's our victory. That's funny. For those that are in SF, just look up and yell. So for those who are on audio, we're showing this on YouTube. You fly a plane around. I think you're still one of the few that actually do this on a regular basis.

Sam Blonde came up with it. $6,000 a day. I don't necessarily think it directly causes me to Google what Monaco is, but all of Monaco's ads are just Monaco dollar sign. Right. It starts the conversation. The whole point of the ad is, what the hell is Monaco? That's a debate to be had, but in some sense, what is the purpose of marketing if not to get attention? Yes, maybe there's one level beyond this. I'll go to an event and people are like, oh, you guys are the ones with the planes. You guys are the ones with the big number on the billboard, blah, blah, blah, 193 million, whatever it is. But they're like, what do you do? But maybe that's the whole point, right? Is that I just needed someone to come and use it as an icebreaker. I will say it's what I said as you walked in as well. It's like, ah, the planes. Yeah, yeah. And now what do you guys do? Well, let me tell you about what we do. As someone who's been in those meetings deciding what we do on the Devin billboards,

we spend so much time on the words. And maybe we should just be like, Devin. I need to do it. It's just so fucking dumb. But like, okay. You know, look, I think Sam had another post this past week where he was just talking about like, I mean, Sam is effectively like our, you know, marketing sort of mastermind here. And yeah, I think there's a little bit of history behind the planes as well. But the TLDR is that his whole shtick is like, I don't have all the answers. You just got to try things. And I think a lot of this is we just try things. And if it sticks, great. If it, like, did we know this was going to blow up as much as it did? I mean, not really. But now that it did. Get an idea. Yeah, exactly. And minimalist, start small, go high, instead of start big, go small. Right. Way easier to start small, go big. Yeah, exactly. Also, you know, it's 6K a day. Like, it's very much just try things. I think people... Cheaper than a billboard. Cheaper than a billboard, for sure. Yeah. And it's fun. Yeah. Not much from time. But, you know, the kind of simplicity here that's worth mentioning is then when we're trying to, like, one-up the plane, we're like, you know what's better than one plane?

Two planes. Two planes? What? My mate went to rockets. Well, okay. I don't have that idea. I would like for that to happen as well at some point, but we'll see. The next SpaceX launch, we just need Monaco. There you go. It should be a start. There you go. There you go. We've got to warm up to our SpaceX friends. Okay. We're on Monaco. Is it worth talking a little bit about Monaco? Do you leave it today? Yeah, yeah, yeah. Because there's just like an intermission between the topics. You decide. Okay. We mentioned Monaco, right? Let's give... I think it's a little bit annoying to mention what the hell is Monaco and then not answer it. Sure. Can you answer it? Sure. Well, I hope so, yeah. Whoa. This is the country. So it's a small little... Oh, no. Oh, no. There you go. There we go. There you go. Okay. This is Monaco. This is the Monaco that I know and love. Yeah, so we are... We're a Series V company here at Basin SF.

we are building effectively like an all-in-one sales platform. Fully AI-native agents doing all the fun stuff, or at least the stuff that humans don't like to do, but that agents do like to do, because they don't really have opinions, but do in a less air-prone fashion. So the salespeople and sales teams can focus on doing what humans are good at, which is closing deals. So presumably, Mark Benioff doesn't like you guys. Probably not. But he's also honestly a very AI-native person. very aggressive and agent force and everything. What are you doing that he doesn't do or he cannot do? Well, okay, so right now we are, we started the company in a sort of like, the market that we went after was not enterprise from the get-go, right? We sort of said, like, we're going to start by, like, if our goal is to build the all-in-one, then we need to kind of build this in an irritative fashion with a group of customers, and we need to grow with them into this. It's much harder to pull someone away from Salesforce than to say, like, hey, if we meet you at the point when you are still just a Google spreadsheet, tracking pipeline, and we sort of show you a more AI-appealed sort of answer to the problem,

then we can grow with you. And in a way, it kind of follows the same model of like Brex or all of like YC companies where you kind of sell to your batch mates, you kind of grow with them, and then you go up market. And so we, I would argue, are the, I would claim that among the best, if not the best, sort of like a startup-focused sales platform. And so all of our customers are like early-stage startups. at this point probably through like series A is kind of where we've primarily been focused, kind of C to series A, and then now we're going more upmarket. And we've kind of learned a lot about the all-in-one in the last year and a half of doing that. And you are the head of AI? Yeah, that's right. What's some cool AI stuff you guys have built? Yeah, I mean, we have built our own agentic harness for sales use cases. So all this, I mean, to call back to the coding harnesses, taking a lot of inspiration from the systems like the codexes and the clods of the world and sort of building bespoke tools that are very sales-focused, bespoke workflows that are all kind of geared toward GTM use cases. So we built our own version of that that powers pretty much every single thing

and that powers the Monocle platform. So from scoring of TAMs, scoring of ICPs, and defining of ICPs, through to pipeline evolution, through to all the mundane kind of things that maybe a salesperson doesn't want to do, which is like, oh, update my meeting, you know, pre-meeting notes, update my summaries, like all these things. You know, we have agents powering all those workflows. So we've done that. It kind of comes out of the box. And, yeah, so long agents, long horizon agents for long horizon agents and short horizon agents in some cases for GTM use cases. And it's pretty domain-specific, built-on foundation models. Would you do any of your own models for this? Yeah, it's a good call out because so far, I mean, I think this is generally true of a lot of like vertical AI companies. It's like you kind of start where you sort of use the frontier models as third-party services to see what's possible because they are like the very frontier. And then I think we've seen this, I guess, in the past week with the whole like Harvey sort of where they like plus 50% to minus 50% margin kind of question.

I mean, I think irrespective of the numbers, I think what is interesting is like at a certain point, things do become interesting from a cost perspective. you start to wonder can you do it faster on open weight models and then a lot of what I think the purpose of the AI pieces are is just to post-train those models build environments for using RL and things of that nature so we're kind of in that phase right now Ooh, fun Yeah, I've been calling this playbook the agent lab playbook but that was kind of my coining of agents start from the user first and then they use whatever models you need to build products, but then you can backfill the model side once you have the user base. And then the model labs kind of go the other way. They start with the big P-trains, and then they try to find users. And it's hopefully working as a split. The monocle would be on the revenue GTM side. That's right, yeah. And I think it's interesting. They would want to serve you while also trying to compete with you as model labs.

That is correct. One of the things that I've been, I dabble in some of this. I think one of the hard problems is actually enrichment. I would say I spend a lot of my time doing that for basically SEO and research and what have you. And I imagine for you, it's actually like a core part of that. That's right. Do you find that models are good at it? Do you have any vendors that you like working with and you want to give them a shout out? Yeah, absolutely. But, yeah, so we have used, we were early adopters of Parallel, and I think they're very strong. Okay, so just your general search. Yeah, I mean. Not like Zoom info or something. No, no, no, no. But, yeah, we use Parallel. We also use Exa. So we'll give a shout to both of those folks. Can you compare them? They're both friends. They're both friends. They're both friends. They're both folks that we like to work with. They'd be very interested if you like one of them. Yeah, I'll keep it. I'll keep it neutral. I'll stay Switzerland in this today. But yeah, and then of course we do our own version of enrichment as well.

So it's like most of our data systems, I mean part of like one of the most valuable pieces of software we've built or most valuable systems is kind of like our world database, like the database of companies, enriched companies, enriched accounts, enriched contacts. So like that whenever we are constructing ICPs for folks, we have the most up-to-date information about who you should reach out to at which company at this point in time. Like I would say that one of our, like I think where we are like world-class hands down is just demand. We are a very good demand company. We help companies increase their meetings booked in a way that actually... So find buyer at the right time, right? And it's kind of like a combination of the... What do you mean at the right time? There's times when... Here's an example. If you are targeting someone that just... You are selling to the piece of sale, let's say. The right time to reach out to them is not when they are sort of in the middle of some... They've been at the same place for the last five years. If they are now in the middle of a job, a good signal is if they've gone from one job to a new job, they're now entering a new place where they're more likely to purchase or be open to a purchase of a new sales platform

because they're in the middle of a job transition. So going from company to company B for the sales, a good signal of, hey, should you reach out to them at this point in time is did you detect that their LinkedIn changed or something like that. How much of this is your expertise put into model and harness versus models are good, there's generic best practices? is like, is there, there's obviously some value in both. Yeah. Is it just the glue getting it reliable, doing it all in one platform, doing it well? It seems like there's also stuff that we wouldn't know that you guys are also domain experts in that's baked in. Yeah, I think without the former, it would be like a fully commoditized. I mean, because there's still a lot of value in the all in one. I think like that is something that we definitely pioneered. Like most companies in the sales space started as just like point solutions. And I think they tried more of like the YC approach of you kind of land and expand. You kind of pick a niche and then you expand from there. And we, from the get-go, I mean, that's why we stayed in stealth for the greater part of a year and some months, was just building because we knew we wanted to be the all-in-one from the start. But, you know, that alone is powerful.

It's meaningful. But I think the real power is the fact that today, Monaco as Monaco has like a world-class go-to-market team, like a truly like world-class go-to-market team selling Monaco, of course. But we also have the same go-to-market people that sell Monaco are the ones that are like what we call forward deploy account executives. They're FDAEs. They're effectively like human salespeople that are like the consultants of sorts that help onboard all of our customers, get them like building agentic workflows, and are effectively there to like help guide them through every part of their own sales go-to-market lifecycle. And so to your question, it's the all-in-one yes, but it's also the fact that we have these like, you know, Sam, and we have brought on a lot of like former Sam's team members and stuff like that who were like the best people in GTM at Brex and other places. so we're codifying all of that in the platform yeah interesting to have you talk very differently than a normal head of AI that I might also talk to because you're also a former founder and you're also used to thinking about these things in this way and it's interesting it's important to have that

in your role yeah cool I think you've introduced Monaco enough there and obviously I think as the Monaco story develops we can revisit maybe the bridge back is also what is coding or agent-native coding or modern software development like inside of Monaco? I think when you're teaching, you're maybe trying to be, here's the lay of the land, but maybe inside of Monaco, you guys are more opinionated on one thing versus the other? Yeah, yeah. So we've made investments in coding agents from basically day one, and we've learned a lot. We have our own scars that we sort of picked up and learning. And I think we've now been getting much deeper into sort of sharing some of our learnings. So, like, in our blog, which might be worth sharing here, we've been increasingly posting some of what, maybe not that. So that one is more like a product, the one at the very top. But the stuff I would say, like, yeah, when AI gets that's written before, like scaling contributions, builder buy, SRE. So, you know, we spend a lot of time looking at different providers and vendors and end up building a lot of things in-house.

So for us something like a background agent platform we have our own version of like a ramp inspect that we call Monaco which is built on the Coder platform but it is what we use to essentially scale up our engineering productivity Coder like Coder Coder.com, yeah. Oh, okay. We actually have sort of adapted it to some of our use cases, and it's where a lot of our agentic development happens. It's just this really well-designed piece of software for essentially isolating of development environments, scaling out to multi-agents, being able to run these long horizon agentic coding agent tasks. So Monocoder, shout out to John Tracy and co. for building that out. And then, yeah, from there we've also built our own versions of a Slack bot, which we call HAL. But effectively HAL is like this part AISRE, part customer success, part QA engineer, because it lives in our Slack, but it serves all the purposes of helping us to debug things

and put up PRs when things are broken, but also use our own salespeople to find out how the product works, find out how to get to the bottom of something. So there's interesting questions we've had to answer around the different personalities that Hal has, depending on whether it's talking about a salesperson versus a developer. So it's something like this that we've built out over time a lot in the last year or so. Shout out to Yunja for Hal and John Tracy as well. So you're building a lot in-house. Yeah. You also use other tools. We do use other tools, yeah. So what is the line? Yeah. The Builders Buy. Yeah, the Builders Buy. It's a good question. And we, I think there's this, like, a thing, a general principle that I generally, which is, like, you build to learn and you buy to keep. Or, sorry, you buy to learn and you build to keep. Yeah, the other way around. So you buy to learn and you build to keep. And I think that's generally true here, where in the early days when we were still trying to figure out, like, you know, get our feet under us a little bit, we were, you know, code reviews, an example.

One of them where we spent a lot of time looking at different code review platforms. It turns out today we still use Devon because we actually still find a lot of value in how this system's been built. So, you know, shout out to the Cognition team. But then along the way, we did try out some, like, SRE tools. And, you know, we tried some sort of, like, off-the-shelf, maybe more, like, background agent platforms. Decided that it was just given our infrastructure, given how we had set up our own cloud platform, it just made more sense to kind of build this out internally. Of course, we're not dogmatic. We are open to revisiting them if something comes out. But here, I think usually we ended up building a house when we felt that there was no mature offering out there. So CodeReview is an interesting example because we did do a whole survey of CodeReview tools maybe over a year ago. And at the time – And I'll show it out like CodeRabbit, right? Yeah, CodeRabbit and them. There's also CodeRabbit, Graphite. for a while before they got acquired. Like Ellipsis, which is a wise company before the fact. But there's a bunch, right? So there were a lot of folks that we did try. And they were okay, but I don't think they crossed the threshold of they're immediately useful yet. And so then we

put it aside a little bit. This was, I'm saying, well over a year ago. Then we kind of went back to this. And sort of organically, again, shout out to the Devon team and Cognition folks, where we tried it again. I think I had gotten that somebody told me, hey, you should try this and you know it seems like it's gotten a lot better and i've been actually during silas's talk during the class last year and it was just good and it was organically like like that's purely vibes right like this is there's no way there was there yeah this this was like i mean okay okay here is what i'll say when we did that first evaluation the first time around where we didn't really pick any vendor um and we were like there's no vendor that we're super happy with i did do like an analysis where i looked at our prs basically did sort of like a you know set of high diagrams like here's what this is solving. Are we happy with the kind of the tradeoff of verbosity and how often it's posting with utility? And so it was more statistically driven than just vibes. The second time around, I wasn't going to push hard on it. The first time, I was kind of like, hey, we should adopt an AI code tool. Second time, I was like, no, I'm not going to do that again. But then organically, people were just like, hey, Devin, is that good?

Like, can you like add me to this? And so today, people are just like getting organically added. So this is a vendor where, you know, one of the cases we're still leaning to the vendor. You know, we'll see in the long term what that looks like. But everything outside of that, we just didn't feel like maturity was there. Yeah. And it's not painful enough that you would, like, we're about Devin and write you on. That's right. That's interesting. Yeah. Good talk for Devin. Yeah, I didn't even know. You know, for the record, I'm only an advisor. I'm not that involved anymore. But, like, it's interesting to hear about it externally because internally there is a certain amount of, let's call it, the bubble where like on Slack you only hear positive things because people only want to circulate positive things but you have no dog in the fight you just choose the best tool I mean if you want to get a Slack for your credit that's totally that'd be cool too I would say no one say no to that I think Silas will hook you up okay so there's that right and then and that informs all your teaching work yes that's right

and I think that's really good and we can bring in more of this as you like because I think this lived experience is very good. I'm happy to go back to HR-ready code bases. So week four and five, kind of matching. They blend a little bit. Whatever strong beliefs you have introduced in Monaco and recommend people to start with. Yeah, for sure. Week four and five, I would say they are a little bit, four is maybe a little bit more conceptual foundations and ways to do this because there are a lot of ways to customize agents. And then I would say week five is where I really kind of lean into, all right, let's think about this and apply this in practice on a true code base. And so here I will definitely give a shout out to the factory team and want to just highlight the value of something like a readiness report. I think that the main piece that I want to reinforce, something like agent-ready code bases is to me like a precursor and a bit of a building block toward everything to come

afterwards. When I say everything comes up, I mean, like, people always get, you know, for a while, and I think people still are, like, super hyped on, like, multi-agents, like, paralyzing agents, like, tens of agents doing all kinds of things. It's, like, you actually need to set a foundation in order to do that well. And the foundation is how verifiable is your code base, how, like, well-structured is your code base, how understandable is it by an agent. And so I think, like, agent readiness to me is, like, almost, like, piece one of the next four to five weeks where, maybe until, like, week eight or nine where I spend more time talking about, like, broadly, you know, AI software teams. and the kind of org-level structure, but definitely building up the building blocks for background agents, the software factory. I think this is where it all kind of gets interesting from that perspective. Yeah. One thing that I observed you talking about for Monaco but it's not here is the SRE side. When you close the loop, you actually want to have the agent read your logs for builds and then ideally read your analytics for live production. That's right. That's right, yeah. Not explicitly, but it's a really good shout-out.

So I think that I'm going to weave that in in, like, two places. One is in the background agents week, and then also in the software factory week. So absolutely right. It's a really good call-out, this idea of, like, trigger-slash-event-based closing of the loop, which, you know, now is one of the probably two or three entry points when people talk about agents kicking off tasks in the software factory. That's absolutely right. That'll come in there. Yeah. It's interesting that people call it trigger- or event-based, which I guess that is the actual technical implementation of it. It just seems too low level almost. So the trigger is usually a cron job, right? Let's be real. Sure, sure, sure. Event-based is usually a cron job that surfaces an event and then it's event-based, but it's all crons. Yeah, yeah, that's right, that's right. Yeah, like web posts or something like that, but yeah, that's absolutely right. Yes, and the technical term, of course, because at the end of the day, Yeah, being sort of like a CS person by trade. And, yeah, we called it those things. And so I guess I fall back on that terminology.

Yeah. And have any AISRE companies made an impression on you? So, okay, I mean, like, I think the Resolve folks are doing well there. The Resolve is doing well. I would say Night Shift is also doing quite a good job there in terms of broadly, like, background agents. I'm a big fan of kind of, like, they're a great lock-back. I think they're doing well in this area as well. Yeah, I would say that where I think SRE is an interesting use case, and this might back to the build versus buy question. I think what's interesting about it is like where does an SRE, an ASRE like really see the most value? It's typically when you have like a tremendous number of like service dependencies and these like massive sort of like systems and tons of Kubernetes and Helm files, like just like tremendously large Kubernetes infrastructure Helm that's like files kind of streamed everywhere around. I think that's why Resolve has done really well in the enterprise, right, where you go and you sell to a Ford. You go and sell to a Walmart, and you kind of say, hey, your service level dependencies are crazy. And you see these diagrams of how things are set up.

And then they kind of go in, and they're like, well, we'll try and make sense of all this. I think that at smaller scales, it's usually still manageable. You can kind of roll your own. But I will say that for the enterprise, I think they've done a good job. Yeah, for people who have never worked at this scale, because it took me a while to see this, it is on the order of, let's call it, tens of thousands of repos, more repos than developers. These are the situations where you don't know what code is running anywhere in the company, because probably the guy who wrote it is no longer here, right? And by the way, it's still low-bearing. That's the scariest thing. That's absolutely right. And then you're the person who's on call, and you're interacting with a system that goes down where that person still isn't there, and there's old out-of-date runbooks, and you're like, what the hell is going on here? It's a scary, scary thing. And yes, at the enterprise level, it's a whole different beast. Yeah, so the kind of college hacker, one person per repo, they've never, no appreciation.

But I think if you just give that idea, you drop them in a simulation of like, okay, like day one on your job, this happens. What now? Yeah. Yeah. So another shout out I'll give here. So the Resolve folks, actually last year, they made an assignment for the class where they effectively created like a sandbox for students to go and then the sandbox was like, I mean, within the sandbox, there was like many, many services that were doing their various things. And there was even like bugs in the services within that sandbox. And then as part of the assignment, The students have to go and actually use the Resolve agent and kind of probe and play the human on-call person probing the system, discovering what's happening. It's got an interesting website. Yeah, it's an interesting website. I do not. It's been a while since I've seen it. Yeah, this is very interactive. Okay, they're being a pager duty now. Yeah, yeah, yeah. Yeah, but also this video. You know, you were shouting out. Interesting, interesting.

Yeah, so, okay, we never get a chance to talk about this side of the world, except for a conversation like this. Any other segments? Because I think AISRE is maybe too broad or even too narrow, depending on how you look at it, right? Like, too broad meaning, well, there's several subdivisions of AISRE that's interesting. Or too narrow being like, well, actually, that's not even the biggest problem that we have. So take it however you like. Yeah. You mean, like, specifically for, like, Monica or just, like, broadly for, like, organizations thinking about this? broadly because that is our audience. Yeah. I think that, again, it's a function of how we just scale. I think that for the enterprise, there are a number of issues that aren't always, I almost feel like the answer is not even always just more agents. And since we're still on the week five agent-ready code bases, when I talk to companies of a certain scale, the things that are holding them back are really more just low-level DevOps, CICD stuff. their merge queues are getting backed up it takes too long to stand up an app

there's just actually more foundational things that have almost nothing to do with throwing in one more SRE bot a bot wouldn't be the answer to the problem because there's more systemic things that they need to figure out so that brings us to the weak face yeah exactly right so the merge queue and beyond yeah this is a love letter of sorts to modern Gentic code review Yeah, I think that the biggest thing that maybe happened between last year and this year was just it became very clear that code review is a bottleneck, especially once people got really AI pilled and the velocity on generation became so high that, you know, you could in theory like put up 30 plus PRs a day. And then what? At what point do you decide that you either have to institute yourself and put yourself in the middle as some way to gate and review things in the traditional way, but then you're literally the choke point of the velocity, and that's not good, right? Then what was the point of generating a scale?

Or you say, well, how do we maybe get the best of both worlds where we're instituting ourselves enough that we have trust in what's getting merged, but we're also able to kind of like merge scale but we're also not kind of like freewheeling it here and anything goes. So a lot of I think what I want to touch on here is just that middle ground, right? Which is like there is a class of problems that if you've put in thoughtful design into your agent readiness, you've put in thoughtful design around your verifiability, there's a class of problems that you can feel comfortable some percentage of the time, essentially like auto-approving. And what has to be true from a kind of a design perspective, from like an architecture perspective, to just auto-approve, and when do you have to insert yourself and say, actually, here, I still want to review this carefully. And even if you do review it carefully, are you still doing line-by-line review, or are you doing something else, right? So, yeah, some of this. One of our hottest topics of this year, we have a guest writer program for Latent Space. This is from Ankit from Aviator, and he has the whole thing on killing the code review. Because that is maybe the final boss.

That's a bottleneck. He has a few gates by which he does this. I don't know if you've talked about it. Yeah, no, I've read this post. Yeah, yeah, yeah. I haven't spoken to him personally, but I've definitely read this. Mostly right? Any criticisms? No, I think this is right. I mean, like, broadly right. I guess you mentioned one thing, like, the last boss. Maybe not, I think there's two choke points. There's the review, and there's probably, like, the intent or kind of the thing at the top, right? Like, this is... And there's some stats as well that I like to pull up. You know, everyone's stats are different. For sure, for sure. But, like, clearly more people are merging. Clearly the number of reviews are not increasing in line with merge. Yeah, that's right. And if you look at any modern sort of company that's really invested in this and thought about this carefully, and some people have already unposted about this, and I'm sure will do post about this as well, but I think recently, like, the Amplitudes and I think Cloudflare has done a good one. I think Augment has also done a good one where they talk about the design of their AI code review system. These are a bunch of really good, if you want to share some of those links, I think they're a fantastically written post. actually shout out to

Cloudflare because I think very like kind of like you know in a sense like legacy company or like you know bigger company but I think they've done a fantastic job in terms of just their content around the AI the AIS DLC so I think they have a good post there but TLDR yes I think this is true I think that there's we're still finding that sweet spot what is auto-approvable what is not really auto-approvable and if it is not auto-approvable is even in that world of something that has to be somewhat human reviewed, what do we review? Are we still doing the line-by-line? Are we doing some combination of the spec? And, you know, I think this, yeah, again, a whole separate question there as well. Are people keeping, are you guys keeping the spec up to date at all times? The spec is no way, right? Yeah, no, well, I mean, okay, in a few different places, right? So, like, there's a spec that as it exists in, like, a linear issue, and there's a spec as it exists in, like, the context of a PR description, right? so you can have your peer description be the place where you describe more or less the entirety of the implementation in a more condensed format

and as long as that's in this up to date thing representing this point in time of the code base, I think that's sufficient actually so we're thoughtful about our peer descriptions we make them be not just like one paragraph, it's more like it is effectively like a spec of what has happened in the PR and there's even like kind of line file level pointers to where things have changed so you not only get the high level of what's happened but even the construction of the files and the service level boundaries that you're touching. So, yeah, I think that's where we've invested quite a bit. So I push back on that because, to me, of course, it's, you know, bare minimum it should be internally consistent with the PR spectrum. No, what I meant is the spec for the feature, the code base, because that is the summation of all PRs. That is like if your PR spec got it wrong or missed something, it's going to be the delta between what the whole code base or the whole history of the code base is supposed to do versus what your PR is trying to do. So I don't expect you to have an answer. Because I really stress about this myself. I try to instruct models to do it.

They're bad at it. That's it. So the humans are the spec gods of like, well, five weeks ago I did this other thing, and now you're reversing it. Have you seen the post from Basis? They had a tweet, little thing, a tweet article. I thought they did a great job talking about different sort of file artifacts they track via Git. And some of them are the specs, the intent of what it's to be, and the other is what they call the canonical state of the code, which is a little bit more maybe what you're describing, which is implicit knowledge, implicit intent that also has to be codified somewhere. So that part, that's all version controlled. I think they found that to work for them, so that might be another thing to share as well. Yeah, people was trying to pull that. This is basically the accounting company. We're having Mitchell at my New York conference soon. Talk to me about that. I like that they are coming up with this. My solution for this is I don't know how ready I am to talk about it because I may never ship it. It's something that is

replacing Git in order to capture all of this. So I call it a program database where it captures all traces and reifies specs through something that we can ignore and keep and it also controls the skills that are attached to it as well. That's cool. Because I think basically Git is only focused on code, and then you just need a source of truth that is not only everything else outside the code, including the implicit, what did you call it? Yeah, like implicit assumptions or like it just strains, intent. Other stuff? Yeah. But also highly concurrent. Right. Because Git, by design, introduces mutexes everywhere, which sucks. Yeah. So you need super concurrency. Have you seen Is it Kid Butler? Is it? Yes. The guy who used to be CTO. Ian, I think is his name. I forget his name. Scott. So, yeah. I think there's a lot of people going after this. And I think there's the question of

really what is the ideal representation of tasks. And I think the winner probably not me linear. Really? Because that's where the tasks are. Yeah. Yeah, yeah. And clearly they have decided to become a coding agent company. Right. And, well, they have all the sorts of truth there as long as you buy into linear. Yeah, yeah, that is true. That is true. I wonder, I have this somewhat kind of held belief that I think a lot of companies are going to evolve into some version of a coding agent company, which is any company that has an interaction with B2B software where you're collecting something about the user or some product insight, I think you can also close the loop. Because ultimately, the purpose of coding agents is to act on some adventure thing that's happening where users are involved or customers are involved, and then use that to fix ideally some asset to the product. I think what Linear is doing, I imagine the same thing Amplitude could do as well, because they're also getting insights about user analytics, user problems. I imagine even if you want to really stretch it, you could even have a Decagon or CRM, do something similar, where here's what's broken, go close the loop.

Yep, all right. So set all those things up for agent-ready code base. instead of code review. Security is week seven? It's not week one? Students. What do you mean security is not important? I make my answer to that. I always say, like, you know, at Amazon, job zero is security. And they always say this. Yes. Anyway, yes, security review at the end. I was at Amazon, and I don't remember that, but I believe... This is the thing Andy Jesse says. Okay, okay, okay, I see, I see. And I was there partly during Jeff's time and partly when I was transitioning to Andy. But, yes, I think this is a topic last year as well, and it's also won this year. And I think that there, Isaac also spoke last year. I find that he's an incredibly thoughtful thinker. Thoughtful thinker. You know, a thoughtful leader in sort of this space. I think SamGrup is an interesting company because they've been around even before sort of agents, right? I mean, the OSS repo was what was well before, like kind of traditional SaaS kind of defense mechanisms. but I think they've done a good job of pivoting

or not pivoting, augmenting their offering into this space. I think it's an interesting time. There's just a lot going on where these attack vectors are now coming to the surface. There's a lot more slop and AI-generated code and I think maybe one of the core insights that I want to drive here is just the idea of dependency tracking and supply management basically. Supply chain. You have to be careful about your dependencies and pin things appropriately and don't just arbitrarily hit npm install and expect the change to go. Yeah, so potentially the most secure code base is one of no dependencies. Yes, that's correct. You look at the dependency, you're like, cool implementation, bro, but I'm just going to make my own. That's absolutely right. That's absolutely right. And that's an interesting insight because it's sort of to the limit. And I think it's probably easier said than done, but if you believe that with code now being commoditized, code being easy and cheap, why not just build everything in-house, right? Like, of course, and that would be the safest thing

because you've seen everything, you control everything instead of, you know, depending on Axios or whatever. But, you know, we'll see how many people actually act on that. That is the safest thing. I agree with you. No, so that statement is dangerous because then you would just make your mistakes instead of relying on other people to discover mistakes. The whole point of open source is that, you know, what is it, like, sunlight is the best disinfected? It's like Linus Horvold's thing. and it's like well yeah you will be very slow to discover your own custom software dependencies because you only have your users and they won't report it to you but if this thing has 100,000 users worldwide someone will find something and then you can depend on it more right hmm yeah this is there's no basically I'm trying to be a voice of reason and you cannot be absolutist in both and I kind of trapped you there I talked to you into the talk track of vendor everything right And there is a bug there. Yeah, yeah, yeah. It's an interesting one. I wonder how much that, how much is that,

I buy some aspects of that, but how much of that is impacted by the fact that everything, I have this belief that most OSS repos, if not all of them, will just become software factories that are effectively agent-run in the future. What happens then where, like, I mean, even, like, even if you look at Warp, right, like Warp, which is a monocoding agent, of course, they're also, you know, selling a software factory product, but they are running their Warp, open source warp terminal as a software factory. And that almost suggests that there's a part where, yes, humans are still involved, but increasingly less so. Certainly not every single person is reviewing every single thing that comes into maybe the same exact level that we did in the past. Does that make this code base more or less vulnerable to securities? It's a bit hard to say, right? I think that's just where there's a little bit of nuance that I think we're still, and even the folks that I know that work on this test are trying to figure out exactly what that stable point is. Yeah. There's a couple of related arguments, and then I'm happy to keep going. Security is like a nonstop, huge, huge, huge surface area thing. There's one argument from Malta of Vercel who's like, okay, guys,

let's take every single highly dependent on software that's written in C, C++, port it to Rust. And it's like, we'll be faster, more secure. And it won't probably take like $6 million, but you'll get it done. And then there's another argument for, This also links to pacing the frontier from the model labs, who are like, okay, let's slow down so that we, or we'll do Daybreak, we'll do Glasswing, we'll make sure everyone's patched, and then we can let loose on rampant acceleration. So I think there's an inherent belief that the models will fix all security, and I think it's just unproven to me, because what if they introduce new bugs, and they're just different bugs? That's right. so I mean well I'm looking for pushback I'm looking for like well no you're an idiot like obviously models are way better at this yeah I guess like that's almost like

in the limit the question is like is the model do we believe the model will create perfect software right if a model is creating perfect software then you're done is there such a thing then that's done right then we're done here at all then we're just going to go home and chill on the beach for the rest of our lives I think there's also the question of is it good enough to first give it for security so let people patch then just delay the release and do it after or is there a problem of oh no it'll be much better at finding vulnerabilities you can full set test it for longer even if you've patched or like how long right you can give it for three months someone can try to exploit it for six months the problem is not solved right right right yeah yeah my whole problem with and again like this is a whole rant that we don't need to go into, but you name 100 less organizations that these are priority, they have access to the cyber model that you've just created 100 attack factors for me to go compromise. By the way, they're pretty easy to compromise. Yeah.

I don't know. Anyway, above our pay grade. It's always fun to get juicy in security. Security is right now very juicy. I think actually the security folks love it. because they usually don't get that much attention, and now they have a ton of attention. Is that true? What do you mean? I mean, that they don't get attention. I find that, at least from a VC perspective, you'd love investing in security companies, and have a pretty good whiz or something. It's done very, very well. Whiz and Socket, literally, I had this conversation with VC two days ago. Apparently, Whiz and Socket are the only security companies in the last five years that have worked. What about a normal? I don't know. It worked for a VC point of view. I see, from an economics perspective. Interesting. Not for me to say. Obviously, many great players in that field, but it's a tough industry, I think. So basically, I think it's not controversial. Best time for any security company is when a high-profile incident just happened.

Yeah. Right? That's right. Then obviously, you're going to sell a bunch of security. Totally. Yeah. So I think that's a fair thing. People should be aware. People should know the basic mitigations. I do think code scanning is important. But I just want to give people the idea that models introduce bugs that you wouldn't make as a human. And if you're not checking those bugs, then... It's not even that they make bugs. It's that you have less awareness of what's being written, right? Yeah, because you know you're reviewing everything. Yeah, forget bugs. Models are pretty good, though. have, on average, I would say less bugs. The problem is also that you don't even know what you're putting out. There's a lot of that. And that's the idea where we thought about mitigations and the combination of security and security as part of code review as well. I think this is also doing those checks. So I think maybe I would say one more thing here, or check my intuition on this

thing, is that not all parts of the code base the creator equal, the parts of the code base where security is very important is data storage and billing. So let's call it data billing off, and off is obviously related to all those things, and those basically don't bytecode, and then everything else will go nuts. Yeah, I think that's absolutely right. And it's really funny how, like, okay, well, if you don't know what the difference between off-end and off-Z is, please learn it. And then, you know, if you don't know, if you have never looked at AWS IAM, which you have, please learn it. Like, these are basic, like, conceptual things to understand that you can prompt for. Once you have the language, you can prompt for it. If you don't have the language, you're fucked. Totally, totally. Yeah, to understand that, I would even extend that to, like, just general, like, starting on a cloud platform of things, right? I mean, there was the, I forget which company had a huge incident where, like, it was just, like, a DNS issue that brought down, was it, do you remember this?

DNS issues? You're going to need to narrow it down. Yeah, I know. I think it was a prominent one, was it? There have also been many prominent DNS issues. I don't remember. We'll list them off. It's always DNS. So DNS is more like a reliability uptime. Sure. In security, I think people are like, did you lose money? Did you leak customer information? Yeah, yeah, yeah. I would probably put that above that for sure. Only DNS because of uptime. Yeah, general uptime. Yeah. Yeah. Background agents. Now you got your buddy from Qualcomm here. Your guests aren't necessarily correlated with that week, right? Yeah, I know. I mean, the nine one was a little bit of a scheduling thing where I've thrown in, you know, a lot and I'm just, I guess. I just isn it definitely week nine I was trying to get him at the end but I think it was just timing wise couldn work it out So it a bit of a weird one where I guess I not lecturing at all week nine But we coming back to those topics in week 10 so it a little bit staggered and a bit of a bizarre way there But, yeah, week eight, Rajesh, who has been leading a lot of the Infra stuff at Cloud Infra, DevEx, and beyond at Cloudflare,

which, as I earlier said, I think they're doing a great job here. and definitely their adoption for a company of their size and a public company of their size is fantastic. And then their content, I think, is also really good. I mean, this is fundamentally about all the ways you can use asynchronous background agents. I think I'm also going to spend quite a bit of time talking about the Stripe Minions, the Spotify, I forget the name of their version of this, the Ramp Inspects, things of that nature that I think are all kind of showing what does it look like when you scale this out. Plugback. Instinct. Yeah. Yes. Yeah, I mean, I guess less on consumer, more on, like, you know, software teams using them at scale. They're all coding agents underhood. It's true. Everything is a coding agent. Yeah, like, GrokBot is one of those interesting things where, like, it's very much presented as consumer-y. It's Grok. You know, it's Cursor underhood. Yeah, yeah, yeah. It's the Cursor team. No, no, for sure. So, yeah, I've seen the, like, you know, Cursor working in the, like, the dot-dot-dots, but they are not, they don't have, like, a code base integration, right?

They do? Okay, I didn't use that. I've only been using it for open table reservations. Exactly. I think that's the point of what they're trying to do. I see. They're trying to be like, well, you can use it for consumer stuff, but here's the extra power if you know how to use it well, because it actually costs our own. I think this goes to all coding agents, right? Like cloud code, cloud design, people using codecs, not knowing they're using codecs, right? a random code interpreter window pops up and you're just asking JTBT a question. At some level, there's coding agent under the hood, right? Yeah, so I'll give you another sort of practical example. So I run all my SEO and AEO inside of Codex and it has a code base, even though a traditional marketer would just check a address or something because I write all these scripts for doing all sorts of crazy stuff. And also I need to store data and also I need to visualize data. And pretty soon, as long as you're doing anything on an ongoing basis that has maybe a lot of steps or a lot of data, you need a code base.

Hmm. That is an interesting take. I think models also like building code bases. No, but that could be a temporary thing. I guess maybe my point is more on the interface, right? Are you going to go so far to say that a GrokBuck will have a place where the power user will be able to see the code under the hood? is that or is it well there is a computer maybe they'll go to cursorize I guess is that what you say well exactly because I'm like otherwise then then you basically cannibalize cursor like if in the limit what you're saying is Grokbot is the cursor then you don't even need like a cursor if all under the hood is just becoming it's the same team so they maybe right now they don't care but yeah it's all Grokbot yeah I buy that I mean I do accept that even things that seem outwardly just like non-technical use cases under the hood are technical can be like written and designed and loaded in code like that That much is absolutely true, right? The question is just more like, is a developer, is that their single point of entry? I think it's not even developer, right? Like you think of a regular person doing marketing that is using Cloud Code or just Cloud in the app

now is working in a code base that they would never know, never touch, but there's a code base manager. I think Cloud Tag is a good one for background agents. It's a big way Anthropic works, and they make it. You should see how they work. Yeah, yeah. Switch out. Okay, AI Native team. We talked a lot about these topics of, like, Gateway. We just put a routing episode out today. And MCP Portal. I would say MCP Portal is, like, the least established. Bullish on MCPs. Bullish on MCPs? There's not bullish or bearish. It just is. It's part of the landscape now, right? Yeah. That's a statement. There was a ball MCPs. So back, MCP is dead. People like extreme. MCP Portals is an unproven business. I thought about starting one. I actually pitched a guy on the topic on potentially starting one, and then I lost interest. I lost interest in a lot of things. But what's in there? So what I'm thinking here is if this is the week, more kind of geared toward scaling this out to,

like, what happens when you take some principles, and a lot of this was informed by, like, the conversation I had with folks that were, you know, leading A&A teams at respective companies, And so here, I think that something that's the portal really becomes interesting once you're at a certain scale, where you effectively have to have a single point of entry to do things like governance, security, auditing, cost optimization. And so, you know, I'd be curious to hear what you see as the future or maybe lack thereof of something like this. But to me, if you believe in an open router or a single point of entry for many things to happen and connect to and be effectively governed, I feel like there's a world where this also can easily exist. My only tweak on this is that this folds into the OLM gateway. They are the exact same platform. That I will give you. I can accept that. That I actually can agree with. And then by the way it also folds into the other thing of what did I say was off-Z, permissioning and all these things

and billing. To the limit, all infra converges to the same exact problems where they have the box in the middle and then you have the box in two sides. The God box will solve everything, right? But it's true only if you understand the job of the God box. And so separating MCP portals and our own gateways means you don't fully accept that they are the same layer of problem. In a very similar way, I also find that people don't understand database querying to have a lot of the similar things where they will separate filtering from ranking from or just like general recommendation. They're the same problem. And so then you just combine... Because when you take all these things, you combine them in the same layer, then you can sort of cross-depend on things in useful ways. If there are different layers, then they cannot. I actually will not disagree with this. I think maybe the... Yeah, so it's slightly pedagogical. I think that this breaking up of separate terms versus one, I think it is absolutely correct that the underlying

architecture is the god box where one thing goes in and there's a bunch of other stuff that is some third party, some first party that all kind of exists on the other side of the God Box. So yes, I agree with this, actually. I have a spicy tweet. Oh, God, where is the God Box tweet? This is the MxN to M plus N architecture. So anyone working at Infra needs to understand. Congratulations, you now have a job in infrastructure or you want to be an Infra engineer. You need to understand that your entire job is making God boxes and then taking them apart. So either, your problem is either you don't have enough, you don't have a good God box or your problem is someone else's God box you don't like so now you have to take it apart and make your own. That's funny. That's very funny. I like that. That's very funny. Into the heat death of the universe. Wow. That's funny. Which is, this is a restatement of like, there's only two ways to make software which is you bundle them, you unbundle them.

Right, right. But like, what are you bundling? It is the things that go together, hopefully. And then maybe over time, people's opinions shift as what goes together, what should not go together. Right, right. So there is that whole thing. Maybe Monaco should be more of a point solution. Maybe Monaco should own everything, right? And there's a question of, in some ways, you're bundling for the kind of startup segment that you serve. In some ways, you're unbundling from Salesforce because that's how you win market share from Salesforce. You're unbundled from Salesforce. that's interesting that's an even broader statement connecting it to broadly all of vertical AI and now bundling unbundling, that's like the whole Craigslist and Airbnb Tinder and all those are separate Tinder like personals basically yeah I thought about it that way but yes, one of those things so we just shipped the open router pod today and open router huge success, lots of people

try to build gateways. None of them are successful as OpenRouter. OpenRouter only builds gateways. So there's an example of they disagree with me that the router layer should also do ops because Arise is doing ops and observability and what have you. And the MCP gateway, they don't have that. They only handle model routing. And so I love being wrong like that because they're like, we are the point solution and we only do this one thing really, really well. And that's our engineering focus, which is good. It's also the marketing focus, which is nice. I think what you've described of the one-stop shop for all the things is ideologically what makes sense. But I think what ends up happening in practice, and part of it is probably also marketing, is that especially when you look at the enterprise, this is broadly under infra, broadly under platform, but then there's separate teams owning different parts of the stack. And I think part of that consolidation at the scale of an enterprise,

if we're going to say MCP portal and the gateway, someone has to come together and say, like, now there's a single endpoint that you're hitting for both of these things. And that fundamentally is, like, a hard thing to, you know, get on board with. So oftentimes it's more, like, political and even, like, people problem rather than, like, from a pure software perspective, this makes a lot of sense. Yeah, yeah. But that's not it, right? Like, software systems sometimes are brought, like, by people problems more than pure software, right? Yeah. My framing of this is typically, you know how you ship your org chart? I forget what the name of this principle is Conway's Law you ship your org chart you buy your org chart when you start a startup or you make a tool who in the company are you best friends for and if it's for everyone it's for no one that's interesting so a conscious part of latent space in AIE is like there's this category of software engineer that really cares a lot about AI is going to make it their day job. Let's only make it for them. Right.

Right. Not a bundling of engineering. Never going to be the dev everyone. Yeah. Look, we talk about databases. We talk about security, but there's an AI angle to it. We'll see. That makes sense. One thing I'll just shout out as well, related to the point, is I think role of the future that is increasingly important, platform engineer. I think this is a huge one. You know, okay, more lore. Temporal, my job before all this, I would have, yeah, so we wanted to own Platform Engineer. If I hadn't started AIE, I would have done Platform Engineer. Because that is the role that decides all of this mission-critical stuff, and then the kids over in applications can do whatever they want on top of our platform. But, you know, the big boys do platform. Yeah, and I think in the modern day, they own cloud, they own DevEx, they own security, psych ops. So I think it's all of the things. Like, everything where you can actually, like, lose the company or, like, make or break, like, yeah, platform guys. And then product guys can do, you know, adjust the CSS.

That's saying. I mean, that's saying you're starting a fight here. I mean, I'm not the CSS adjuster, but I appreciate everyone. I mean, it's a fun time. What's the future? I feel like all the guys can adjust all the things. Yeah, yeah, yeah. So there's a few aspects here. Okay, so there's a little bit of spillover from the previous week. So yeah, coding agents, big teams, some of the concepts. But then, you know, we'll touch on the software factory. I want people to really start thinking about this world of this machine that builds the machine and where, if at all, are people still in the loop there. You know, today I think the answer is some combination of intent and kind of review or maybe the places, like sort of intent where you're defining the specs and what needs to be built and review where you're doing some amount of your own checking of the system. And, of course, this also. I think that bottleneck is going to be loosened up before even the intent piece does. But for now, those are the two choke points, I think, in a fully automated system. So, yeah, I just want to give people's voice to that. For one of the USS partners, actually, Warp, they're doing this thing as part of their contribution to the USS program

where they are going to essentially make a version of Warp that is running to the factory, and students will just be, like, building the factory, contributing to the factory, and keep it running. Like work in our factory. Work in our factory, exactly. And so that's going to be one of the ways. and then they're going to find things that are interesting there and pull it back in. This is one of those things where I'm not sure. Obviously, Factory itself is very keen on this. I feel like everyone wants to build their own. Yeah, I think that is a good question. And I think they're, of course, entitled to try and sell the products. I think it's a hard system to own its entirety. It's a lot. Decide everything for me, please. Yeah, it is a lot. And I haven't gone too deep in understanding how they approach their GTM here, but I suspect you would have to do it piecemeal, right? You'd have to be like, hey, we first hear your coding agent. Oh, then also, like you basically land and expand, right? I mean, we talked about this bundling and unbundling piece. You don't start by saying, here's your whole factory. There's no way you could get a Ford on board with that, you know what I mean, where you're going to go on and just plop the whole thing on their desk.

That's at least how I would think about it, is you first be their coding agent of choice. You build out some background agent for them. you, you own some of their SRE pieces, and then you're like, well, now we can just weave it all together if you define everything in code or something like that. Or like, you know, configs. There was a fun blog post on this from OpenAI. Our buddy, Ryan. We had Ryan on the podcast. Basically, they spent five months making an internal tool with zero lines of manually written code. They kind of walk through, one, it's called harness engineering, so you can learn harness engineering. but the learnings of there was kind of a slump or a slumber at first and then you offload and you had peer review. At this point, it's from February but it's a vision of a software factory. You are full sending, letting agents fully write everything and how to do that well. Yeah, absolutely. Some version of this is going to exist. To fix this point, I do wonder about the sales piece here,

like the actual business of this. That's an interesting question. But from a principal's perspective, I think this is a conversion of the future. And then beyond that, I think there's going to be some amount of dabbling into and speaking about, like, okay, what does it mean to, yeah, what does it mean to be a developer in the future, right? Like, are we, people talk about taste. People talk about, like, you know, understanding of systems and architecture and, yeah, like software taste. I want to, like, go a little bit deeper here on maybe starting to, like, at least put some answers to the question around this. Yeah. So this is your answer to where AI software engineering goes next. Yeah, so I mean, the factory is like the tactical, but I think beyond that there's a question around the role, right? Like what does the role evolve into? If the factory is running and, you know, full steam ahead doing its thing, the conveyor belt is churning out code, what do we do in all this, right? And I think you and I, we had this discussion with Jerry at the dinner once upon a time a few weeks back with Dex and Jerry around the factory. But yeah, I think that there was this part of, at least the discussion points there, which I'm pretty much on board with, is like

increasing levels of abstraction and then somehow there's someone still, even a CEO of a company has to have enough oversight and understanding of different constituent parts to be able to make a judgment on direction, right? To be able to course correct, to be able to set metrics and things of that nature. I think there's a version of that translates of that. There's a version of that analogy that translates to someone who oversees the technical. We call it a manager of agents. We call it different terms. They're all buzzwordy, but in some respects, someone who understands product requirements, constraints, technical constraints, loops through all of those and knows how to adjudicate. I will say that in the chair before you, we just had Anthropics head of products and head of engineering for the platform. I was like, when are your jobs going to come by? They're like, not yet. If Anthropics is not there yet, you should not feel like it's imminent. Good. Did they say how long or are they just not yet? Not yet, but who knows.

There was definitely a lot of like, okay, I'm doing her job, she's doing my job. I think it's also an interesting perspective for students. You said there's a lot of sophomore undergrads. You've got two, three years to go to plan out what you want to be doing. There's different ways to look at it from that, right? You're not straight in industry. The thing that I care, I'm thinking a lot about is, one part of this discussion that resonates more with me is the loop stuff, where like, okay, you started writing prompts. Now you no longer write prompts. You write prompts, you write prompts. That's right. And then you just keep going up. And I think to me the sort of recursive concentric circles nature, going outward but also going inward, is something that builds the factory. I think you build up to the factory, not build down. I think a lot of people will eventually start building down. You start from the top level, like, what is my goal? And then you figure out the systems. That currently creates a ton of swap, and you should not do that. But someday you will.

That would be my question to prompt you. What is the sort of prime question that people exist to solve? That is your thing, right? So let's say an engineer today, you are measured by your PR merge rate, PR contribution rate. Well, two years from now, that's going to be ridiculous. Okay, well, what's your main thing? Money? People always go to money too quickly. It's something in between money and PRs. Test process action, test scores from your features. Even that's like a product decision, not really a code decision, right? Like, I don't know. But if the engineering is evolving into, like, a PM engineer, a crowd engineer, you know, hybrid, then I think that's my answer to the question of, like, what does the role evolve into? I think it goes without saying that I would today recommend to students spend more time thinking about holistic systems, not just your, like, single part of the engineering puzzle, but how that ties in with, you know,

and this is, like, in practices of happening, like, students for their capstone projects, I encourage them to go out and, like, in fact, would I give them as the seed of ideas for final projects? is the YC request for startups. Someone was like, go build a company. You know what I mean? Go build something interesting. What did people build last year? I mean, a variety of things. Someone came to the E department, was in the E school, was studying EE, and they built effectively a translator for parsing of manuals that have memory addresses in C and then converting that into code that they could then interpret and even visualize. so it's not specific to their use case someone was like a designer and is now actually at Vercel and she like had you know like take a picture of my design notebook and generate me an app based on what I take a picture of right so literally she was like for her demo I was like here you go snaps a picture of like some random like scribbles that she was doing and she got a fully fleshed out Vercel app and from that and I think because of that probably got a job at Vercel doing that doing some versions that's a high bar actually Vercel

very known for their design engineers yeah yeah and I think she's now working on Eve from what I remember No Eve in your... Eve is an OSS partner. One of the ways for Siles partnering with us is that they are having students build with Eve. This is the concept of meta-harnesses, right? Harnesses of harnesses. You can always go one level deeper. I do think it makes sense, at least for me, in the same way that model routers wrap model APIs, you need to wrap harnesses as well. That's the opportunity. I think at some level, students should also stay grounded and like there's a lot of stuff happening at the edge there's also a lot of companies that are still like single call chat gpt in production playing around haven't tried it um you know there there is a lot of that so there's there's still time for what to work but i think your point on taste is good have taste try stuff like uh whether each company has their own version of a software factory uh you're building tools to help companies

do it, like staying on top of it, actually using the tools, playing with it, like you'll, you'll figure it out along the way, you know? Yeah. And I think the, the, uh, a hundred percent of all that, I mean, first to your point that are a lot of companies still in the prompt chat GPT or some version of, you know, chat GPT, maybe they just got a cursor license and are just now getting people on boarded. Um, that is way more true than we believe certainly here in the Valley. Um, that is, I can, I can say 100% confidence. Um, but yes, I think for the students being exposed to this, is just at a certain point, the fact that they know this and are on the edge is what will eventually allow them to also be some of the more interesting and more valuable contributors when they go to maybe places that maybe aren't up to stuff in terms of being on the frontier. Yeah. One of this kind of progression-type talk, still very influential, about a year old, Steve Yagi, talking about the levels of cyclists. He's all the way through level 10 now. 11. He's given up and he's moved to a farm. But yeah, you want to say, Sorry. I'm making fun of you.

He built Gastown. Yeah, I remember. All these things. Moved to Gastown. Now he's burned down the city. He'd say, I'm building my game. No more Gastown. I thought that, I mean, I remember talking to him at the engineering fair earlier this year. And wasn't his whole thing, like, the only useful thing that was built at Gastown was Gastown? Yeah. Right. That's a horrible failure. Horrible. Totally. Totally. And so, like, people should be aware that, like, psychosis is real. and if you're not holding yourself accountable to delivering real products, you're just screwing well with all of them. Yeah. This is like a, maybe, I mean, relevant to it, but I wonder how much of, I mean, I played a little bit with Gastown, very involved system, maybe a little ahead of time, but I'd also wonder if they were like bottlenecked a little bit by the marketing in that it was a very difficult thing to grok from a terminology perspective. Like there's just a lot going on. It made sense to him. I think the Gastown marketing also is one of the things that propelled it, right? Like, what was right before Gastown? Beats.

Yeah, Beats. Unless you're talking about someone else's stuff. No, no. There was the whole open claw. There was something else, and there was Gastown. And the whole manifesto also really helped spur it into the conceptual. The people that would never use Gastown but learned about Gastown because you can read the manifesto. There's a marketing angle to it. Yeah, I mean, so Steve is not necessarily someone to take literally, but directionally correct, you know? And so I think he's directionally correct on this one. And so if you're at the IDE stage, he's trying to move into the power user stage. If you're a single agent, you're trying to move to multi-agent and blah, blah, blah. And you probably should try to build your own orchestrator, right? And that's all that. Arguably, this is kind of out of date now. There is more to the software factory beyond this. but this article predates the software factory. I do think one of those things where I'm open to, as an employer, I have this problem in my company today, which is I do not want to hire anyone who hasn't been through psychosis

and come out the other end, because I am currently paying for somebody to go through psychosis. And it's very expensive from his salary, very expensive from tokens, and I'm getting nothing out of it, and my feedback doesn't get through. and so like it's almost like I would encourage your students to have psychosis and then come out and go like okay what was wrong with me right because like that's what Steve just went through like a very public two year journey of this yeah it's weird on the student side there's like a lot of industry you think you're being productive you're not it's probably the student version right you're notion charting and making all these optimizations personal ass and it's like dang I went through the psychosis but nothing has changed right right so go through it go through it right Yeah. I haven't seen this. I've seen a version of this from Boris, actually. Obviously more geared toward Claude use cases, but I remember he made a Claude artifact, which showed I think more at the team level, but I think he had four stages, not eight stages, but it kind of roughly followed the same progression of no adoption through to... Yeah, only four is good.

This is too many levels. Yeah, well, we'll see if Viva can pull it up. But other than that, zooming out, have we discussed all the things? That's the one. Oh, that's one? Okay, here we go. It is an architect. Okay. So it's gated, assisted, parallel. What's the last one? Supervised autonomy and AI native. So it's zero index. That's five levels, right? All right. I think so at zero. Yeah. Wow. Actually, it's pretty good. I need to be through this in more depth. And use AI native. Yeah. I would give a shout-out to dynamic workflows. This is one of those things that is a sub-agent pattern that the OpenAI guys are also exploring. that is clearly fitting. And so basically, you're writing a harness on demand for the thing that you're trying to do. And so this is maybe, like a lot of sub-agent conversation is just you're copying the parent agent, but you actually should be writing way more code and then running that as a sub-agent.

And that is what the end topic people have chosen to call dynamic workflows. Okay. But, you know. Yeah, we built a version of this as well. Yeah, it's a good pattern. for monaco what does it do yeah i mean on the fly sort of like harness sort of use cases for yeah oh that's great yeah um i will note that when boris wrote this it's for team adoption for people that are listening and also he says you know anthropic is on stage three we're still trying to get to stage four wow he's like i am at stage four i'm the guy but you know it's it's not as simple as steve's like steve is a very good personal do this do this do this uh this is a much more if you're running a team and you want to AI pill your team and not just be a user of one. Anthropic is still at stage 3. Don't be like... You should teach this. This is great. Everything that we talked about. I can't believe I haven't come across this. Very, very good.

Okay. That was a wild tour. One semester of Stanford in one hour. Thank you for doing this. Any parting thoughts? Any call section? No, this is, thanks for, I mean, a ton of fun. It's been nine months in the making. I'm excited. This is end of week one. A lot to do, a lot to learn. I'm excited to see what happens with, especially the OSS partners, and just getting students deep in the code of some complex code bases, having them build some stuff. So, you know, it'll be fun. Keep you guys posted. You're doing another version next year? You need credits for students. If OSS people want to get involved, they can reach out? Sure. Those credits matter. Do they want it? Do they actually want the credits? You have credit cards. I don't think a lot of students want to spend that. I'm looking at you. I think. I remember there's a certain number of credits for sure, but then on top of that, he's like, look, if they want to go hog wild with this, let me know and we'll have these one-off experiments where they can just scale it to a thousand plus agents. This is Unifarm Factor.

Unifarm Factor. Those kind of people are valuable. This is Wilson from Cursor, who is like, I'm going to write a browser. Those people who are ambitious and can actually execute, those are very valuable, and that's basically a recruiting call. That's right. Yeah, that's a good point. Yeah, but I do think students should understand how to be ambitious because respectfully snapping a photo and turning it into an app is not that ambitious in this day and age. Writing a browser is, and there actually is a lot more projects like that. Yeah, I think that's right. It's all about raising the floor. I mean, I think, yeah, last year it was, you know, credit to the students of last year, but, yeah, this year, obviously, we're going to expect more. That's a fail now. We're going to expect more. I mean, yeah, but that's the whole point, right? We're leveling up, right? Last year it was like, yeah, hey, have you heard of prompting? Like, here's what it is. I spent some time talking about, like, you know, K-shot prompting as one of the lessons, and, like, I don't even think you guys even think about that now, so. I think there's a version of thinking you stick out in your cohort of classes,

and then there's also a version of you're also in early engineering roles with a bunch of people in industry, people still trying to get jobs. It's not to call out one demo, but you can do a lot to stick out. I'll give students a freebie of week 11. Make benchmarks. Make benchmarks and evals. This will get you a good one. It's going to get you in the door of every frontier lab and every top agent lab. Good engineering blog posts with some rigor on anything interesting. There's a lot of benchmarking, a lot of hanging fruit there, but the engineering work posts do really well, right? You went deep on one thing. Yeah, and probably also there's a lot of money to be made in that as well. So not just in the doors, but if you make that into businesses, you can really, really cash in on that. Yeah, people don't know. C-Bench was an undergrad project. Yes, yes, that's right. From Princeton, right? Yes, yeah, yeah. great okay awesome that's it

thank you so much yeah thank you guys