2026年のソフトウェアファクトリー標準 — Zach Lloyd（Warp.dev）

Hey, it's me again. I have heard a lot of positive feedback about our episode intros, so I'm going to keep trying to do them as far as time allows. Today we're going to dive into Warp with Zach Lloyd. Zach has been working on Warp for a while and it's pivoted into what it is today, which is basically the definitive software factory that is also open source. and he has this whole definition, I think, which has done really well. His talk at AIE has been going viral as one of the top talks of all time. And I think, effectively, everyone is trying to set this up for themselves. They have a solution for you, but it's also kind of an open spec that is becoming table stakes for what you're going to want. And I think you're going to want cloud, you're going to want a control plane, you're going to want some other sort of basic security features and team features that you take for granted. So we're releasing this episode as part of a preview for AIE Code. AIE Code is the last AIE flagship of the year.

It's going to be in San Francisco where we're gathering all the top agentic engineers and talking about software factories, among other things. System 1 models, Diogo is coming back and raising your ambitions. So Tariq and Jason and Brandon from Artcraft is doing his first conference talk ever. So if you're interested, Zach will also be giving a great keynote on software factories and sort of recapping the year and previewing what's next. Join us, and it'll be lots of fun. See ya. We're in a remote studio, catching up with Zach Lloyd for more. Welcome. Good to be here. Thanks for having me. Zach, you've been going viral on AIE. What's up? You've hit a nerve. You've hit a nerve. People are really interested in software factories, and that is an area that I've been focusing on, that Warp has been focusing on. I think we have some strong kind of interesting takes on what a software factory actually is and what it means for being a software developer.

So, yeah, and people have been into it. Your H1 on the landing page says open infrastructure for cloud software factories. I think maybe since you spend more time than I do talking about telling people about factories, what do people think is a software factory and what do they actually need? When I talk to companies or developers, most of the world is trying to build some sort of internal software factory or set of software factories. What I think it means, or maybe I'll just define what it means to me, is like a agentic system in the cloud where you're defining what are the roles of various agents across building software for your team or for your product. And there's a certain amount of autonomy in that system. So it's like the agents can do things on their own. And that system to me should be something that's very measurable, very closed loop, which over time you as a team can like tune, measure, benchmark, and just ultimately get confidence that as people switch to this way

of doing agentic development, that they're doing it in the most scalable and efficient way. And what do people come to you with that perhaps is a common misconception, right? Like, they're like, okay, if I do whatever, if I buy whatever, I'm going to come out the other end with, like, this amazing money machine. Yeah, so when people come to us and they're trying to build their own software factory, like, often they will be, like, stringing together different automations. Sometimes people are conceiving of software factories as just, like, Just like not a specific thing, but it's just like we're using agents a lot to ship software, and we're calling that our internal software factory. And what we try to do to make that more concrete is be like, no, no, no. You should actually think of a software factory as a very specific thing. Like you should define your factory in code. You should make sure that the factory has all the workflows, skills, MCPs. everything is like it's like a concrete thing that you should be able to run at some point in time

and measure how it's doing because people have all sorts of kind of random ideas around what these factories are and so for for warp factories which is our platform it's like well we say no no it's actually it's a real thing you can orchestrate it you can measure it you have one factory per product surface and we try to like take it out of the realm of this being like this nebulous uh it's collection of like arbitrary automations and interactive agents into a thing that's an actual well the way to think about a software factory or the way that we think about it what you're looking at here is warp software factory platform is a factory operates on some product surface so imagine you're building like your mobile app or you're building you know your server app or whatever and the factory itself is operating on a set of repos and has a set of skills it needs to a set of MCPs, and then it has like a workflow associated with it. And this workflow, you know, roughly matches what, you know, your team is probably already doing to make changes to that code base.

So there could be a step where you're doing triage. It could be where you're doing specking, implementation. The important thing is like there's some kind of defined workflow and that that workflow itself is defined in code. so for us we basically have a yaml file which says okay these are all the repos that you can be using these are all the mcps all the integrations that are hooked up so you basically have this whole thing defined someplace the way to think about using the factory is that there's actually sort of two personas so there's someone who is building the product they're doing product engineering they're shipping features through a factory and for those people the way they use the factory the way that is through their existing tools so they could be slacking the factory to put work into it they could be using a task tracker like linear jira they could be interacting with the factory through github they could be calling it via api or mcp and the metaphor is like you're

putting a work item into the factory and when it goes into the factory that starts like a predefined workflow and that workflow is defined in code that makes sense so far i love defining things in code yes so okay so you have this whole thing set up in code and then once you have that in place and so the again metaphors like someone starts a task in slack or in linear or through an automation like you know there's a server error it all goes through one system is the kind of key thing and once you have that working you basically get a whole bunch of benefits that flow out of it and so one of the big benefits is that wherever these agents are running and you're sort of seeing a real-time view across warp's internal team how are we using our own factories you get a full trace of what everyone is doing with coding agents across any product that has a factory for it and this is cool not just for like oh i want to see what people are doing like i want to see what Softfic is doing, which is cool, but it also starts centralizing all of the agent conversations

and agent artifacts and all of the work that is being done by coding agents across your team. And so when I talk about like having a closed loop system, these agent conversations, these agent traces are like the key aspect of how this system improves. And so for instance, just to look at one of these agent traces. Sathvik is working on our mobile app here. Sathvik was doing this in Slack, so I could pull this up in Slack if I wanted to. But the thing is actually generating, it has a full record of the conversation, it has a full record of all of the, these are computer use screenshots that are coming out of building this thing. You can see a full record of exactly what the different pieces of the orchestration agent were doing. And so you have this, but you have it not just for one thing, you have it for every single thing. So on your team today, when I talk to companies, for the most part, they don't have something like this because most of their engineers

are individually using Cloud Code in their terminal or Codex, the Codex app or something like that in their own local setups. But they want to get to a world where they have this centralization. When you have this centralization, you can basically get all this interesting data on how your agentic coding is working across your company And you can think of this data along different dimensions So you can think of this data as like okay what our team output in the world of using coding agents And so that's things like, you know, traditional Dora metrics, how long does it take to ship something? It's things like how automated are you? And so we have a sort of metric around like, for every PR that gets merged, how many times did a human have to do something with it? what's your throughput it's metrics around cost and so we're using this like constantly and when we talk to companies this cost thing is very very important where you want to be able to tune your factory with the right models the right skills to basically bring this cost per pr down by time

you want to be able to slice the cost by person you want to be able to slice it by automation by by where in the stlc are you spending money the final thing that's that's really cool once you have this sort of closed-loop system going is that you can actually look underneath the hood at how well your coding agents are doing along different dimensions. And so you can think of this as like online evals, or we call it scorers. And these can go from very general things to like, are your agents being too verbose? Are they being efficient? Is the code quality good? We have work to do on this. Do I as a user define this, or do you provide this as a platform? Yeah, so both. So we provide out of the box a set of ones that we think are like generically useful. But then, you know, as we work with companies, you know, they may have specific failure modes for their agents that they're interested in tracking. They may be like, hey, the agent making a specific type of thing that is like change.

It is like breaking our build. They may want something, you know, very specific to their workflow. So, again, this is all defined in code and all customizable. And what you do is like when you set up one of these, what's actually happening behind the scenes is that there's some set of runs that you, you know, when I say runs, I mean like historical agent conversations that have been stored that an agent then goes and like retroactively grades. and so this is a good example of one where it's grading for redundant tests which is like this is a common failure mode if you're using coding agents like sometimes they write a bunch of tests that you don't need they write it in the course of like figuring out how to like implement something and then they leave it in the code and so this like blows your code base it wastes tokens and so you know say you wanted to check for that you you know you basically create one of these evals and just to show you like a little more about what the eval looks like the eval is again think of This LLM is a judge, so you're picking a judge model.

You're then seeing, okay, where do these run? What's the prompt for judging it? And then a sampling rate, because you might not, like, these are agents that are running. You might, you got to be aware of the cost of them. But when you have this set up, think of it then as, like, you have all these smart agents that are looking across your team at how all your coding sessions are going, and they're saying, okay, sometimes there are agents that are writing too many tests. And so you get this record and this understanding of where you should look to sort of improve the way that your team is using coding agents. So far, so good. Because I want to show a couple other things just to make it more specific what you can do with this. Overall good. I'm a little bit wondering if basically there's different levels of warp where this is the CTO level, what you're showing us. right? You're running the company, you've got a bunch of coders. This is like, the people who use this, it's partly the engineering manager, the CTO,

but it's also like very commonly like the platform team, but it's not your like day-to-day engineer typically. But like when we work with companies, you know, they'll often have an internal team that's been trying to like, it's like the DevEx team or the platform team, and they're trying to figure out how to optimize coding agents. So what I'm showing you here, yeah, is like the tool set that they are using to be like, how do we get the most out of cloud code at our company or codex or something like that? That part is clear. Yeah. Cool. Yeah. And so like a couple other things that I would just want to emphasize, not to make this just like completely about warp factories, but like the general principle here for anyone who's trying to build their internal factory, whether it's on warp factories or not is like you want to get to a spot where everything is sort of stored. Everything is observable. Everything is measurable. Because that's when you can actually start to apply engineering rigor to how you make coding agents better.

You probably see this on Twitter all the time. Everyone is always tweeting about, like, you should check out this new model. You should use this new harness. What's better? Is it Pi or Cloud Code or Cursor or whatever? And so what our opinion on this is and what the world sort of needs is, like, out of the realm of vibes for understanding these coding agents and actually into the world of measurement. And so we think that this is the direction that companies are going to want to go. They're going to want to, like, put in place a system that's like Warp Factories, whether they actually use Warp Factories or they build their own, which collects all this information and then provides all this way to see how your coding agents are doing. I guess one other thing I like to kind of take for granted that's part of this system is like you can create all these different automations around it. So like I mentioned people work in Slack, but people are also setting up different trigger-based ways of putting work into the factory. Once you have this view that I showed earlier of like scoring how your factory is doing,

you can also set up another kind of loop of agents and everything is kind of loops these days where those agents are looking over they're looking for failures in your scores like they're looking for things that aren't going well and trying to suggest changes to your system to make it go better and so like a really simple example here we call this self-improvement is like okay our self-improvement agents were looking at how well our triage step was doing. So we have a triage step and they're saying, okay, we have 54 failure cases of this particular form where it would have been better if our triager had been able to spawn a subagent to do its investigation. And so what they're able to do, and like, you can go look at like why it thinks this if you want. But the cool thing is because you have your whole factory defined as code and these are coding agents, they can essentially make a pull request against a factory definition to try to improve the factory itself

and so this is like a like a full learning loop that's built into the system to try to actually self-optimize and like we don't have this yet but the next thing that we want to do with this is like actually for any suggested change like this you could go back replay the runs and make sure that this actually would have improved them. So the possibilities of like how you can engineer this stuff get pretty cool once you have the right primitives in place. And I'll show one last thing, and then we could sort of maybe talk more generally about the whole approach here. But the other thing that is, you know, I think companies should strive for that we find very useful internally is like the ability to actually do what we call benchmarking. And so benchmarking in this context, in the factory context, means since we have the ability to take any coding agent that's running your system, you can sort of add it to a benchmark suite, which means you might curate a set of coding agent runs that were related to doing front-end coding or something like that.

And then once you have that, you have the ability to sort of replay those same tasks against a matrix of different configurations. And so, you know, I'm a big believer in not just allowing you to vary the model, but also to vary the harness. And so, you know, in Warp, we have our own harness, but we actually will let you benchmark against Cloud Code or Codex or Pi. You can pick the model. And when you do something like this, what you end up getting is a result that looks very similar to, like, the published results for Sweebench or Terminal Bench, but on your company's own real runs. For instance we looking at a cost versus correctness tradeoff for a certain set here And like you want to be in the upper left corner so you want like lowest cost highest correctness These models are now slightly older because the world's changing so fast. But the idea is basically, you know, we could see, okay,

GPT-5-6-SOL here was like 95% correct on this task set, but it cost $6 on average, and GLM-5-3-SFLASH was 93% and cost $2. and then the cool thing about this is you can take this information and feed it into a model routing strategy or harness routing strategy, honestly, that's defined specifically for your own workflows in code base. So if I had, again, to zoom out, the really big picture thing that I think is so cool and powerful here is you're getting out of the realm of guessing and you're getting into what we call factory engineering, where it's like you are taking a real engineering mindset. You're like, how do I, we have this amazing new primitive of coding agents. How do I know as a team, as a company, that we're actually deploying them in the right way? And so, you know, we're trying to build this infrastructure and tools for companies to do this. And like this approach is really, really resonating with a lot of companies that have been trying to build, you know, their own versions of this.

So that's like a quick tour of like, what do I mean when I talk about factory? I can sort of stop talking for a sec. I do think that, you know, the power move, I guess, is to sort of thought lead so hard that everyone ends up trying to build warp even if they don't use warp. Yeah. So, you know, you have on your landing page something I like. In your talks, you've done a few of these as well. Something I really like also is any surface, right? It really has to be any surface. It has to be a factory's code. You have to have the eval loop, like... What is the whole, like people calling this the ADLC. I don't know if you use that term. It looks like you use SDLC still, you know, all that stuff. This is how I would think about it. Again, whether it's warp or not, if it's, I would obviously, you know, warp is this is what we're fully focused on. We have an awesome platform for it. But just in general, what I would think about if you are trying to build something in this space is A, I would view it as like a stack. So as infrastructure at the bottom of the stack,

I think it's really foundational and important that you define your factory declaratively as code. So again, the analogy is like Terraform here. You get the same benefits, but actually more. So the benefits you get from defining it as code is like you get a versioned view of like what was your SDLC, AI, DLC, PDLC, whatever DLC you want. You get a version view of that, and then you get the ability to create branches off it, which you can test. You get the ability, if you change it, to revert it. So all the obvious things about source control that are great. And then you get the added benefit that's extremely cool, where because it's in code, coding agents can suggest improvements to it. So to me, this is like the foundation of a self-improving software factory setup is define your factory declaratively. The next step up in this stack, as you think about architecting it, is someplace to gather all of the information that agents are producing.

Because what you want to be able to do is examine that information, look for failure modes, or replay those same past tasks with different variables to sort of, I don't know, scientifically is the right word, but from a measurement standpoint, really optimize your system. So that's the mindset. So you need the data to do that. You need the agent conversations and you need the agent artifacts. You need a record of all of the interactions from all of the tools to get to a result. And so what I mean by that is like you might have a factory conversation that starts in Slack and then continues into linear and then continues into GitHub. And there's a code review comment. and that whole interaction is going to result in some piece of merged code. In fact, it might even result in something that gets shipped and then has a server crashed. And you want the ability to retroactively go back with an agent most likely, but you could do it as a person and diagnose how could that have been better.

What was the failure in it? You also need, this part will seem fairly obvious, but you need a way of getting data into the system, so you need some way of plugging in MCPs. And then we find that as companies get bigger, They want different access for different parts of their organization. So they want things like what MCPs can in the marketing team access versus the sales team versus the engineering team. And so, like, again, you'll find there's a bunch of complexity depending on how big your organization is if you're trying to roll something like this out. You really want a compute layer. You should think of that as a separate thing. And so there's obviously a lot of agent sandbox providers at this point. So if you're going to use a third-party one, I would just make that pluggable. We use Namespace as our primary one. Ooh. The launch of AI Europe. Namespace is awesome. But this should be pluggable. I think that's really important. So if you wanted to use Daytona or Modal or E2B or whatever, like, I would architect it. So this is the pluggable component of it.

I would make sure, depending on your requirements, that you can do different platforms. So, again, by default, you know, I think you'll probably want to use like a Linux type sandbox. But for computer use, like the way that we test like iOS apps is you're going to want to have the ability to run a Mac VM and actually like have it run the simulator and click around. We have a very open approach to inference. And this is like this is very much with our ethos of trying to be super duper flexible. so we find that a lot of companies right now will have pre-existing commits with anthropic or open ai or google that they want to burn down and so we're we provide like a very big variety of like you can plug in your own inference provider or you can buy inference through us you can even plug in your own harness which i think is a pretty big deal so you know you can orchestrate Things like Pi, like I mentioned earlier, basically any ACP compatible harness.

And then you want this improvement layer. And so this is some of the stuff that I showed in the demo where it's like the real power of this whole system is the ability to improve it. And so just to give some stats at Warp, when we first started using the factory approach, it was costing us about $80 per PR because we were using like Opus where we shouldn't have. our triage agent was using like a really high powered model you know we were doing a bunch of redundant work with our the shape of agents like we were doing too many sub agents relative to just having fewer agents and so we basically did a whole bunch of internal tuning and testing using these built-in features and we were able to get the cost per pr down to like 20 bucks or 15 bucks something like that well we can measurably improve the throughput of our prs over the last month it's gone up by 40 percent we measure like the revert rate and it hasn't gone up and so i really feel like a lot of confidence in this general approach of like observe measure tune finally you need some

sort of like orchestrator sometimes people talk about the orchestrator like it's the whole platform i think it's a piece of it what i mean by orchestrator is like you need some system that will take the factory definition code and then actually move work along the different stages calling out to the right models, loading the right skills, et cetera. You need something that will fire the right automations. And then finally, you need a bunch of access points. And again, our philosophy with this, which you can, I could see people have different philosophies. Our philosophy is to sort of bring the factory into a company's existing knowledge work tools. So think of like, you know, the most obvious one is Slack. And like we're doing like 60% of our work is starting in Slack. But sometimes it makes sense to start in the task tracker sometimes it makes sense for a developer who is more comfortable using like cloud code in their terminal to do a bunch of work there and then submit a spec into the factory. And so, you know, our philosophy is to try to

make this whole thing API driven and MCP driven so that the factory itself is like a piece of kind of headless infrastructure where it has all these different entry points So I think this stack view of it when I shown it when we talk to companies, seems to land very well. It's like, okay, yes, this makes sense. This is a new thing in the developer infrastructure stack. It kind of sits one level back from where CICD is. But I think most companies will have something like this for software development. Yeah, that's a lot of layers. actually i've never actually seen it spelled out like like you have i wonder because i think most you know i advise cognition and i've seen i you know cursor what all these things they all try to do some bundle of like the vert like from bottom to top and there's a little bit of bundling and bundling with all software and so there's a critical question of like look i'm trying to do my job here as CTO of a, you know, B2B SaaS.

Like, do I need, like, all these, like, six layers? I think it's, like, who's this for? Maybe is kind of the question. So when we talk to companies, where this resonates the most is it has to be a company that has some amount of scale. Like, if you're, like, a few people, honestly, this is overkill. Don't do this. use the Cloud Code sub or the Codex sub or whatever, and just you'll go faster. You probably don't need... You could use this, yeah. Same for, like, any, like, larger team stack, right? Like, at some point, you switch from, like, Fire, YOLO, whatever, to we actually need a platform system here to accelerate everybody. Exactly. Yeah, so you have to be a certain scale. We start seeing that maybe around, like, five to ten engineers would maybe be the floor where this makes sense. and then all the way up to very large companies, they want something like this. The bigger the company,

the more this requires implementation work. I'm sure if you're working with Cognition or I don't know if ComisCursor does, but to get something stood up like this within a company's own infrastructure, it's not going to be purely plug and play. There's the sort of complexity of a team's internal developer setup just grows with size within like but there's a bunch of companies where you know maybe like it's like a 10 to 100 engineers fast growing like growth stage startup where it is pretty self-serve actually where it's like you just if they're using like the happy stack of like slack you know linear jira hang on did you call it the happy stack i call it like yeah that's like our i don't know uh no shade to teams or whatever, like, sure. But, like, there's, like, you know, like a set of integrations that I find very common at startups where you can plug something like this in and get, like, immediate value without too much setup.

I don't know how much you're working like this, but if you're working with, like, Devin or something, you can kind of get some of the same experience where when you're working in an environment that's natively multiplayer, so, like, Slack, for instance, or Linear, There's all these benefits that you might not realize just by having, like, this central system of record where you can have, you know, for, like, the Slack integration, for instance, it's, like, you can get some flow of information coming in through, like, a server crash channel and some through a user feedback channel. You can just feed that directly into the factory to have it do the work. The factory can loop in multiple people on your team depending on who knows the right part of the code base. And so, yeah, I don't think you have to be that big for this to make sense, but it definitely doesn't make sense for a small team. And it really makes the most sense for a company that's like, I want to put in place the right infrastructure so that as models and agents continue to change, we have a system that we're building upon.

There's a lot to go on here. One follow-up question, and then I'm going to have you plug your AIE talks and just generally where to follow up with you. If you, what is the most steel man criticism of setting up all this? Like when you encounter criticism or pushback, what should people know as like, here is like what people are running into? I'll answer this in like, there's a certain classic customer where I don't think it makes sense to like go on to this particular thing. And that's like, and beyond the size thing. Yeah. Where we have challenges right now are companies with very, very bespoke infrastructure. or companies with, like, are kind of not a good fifth customer for this, for instance, would be, like, Uber or Ramp. Like, they're building their own thing. They have very large internal developer teams. But apart from the size thing, I'm thinking, like, I mean, there's definitely, like, the, for some people, they should, it's just going to be more economical to just use the subs,

like, for small, very small teams. But that comes back to the size thing. Okay, actually, yeah, I don't want to actually omit the subs stuff. because I think a lot of people go straight to pay by API, and that is between 5 to 50 times more expensive than just getting a whole bunch of subs. Do people do that? Is that, like, so against terms of service that, like, just don't bother? Or, like, talk to, like, a lot of small companies we talk to where they're like, oh, this system seems amazing, we want this, but we're on these, like, collection of $200 subs that we're rotating through. I'm like, it's probably not like 17 different subs, right? And then you can't imagine that's allowed. But I don't, the truth is I just don't really know. But, like, they are. And it's, like, and it makes, like, honestly, it's, like, if you can do that, that's going to be the cheapest way. But you do lose a whole bunch of other benefits if you do that. And it's not a scalable thing. and if your company scales to a certain point,

you're definitely not going to be allowed to do that. But it makes sense. Yeah, I'm just saying that's the extent of subsidy. Like, you get, for like a $200 plan, you get $8,000 of API, right? Yeah, it's crazy. That's not money to sneeze at. No, I don't know. And it's like, it's one of the, I mean, just from like a business perspective, it's one of the reasons that, you know, I don't want to be in like the pure coding agent business because you just can't build a competitive coding agent on top of a thing that costs so much more money. And that's just the reality. I mean, Cursor was brilliant. They did it. And then I think that they had a perfect exit and became a model lab because I think they fully realized the economics of it. And so, yeah, the cool thing about Warp Factory is we're not trying to pitch a token reselling product. We're trying to pitch basically a coding agent optimization platform

and a coding agent productivity platform. So we're not in that game with this product, which is a strength. And that game, I know there's still a few independent coding agents out there. I don't know how they're going to do it. It's basically Cognition and OpenCode. That's it. Oh, and Factory, of course. Let's wrap up with a little plug for your talk, and people will come see you and we'll live stream it and all that. But give us a general arc of the sort of series that you've been developing and where you're going to talk about next. What the talk is going to focus on is what it looks like to be an engineer in this world of increasing factory engineering. And like what are your roles? What's your job going to be? What are the two types of engineering, product engineering, factory engineering? And then what is this new discipline, if you want to use the word, of factory engineering actually look like? What are the tools? How do you, as an engineer, build the right skills so that you're engineering the system that builds the software and not just building the software directly?

That is a perfect recap and overview of if people want to talk to you, where should they find you? If you're hiring, what are you hiring for? Anything that's a good call to action? Yeah, so come check out warp.dev. So most stuff is there. We're hiring engineers. We're hiring on our go-to-market team. We're hiring on the data team. We're hiring a designer. So hiring for a whole bunch of roles. If you're interested in my musings on factories, I put out a lot of content on my Twitter now, so Zach Lloyd tweets. So come follow that, and I'll keep sharing how I'm thinking about it. We'll do, we'll do. All right, thanks so much for the talk. Cool. Cool. Thanks for having me.