Stripeはなぜ社内ドキュメント50万件を5,000件に絞ったのか

What's up, folks? I'm Demetrios. We made it to sunny San Francisco. And whenever I'm here, I enjoy chatting with my friends who are working on AI agents because they are seeing what is at the forefront of this whole industry. So let's go hop in Waymo and have a conversation. What's up, dude? All right. I'm here with Dave. We're doing podcasts in a Waymo. Waymo cast. you're the CEO and founder of Falconer yes and last time we talked when you were on the podcast it was all about documentation and writing for developers and I remember in that specific conversation I asked you what your feeling is on documentation now because it feels like you have to put a lot of onerous on the person that's writing the documentation. But these days,

dude, and by the way, you literally wrote the book on documentation, the literal book, Documentation for Developers, you wrote. And so it's like, where are we at now? Because documentation has changed so much since three years ago when we chatted. Yeah. And I think at the very end of that conversation, you asked me what I'm excited about or something. I'm like, actually, as much as I talk about external documentation, I'm more excited about internal documentation. And that was just like a weird kink. Like, why would you want internal docs, like, for your teammates? You know, dozens of teammates, hundreds of thousands, when you can have millions of Stripe developers. And that ended up being, like, a very lucky, weird thing to be into because that's what powers agents. I have a lot of feelings, like, it was a roller coaster at first because it's like, wow, it used to take me two weeks to write a changelog for a release. and now I can do that in 30 seconds. Like, what is going to happen to my craft? Yeah. And then I realized, well, I always hated writing the change logs.

So please, like, generate those. And then it became very clear that every, like, documentation, I think, is worse than it ever has been. And, like, we're guilty of this kind of on purpose because we've been asking ourselves the past couple years, and I've been talking to friends at the big labs that care, like, documentation has to change. Yeah. We don't know how. We don't know why. Or I mean, we don't know like how to do it. But like agents need very different content than humans. And I think what everybody's doing is like, oh, I used to write bad documentation before. Now I can have agents write bad documentation for me. And so you go to a random site and it's just like walls of text. It's just flop. It doesn't help you. It doesn't help the agents and it doesn't help the human. so I'm like more fired up than ever there's more opportunity now because we're pushing out so much more documentation yeah and people justifiably don't have to read it as much

so if they do read it who cares if it's 99% agent traffic if I'm going there as a human it is extremely important I need something the agent couldn't get me I need to visualize something so it's more important than ever that if it is a human and you're expecting them to put their own time into actually going and looking at it, it better be good. Yeah, it's the delivery method. If I'm an agent, I just want markdown. I'm going to read every single character on the page. Just give me markdown. Give me endless amounts of markdown. All the context. It doesn't matter. Just give it to me. If I'm a human, I don't want to read markdown. I don't want to read it all. So don't even give me text. Read my mind. Show it to me in a format that I like. Some visuals. Yeah, and give the markdown to the clankers. I don't want to see that. But right now it's all meshed together and people just conflate it like, oh, agents and humans need the same thing. It's just like the opposite is true. Like just slough it off to my agents that are going to be doing the actual implementation.

Like I want the big picture. I have different needs than my agents. It's funny because I, up until now, have been saying that, oh, well, you know, you can write documentation that humans can read, but agents can't. But you can't do it the other way around, where agents can read it and humans can't. But it's almost like, oh, now seeing your point, it is true that just because I can read it doesn't mean I want to. and if it's not optimized for the human it's going to be 10 times harder for me to actually sit there and read it yeah we have multiple senses and so the cool thing is like as we make video as we make graphics like agents can read that too but agents don't have the feeling of oh shit my boss wants me to do something or i need to like implement billing before we ship tomorrow and that feeling of dread when you can't find a crisp answer do you support this form of off do support this currency yes or no and you're like okay this must be a simple like thing i can verify

and that feeling of dread when you land on the docks and they're terrible and it's like a wall of text like agents don't have that feeling they'll find the needle in the haystack if it exists yeah but if you're a human you know you have that feeling of dread and now you have that bad impression of this company um versus turning that into a positive experience of like hey come on in this is it's it's almost functioning like a marketing site like yeah hey come on especially Especially for developers. Right, exactly. We put time into this. You can do this with your eyes closed. Come back the next time. Tell all your friends. Yeah. So, like, really that, like, human part of it is more important than ever. And that's what people, I think, are the worst at because it's the hardest thing to do, ironically. And there's another way that I've seen the ability for agents to grasp what your product does or how to use your product. it's not so clear like how we write documentation for agents you can just put everything in there but then i've seen a company that does like evals on how well agents can use different

tools i think they're 2027.dev they're doing super cool stuff because they're saying look the common tools out there, the Vercels, the Tailscale, whatever it may be, I'm giving my agent a lot of different tasks that need to use these tools. Let's see how well the agent can use it. And some tools the agent uses really, really easily. And we don't really know why other tools it doesn't. And I'm sure there's a lot of like, well, maybe it's because there's great documentation and maybe it's because that tool has just been around and there's more of that code out there in the world. Or maybe that's like a minus. Who knows? But what I'm trying to say is it's not clear why. Why does it work so well on some tools and then not so well on others? Yes. Oh, that's so good. And this reminds me why this is like a very, very hard problem to solve. Like the old way it was a hard problem and we took the time to solve it at Stripe when it was just humans

because it's information architecture. it's search, it's writing quality, it's layout, it's design, it's author experience. Now it's very, very hard in a different way, but what you're describing, how do the agents even read what you're writing in the first place? So the legibility. And then there's these very interesting mini-research people have done. The head of, she's high up at Stripe, Michelle Boo. she talked about like agents ignoring their instructions like in the docs and like how do you like prompt inject the agents to follow what's in in on the page and ignore their training data because agents for the sake of speed they they cannot travel faster than the speed of light so they have to lean on their biases what do i already know about stripe and i can just like ignore like i already making these assumptions and givens and so they using outdated data they think they don have to fetch the latest thing on this page And like an example for Vercel is all of a sudden they have this big Docker launch

And, like, the agents were still recommending competitors because they were leaning on their training data. I already know this thing. So I'm going to give the question asker, the person that's asking me, like, does Vercel run on Docker, an answer that is wrong. Because I think I already know it. Yeah. And meanwhile, you want it to go crawl all the thousands of Vercel's latest pages to give you the right answer. So it's a really, really interesting problem to solve. Like, how do you make it legible? And how do you get them to follow certain instructions properly? Yeah, the most up-to-date information. Because, sure, they can search. But if they think they already know it, then they're going to lean on, oh, well, yeah, I already know this, I'm not going to search. Yeah, you have to make a trade-off where do you want speed? If you want speed, the agent's going to make assumptions, lean on his training data, its inherent biases. And everybody is shipping 10x more, writing 10x more. So think about in two, three years from now, like it doesn't matter how fast the agents are.

They're not going to be faster than the speed of light. Are they going to read through your 100,000 docs and quick starts? Like every time they get a question? No, they're going to lean on what they already know. So it's a very hard problem to solve that you just cannot lean on today's or yesterday's practices, which I think everybody is right now. Wow, yeah, because the problem is only going to get exponentially greater. So much, so much, yeah. We did talk about internal docs, too, the last time that we chatted. I have since kind of gone down a rabbit hole on the whole idea of creating a decision doc. Love it. If there are things that are done at your company, you have this decision doc on why it was done like that, who decided it, what the rationale was behind it, and so you can see that trail. It still feels like you're going to want to, like the value of that has never been greater because then an agent can see it and it won't try and do it a different way, right?

Absolutely, and not even an agent, but you want to see adults throw tantrums at work. It's repeating themselves or somebody not remembering the decision they made or somebody like, we have this feature for our agent, remember this. And I was just arguing about it for our lunch cuisine today. I wasn't going to make lunch. And I'm like, Falconer, remember what my cuisine is, tell the team. It's been used against me multiple times. Like when I remember one time I was like, you know, said some offhand comment about a company. I was like, no, we would never use that. Like that's the antithesis of our philosophy. And later on, they were becoming a prospect and a lead. And I'm like, guys, this customer, this company is so freaking cool. Like we should do a case study. And somebody on my team was like, you said you never wanted to be associated with this brand. I was like, I would never say that. And she's like, at Falcon, what did Dave say? And it was like, I never want to be associated with this. And I'm like, you just like, you forget your own decisions, let alone what your team makes,

let alone like all the little offhand things you want your agent to use as checkpoints. So that is a, like, I think we're just at the beginning of like logging decisions, but yes, like that is the ultimate. And when somebody wants to onboard and they say, why do we do it this way? What is the reason and rationale behind that? And you can see that decision and say, you see that we used to do it this way. Then someone proposed this. And ultimately we went that because the pros are this and the cons were this. Exactly. But then when you see that whole transformation, if something has changed, you can know maybe it's time to re-update this decision too. Absolutely. And that happens all the time. Smart people come in. And Frontier models do this too. You want it to fix one bug and it tries to refactor your whole code base because you made all this. And it's like, no, don't touch it. It's there for a reason. And an engineer comes in and is like, why are we using Mongo? We need to rewrite all this in MySQL. and somebody's like, here are the many arguments we've had over the years,

and then it starts to make sense. But, yeah, absolutely. Or it can be, well, yes, MySQL has since changed these things, so it's worth us re-evaluating. This one tiny thing that this vendor doesn't do that makes it a showstopper for us. And it's really easy to forget those, and it's really easy to argue about those. And it's just not in code. It's in people's heads. Yeah, which is another thing that makes it so hard because you go, you say things and in Slack, it usually plays out in Slack. You're having this conversation. It's like, well, why are we doing it that way? Oh, because this, this and this. And you have this whole thread and you're like, but that actually doesn't matter anymore, does it? And so there's people that are having to chime in, spend their time. And that potentially is not the most valuable use of their time. Yeah. Trying to defend decisions that were made in the past. what are other internal documentation things that you've been really enjoying? I think, I mean, the decisions are incredible because we just, we log things and you just need

a timestamp. It's funny how people just run in front of Waymo's because they know the Waymo will start. It has to stop. Yeah. You get like cut off because this is like the nicest driver around. So the cars just go around, people run in front. They know it's safer than everybody else. They don't care. Stop sign. It's my right of way always. You're not going to get flipped off by the Waymo. I think before we get to the cool shit, decision records are incredible, and I love, and I'm telling everybody, just write stuff down with a time stamp. You don't need to perfect it or make it beautiful. You can do that later. If you write it down and keep it in chronological order, you can do stuff with that later. I'm not even, I think internally people just have to clean up their junk. They don't realize that 95 probably, and this is proved out true from doing this exercise at Uber and Stripe and then a bunch of clients when I was advising, probably like 1% to 2% of the stuff you have is actually useful.

95%, let's be generous, is copies of docs, outdated docs for deprecated features. It's dubs, like just scratch pads. And all of these are Google Docs. They're Notion Docs. Whether people still read them, they're in the context for the agents, and they're in your search. So when you go to search, and they're bloating it. And then a lot of these search tools provide weight for age. So the older it is, the more I remember we were working with this, this customer that had like 8000 employees. And the first thing I do when I search, they were using notion to like test like how good their searches. I'm like, I'm new to this company. I'm a new engineer. How do I get started? And the top hit was titled Jason's intern onboarding doc. Do not use this, but please read. And then you click in that and it's like, this is the top hit because I was the first person to write an onboarding guide. but this is the true doc like click this one this is the real omar doc and of course you you click

that and it's a dead link but like it's just like people just like well what are you gonna do you know search is search and and i i just refuse to take that as an answer get rid of all of the junk reconcile it and now you start to and the cool thing with agents is once you have this baseline like stripe we took 500 000 docs got them down to like 5 000 we actually cared about wow now Now, like at the time we had people, you know, keeping things up to date and it was, it was terrible, impossible. Now you can, you get it down to those 5,000 docs. That's like very doable for an agent to like keep that in sync with the code base with tasks. It's not going to be perfect, but you're, you're going to be like 90% of the way there. And I can tell you how many docs I don delete because I like there probably one sentence in there that actually valuable Yeah And I know that it not important but that one sentence potentially could have value in the future so that I am too scared to delete it Yeah. Everybody has that.

Yeah. And you remember, oh, that took me so long to create. Like, I don't want to. It's like cleaning out your garage, but it feels so, so good to delete stuff because right when you delete it, you never think about it again. You know, like, and if it was important, who knows? You won't know. Well, even when I do think about it, because of that search problem, I can't find it. So I'm sitting there, I'm like, oh, didn't I create something like this before? And so I go searching for it, and then I can't find it, and it's so frustrating because I know I've made it. Yeah. I just don't know where it is. Yeah, and other cool stuff, too, like people internally love a good flow chart or a diagram. Don't give me a 10-page software development lifecycle doc that was written by an engineer that's no longer here, and people give it to me and they're like, this is mostly out of date, but it's directionally good. Like, give this a read. Like, oh, FML. Give me a diagram that shows the major pieces, and you can accomplish so much with just, like,

the width of the page showing me, okay, we start here with a PRD, and then that moves to a prototype, and then that gets specced out. And then like that alone is so much more useful. So that's that's probably to answer your question. Like I would say, look at the stuff that can just be distilled down to a diagram, a workflow, a checklist and like get rid of all the old context. Just delete it. Saying that, it makes me think about bringing up the things in the moment and how with the way we work. That's one type of way of doing it, but then you've also got looking at the code base and how is the code base actually built. But when you get to a certain scale, I'm sure that the code base also is another thing that it is massive amounts of code and repos and everything that you now have this context flow problem on that side too. So not only do you have the Notion docs or the Google docs that give you English, but then you've got code that goes with it.

And if the majority of the stuff doesn't match or it just is overwhelming to the agent, you're really rolling the dice on if you're going to get anything valuable back from the query that you've asked. Yeah, absolutely. And code is pretty pristine. That's what the agents are trained on. what they're optimized for. And if you have dead code, delete it. But agents are going to know like how to traverse that and what to get out of it. But yeah, you're right. They start to get confused when they're taking, okay, this is the code base, but your name is tied to this documents conflicting with what's in the code. And it's conflicting in this old JIRA or linear task. So like, I'm just going to flip a coin because I'm meant to please you and your name is on the code and your name is on the task, your name's on the doc. So I'm just going to make a call here. And, and it's so great. Cause I'm like a magnet for friends at companies where they're like, like I just heard the other day, somebody was like, we had the worst incident I've ever experienced in my career. It was weeks of, of, of cleanup because an agent made the wrong call based on a document. Like it is,

this happened to me at Uber, except it was humans not updating the documents. Like agents are going to do the same thing but worse because they're going to write something down and they're going to make it seem and you're going to trust it because you're going to think like, well, it's here. I don't have to go ask the person who actually owns it because something is written down by Claude, so I'm going to trust it. It's more dangerous. Dude, that just makes me think about the people or the ones who have to actually clean up the mess once it happens because this is almost like a latent problem until it's not. And then you get some kind of an incident and somebody can't figure out what the hell is happening because the documentation has been a mess for ages. Yeah. And, or the agent is trying to figure it out and it's making mistakes because of documentation. And so you almost don't realize how bad it is until something goes wrong. Yeah. And we have been in a period of the last couple of years where you're allowed to blame Claude.

You're allowed to blame the agents if something goes wrong. But as a few performance review cycles come around, people realize like, oh, I own reliability and my team's docs went down. Or some other team's docs caused it that were written by Claude. But I own reliability. I'm getting dinged in performance reviews. All of a sudden they're going to start to care. things aren't going to be as like as laissez-faire and YOLO as before of just like trading you know blindly trading what Claude says, blindly publishing it because now you're responsible so that's the, when people ask me about docs, like what tool should we use should we do a style guide, I'm like start with accountability because if you put a name to a doc set, just like a code base or a repo, people will figure it out how to care and the next step of that is during performance reviews like put it on the job ladder like are you responsible for the documentation uh people will figure it out it will all cascade like smart people will figure it out and i think that's just kind of been blown up the last couple years um

but i already see it changing like ownership accountability people know how to measure performance when you're pairing with agents um and so i am optimistic that like you'll start to see that like oh shit you cause an incident somebody else's team wrote the docs but it's your responsibility it's your metric that you own like that's on you yeah um well and that whereas before when if or when clause quad or uh astra or whatever it may be codex causes the incident it's kind of like oh because i was testing the limits of the coding agent and now that's not an excuse like you have to still be an engineer and you have to be the owner of whatever is happening like you're saying like you have to own reliability and so if that means that you can do it moving fast with 50 million coding agents cool go for it because you're able to architect your software factory in a way that you know it's going to be all right but if you can't

like you can't use that as an excuse now my dog ate my homework i'm not able to do it Yeah, exactly. Especially if you're B2B, there are contracts at stake. There's compliance audits. There's hard stuff at stake where it's not like the customer's going to be like, ah, yeah, got us too. No, you have to figure out a way to pay for over GitHub if it's going down. It used to be okay if US East went down because everybody went down, and it's like, well, you kind of have to figure out a way to be highly available no matter what. Like your customers don't give a shit if it was Amazon's fault. Use it as an excuse. Yeah. So I'm starting to see responsibility, accountability come back, which is a good sign, I think, for everybody. What are some other tricks or ways that you've seen it done well? You've said put names to the accountability. Just have a document that says who owns what, and that will figure itself out.

Another one is put owners on documentation. Mm-hmm. Are there other things that you've seen to work well? You have to treat documentation as part of the product. Like, that one is evergreen. If the product is ready and you ship it and it's not documented, then you don't care about docs. You might as well not even do them because, like, who cares? And internally, you have to treat it the same way. Like there is this amazing it like the the the Holy scripture of documentation people that Jeff Bezos had written some memo Um I don know how to pronounce the name Who the Gaston guy Steve Yege Yege yeah He the one that actually talked about it and it so funny It's written in like a readme in a GitHub repo buried somewhere. But at one point, Bezos was like, communication is a defect. If people are talking to each other, that means that you don't have properly designed interfaces that are well documented because you should communicate through interfaces and well

documented APIs, not through meetings. And I love that because it's like the introverts manifesto of like, don't talk to me. I don't want to talk to you, like document your stuff and then we'll socialize over lunch, but I don't want to socialize or, you know, have to stop what I'm doing to explain how this, this endpoint works or how to get going. And, um, and so that is, I think that was like the the early microservices type of philosophy but i think that still is actually more important than ever your team should be an interface for the rest of the company so how do you what's your operating model what are the tools you use if you do have internal services or infra like how do i connect my tools to it because the cool thing is now it's like agents can do more human-like things so they want to know what your working hours are what your communication preferences are, what your roles or responsibilities are, because they can do so much with that information. Like, don't schedule a meeting to talk about stuff that could be written down. Yeah. Yeah. So you talk to so many people across the space, like, and I'm sure you have

opinions and they have opinions on the future of documentation. Like, what's the opinion you've you formed recently. So when it comes to, dude, underappreciated asset is what I think. I like that. Yeah. Doc should be an asset, right? And that's why I love talking to you because you are so deep in this practice and this craft that you understand all the ways that we can make it an asset rather than a liability and I think the first time that we chatted I quote this all the time to folks how you would think about little micro almost philosophical things or psychological things that the people that are reading the documentation would have to go through and that was with the escape hatches and so what you said was that you would always in any documentation that you wrote, have a few paragraphs in, is this what you were looking for?

Or was it something along the lines of this, which is similar, almost like a recommender system of docs that you could have meant when you were looking for this so that somebody could traverse easily and not have to read through the whole documentation. That is what I look at as a document being an asset because I am not wasting my time when I am looking through the docs. And the fact that it's so easy to churn out so much slop, we all are getting sick of reading slop. And I now have come to a point where I say to people, if it takes me longer to read your document than it took you to write it, I'm not going to read it. I call that a brovella, like a novella. If it took longer to read than it took you to generate, like, you're just, you're forcing your lack of rigor on me. Exactly. And it does, like, if you go to any random, this is why I'm so angry and fired up about, like, docs today,

because if you go to any renowned company's doc site, it is that slop fluff, and it feels like I'm getting the Yankee Candle. You know, you walk into, like, a Bed Bath & Beyond, it's like, ooh, somebody's baking cookies. Shit, that's just, like, the cookie Yankee. It tricked me, yeah. And, yeah, like you get repulsed after a while. That's what I feel disrespected. Right. Somebody just disrespected my time so hard that they look through and they think that I'm going to spend my time, which is the only thing I can't get back. Absolutely. It's the most precious thing in our lives. And now I have to sit there reading some slop that you generated? Yeah. That for me is why I feel like it is an underappreciated asset because as of now, it's okay to do that. The norms haven't changed, but I get the feeling that enough of us are going to read enough crap that it's going to start really dinging your reputation in ways that you don't want.

And so the next order effect of that is going to be that you're not going to put it out there or you're at least going to edit it before you generate it. Yeah, yeah, that's a good way to put it. Like, the rigor is missing. Like, you can get the initial output from an agent, but the human rigor, because you can empathize with the end reader. So, like, what would you want to read? I want to feel smart. I want to feel accomplished. I want to feel like a moron, you know, reading your instructions because they don't understand it. Exactly. Well, dude, this has been great, man. I appreciate you coming on here and talking docs with me. Yeah, anytime. Are you coming to, can you come to Agent Cone? I was if this was Friday, but now that it's Monday, I have to go home. Oh. Yeah. Yeah, because you're based on a... I'm in Santa Barbara. That's right. Yeah. Santa Barbara? So I spend like half my week up here. It's like an hour away, right? It's close to San Jose though, right? 400 miles south. Oh, 400 miles. It's 100 miles north of LA.

Oh, Santa Barbara is closer to LA? Yeah. Oh, I'm thinking of Santa Clara. that's why i was like ah santa claire's close right no santa barbara santa barbara yeah so i i fly up um every week well all the more why this is so cool that you did this and we made it happen yeah this is fun how many of these are you have you done are you doing i have one more yeah i just did one with uh jesse who created superpowers do you know that speaking of writing and all that fun stuff that sounds familiar the superpower skill pack super cool yeah i saw you you talked to zach which was cool he's one of our angels oh yeah yeah oh nice he's dude his stuff was really good yeah and it's a while back now but yeah it was very cool and he's a good writer yeah oh yeah he takes it seriously he writes a lot about software factories which is cool yeah yeah and really nice and then we have our our big ga launch on wednesday so oh dude that's huge tell me when

it goes out so i can you know like comment oh well yeah that would be that would be huge very cool yeah and we'll i'll send you this um like this recording hopefully in give me like two weeks potentially three uh for it to go out you know but if you want it so that you can make like you know shorts and all that fun stuff oh amazing yeah that'll be killer oh i'm i'm remembering now you you uh posted about my neural link content that part does i actually believe it i was like i had to think what could disrupt what could disrupt me neural link shit oh wait but now no there's too much noise yeah talk about context yeah like context bloat and we really got to get better because oh man just like meditating when i'm trying not to think yeah all i'm doing is thinking do we have the right mustard for the vinaigrette i'm making later exactly dude how long do we have

with climate change yeah i'm trying to think of like all the stuff that pops in my head when trying to meditate oh man yeah just such random shit and then you have one thing that might be valuable should I stop meditating because I need to I don't want to forget this one thanks dude