トークン使い放題の時代は終わった――「Sonnet maxxing」とAIコーディングのROI

Token Maxxing Is Over and Here Is What Comes Next

トークン使い放題の時代は終わった――「Sonnet maxxing」とAIコーディングのROI

トークン使い放題の時代は終わった――「Sonnet maxxing」とAIコーディングのROI

Agentic Conversations (formally mlops.community)

0:0035:59

要約

元HelloFreshのプリンシパルエンジニアPepijn Senders(Pep)が、AI未経験の約200人のエンジニアに6か月でプラットフォームを再構築させた経験を語る。意図的にOpusではなくSonnetを選び、人間を開発ループに残した。トークン消費と成果を散布図で可視化してROIを議論する方法や、複数人で行うリファインメント、プロンプトの言葉選びがモデルを左右する実験、Claude Code上のメタハーネス構築も紹介された。

この回のポイント

  • ●Pepは推論ステップが少なく早く止まるSonnetを選び、開発者が途中で判断に関与し続けるようにした。銀行ソフトで業務ルールが厳しいことが理由で、80万行のコードのうち約6割をSonnetが書いたという。
  • ●トークン消費をX軸、機能数・行数・マージしたPR数などの成果をY軸にした散布図を作成した。結論ではなく「何を作っているのか」という対話のきっかけとして使うのが狙い。
  • ●ROIは「2倍・3倍の生産性」ではなく「本来30〜40人要る仕事を20人でやれた=節約できた人員数」で語るべきだと述べている。HelloFreshでは自社の関心である機能ごとの出荷速度で語れず、証明に失敗したと振り返った。
  • ●AIが提示する選択肢は、気づかないうちに選択を狭める。自分のコードベースで理解できない関数がすぐ見つかるかを試すことを勧めている。
  • ●全員参加のリファインメントセッションで、エージェントの質問にチームで答える運用が、うのみにする受け入れを防ぐ。
  • ●同じ一発プロンプトを設計者とエンジニアに与えた実験で、「React」「scalable」「maintainable」といった語が出力を大きく変えた。
  • ●コードの85%が「リファインメント→仕様→実装→検証」の同じ流れだったため、Claude Code上にメタハーネスを構築し、各段階ごとのトークンなどを計測している。

章立て

  1. Pepの紹介とSonnet maxing

    PepはEposcaのプリンシパルエンジニア。AI未経験の約200人にOpusでなくSonnetを使わせた理由が説明される。

  2. 6か月でのプラットフォーム再構築

    80万行規模の再構築で、約6割をSonnetが書いた。人間を判断ループに残すため、Sonnetが途中で止まる性質を活かした。

  3. AIコーディングのROI証明

    AI導入前後で比較できるダッシュボードを作った。1%良い画面に2万ドル分のトークンを使う価値があるかを問う視点も語られる。

  4. HelloFreshでの挫折とトークン可視化

    HelloFreshでは機能ごとの価値で語れず証明に苦戦した。トークン消費と成果の散布図は、対話を始めるための道具だと説明された。

  5. 人員換算のROIと人間の主導権

    生産性の倍率でなく節約人員で語る考え方を示す。人間がコードを理解し続けることの重要性と、人間が方向修正した箇所の計測も語られる。

  6. マルチプレイヤー式リファインメント

    AIの提案に流されないよう、チーム全員でエージェントの質問を議論する。成果物は要約や意思決定の追跡にも使われる。

  7. 言葉選びがモデルを動かす

    設計者とエンジニアで出力が異なった実験を紹介する。設計者向けにSvelteと出力スタイルを設定した工夫にも触れる。

  8. 社内メタハーネス

    コンテキストを整理し、段階別にClaude Codeを起動し直す仕組みを説明する。各段階のメトリクスも取得する。

解説記事

元HelloFreshのプリンシパルエンジニアで、現在はEposcaに勤めるPepijn Senders(Pep)が、Demetrios Brinkmannとの対談でAIコーディング導入の実践知を語った回。AIをほとんど使ったことのない約200人のエンジニアを抱える組織で、あえて推論の浅いSonnetを選んだ理由や、トークン消費をどう正当化するかが中心の話題だった。

「Sonnet maxing」という選択

Pepは今年2月、AI未経験のエンジニア約200人がいる企業に加わった。それまで使っていたGitHub Copilotでもトークン消費はほぼゼロだったという。課題は、既存業務に多くの時間を取られる状況で生産性を上げることだった。

そこで彼が選んだのが、思考ステップが少なく早く止まるSonnetだ。扱うのは銀行ソフトで、業務ルールが厳しい。Opusなら「取引エンジンを作って」という指示をそのまま進めるかもしれないが、Sonnetは最初の段階で詰まり、人間が舵を取る必要が出る。Pepはこの性質が開発者をループに残すと考えた。結果として6か月で約80万行のコードを書き、その約6割をSonnetが担ったと述べている。

トークン消費をどう説明するか

Pepはトークンを使い放題にする風潮は終わりつつあると見ている。企業にはCFOがいて、請求も無限ではない。そこでAI導入前後が明確に比べられる強みを活かし、ダッシュボードを作った。

特に使うのが散布図だ。X軸にトークン消費、Y軸に出荷した機能数やマージしたPR数を取る。2万ドル使って大量のコードを出す人もいれば、1.5万ドルでPRが1件の人もいるが、後者は調査中かもしれない。Pepはこれを評価や処分の材料ではなく「対話の入口」と強調する。個人の用事にトークンを使うケースもあり、7割対3割程度で私的利用が混じっていた時期があると冗談交じりに話した。

ROIの伝え方では、生産性が2倍・3倍といった倍率は使わず、「30〜40人必要だった仕事を20人でこなした」と人員数で語ると述べる。一方、HelloFreshでは機能ごとの価値で語れず、6〜7か月戦っても十分に証明できなかったと率直に振り返った。収益に直結する機能にAIを当てて出荷が2倍速くなった、という物語にすべきだったという反省だ。

人間が主導権を持ち続ける

Pepが最も懸念するのは、LLMにコードを任せきりにして、何が起きているか分からなくなることだ。自分のリポジトリを開き、用途が分からない関数を探せば、すぐ見つかるはずだと言う。未知の未知は本番障害の火種になる。

また、AIが示す選択肢は、気づかないうちに道を決めてしまう。Opusなどが理解しにくい質問をしてきても、つい受け入れてしまうという話題も出た。対策として、社内ハーネスで人間がモデルのデータ傾向に逆らって方向修正した場面を計測し、チームごとの傾向を見ている。ただしそれが良い兆候か悪い兆候かは状況次第で、ここでも結論ではなく対話のきっかけだとしている。

さらに、1対1でエージェントと進めると推奨をそのまま受け入れやすいため、チームで行う「マルチプレイヤー」のリファインメントを導入した。PRDを取り込み、全員で画面を開いてエージェントの質問に答える。プロダクトマネージャーも参加でき、集団の知識が「良さそうだから」という安易な承認を防ぐ。記録は要約に使え、意思決定の理由もさかのぼれる。

言葉がモデルを動かす

Pepは、同じ一発プロンプトでページを作らせる実験をした。設計者の多くはHTML・JavaScript・CSSを1ファイルにまとめたが、エンジニアはReactやVueのアプリにした。エンジニアのプロンプトにある「React」「scalable」「maintainable」といった語が方向を変えたという。さらにOpusは誤ったトークンを遠くまで推し進めるのに対し、Sonnetは影響が小さい、とも述べた。

設計者向けには、専用リポジトリでSvelteを使わせ、出力スタイルも設計者向けの言葉遣いに変えた。ミスしにくい環境を先に用意する発想だ。

メタハーネスの構築

最後に、Claude Codeを束ねる社内のメタハーネスが紹介された。きっかけは、スキルを大量に入れたせいで開始時点でコンテキストが4万トークンに達していた同僚の例だ。Pepはコンテキストに何を入れるかを厳しく管理したい立場だ。

組織内の多くの人は決まった手順に従って働く。調べると、コードの85%が「リファインメント、仕様、実装、検証」という同じ流れで書かれていた。そこで各段階に最適なコンテキストとスキルを与える仕組みにした。システムプロンプトに「終わったら特定ファイルを触る」と指示し、ウォッチャーがそれを検知してClaude Codeを閉じ、次の段階の新しいセッションを開く。段階ごとにトークン量などの構造化データも取れる。

まとめ

この回の核心は、AIを最大限に走らせるより、人間の理解と判断を残したまま成果を測ることにある。日本企業でも、トークン費用の説明責任やAI導入の定着は共通の課題だろう。人員換算のROI、散布図を対話に使う姿勢、全員参加のリファインメント、段階別のコンテキスト管理は、すぐ試せる示唆だ。ただし数値はPep自身の経験談で、一般化には注意が要る。

文字起こし(英語・自動生成)

I feel like sometimes, like, for example, Opus 5 or something, like, Opus would ask me something and I don't even know what's going on anymore. Yeah. And, like, I really wonder why nobody's, like, why the fuck are we, like, listening to this model who doesn't even speak English? Right? Why am I not taking any second guesses here and be like, okay, what is actually, what's happening here? That's the thing with Fable, man. So many times you're just kind of like, I mean, I guess that looks good. Are you speaking English? Yeah, exactly. Or are we just like talking in some made-up language now? Take your repository, whatever you're working on, open the code base, and stop when you don't understand anymore what's going on, when you cannot place a function. Like, what is this function doing? Why is it there? What's the business value of this piece of code? I will guarantee you, and even now, even when I'm really trying to make sure that we are in control, there will be parts of code where people will have no clue what's going on. But I will guarantee you it will not take you a long time before you will find that function. Yeah, first line. First line, yeah.

I guess I lost. Pep, give me the lowdown on what are you working on just so we can ground the conversation and let people know who you are. For sure. Yeah, so I'm a principal engineer at Eposca right now. It's like a customer benefits company. Before that, I was a principal engineer at HelloFresh. This one you might know a little better. I've always been, like, trying to be as generalist as possible in my practice. And then my baby was born February 2025. Congrats. I took, you know, as a European, parental leave. You're still on parental leave? No, no, no, not anymore, not anymore. But in this time, of course, that was around the time that AI was, like, really becoming very, quite effective at coding. 2025, yeah. 2025, long time ago. Like, in AI years, very long ago.

Coming back from parental leave three months later, talked to the CTO. It's like, we've got to change stuff today. Now I'm back. Yeah, yeah, yeah. And now I'm back and just working. And I'm also on the side of my helping out Dex at Humilayer. He was on stage earlier. He's very interested in, like, seeing how AI is impacting the engineer's life, how you can actually get products from working with AI that make sense, that don't go off the rails. And, again, Sonnet maxing is one of the things that I say. Let's go deeper there. What is Sonnet? Why the enamoration with Sonnet, I guess? So imagine, right, everybody at this conference is, like, already super AI-filled. You come into a company February this year. Nobody has ever used AI before for coding. 200 engineers, and your task is to, like, increase productivity free for X.

You know, this company was, like, spending 60% of the time on KTLO tasks, not creating enough shareholder value. and it's like if you would give everybody like Opus 4 or 8 or 6 whatever was out at the time I didn't know where I would have been you know six months later so for me like a model that doesn't go for as many thinking steps, doesn't do as much reasoning stops earlier keeps the developers more in the loop there's way more to this productivity gains that you can show and then you can say like hey, do we want more of this? It's just like... There's enough. Yeah. So we basically rebuilt an entire platform in six months, wrote like 800,000 lines of code with that team, part of the team, like 50 people with that team. So that was like quite a big thing. They launched it actually two days ago at a big conference in Germany. So everything's fine. You know, there's some gaps here and there,

but people have been staying in the loop. and 60% of that project was written by Sonnet. Yeah. Of course, it's awesome. Opus, maybe a little bit of haiku here and there. But, yeah, we were using Sonnet for the most part because I really believe that if you're in a loop enough, then it's fine. And I wanted people to be in a loop because it's just, like, hard matter that they're working with. Banking software, hard business rules. And if you tell Opus, like, hey, go build this transaction engine, And he's like, okay, sure, I'll build you a transaction engine. And Sonnet will get stuck at the first step. And they're like, okay, I actually have to steer us a little bit. I actually have to think more. So, like, I try to always tell these people in this company, like, hey, don't take off your thinking hats. It's like, stay on the table. And we carried, like, systems around that, like, you know, like the Spectre of Development, RPI kind of stuff, where Sonnet was, like, consulted for the right things, and it was, like, bringing up human feedback at the right time as well. Was the adoption from the engineers easier, too, with Sonnet?

or were they already interested because they were playing around with it on the weekends i would say it was they were very german uh sorry for the people that are listening i live in germany i know exactly what you mean yeah yeah so i will not go deeper into that one uh but no everybody was super excited to get get on get on board they there was like nobody doing like the adoption preaching or whatever at the company. So nobody was really like, hey, let's go. So they were using GitHub Copilot before this. Okay. And their token spend was almost zero. Yeah. So. They weren't really using it. No, they weren't really using it. Yeah. GitHub Copilot, but it wasn't like anyone had told them, shown them the magic of building a software factory with AI. And so you found that it was a much smaller step to use Sonnet to get them just like, hey, ask a few questions, try and get it to write you some code.

And then it would write a little bit of code. It would come back to them. They would be much more in the loop and therefore be much more trusting. Exactly. And the spend was. So one thing that I try to do is like maybe something that we can get into as well. So, like, I'm very passionate about showing the ROI of using AI for coding. Just because I think it's very interesting that for some time, I'm pretty sure we're over that time right now, but everybody's like token maxing for a little bit. It doesn't work if you have enterprise billing. Yeah, or a CFO. Or a CFO or both, right? So, like, I basically built a dashboard where we could calculate the amount of, like, full-time engineers we're gaining by using AI. Also because we had a very clear cutoff point. Okay, here we are not using AI, and here we are using AI, again, this year in February. So we can really see, like, the difference. Like, we can actually see the difference in productivity before and after.

Yeah, tell me more about these metrics that you're looking at. So there's a couple ones that I think are interesting. So we are, of course, like lines of code and those kind of things are very difficult to actually, they don't really tell a story as well as they cannot really tell a story. So basically we were trying to like, like one thing that I really like to do is like to see, okay, this team is going to build like a new home screen for the app. And then I can actually see for that specific feature how much time they spend on it and how many tokens they spend on it. but if that new home screen is like maybe let's say 1% more valuable, was that worth spending $20,000 in tokens? Right? That's like a bet. Exactly. And I never think that these metrics should be like, okay, that person is now fired because he used $20,000 tokens on a home screen. But it should always be a conversation, right? And one thing that I – You're saying it the other way. I understood it as it's like, wow, you have the possibility of hitting a really big home run when you're using these.

But you're saying like, eh, was it really worth it to do it and use this much money for this, like, not that valuable of a thing? Exactly. Exactly. And when I was at HelloFresh, like, there was a way bigger engineering org, and I was also doing, like, the AI. and they moved there again from like May 2025 which is like a very long time ago in AI years and I was first such a company where and now I can speak freely because I'm not you know I don't sign any NDAs but it a very product driven organization there is like a gazillion different funnels for every single customer profile A lot of different variations Everything gets A tested Everything gets weighed 1,000 times before it actually gets even built. The product roadmats are the leading indicator of everything that's going to happen at the company. Everything is measured. How much customer value can we get out of this work item? Everything is measured

to that. If Pep comes back from rental leave and says guys you should start using ai for coding they're like well what's like the game is there more customer value at the other side of this we don't know so i was fighting with them for like six seven months like trying to like actually prove that roi and i managed to do it at the end but how and how did you prove it i'm sure it was a million conversations and it's really hard to just summarize in you know five minutes but what were some of these big talking points that you had so for me it was all about speed of course productivity increase and then again also like you know variations that i could show and like the things that they could ship but honestly in the end i think i kind of failed at this company because i wasn't actually able to show the full story which i'm got pretty good at right now yeah because i think i should have really focused more like what the the company was focused on they were really okay which feature is bringing the us money i should have told a story like okay if we put ai on this feature then we ship it like

two times faster right oh that could have been a story okay not a lot of guests on here are so open and vulnerable so i appreciate you talking about that and saying you know because it's not all roses and tie-dye no no we're all learning right that's crazy yeah yeah and so now you're recognizing the biggest feature that is going to really be generating the, basically what I was saying before is there are some bets that you can take that will give you outsized returns. So focus your energy there, focus your token spend on those, as opposed to making a home screen 1% better. But it's always been like this. That's the thing. I feel like with these token mixing budgets, people just forgot about that, which, again, we saw as well. I was looking at the dashboards, and I was like somebody was spending billions of tokens a week, and I was like, what are you doing? I was like, yes, I'm building my own tooling. I was like, why? On the company, Bill?

Okay, nice. I'm not your manager, but maybe. And so first thing you did was you put together some dashboards on just token spend? Yeah, yeah, yeah. Yeah, my favorite dashboard for this, you can see my idea. A lot of people, when they look at it, they're like, wow, that's actually – again, it's not like a conclusion, but it's the beginning of a conversation. And I don't want to throw any engineers under their bus right now. They're using tokens for their personal projects on the company bill. I hope you can still do it for the foreseeable future. But what – You're speaking to me so much. All right, yeah. So we basically built like a scatter chart of person versus like production. Or token spend versus production. You can see like the token spend on the x-axis and how much they are producing on the y-axis. Like number of features shipped, number of lines shipped, number of PRs merged. And, you know, I'm spending like 20K, but I'm also shipping like a million lines of code a month, right?

And then somebody also then spending 15K, but you only merged one PR in four weeks. again it could be like he's just researching right that's like yeah but at least you that's like a conversation starter and i think we should have those conversations and i think we should be more open and honestly i feel like we should share these metrics with each other because i know i don't want to be at the mercy of these vial companies just like sending like a you know one million one million dollar check every month which is the situation for a lot of people are there any other data visualizations or just data that you've been able to catch the outliers both good and bad that help you see like oh wow we should have a conversation yeah it's very important it's the start of a conversation yeah it's not accusing you of anything but what are you building yeah yeah exactly and again there is a lot i would say in my experience it's all it's it's become better now that we're like talking about these metrics better and people are more realizing what uh yeah you know they're actually spending a lot of dollars for the company and the company

has to know what the roi is but before that was big like i would say like 70 30 percent 30 percent were people just writing their resumes uh tinder pickup lines on the company dime on the company asking cloud code like i have a date tonight what should i do oh man and then you start to see that and you recognize like uh-oh yeah we need this and then i start thinking about my own prompts. Okay, fuck. Actually, maybe I'm also not. Yeah, you know. Use your personal computer. Yeah, yeah, yeah. Use your personal account. But to your question, like, which are the visualizations? So, again, I think, like, this is, like, really thinking about, okay, what is, like, using AI, giving our team, giving back to our team? And I don't like to think in, like, productivity increase. Like, okay, we're 2x, 3x better. But I'd rather think, okay, like, how many, like, if we would have to do this with, now we're doing this with 20 engineers. Like before AI, we might have needed like 30, 40 engineers. So now you're suddenly saving the company 20, like 10, 20 headcount, right?

And that's like their ROI. And the almost like false dichotomy of that, maybe that's not the right term, is that when you have 40 engineers, you're organizing 40 people. And so it's not a guarantee that with those 40 people, you're going to get the same output and same outcome. And so it's a little bit like, yeah, I could see that. You can kind of like say, all right, there's metrics of how many people would we need if we wanted to do this? Are there other things? Because I'm thinking about when you're going through this, you potentially want to go and get the folks that are actually shipping the most. And really in the quadrant that you drew, they're in the top right, right? That's the best quadrant to be in. Yep, yep, yep. Most spend, most shipping. That's the other side of the story, right?

They kind of want to get people to that position. but are you going then and talking to those people and saying what are you doing? yeah learning from them and also the thing with tokens pandemic is that a lot of companies have right now post token maxing days and CFOs freaking out and wanting to know what's happening there I feel like these topics are still so early but it's like and like, you know, I was talking to somebody at the dinner two days ago, and they were like, yeah, we have 60K per month token spent. Just their team? No, no, individually. Oh, wow. Yeah. So do whatever you want with it. Do whatever you want with it. We don't have that kind of luxury. Wow, that makes me uncomfortable. Yeah, that's a lot of tokens. But I don't know, like if somebody is like super successful at like producing value, right, with AI and they can show this why not give them more money because they are basically

maximizing themselves right where other people might actually be more on the negative side where they're using a lot but maybe they're I know there's like a lot of a lot of stories here and again that's why it's always a start of a conversation not like a conclusion yeah because you may be saying you're not being productive by these metrics we have but if you grabbed other metrics, they are being productive in ways that you just can't see. Or they're super new to the domain and they're just using AI to learn about it. Also happens. And then they're not producing as much for maybe three months, but in three months from now, they're going to be kicking it up. Yeah, I can see that. So, alright, what else you got? These are some kind of spicy takes and I like it. Yeah, the spicy takes are there. So, So, I think what else? In this space or just generally? Actually, I would love to hear, yeah, what basically under the guise of spicy takes in this space,

maybe not necessarily ROI because we've beat a dead horse on that one. Yeah, yeah. And that's what everybody wants to know, right? Again closing the loop on it it very hard to know what token spend is quality token spend and what is not i really like your quadrant idea of who shipping who faster and more and who spending like less and more but are there others things that you think about uh that aren't related to the like the company roi things so yeah I think a lot about just like making sure that humans stay in control and stay the driver in production of codes I've seen many problems over time where ALM just take over and you don't know what's going on anymore

Yeah, that's basically it. I've, like, everybody at home can try this. Take your repository, whatever you're working on, open a code base, and stop when you don't understand anymore what's going on, when you cannot place a function. Like, what is this function doing? Why is it there? What's the business value of this piece of code? I will guarantee you, if you've, and even now, Like, even when I'm, like, really trying to, like, make sure that we are in control, there will be parts of code where people will have no clue what's going on. But I will guarantee you it will not take you a long time before you will find that function. Yeah, first line. The first line, yeah. I guess I lost. It's really freaky. And I think, you know, I think Dexel said it nicely on stage. Like, yeah, you can, like, improve those. You can, like, point the agent at, like, saying, like, hey, you have to improve this. But if you don't even know that it's there, it's like a ticking time bomb.

Unknown unknowns. It's the unknown unknowns. And I just have way too many examples of this blowing up in production. And, you know, I've worked on, like, quite critical systems with a lot of, like, customer data. I don't want to beat her. Well, and so what are you doing then? I know Sonnet is one method of trying to make sure that things, it's constantly asking you questions. But it's so easy to get out of your depth, especially when it suggests you to do something that you're like, well, I was going to do it this way, but I guess it probably knows best, so whatever. And especially if it's not like a language that I'm super proficient at or it's something that I'm not really that into and I just want to get it done quick and boom. yeah and then you got a problem yeah exactly I really want to flag those things we have a company harness now and we're trying to measure so we have a lot of steps where people have to go through it's kind of like the grill me but then our own version that we had before

and we're trying to see when is a human steering against AI and we also use different models for this stage for example we might use Opus actually or we might use Fable, depending on the complexity of the problem that they're working on. But we want to know when we're steering away from the data set that the LM has. Because I'm really convinced that these, like, you know, they're trained on data. Yeah. And, you know, we want to do something, you know, most of software engineering is super generic, right? It's just like, okay, build this fucking modal or, you know, build this database adapter. It doesn't really matter. It's all, like, very reproducible by AI. They can just turn it out for you, whatever. But there's a lot of moments, actually, that you have to steer back. And so, like, actually, no, we need something different. And I'm getting a little bit in the weeds now, but we really try to measure, like, okay, when is, like, the humans are steering back? We have, like, some diasporas in that as well, seeing, like, which teams are doing that more and less. Again, conversation starters, not, like. But that's a positive signal. Or no.

So, it's a positive signal. It depends. Again, like, it's a conversation starter. Yeah. Yeah. So, we cannot, you know. Being able to steer is really like understanding that we don't want the LLM-generated stuff. And so it's almost like I would look at it. I guess you're going to find the pros and cons on each. It's interesting because we have, like, graphs over time that can show this. Yeah. And also with models and, like, for example, like, I feel like sometimes, like, for example, with Opus 5 or something, like, Opus would ask me something and I don't even know what's going on anymore. Yeah. It's such a... and like why i really wonder why nobody's like wait why the fuck are really listening to this model doesn't even speak english right why am i not taking any second guesses here and be like okay what what is actually what's happening here yeah that's the thing with fable man so many times you're just kind of like i mean i guess that looks good are you speaking english are you or are we

just like talking in some made-up language now yeah so again i've seen just so many different things and we also look back at like sessions a lot to kind of like look at like how our features build and also like now we can try to look at session history for post-mortems for example there was like a authentication problem and I was like looking at the code and the ADRs and all the documents that were put out and they're like I was like what the hell how did this how how did this get decided that this is a good idea how and this is the funny piece is that when you're given the selection of choices, it kind of is saying without saying it that here's what you can do. Yeah. And it doesn't it makes you think that like these are the choices. And if you're not going to put in the energy to figure out what other choices there are, you kind of go down the path that it wants. Yeah. And so by having those questions that are being fed to you, like do you want to work i suggest you do it like this and it usually will recommend something yeah

it always does that will throw you on a path that maybe you didn't want to be on yeah yeah it's it's hard like the it's it's almost the the one of the ways i found to have give some agency back uh to the engineers to kind of like help them steer for that and the way that we actually do it right now it's like because i feel like when you're one-on-one prompting with an agent you're more likely to say, okay, recommend this, recommend this, recommend this. I'm in. What we do right now is actually we make it like a multi-facet story where it's not just engineering questions. It's like product, UX, whatever. Basically, you ingest the feature, the PRD or whatever you want to call it, the product brief or whatever. The idea is with the teams that we're working on right now, they come together on Monday and we call it the refinement session like we used to call it as well in the past where you had this annoying engineer that put their hand and be like, hey, we cannot build this, you know, or it's too difficult. Yeah. That's the same vibe that I want to get back. So basically, we're shifting left. We're putting everybody in the same room.

We're discussing what needs to be built. And then AI as a partner is trying to, like, grab, like, the hard parts from the requirements. Like, the hard parts to implement. The hard parts to, that were, like, the code base would get, you know, where it would be difficult to implement in the current code base that we have. okay so talk to me more about this because this is very much an organizational way of working with definitely ai yeah so i'm one-on-one with my agent talking about what i'm going to build and i have this feature idea or i have this way yeah i'm context building and then i go into a meeting with the rest of the team and i show them the no no you do live okay so basically you have the or either it's asynchronous or during a call, just like open up the agent screen, make it pretty in some way, and you just talk about the future. Just like how you would do like a refinement session before AI. Interesting. Yeah. And then the agent's able to get more questions, able to drive the conversation more, because there are a lot of conversation starters that the agent will give you.

And then in a group setting, I feel like peer pressure, people are less likely to be like, oh, yeah, I recommend it. or as suggested. That's the social cues, yeah, because you can't just go through every step. Exactly. There's a lot of value there because a product manager can be there. It's like, hey, they can also steer the conversation, right, because I feel like if you're in your silo, it's never been good to work in a silo. Agile is great for that reason because there's so many ceremonies, there's so many points where people can actually talk and get out of that silo. Dude, okay, so this is brilliant. So you literally are going through these questions with the rest of the team. And then it's almost like fun because, oh, cool, what's it saying now? It's suggesting that we do it this way. And everybody, that group knowledge is able to not allow the just looks good to me. I'm going to accept everything. Exactly Okay And it works really well And like the artifacts that get generated over time for this actually are very valuable for somebody like me who like sitting on top of this gigantic project of like building like

rebuilding, like, a huge part of, like, their platform in six months. Because you're also recording those calls, and then you have the transcript. No, no, no. We just have the transcript. So, like, yeah, record it, but then I'm not going to go for, like, the actual call, but I will, you know, also, like, use an LLM to kind of say, like, hey, can you give me a summary of the refinement sessions that happened this morning? What's the general direction? What should I reach out for? Nice. And if you want to dig into something specific, how is this decision made? Why did we decide to use this or that? Yeah. Honestly, I've been too late. There's some of these decisions sometimes, but I can see myself use that in a better way, where I can actually be ahead of things getting implemented in an unscalable way or, you know, non-desirable, whatever. Okay, so shifting that left, what are some other ways you're thinking about shifting left or just, like, organizational changes? Yeah, I have no hot take.

Oh. So we have to demystify LLMs for us to be productive with LLMs. Otherwise, we're always going to treat them as, like, some magic boxes. It's okay. I'll put something in that's going to do it for me. Done. But if you actually really demystify it and, like, know that the input that you give it is very – it's actually very – like, the output will be very different depending on how you're using the tool. I did a little experiment because all of you exoners were, like, knocking on my door. It's like they want to write code. I was like, sure. But let's try it out first. So I basically gave the same task to a bunch of front-end engineers and a bunch of designers to, like, generate a page on the website. And I compared the results between the two groups and the prompts that they used. I told them one prompt, you can do only one shot to generate this page.

And the results were stunning because, sure, they look the same on the outside. but all the designers but one were like single HTML page with JavaScript and CSS in line. All the engineers had like React apps, whatever, Vue, doesn't matter. And there's like a very small thing if you look at the prompt that actually is changing the way that these outcomes are different from each other. and that's like the engineer would write just React or scalable or maintainable like any of those tokens steer the model in a completely different direction so if you take that very simple example and take that and then apply that to the way that you want to work with a lens at scale then you really realize that you have to make sure that the right tokens get put in for the right task because they really steer the outcome a lot.

And then going back to the Sonnet Maxing, with a wrong token in Sonnet is not as bad as a wrong token in Opus. Because Opus will take that token way further than Sonnet would. Yeah, which in a way is like a double-edged sword because you want Opus to go further and not ask you. but if it's going the wrong direction you're screwed exactly this is something that i was literally just talking about with bouquet on the last segment and we were calling it like magic words and so it's being able to place the model into the right latent space yeah to understand that oh when i make this it needs to be maintainable or it needs to be scalable so i'm going to use these constructs instead of those. Yeah. Yeah. Yeah. And to know that, that's the key, is that you have to understand the value of these things. And so that's where either you learn that for that specific thing that you're trying to do.

So if it's a UX designer that's trying to write code, they have to now learn that. And that, because it's not like so hard to learn. It's just like add a few extra words to your prompt, right? Yeah. But you got to know those words. No, like, with the thing that we did in the end, it was actually the designers are writing code now. Yeah. But they have, like, a repo where they're doing it in, where we set the context, and basically I did an evil thing where I basically said that every designer writes in Svelte. Because Svelte is, like, it's, like, not as safe as Rust, but it's a lot safer than React, for example. It's, like, way harder to make mistakes. So, like, we just tell Claude in that repo, like, hey, whatever the designer asks you to write it in Svelte with these, like, principles, yada, yada, yada. And then we have another, like, output style thing that is supporting cloud code, and then it basically talks back to them in designer speak. Brilliant. Like, yo, I built your page. It's done. Right? Find it here. Smart. Smart, man. Well, before we sign off, you did mention you have a company harness.

What does that entail? So it's like a meta harness they call it right now. Harness above the harnesses. Exactly. So we're orchestrating plot codes. I did a talk in Berlin. I was very harsh on people there. It was like around the time of the rise of the skills, the doomsday. Like I want to control what goes into the context window. Like I'm really, yeah, I'm really anal about it. Yeah. Making sure that you don't just get random skills thrown in there. Yeah. So like I will give you an example. We had this workflow where we used the classic commands that Cloud Code used to have. Still are supportive. And some guy was like, yeah, I just cannot get good results. Okay, let me open up your Cloud Code. Let me see what you have. And he basically went into some skills repository and downloaded every single skill. So he had superpowers, all that shit. Everything was turned on.

So it's like the same skills trying to do things and they're fighting between each other. Exactly, exactly. And, like, his context window was already at 40K tokens before he even started. So I was like, yeah, well, that's why. So going back to the company harness. So everybody here, probably this conference, is, like, more on the innovator side of things. They're, like, want to do things themselves. They want to, like, explore whatever. If you go to, like, just a normal company, that's not the case. A lot of people are just, like, very happy to just do whatever, you know, we tell them to do. They just take the ticket, bill it out, ship it. And I didn't think it was as bad. It's not bad, but I didn't think it would be asked that many people that actually would just follow the process. When I would give them the process, we gave them, like, the process, and then we actually built it out into a company harness. But while building, we found that 85% of the code was written in exactly the same way. So refinement, specification, implementation, verification.

Okay. So we could see that after the fact because the artifacts were all there. We could measure how much code they were generated from that. And it's like, okay, let's just, like, make this a harness and make sure that whatever somebody is in that stage is the most optimal context window, the most optimal skills for whatever they need for that specific stage. We measure exactly what happens there so we can see, okay, This team spent this much tokens for this specific stage, for this task. But you started from scratch, right? You're not pulling up pie and trying to build on top of it. No, it's very simple. This is a cool trick you can do with Cloud Code, and you can make it your own meta harness, where basically what we do is we actually start Cloud Code. We inject a small thing into the system prompts, and, okay, if you're done refining, go touch this file. Then we have a watcher that watches that file, and then whenever that file is touched, we close clock code and open a new session for the next stage.

And then we capture all the metrics from that, and we get structured data, all that stuff. Brilliant, dude. Well, this has been awesome. Yeah, thank you. We've got to go to the next talk. Next talk. We're going to send it off. We will see you all soon, and I'll be back with more folks in the podcast room. But until then, enjoy this talk from the main stage. Thank you. Dude. Wow. That was freaking awesome, bro.

番組の概要欄(原文)

<p>What happens when you give 200 engineers who have barely touched AI a coding assistant and a six-month deadline to rebuild their platform? <em>Pepijn Senders</em>, a principal engineer who previously led AI enablement at <a href="https://www.hellofresh.com/" target="_blank" rel="ugc noopener noreferrer">HelloFresh</a>, has done exactly that, and he chose Sonnet over Opus on purpose.</p><p><br></p><p>His reasoning is what he calls &quot;Sonnet maxxing.&quot; A model that reasons less and stops sooner keeps developers involved, and that matters a lot when the codebase is banking software full of hard business rules. The result was 800,000 lines of code, about 60% of them written by Sonnet, shipped by a team that never stopped thinking for itself. Pep also explains how to justify the spend once the token-maxxing era ends and the CFO starts asking questions. He covers a scatter chart that plots token spend against output, the uncomfortable question of whether a 1% better home screen was worth $20K in tokens, why he measures gains in headcount rather than 2x or 3x multipliers, and where he fell short when trying to prove ROI at HelloFresh.</p><p><br></p><p>We also get into a codebase test you can run yourself, why AI recommendations quietly narrow your choices, and multiplayer refinement sessions where the whole team works through the agent&#39;s questions together. Pep shares an experiment where designers and engineers got the same one-shot prompt, and it showed how words like &quot;scalable&quot; and &quot;maintainable&quot; steer a model. We finish with how his team built a meta harness on top of Claude Code, after finding that 85% of their code followed the same path: refinement, specification, implementation, verification.</p><p><br></p><p>Demetrios Brinkmann: <a href="https://www.linkedin.com/in/dpbrinkm" target="_blank" rel="ugc noopener noreferrer">https://www.linkedin.com/in/dpbrinkm</a></p><p>Pepijn Senders: <a href="https://www.linkedin.com/in/pepijn-senders" target="_blank" rel="ugc noopener noreferrer">https://www.linkedin.com/in/pepijn-senders</a></p><p><br></p><p>Timestamps:</p><p>[0:00] Can you still read your codebase</p><p>[1:15] Meet Pep from HelloFresh</p><p>[2:35] What is Sonnet Maxing</p><p>[3:41] Rebuilding a platform in six months</p><p>[5:02] Why Sonnet made adoption easier than Copilot</p><p>[6:37] Proving the ROI of AI coding</p><p>[8:50] Fighting for AI ROI at HelloFresh</p><p>[11:02] Token spend vs output</p><p>[13:24] Tinder pickup lines on the company dime</p><p>[14:13] Measuring ROI in headcount saved</p><p>[18:26] Keeping humans in control of AI code</p><p>[22:26] When Opus and Fable stop speaking English</p><p>[24:19] Multiplayer refinement sessions with AI</p><p>[28:16] Magic words that steer the model</p><p>[32:32] Inside the company Meta harnesses</p>

出典・この記事について

番組
Agentic Conversations (formally mlops.community)(Demetrios)
元の回
Token Maxxing Is Over and Here Is What Comes Next
配信日
要約作成日
公式サイト
mlops.community

この要約・解説記事は、元の音声を文字起こしした内容をもとに AI(Claude)が日本語で作成したものです。 固有名詞や数値に誤りを含む場合があるため、正確な内容は元の音声でご確認ください。制作方法について

X でシェアSpotify で聴くApple Podcasts で聴く

関連エピソード