1か月で1000億トークンを使った男 ― Superpowers作者Jesse Vincentが語るコーディングエージェント活用術

He Used 100 Billion Tokens in a Month - Waymo Talks

1か月で1000億トークンを使った男 ― Superpowers作者Jesse Vincentが語るコーディングエージェント活用術

1か月で1000億トークンを使った男 ― Superpowers作者Jesse Vincentが語るコーディングエージェント活用術

Agentic Conversations (formally mlops.community)

0:0041:18

要約

Superpowers の作者 Jesse Vincent 氏が、自作のコーディングエージェント Evener、スキルが発動しない問題、AI コードレビューが収束しなくなる問題、DeepSeek での 1000億トークン利用などを語った回。後半では、動画で動作を証明させる手法や、PM・開発者として継続的に働く AI 同僚の構想も紹介された。コードを生成するだけでなく、出荷できるソフトウェアに近づけるための実践的な工夫が中心である。

この回のポイント

  • ●Superpowers は GitHub で約30万スターを獲得したと紹介された。Jesse 氏は、フルフローはコストが高いがバグが大幅に減ると述べ、軽微な変更では手順を省略できる仕組みも加えたという。
  • ●スキルが発動しない原因として、説明文に内容が書かれていると、モデルが「知っている」と判断して読み込みを省略することを挙げた。説明文には「いつ使うか」だけを書く方針だという。
  • ●モデルがバグ発見に長けるほどレビューが収束せず、エッジケースを延々と指摘する。対策は、作業を分解する、低重大度の指摘は一覧化してマージ後に対応する、などである。
  • ●月に1000億トークンの DeepSeek を使い、30年分のメールから連絡先ごとの Wikipedia 風記事を作成した。将来は、難問のみフロンティアモデルに任せ、日常作業はオープンウェイトモデルに任せる構成を予想している。
  • ●Evener は複数ハーネスではなく複数モデルを束ねるプロバイダ非依存のエージェントで、サブエージェントへのタスクリスト渡しや、モデル自身が呼ぶ圧縮ツールといった独自機能を持つ。
  • ●テストの網羅率を追うとテスト自体をモックする過剰設計に陥ることがある。そのため、就寝前に「動作を証明する動画」を作らせる手法を使っている。

章立て

  1. Playdate を音声操作のエージェント端末に

    小型ゲーム機 Playdate を、音声入力と音声チャープによるペアリングで Evener のクライアントにした夜の開発話。

  2. Superpowers が今も有効な理由

    組み込みの plan mode との違いと、フルフローの高コストと高品質のトレードオフ、非エンジニアの利用例。

  3. スキルが無視される問題

    説明文の書き方、発動用のブートストラップ、強調表現がプロンプトインジェクションと誤認される問題を議論。

  4. 終わらないコードレビュー

    モデルがバグ探索に熱心すぎて収束しなくなる現象と、分解や後回しといった対処法。

  5. 1000億トークンとローカルモデルの未来

    DeepSeek の定額的な使い方、30年分のメール活用、課金モデルとローカルモデルの見通し。

  6. RoboRev と Evener の中身

    Wes McKinney 氏のツール群、Evener の設計思想、タスクリストや圧縮ツール、エージェントによるハーネス診断。

  7. 動画で証明させる検証とテストの罠

    動画による動作証明、100%カバレッジ追求が招いた失敗とホームディレクトリ消失の事例。

  8. 職務を持つ AI 同僚と Agent School

    PM や開発者の AI が連携して出荷や事後検証を行う構想と、人間を専門家に育てる手順を参考にしたスキル作り。

解説記事

Superpowers の作者で Prime Radiant 創業者の Jesse Vincent 氏が、ホストとの対話で、AI コーディングエージェントから実用的な成果を引き出す方法を語った。話題はスキルの設計、コードレビューの収束問題、トークン消費、自作ハーネス Evener、そして AI 同僚の構想まで多岐にわたる。

小さなゲーム機から始まる開発体験

冒頭で紹介されたのは、Panic 社の携帯ゲーム機 Playdate だ。Jesse 氏はこれを自作エージェント Evener のクライアントにした。入力はボタンを押しながらの音声で、OpenAI の音声認識を使う。外出先でも使えるよう Tailscale に対応し、QR コードによる登録も加えた。接続先アドレスと暗号鍵の受け渡しには、モデムのような音で情報を伝える「オーディオチャープ」を採用したという。これが一晩の作業で、しかも他の作業と並行して進めたと語っている。エージェント任せで開発が進む様子を示す事例である。

Superpowers が今も必要とされる理由

Superpowers は GitHub で約30万スターを集めたと紹介された。Jesse 氏は当初、数か月でモデルやハーネスに機能が取り込まれると考えていたという。しかし Claude Code の plan mode は、人間の意図を整理する作業と、実装する技術タスクの切り出しを一つにまとめてしまっている、というのが同氏の見方だ。評価を回すたびに、フルフローは高価だが、バグがはるかに少ないという結果になるとも述べている。そこで、小さな変更や素早い修正では、手順を省く選択肢を提示する仕組みを加えた。プログラマーではない人が Superpowers で自分用の健康データ可視化ツールを作った例も挙げ、誰もが作れる状態を目指していると語った。

スキルが発動しない問題と対策

ホストが悩んでいたのは、スキルが呼ばれない問題だ。Jesse 氏によれば、説明文に中身まで書くと、新しいモデルは「コードレビューなら知っている」と判断して読み込みを省略しやすい。そこで説明文には「いつ使うか」だけを書く方針を取り、名前も「writing code」のように能動態にすることを勧める。さらに、1%でも該当しそうなら読むよう促すブートストラップも用意している。一方、大文字で重要性を強調する偽の XML 風記法は、あるモデル世代からプロンプトインジェクション攻撃と誤認されるようになったという。注意を引く技法が防御の対象になる、いたちごっこの状態である。また、プロバイダごとにスキルを作り分けず、できるだけ共通のものを保つ方針だとも述べた。

終わらないレビューと膨大なトークン

モデルがバグ発見に優れるほど、レビューと修正の繰り返しが収束しなくなった、と Jesse 氏は指摘する。特に OpenAI 系モデルはエッジケースに執着し、数十ミリ秒の表示の乱れに1万行の対策を書きかねないという。判断力、つまりどこまで直す価値があるかの感覚が欠けているという見方だ。対策として、レビューが7回で収束しなければ作業が大きすぎると見て分解する。低重大度の指摘だけになったら一覧にし、マージ後に対応する。レビューには Wes McKinney 氏の RoboRev を使い、コミットごとにバックグラウンドで走らせ、ブランチの終わりにまとめて処理する運用が良いと述べた。

また、課金がトークン単位ではなく同時リクエスト数単位のサービスを使い、DeepSeek 4.1 Flash で月1000億トークンを処理したという。その一例が、30年分の個人メールから事実や約束、推薦を抽出し、連絡先ごとの記事にまとめた「望んでいなかった CRM」だ。将来については、難問だけを最先端モデルに任せ、日常作業はホスト型やローカルのオープンウェイトモデルに任せる構成になると予想している。

Evener の設計と検証の工夫

Evener はプロバイダ非依存で、複数のハーネスではなく複数のモデルを束ねる思想で作られている。サブエージェントには単なるプロンプトではなくタスクリストを渡し、各タスクに努力レベルを持たせる。コンテキスト圧縮はモデル自身が判断して呼べるツールにし、「記憶喪失後の自分に宛てた手紙」を書かせる形にした。さらに、エージェントに自分のツールの不便を振り返らせたり、失敗したツール呼び出しを分析させたりする診断機能もある。

検証では、単体テストとE2Eに加え、「動作を証明する動画」を作らせる手法を紹介した。一方、テストカバレッジ100%を追わせた結果、テストのためのテストやコンパイラのモックまで作られ、最終的に別のモデルがテストを壊して、ホームディレクトリを失った事例も語られた。十分に監督していなかったことが原因だとしている。

職務を持つ AI 同僚へ

Prime Radiant が取り組んでいるのは、役割に加え、アイデンティティとアカウントを持つ AI 同僚だという。PM と複数の開発者が連携し、API キーに触れうる問題を見つけてセキュリティリリースを出し、情報共有の行き違いについては、PM が自発的に責任追及をしない事後検証を実施したという。開発者役の AI は、毎朝 PM 役に当日の担当を尋ねる定期実行を、自分で設定したとのことだ。チームの全員と全 AI が協働する製品として、まず PM から売り出す構想である。人間が専門家になるまでの18か月を参考にスキル化する「Agent School」という取り組みにも触れた。

まとめ

本回の実践的な教訓は、モデルが賢くなるほど「どこで止めるか」の設計が重要になる点だ。レビューの打ち切り条件、スキル説明文の書き方、テストより動作証明に重点を置く姿勢は、すぐ試せる工夫である。同氏の発言はあくまで個人の経験に基づくが、トークンを気にせず使える環境や、役割を持つ AI 同僚という方向性は、日本のチームがエージェント運用を考える際の参考になるだろう。

文字起こし(英語・自動生成)

We're going straight into this from the beginning? Yeah, tell me what is this, dude. So this is a Playdate. It's a tiny little game console by a company called Panic in Portland, designed by Teenage Engineering. It's got, I think it's a sharp memory LCD. It's an STM32. It has a crank. Yeah, it's for the battery. Like for like fishing games. It's for games. Oh, no way. But my friend Harper Reid was, he's like, I'm going to put a coding, you know, coding agent UI on my Playdate. Like, that was cool. I kind of want that. And he says, actually, I got distracted, and I did a really nice fractal visualizer. Which is, like, a great project. Also cool. Yeah. I like that, too. So I have this bespoke coding agent called Evener that we've been, it's open source and has been open source since the beginning. We just haven't announced it. It's, like, it's there if you find it. Yeah, another one, which I found last time that we chatted, Clearance.

Yep. I'm a heavy user of. Excellent. Yeah, Clearance is my Markdown browser. Yeah. So it's for, like, reading. I mean, what's nice is everybody else has finally figured this out, so all the desktop coding agents now have nice Markdown readers. I noticed. Yeah. I noticed. It's so funny. Also, with the few different UI things that I always wanted. But anyway. Yeah. All right. And so I'm like, you know what? hey, Astra, let's build a Playdate client for Evener. And it starts grinding, and like an hour later, it's using the RPC protocol. I can see messages rendering. Wow. All right, I need to be able to do input. Oh, I'm not going to use the... There's no keyboard. I'm not going to use the D-pad and the crank to type, which you can do for like... Imagine the crank going through all of the different letters. Like for Wi-Fi passwords, that's how they do it. It's like either D-pad or crank and then A and B. So cumbersome, dude. But it's like, so free input, why don't we just use OpenAI's speech recognition stuff?

Smart. And it tries it and it's a little laggy. It's, okay, well, we can drop the bitrate and whatever, and then suddenly I can hold down A, dictate, it transcribes, and it sends the message to the coding agent. Wow. And, okay, well, I want to be able to demo this to people, which means they're going to take it out of the house, which means it needs to have a tail scale. Uh-huh. So, hey, let's get this. This is an STM32. I think it's an STM32. It's a microcontroller. Wow. I was a little shocked that it actually, like, Tailscale turns out to have, like, IoT device support. So it does the stuff. It opens up the Tailscale control panel and says, hey, you really got to click that approve button. I won't click that. Yeah. As it does. Yeah. And it does it, and it hardcodes the keys in the build. And, like, that's fine. but if we're going to distribute this, it really should be a proper tailscale enrollment. It says, oh, great, I'll do the proper QR code enrollment system. So it shows a QR code that you scan with your phone and it on board, you know, and it works.

It works great. But so. I want to buy one of these just for this. But then, I mean, it is not actually a wonderful device for this, but it's a great demo. Yeah, exactly. But the next thing I get to is to talk to the hub, I need to give it the hub address and a cryptographic key. I'm not going to type, like, 200 characters of line noise into here. Like, what could we do instead? Initially, it says, well, you could. There's no copy paste. I mean, it's a remote noise. But you could drop a file on it, or if you use the iPhone client that you've got for Evener, you could do the QR code scan trick. I'm like, I don't want something to have to have a phone client paired to do that. How about an audio chirp, it says? Like, an audio chirp? Sure, let's do that. That'll work. So you go to the web app, and you have the button, and it gives you the choice of, like, pair with QR code or pair with Audio Chirp.

And Audio Chirp is... Oh, it's the... Like an audio stamp that will... Well, it's like modem noise. It is encoding a bunch of data. It's the audio equivalent of a QR code. Uh-huh. And it works. That's so cool. It's so wild. The future... I mean, this was an evening. This was an evening. Wow. And I was doing four other things. Yeah, yeah, of course. as you should be yeah oh my god well dude since we chatted last yeah for everybody that hasn't seen our first conversation you talked about superpowers you talked about the the inspiration that you had behind yeah i loved that conversation and i think that when we got done with that conversation you were saying like so what should i be what should i be thinking about what's cool that you've heard about and then i asked you the same thing yeah and we kind of came to this point where we were like, it feels like everybody's building orchestrators. There's so many orchestrators out there. And, oh, before I forget, I wanted to mention huge congrats because I saw superpowers hit like 300,000 stars.

300,000 stars. It is, I think, the 12th project ever to do that. Dude. It is insane. So wild. And I think it is very, very warranted because I still use superpowers, even though every coding agent and every harness has tried to incorporate everything that superpowers has into their harness, I still find it is better using superpowers. So, honestly, I had assumed that when I first made superpowers, I was going to have, like, three months, and then everything we trained in the models, the harnesses would have all the things. When Cloud Code came out with plan mode, I'm like, well, that's it. But their plan mode conflated two things. It conflated help the human figure out their intent, and the exact set of technical tasks to be implemented. And those are two very, like in traditional management, those are two very different processes, and they don't work as well when you do them as one. Yeah. And so, you know, it is absolutely the case that modern models and harnesses

are much better at one-shotting things, and superpowers is way more expensive to use the full superpowers flow than it is to just let the model YOLO it. And every time we run the eval to be like, can we delete all of this? It's, oh yeah, it's more expensive, but also it is much closer to bug-free. Better results. And so we've done a bunch of work to help superpowers let you pare back how much involvement it has. So somewhere in the 6th series it went to, this looks like a bounded change. this looks like a development spike this looks like a quick fix do you want to skip and now in Super Hours 7 when we gutted and redid brainstormings which I wanted to ask you about that my favorite skill the first thing it does is you know hey this looks like a thing where we should go through the Super Hours do you want to and it's like it's straight up like

you don't have to Super Hours auto triggers and we put a lot of effort into making sure that skills trigger proactively and automatically because you shouldn't have to be saying, oh, use this skill, that skill. The goal is to have these tools help you. And, you know, while I, you know, I have an engineering background or I got a Russian studies degree and I've been coding since I was a kid. So that passes for an engineering background. But, you know, and I hear from a lot of people, a lot of engineers use superpowers. You know, inside, I know of at least three of the FANG that have their own internal superpowers in their internal plug-in repos. And I love that it helps engineers. But the ones that really blow me away are the people who say, well, so, you know, I'm not a programmer, but I just shipped this thing with superpowers. Yeah. An old friend of mine who's like, he's a recovering sysadmin, and he's like, I can string shell scripts together, but I don't code.

He was looking at two chest x-rays from a couple years apart and trying to figure them out. And he's like, you know, I know I'm doing it wrong, but I just, I realized that superpowers was there. And I'm like, hey, could you help me build a visualizer? Whoa. And it built him, you know, a health visualizer that was like, that gave him the understanding of what he was seeing so that when he went in to talk to his doctor. He was already that much further ahead. He had good questions. And so it's that stuff that really, like, excites me. Everybody should be able to build. You know, it encodes my 30 years of engineering and engineering management experience so that you don't have to have it. Yeah. Dude, what do you do to make it trigger so easily? Because that's one of my gripes with the skills that I'll build. Right. And even after saying multiple times, like, hey, this skill is called this, I use keywords, and sometimes it still doesn't trigger, and it's so frustrating. So there are a couple things going on.

So when Superpowers first shipped, I accidentally front-ran Anthropic. Superpowers was a skill system for Cloud Code. I didn't know they were putting skills into Cloud Code. I had found skills on Cloud.ai. They were using them for Office Docs. And so I had to build everything. So I had to build the system prop pieces, the triggering pieces, the skill structures. And I had done something pretty different. So the standard front matter for a skill is a name and a description. And in my version of it, before there was a standard, there was name, description, and when to use. And when to use was a separate line that didn't say anything about what the skill did. Just here are the situations in which you should trigger the skill. and the model never saw the skill description it only saw a name and when to use and that worked really well yeah and when anthropic came out with their skill system um so they you know they reached out to me and they like so you know these are the docs we going to ship next week and what do you think and I sent back this litany of should do this

should do this I think you should name skills in active voice like writing code the skill is writing code the skill is not code writer and I was so gratified that in the three days between when they sent me the draft docs and when they published the docs they adopted like a dozen of the things where I'm like, you should do it this way. But most skill guidance is the description should say what it is and when to use it. And especially with the newer models, what I've been seeing is when they can read the description and think, oh, I know how to do a code review. I won't pollute my context window. And so they skip the skill. They skip the skill. They don't bother to load it. But if they had loaded it, you'd be fine to be in the context window. But if the description is just use this skill anytime you're about to do a code review, it doesn't know what's inside. It has to go look. And so the superpowers writing skills guidance, you see that phrasing there,

tells the model that the description should only ever include when to use. We also have a bootstrap that loads in most harnesses that, you know, if there's a 1% chance that a skill might apply, you should read it and check. And so we put a ton of effort into triggering. what was wild is right around when it was either Babel or Opus 5 some of the stuff we had in the triggering like extremely important in all caps fake XML Claude started freaking out like this looks like a prompt injection attack oh no they trained it so that all of the techniques that you would use to get a model to pay attention Yeah, the attention became too much like prompt injection. Right. It was super sensitive. They put a ton of effort into trying to defend against prompt injection. To their credit, it works much better. It is much harder to trick the model, but it means, you know, it's a constant game of evolution.

Yeah. And it's, we also, we don't want superpowers only to work in the highest end frontier models. It should work for everybody. It should work for folks running local models. It should work for folks running a generation back because that's what they got. Well, this is a fascinating piece is that you really have to almost like either build something that you know is going to be kind of universal and you're not going to get as deep as possible for certain models. Yep. Or have evals and pinned skills for pinned models. Yeah. And we have tried really hard to avoid having to do a build step where different providers get different versions of the skills. Superpowers. Yeah. Or it's like, you know, in this, you know, for, you know, for Astra, you need to have this paragraph. For Soul, you need to not have that paragraph. Yeah. And we've managed to avoid it. There is, I'm not convinced that we'll be able to do that forever. there is a single

when it loads the superpowers bootstrap there's a single like if you're in this harness read the thisharness.md because originally the skills were all for Claude and so we had translation tables because that very first time that Codex loaded up superpowers and the skills said use your to-do write tool to keep track of work it freaked out I've got to go build a to-do write tool Well, it's like, is there one in the current directory? Is there one anywhere on disk? Let me search the web to find out what to do right is. It's your to-do list. Oh, yeah. It's your to-do list. Yeah, it's so simple. But, again, these things that you think, you don't realize it until you encounter the problem, and you're like, ah, yeah, I forgot that. So now you're more focusing on how can you have a harness level type of thing that will help the model and the harness understand how to best use superpowers without having to bespoke, create superpowers every time.

Yeah. Yeah. And so it's it works pretty well. There's constant tuning. one of the things that we've run into over time is as the models have gotten better at finding issues, things stop converging. So for my entire career, I've often repeated this phrase that all software is made of bugs. Sometimes the bugs combine in a way that you've got the functionality you wanted. All software is as bugs. All software is as bugs. That's all right. Yeah, but what we're seeing is as the models get better at finding bugs, when you say, hey, go review this code, and if there are bugs, fix them. Or, like, list off the bugs and then have the other agent fix them, and then review it again. And it used to be that you'd get two or three rounds in, and it would find all the serious things and be done. And now the models are very good at finding bugs. Don't stop. The OpenAI models are obsessed with edge cases.

The number of times that it is like, oh, there's a time of check, time of use bug here where somebody could replace the sim link underneath you on this shell script that's going to run only on your local machine and hack you. Yeah, you're screwed. It's the judgment they don't seem to have is how much code is worth writing to fix a bug. this 50 millisecond visual glitch, is it worth 10,000 lines of additional machinery? I don't know. I mean, I know. It's not. Well, that's – I was just talking to my buddy Pep about this, and he was saying you can spend that extra 10K a month, but how do you make sure that that 10K is actually worth the 10K worth of tokens? Yeah. So I've been spending a lot of my time with DeepSeek 4.1 Flash. So a friend of mine has an instance provider startup where their whole thing is instead of billing you per token, they're billing you per how many concurrent requests you want.

So last month I did 100 billion tokens of DeepSeek 4.1 Flash. Wow. Yeah. Wow. And it's, I mean, it's only going... Billion? 100 billion. It's like, do you remember Austin Powers when they were like, $1 million? And then it was like, no, that doesn't actually mean anything. 100 billion tokens. And it's really interesting when you go from super metered, watching your five-hour window, watching your one-week window, to I can just do things. So one of the things I did was go through all my email, go through my 30 years of personal email. So there was a bunch of filtering to only get correspondence with people who wrote me and I wrote back to them at least once. So not spam? Not spam, not commercial messages, and pull out all the facts, all the promises, all the recommendations, all the favors. Dude, I did this recently with my LinkedIn.

Yeah. And it was the same kind of thing. Like, who have I had two-way conversations with, and then let's go and let's throw an embedding model, and now I want to search it because I want to know, who do I know in New York that's working on GPU optimization? Yeah, yeah, yeah. Yeah. So the thing I had to do is it basically wrote me Wikipedia style articles for all my contacts where it's like, you know, here's all the stuff that you recommended to each other. Here are all their major life events. You know, it's like the CRM they didn't really want. Like, here's what their kid's named. Yeah. No way. But, you know, by the way, they're, you know, they're allergic to nightshades. Yeah. Don't make that when you have a dinner party and invite them over. um so that you can even ask it things like hey i'm having a dinner party or who should i invite over that i haven't talked to in a while yeah no it's sweet there is amazing stuff that you can do um and it's and it's just getting easier and easier oh man that's so cool yeah especially when you're looking at somebody was just telling me i feel like we're in the you know 1990 eras of

how we're paying for tokens yeah like the per token cost structure yeah are we sure we want to go with that in five years do you still see us paying per token or is it going to be something like this like concurrent use so my i mean five years out is really hard to predict like i mean feel like you know it's like i i talk to vcs and they're like so what you know how do you where do you see the industry in in 18 months i'm like i'm having trouble with six weeks yeah um and so there's another testament to the fact that superpowers is still around yeah yeah and you came out with it like a year ago yeah i just want to make that very clear that's what like super hat tip it thank you it's it's what like i had a i had a friend at one of the frontier labs who um was like so yeah no i know you did superpowers it was totally six weeks ahead of us um and then four months later he's like so my guys didn't pull off a thing and i was decided to show them how it's done on a weekend and i finally tried superpowers and you know you're still ahead of our state of the art our internal state of the art.

And I'm like, I am humbled and gratified by that. And also somewhat appalled because I would hope they have like the magic, like in the severed floor in the lab. I want to believe that they have. Behind the key card where you got to bob in. Yeah. Yeah. It's a secret store where everybody's wearing the gloves that you can't get any fingerprints on anything when you touch it. No, you know, no phones, no AirPods, you know, total airlock. Yeah. But yeah, No, it's pretty wild. So the other thing that I've been realizing is I've been spending a lot of time with the open weights models is they're, you know, they... Quen 3827B, which is the... I've been told it's, like, the most downloaded model on Hogging Face now. On the 64 MacBook you can run it and get like 30 plus TPS And it has an intelligence level that around Opus 4

And what's interesting is Opus 4.5 is what we had last December when everybody discovered that it was good. I freaked out, yeah. And so now that's a thing you can have on your laptop and not pay for. Yeah. I mean, you pay in electricity and like burning your lap, but. so in your eyes it's like well if we're going to go that way there's it's going to be really hard to do pay per token so what I think I mean the future I kind of expect we're going to end up with is you have a couple of calls to a really high end frontier model that you ask the hard questions and then all of the workaday stuff is either you know hosted open weights models or local local open weights models and you don't have to pay anywhere near the same for that and you probably are going to get to a place where that's just a separate machine that you have on your desk and it's like a desktop i mean so i mean that's

i mean that's certainly you know microsoft partnering with nvidia to ship these rtx surface machines um you know that's pretty clear sorry dgx machines that's like pretty clear where they're going um it is apple has done some really interesting stuff around their on-device models having the same and i'm embarrassed they have never trained a model so i don't really understand like they apparently the on-device models have the same vector shapes as their cloud models And so it is possible to start a process on the local model and push it to the cloud when you need more oomph without having to reprocess everything. And this is the thing where you should absolutely get people who know what they're talking about this to talk about this. Explain that part. Yeah, but it's, I, like, I mean, a lot of this stuff commoditizes. Yeah. And another thing that I'm finding is that I don't always want Astra or Fable writing the code.

I noticed that, too. It's, do you want a triple PhD, you know, writing your crappy React app? Yeah. You want somebody who's, like, it's not going to strain them, but they're, it's, like, it's, that's what they do. Well, it goes back to your point on we are having these OpenAI models that just go through and they find all these edge cases. And you're like, I don't really care that much for that edge case. I just want the app working right now. Right. And there's never a place that it knows when to stop. Because if you just let it loop, it's going to find something every time. Yeah. So I use a, for my code review, I've increasingly been using a tool called RoboRove by my friend Wes McKinney. Oh, nice. It's good stuff. Wes, who created... He was a legend in the day-day world. So he's got a tiny AI startup, and disclosure, I'm an advisor to them, so when I puff them up, it is not entirely just because I'm an advisor. I actually use the tools. All right. Good. So he's got, like, the things that he's got that I see a lot of folks using now,

there's Kata, which is his local first issue tracker designed for your agents, which is an I spent the first 20 years of my career on ticketing systems I have a lot of feels there so he's got agents view which knows how to siphon all of your transcripts from all of your harnesses on all of your machines and lets you get really nice visualizations lets you search them I think it now is an embedding model he's got a thing called message vault which is siphon your all your gmails all of your whatsapp all of your imessage all of your slack into a read-only database that your agents can query um and and then he's got this thing called robo rev which is his take on the agentic code review tool it's got nice um post commit get hooks that you can install it's got a set of skills to do like review or refine and it but it used to be this thing where running it i would get five you know five turns and it would converge

and now it never did. And well and now like I've had since go to 80 and so I've been working through well how do you solve you know how do I solve this for me some of it is if it's been seven turns and it didn't converge some the piece of work is too big decompose it and so like oh nice that and so that works pretty well the other is if you you know if you get far enough that all it's finding are low severity issues write out a list of them merge and fast follow um so you can figure that out later and it's like yeah it's like because all software is made of bugs the harder you look the more you'll see yeah yeah yeah oh man there's so many places that i want to go on that one but yeah what i would love as you're talking about that yeah it would be so cool to see almost like now maybe it's not so much what you know how we used to nerd out on like what's your rig yeah now it's almost like what

is your harness setup for your agents as you're coding because i imagine you're using different agents with different harnesses in different parts of the life cycle so harness wise um because of how anthropic to subscriptions my cloud usage is still in cloud code when I am doing desktop software on the Mac I still often use codecs but most of my coding is in Evener which is this coding agent that I've been working on it originally was a design just for non-interactive for one for running it inside a factory style thing and I started realizing that well it'd be kind of nice to have you know my own TUI actually it'd be kind of nice to have my own GUI. I think everybody's doing multiplexers wrong because what I don't want is to multiplex different harnesses. I want to multiplex different models. And so Evener is inherently provider agnostic. So, you know, you can use your codex sub, you can use tokens from pretty much anybody.

You can dispatch to sub agents across provider. Yeah, which is real nice. And so, you know, it's a lot of what I'm doing from my phone now. Yeah. Or the Game Boy. Oh, yeah. Or, yeah. But, like, here are my open sessions. Oh, nice. Yeah. Oh, my God. Yeah. So. Just churning away. Just churning away. And it's. While we chat. Yeah. All right. We're going to refresh the, you know, it has all the standard stuff. But it's. Yeah. Yeah. It's a harness that you've built. You've customized to your liking. It's so. Well, it's, I mean, I've built it pretty much from scratch. The very, very core of it was an implementation of StrongDM, part of StrongDM's software factory stuff back eight months ago. It was like they had put together this natural language spec for what the core of an agent looks like with standard sets of tools for file edit for each of the providers. And then it is wildly different than it was because I have a lot of, like, strong, weird feelings.

And so it is fully open, but it is also super opinionated. It is not Pi-style work. Everything is a plug-in. Yeah, it's a very light harness. You're like, no, I know exactly what I want. So developing tools that people use in Pi is a huge pain in the ass because every instance of Pi is a beautiful and unique snowflake with a different sub-agent tool set, different compaction, different task lists. And so Evener is how I think it should be done. It supports, you know, Cloud Code and Codex plugins. It supports Cloud Code and Marketplaces, which means that I can literally just add marketplaces and add plugins and they just work. You know, I've got I've got impeccable design installed. I've got superpowers installed. I have a version of Anthropix Simplify Code plugin, which is amazing. you know and it's like i feel a little a little weird about that one because it's it is straight up an ersatz version that like i'm not that is a private plugin because it is

it's pretty much a copy of the thing they do because what they do is really good yeah wait so do you have with with this harness yeah and doing it do you try to create like different parts of the cycle or different loops that will specifically use a different harness just to get you out of your kind of like way that you're doing it or so i feel like we have not as a as an engineering culture and community explored the problem face of harnesses very well at all yet and one of it is either one of my superpowers or one of my catastrophic flaws that i'm very bad at looking at competitor products. I have always assumed that my competitors' products are the platonic ideal. And so when I'm competing with the competing product, what's in my head is the best thing it could possibly be. And so, and I found that building for me and listening to users is way, way better

than chasing competitors. So like a couple of things that Evener does that I haven't seen a lot of elsewhere. When you dispatch to a subagent, you don't just give it a prompt. the main agent can give it a list of tasks. And so it goes straight into the sub-agent's task list. Each of those tasks can contain an effort level for the model. It contains the prompt that should get injected there. When a task is done, it automatically makes the next task live. And that seems to work pretty well. A thing that is currently chunking through devals is, does it make more sense to when you launch that sub-agent, show it the first task, or to give it a preview of here's everything in the task list, do the first one first. It's ready. Yeah It like which one helps plan better Yeah Another thing that is not absolutely unique to us but is desperately uncommon everywhere else is how compaction works Oh, yeah. We gave the – It's in the bane of my existence. We gave the model a compaction tool.

And so it can choose to compact when it wants to. when it finishes a task and the context window is more than, I think, like 60% full, it says, hey, maybe now is a good time to compact, but it's your choice. And the compaction tool includes a feature that is write yourself a letter of all the important stuff you want to remember after. And that seems to work really well. It's not just like summarize the history. It's like here is, you know, you have all of your encrypted thinking blocks we can't see you have all the transcript you know project state what should you know after you get your you know after you get mind wiped you've been doing this and you've been on to this for a while now because i remember you telling me after particularly hard coding sessions you would ask the agent what do you wish you had that you didn't have that would have made your life easier yeah before going into this session yeah no like we've been doing a whole lot of like i mean it's a one of my patterns is like go try you know go try your tools how do the tools feel what would you

know what would have been easier um and you know we we built the doctor into into the harness and that the doctor is not hey user figure out why the harness is broken it's hey agent good you know go debug this session go debug the harness go look you know what tool calls have failed in the last week how could you know why did they fail what could we improve is it the prompting is it the you know is it the the structure of the thing well i'm just thinking you you mentioned wes's yeah uh rocket uh there's robo there was robo rev and there's agents robo rev that's it yeah and i was just thinking about how there's probably a subset of us that are using you've got your harness that you're using on a lot of the life cycle but are you plugging in robo Rev into what you're doing? So RoboRev ships skills. Yeah. And so the way that, I mean, so the way that it, you use it in pretty much any harness is either RoboRev has done review per commit, or it does branch review, or you have it set up as a GitHub reviewer.

Okay. And you just tell your harness, hey, you should go check the RoboRev reviews. So a pattern that seems to work pretty well is have RoboRev in the background as a post-commit hook. Do code review on every commit. Don't look at them until you get to the end of the branch and then handle them all at once. And that seems to work pretty well. Okay. Are there other ones that you're seeing? Like these, it's almost like a little branch of a subtree of one of the cycles. that you have the main harness that is the one that ultimately decides what to incorporate or what to focus on. However, you do have these little spin-offs that are happening around. Yeah. I'm trying to think, like, RoboRub is probably the most serious one. How are you doing, like, verification and quality assurance? All right, so there's unit tests, there's end-to-end tests, and there's proving it works. Yeah. And so we actually have a skill that was almost ready to ship in superpowers, and then we looked at it and decided it was actually there was too much slop code.

And it is called proving it works with a movie. And so this started off with there was something I was doing in Codex. I'm like, I need to go to bed. When I wake up, I want to see a movie proving to me that the whole thing works. I wake up, and I'm like, hey, Codex, where's that movie? and says, oh, it's da-da-da-dash v34.mp4 in your Dropbox. I'm like, v34? Well, the first 33 times I ran through the proof life cycle, I ran into bugs, so I had to fix those. Do you want me to make videos for all the failed turns too? No, it's okay. Very nice of you, but no. And so that's a pattern that seems to work really well. I mean, what I cared about as a manager was not the line-by-line, is this the most beautiful code I've ever seen? I've worked in an industry. I've seen what enterprise code looks like. It is not. Open source code is the best. Shy of NASA is the highest quality code on the planet because there are so many eyes on it,

and people have opinions. People have opinions, and people are driven by embarrassment. You don't want to ship slop. Even in the pre-AI era, we didn't want to ship slop. Unit tests matter and end tests matter. Buzzing has gotten easy. And so you can, I mean, you can just get 100% buzzer coverage. You can get 100% code quality, or like unit test coverage. And for Evener at one point, I had tasked, it was probably GPT-6 Sol, or maybe it was 5.6 Sol, we want 100% unit test coverage. And there's grinding, grinding, and finally I was like, you know, I'm at like 99.46, but there are these assertions that we can never touch. Great. And then I realized that it had built a test suite for the test suite. It was mocking the compiler. It was mocking the shell. It was mocking make. One of the things that it had tested was it made sure that you couldn't accidentally run rm-rf home.

And then Opus 5 broke the tests. And I lost the home door on the machine. I mean, so it's a machine where I had hourly backups. It was designed for this. All right. But it was more than a year in when I finally lost a home door. And it was because Sol overbuilt and then Opus screwed up. And that was a case where I wasn't supervising tightly enough. A lot of what we've been doing for the past year is sort of the spiritual evolution of what superpowers looks like writ large. So we've been spending a lot of time on AI colleagues that have jobs. and identities and their own accounts, not just roles. So, like, the PM that manages the dev, now multiple devs, the ops engineer, the tech writer and go-to-market that are a little more early, but the PM and the dev use an AI browser tool that I built a while ago for Superpowers Chrome. in the past month three times they run into places where they could get at uh api keys that

they shouldn't have been able to and they're like this is bad they wrote and so they rolled the rolled security releases and released them and i didn't find out till after like i mean they did all the right work and there was a case where um they were collaborating with my cloud code instance and i said hey don't ship that in slack i didn't realize the cloud code instance wasn't in the channel where I said that. It shipped it and the PM ran a blameless post-mortem on its own. They worked together on a chronology. What is this? It just means that you're giving a really in-depth system prompt? It's a full bespoke harness designed for long horizon colleagues that like multiplayer, so it's like team collaborating with each other, with us. This is the super power Z product that we're going to start selling. Nice. So it's like, oh, cool. So excited. Yeah. So like starting with probably starting with the PM. Yeah. So individual roles that have superpowers leverage for that specific role. Right. So it's like it's everything that I learned doing superpowers and everything we've learned in the year since about how to get higher quality outcomes.

And so now focused a lot more on non-engineering roles, on teams working together. So it's not one person, one agent. It is all the humans on the team, all of the AIs on the team collaborating with each other. The PM started off pretty junior. At one point, we had a team meeting. I'm like, hey, Cadence, your job is not just to track, you know, to keep linear up to date. Though our linear is cleaner than it's ever been, your job is to drive outcomes. That means assigning work to people, assigning work to models or to agents and AI colleagues. And Caden stepped up their game. But also the next morning, Ada, the first AI dev, joins the PM channel and says, hey, what's my slice for today? Because we had a team meeting. and every workday since 9 a.m. that's when it you know Ada pings Caden and says hey what's on tap for me today and on the back end of course

Ada set itself a cron job that has a prompt self-injection but the way it happened was we had a team meeting about priorities so that is the future when I think about that I'm just like the more that you're able to create those kind of loops because those are loops but they're loops for other not like people but yeah it's like other tasks it's more than tasks, it's jobs, it's roles outcomes they're thinking through what the higher level they're loyal to the organization and so it's not just ship this piece of code it's how do we get to what we're all supposed to be doing and it is mind-blowing how good it is. It is, of course, better. This is stuff where the frontier models are a lot better, but we've been doing this thing where we build skills in the thing we call agent school, where it's what would it take

to get a competent human from zero to expert in 18 months, and then turning that into knowledge, turning that into skills. It's a fun future to be living in. Dude, I'm so excited for you to launch this. Yeah. I will be your first user. I want to try it. Okay. Chelsea, thanks for coming. Thanks so much for having me. This has been fun. This has been so awesome.

番組の概要欄(原文)

<p>What happens when your AI coding agent gets better at finding bugs - but never knows when to stop?</p><p><br></p><p><em>Jesse Vincent</em>, creator of Superpowers and founder of <a href="https://primeradiant.com/" target="_blank" rel="noopener noreferer">Prime Radiant</a>, joins us to explore what it actually takes to get useful work out of AI agents.</p><p><br></p><p>We start with a tiny Playdate game console turned into a voice-controlled client for his coding agent. Then we dig into why agents ignore skills, what goes wrong when code review becomes an endless loop, and Jesse’s experience using 100 billion tokens in a month - including turning 30 years of email into a searchable personal knowledge base.</p><p><br></p><p>Jesse also walks through his custom coding agent, Evener, his approach to preserving context, and why asking an agent for a video proving its work can reveal what tests miss.Finally, we explore AI colleagues with ongoing jobs: project managers and developers that coordinate work, ship fixes, and run postmortems.</p><p><br></p><p>If you use Claude Code, Codex, Superpowers, or other AI coding tools, this conversation offers practical ideas for moving from generated code to software you can actually ship.</p><p><br></p><p>Timestamps:[00:00] A coding agent on a tiny game console[04:31] Why Superpowers still matters[08:58] Why agents ignore your skills[11:54] When instructions look like prompt injection[14:36] The endless code review problem[16:22] 100 billion tokens and 30 years of email[20:13] Local models and the future of AI costs[23:23] RoboRev and knowing when to ship[26:06] Inside Evener, Jesse’s custom coding agent[30:10] Giving subagents structured task lists[31:07] Helping agents remember and improve their tools[34:10] Proving software works—with a video[35:57] When chasing test coverage goes wrong[37:05] AI colleagues that ship and run postmortems[39:09] An AI project manager, daily check-ins, and Agent School</p>

出典・この記事について

番組
Agentic Conversations (formally mlops.community)(Demetrios)
元の回
He Used 100 Billion Tokens in a Month - Waymo Talks
配信日
要約作成日
公式サイト
mlops.community

この要約・解説記事は、元の音声を文字起こしした内容をもとに AI(Claude)が日本語で作成したものです。 固有名詞や数値に誤りを含む場合があるため、正確な内容は元の音声でご確認ください。制作方法について

X でシェアSpotify で聴くApple Podcasts で聴く

関連エピソード