AIエージェントには見張り役が要る：意図ベースのツール選択、WebMCP、検証の仕組み

I am very lucky nowadays because I got Grok in my car, so I can chat to Grok in my car and have a wonderful time. We don't need radio anymore. We can talk to AI about things. Are humans going to stop being the drivers of computers, effectively? With the whole super intelligence situation, how are we going to deal with that for the AIF? I mean, like, is he just going to be there ASIF now? I got a design. So I'm into calligraphy. And so I created, like, a calligraphy design. probably five years ago and i've been flip-flopping about getting it and now yeah i ended up got it

getting it and it's it was it basically says um let's talk about jev oh great that's like you designed this like years ago that's like amazing you know how how how reminiscence my friend that's like crazy let's talk about jeff yeah like my goodness but what are what are you feeling on jeff it feels like we've kind of we're on the down slope of the hype train a lot of stuff has come out where you at it's exciting yeah break it down why okay so what i was doing in my car the other morning when I was driving home from the gym as you do um I am very lucky nowadays because I got grok in my car so I can chat to grok in my car and have a wonderful time we don't need radio anymore we can talk to AI about things that's what I was doing when I was

going to Japan I just was talking to AI and trying to learn Japanese phrases that would be useful to me i did like i've been off that's another topic of learning languages by the way like let's talk about it in a moment because i've been trying to use ai to learn languages as well which is related to my gym adventures because my gym instructor or like two of the gym instructors speak spanish so for some reason that i think they speak english during the class but they can also speak spanish so I was like oh maybe I want to know Spanish but but anyway on the dev part what I was thinking about is you know how we've been speaking about security and like MCP tool calling security as well as having the next layer of security is auditability right so being able to make have an audit trail of what happened okay so then i had this idea okay i have not tried this idea

just be very clear this was a morning conversation i like okay and then i talked to grok about it but grok said that there are some papers out there in the world that's come out so i'm not completely crazy but i love how it's just like there are some yeah you want to know what they are we'll figure that out later but they're out they're told me stuff it gave me names of authors do i remember those i just was happy that i wasn't crazy that was like step one what's the theory what is the so imagine you have an agent a tool calling agent but the agent doesn't know what tools are available to it okay so if you have that scenario the agent cannot make the recommendation to call a tool that it is not allowed to call it also cannot doesn't know exists yeah yeah So instead of the agent defining saying which tool to call, the agent defines what he wants to do and intention, right? What do I want to achieve? And then you use Jev with the intention, and then you have the selection you can make is different tools.

So you have Jev recommending the tool based on the intention and the need. Okay. And what's kind of fun about this is that then we can actually start evaluating and fine tuning. Like, okay, do we need better tool descriptions? Do we need better, like a better way for the agent to define its intent and the action that he wants to take for what purpose? So I thought that was kind of interesting. Then it also becomes, it's also auditable beyond which tools were called when. it is then auditable to why was the tool called. This is assuming that the agent that pings over to your MCP server is going to give up that information, right? So this is the idea that they doesn't send, having JEV making, effectively giving your agent a single tool, which is the tool finding. The JEV tool?

yeah it's a tool for finding tools that you use dev underneath and the contractors provide me with your desired action and effect and why you are making this why you want to use this tool then you can find if it has access to any tool like that or otherwise you can you can come back and say there is no tool for doing what you want to do like there is nothing available to you yeah impossible task you just gave me yeah like there's nothing for you but there was I mean there are several things out there there's like how can we use JEV for better tool selection for the task so that's when I found out I wasn't crazy that obviously that's a use case yeah I've seen I but I was thinking about it from a how can we effectively not even make the agent aware of all the tools and also how can we make it so that it becomes auditable at the level of

intent well did you see recently pi the harness the creators of pi i think the dude's name is marco he put out a blog post about how pi now is making mcp a first class citizen and the reason And it means that it's in the core harness, whereas before it was like an extension or an add-on. And the reason that he was saying this or one of the things that you get for free in the blog post, if I understood it correctly, was that now you can use JEV also with MCP and do what you're talking about, like classifying tool calls or just using JEV. in the MCP selection or the tool selection, all of that fun stuff. So I thought that was fun to think about and see how Jeb almost like propelled MCP in a way.

And the use cases are plentiful out there on the Internet. What I saw and I thought was one of the most interesting takes out of all of this noise was Steve from Builder.io. I don't know if you ever follow him, but I love his content. It is top-notch stuff. He broke down all of these different posts and what they were saying they were using Jev for and kind of like demystified him and said, look, this is probably bullshit on some of them. And then on others, he said, well, we've been hearing a lot about Jev's computer use skills. so he ran benchmarks to see if it is going to save you a ton of money and if it is actually just as good and what he found was like it's not that good come in the uh accuracy i think some of those benchmarks were like it only got the stuff right 33 of the time

and what to use in like a computer use yeah like effectively a tool selection my price and the the other piece was that that 33 was after a bunch of retries so like the one the first time it was 18 and then you would have to retry and get to 33 so it ended up not being that cheap either like the whole value prop of jev is hey it's faster and you get the same amount of accuracy and it's super cheap right but after he ran these benchmarks he said yeah not quite what is actually the way to do it though is something that he was calling like a blended jev or a hybrid jev type of mode because he was saying look luna or the smallest model whatever haiku plus jev so you use jev first if it doesn't work then you fall back to luna or i have to make it more like you know so that you just have that reasoning based efficient

rather than a yeah and a lot of times the input that you are constricted to or confined to with jev is actually one of the downfalls it's like a double-edged sword because it's great that you can do this with this classifier model but then you can't just feed it a screenshot right no okay so that's why computer use becomes really complicated no it's it's it's more that's where if you have the fallback of a smaller model you get the accuracy higher yeah when jev can't figure it out and like but i think it's interesting like on the topic of like dev and inputs i feel like do you there must be like a new art you know how we talked about prompt engineering before right yeah and in context engineering and whatever else we want to call those things i think that

dev is going to have that entire like learning curve for people as well like how do i actually get the most all of a sudden get good stuff out of it yeah people are becoming machine learning engineers i like i saw a funny take where it was something along the lines of wow jev made a thousand ai bros on twitter have to learn classical machine learning it's like i think that's another fun part about ai or the ai world is that there is it's coming like there's so many parts to it the touches on things that are not new constantly and I find that wonderful and fascinating how we are running into problems you know whether they are perceived or real and when I say they perceived it like oh it a problem and I need to figure out to solve it But actually it already been solved

Or whether it is like, in this scenario, say, like, Jeb, it's like, oh, now we're going to introduce all of these classified concepts. And now people have to learn a different way of thinking that they maybe haven't used in their system designs or applications before. well one thing that i feel like jev could be really useful for and i haven't seen people talking about it as much in this computer use scenario is or just browser use whatever it may be is if you combine it with web mcp okay because did we talk about web mcp last week yes we talked Yeah, we talked a bit about it. Okay, so you're thinking that if you have WebMCP, which effectively allows a web-hosted site to expose tools to an agent that is interacting with it. Exactly. How are you thinking you combine JEV with this? Well, just the way that you were talking about,

JEV is classifying which tool to use. Yeah. Okay, yeah, that would make it a lot easier. Yeah, wouldn't it? Right now, exactly. it has to figure out stuff without any tools to grapple with in the computer use it's just kind of like determining do i press button now do i press that link do i go up and press something in the header and it's weighing its options but if you said hey look here's the four happy paths that people usually take on our landing page and we expose those as tools so maybe it's an add to cart tool it is a like go see the catalog tool whatever i'm using an e-commerce example yeah then jeb can classify that based on the intent like we were talking about before yeah okay and then i think to me sort of make then it also makes sense in the world of like if you have the you know you

have dev for decision making then you have a lighter model that is the action that powers the more agentic behavior overall right so then you can even okay i don't know if this is a crazy idea or a sane idea but actually have if you have your like luna or haiku or even if you just go sonnet or whatever like you have that model that can even enhance the options that are passed to jav for the context as in the intention of the user or the task so i don't know like could you then construct a request to jav with help from your another model yeah when it's interacting with a web mcp server you mean or yeah like any tool selection effectively but i mean like if you are on the i mean the most typical uh use case right it's a pet store demo it's a

pet store demo right and we're in the pet store demo like if we have the context of whether we are shopping for cat food or dog food right or something like that yeah maybe if we know the job the ultimate task we're up for is buy cat food maybe by having the inference model provide that context of the overarching agent task to the input for gem or maybe you don't even need inference maybe you just need a dumb agent harness that doesn't need inference for it when i say dumb agent harness i mean something that just puts these things together it might not need inference tonight you know what you're you're making an interesting point because where should that responsibility lie should we as the agent builders or whatever the agent harness we're using should that be on us or should it be on the web mcp builder to say look there is this pet store and

a lot of things that people want to do when they come on this pet store website is buy dog food so that is a tool that i give you and if you call that tool buy dog food it just is straight to uh you know like pass go and you're good you're straight and maybe you like then get to different place okay i have a question about web mcp now i have a fun question because i don't actually know if this is how it can work or does work so imagine you have buy dog food tool yeah so you landed on their pet store shop this it'll buy dog food okay now you're at so you've gone through the first door of buy dog food so now we're in the virtual dog food aisle if we're in the virtual dog food aisle can in web mcp it now surface new tools on that yes exactly right exactly so each Page is a new, you get new tools. So it is, for example, you're just saying, I'm this pet shop store,

and what I'm going to expose as these four main tools when someone hits my landing page is buy dog food, buy cat food, buy rabbit food, or buy, like, accessories. And then the fifth tool is, like, search tools because I have a million other tools there, but that's kind of the code mode type of method where you don't want to overload it with tools. You want to give the four top ones and then search. And so you then say, okay, buy dog food. Boom, you land in the dog food page. And now you're met with four new tools, which are like big dog food, small dog food, vegan dog food. Yeah, whatever. You get all these different descriptions, or it's probably more like select type of dog food, select size and all that, and then add to basket. Yeah. And then, boom, you go to basket, and then the next one is decide how you're paying, just like you kind of would with a human thing.

but each new stage after you call a tool you are met with new tools yeah yeah okay i just had like a picture in my head there was like the skill trees in computer games where you get one skill and you unlock another skill in the skill tree it really is that yeah it's very much a decision tree if you look at it that way. But then there's also some other cool things that I saw folks doing with WebMCP servers that are, in a way, built uniquely for the capabilities that WebMCP has. And I realized this because I went back and I listened to the interview that I did with the keynote speakers, Frederick and Dominic from AgentCon Amsterdam, and they were talking about how one of them, if I'm not mistaken, said that there is a WebMCP server issue that just got merged

that is specifically for debugging. so you can create a server on your website yeah that is only for debugging and so that agents can then go and use your website and debug like basically test it out and make sure everything's working properly like a QA for a QA agent can like QA the site and click all the things you shouldn't click and see how it breaks and it's not exposed to it's not like you would expose that to the general public but you would want to use that to sanity check if your web mcp server or if you're just your website in general is working as you think it should be working i think okay the more we speak about web mcp i realized you know how we talked about web mcp last week and we've talked about at one point is humans are are humans going to stop being the drivers right

of computers effectively and are we gonna get are we gonna go completely headless and i think spoiler alert for anybody that didn't listen no we decide no right that's pretty yeah clear we think humans like looking at stuff uh and being part of it but maybe not everything there are Some things are really boring. But what I did find really fascinating, you know how you said earlier around dev is making people think about machine learning concepts that they may not have been exposed to before. Now with WebMCP, something that people who've been writing and creating APIs for the last couple of decades, something that has always been hard and frustrating is to move the idea of APIs to be workflow enablers rather than oh they sort of crud create you know yeah update delete retrieve yes it's like not like this is you have to figure out your workflow

because a lot of APIs were sort of exposing the raw tools, if you will. And you then had to assemble your own workflow around the available tools. It's sort of Lego without instructions. But something that emerged in API design probably over the last bit more than five years, and it was there before, but it started to get popularized, was to do workflow-based APIs. So thinking of how you enable, well, what is the program that is cooling our system actually trying to achieve, and how can we design the APIs to achieve the workflow instead of just interacting with data or functions? And now when you talked about WebMCP, I think it's that shift of thinking going from, I'm going to turn my APIs into MCP tools.

I think actually fundamentally whatever it is you're building, whether you are building APIs or MCP tools, I think just focusing more on the workflow is better. Yeah, it's the jobs to be done framework, right? Where people say, oh, you don't want a drill. You want a picture on your wall. Yeah. or you're like you want clean floors you don't want a vacuum cleaner you want clean floors so it just so happens that the vacuum cleaner is the tool to get you clean floors and there's others that will help depending on what kind of floors you have and so you have to be able to choose which tool is right for the job in that moment but the or you can hire a cleaner and never buy a vacuum cleaner yeah yeah and that's like hosted fast software right or the i do like that metaphor and the thinking around it of you just want the outcome for example i want the dog food

to show up at my house yeah i tell my agent i don't care like what it needs to do ideally the agent is able to then go and seamlessly using the marketing term get that done very yeah very quickly and with maybe one tool from a web mcp server but the bigger but then you kind of go into this question of does it matter if it has the web mcp server if it can do it on its own already without it yeah that's the other that was interesting yeah question but i think that fundamentally if we can design better for workflows whether that is better user experiences better APIs but the web mcp tools fundamentally I think how you achieve it is it sounds like it's not necessarily that important it's going to be important

because things only take off if people adopt them but I like the idea of getting people into workflow thinking and like how do I achieve something and I just you know we talked about the pet food and also being able to make decisions right when I shop for my cat's pet food because I am a last minute kind of person do I not tend to try and get my cat food like the day before they're about to run out so sometimes I have to pick a more expensive cat food so I can get it delivered tomorrow because the other one that they normally have I would have to wait two days or two days how long but you know that is sometimes the problem and i think it's nice like could like you have the the goal of the agent is like you i need to get cat food tomorrow right so even if there is cheaper food available in two days but i need it tomorrow right so having having

constraints yeah more constraints or more context around it yeah i haven't figured that out that is something not with cat food but for me it's with coffee i need coffee beans and i tend to order my coffee beans like just a little bit too late and what happens is i have to go and ask my neighbor for coffee beans to hold me over while i have like one or two days of no coffee and i always think like remember back in the day when internet of things you was coming out and people were like wow it's going to be so incredible because we're never going to have to order milk again it'll just magically appear at our doorstep when we run it out of milk and that didn't happen and i think we went about it all wrong but i'm not sure how we could go about it so that i don't have to think about ordering coffee it just comes i mean the subscription is sort of but i've tried subscriptions and the problem with subscriptions is i either get too many coffee

beans too much because sometimes i go and i go on a trip and i'm gone for two weeks and you don't want old coffee beans you want fresh ones right well it's not even that it's just that all of a sudden i have three bags of coffee beans at my house because normally if i'm home i'm drinking a lot of coffee and so i'm ordering a subscription that last week the amount of coffee you were drinking yeah you gotta get that l-feigning to counterbalance it uh so but the so i don't know where we're going with that but it is something that i've been thinking about is how can i take that mental task yeah and it's not only coffee it's just mental tasks i am just in my head thinking about the amount of coffee you consume and how this is a problem that follows you around in your life whether that's in japan and you've got too many coffee cups that you've carried around because you don't have a rubbish bin or whether it is i need to figure out how i have coffee beans

in my house the right amount of coffee beans i don't want too many i don't want to overflow with coffee beans because then i feel like i need to drink more coffee and that's also another thing I just want the right amount of coffee beans. And I was listening to a podcast from those YC dudes. I think it's like Michael and Dalton. And they were talking about how to create. They want to see startups being created that are trying to bring the luxury experiences that rich people have into the average everyday person's world. okay and in this example uh it was kind of a douchey thing to say what's the coffee delivery no selection of the right coffee for you no just take your mind off coffee for a moment let's go with this one what it was is they said you know i am a person who does not have to worry about

things being empty in my house anymore because i've reached a certain level of wealth that i can have somebody else do that for me and i wonder if there is a way to bring that experience to the average person like me you know so those guys they can afford to have like a house manager i do not have a house manager as we've established because of my poor coffee timing wouldn't it be great do you have your daughter though i think you need to train your daughter for this that's a much cheaper and less energy wasting way than starting a business around how can i bring perfect coffee beans to your door at the right moments i don't know she might have like compensation if she wants for a responsibility like this always keep a tab on the coffee level coffee bean yeah no but it is to be fair though like this conversation so we hosted like a few

weeks ago like a coffee lean coffee after dark in atlanta as part of like it was our first AIF meetup and we had like conversations but what you were saying though one of the guys he said that he really just wants an AI agent that can be a what do they call them those people who plan vacations for your travel agent yeah proper personalized travel agent that doesn't just you know come up with suggestions but books managers okay uh flights are delayed or like something happens also you know a good travel agent like will help you book new ones like replan your trip yeah and he's like he's like that's what i want right and i feel like this topic of having ai help you with managing your household or even like bring those luxury experience like a personal

personal travel agent is one of those home manager is the probably one of those personal tutor you know we said we were trying to learn languages with help of ai right the having a personal language tutor is a level of luxury yeah i agree i agree 100 and you see that it's funny because it feels like uh saw a twitter thread where or a post on x that someone said everyone is building the same thing and it does feel like everyone right now is building what i've heard termed the personal assistants or life assistants. Muse. Was it MetaMuse? Muse came out. You've got, it was like Inception, I think, or Instinct. That's it. That was the first one that was valued at $3 billion in their seed round, and everybody was looking for the promo code to get involved.

And then recently, Dots. Yeah. OpenAI just released it. I'm actually very stoked about dots not because of the life assistant thing but because they finally brought in the notion like document editing to the interface but that's a bit that's a different story I've been asking everyone I know that works at open AI and quad or anthropic yeah to create that type of an experience because i noticed i'm dealing much more with the chat as a document and a living document than i am uh wanting to take what i get from the chat and then pipe it over to a google sheet or pipe it over to my notion workspace yeah i just want to have that inherent in my and and be able to organize the information yeah and tag in the agent when i want like highlight something and tag it in or leave a comment and do all that stuff or drag a certain

text block or create a um some kind of a visualization with data or at someone and tag them in or at a database and tag it in and so i wanted that whole like notion vibe but inside of my chat and turns out open eye open ai listen to me because now they have that they listen to you but they're hopefully more people who felt the same way it must have been me i'm the for you the single user have you ever heard that expression squeaky wheel gets the oil yes i was quite squeaky on this one about it and hopefully it was because of me that they decided to do that and i like to view you as a champion of the underserved i'm i'm like a visionary i think is what they would call me yeah should i just say that i sound like such i felt

like a douchebag saying no i think you said you're a visionary and not a dover that's fine exactly exactly you said that other people did the things yeah openly i made the thing you just but it was my idea all right don't forget uh if you have loads of ideas all of them are yours at one point or another it's fine i know it is that that's a whole nother topic of inspiration and where ideas come from but uh that's for different how uh how is i have a funny question not funny i don know it a question with the whole like super intelligence situation how are we going to deal with that for the aif i mean like is it just going to be the asif now i don know what to do with it there was never yeah there was never a a i uh a g i a f you know like it was always just a i so whatever i think

it's interesting right because people are obviously arguing AI has always been the umbrella term and super intelligence is just a type of AI exactly you know what I wanted to actually talk about though is this idea of building verifiers in your code and in your and it goes back to because I remember as I was mentioning the QA assistant for or just the QA WebMCP type of server that you can create and what I recognize is I want to learn from people who are building their verifiers how they're building their verifiers because it feels like that's where that single point on how you're verifying the code that is being churned out by your agents yeah what does that look like what is that whole process and what

are the new rituals that we're creating because we used to have it where you would submit prs and now maybe you can get rid of a lot of prs by creating really good verifiers and verifiers i guess another term for it could just be like a tests or a really robust cicd pipeline but i do think this is interesting because people again people have been dusting off an old approach that had um i mean this is what i've learned from some of my friends and colleagues is so do you know about cucumber testing no what is it okay it's very much a java-based testing framework i wonder if it works for other programming languages but let's go back to the basics of what it's trying to solve the fact it's called cucumber is probably rather irrelevant but it is funny because it's so old essay framework but instead of testing like method functionality

or like feature functionality you are testing like business outcomes like more you are testing against you you start off defining again coming back to idea workflows and I think some people back when it was popularized might get this wrong but I think people refer to us like business driven development it was a time when it was also the time when you often had in large companies like a very divided like here's the tech engineers the software engineers people doing that and the businesses over there and then you had business analysts and people who sort of translated business needs into requirements for software and this idea was effectively take the requirements and sort of encode them into the tests, if you will.

And I've seen people been using this because that allows you to maintain the business impact and outcome-based information into your code, effectively. So you don't have to ask yourself, well, why did the agent write this code? Because there's a test that is using the code to verify that the intended outcome actually happens. So what could be a really simple idea? Maybe think of onboard, set up a new user in the system. setting up a new user in a system would be more than just creating the user in the database right it would probably involve creating the user set up permissions send like email to like notify them that they have an account to activate blah blah blah whatever right it would probably make sure

they're part of a certain team and whatnot and the idea that my friends showed me was that that would I mean, one of the big questions they said that it comes before quality is why was this code created in the first place? Being able to go back, OK, this code was created to solve this particular problem. And this is especially relevant maybe in very large code bases and like code bases where you are working to try and understand why the coding agent even wrote that code for what outcome and purpose. yeah it makes me think about the audibility of it and tracing back the decision that was made to actually include this code into the code because there was some yeah there was some spark that caused this to actually materialize yeah and it and i think like what's interesting about

writing these type of tests is that you have to be you know how I said back in big enterprise like 15 years ago you had the business you had the software engineers and then you had business analysts in the middle you have to operate more like a business analyst yourself you have to have a deeper understanding of the actual business not just how to execute the code as a software engineer or builder or whatever you want to call it yeah and I think that was interesting to me in how we have seen these like in-between technical people become very powerful in their era of AI because they have the ability to express their intent because they've had to they've had to express it over so many years already to humans and since AI operates on natural language they have now been able to interact with this much better than they have

before say i think that to me is quite fascinating well so the verifiers that i was talking about were way it was a totally different feel or ballpark than what you're talking about but i do like that you took it here because in a way it reminds me of shift left and trying to shift left so far left that you're at the inspiration of what then comes out of it yeah so that's great the verifiers i was thinking about joe are and how to architect verifiers and what i've heard folks saying that they do are things like just making sure that before any pr gets submitted you spin off a sub agent and that sub agent goes and it tests the whole new changes and your whole application to

make sure that nothing broke and everything is showing up the way that it should and it will write a document with screenshots that verifies okay I clicked this button and it should go here and it does go here here's a screenshot of me being here and oh I clicked this button it looks like this is off now I need to go and change I broke this page or I broke this link or I broke whatever And so you're able to have a, again, it's like this QA agent that is going and verifying the code that's trying to be changed before it gets to the whole PR and CICD pipeline. Like limit the amount of human review so that we can reduce it. I think like even just the simple things that are not agent driven quality controls is so often not checked by people.

Say, for example, a common thing that you would have in a software project is that you have limping, testing, etc. Right. That's like some basic stuff. but the amount of like in open source where in an open source project this exists the amount of PRs I see get opened that don't even pass the existing sort of testing harness right yeah and I find that interesting right like that's another human problem like the human is clearly being told by the PR that stuff's not working or their coding agent or whichever we want to see it as right but then no action is taken taken to address it so that's another like how do you do that deal with that as a maintainer like I try to like ping people and be like um do you want to

fix this or sometimes it's hard because sometimes part of what they've done is actually not bad then I'm like do I need to go rescue this and like bring it in because there is something useful in there yeah that man that's something we should talk about next week is just open source projects right now and how you are fielding all of the AI generated code that's coming through and how you are looking at maintainers or just fly-by-night contributors that want to get something merged and they'll throw some slop at you and you get inundated with slop and you have to set hard rules and what those rules are so let's do that next week i gotta jump now though yeah but i like the idea of trying to figure out how can we we have got some help from a google gemini tool to try and help us with like reviews but it's it's still like i wonder can you even enable people

even more like you were saying can we try and even help coding agents help themselves with a testing agent build a verifier that will explain to the agent why this code is going to automatically be rejected if you submit it that type of thing there's yeah awesome well cool i will see you next week yes hopefully i see you too no i will see you yeah