営業・マーケ担当者はエージェントに関心がない：HubSpot CPTO ダンカン・レノックスが語る実用的なAIエージェント戦略

Generative AI is transforming industries at an unprecedented pace. But as AI changes how you or your team work, one thing is clear. Your skills also need to evolve. At DataCamp, we offer everything you or your team need to adapt and thrive with AI. Whether it's business users looking to get the most out of ChatGPT and Copilot, or developers and data scientists looking to fine-tune models, you can learn the entire AI skills spectrum on DataCamp. Power your AI transformation today. Start learning at datacamp.com If you deliver too much information as part of context, that can actually create a huge problem as well. So the technology is getting better and better every day. And I think for some of this Pimpler discovery and buying journeys, like we actually make it more convenient for the end user customer to interact with an agent and do a lot of the process that way. I love hearing all the latest news about AI agents, But not everyone thinks that way. Sales and marketing people, on average, get less excited about the tech than I do.

This puts HubSpot in an interesting position, since they're busy creating agentic tools for that very audience. We're not simply doing it for ourselves, though, of course, we do do that. We're doing it for 300,000 businesses that are HubSpot customers today. Can I trust what these agents are going to do on my behalf with my customers? How do I understand how much it's going to cost me to do this? A lot of customers are very nervous about AI, generally. Our guest is Duncan Lennox, the Chief Product and Technology Officer at HubSpot. He's responsible for both HubSpot's SaaS platform and the agents that run on top of it. Duncan's a seasoned tech exec, having had stints at AWS and Google Cloud. I want to pick his brains on how you sell agents to salespeople who don't care about agents. Hi, Duncan. Welcome to the show. Hey, Richie. Great to be here. Thanks for having me. Yeah, great to have you here. Now, I've seen over the last few months there have been quite a few startups pitching the idea of AI, SDRs, AI, BDRs. How close are we to completely automating sales? I think we're a little bit away from completely automating sales.

I mean, the way we think about it is a little bit different. I think there's a lot of productivity time that salespeople waste on admin tasks today, whether that's updating the CRM or researching a prospect. and we see tremendous opportunity to kind of give them all that time back and let them focus on the actual human-to-human interaction part of sales. So the technology is getting better and better every day. And I think for some of the simpler discovery and buying journeys, like we actually make it more convenient for the end user customer to interact with an agent and do a lot of the process that way. But for more complex, higher-end sales, it's really about how can we help the sales rep spend more time with the customer and take away the drudgery that consumes a lot of their time today. Okay, I can certainly see how if you're trying to automate a multi-million dollar enterprise deal that normally takes months, then you're probably not going to be able to do that entirely with bots. You mentioned that there are simpler flows. Do you want to talk us through how that works then, like where the agents come in for a simple sales deal?

Yeah, absolutely. So if you look at typically how a lot of those kind of buying processes work, So let's say it's more complex than a consumer product, which typically people are already used to going to a website, buying it directly, and so on. Let's say it's a little more complex than that, but not your multimillion-dollar sales example. I mean, the beauty of the AISDR type of approach, or inbound agent, as we call it, is that ability to help the customer really, a prospect really understand, go through the discovery journey, ask a lot of questions, understand whether there's a good fit to the products and services, and then automate much or, in some cases, all of the whole buying journey for them. So it really turns into a convenience thing from the end-user customer point of view. Okay. I like that idea. So it's basically like it's the next iteration of your sort of support chat box. It's just like finding out how can I help you and then hopefully redirecting you to the right place. It's very much an extension of that, and frequently for a lot of companies, including our own, that we started with the agent technology

in the customer support space. So like that's something we've been doing for a while. One of the reasons was there was really good product market fit early for that. Like at the end of the day, the person that has the support issue, the query, whatever it is, they just want to get an answer fast. And it's not that they need to speak to a person necessarily. In fact, often that really slows down the process, right? Because you've got to call up, you've got to sit in a call queue, then you've got to explain everything about what your problem is. then the support person has to run through all their classic things like you know did you try unplugging it and plugging it back in or you know whatever the equivalent might be and eventually hopefully you get to an answer but sometimes what happens is it's like well i need to hand you off to somebody else that's a little bit more of a problem and you know what's the first thing that happens when you get handed off to the second person you got to start the whole thing all over again like why are you calling essentially you know so what we call our customer agent really eliminates all of that and just makes it a much easier, faster experience for the person that has the query that's calling in. And that let us build and perfect the technology of the ability to handle

that. And one of the things that we have to do as HubSpot is we're not simply doing it for ourselves, though, of course, we do do that. We're doing it for 300,000 businesses that are HubSpot customers today. So everybody's set of products and services are a bit different. The type of customer they have is a little bit different. How they want to engage even is different. Is it WhatsApp? Are they sending an email, you know, et cetera. So building that out and making something that's really good in that space and then is a natural kind of foundation to extend into sales. So on the customer support side, we are at, you know, in the low 70% resolution rate for our customer agent, Meaning like more than seven times out of 10, it can resolve the issue that the customer is inquiring about without the need to hand it off to a human. And a little bit like the sales example earlier, too, the ones that are getting handed off to people to handle then are typically the more complex ones, the more impactful ones, where it really is helpful to actually get to speak to a live person.

So taking that technology and then extending it earlier in the buying journey, you know, the customer support agent is working post-sale. You start extending it much earlier into that whole discovery process and buying cycle and applying the same kind of technology. Okay, and that's kind of cool, I guess, that it works 70% of the time in customer support, but it's completely automated. And that's on average, by the way, like just as well. So we've got customers that have resolution rates that are 80, 85, 90, 92 percent. It's a little bit dependent, but we're getting over 70 percent average across the board. So which is which is pretty good. It's an industry kind of leading place to be in terms of resolution rates. OK, I suppose if your resolution rates get higher, it means you're probably having like a lot of like really stupid billing problems that are the same thing over and over again. And it's probably a good sign that you should like go and fix your product to that point, I guess. Is there that kind of feedback loop then from you've got an agent trying to solve this?

No, there is. I mean, and I think actually when you can tie into the data much easier when you have something like customer agent, because it's creating and tagging all of these incidents as it goes and so forth. And it's really interesting, the issues that pop up, because you'll get kind of two broad classes of issues that pop up. I mean, of kind of the more challenging ones. And some of them will be a good indicator, as you said, of like, actually, there's an opportunity to make the process better for customers. So they never run into this issue. And it could be something as simple as figuring out, like, well, how do I reset my password? And it could be, so that could be a design user interface issue of like, we're not making that obvious enough to customers. So when they get into that situation, it's super easy. Or maybe our actual password reset process is actually not very effective. Like, you know, and the other class of problems then is more the ones that happen periodically. Like, you know, you may have a big event or something changes externally that causes customers to start using your product in a different way or et cetera.

So it's creating a new issue that you've not really seen before and that starts to pop up. And one of the things that we do with Customer Agent is we have another agent that kind of works in the background behind it called Knowledge Base Agent. And it's watching for what new issues are popping up with regularity. And then it's recommending new knowledge-based articles that you can write that both get made available directly to customers and people, but also then feed the customer agents so it's able to do a better job of responding to those issues. So you do get this feedback loop going of improving the system all the time and then having that flow over to improving your processes and indeed your products and services as well. Okay, that sounds like a very cool, like positive feedback loop. Since you mentioned quantitative benefits, I think there's a lot of hype around like making use of agents, but it's sometimes difficult to show what the return on investment is. Have you got any data from your customers to say, this is what we've achieved using the agents?

Yeah, no, we do. And I'd say, you know, I'd say, look, it's still early, like, you know, much as all of us are immersed in this every day and love it all. For most of our customers, it's still pretty early days for the use of AI in their in their business. And I think particularly in our case, because HubSpot's customers tend to be small businesses and that don't have the expertise, the time, the resources to really learn this stuff. They're just trying to grow their business, you know. But all that being said, we do get to do this interesting cohort analysis because across more than 300,000 businesses of all shapes and sizes across 140 countries, we can see and look at the data of customers who are using our agents versus those that are not. And we see very clear improvements in the go-to-market outcomes that matter. So at the end of the day, for our customers, we're in the go-to-market platform business. So we're helping them get more leads into the top of their funnel, book more meetings with prospects, close more deals, support those customers, etc.

When we look at, say, something like MQLs, marketing qualified leads, we see over three times more MQLs created by customers that are using our agents and our AI as part of what they're doing in their campaigns by 3.6x, actually, on winning more deals. There's nothing more fundamental for most of our customers than the actual deals that are closed won. We see about a 3.2x improvement in the cohorts that are using AI and agents. And then we've been talking a lot about customer agent. We see over twice as many tickets resolved for customers using customer agent versus handling it entirely manually. I mean, more than three times improvement on that sort of early to mid sales funnel. That's nothing to be sniffed at. It's dramatic. It's dramatic. And, you know, look, that's an average. So some are getting even more than that. Some are maybe not quite seeing that yet. like. But it's very clear that when you're leveraging agent solutions like this, the right

way, I should say as well, and we can talk a little bit about what we've seen and what really has an impact on the quality of that outcome, but it is pretty dramatic. Okay. So that's the kind of the positive side of things where it works. I'm curious, are there any places where you're like, okay, agents don't really show an impact yet. This is where you really want the humans to spend the time? I think a lot of it comes down to not necessarily a particular area, but maybe the class of problem within an area. So if I go back to the customer agent example or we use the AISDR example, there are inevitably for customers the high risk, high reward, high impact type engagements where the complexity of what's involved or the criticality of the decision that's getting made is just that it's better having at least a human in the loop. It doesn't mean the agent can't be assisting the person on our customer side in doing that. And so like an obvious one there would be if you see a if a customer is not getting their issue resolved

and in customer support through customer agent like we misunderstanding it or we not getting them to to a resolution that ability to escalate that to a person like becomes an issue So it more that It more knowing when to escalate when to have people in the loop rather than necessarily a full class of problems that just doesn't work today. Because the technology has gotten quite sophisticated, as you know. So we can apply it to most anything. It's just a question of how much of a bucket of problems could be resolved, you know, primarily with the agent. Okay, so basically agents throughout the whole sort of go-to-market life cycle, but you probably want agents to do the simpler things and then have humans for the whole complex stuff. Is that the general? I think potentially, yes. I do think it's still early, and I think the bigger challenge for our customers today is really wrapping their head around trust. Can I trust what these agents are going to do on my behalf with my customers? How do I understand how much it's going to cost me to do this?

And, you know, you talked about ROI, making sure they understand that. So a lot of the work we do beyond, of course, building the high quality agent in the first place comes down to, you know, what we call trust hill climate. How do we help customers try out these agents in a very low risk way where they stay very much in control at all times? I get to see what the output is going to be like. maybe then start to use it and try it out in lower risk scenarios for them. And over time, as they get more comfortable, as it gets more fine-tuned, be able to delegate more and more authority to the agent. And with Customer Agent, we've built a lot of tools to let our customers do this, for example. So not only things like where you can test out the agent like yourself, act as a live customer and actually see how the agent would respond to you and so forth. Then as you get comfortable with maybe having it handled some queries, we do things like you can choose the time of day that the customer agent responds versus the humans responding.

So people will often do things like nights and weekends when they don't have coverage right now. So let's give our customers a better experience during that time. But it tends to be a quieter period. And then over time, kind of, you know, ramping that up as they go. So a lot of that turns out to be a big part of making customers successful using the technology. It's a lot more than just building the technology itself. Absolutely. I mean, you mentioned trust. Now, the data-framed audience skews very, like, technology excited about. So, like, a lot of people love agents. But you said a lot of your customers are small businesses. Do you get that same level of enthusiasm for AI with your customer base? I think what we find is that a lot of customers are very nervous about AI, generally. And this is a general issue. It's nothing to do with go-to-market or, you know, any particular agent. And they're looking to understand. And it really does come back to that trust kind of notion of, is this going to work for my business?

Is this going to help me? We see that as a big part of our value proposition for customers. that we hide and abstract away a lot of the underlying agent technology. Like we have our own agent harness. We've built our own, you know, context engine on top of all of the structured and unstructured data that, you know, that we have for an individual customer. And then we're using that to drive the right outcomes with the agents. And from their point of view, they don't have to worry about how do I provide good context? Which model should I use? You know, what is an agent like kind of thing? So because at the end of the day, again and again, what we hear from them is, I'm just trying to build awareness, win deals, delight customers. Like these are the three core, we call them the golden journeys. These are the core things that they're trying to do. And AI or any technology is really just in service of that for them. So they're not nerding out on it generally the way you and I might. Okay, that makes a lot of sense. It's like if you work in sales and marketing or customer success,

then you just want to make some money, like close some deals and help your customers rather than like using the technology just for the sake of it. Yeah. I mean, of course, there's always, you know, classic visionaries and early adopters who saw the opportunity of the technology early and kind of glanced on to it. And, you know, it's super fun always talking to those folks and they give us tons of great feedback and input. But, yeah, most customers are exactly as you say. I'm just – I got a business problem to solve here. And if you can help me solve it, HubSpot, and if leveraging Identic AI is a great way to do that, then that's fantastic. But at the end of the day, it's the outcomes that matter. Okay. Now, I'm kind of curious about HubSpot's role in this because HubSpot is traditionally a piece of SaaS software. And at the start here, there was talk of this SaaS-pocalypse and that thing that was overblown with the idea that agents were going to replace SaaS software, but yet you're very much pushing agents. So how do the two sides of HubSpot sort of reconcile?

You know, I think it ties back to the outcomes piece again. I think for me, I've been building software for a long time, and this is certainly the most transformative technology I've ever gotten to use as part of building software, which can create challenges because it changes things from how it used to work. But it's also the most unbelievable opportunity to have bigger impact. If you think about HubSpot, we just celebrated our 20th anniversary back in June. And so for the first two decades of the company, we started as a marketing automation company, grew into building out a full CRM, and for a number of years now, we've been a full go-to-market platform. So we cover everything from, you know, the first touch of a prospect to your website or responding to an email campaign, all the way through that buying journey, post-close, supporting them, continuing to grow and delight that customer. But for most of the 20 years, what that meant was we were building tools and workflows that let go-to-market teams drive their outcomes.

And for the last couple of years, what that's been really transitioning into is, well, now it's really all about a platform that can help you manage hybrid teams of humans and agents working together and where we can actually do a lot of the work for you. So like an example of that would be, we just had our big conference a few weeks ago. And one of the big things that we announced that that is a product called Marketing Studio. So it's squarely targeted as marketers, as you would guess from the name. It's really all about how do you build campaigns and run them. But it does that in a very new data-driven and agentic way. So it's proactive. It's looking at all the data from every campaign, from every sales cycle, from every that you have across all of your customers. It's recommending areas where maybe you need to do work. Like we might be saying your visibility inside of LLMs like ChatGPT or, you know, perplexity are low. We recommend you build a campaign there targeting this segment.

You know, would you like me to put the brief together for you? So campaign agent, a new agent we launched builds the full marketing campaign for you. Then if you want us to, content agent kicks in and can actually build all the assets, like the ad to go inside a chat GPT ads or the landing page, the email, et cetera. And then nurture agent is another new agent that we just launched that as leads come in, can build customized email follow up flows tailored to each individual person. And all of those things I just said are the things you would always have done with marketing campaigns, essentially. But people would have had to do all of those things along the way. And now we can help the marketing teams by doing a lot of that work for them. But the last thing I'd say on that is we're very much, you know, building agentically because that is the best way to drive the best outcomes for customers now. But we're hiding all of the complexity of that. So what I didn't mention is that whole flow that I just talked about, you can do all of that just by talking to Breeze Assistant, which is our co-pilot that's available everywhere in the product.

And it even figures out what agents to use and how to get things done. So as a small business owner, you don't need to know even those set of agents that I just named. You don't even need to know they're there or how they work. So what we see is that there's never been more opportunity for us to do more on behalf of customers. And it's about taking that platform that we've always had. We've up-leveled it with an agent harness, with the growth context engine, and a set of agents that we've built. And now we can just do more on top of that for customers and in the form of agents. But they don't need to necessarily know or care about that unless they're interested in understanding it. Okay, so I guess the business goals have not changed at all. It's like people still want to do the same things. We're back to like you just want to close deals. I always think it's like, you know, on one level, it's like, you know, here comes the new boss, same as the old boss, right? Like, so, and, you know, we can get so sidetracked by the technology. And it's amazing. Like, what Frontier Models can do, like, is astounding and it's improving all the time.

But, you know, the best way to build products, to build software, has always been deeply understand the problems that customers have. And people often think they understand them, but they haven't deeply peeled the onion of what's really going on there. And then how can we apply technology to find better ways to solve those problems? And that sounds really obvious, maybe on one level, but there's a tendency for us to get carried away and excited, especially maybe as engineers, as product nerds, by the technology and put the cart before the horse. Yeah, definitely. I like the idea that you solve the problem first and the technology is supposed to help you out with that rather than the other way around. I feel like there's a few different ways of agents and SaaS software interacting. So you mentioned there's the Breeze Assistant, so that's basically I guess a chat interface to solving all your problems. You've got the traditional HubSpot interface, but I guess there's also, can you use other AI platforms? So if you're

a copilot user or cloud user, OpenAI user, and then accessing HubSpot from there? When do you want each of the different interfaces? Well, the way we think about it is quite straightforward. So the number one job for us is we have to deliver the best, cheapest, fastest go-to-market outcomes. That's what you have to be able to get out of the HubSpot platform. That's our job. The second part then is we want to meet customers where they are. So we have a huge range of customers. And we spent a lot of time digging into the data. And they're really in a variety of places on what you could think of as the AI maturity kind of spectrum. And interestingly enough, it's not really divided by, neatly segmented by geography or company size. It's like at all geographies and company size and market segments, you're people who are, you know, we're really early adopters, real innovators, very comfortable inside of Claude or ChatGPT to do a lot of things. And then you've got the vast majority of our users, I would say, still who are less comfortable with that, at least yet, and are more comfortable with something like Breeze Assistant because they're used to being inside of HubSpot.

They're go-to-market teams. They're familiar with HubSpot. It's always there. Breeze Assistant has the big advantage that it knows everything about you and your business because it lives inside of HubSpot. But then you've got a lot of people who are still very comfortable in what I might call the traditional web UI or our mobile app and so forth. So for us, what it means is we have to be able to meet customers where they are. And we're kind of doubling down on that. And that includes if they want to do it inside of, you know, one of the LLM. So this is the so-called headless strategy. Right. So where we build things in such a way that like if a user doesn't want to, they essentially never have to log into HubSpot. Like if they're a sales rep or, you know, they're a marketer and they want to work inside of Claude or ChatGPT or Copilot or Gemini, we've built connectors, you know, a year or more ago into each of those products. Like HubSpot is fully exposed inside of those products. And we think the best combination of that is HubSpot is still doing the work for driving those core outcomes.

And then you're using the SHAP products like the CLODs and the CHAT TPTs to build around that and do other things. It orchestration and so on So we committed to this notion of an open ecosystem that we going to deliver UI parity into our headless environment In other words anything that you could do through the UI in HubSpot, you'll be able to do inside of one of these LLMs. And it's all about meeting customers where they are. Okay, I like that there is that choice. And depending on what your preferences are, then you can do it. And in theory, it should work just as well at whatever interface you use. earlier on you were talking about uh building trust and i suppose the biggest way to build trust with agents is to make sure they actually reliably work do you want to talk me through like how you make sure that uh all the the hubspot agents actually give the right answer yeah yeah yeah we we do it we do a ton of work around this as you can imagine right so it uh there's a lot of things we've learned along the way i would say you know maybe maybe i'll cover two or three uh

key things we've learned along the way. We can dive into any of them in more detail. But the first one is that the models are very powerful, but they're not all-knowing and they're not magic. So on their own, although they know a lot about the world, they really don't know anything about you and your business. And like that turns out, perhaps not too surprisingly, to have a big impact on the quality of the outcomes that you get out of So what we found is being able to provide the right context, like in real time, at the point of need, when we're going to go execute a task or drive an outcome. Let's say it's resolving a, you know, a customer ticket like with customer agent, you know, or it is part of a, you know, researching a prospect for a call that a sales rep is going to have a demo that's coming up tomorrow. that being able to identify and inject the right precise context into that prep task or resolving that ticket has a huge impact on the quality of the outcome and the ability to resolve it.

And interestingly enough, what we found very clearly, and there's been a bunch of academic work that's shown this to be true more generally, if you deliver too much information as part of context, that can actually create a huge problem as well. It's a little bit like a LLM version of the classic ML overfitting kind of problem, you know. So and so getting that precise, correct, right amount of context is a huge indicator of the quality of the outcome you'll get. It also, by the way, affects how much it's going to cost, because if you throw a massive amount of context at a model, of course, that's consuming a lot of tokens as well. And in the worst case scenario, you might actually be getting a worse output for more money as a result of all of that. That's probably the single biggest factor that we've seen that impacts quality and safety. The second one is we built our own agent harness. And we've been doing this for about three years now, so long before the term agent harness had even been created. And what that lets us do is really two things.

The first one is it lets us be model agnostic. So as amazing as all the frontier models are, they have what's referred to as jaggedness. You get to edges where suddenly they tip over and they're not very good at things. And those are in different places for different models. So the net net of that is there isn't really a single model that's the best at everything across go to market outcomes. So our kind of AI model agnosticism lets us test and use the right model for each of the right tasks individually. and indeed because the state of the art is changing so much and so frequently as things change if a different model pops to the top in driving the best outcomes we can just route to that model instead and so that that's that's the second thing and i'd say the third and final thing on it is we spent a lot of time using the two decades of go-to-market expertise that we have and we built out thousands of evals like the actual definitions of what good and great looks

like for different go-to-market outcomes as well as what bad looks like and we have a bunch of automated testing that's running continuously using our harness and so on to run those things and that really lets us capture a high quality high safety of everything that our agents are doing actually a few surprising things in there so i think uh a lot of the story around context uh at least i see is people saying oh you need to provide more context that's going to be better answers but you're actually saying that you don't want to give it too much as i suppose it's when like if your boss just gives you like a hundred different instructions at once and a load of background that you didn't really need to know about then you're going to get confused about what you're supposed to do so that's interesting that you're trying to tune it to just the right amount of context do you have a sense of how you get to that yeah i mean i i think well i think that it's it's really where the combination of our experience over two decades and all the data that we have available in the system kind of comes together. And that is something that's a little bit unique about HubSpot.

You know, I was mentioning earlier that, you know, it is this end-to-end platform. So if you think of an individual business, let's say we're a plumbing business, you know, in a Midwest city, a suburb of a Midwest city, just to pick an example, you know, 25 employees. But you've been using the HubSpot platform for your marketing campaigns, whether that's email, whether that's ads you're running, blog posts, et cetera, all the way through then, you know, obviously through the service cycle of customers choosing you, working with you and post that engagement. And what that means is we have essentially like a lot of data about what those full end to end journeys look like. And that includes things that went well, where you like won business and went and supported that customer and all went really well, or prospects ultimately decided to go somewhere else. So we can see what, for your business, both good and bad look like. And then do that times hundreds of thousands of businesses over the last two decades.

And we've evolved a lot of expertise in what good and great looks like. And then we can quantify that down in the evals and show the context that is the right context to ingest around that. And that's a very data-driven process. If you have these evals that are, you know, very data driven ways to evaluate good and great outcomes, you can start to train, you know, our context engine to what is most effective here. What set of inputs here gets us into that sweet spot that is sufficient context to give precision, but not too much to lead into the overfitting problem. Okay, yeah, I guess the evals is the important thing there. It's like as long as you're testing, does this work or not, that's going to give you some feedback as to like, well, can we improve it? You know, evals are, you know, the new PRDs or, you know, it's like it's not just it's not just the modern equivalent of QA, like the quality assurance work. We used to always do with software where you'd write tests that would run automatically. And these are much more sophisticated because the LLMs can reason and they can do much more sophisticated work.

work. But the contrary example is if you don't have these high quality evals, how would you know if what you're doing is any good? I mean, at the end of the day. And again, for us, this is a huge part of the value we provide and that we see as our responsibility. Again, if you're a small business, you can see if leads are growing, if deals are closing, et cetera. But having the time, the expertise to redefine with precision in a data-driven way that can be measured and iterated on that's really our job to do a lot of that for you and that's what we've been building into the agents and then the platform that powers all the agents it's one of those things where like uh testing has never been like the most exciting part of the job it's actually it's a very cool kind of field at the moment with like how do you test these uh these uh agentic systems well actually do you want to make it more concrete have you got some examples of like what good evaluations look like? Yeah, I mean, like I think if you, like there's a lot of different pieces to it and it's very dependent on the area. So like let me give you an example of something that is

a big part of what HubSpot is typically doing for customers, which is emails, writing emails. And that could be in different, multiple different contexts. It could be a personalized follow-up and nurtured email as part of a campaign. You know, you expressed interest in a campaign that we ran and so now we're periodically sending you follow-up emails we're trying to get that cadence right so we're not irritating you that we're adding value and that there's other ones other emails would be helping a sales rep write a follow-up email after a demo like what's the next step you know and then of course we talked about customer support that all works customer agent handles email as well as all kinds of other chat live chats and so forth too across the board there and you're getting into something that is quite fuzzy and subjective in many ways, which is, well, what does it mean to write a high quality email? And the use case is different in each of those cases, but there are core components to it that are common. Like, so, for example, one of the ones that we spend quite a bit of time on is across

a number of metrics that we quantify is a humanity score. And the humanity score is literally looking at how human does this sound? Like, does this sound like a real person wrote it? Take the example of a sales rep follow up, right? So if we're trying to take, you know, admin burden off of the sales rep, one of the ways we can do that is write the follow up email for them. And what happens there is the rack will have been using our hotspot note taker when they were doing their demo with the prospect. So the note taker will have captured everything that was said during the meeting, does a bunch of analysis on it, gives you a summary and updates the CRM and a bunch of other kind of useful things. But it's then using that to help write the email and the follow up email. But it doesn't just look at what happened in that meeting. It looks at every interaction that you've had with that prospect as a company and as a sales rep. And it uses that as well. But then the humanity score comes in to actually be able to quantify, generally, does this sound like a human wrote it?

Like there's been, you know, you see all this AI slot. We see it on LinkedIn every day, et cetera. Then it's like the second you start reading it, you know AI wrote this. Like it just doesn't talk the way a normal person sounds. Like, you know, and so humanity score is about capturing that. But then taking it to a level above that, which is, is it capturing the style and tone of this individual sales rep as well? And one of the ways we improve there is we have a pretty high bar on it of what we're delivering. But the rep obviously gets to see the email before it's sent and they can make edits to it. And when they make edits to it, the agent learns from that. and we update evals and memory so that we know now next time, okay, here's an extra nuance of the style of this rep that we can apply automatically next time around. And using that approach, over about a period of six months earlier this year, we went from, on average, reps making edits to the emails that we wrote for them two-thirds of the time

and only being comfortable sending them unedited one-third of the time. we flipped that entirely where now it's more than two out of three times it's good enough they're like yep send so so that's this kind of iterative data-driven way leveraging evals that we are constantly kind of working on the quality of the output it's fascinating stuff and i'll say i love the idea of having a humanity score to say is this does this stuff sound like a real human it's quite often difficult uh for marketing so i had a case recently where we've been creating uh like a landing page for a webinar campaign and i told my colleagues like you know this all this text it reads like ai slop what is it he's like i wrote every word of that myself like this is how bark to sound sometimes so uh yeah it is it's tricky but i love the idea that you're trying to sort of pass out and make sure that you're not just creating slop everywhere yeah and it's not and it's not about you know as well it's not about trying to fool people like it's about trying to be authentic, like, you know, and connect with people. Like, I mean, in the Rep case, you know, we're not

trying to replace that Rep. We're going to help that Rep be more effective, but it still has to feel like a human to human connection as part of it. It's kind of like the old uncanny valley in animation, right? Like, but if you try to make it, you know, in the old days, like when the early days of Toy Story and Pixar they deliberately went with this cartoon approach because if you try to do something that looks realistically human like people brains like can tell like there something wrong here it almost becomes creepy and ai slap i think is a little bit like that although i will say as a as a personal lover of the m dash i hate that that has now become a problem because like we can't use m dashes anymore because people just assume ai wrote it if you if you do that so yeah it's uh it's been a sad scapegoat is a very useful piece of punctuation. I want to go back to something you mentioned earlier about, you talked about using frontier models, and I'm curious if they still have a role in, like, generating text for sales and marketing. I feel like a lot of the frontier models have been focused a lot on coding, cybersecurity, and scientific things.

Do you still need frontier models for writing your marketing copy? Yeah, absolutely. I mean, I think you don't have to be using the biggest, most cutting edge model from an open AI or an Anthropic, like for a lot of the tasks that are there. I mean, a lot of the tasks that we're doing in go to market, I mean, I call them the meat and potatoes, like a go to market outcomes. And those things, because the models have gotten so good, you can use a smaller model in the same model family to get high quality outcomes. And that's, again, a big part of the value that we bring for customers, because we're using all those evals and that agent harness and we can figure out, actually, you don't need to throw the full power of like, you know, a fable, you know, or an Astra of writing this email to your point that if you use the right context, you can use a much smaller, cheaper model. And that means we can deliver more value for customers. So that's certainly true. Absolutely. Yeah. So the small models kind of work. And this leads to the idea of like saving money, which I think is a very hot topic at the moment.

A lot of people have been stung by high AI bills this year. So do you want to talk me through, like, how do you save money on your AI bill and, you know, get good value for money? Yeah, I mean, well, here's two parts to that. You could, you know, there's the like, what's our job to do as part of that? And then how can customers think about it? I mean, the answers are largely the same, but we're trying to do most of the heavy lifting on our customers' behalf, right? So it does come back to what we were talking about earlier with context, evals, and an agent harness that you can run yourselves. Because those three combos together give you a lot of control over the number of tokens that are being consumed, both input tokens and output tokens. And as long as you can measure the quality of the output that you're getting against that, then you're going to be able to optimize for quality and cost. I mean, the mental model I always use is you have to kind of triangulate three things, quality, performance, how fast you get the output, and then the cost represented for that.

And the reality is, if you think of that triangle or that three-legged stool, the answer or the mix of those things is different depending on what you're doing. So, you know, if you're if you're if you're doing something like responding to a customer support ticket, you need to make sure that the quality bar is very high because it's going to go out directly to one of your customers. That still doesn't mean you need the biggest model in the world. It's still going to be about more about injecting the right context. But there might be other scenarios where we're taking on something that's very complex, like planning an entire marketing campaign and building out a brief for that and then constructing the assets, etc. That's a complex set of tasks. You're going to need more sophisticated models for that. But you still need to be able to balance costs. So having an environment that either gives you control over that, like if it's something you're doing yourselves, I think is critical. For us with go-to-market outcomes, we take on that burden for you. So we'll essentially, you know, our promise to customers is we'll help you deliver the highest quality, best value, fastest go-to-market outcome for your needs.

Okay, I like that. So as long as you're providing the right context and, you know, you've got all the evaluations in place, you're not going to be wasting money then because the AI is not working. So, yeah, at that point, you should be fairly cost efficient. I guess the flip side to this is if you are serving AI features or AI products, then the pricing model seems to have changed. So we've gone from subscription pricing to usage-based pricing. And the next one that gets talked about is outcome-based pricing. Now, I know HubSpot has some outcome-based pricing. Do you want to talk us through what that is and where you can use it? Yeah, and it's really interesting. And I think everybody's still trying to figure this out, right? for what is the right way to deliver value and then get paid for that value appropriately. And overall, our model is a hybrid model of seats and credits. So you buy a certain number of seats for the platform. That comes with a built-in bucket of credits every month that you can use on things that consume credits. And then you can purchase more credits if you need them.

And the essential idea there is the core functionality to drive marketing or sales or et cetera is there for you. As you get more value out of output and outcomes-based approaches, you consume credits and you can buy more credits for that. But you're doing that because you're seeing the ROI. You're seeing the value exchanges there. Yes, it costs this many credits to get a qualified lead or to resolve a customer ticket. And that is fair value to me. So my cost of that scales with the actual usage and outcomes that I'm getting out of it. So as part of that, not everything, maybe even most things, are hard to quantify what is the outcome and the quality level of that outcome to make outcome-based pricing fair in every scenario. So we're still experimenting with it, I would say. But there's two key agents where we're all in on outcome pricing. Prospecting agents, which is about getting you qualified leads, and then customer agents, which we've talked about, which is about resolving support tickets.

And we moved both of those to outcome-based pricing back in the spring. So with customer agent, you used to pay for every conversation, essentially a back and forth with the customer, regardless of whether we resolve the issue or not or whether it ultimately needed to get escalated to a human. We moved the pricing to you pay for resolve tickets only. So and that's the 70 percent or so on average that we're resolving for customers. So in other words, if we are unable to resolve the issue to the customer satisfaction and we need to escalate it to a human, you pay us nothing. And then similarly on prospecting agent, you can tell us how many qualified leads a day you want or you want for sales rep, again, to stay within your budget. And prospecting agent, which is doing all kinds of kind of complex signal and intent-based prospecting on your behalf. So it's looking at target companies and target individuals and, you know, what's going on. You know, are there are there signals there that these people are ready to buy your product or services, etc.

In that one, we used to charge for the number of companies, essentially, there were contacts that we were monitoring on your behalf looking for, you know, new intent signal. And we shifted that to it's only if something hits a bar of a qualified lead and we hand it off to you. You only pay for those qualified leads. Okay, yeah, I can certainly see how something like writing emails is a bit fuzzier on whether it was a genuinely good email or not. So charging for outcomes is going to be harder there. So it makes a lot more sense like in the customer support case where you can say, okay, we've got a resolved ticket. There's no ambiguity. Is there a threshold at which point it makes sense? Like you mentioned that 70% of resolved tickets sort of works for you. I presume if it's only 50% resolved tickets, then the economics are maybe not there. Is there that threshold then that you have to get the agents reliable enough in order to have outcome-based pricing? Well, it's super interesting because it puts much more of the effort back on us as the vendor to have a very high-quality bar.

That seems like a very good thing for customers, keeps us honest, if you will, because we're going to get paid more the more of these tickets we can successfully resolve for you. So if we're only at 50% for a customer, the economics for the customer are probably still very good because they're only paying for those 50%. And those are, by definition, resolved tickets. So they're tickets that a person would have had to deal with otherwise. But for us, it aligns incentives, if you will, because getting that to 70%, getting that to 90% is a win for us and a win for the customer. The customer is getting more of their customers' issues resolved faster. and we're obviously generating more revenue from our individual customer because we're doing more on their behalf. So, you know, there's a lot of incentive alignment, I think, around the whole model of where you can define clearly an outcome and a quality bar. I think it works quite well. It just turns out to be quite complex for things that are more complicated, right? How do you start

to define on them. But we're continuing to iterate on us and, you know, looking for opportunities around it as we go. Okay, that makes a lot of sense. I love that. There's that alignment between vendor and customer. I guess that's a good situation. All right. So we talked about quite a few different things around like having more AI, more automation throughout their go-to-market sort of cycle. What do you think the end goal is or like where do you think things are trending towards? You know, it's so hard to make predictions with this technology. Like, you know, again, it's changing and moving so fast. We very much believe, though, at the end of the day, impressive as the technology is, it's really about augmenting what humans can get done rather than replacing them. And particularly for go-to-market, it's such a human connection oriented business, if you will, right? Sales and marketing and customer service. And at the end of the day, I think it ends up being we're creating more time for the humans in marketing and sales

or customer support to be able to spend time with their customers, deliver more differentiated experiences like higher value add experiences and for both for their customers and for our customers themselves to go to market teams, taking away more and more of the drudgery of the work that they used to have to do. And this is what we've seen again and again with technology as it It comes along and evolves ultimately. Like what happens is it automates away some aspects of some roles along the way, but it now frees up time and creates these new higher value roles and opportunities that create better experiences. And that's certainly the road that I think we're on. Okay, I mean, that sounds like a nice outcome. If you don't have any drudge work as an else person, as a marketing person, you get to focus on actually helping the customer. Okay. So just finally, I always want more people to learn from. So who's working most excited about at the moment? Well, you know, I still love nerding out and all this stuff. So I do spend a bunch of time, you know, reading academic papers on what's going on with frontier research. And, you know, each of the labs puts out good papers themselves in addition to the academic work that's out there.

But I'll tell you right now, we talked a bunch about evals and quality evals as a key to a lot of this. And I think writing evals, which is a brand new skill, I think is one of the single most valuable things product teams in particular can be doing right now and practitioners generally. and you know Lenny Wachitski in his newsletter and in his podcast has been talking a lot about evals recently and just had some just gold dust on specifics and experience from the field of how to write really high quality evals so I'm really enjoying reading Lenny's newsletter and that's definitely one I'd recommend to people oh yeah Lenny's newsletter is absolutely amazing stuff and yeah evals is an incredibly important topic so that's good stuff alright thank you so much for your time thanks Riffy I really enjoyed it