カスタマーサポートは AI で最も厄介な問題だ

From radars that detect threats to technologies that safeguard critical missions, your engineering expertise can make a difference at Raytheon. Bring your skills to a work that matters. Visit careers.rtx.com slash Raytheon. Hello and welcome to the Stack Overflow podcast, a place to talk all things software and technology. I'm your host, Ryan Donovan, and today we're talking what AI engineering looks like at a company that has a long-standing machine learning program. My guest for that is Ahmed Aldali, who is the CTO at Airbnb. So welcome to the show, Ahmed. Yeah, thanks for having me. So before we get into the topic today, give us a little flyover of how you got into software and technology. Well, I grew up in Winnipeg. It's a very, very cold place, and by virtue of not being able to go outside, I spent a lot of the time inside on my computer. I fell in love with gaming. I was an avid video gamer.

Starcraft was my choice of games. And I wanted to be a game developer. That's what got me into computer science and machine learning and AI and was really out of a passion of video gaming. And then I went to University of Waterloo for computer engineering and moved down to the Bay Area early 2000s. I worked at a company that was at the time maybe had like one product success that staved it from bankruptcy. And it turned out to be a really smart, really fortuitous decision and ended up working at Apple for 16 years and then on all sorts of projects and technologies. And then I went to Meta for six years and then I ended up at Airbnb. Airbnb's had a long, fairly well-known machine learning program. I remember I was running an engineering blog about eight, nine years ago at a previous company and looked at Airbnb for inspiration. Obviously, now we're in the agentic AI era, which is a slightly different thing. But I wonder what your program there looks like, having built on such a foundation of machine learning.

Yeah, I mean, we're really lucky. We have a lot of very AI native problems to work on. This is very multimodal in nature. You know, when people are searching for inventory, it tends to have images, it has text, it has all sorts of modalities that you sort of need to interact with. So we've had a pretty strong machine learning program that was mostly focused on search ranking and recommendations. We've evolved that program, obviously, to include a lot of the latest advances in generative AI, which also meant covering more parts of the marketplace. So now when I look at our AI program, we have agents that are fully deployed in offline and online fashion. So in offline, for example, we have a lot of agents that are helping us optimize the marketplace. and there we built our program out to tackle the hardest problem first, the problem with the most gnarly eval use case that really needs to work well. To build the muscle, the institutional muscle, the engineering rigor that's required to sort of deploy these systems, we picked customer support, which is actually the hardest example

that you can pick because you have to be correct. So we've been sort of building out the core strength that was really focused on search and recommendations, But now we're kind of covering almost the entire engineering organization beyond just what we did for the guest experience, but now what we do for hosts, but also what we do for the marketplace health and also now for customer support. What about customer support makes it the gnarliest problem besides people being salty? It's actually a two-sided marketplace. So you have to sort of facilitate resolution between two entities. So you're like dealing with like a multi-party resolution. And a lot of customer support normally is like single-ended. It's you with a complaint about the company, not about another host. And so in that context, you know, many people train models to complete tasks that are kind of single player. But we've had to sort of solve the problem in a multiplayer way. We have multiple parties. And sometimes the facilitation of the conflict is about complaints between parties, not necessarily between us and the party. So we're kind of facilitating. So that makes it really hard. And then also, you know, these are high ADR cases.

So average daily rate cases. So you're paying a lot. these ticket items are large. When you're dealing with a resolution, the consequences and the stakes are also quite high, which makes the problem particularly challenging. The resolving between two parties is more than just a sort of text generation problem, right? More than just information retrieval. Do you do any sort of predictive modeling as in like to sort of predict what a particular host will accept or a particular guest will accept? We attempt to facilitate a resolution according to our policy. You know, we're early here and I think the industry is early. I don't think this is a research topic, actually. It's like research where you have like two conflicting objectives and you need to convert it to a result. So in that context, we are just starting to scratch the surface of that complexity. But again, we're working to adhere to our company policies and helping facilitate a resolution that brings everybody to like a good state. Yeah, sort of a negotiator bot, right? Yes. Even the information retrieval scenarios mean there are a lot of regulations,

very complicated business to operate in. And so we have to be very accurate and very correct in our responses versus like, you know, informational like chatbots and other industries can answer something, but they don't have to be 100% correct. And we have like multiple languages. It's a global deployment. So even the complexity of informational queries, which could appear simple in deployment, like you just have to do RAG, is actually quite complex to do it well. And one of the things that I love about Airbnb and I think is very unique relative to what I've seen in the Valley is this obsession for quality and for finish and for the user experience. So we don't just make it hard because we want to be accurate and we want to be correct. We also make it hard because we want the product to feel good and to have the right polish. We have very, very strong requirements around latency and ease of use, which puts a lot of stress on how we engineer the system. We can't just cheat and use the largest model that happens to decode at extremely slow speeds. We want something really fast, responsive, and gives the user a feeling that they're not waiting for an answer.

There you're making an interesting trade-off because you want the latency, but you also want a model that isn't going to make errors, right? You want something that is able to hold the conversation at speed, at budget. How do you weigh those trade-offs and what sort of results does that trade-off model have? So this is where machine learning, for better or worse, starts with evals, evals, and evals. And the reason why I was excited about taking on customer support use cases like the first primary one, we also leveraged it in search. We leveraged it in all the places we've historically leveraged it. But for generative AI, for conversational agents, we leveraged. Our first launch was really focused on customer support. easy to build evals for. We have production traffic. We know what people are calling for. We're able to build really specific, really carefully constructed evaluations to understand our performance in that domain. And once you have a domain and you have that kind of internal flywheel in that internal eval system, then you're able to sort of look at the Pareto of model performance

intelligence versus cost versus latency and find the right teardyme for yourself. Now for us, And because we have a very strong machine learning history and lineage and capability we also able to post our own models So in that context you can not just live with the existing Pareto you can create your own Pareto So for us, we were able to leverage a model that we think hits the right decode speed, but then specialize it for our use case because we have these strong evals that give us a sense of the production quality. We're able to now go work backwards to build a model that's highly specialized. That is why we have, I think, the best customer support agent in the industry, supported by third-party evals, to be clear. That's just my own assessment. But I think we have the strongest one. And that's because we're able to leverage our own infrastructure to sort of hill climb and improve for our use case. You mentioned that evals are the heart of an EML program. How much of those eval programs did you have before generative AI?

The philosophy existed, right? Like the ability to train models and have holdouts sets and evaluate your system without giving access to the direct question and answers that you were evaluating your system against. That rigor and culture and philosophy existed. The tools did it, right? Obviously, it's a different technology, right? Different technology, yeah, that's right. So we've had to build a lot of that ourselves. And we've leveraged also the beautiful thing, and I have a long lineage and strong, strong belief in open source, is that we can leverage what the ecosystem is also built. And there is a pretty rich ecosystem that's already out there for us to leverage in terms of being able to adopt our own infrastructure for our needs. So putting together a good eval, you talked about having the customer service, you have a good data set. I heard of people talking about having a sort of golden data set as almost like a benchmarking. How do you generalize that? I think that's an old school perspective. In the old days, you had to label data. And so in that context, I mean, we still label data, to be clear, but there are different techniques. are more scalable. But I think nowadays, my perspective on evals is the moving target.

You do not have a fixed data set that you measure against in every release. And your goal, if you're really good, is to saturate and then collect harder. And so you're always hill climbing. The process of hill climbing is have a benchmark, you train a model, you perform poorly on the benchmark. If you did a good job, you should perform poorly. And of course, you should segment your evals to have an increasing taxonomy. of complexity, right? So if all you do is like have like one number to flatten all of your use cases, like not very informative of what you need to help climb on, you need a taxonomy and then you need to like have appropriately difficult questions. And this is true of the frontier, but it's also true in very application specific domains. The frontier now is like always in search of difficult evals. In fact, some startup strategies are entirely based on evals. They don't train any models. They just put out evals and then they try to get the frontier labs to post train on their evals, So that is a strategy that exists in the industry. We also have that philosophy where if I see a team tell me we got 95% on this eval and we got 92% with the old model, I tell them to redo their evals because that's saturated.

I think the rate of saturation now with AI is much faster than in history. In the old days, you would fight for percentages, like 1%, 2%, 3% with a big gain, and you're kind of very painfully hill climbing. Now we're starting to see, like, I can saturate an eval or performance in, like, months versus what used to be where it was, like, a year and we pushed 5%. So when you tell them to redo their evals, what does that mean? Do they scrap it, start over? Do they have a base to start hill climbing on? They take a step back and then they start to figure out where do we perform well and where don't we perform well? And how do we collect more use cases where we don't perform well? How do we enhance our taxonomy and climb to the harder use cases? So, for example, if a team said, hey, we're getting 95% of the tickets are resolved with AI. Well, the truth is, actually, if you look at what we've done with our agents, there are use cases where we defer to the human. So we don't cover 100% intentionally. There are certain use cases we're uncomfortable dealing with.

That should be part of our eval. That should be part of our taxonomy. So a team that's achieving 90% isn't really honest about where we are doing well and where we're not doing well. And so I would much rather see 50% and we understand exactly where we focus rather than 90% and we're celebrating. And we're like not sure. Like, how do we get to 100? You'll never get to 100, right? It's a statistic. So I don't think it's possible. I think if you get to 100, something's up. So a lot of evals are sort of looking for data that hasn't been trained into the model to figure out how to get an eval that is novel responses, right? Do you look for that? Or like you said, would you be happier if it was just like there was a lower level where you kicked it to a human? Well, I think in our case, because we are, what I love about this place is quality, quality, quality. So I think we're always going to focus on making sure that the end user has the best experience, right? We're not deploying these systems for the sake of like maximalizing AI. We're trying to deploy these systems to improve the user experience. So there will always be cases where we feel a human touch is better.

And it's actually communicating care for the consumer. So I don't think the goal is just to displace all calls with an AI unless we believe that that is fundamentally a better experience for the caller. So that aside, when it comes to how do you mine or discover where your system performs well and doesn't perform well, there are many tactics to do that. Including, by the way, deploying human labelers and users, adversarial testing, red teaming of your model. with synthetically generated agents that are simulating use cases across your system. And you can do it via really targeted data sampling from production. So there's multiple tactics that you can use to sort of remine and rediscover the sort of jaggedness of your system. Every intelligence deployed into production is jagged in interesting ways. The Astra, I can play with Astra and find where it's jagged through those techniques. And those techniques apply whether you're a frontier lab, building your model or whether you're like applying the AI to a system, to a production,

to a use case. It's very similar philosophies and mentalities. Yeah, I want to get introspective on Airbnb. I know this is the sort of external use case for AI, but a lot of companies are bringing AI inside, getting agentic SDLCs. And I'm curious, I was at a conference last week and somebody said agentic AI coding tools are really accelerating individual productivity, but they're really doing a number on team coherence. What have you found in terms of that, the individual and the team productivities? There's a couple of things I don't think get talked about that I think are interesting. Number one, I think actually a lot of philosophies that were pre-AI on what made great collaboration true exists post-AI. And I think we treat it like it's a novel thing. I think what ended up happening is an exponent in capability created an exponent in misbehavior almost. And so I think that is actually an under-talked-about thing where it was never acceptable.

I mean, I've been in meetings. I'm sure other people have, too, where you're like, in the meeting, you see a document. You're like, okay, it's 42 pages. And clearly, it was written by 42 people. And nobody at any point thought like we going to compress this thing because really it just saying three things We don know what the purpose of the meeting is and we don know why we there That happened before AI Now with AI a single person can produce the 42 document They don know what in the document and they're sharing it with you. And then you're getting your CLOT agent or your GPT agent or your open source, whatever, to analyze their documents. So we're just all wrappers of AI. But that behavior and that social etiquette of working together, it was never acceptable before. It's not really acceptable today, but it's just happening a lot more because the tools are facilitating for that behavior. And so the real question is, can you add a countermeasure where there's an inherent expectation in your culture that the thing that you produce, you're able to defend? I think that's the fundamental thing that sometimes we lack. A lot of people accept it. They're like, well, the agent coded it

or the agent generated this. I think someone called it a slot bomb. But it wasn't really me. It was the coding agent. Well, it's actually you. You should be responsible for it and you should know what it built. Who's using the coding agent? Yeah, yeah. And so I think that's very true before and after. How you codify that and whether that's like a top-down edict is still kind of very tricky because on the one hand, you want to encourage adoption of these tools because they do create a massive improvement in your productivity. On the other hand, you need to create these social norms where people aren't sharing slop docs or slop PRs where they're not paying attention to what's actually being built. They're not putting a lot of care and judgment. To me, the difference between using a coding agent well and poorly is whether you apply judgment to that process or not. And when individuals don't apply judgment, that is when you get really bad outcomes, in my opinion. Yeah, I think with AI in general, it is very easy to just produce something. It's harder to understand it. Yeah, exactly.

I think you can, though. You can expend the energy to work with agents to apply your judgment and your understanding of the artifact it's producing. And you can produce high-quality output with coding agents that did all the coding. You can have it do all the coding and still produce something reasonable. And there's a way to do it where you produce something very unreasonable. And it comes down to the effort you applied to generate that artifact and the judgment that you're applying in either your loops or whatever your preferred approach for deploying agents apply. By the way, the other dynamic that's happening that's under-talked about is sometimes I hear a sentiment that actually early-stage engineers are fundamentally more AI-native and more capable of deploying these tools at scale. And I think it's actually counterintuitive, but it's a complicated situation. The senior engineers have judgment. They've shipped things. They've written artisanal code. They've handwritten code. They've deployed it into production. They've been burned. They broke prod several times, right? It's almost like they've been through apprenticeship. They were a blacksmith at some point.

They were in a blacksmith's practice. They learned how to make a sword. And then suddenly there's machines making swords now. And so if you work at the factory, do you really know how to make a sword? Or are you just kind of watching the machine stamp out swords? So there is this like weird dynamic in the industry we're still figuring out. And I haven't figured out the perfect answer to this. But I don't really hear it talked about. And a senior engineer can now deploy agents and produce like really effective code with low defect rates. and a junior engineer can leverage the same techniques but generate higher defect code because of the lack of judgment or experience that they haven't had the time. And so what that does to software engineering over the next couple of years is kind of an unknown. You know, with a lot of folks I talk to are building software factories, right? To go with the factory analogy, they're building plowshares and not swords. But the judgment has to be engineered into the system pretty sort of explicitly, right? And I think codifying that sort of type of knowledge has always been hard in engineering. We've been experimenting with a few things that are interesting, but I don't think are like cures.

You know, we have this like concept of context graph that encodes institutional knowledge. It's like a shared memory bank, like a company brain, if you will, which we're hoping to like be able to encode our norms and our practices and take a generalist who may not be familiar with the specialist part of the code, the code base, and allow them to be able to interact with their coding agents to land things that they normally would have to coordinate with other people. So there are techniques we're trying to basically build memory around the agent to improve practices around how you use agents and produce better outcomes when you are using agents. But I think this is kind of like a burgeoning field. I think a lot of people are still experimenting, and so are we. Another issue I've seen in a lot of larger engineering organizations, You lock down the tools, the languages, the things that any developer can use, right? With AI, it seems like I've talked to folks who their organization just has thousands of skills, many that do the same thing, little scripts that all do the same thing,

and it's sort of impossible to track and standardize. Is that an issue or is that a good problem to have that everybody's building stuff? I think I could see it both as a good thing and a bad thing. So in the context of it being a good thing, I think the fact that you have people experimenting at scale that way means that you are on the right boundary where you're leveraging AI very aggressively. You're able to generate, like people are generating thousands of skills, which means people are using AI enough to even understand what a skill is. Which, by the way, a lot of companies still haven't really unlocked that level of participation and focus. Like people talk about, okay, we have 99% monthly active users on AI. That's not really an interesting metric. What you really want to know is intensity of use and impact, downstream impact. So from that context, I think it's a good thing if you have this problem. That means that your organization is appropriately leaned in to leveraging AI to automate their own workflows. Where it is bad is probably token efficiency. If at the end of the day, everybody's doing the same dashboard, and you're using Astra to produce your dashboard, and everybody's using it.

And so it's like, what are you really getting to at the end? Are you actually producing a better outcome for the business? Is that right? So I think the next paradigm where this becomes like a major focus is really token efficiency and our people leveraging the common workflows so that when you improve one, you improve many people's workflows versus there's so many tools and so disparate and they're so disjoint that like any one improvement doesn't really get you anywhere. You need to improve all of these little skills everywhere. And I think those things, personally, I think those are temporary problems over time. Organizations learn to bet on the skill that provides the highest value, and they're sort of like pods that begin to aggregate. We had that problem at some point, and we still do to a small degree, but we do have skills that are getting a lot of usage. So, for example, somebody built a skill around a data scientist, an agent specialization for data science. First, they launched it, and then they experimented with it, and then now we have like 300, 400 people on it, and it's growing every week. And that becomes like the standard skill that everyone wants to improve. And so the more you're able,

I think you do want a bottom-up environment where people feel like they can create an experiment with their own workflows, they can customize what works for them. But ultimately, like a winner emerges, that's the one you want to really bet on. There's a discovery and cataloging problem. I mean, when I worked at a large engineering organization, there was no catalog of what microservices there were, for example. I don't think large engineering organizations are necessarily very good about that, unless there's concerted effort. That's right. Certainly you can discover the skills we have but we have enough skills that you don know which one you should use The discovery is not necessarily the issue of figuring out which one Selection is the problem for us I think Thinking about what is effective with agentic engineering how were you thinking about measuring performance and productivity and quality of results? Yeah. At no point did I ever think that token use was the correct rubric for productivity. It's an aversion. I've had that version for the longest time. So at no point did I ever tell the team, like, token use is, like,

what I focus on. It's the new lines of code, right? Yeah, exactly. It's exactly right. But I do think there is a tipping threshold. It's like a hysteresis, you know, at some point, you got to trip the wire before it doesn't matter. So I do think it is alarming if you have, like, zero token use. Like, that is alarming. So in that context, like, that's the only time I would ever care about, So using that as a rubric would be zero use. Zero use communicates something. Anything above zero is like, I don't care. What I do care about, and I think what we care about, what we focus on is outcomes and speed of execution. And that's where the age-old problem of how do you set goals comes in. It is not an AI-native problem, but the atom of reasoning when you are an AI-native company needs to be like velocity and clear goal setting. If you build clarity, you build pods that are independent, that have agency. And what I mean by clarity is like the objective and the result is clear. And the pod has the autonomy. And you have a lot of people coordinate together, things slow down.

The human tax becomes larger than the building tax. So you have to keep the human tax really low. And you have to focus on, do they have clarity on what they're expected to achieve? And then you compress and you hope that in pre-AI, this would take this long. and then it's going to take us this long now. That is why our rubric, for example, was we shipped 80% more features this year than we did last year with similar complexity, which tells you that that is actually the metric we care most about is are we improving our product at a rate that we can be proud of, that we feel is kind of commiserate with the speed that AI gives you. Still improving, we still have a lot more work to do, but that's kind of the mentality you want to bring is are we moving at the pace that we should be and is our human tax low enough that the cost is mostly associated to building and iterating? I think there's a concern with the increased speed of delivering code to production that there's a reduction in quality too. So I wonder if you notice any sort of change in incidents, production outages?

I love that question. Not enough people are asking this. This is like my biggest fear. I go to sleep at night wondering how many slot grenades have people put in today? So it's definitely something I worry about and I keep my eye on. I am very on top of goals for reliability. We're very aggressive on making sure that that stays where it is pre-AI and we don't suffer as a result. So we have very strict guidelines on how it is. If we trip too many of them too quickly, we actually do a concept for code yellow and then eventually code reds so that we maintain focus on testing, on testability, on observability, observability on remediation and making sure we understand every outage. We have very strong culture around SEVs. When we have an outage or a defect, we have a pretty rigorous process that I attend to make sure we prioritize remediation. I'm very proud of this. Our agentic coding usage has been very geometric, but our defect ratios or defects in production

has been linear. So it has increased. We have not been able to hold it. but has increased linearly when our usage for AI has grown exponentially. And I know for a fact other companies have had geometric growths in their defect rate, so I think our hygiene is reasonable today. But it's something you have to stay very vigilant on. To go back to the productivity question, are things like Dora metrics still applicable, or do we need new ways to measure the velocity of a development team? The process of building product and features and deploying things into production is an inherently multidisciplinary problem. So in addition to rethinking how software engineering velocity works, you kind of have to think about the whole software development lifecycle. And that's where I don't know the answer of how we measure velocity for an entire product feature iteration. Because I think every product feature has a different blast radius that are associated to it. And so some features, for example, require legal sign-off

and legals involvement. Some features require work with marketing and require an onstage presence and touch every part of the app. Some of them is regulatory and very explicit. So I think you kind of need to figure out the atom of the pod. If the pod requires eight multidisciplinary teams to come together, that is a very difficult... The human tax goes up, so the velocity expectations need to modulate to the complexity of the feature. But if you're shipping something on a homepage that's a different button, that should be incredibly fast. A software engineer, a designer, and a PM, that's it. Actually, you don't even know if you need all three, for example. Are there many parts of a piece of business-critical software that are just committed and quitted, doesn't need anybody but an engineer on it? Our infrastructure stack is a perfect example where we don't have designers and PMs and legal and a bunch of other teams overseeing. So for those, we can measure the velocity of migrations. A migration, for example, is amazing because certain migrations might have taken us 6-12 months before.

And what's amazing about our infra team here is I challenge them to think about things that they thought previously were impossible that suddenly become not so impossible. And let's go do that. So large code migrations are suddenly possible. That is a project, for example, where it is pure software development. It doesn't require a lot of cross-functional development. And that's where I think you can measure very directly the speed of development. All right, everyone. It's that time of the show where we shout out somebody who came on to Stack Overflow, dropped a little knowledge, shared some curiosity, and earned themselves a badge. Today, we're shouting out a celebrity, somebody who dropped a question that was so good, It got 10,000 views and earned the famous question badge. So congrats to TechGuy for answering, is WebView2 supported in the Citrix server? If you're curious about that, we'll have the answer for you in the show notes. I'm Ryan Donovan. I host the podcast, edit the blog here at Stack Overflow.

If you have comments, topics to cover, questions, concerns, you want to say hi, you can email me at podcast at stackoverflow.com. And if you want to reach out to me directly, you can find me on LinkedIn. I'm Ahmed Aldale. I'm the CTO of Airbnb. You can find me on Twitter or LinkedIn. It's Ahmed Aldale. Yeah, thanks for having me. Thank you for listening, everyone, and we'll talk to you next time. Bye.