AI採用の「破滅ループ」から抜け出す方法：Greenhouse CEO Daniel Chait と Voice AI責任者 Ophir Samson に聞く

Generative AI is transforming industries at an unprecedented pace. But as AI changes how you or your team work, one thing is clear. Your skills also need to evolve. At DataCamp, we offer everything you or your team need to adapt and thrive with AI. Whether it's business users looking to get the most out of ChatGPT and Copilot, or developers and data scientists looking to fine-tune models, you can learn the entire AI skills spectrum on DataCamp. Power your AI transformation today. Start learning at datacamp.com. optimistic that we can turn the tide and make things much much better using ai and using these

technologies to bring people together and help people find jobs somehow it's both really difficult to get hired right now and also really difficult to hire good people when everyone uses the same ai tools to polish their resume and or to apply to jobs it's difficult for good candidates to stand out this is the ai hiring doom loop and today we're going to try and escape it will more ai help us out from the mess created by AI. It is an unusual time in the sense that historically you had either like the job market would be strong or weak and one side or the other would sort of feel like they have the upper hand. But this is really the first time that I can remember when both sides were dissatisfied and didn't see it getting better. Yes, it's a bit more effort. But yes, you have to put in 10, 15, 20 minutes of your time and think through some answers. But you know who's not doing that? The 1,500 other bot applications who are still competing for the attention of the recruiter. So if you're a good candidate, you're kind of being drowned out by just so much noise.

I have two guests for you from hiring platform company Greenhouse. Daniel Jake is the CEO and co-founder. He spent more than two decades as a tech executive, managing hiring teams, and as an HR entrepreneur. Ophir Sampson was CEO at Ezra AI Labs, now acquired by Greenhouse, where he runs the Voice AI team. Before that, he founded Voice AI coaching company, Julia AI Labs, and he's run engineering teams at General Motors, Uber, and Aurora. Let's find out how to save recruitment. Hi Daniel, hi Ophir, welcome to the show. Hey, good to see you. Hey. Yeah, great to have you here. so we've got the point where you've got bots to run interviews you've got bots to help uh people who are interviewing for a job are we heading towards a world where it's just bots interviewing bots no humans are needed yeah look i mean i think um there's real there's real concern among job seekers about the state of the market today it doesn't feel good to get a job right now and

i have a ton of empathy for anyone trying to get a job because to your point none of us ask for all this AI stuff to happen and it happens so quickly that almost everything about hiring is wildly different than it was just a few years ago and it's really difficult to navigate and so you've got this you know never-ending stream of worries about how it's all going to play out and AI is kind of at the center of a lot of that so I get why there's a lot of worry. I think at the same time though one of the interesting things is there's a whole bunch of power and possibility that AI is also opening up. And so I'm personally extremely optimistic that we can turn the tide and we can make things much, much better using AI and using these technologies to bring people together and help people find jobs. Okay, yeah, so it's this double-edged sword where there's some new challenges but also some new opportunities. Maybe we'll take the challenges first. And I think this happens on both sides. It's like the job seeker side and the recruiter side. But maybe let's do job seekers first. So, Elfair, do you want to talk us through, like, what are the challenges for job seekers right now?

Yeah, I mean, I echo what Daniel said. It's a really difficult time to be on the job market right now. If you are applying for a job that you've seen a job post for and you send in your application, you're likely competing with 1,000, 2,000 other applications, 60-70% of which are bot applications or fake applications 100% of which are AI generated resumes and so if you're a good candidate who's applying for a role that you're genuinely interested in getting you're kind of being drowned out by just so much noise and so your likelihood of being noticed by a recruiter is minuscule it's tiny And so that's kind of one half of the problem that we're trying to solve here. How do we give good candidates who are genuinely interested in a role that they're applying for the opportunity to be heard, to be understood, to stand out and to dramatically improve their likelihood of making it into that job than what the current state allows them to have?

And we'll get more into the sort of how we do that and, you know, how we're using voice later on. But that is, you know, the problem that we're trying to solve. Absolutely. I'm certainly standing out in a field of like a thousand other resumes, especially when you've got bots that are tailoring things exactly to the job description because they don't need to be a real human or tied to a real experience. Then, yeah, that's certainly a challenge. Having a thousand applications, that sounds like something is good for recruiters, but I can imagine it's also a challenge. Daniel, do you want to talk me through where the problems come in? Yeah, so it's interesting because you're right. I mean, typically you would think, oh, the companies are out there looking for talent. The more talent that they have access to, the better. But it turns out, no. It turns out that kind of the mirror image of the problems that job seekers are facing are also being experienced by companies that are looking at a hire. So what we're seeing across our customer base is typically they'll open up a job and they'll be inundated with hundreds or thousands of applications that they have very they have a difficulty telling the difference between.

And so kind of like Ophir was saying, a lot of them are kind of AI generated auto applies and sort of everyone's resume and is kind of being polished into sameness by the same AI technology that we're all using in other areas of work. and so those unique candidates that are best for the job just aren't standing out really in any way from all the rest of the noise and so when you have a thousand applicants to pile through where do you start like how do you even tell the difference and how do you even know like of the people who applied surely there are some who very much want this job who've thought hard about it who targeted your company as like a great place that they want to be and like this is the job they're most interested and then there's a bunch of other applications in that same pile that look very similar that have never heard of your company that didn't even really apply intentionally that maybe a bot applied for them and so the companies really are stuck like not knowing where to start or how to even get through that whole that whole pile of of resumes either and so it is an it is

an unusual time in the sense that historically you know you had either like the job market would be strong or weak and one side of the or the other would sort of feel like they have the upper hand but this is really the first time that i can remember when both sides were were just dissatisfied and didn't see it getting better yeah it's kind of a situation where both sides of us are struggling so i mean a lot of people don't know where to get started okay so let's see if we can figure this out uh where do you start uh if you're applying for a job that you know that you want and you're um and you have the opportunity to be interviewed by our voice AI system, which we call Greenhouse Voice AI, we recommend starting them. The reason for that is your likelihood of success is dramatically higher if you take the effort to do that than if not. So we're seeing that candidates who do a voice AI interview

are three times as likely to reach a final round than candidates who haven't done the voice AI interview. So I think a lot of candidates are justifiably skeptical of AI in the process, whether it is AI interviewers or other implementations of AI, just because there's a lot of AI slop out there. There's a lot of really bad tools. but when it's implemented correctly and it's deployed by a recruiter or a company that is really thoughtful about how to give transparency to candidates and you are on the receiving end of that then we really encourage you to do it because you will be noticed in the way that you would be noticed otherwise yes it's a bit more effort Like, yes, you have to put in 10, 15, 20 minutes of your time and think through some answers. But you know who's not doing that? The 1,500 are the bot applications who are still competing for the attention of the recruiter.

So it's a way of getting to know you as a candidate in a far, far greater level of detail than the recruiter will be able to understand just based on your resume. And so that would be the starting point that we'd recommend. And maybe if I could just pick up on something that Ophir said, because, you know, I think part of what's so frustrating about being a job seeker right now is there haven't been great things for you to do to give yourself a better chance. Like, why are they sending out so many applications? Why are they signing up for these $19 a month auto apply tools? That's what that's what they can do. And they just want to get a job. and like there hasn't traditionally been until now an effective way for them to kind of put more effort in and give themselves a better chance at getting a good job everybody understands i think most job seekers are okay with the fact that good jobs are going to be competitive to get you're not going to be the only applicant i think the challenge is when it's not when it goes beyond being competitive and you're just lost in a sea kind of a resume black hole what what do you even

do to get out of it like how much how do you if you want to try harder like how do you try harder and so what a fear is talking about is you know the voice ai is a great opportunity for this person who's applying for a job who really wants it to put in some effort yes but in ways that help get past the resume and and really show who they are what they're good at and i think we've all had the experience when when writing a resume of like gosh i only have this limited space to put my whole life into it has to be in these bullet points like it's not really saying what i want to be able to say and so i think the opportunity to to speak and to tell your story and to answer some questions and to by the way ask your own questions it really does give people a new and better way to show what they're all about being able to tell your own story and explain why you care it seems like a useful thing um i i'm curious a bit about the the logistics though so whereabouts does this voice uh the ai voice interview fit into the hiring process is this like at the point where

you submit your application? Is it like instead of the screening interview? I presume you don't have like a final round interview with AI. Yeah, Alfred, do you want to tell us where it fits into the flow? Different companies implement in different ways. So the majority of the use cases are that the company will receive a bunch of inbound applicants. So they'll get like 2000 applications. And then the recruiter will choose a subset of those to invite to a voice interview. Now, the word interview is actually an industry term. It's a bit of a misnomer, though, because it's not instead of an interview. So when I originally built a company that Greenhouse acquired, my goal was to not replace a human interview, but to supplement the resume and to supplement the written application And so it kind of comes in a point of the pipeline where the recruiter wants to know more about you beyond what you just kind of written

and is kind of choosing who are the 30 40 people that they want to kind of spend their human time with so it really is a way of like supplementing the resume and is normally kind of very early in the pipeline there's some recruiters and companies who use it a bit later on when they want to understand more about candidates in between different stages of interviews but it really is normally at the kind of front end of the application okay yeah that that makes sense having it earlier on rather than later okay so i i feel like having uh being interviewed by a it's going to be polarizing so you can be some people are very enthusiastic others are going to be very concerned about this maybe feeling like it's just a sort of a cost-saving measure on the HR department. Do you want to talk me through which of these concerns are legitimate? Yeah, look, you know, people have the concerns that they have, and I'm not here to say that they're not legitimate. I totally understand. I'm extremely empathetic to these worries, as I started this conversation by saying. So, yeah, you know, we've interviewed, we've surveyed

many thousands of job seekers and asked their opinions and collected lots of data about what are your attitudes and what are your fears and what are your worries about about these technologies and generally they come under three kind of big buckets the first thing people worry about is you know the experience is going to be really clunky and awkward like we've all had that experience where you're shouting representative you know at the phone company you know trying to get through to someone and it just feels like these technologies historically have not felt great. I think the second is that people wonder what's actually going to happen out of this conversation. Is this going to be a machine that's going to make a decision arbitrarily based on something I happen to say during the conversation or is it going to be fair? And what's really like the decision making sort of framework behind all of that? And then the third kind of concern that people have is sort of like you said, like, who's this really built for? Is this built for me as a job seeker or is this built in some kind of way to like save money or dispatch with me or kind of keep me away?

And so what I can tell you is when we went to look for the right solution to take advantage of this amazing new technologies and innovations in AI and in voice technology, those three things were very much in our mind. And when we connected with Ezra AI Labs and started to hear what Ophira had built, it was really apparent that they had built a technology that was first and foremost meant to give a very comfortable and naturalistic experience to the job seeker. that it wasn't going to be that kind of stilted thing where you're interrupting each other constantly or some like creepy like robot cartoon talking to you or something like that that it was going to be built on a solid defensible foundation of structured hiring with clear decision-making rubrics and evidence that's being presented to humans to make decisions and that it's going to be really candidate-centric that it's going to help the candidates ask their own questions interview the company and about what they want to know about and really transparent about, hey, you can opt out

of this if you want. Here's what's going to happen next. And what we found is that when you take care of those main concerns of candidates, it really does overcome most of their worries. And the feedback that we've collected is really excellent. The customers or the job seekers of our customers have rated the experience very, very highly. And so I get the worries. We have the very front and center but again i feel like the choice is do you want to have your resume go into a black hole never to be seen again or do you want to have the opportunity to present who you are and ask your own questions i think when people are are confronted with that choice they're very enthusiastic about it okay yeah i mean certainly uh i think everyone's had a terrible experience with those like uh phone systems where it's like press seven to speak to this department maybe yeah you certainly don't want that in a job interview on the other concern about the decision making process what what's the output from this ai interview uh is it uh just there's a transcript

and a summary to be given to a human to make a decision or does the ai make a hiring decision by itself yeah so hiring um so a human always makes a decision so we're very clear that there is no AI will make a decision about the fate of a candidate that is going to call to our principles. Another part which is call to our principles is to eliminate bias and to do this in a highly structured way. So we have independent third-party bias auditors that evaluate us every single month and test our systems across tens of thousands of interviews to make sure that we are conducting the evaluations in a bias-free way across every single protected class. We also, you know, we see this very much as like wave two of AI interviewing with wave one being the clunky stuff that Daniel mentioned earlier. and a part of kind of wave two is not just like taking the transcript of an interview and giving

it to you know an LLM and asking like is this a good candidate or not and just like getting a score of 68 great candidate that's not what we do we every single candidate has evaluated relative to the specific rubrics of the recruiter set before the interview began. So the recruiter will set three, four, five other questions that they like, specific questions that they want every candidate to be asked, and every candidate will be asked those questions. And along with every question, they also set rubrics for what determines a great, a good, an okay, and a poor answer to each one of those questions. And they also describe to VoiceAI what they're looking for in an ideal candidate. So we have a ton of structured rubrics, information that is specific to that recruiter, to that job, to that company, that culture, that team, through which to evaluate the specific candidate that took an interview for that role. So every evaluation is done relative to that structure.

And that is a way that we guarantee consistency across every single interview for any different role. And the other kind of benefit of that is, or the other implication of that is, if you're setting up an interview for role A, that evaluation, that setup will look completely different than role B, and which will look different from role C, it's a different company. So every use case is a snowflake or kind of built on the foundation of highly structured, highly rigorous, kind of bias-free AI that is not making a decision, but kind of giving the recruiter all the information they need to make a really thoughtful decision. Okay, yes. Certainly consistency is a really big problem with hiring. So if you're hiring for lots of the same role over a billion, I'm thinking something like account executives maybe, where it's kind of a fairly similar role, but you're going to want to hire lots of times. Is that the sweet spot for this kind of tool?

Or are there any particular roles where this voice interview works better or worse than others? Not necessarily. I mean, I think to Ophir's point about the kind of generations and the evolutions, I think there was a time not so long ago when companies kind of thought, well, I'll have AI speak to a blue-collar worker, for like a food service job or something to that effect. But I would never have them talk to an employee that's going to come work in the office as, say, a salesperson. I think the technology has gotten really good now. The experiences are really good to the point where that's not really the thinking anymore. And so it's really more about the company wanting to make sure that they don't overlook the talent that's already in their pipeline. People who, as we talked about, are applying to jobs and kind of getting lost in the shuffle, they want to find those great talents. So our customers are using it to hire salespeople, engineers, people all over the organization. And it's really more about wanting to take control of their hiring experience

and bring some science to it versus it is about necessarily the type of candidate. And if you think about what Ophira was saying a minute ago, just to build on that, Like there are decades of science that have shown the most effective ways to make hiring decisions are also based in fairness. That that idea of having a clear assessment rubric for this role that ties directly to the requirements of the role doesn't have to do with your credentials. I don't really need to care about, you know, did you go to a fancy school or things like that? I want to know, can you do the job? If you build out a rubric, meaning a clear set of identified questions that every candidate is going to be asked in the same order and that the answers are going to be evaluated in the same way across every candidate, you're going to get better decisions. You're going to get fairer decisions. and what I was most impressed with when we looked at what Ophir had built was before we even get to

the point where we're talking to a candidate the technology interviews the hiring team and ask them to build that whole framework out and so these are the things that our best customers struggle to do because it takes a huge amount of time and doing that consistently is something that you have to be really thoughtful in order to do. And that now with AI, automatically, every time you open up a job, the team can just build out a fair scorecard. Here's the questions that everyone's going to ask. Here's the way everyone's going to evaluate it. And the AI can then go dutifully follow that plan with every candidate, giving everyone an equal chance. And I think that's sort of behind the scenes setup is something that might not be as visible to the job seeker, but I think should give people a lot of comfort that when you do approach technology through the lens of fairness and good decision making that we actually can make things much more consistent and fair than when people have been doing it which we've known for forever have created lots of iron problems it seems like when you do a lot of planning up front and decide what you want before you start

hiring you're gonna have a better outcome i think that's the same uh in many fields if you if you plan things first you're gonna get a better outcome uh and certainly i don't think this is related to the idea of structured hiring that you're a big fan of, so we'll come back to this later, but while we're on the subject of fairness, I think there's going to be some issues around friends for voice AI I'm sure, because I spend a lot of time trying to get people to come and speak on this podcast and webinars and things like that, it's a very divisive thing like some people love it and other people struggle, so perhaps if English is not your first language, you're talking a different language, it makes it harder if you've got a stutter or a lisp or some other variant speech, then it's going to be tricky. How does the voice AI deal with, I guess, non-standard speech? Yeah, absolutely. So it's something that we take very, very seriously, and it's kind of core to our engineering from literally day one of writing code for this. So firstly, we support interviews across many languages.

we have the world's most cutting edge transcription models and speech to text models that work extremely well across different dialects and accents and we've tested it in highly you know we've really kind of stretched the limits in our testing of accents and dialects and across you know thousands of interviews um but as good as the technology is there's no technology that's perfect the question then becomes how are you making sure that you're not negatively judging someone based on those characteristics even if you you know protect for 99 percent of what you said and in the technology up front you also want to make sure that you not negatively assessing someone if in that 0 of cases that fall through the cracks And to that end we only judge the content of what someone says. So we do not judge any kind of communication style, any kind of fluencies or disfluencies.

None of that will impact the score that a candidate gets. Again, that was absolutely critical to the value set that I had on day one of building Etra and value set that we have now at Greenhouse. Okay, so it's largely being scored on the transcription rather than the actual voice itself. Yeah, it's scored as a literal comparison between what the recruiter wanted to see, like how they define their rubrics of what makes a great answer, bad answer to each question, and what the candidate actually said. So the content of that, you know, there's lots of things. Obviously, recruiters care about speaking style and communication style and presentation and things like that. Those are the kinds of judgments that the human should make, right? So we can bifurcate the problem between what are the kind of repetitive tasks that can be handled by an AI really well with cutting-edge technology.

So, you know, handling questions that come from candidates about the salary, the benefits, do I need to be in the office every day of the week? So, like, the voice AI system can handle those questions and does across, like, hundreds of interviews for a role. That's not a good use of the recruiter's time. A better use of the recruiter's time is judging things like communication, you know, culture fit, things that are, like, really human and those are things that we don't believe that the AI can or should do and so that's not something that we touch. Okay, all right. So I like the idea that it's focused on just assessing content against the predefined criteria. How does it work when you've got technical terms? I think the classic example is with pharmacists or something where there's a lot of drugs which sound very similar but have completely different effects and if you prescribe the wrong one, you get the name slightly wrong and then someone dies. So I can see how little changes in the words,

if the transcription is not perfect for technical terms, then it can have big impacts on the scoring. How do you deal with that? Agus, over here again. Yeah, so we've done a huge amount of testing across industries that have highly technical terms as well. And, you know, fortunately, we're not prescribing drugs, so there's a different set of standards. but we have in our code in our kind of engineering approach we have multiple ways of transcribing and you know extracting the meaning of what someone is saying so that we minimize the rate of error there is no technology in the world that won't ever you know misunderstand words in the same way that humans won't either and that's one of the really important things for us at voice AI to not overpromise the capabilities of AI. And that is why we want to keep the human in the loop, because a human recruiter for that healthcare company, that pharmaceutical company, he or she would be the one that actually recognizes

the specific terms of their drug, just in case the transcription models don't work as planned. So it's a multi-level approach of excellent engineering. I feel like we have the best or I can show we have the best transcription models out there plus the level of human verification to make sure that the end-to-end product is trustable. Okay, yeah, so I mean, I guess if the human's making the final decision at the end, rather than the AI, then you've got a bit of grace for, okay, even if the transcription is not perfect, then you can sort of correct for that. Okay, nice. So suppose you're interested in this how do you know whether it's working well like what sort of metrics for success are there and like how does using ai rather than the human affect those metrics uh daniel yeah as a company you know thinking about this investment of technology what they're thinking about are a few things first um of course they're thinking about efficiency and they want to make sure that

they can run their hiring processes you know as efficiently as possible and get through all the applications in a timely way, for sure. But more importantly than that is what they really are looking for is access to top talent. The customer wants to know when they open up a role, can they fix this problem that they have right now where there's not a reliable signal in their pipeline about who's good and who isn't? There's not a reliable amount of trust in the pipeline about who's genuinely interested in the job versus the ones that aren't. And even about misrepresentation, who in the pipeline might be misrepresenting themselves versus who are the vast majority of legitimate job seekers out there. And the customers that we are working with, they're thinking about those things. They want some assurance that the candidates that they're going to spend their own time with pass those tests. Okay, yeah. So certainly I can see how saving a bit of money up front is kind of useful because it

helps you scale by more candidates but really like the the big test is are you hiring good people so it's like measuring like i guess first performance review scores or something after after people get hired is that about right yeah i mean quality of hire is notoriously difficult and you know i think we could spend the next week on this podcast you know arguing about different methodologies i think every company and you know every organization to think about that question a little bit differently but however you think about your your quality of your talent um surely it starts with not overlooking most of it, not discarding most of the applicants, but making sure that you give everyone a fair shot. Okay, yeah, so having better top of funnel is going to hopefully lead to better quality further down the line. I'd say also that, you know, other things that recruiters track is of the candidates who've been interviewed by Voice AI or spoken to Voice AI, what's the pass-through rates of those candidates to subsequent rounds? Because if it is, you know, Fundamentally, our core hypothesis here is that it can find the needles in your haystack.

So if it's actually finding you needles in your haystack, then those needles should be progressing to subsequent parts of your pipeline at a higher rate than the rest of the haystack. And they are. I mean, they're progressing at three times higher rates than candidates who haven't been interviewed by the system. Also, they look at a total amount of time spent by recruiter on a hire, and that is going down by about three and a half times. So recruits are spending about three and a half times less per hire than they were before. So there's a huge amount of time savings. And then the last thing I mentioned, in addition to everything that Daniel said, is that candidate experience, and Daniel touched on this earlier, But we're seeing currently just over 90% of candidates are rating the experience as excellent. And only around somewhere between 6% and 10% of real candidates are dropping out. So every candidate is given the opportunity to opt out if they want.

It's never forced on a candidate. But the opt-out rate is also very, very low. So recruiters track everything that Daniel mentioned, everything I said, to be highly thoughtful about the implementation of this. So you mentioned you want to compare performance of people who are using this versus not. Is it typical to do an A-B test then? Do you have 20% of the year people through this to begin with and see how that works? Yeah, many of our customers do that. They take an open job that they have. They get an inbound pipeline, inbound of, say, 1,000, 2,000 applications. They split them into two cohorts, with VoiceAI and without VoiceAI, and they see that in the cohort that uses VoiceAI, they're spending far less time, as I mentioned, about three and a half less time per hire than they were in the cohort that didn't have VoiceAI. Okay, nice.

All right. So while we're talking about methodology, can we go back to this idea of structured hiring? Daniel, do you want to talk me through what does this involve and what the steps are? Yeah, I mean, the basic idea of structured hiring is old, right? It's tried and tested science that basically says the way to run a hiring process is to define up front the criteria that you're looking for. In Greenhouse, we call that a scorecard. Here's a set of attributes that we want to assess candidates on to know who to hire. And then to define for each of those attributes, what are more different kind of interviews or assessment process you might go through in order to rate the candidates on those attributes. So, for example, for this job, we want to know if you can, you know, problem solve in a certain way. I want to know if you have certain technical skills. I want to know if you have certain behaviors that you're going to exhibit. And I'm going to design certain questions to get at that. And then the key is you put every candidate through the same exercise, through the same set of questions and evaluate them on the same scorecard.

If you do that, that leads to much more reliable and much more valid data to help make decisions on. And the reason we do those things is because, you know, it's really hard to make decisions about people in a hiring context. I mean, our brains are hardwired for all of these kinds of biases. And, you know, people hear bias and they have certain ideas in their mind about what that does or doesn't mean. Unfortunately, some of that's been politicized. But it's pretty simple if you think about the fact that our brains are very, very good at rapid pattern matching. Right. These are like things that we've evolved over millions of years to be able to do. We all live in the world and we go out there and we have experiences and those experiences give us certain associations that may or may not lead to the best decisions, but they happen really quickly and almost unconsciously. And so one fun example I always like to give is, you know, if I ask you to picture a genius, you know, most likely most people I talk to in America or Western Europe will picture Albert Einstein.

Yeah, definitely. Right. And so it's like with this very simple thing, it's like, oh, I associated this like white haired guy with a buffy mustache and the concept of genius. Or the association between mustache and genius. Exactly. You should be a huge fan of that. But, you know, or if I ask you to look at an optical illusion, most people will see those optical illusions and their brains will be tricked. But it's not because your eyes are broken. it's because your brain is like finding these patterns almost against your will and and and helping you and sort of leading you to make these decisions that are wrong about what's actually on the paper and so i give those examples to say it's the same when you go meet another person and you immediately have a set of associations about what they're what they're going to be like to work with whether or not you want to or not and so if you don't have that basis of structure underneath it what you're left with is i don't know whatever the boss thinks or hey you went to the same school i went to or some of the things you were asking about richie earlier maybe the person has a speech impediment or maybe there's something about the the way that they're

dressed there's a big debate right now on linkedin if you're following it around iced coffee and is it okay to bring iced coffee to a job interview and the attitudes people have about this are all over the map and the answer is like is it a good thing or a bad thing to bring iced coffee shouldn't matter it's obviously not a criteria for almost any job and yet people have really strong feelings about whether bringing iced coffee is appropriate or not for a job interview and so hopefully that tells us that no if you don't put iced coffee as one of your criteria on your scorecard and you don't test specifically for that you'll make better decisions so that's really the heart of what we mean by structured hiring. And just to add one point, from a technical standpoint within the evaluations in the voice AI we don consider don consider the resume So two candidates one candidate who been to you know who been to an amazing Ivy League university, another candidate who hasn't been to

that kind of educational experience, hasn't had that educational experience, if they're answering the questions well, equally well, they'll get the same score. It doesn't consider where you've been to. So, yeah, we kind of manifest, like, all of the philosophy of um you know the conceptual philosophical frameworks that the the daniel mentioned in literally into the code that we write and i was actually talking to a recruiter in south america who was talking about this exact issue that she was helping really talented technical talent in south america try to get jobs in in the u.s and it was one of the difficulties that you faced was that most of the recruiters and most of the teams hadn't heard of these universities in Argentina. Right. I mean, I don't know if you could tell me what the top technical school right now is in Peru. I certainly can't. And so the point is, if you can answer the questions correctly and show what you're capable

of, it shouldn't matter that I've never heard of the school you went to. And so that's really the heart of this idea of structured hiring that we're trying to bring in. okay yeah and certainly there's so many different cognitive biases we have and so much weird information uh around hiring like the ice and coffee example is one i i heard there's also a thing where like taller people are more likely to get promoted into managerial positions and i said this is someone who used to have a boss who was six foot seven and it's like an okay boss but not amazing i feel like uh there might be some bias there but yeah if you're taking interviews uh on a video call then you can't tell how tall people are so that gets rid of that and that's why you know at the start of the conversation why i said you look i think there's a lot of risks and a lot of concerns about ai that we all need to be cognizant of and we spend a lot of time on that and we bring the receipts we publish our audits monthly as i hope you mentioned but i think also there's an opportunity to do much much better and i think these kinds of things that we're talking about these are mistakes that humans have been making for hundreds of years and so the status not the status quo is great it's not like getting a job has always been perfectly fair um there was

a study in 2003 that showed a tremendous amount of bias just in reading resumes and deciding who to interview at all right that showed that if you sent these experiment these experimenters sent out a few thousand resumes that were identical other than the name at the top and half of the resumes they sent out the name at the top was a stereotypically white sounding name and the other half was a stereotypically black sounding name and they measured who got invited for an interview and they found was a 50 percentage point difference i mean that strikes that that's a real like bias and it's having a huge effect on recruiting it's negative for the person who's trying to get hired and also for the company because it's not hired the right people you got it and so so my point is that these have been these have been massive issues in hiring for for literally generations and as of your mentioned now we have the opportunity to teach a computer program, don't look at the name on the top of the resume. It's not relevant to the job.

Here's the rubric. Here's the things we care about. I think we can skip resume review altogether, define that 50% of bias completely out of existence, make the process work better for everyone, and help people get jobs. So I'm very optimistic that there's a lot of benefit to be gained, even as we need to be very cognizant of the risks and downsides. And we're also not making the claim that AI is not bias. That's a very, very broad claim. And we're not making, we're not hiding the fact that we use models from various frontier labs. We are not making the claim that those LLMs have zero bias into it, because then you're in the realm of how are they trained, does it? That's not the case we're making. The case we're making as the is that the software that you receive as a user of this is a combination of two things. One, the various frontier models that we're using

and different elements that we're using. And two, is a thoughtful implementation of them that mitigates the bias. And it's on two, the second part, that's where we can do a lot of work to make sure that that 50% gap that Daniel mentioned, based on racial differences are eliminated as well as gender-based differences, location, all the protected characteristics. We're very kind of careful about which claims that we're making. We're not trying to speak on behalf of like all of AI because clearly there are questions of kind of bias and training data in those. Absolutely, yeah. So it's not going to completely solve all your problems, but at least it gives you a framework to think about them. Actually, while we're on the subject, there are quite a few regulations now around AI usage, and particularly with the EU-AI Act, HR use cases are considered high risk. So if you want to adopt this voice AI interviews, is that going to expose you to some regulatory problems?

What do you need to deal with here? Yeah, well, certainly, I mean, I think with good reason. These are things that affect people's lives, and we need to be very careful about it. So absolutely what we've seen is our customers are having their legal teams and even their security and privacy teams involved in these decisions early on and often. We have a ton of resources available that we've published about this exact question, helping customers think through these issues of what do they need to pay attention to and how can they safely adopt AI technologies. The fact is that this framework, whether it's the EU Act or other AI laws, while they do impose regulatory requirements, they also give everyone more confidence that things are being done well. And so we've been very encouraging of legal oversight and regulatory review of these technologies and our customers that have gone through that. They're very happy to be able to say that they've bought these technologies, implemented them in accordance with known laws and published standards.

And our own legal team has actually done a huge amount of work to make it easy to understand and clear what the requirements actually are. And, of course, our product teams led by Ophir are very, very good at making sure that what we do is compliant with the law. Okay. I should point out, as opposed to human interviews that are out there today saying who knows what to who knows who outside of control. And, again, so it does come back sometimes to this question of, like, as compared to what. And I think, you know, I built a career on the notion that most hiring has a lot of problems on it that we need to fix. And this is one of them. So I do think that companies who are thinking about these questions of fairness and of the implications of these technologies on job seekers, you know, I do think that they're starting to see the real benefit of bringing technology to the problem that used to be done by humans who have all kinds of issues.

Absolutely. I mean, certainly when human interviews aren't transcribed, then, you know, all sorts of things get said far too often. Actually, I have one more possible objection for this. So the rod of possibilities for faking things now, you can have a tool called Interview Hamo, which is like an AI support thing for people being interviewed. So it like finds out what the interview is asking you and then it gives you like some text to read from. or you can have like a friend in the room you can clone your voice and have someone else say things or something there are all sorts of ways of faking this how do you deal with this? I can take this one we have built a lot of anti-cheat technology into the platform anti-cheat and also anti-fraud technology I'll say one thing is we were mentioning the bot applications earlier, we're seeing with some role that 50% to 70%

of inbound applications are just not real people. So just by virtue of putting an AI interview in the pipeline, you're already going to, in one fell swoop, kill off a lot of those BAS applications. Then within the interview itself, we have anti-fraud, anti-cheat detection. It will give the recruiter a heads up and a flag with a confidence score and a justification of that confidence score of whether we think a candidate is reading from a script, whether that script is a chat GPT or a Claude or one of the cheating tools that you mentioned. So we detect that. Importantly for us is we don't let that affect the score of the candidate. This is kind of in the realm of the types of things that still require human judgment.

Because there's no cheat detection, no fraud detection, the lie detection system, even the FBI's by the polygraphs. Nothing is completely foolproof. So our approach to this is, hey, we flagged this as really suspicious. Here are the five reasons that we think it's really suspicious. Here are timestamps in the interview to look at for yourself. And you make the determination. We're still going to give you the score to how well they did based on the content of what you're saying, of what the candidate said. You make the final decision. Oh, yeah. So if they actually say the word M- out loud, then they're probably reading for a prompt. OK, I like that. Again, it's giving information to the recruiters. so the human makes the decision about do they think this is fraudulent behavior or not. And I suppose if you're going to cheat, then you're going to get caught out pretty soon, either in a later interview stage or fairly soon after you get hired, if you're just making stuff up. Well, it's a massive problem. It's a huge problem.

And it's becoming extremely important for all of our customers to kind of recognize that up front before they get to the point of actually hiring a candidate. and so yeah i mean it's something that we we spend a huge amount of time on okay nice all right uh just to wrap up i always want more people to learn from so whose work are you most excited about now oh my goodness well ours of course look i think there's a ton of really interesting stuff happening i would point to the work uh that the recruiting team at zapier is doing um as really really interesting as well as i think paradigm strategies is doing some really really interesting work around um measuring culture and bias within the organization and then i'd be remiss if i and talk about the team at Anthropics. Sounds like they're both doing great work there. Excellent. Of it, who are you looking at? Yeah, I just kind of echo what Dan said about Zapier. So they've published a few cases. For anyone who wants to really take a deeper dive into would this work for their organization, what are best practices for implementing this,

how do other kind of leading recruiters use it, then Zapier published some case studies showing that. So that's some kind of good reading. Anyone who is kind of interested in the more technical side of voice and wants to kind of go down in an engineering direction, I recommend the reading of a company called PipeCat and Daily. But yeah, if anyone wants to see the product, we'd be happy to show it to them as well. Nice. All right. So we've got two votes for what Zapier are doing. Sounds like it's worth looking into their recruiting process. All right. Thank you so much for your time, Daniel and Alfea. It was great chatting with you. Thanks a bunch. It has been great. Thank you.