データとセマンティックレイヤーを整えて製薬AIエージェントを正解に導く――Flatiron Health リチャード・ノーキ氏

Welcome, everyone, to the Emerge AI in Business podcast. Today's guest is Richard Canocchi, Head of Data Science and AI at Flatiron Health. Flatiron Health develops technology and data solutions that help oncology teams generate and apply real-world evidence to improve cancer care and research. Its platform connects clinical expertise with structured data and AI-enabled insights to support treatment, regulatory, and scientific decision-making. In this episode, Richard explains why AI agents in pharma are only as reliable as the data foundation and semantic layer beneath them. He describes how his team built its semantic layer from years of human analysis, used bad history to benchmark its agents, and found that having an agent resolve ambiguity in a question before answering closed a large share of its early accuracy gap. Today's episode is sponsored by Flatiron.

Emerge works with a select group of AI vendors to reach Fortune 500 decision makers through research, media, and direct access. If you want to be considered, download our media kit at emerge.com.ad1. That's emerj.com.ad1. Now the conversation with Richard. So, Richard, welcome to the show. Hi, thanks. It's great to be here. Yeah, we've got a lot to unpack in a very fast-moving industry here in life sciences, and the revolutionary changes we're seeing in life sciences in part have to do with the fact that the AI stack means something very different now than it might have half a decade ago or even three years ago in life sciences. I'd love to get your take, however you want to explain it, sort of what the layers of the AI stack are within sort of the big drug makers

and kind of what's changed recently. I agree. This space is always evolving really quickly these days. I would say when I think of the layers of the AI stack, there's kind of three central pillars to it. There's the data foundation, the semantic layer, and the agentic layer. So maybe to elaborate on each of those. At the data foundation, I think, you know, a lot of the focus on discourse these days is on AI agents themselves. But at the end of the day, if you don't have a solid data layer for those agents to work off of, they're not going to come to the right conclusions. And that's even more important now with AI agents, because if a human has been working with that data for, you know, a few years, they know the ins and outs of it, the nuances, where there might be gaps and things like that. They can fill that in as they need to with that institutional knowledge they've built up. an AI agent just has access to the context that you give it. And so anywhere where there's those holes and things like that, it's going to kind of elevate that issue to a new level.

And so that strong data foundation is even more important than it's been in the past. And, you know, like at Plattern, we spend a lot of time thinking about that. That's kind of our bread and butter is curating high quality real world data. We've published like different frameworks on how we do that. So then maybe moving on to the semantic layer, that's really like all about how AI understands the data, right? And I think of that as having two or maybe even three parts to it. The most basic version of that is just the semantic model. So, you know, does the AI understand what data tables exist, what columns exist, what the data types are? Is there null values in some of those things like that? But I think a common failure mode that I see happen is people just stop at that level. They think if you just tell the AI what's in the data, that's going to be good enough. And what we've seen is that's really not the case. There's a lot of nuances that get built up when you're curating the data, a lot of choices that get made along the way that impact how that data should be applied and what it's

useful for. So you really need to build up appropriate tools for surfacing that context and those nuances to the agents through things like RAG systems or MCP tools. And then the last piece that I mentioned was that agentic layer. That's the harnesses that are doing the actual work, things like building patient cohorts or literature review or analyzing the data. There's a lot of different important aspects to consider when building out that layer, things like how transparent is it? How do you trust that it's high quality? How do you make sure that it's interoperable with other systems? I imagine we'll get into all sorts of aspects. Oh, absolutely. I'm going to double click on some of this stuff right now for sure. So if we could, I'd love to just poke around on some of what you just laid on the table. So you talked about kind of the data foundation and then the semantic layer, you know, in the life sciences world where you live, you know, there's entire companies that just sell ontologies, complex ontologies of a gajillion molecules and all the thousand names that they can have, you know, interact with each other and whatever else. And to your point, just feeding the data without saying, okay, these things cluster in this way.

We kind of mean this when we say this here, et cetera. Do you see that semantic layer being different by companies, by teams? Because some people would say, okay, how is that any different than the data layer? And obviously there's a distinction. How do you kind of explain that distinction of the semantic layer that adds the richness, that lets us get agentic? It's not just the raw data. It's that middle sort of meaning piece. How do you explain that to leaders that are just asking about it? Yeah, for sure. I think that the semantic layer is going to mean something different depending on who you are that's working with the AI agents and what your needs are. So, you know, like a data partner like Flatiron, we're going to have semantic layers that are really focused on, for one, like what's in our data set? How do you, you know, like I was saying, how do you understand all the nuances of how that data was curated and what that means for when you should use certain columns in an analysis versus use other approaches? I also think that there's sort of universally applicable semantic layers. So, you know, some of the things that you were mentioning where there's ontologies that describe

here's some biological or molecular concepts or, you know, clinical concepts, those things are universal. And so it's important to get them right, but you can maybe get those from an outside source. You don't necessarily have to go. Off the shelf, off the shelf somewhere versus, yeah, the messiness of your own data is almost a little bit of a difference. Yeah, right. And then within pharma, you know, it depends on who the stakeholder is. A semantic layer there, it could mean some of the things that we've been talking about already, but it could mean being able to describe or define what an active prescriber or an engaged healthcare provider or something like that means by their commercial team's definition or something along those lines. So, yeah, there's no one-size solution for all of these things, but it's something that is, I think, critical to build out as like a fundamental layer before you can really make progress in building the agents on top of it. Yeah, I want to get into some of the barriers that come up when these things aren't really structured properly, because many life sciences firms are still really in the journey of,

let's say, getting the data house in order, per se. We're not fully at some kind of agentic, 100-year-old pharma firms. Not there yet, but I think the vision is somewhat clear, and a lot of the pieces are in place. But right before we click into that, in this semantic layer, my guess is that involves a lot of strategic decisions that have to be made So at some point when we sort of unifying and kind of de as well as we can these various and sundry data sources there have to be different kinds of leaders and different kinds of boots on the ground experts to get together and say okay at a high level how do we want to parse the meaning of these things? Or how do we want to categorize these things? Or how do we want to, like, it almost feels like there's a sit down strategy part of, you know, semantics isn't just, I don't know, cluster the things that are similar. That might not be what's going to help the business. These are really deliberate, intellectual, strategic, product, patient level, business workflow level decisions. Is this a bit of a strategy hands-on to get that semantic layer well? Yeah, absolutely. But I think the thing that helps a lot there is, at least in Flatiron's position, and I would assume that most big pharma companies are in this position too,

we had many, many years of data built up where we had done this manually with humans. And so you You don't have to reinvent the wheel. We leaned really heavily into those past analyses that we've done or past SQL queries that we've generated with humans where we had this giant backlog that we could lean on to, for one, build out the semantic layer in the first place, to list out here are all the nuances, here's all the best practices. But we had things like we have a knowledge center that's available internally into our customers that lay out a lot of those nuances. We have internal playbooks. So we lean on those to build the semantic layer. And then you can also lean on that historical data to pressure test. Is this system that you're building working well? Is it generating the same kind of SQL query or analysis or using the same columns that you would expect it to based on what you've seen humans doing in those same situations in the past? Yeah, so you want to vacuum up all the context that exists, structure, stratify it, maybe be able to use that as an initial semantic swing and say, okay, does this have what we need? Are there places where we need more refinement, etc.? Like you said, backlogs of all kinds of processes and evidence of what humans use it for. I think that's a useful distinction.

We get a little bit more into like the places where, let's say, not having this in order today is keeping progress from happening as fast as researchers, company leadership execs would sort of hope. You know, when you look at scenarios where the data layer isn't quite where it needs to be, or maybe it's mostly there, but the semantic layer is still a little bit up in the air. You know, you mentioned that this sort of might bar us from, you know, building patient lists or I imagine having any confidence in what our agentic systems could do. But what are some concrete examples that could show what kind of not getting data right can do to hold back progress on projects? Yeah, I'm thinking of a really simple example right now, actually, which is one of the first things that Flatiron did with AI agents is to build a tool that builds cohorts of patients. And this was initially like an internal tool that we used. And the idea was our customers would come to our life science partners team with a question about, you know, do you have patients that meet certain inclusion exclusion criteria? And then that life science partnership team would have to go to our data analyst team with that question.

And usually it wouldn't be scoped very well. One of the biggest things we found is people don't always ask questions in a way that is programmatic and robotic and that an AI agent is going to be able to say this is a perfectly scoped question. So I can go ahead and answer it, right? And so before AI agents, what that looked like was there'd be multiple rounds of back and forth between the clients and our life science partners and our data analyst team back and forth trying to understand what is the real ask here and how does it relate to what's in the data and how do we identify what's ambiguous in that question and turn it into that well scope problem. Right now with with AI agents, now that we have that system in place with that strong semantic layer in place, the agent can look at what's in the database. It can look at how that data was curated. It can look at the best practices and knowledge. And the first thing you have it do every time is reason about where there's ambiguities in this question to make sure that the person who's working with the AI agent and the AI agent itself are on the same page and can get to that right goal.

And this is actually hugely impactful. When we first started building this system and we were comparing it to that old data that we had to benchmark it, this was like two years ago at this point. So it was with older models. The performance was like 60% accurate if you didn't go through that process of looking at the semantic layer and looking where there's ambiguities and clarifying that with the user. anything like that. Because 30% of that inaccuracy, 30 percentage points, right? Not 30% of it, but 30 of those percentage points was coming from just the fact that the user was asking a question and the AI was interpreting it in a different way because they had different understandings of what was in the data and what it meant. Yeah. So, okay. I'm going to nutshell like the basic concept here and then make sure I'm on the right page. And then I'd love to get specific on some of the kinds of tasks, projects, workflows, where this would be really impactful. What you're getting at from what I'm hearing is like dialing in that semantic layer. When did this data come in? What's the context about it? Kind of who put it in where, what, how, the context information that matters,

the semantic meaning dot connecting stuff that ends up mattering. When that's all there, people don't have to speak like robots and people don't have to back think about the 17 kinds of context required to ask this question. They can just ask the question and essentially move forward on whatever the process is. Because for humans, you know, where all of our context lives just hidden inside of our neurons, we don't naturally unpack every ornate detail of it. If that's already sat inside of the nested hierarchies of silicon instead of our brains, then we just say the thing that comes to mind. We're more likely to be able to just move forward on a workflow or have an AI that clarifies a bit of vagary, but it already understands the other areas of vagary. And so we can speak fast, move fast the way humans want to, to get projects to move forward without having to make it like a SQL query. This plus this plus this. Am I kind of roughly on the right page, Richard? Yeah, exactly. I think that's a huge part of it. The other aspect of it that I think is important to highlight, too, is it helps guarantee the quality of the output, right?

Because not only does it let people be able to just speak a natural language and help the AI understand it, but it also prevents situations from coming up where the AI assumes that it can do a certain thing with the data in a certain way, and it turns out it can't because there's some nuance in the data that is leading to a blocker or that would lead you to the wrong conclusion. Yeah, so having that context already kind of nested into the system so that it can clarify some degrees of vagaries, but for the most part, kind of gets what is meant by this phrase, gets what is meant by this, understands how confident it can be about this or that before it responds, et cetera, et cetera. What are just examples of workflows that are common within the kinds of people using these systems? I don't know if you want to talk about clinical trials, processes, or drug development stuff, or wherever you want to start. What are the kinds of things humans would want to work with their AIs to get done where this context understanding is really impactful? There's a lot of different directions you could go with that. There's, first of all,

basic productivity improvements internally, right? It's not super shiny, but is what really helps you transform from something where you're spending most of your time on routine, repetitive tasks to something where you can send off emails, reports, slides, meeting notes, things like that to the AI system and have you focus your time on where you really add the most value So there that lens of it There also like the more glamorous or exciting parts related to the scientific impact like you know how can we use AI to generate predictive models to understand which patients are going to be better responders to a certain treatment and how do you use that for trial enrichment strategies and those kind of things. They're both evolving a lot these days. Curious if you want to lean into one area more than the other here. Yeah, yeah, no, I'm picking up what you're putting down. I mean, I'm imagining, I'll just speak this extemporaneously. If there's any that you want to kind of correct or critique, let me know. But I'm just going to talk about workflows that, you know, we see people working

on. So, you know, there's all kinds of trial report writing stuff with these super regulated multi-thousand page documents that need to be worked on to yada yada. And there was so much trepidation around using AI in almost all parts of that because any bits of hallucination would be terrifying from a regulatory standpoint. Everything needs to be hyper exact and triple checkable and all that stuff. Being able to have a system we can be confident isn't going to hallucinate and knows what we mean by certain things might not mean we can automate the whole report, but maybe we can have it draft versions of these sections of the report or check the most recent regulation and how it might interact with these three chapters or segments of the report and know that what it's giving to us at least will let us focus our human attention in a more robust way. Those might be a thousand little knickknack questions of drafting, reviewing, checking with regulatory circumstances or whatever, where if we didn't have confidence in the AI, we would just want to hop in Google and just do all the legwork ourselves versus maybe all those

little things, those unglamorous, as you had said, which is a good phrase. I think much of the boots on the ground work is unglamorous. Much of that maybe is easier to kind of push through when we have confidence in that context. Let me know if I'm kind of picking up what you're putting down. Yeah, I think that's right. Yeah. For that first aspect of AI and that personal productivity, that's very similar to how I think of it. Yes, there are challenges with that. Like you mentioned, like avoiding hallucinations. We can talk about strategies for that if it's interesting. But generally, I think for that particular application of AI, the most compelling aspect of it is the ability for AI to reduce handoffs between different functions and reduce that coordination overhead. So yes, part of it is you don't have to get into Google Suite and start typing out slides or documents as much anymore. You can use it to generate first drafts. But also just the ability to help people own and iterate on a process from one end to the other without having to pause and pull in other team members at every single step. I think that's one of the most valuable aspects of generative AI.

Yeah, we talked a little bit about the barriers we're leaning into here. What are the opportunities? What are the upsides? and it sounds like one of them for you looking at a bunch of these use cases in the world, in the wild here, is having so much context baked around the AI that instead of having the six humans that own the little piece of the elephant for context, one person can just get a lot done because they can get enough of that buffered context around the edges to just keep plowing forward and just make a lot of headway without having to stop Jane by the water cooler and stop Jacob in the Thursday meeting and make him read something or whatever the case may be. This needs to be what you're articulating here. Yeah, that's exactly right. And I think that goes back to the importance of that semantic layer, right? Because if you don't have that, right, you can't just plow forward like that. But if you have a really strong semantic layer, it enables you to own more parts of the process, keep pushing through those handoffs. And that's not to say, you know, those other functions shouldn't be involved in any parts of the process, but it helps you get to that first draft much more efficiently. It helps you push that further so that when it comes time to review and pull in those

other functions, it's in a cleaner state and you can make decisions faster based off of that. Yeah, I guess I'm thinking for the audience here, almost any workflow or project that you're involved in where there's a lot of touches. You had said coordination overhead, Richard, I think is a good way of phrasing it. Like where are the places where there's just a lot of coordination overhead because the context overhead is massive. And hypothetically, if the semantic layer is nailed after the data layer is nailed, that becomes just less friction through the entire kick-caboodle. Exactly. All those coordination points, those all introduced moments where, you know, for one, you're having to sit there and schedule things, wait for somebody to open up, for them to have time to pick it up. They have the context switch. It's an opportunity for information and context between people to drop off and for mistakes to get made. It's an opportunity for people to feel like they need to step back and reinvent the wheel. Like a lot of the times, if one group is working on a part of the project and they have to hand it off to another. We see it so often that they don't always take what was done before

and build on top of it, right? A lot of times they want to take a step back and build it. And if you have this system built in a way where you can eliminate as many of those handouts as you can, you can avoid all those downsides of that coordination overhead. Yeah, coordination overhead, too many cooks in the kitchen, right? It seems like if people want to find low-hanging fruit for where their own workflows could just be streamlined by getting this stuff right, looking out for where that friction lives might be initial signals as to, okay, this is what might be better if we got this process right. And leaning into that, Richard, from your vantage point, I mean, having seen Flatiron's evolution, also just getting a sense of what's happening within the giant, you guys work with some of the biggest companies in pharma, getting a sense of companies that are maybe more on the road to this path of making agentic real, you know, where we can have confidence in what AI does, we can have confidence in the context support we're getting, we can have context and running the football on certain kinds of projects without needing to pull in a bunch of people. What should leaders be doing that would put them in a position to take advantage of that new paradigm, to get these data layers right? Like there's people tuned in now, maybe it's a chief

data officer, but it could be somebody a little bit lower in the totem pole there who's kind of saying, hey, this is the vision of ultimately where we've got to be. If Agentex is going to be real within our company, this is basically a part of the framework of how that's going to actually look. What should technical leaders be considering now to get on that path? Like what are the strategy decisions? What are the meetings they need to be having? What are the foundation level things they can do to be ready for this kind of a future you've painted? Yeah, there's a few different things coming to mind. We talked about data quality already. I'm not going to spend too much time on that since we talked about it already, but that's a very important aspect that you need to get right. But since we already talked about it, maybe we'll focus on some of the other aspects here. The other two things that are coming to mind is one is dealing with siloed systems. I think that comes into play in two different ways. One is with the data itself, and then the other is with the AI agents. So like on the data side, if all of your data lives in disparate systems, and you're going to want to connect those all to AI agents, you're going to have to duplicate your effort for every data silo you have. You're going to have to have different semantic layers, you're going to have to have different access control

systems, different connectors to all the agentic systems. And given how important it is to make sure that that is all well governed and that agents have access to the data that they should have access to and not the data that they shouldn't, having that all in one place centralized helps reduce the overhead for that greatly. And then on the agent side of it, I think this is a piece that is maybe less obvious, or at least it's an area where Flattern made mistakes early on in our journey, is even within a single company, there's usually different groups that are building their own agents and their own harnesses, or it's like figuring out how to connect it into the UI, especially at big pharma companies. And what you end up with is having a lot of bespoke solutions that can talk to each other at all And so one of the things that we are spending a lot of time thinking about lately is how do you take your AI agents and decouple them from the UIs and like the interfaces that you're engaging with them through and create like these headless situations where you have

agents and tools that are hosted somewhere that can plug into any system. Sometimes that's your own systems or sometimes it's even like as a data vendor thinking about, how do we make it easy for a big pharma company who is building out their own AI agents and their own AI interfaces to pull our agents or our tools into that system so they don't have to go to a different UI for every data vendor that they have. And to be clear, that works for some pharma companies that are more well-resourced. But that's not to say you shouldn't ever have a custom-built UI or something like that, because smaller or less-resourced teams might need something like that and might want to engage in that way. So it's really being flexible and realizing what are your needs, where are they at on that spectrum, and how tightly coupled should those agents be to a particular chat interface versus being able to plug in the different systems on the fly. Okay, I'm going to try to put this together on the table and just sort of see your best ideas for making this actionable for the audience. On the one hand, you're talking about the world of data silos that certainly are enterprise life sciences people will be fully aware of the world of data silos.

You're saying, knowing ahead of time, and maybe this is doing the strategy work or getting the actual bench folks and leadership and the chief data kind of AI folks together and really saying, okay, at a high level, ideally, who should be able to access what at any given time? What things might need to be formatted or structured in different ways to actually be useful when they're part of this person's workflow? Because they're pulling this in and they're doing this manual thing every time this happens. This is kind of ridiculous and way too high friction. So who needs to access what? What needs to be formatted how? And what would that skeleton look like so that as data flows through and is getting work done, we all feel like, yes, it isn't governed well, but it's also serving what we need to do as a business. Was that kind of your first point on some level is get to that? When you look at what it takes to get there, of course, easier said than done, but it is shocking that sometimes people just deal with that friction for years until a strategy kind of comes about from it. when you see some degree of decisions get made up top that make all that easier, you know,

to really kind of nail that first bullet you mentioned, what happens there? Is it generally driven by leaders? Are there different kinds of people in the room? Like who needs to be there to have that strategy combo to make it smoother for everybody? I guess a few thoughts here. I would say generally when I've chatted with our industry leaders about this, for the most part, it's like you said, it's easier said than done, right? These are systems with like historical setups, it's not always easy to change them while you're using them. So I would say for one, there seems to be a common theme that all pharma companies are working towards this, but it's a work in progress across the board. But generally when it's going well, it seems to be in situations where it's coming from high up shared infrastructure leadership within the pharma companies that have a more holistic view that spans the company, right? So it's rather than, you know, a team that's maybe owning one particular database and thinking just about that database. Really, what we're talking about here is you need a common system for governing access and establishing access patterns. And that needs to be something that is a more holistic view that can

be applied across all the data sets. And so it kind of requires that it come from a higher up level, I think. Yeah, buy in at a high enough level to be able to pull on the people that have friction points in all the different junctures and get them together enough to get rid of some of that friction at a higher level. And you sort of said the same thing here in closing, kind of the same thing on some level around agents. Cool. Where are we using them? Where should they be isolated to some little chat interface versus where should they be more permeable into multiple workflows at once? That again, seems to me, Richard, to be something where a little bit of a higher up view would have to come in and say, okay, what do we want to be kind of spidered into where should these tendrils reach, where should they not reach, what should be connected. Same kind of an idea with the agentic lens. Is that right? Yeah, definitely. With the only caveat that some of that space is a lot more novel than the data governance is. So there's a lot of new solutions for doing these kind of things. At Flattern, we use, for instance, like AgentCore to host our agents and our MCP tools. But there's a bunch of different solutions

for doing that. And so I think, again, there, it's important to take that holistic view of what do each of our teams need from our agents and what kind of engagement models do we want to establish between how those agents are hosted and how they get pulled in? Like, are we using MCP? Are we using agent to agent protocol? Do they need to just connect within our systems or do they need to be able to connect externally? Do we allow external connections to come into our systems or so, like, how do you do that in a secure way? And again, a governed way where you're confident the proper access patterns are in place. So it's really like a significant infrastructure design challenge that it's important to think through ahead of time, because otherwise you'll end up in a situation where, like I said, you've built a bunch of bespoke agentic systems that work for one purpose and can't be pulled into like a broader system. And ultimately, you really unlock the benefits of these AI systems once you can start connecting all of these agents and tools that you've developed together to scale the capabilities

that they have. Yeah. So as you said, it's a lot more novel than just the basic kind of data governance structure stuff, but it is still the same holistic view. How should this be set up in the future for all of us to be able to succeed and work together kind of with the combination of people and machine. So that's a useful lens in parting here, how to get ourselves on the right page, not only with data, but for this agentic future that we're entering. And hopefully, Richard, our audience is a little bit more prepared for that future. Thanks to some of your ideas here today. So I know that's all we had for time, but it's been a real pleasure being able to have you on the show. And thanks so much for being here. All right, great. Thanks for having me. Wrapping up today's episode, let's look at the three key takeaways from our conversation with Richard. First, AI agents only know the context that they're given, so gaps in the data foundation turn into wrong answers far faster than they would with experienced human analysts.

Second, a semantic layer has to capture how data was curated and how it should be used, not just what tables and columns exist. And past human analysis are the best material to build and benchmark it. And finally, leadership centralize governance of data access and build agents that can plug into any system, rather than letting each team create siloed bespoke tools. Merge works with a select group of AI vendors to reach Fortune 500 decision makers through research, media, and direct access. If you want to be considered, download our media kit at emerge.com.ad1. That's E-M-E-R-J.com.ad1. For further executive level analysis and to join our network of leaders delivering workflow impact with AI, visit Emerge.com. On behalf of the team at Emerge, we'll see you on the next episode.