エンタープライズAIはなぜROIを出せないのか｜DevRev CTO Ahmed Bashir

And the problem, and I've heard this from a lot of people, is that enterprise AI demos work amazingly, but once you get them into the enterprise and the complexity of the enterprise, everyone's disappointed. While everybody claimed to have industry-leading AI, CIOs were betting on AI heavily, and very few were actually unlocking real ROI. There was a gap in the level of governance, the level of orchestration, the level of synaptic understanding at scale, and that's exactly what shared memory provides. I think that the major observation is that a more sophisticated model cannot always make up for not having a sophisticated memory layer. Swapping in a smarter model didn't win you that much. That what mattered was the plumbing of the system. Is that right? And if so, did that surprise you? If you think about computers with CPU and RAM, just getting a computer with a really, really high-performance CPU cannot counter the ill effects of not having enough RAM. We talked on our first call about re-digitization,

and you had an analogy or an example of how music digitized for CDs but then had to be re-digitized when it moved onto the Internet or into applications. Can you talk about what re-digitization means in this case? Yeah. Yeah, absolutely. So, you know, obviously we went from vinyl to tapes and then ultimately CDs. And when you talk about the compact disc, it was a digital format, right? I mean, we went to a format. I mean, these were optical drives, but they were storing digital content. And so to some degree, you know, we had the satisfaction of knowing that we had moved to the digital era, but it wasn't going to support the iPod. it wasn't going to support the shuffle it wasn't going to support spotify playlists for example i'm just using those as examples so we needed to fundamentally rethink uh you know what digital

music needed to be in order to support all the new scenarios that uh we were going to be able to support with uh the portability that was to come with the mobile era right so with the mobile era. You had the iPod shuffled, you know, the one that you could clip onto your, to your, you know, at leisure apparel and, and on you go. And of course, you know, now you have AirPods and you have music on your watch and everything else that we've seen over the last 20 years. And it's all because we were willing to re-digitize. And of course, this has happened several times. You know, when we moved to the cloud, we didn't just take databases as they were on premise and just move them over, we created an entirely new data interface through web applications and APIs and microservices. But then also, we took that information, we created a change data capture mechanism so that we can move the same exact data every few minutes, every few hours over to create a reservoir, what we call the data lake, which then gave way to a snowflake, which gave way to

a Databricks, which gave way to a BigQuery. And so we created this entire alternate universe where analysts could work and scientists and engineers, data engineers could work and, you know, define new features and find new opportunities and pattern match and understand trends and usage and run their business without disturbing the customer and the customer interactions, right? So we've gone through this time and again, and I think we're going through the next one right now. Flagship product is called Computer. Is that right? And it's a conversational, quote unquote, teammate that syncs structured and unstructured data from existing tools and builds a knowledge graph. Yeah, I mean, absolutely, Craig. We'll start with the name. Craig, I think it's a nod to the fact that this is we're going through sort of a re-digitization at this point. We've done this before, you know, when we went from, you know, having, you know, accounts and books to spreadsheets and, you know, spreadsheets to, you know, more sophisticated accounting software to cloud and web software.

And then, of course, from cloud to, you know, what we see with Databricks and ML engineer and data scientist and business analyst type software, which needed its own, you know, data format. And again, we need yet another data format that is AI native, something that can structure and index and join and relate and negate information so that it can provide really, really high value search results, answers, and of course take the right actions. And so this new expectation, this new demand from AI is not just about being able to speak in natural language or make up very interesting and compelling sentences. We also have to combine it with truly understanding the work that we do, the role that we play within the business, the customers we serve, the products and services we create. These relationships have to be understood. There has to be a meaningful governance and orchestration layer on top of it. And fundamentally, it has to help us with our search, with our answers, with our actions.

And so that's what our focus has been. And we think that this constitutes sort of a re-digitization. We think this is the modern computer. The idea is that the modern computer is AI first, and it is a fundamental reshift in what we expect from the computer in the workplace. And for that reason, we've sort of taken that name and, you know, that sort of defines what our business is doing right now. I think there's sort of three layers. I mean, when you think about the enterprise data architecture, the enterprise AI data architecture, you've got obviously all these systems where data exists today. So sort of the digital data of today is found in all these various tools and products that you've talked about. And then, of course, you've got these new reasoning LLMs that have come up. But there's more to the architecture. There's the semantics. There's the orchestration.

There's the governance layer. And altogether, we call this shared memory. And so what are some characteristics of shared memory? Well, first of all, they have to be able to sort of relate information across these tools because the reality is a lot of times answers they were searching for are across tools, across departments even sometimes. And so you have to be able to figure out how to connect with multiple tools. That's one characteristic. The other thing is when you connect, you have to be able to connect bidirectionally so that your answers can then give way to actions, which are increasingly agentic actions. And so you have to be able to do writebacks. That means you need to have a bidirectional channel. It's not a single, you know, unidirectional channel. And so that's the second major characteristic of this shared memory that we're talking about, this re-digitization that's happening. And then the third major characteristic is permissions. You cannot shed permissions as you create this. This is one of the most important and most complicated aspects because, you know, some of these more sophisticated tools,

they themselves are an amalgamation of several tools or several iterations of software that have come together over the last two decades or more. And so you might, within one tool, have seven different permissioning systems, and you have to be able to respect them, and you have to be able to bring the information together with the identities that already exist, without necessarily forgetting the identities or, more critically, forgetting the relationship between information. You might have dependent fields. You might have conditional logic on what field values might be supported, right? For example, if your state is California, then San Jose is the city that you can select. But if your state is Texas, then perhaps San Jose is not a city in Texas. So as an example, you have to understand permissions. You have to understand attributes, objects, and conditional logic that governs permissible values, permissible stage changes. So all in all, schema awareness, permission awareness, bidirectionality, and also ability to connect a diverse subset of systems.

These are all some of the prominent qualities of what it looks like to re-digitize. There's more, but those are the prominent qualities, I would say. And then computer has a conversational interface. So you inquire using natural language, and then what it writes, SQL queries on the back end, if the data is in a SQL database, or what happens on the back end? Yeah, that's a great question. And so what really fascinating about computer is computers aim is to give you the absolute highest degree of precision efficiency and safety And so you know in a world where there no shortage of chatbots and conversational interfaces what we wanted to do was create something that was ready for the enterprise, that understands your team. So it's team first, it's multi-model, it's multi-modal, it's multi-surface. And so,

yes, it's a conversational interface, but it has a visible catalog in the background. And so even though you're interacting using conversational interface, you can click and see a record, you can see a list, you can see a dashboard. And that's part of the safety proposition is the fact that it's not an invisible catalog. Invisible catalog is quite unnatural for humans. Humans feel safer when you can touch and see the attributes and see the assignee and the priority. And it's not just a conversation that you're having about it. You can see it. You can touch it and change it and click on it and so on. So that's something that's built in in the background as well. You click on the citations and it's all grounded and visible. And that's a big part of it. But yes, I mean, for the day-to-day user, it's a conversational experience, but one that's available across many surfaces. So you can embed it into your help center or your portal or your mobile app or your website. And so it can take on a lot of different formations and surfaces.

But yes, one of the most common ways to use it is on your phone or as a desktop app or as a web app. And it's quite powerful in the sense that, yes, it gives you that sort of gen UI, generative UI main panel that allows you to interact using natural language. But what's really interesting is that alongside that main panel, you also have the left panel, which gives you collaborative use cases. so you can basically have conversations where you're talking to computer but alongside your team members which is a very powerful way of collaboratively getting outcomes and of course those outcomes can take the form of the right pane the right pane is what we call the canvas where things that you produce or inputs to the conversation can can live so I can attach PDFs if I want to, or I can actually produce presentations, spreadsheets, websites, and I can even launch those websites and make them go live.

So it's not just something that's sort of bounded by the lifespan of this session, but obviously any artifacts that I created are now part of the catalog, as you would imagine, but more importantly, can be shared publicly or with large groups of people through this canvas. And so there's some important nuances to the overall experience, the fact that it's multi-surface, multi-modal, but then also multiplayer. And then, of course, sort of the fact that it's also quite generative in the sense that you can produce content that then makes its way back into the catalog naturally, but then also can make its way as something that you launch or share with large groups of people, groups that exist or publicly as you wish. So, Craig, you know, in a nutshell, Enterprise Bench is a benchmark. And, you know, the question would be, why another benchmark? And actually, the answer is quite profound. You know, I found myself at several conferences all of 2025. This was sort of, you know, what we thought was sort of the peak of the AI hype cycle at that time.

And we didn't realize that it was going to continue all the way through 2026. But at that moment in time, I saw a lot of exhibits where people were talking about having industry-leading AI. And this was sort of what I would call a very reductive and misleading premise that a company can claim at that early juncture to have industry-leading AI. And how is it that everybody has industry-leading AI? And you might say that it's false marketing by a lot of companies that were trying to AI wash their businesses. And probably there's some truth to that. But the reality is also that people automatically assume that if they were using the quote unquote best models, they had industry leading AI. So if they were using an industry leading model, you had industry leading AI. And of course, you know, a sophisticated buyer would see through that type of claim. But at that moment in time, there was really nothing to counter that type of reductive thinking. And so, you know, we wanted to make sure that there was a a true and highly authentic view of what makes enterprises work with AI.

Because the truth was, while everybody claimed to have industry-leading AI, CIOs were betting on AI heavily and very few were actually unlocking real ROI. And so you had this sort of dichotomy in the market where it seems like you have so many different options to industry-leading AI, and yet you had such little chance of success. And so why was that? In the reality, something that we believed in, but we wanted to make it more explicitly obvious to us for the market was that it's quite simple. It's that while the rest of the benchmarks that were being done were focusing on increasing the level of complexity of tasks, what we observed was that tasks are not simple in the workplace, but they're also not in the Six Sigma or the Three Sigma, meaning they're not the most complex things that humans can do. They are of progressive complexity. They can get complicated. Sometimes they're rather simple. What's hard is not always the task. It's

sometimes the data. Enterprises are organizationally complex. That means they use a lot of tools. They have a lot of versions. They have a lot of permissions. They have a lot of roles, jobs. They have a lot of fragmentation. That is what makes it hard is because they're organizationally complex. And what we observed in the market was that benchmarks that were prominent at the time did not represent enterprise complexity very well. They only represented task complexity. And so what we set out to do was to create a benchmark that had different levels of task complexity, but more importantly, it authentically represented the complexity of the enterprise. That means data is not just available on a platter for you to input into an LLM. That's just never going to be the case. You have to find it. You have to validate it. You have to join it, structure it. You have to understand it. That is the hardest part of the whole thing.

You have to figure out where the real information exists. And then you have to be able to bring it together to answer the question or to take the action. And so we became the foremost and really unique in this regard that we've not only defined enterprise complexity, but we came up with a way programmatically to represent it as part of a benchmark. Meaning we went from a system that was effectively putting data on flat files and saying, here it is, input it as part of this task, to saying, go figure out where it is and then perform the task. And that became sort of the thing that we became famous for was that Enterprise Bench understands what makes enterprise AI complex and represents why ROI has been hard to come by despite such sophisticated and powerful LLMs. and that was very clear. You needed a better memory layer. Let's talk about something a lot of us have probably done. Typed a question into Google or ChatTPT

and asked it to recommend a piece of software. The next time you do that, it's worth asking one more question. What data is that recommendation based on? Because here's the catch. The recommendation you get back is only as good as the data behind it. And a lot of that data is thin. That's where G2 comes in. G2 is the world's largest and most trusted source on B2B software, built on real, verified reviews from real users at real companies. Across every category, including the fast-growing world of AI tools, whether you're comparing established platforms or the latest AI tools, G2 helps you see what actual users love, what they don't, and what you should know before you buy. There's a reason teams at every Fortune 500 company use G2 to make software decisions. Peer proof still beats a hunch. Before your next software

decision, go see what actual users are saying on g2.com. And if you haven't, go back and listen to the podcast interview I've done with the company. And the problem, and I've heard this from a lot of people, is that enterprise AI demos work amazingly, but once you get them into the enterprise and the complexity of the enterprise, everyone's disappointed. And the benchmarks that people are generally presented with are LLM benchmarks right or the underlying model benchmarks that measure you know the ability, as you said, to perform complex reasoning tasks and things like that. They don't, and what you're saying is that for those benchmarks, the data is prepackaged and delivered. and of course then the model does amazingly.

But if the data is messy the way it is in most enterprises, the performance isn't as high. So Enterprise Benchmark was designed to test whole systems or to test models under a more realistic scenario with data buried all over the place. Yeah, it's the harness. It's the entire agent harness. Right. So we're looking at it because that's what you're going to use, right? So if you're going to use a Hermes agent using a particular model, then that's the configuration that we test. And we can put a link to the leaderboard so people can see more about it. But, yeah, effectively what you want to do is test in totality the experience, right? Not just like small components. You want to because, I mean, in terms of component benchmarking, there's others out there. Really, what you want to do is figure out what is it that, you know, the enterprise is going to consume?

What is that logical unit? And that logical unit is the entire agent harness. It's the agent with however is going to manage the data with however is going to manage the model interaction. And so that's what we want to do. And what's really fascinating is that right now we're doing a we're doing a sort of a focused job on precision and efficiency. So, you know, just not to bury the lead, but right now what we're seeing is that on a relative scale, the accuracy improvement between cloud code using an Opus family model and computer using the same Opus family model is a relative delta of 50%, meaning computers with 50% higher accuracy than cloud code using the exact same model. You might say, well, why? Well, it's the same actual pattern that we're seeing in the market right now. When we talk about CIOs not seeing ROI, it's because the outcomes are hard to come by without shared memory. Shared memory gives you the semantics, the governance, and the orchestration.

Just to give you a very concrete example, I'm sure this is something that even our viewers would relate to. We see patterns where leadership sees a working demo of some kind of ticket clustering or some sort of financial report correlation, some sort of an assessment of deals that went sideways. And so the assessment, the correlation, the classification, the summarization, all of that was done rather easily. And this is remarkable because these things were rather hard for humans. They were hard because of the scale, right? Because you have to bring a lot of judgment along with also synthesizing a lot of information in order to perform that task. But for the GPU, it was rather easy, particularly if some of the, let's just say, non-deterministic logic could just be described in plain English language. And that made it quite easy. It just felt like it was doing your job. But the truth is, in those same circumstances, were we willing to take the findings and update the CRM?

Were we willing to update the status page? Were we willing to update the financial records? And the answer is definitively no. And we know that because we've seen the MIT report, we've seen, you know, IDC reports, we've seen reports where CIOs are saying that only 28% of them saw the value that they were expecting. And so absolutely, there's a huge percentage of people that were disappointed in the final result. And so what could that possibly mean? And what it means is that there was a gap in the level of governance, the level of orchestration, the level of semantic understanding at scale. And that's exactly what shared memory provides. It's not about the, you know, a delta or a misunderstanding of the model. The model's capabilities are actually well understood right now. I mean, there's more to understand, but they're understood well enough to use them under the right governance, under the right orchestration, under the right semantic understanding of your business. That is not going to be at the reasoning layer.

That's going to be at the shared memory layer, which is something that obviously our business and other businesses have really focused on over the last couple of years. For us, it's been five and a half years. This episode is sponsored by MindTrip. AI agents have a last mile problem, and travel can see it most clearly. Planning a trip takes one prompt now, but booking it, you're still juggling tabs, comparing prices and points, and managing multiple accounts. MindTrip is closing that gap. It's a travel-specific personal assistant that uses agentic AI to plan your trip and book it for you, too. It learns how you like to travel, helps you make the most of your loyalty points and perks, and books directly through your existing accounts. Most importantly, your credentials and credit cards stay on your device, and you approve every booking. And because MindTrip understands your entire trip, it keeps working after you book, monitoring prices, anticipating your needs, and helping when plans change.

Download MindTrip for iOS to try it yourself or learn more at MindTrip.ai slash book anything. You know, yeah, so I think that the major observation is that a more sophisticated model cannot always make up for not having a sophisticated memory layer. and I think that's not surprising because you know if you think about computers with CPU and RAM if you will just getting a computer with a really really high performance CPU cannot counter the ill effects of not having enough RAM because CPU and RAM perform different functions I mean there is a relationship but they perform different functions and you kind of need a high performing computer to have sufficient CPU and sufficient RAM memory, right? So I think that, you know, having known that, you know,

each component plays a role in the overall success is important in, you know, expectations management. So, I mean, obviously, you know, as a company that's built around the concept of building enterprise-grade shared memory, we sort of felt like this thesis would land and this is where we expected things to be. But, you know, to be able to witness it is really important. Craig, I'll mention one other thing about tasks, since you did ask me some questions about the tasks themselves. One of the things that we're trying to do here is effectively also define levels of autonomy. And so we kind of classify levels of autonomy as levels one to four. And right now, the benchmark only includes level one and level two tasks. and so uh in as we continuously evolve the benchmark we're going to introduce level three and then progressively l4 as well uh but it's pretty straightforward l1 is a task that's really about searching for information within a single domain uh level two gets into uh multi-domain

uh results and uh you know transitioning from what i would call a search to an answer where you're not just synthesizing information from different sources within a single domain, but perhaps going from single domain to increasingly multi-domain, and also trying to understand how to relate information, maybe in a more sophisticated way, which is the difference between just doing a search and getting an answer. And that's L1 going to L2. L3 goes from, you know, actioning on data alone to actually reasoning towards some level of what I would call single domain actions. So like getting outcomes by performing actions within a domain. But in a way that is perhaps part of a larger outcome that is still, you know, larger than the scope of the action that was taken here in the task. And then, of course, L4 is end-to-end multi-domain actions that complete a job. And so that is sort of a very high degree of autonomy.

And so what we've done is effectively defined these L1 to L4 levels of autonomy in a way that allows us progressively across all departments. So this is now department agnostic, allows us to define autonomy and then of course define tasks in each department that align with that level of autonomy so that people can also start to educate themselves on you know how effective different agents different models can be in performing varying degrees of autonomy across departments Because AI is across departments so we felt like this was the most well way to actually organize our benchmark which is, I think, another big part of the benchmark's contribution to the open discourse around enterprise AI is defining levels of autonomy and also figuring out when to distinguish across departments and when not to. You know, one very common way of bringing data into the context for the LLMs is using a technology called MCP.

And so MCP allows you to make systems compatible with LLMs by allowing agents to make a decision on what information to selectively grab from different systems that have MCP servers. And so a lot of the configurations that we test rely on MCP to bring information about accounts, tickets, documents into the context so that you can answer the question and define the task. Right. So and by the way, like comparing computer, same model, computer versus cloud code, you know, we saw, you know, on average, anywhere from like 75 to 80 percent reduction, fewer tokens used by computer versus cloud code. And you might say, well, why would that be? And the reason is because oftentimes with MCP, the agent is burdened with the idea of figuring out what the best context is.

And when it's burdened with that responsibility, sometimes it can figure it out that it will often use a lot of tokens. and often if it spends enough tokens, it will perhaps too eagerly give up because it feels like it's already wasted enough time and money trying to get enough context and that it needs to just stop and now just proceed to the next step because after all, it came up with a plan and no one step in the plan can effectively sabotage the rest of the plan. So what happens is, as you have increasing levels of data or ambiguity in the data, it's very difficult for a technology, like a generic technology, for example, MCP, to know when you have, quote, unquote, all the information you need. Just imagine you've got, I don't know, 100,000 accounts, and you're paginating through them 50 at a time. How do you know if you need to go to the next page? You might find something that's relevant in the next page. it's kind of like going through um uh you know a reddit page or a google search results i mean

there could be something on page five that you need but at some point you just have to stop and so if you need an exact answer the fact that you must stop without knowing whether you've seen at all is a common reason why accuracy suffers and there's many different versions of this that it gets worse and worse when you go from one system to three, three to five, because now you have to make a determination, do I have what I need across all five systems? And honestly, there's just no answer because there's no shared memory that can definitively answer for you. You can contrast that with what shared memory with computer does, which is it actually digitizes that information in a native format. And it allows you to actually take this sort of reasoning task, which is probabilistic, and just remove it from the responsibility of the LLM entirely and say, you know what, to actually figure out how to create the context, we're just going to use databases and we're going to use CPUs. And so that's going to save you a ton of tokens because you're just using CPU

and you're just running SQL. And SQL is deterministic and is grounded. And, you know, It's something that's existed for 40, 50 years. And so we use that as the basis of how we define context, which then goes into the LLM. So, you know, the secret is don't use LLMs for things that you can't, you know, absorb some level of probability or, you know, you don't want to take chances around. So that's what we do. We use databases where you don't want to take a chance. And, you know, building context that's exhaustive or repeatedly, you know, consistent is something that is a big part of how enterprises choose to take actions or get answers, which is why it's so well suited to databases and has been for decades. And so we preserve that. We don't abandon the database, you know, in lieu of the LLMs. We sort of combine databases and LLMs to get the best of both worlds.

And when I say databases, I don't just mean traditional type of database. These are all brand new types of experiences, some that are more important for transactive type things, some that are more analytical, some that are more semantic or vector databases or graph databases for pattern matching, what have you. So, you know, this is a modern idea of databases that sort of takes whatever shape it needs because our conversational interactions can take many shapes. So, you know, the database itself must take those shapes as well. Yeah. And what you're saying is that by using the more deterministic queries of databases, you're relying on CPUs, which are much cheaper to run, and save the GPUs for when you need LLM reasoning or something on the data. That's right. Is that right?

Exactly. And of course, using the CPU is going to save you all those tokens, which are expensive. But also, you know, databases have pre-built understanding of relationships, indexes. They have structure. They've created a well sort of tuned way of joining information and negating information and aggregating information and projecting information. And so all of these things are well defined in SQL. So why reinvent the wheel coming out of U of I? I mean, we had, you know, I had taken a class where all we did was study the beginnings of databases, starting with the system R paper and, you know, learning about all the different subsystems, the buffer subsystem, the transformation layer, the query optimization layer. And so I had this deep reverence for that technology. It's just that, you know, after doing that for a few years, it was pretty apparent to me that, you know, the tide was turning. And, you know, that's when I shifted as well.

I shifted over to Apple right sort of on the heels of their iPhone announcement. And I spent almost 13 years at Apple, 12 and a half, 13 years there. I was fortunate enough to be part of the founding team that launched in the same year, iMessage and iCloud. You know, I was the solo engineer on the iCloud backup and restore feature on the back end. And there was two engineers on the front end working on the iPhone integration and the integration with the iPhone sort of onboarding experience that we had launched for the first time that year. So for the first time, you could actually set up your iPhone without needing a Mac, which was a big deal. And you could actually save all the contents of your phone without needing a Mac. Because prior to that, you would set it up on an iTunes-based Mac. And you didn't need that anymore. You could basically buy a phone and be completely PC-free effectively. And even like sort of towards the tail end of my tenure at Apple, I had this deep interest to do startups

because that's really what it means to live in South Silicon Valley and to be a technologist. So, you know, I had this deep interest and obviously I was very happy with what I was doing at Apple, but certainly had this hunger. And, you know, we started to see this with models and, you know, some of the things that were coming out, you know, towards the end of the last decade and, of course, the early part of this decade. And so when the opportunity came about and, you know, there was an opportunity to work with Dirich and the team, they had just announced the startup, I gave yours a call and I said, hey, let's do this because the company was backed by Vinod Khosla, who was deeply interested in investing in AI. He had just invested in OpenAI. And, you know, we sort of represented the memory layer, whereas, you know, OpenAI kind of represented the model layer. So the conjunction of the memory and the model together was how enterprise AI was going to really take off. And, you know, these things take years and years to develop. And certainly we've spent, you know, I don't know, almost a quarter

billion probably to build this memory layer that we're quite proud of and several aspects of it are quite patented as well. So that's what we've been up to over the last five years. So, you know, that's, you know, maybe five minutes on my background.