Back to Podcasts
Gradient Dissent

Arvind Jain on Building Glean and the Future of Enterprise AI

Gradient Dissent

Tuesday, August 5, 202543m
Arvind Jain on Building Glean and the Future of Enterprise AI

Arvind Jain on Building Glean and the Future of Enterprise AI

Gradient Dissent

0:0043:41

What You'll Learn

  • Glean uses LLMs to build an enterprise AI assistant that connects employees to internal data and knowledge
  • The company started in 2019 and saw the potential of transformer models for enterprise search early on
  • Glean builds custom semantic embedding models to understand each customer's data and language, while using off-the-shelf LLMs for reasoning and generation
  • Security and access controls are a key focus to ensure sensitive data is only accessible to authorized users
  • Arvind Jain emphasizes the importance of leveraging existing AI innovations rather than reinventing the wheel

AI Summary

Arvind Jain, the CEO of Glean, discusses how his company is using large language models (LLMs) to build an enterprise AI assistant that helps employees find information and get answers within their organization. Glean started in 2019 before the recent LLM hype, and the team saw the potential of transformer-based models for enterprise search. They built custom semantic embedding models to understand their customers' data and language, while leveraging out-of-the-box LLMs for tasks like reasoning and generation. Glean also focuses on security and access controls to ensure sensitive data is only accessible to authorized users.

Key Points

  • 1Glean uses LLMs to build an enterprise AI assistant that connects employees to internal data and knowledge
  • 2The company started in 2019 and saw the potential of transformer models for enterprise search early on
  • 3Glean builds custom semantic embedding models to understand each customer's data and language, while using off-the-shelf LLMs for reasoning and generation
  • 4Security and access controls are a key focus to ensure sensitive data is only accessible to authorized users
  • 5Arvind Jain emphasizes the importance of leveraging existing AI innovations rather than reinventing the wheel

Topics Discussed

#Large language models#Enterprise AI#Information retrieval#Natural language processing#AI security

Frequently Asked Questions

What is "Arvind Jain on Building Glean and the Future of Enterprise AI" about?

Arvind Jain, the CEO of Glean, discusses how his company is using large language models (LLMs) to build an enterprise AI assistant that helps employees find information and get answers within their organization. Glean started in 2019 before the recent LLM hype, and the team saw the potential of transformer-based models for enterprise search. They built custom semantic embedding models to understand their customers' data and language, while leveraging out-of-the-box LLMs for tasks like reasoning and generation. Glean also focuses on security and access controls to ensure sensitive data is only accessible to authorized users.

What topics are discussed in this episode?

This episode covers the following topics: Large language models, Enterprise AI, Information retrieval, Natural language processing, AI security.

What is key insight #1 from this episode?

Glean uses LLMs to build an enterprise AI assistant that connects employees to internal data and knowledge

What is key insight #2 from this episode?

The company started in 2019 and saw the potential of transformer models for enterprise search early on

What is key insight #3 from this episode?

Glean builds custom semantic embedding models to understand each customer's data and language, while using off-the-shelf LLMs for reasoning and generation

What is key insight #4 from this episode?

Security and access controls are a key focus to ensure sensitive data is only accessible to authorized users

Who should listen to this episode?

This episode is recommended for anyone interested in Large language models, Enterprise AI, Information retrieval, and those who want to stay updated on the latest developments in AI and technology.

Episode Description

In this episode of Gradient Dissent, Lukas Biewald sits down with Arvind Jain, CEO and founder of Glean. They discuss Glean's evolution from solving enterprise search to building agentic AI tools that understand internal knowledge and workflows. Arvind shares how his early use of transformer models in 2019 laid the foundation for Glean’s success, well before the term "generative AI" was mainstream. They explore the technical and organizational challenges behind enterprise LLMs—including security, hallucination suppression—and when it makes sense to fine-tune models. Arvind also reflects on his previous startup Rubrik and explains how Glean’s AI platform aims to reshape how teams operate, from personalized agents to ever-fresh internal documentation. Follow Arvind Jain: https://x.com/jainarvind Follow Weights & Biases: https://x.com/weights_biases Timestamps:  [00:01:00] What Glean is and how it works  [00:02:39] Starting Glean before the LLM boom  [00:04:10] Using transformers early in enterprise search  [00:06:48] Semantic search vs. generative answers  [00:08:13] When to fine-tune vs. use out-of-box models  [00:12:38] The value of small, purpose-trained models  [00:13:04] Enterprise security and embedding risks [00:16:31] Lessons from Rubrik and starting Glean  [00:19:31] The contrarian bet on enterprise search  [00:22:57] Culture and lessons learned from Google  [00:25:13] Everyone will have their own AI-powered "team"  [00:28:43] Using AI to keep documentation evergreen  [00:31:22] AI-generated churn and risk analysis  [00:33:55] Measuring model improvement with golden sets [00:36:05] Suppressing hallucinations with citations  [00:39:22] Agents that can ping humans for help  [00:40:41] AI as a force multiplier, not a replacement  [00:42:26] The enduring value of hard work

Full Transcript

You're listening to Gradient Dissent, a show about making machine learning work in the real world. And I'm your host, Lucas B. Wald. This is a conversation with Arvind Jain, who I've known for a while. Arvind is the CEO of Glean, which is one of the most successful enterprise companies using LLMs. Maybe right now the most successful enterprise company that foundationally uses LLMs. And I really want you to not be fooled by Arvind's humble style. He's one of the most successful founders in Silicon Valley. Before, Galini founded Rubrik, which IPO'd for, I believe, a value of over $20 billion. So he's been phenomenally successful twice. Very low-key, but clearly very driven and has some real insights. I hope you enjoy this conversation. Arvind, thanks for taking the time to talk. It's been a while. So I think, you know, you have an incredibly successful company. So I feel, you know, maybe embarrassed asking this, but I think it's not a consumer brand. So people might not have heard of it. Can you explain what Glean does, please? Yeah, absolutely. Well, first, Lucas, thank you so much for inviting me to this. Glean is a, it's an enterprise company, enterprise AI company. What we do, I mean, think of us as ChatGPD, but inside your company. Glean connects you with all of your internal systems, your data, your knowledge inside your company, and then gives you a chat GPT-like experience where people come, ask questions, and Glean will answer those questions for them. And it's going to use all of the world's knowledge, but also additionally, it's going to use all of your internal company's data and knowledge and context to help answer those questions for you. So that's, like, simply put, like, you know, that's what we do. Of course, like, you know, as AI, you know, continues to advance the same platform that we use to build our Glean Assistant. You can also use that as an agent building platform to build all kinds of AI agents on our platform. You know, it's kind of interesting about your company is it's been, you know, one of the real successes, I think, or at least from my perspective, one of the real successes in terms of real LM applications for enterprise that works well. But if you look at the timing of when you started it, before it was obvious that LLMs were going to work so well, at least to most people. How did your vision for the company start and how did it kind of evolve through this period of LLMs taking off? Yeah. So Glean started in early 2019. you know, in those days, nobody was talking about large language models. And the term generated AI did not exist to my knowledge at the time. And we were not thinking about that either. And we had a problem at hand that we were interested in solving, and which was search. We, you know, we felt like, you know, this is my experience in every single job that I had before Glean. it's so hard to find things inside the company. When you do your work, you need some information, you're looking for a document. It's extremely hard to actually find information. And it's because as enterprises, we have so much information, so much data, and it's spread across many, many different systems. I guess, you know, in enterprises, you tend to have like hundreds or thousands of different systems. and stuff is all over the place. And so when I'm looking for something, I have a quick question, I need an answer, or I'm looking for a dog that I know exists, but I still cannot get to it, that caused a lot of frustration. And in fact, there have been a lot of research studies that have been done. People would spend one third of their working time just trying to find things. And so that was the first problem that we wanted to solve. We wanted to build a Google for people in their work lives. one place where you go, ask your questions, and we want to bring the right information back to you. And sometimes, you know, it may not even be the information. It may be just looking to connect with the right people who can help you on a given topic. So, like, we become that one place. We'll come and find data, find documents, find people, all in one place. You don't have to think anymore where to go and look for things. So that was the first part that we wanted to build. The interesting thing is that in early 2019, transformers were already a thing. Like it was not like the rest of the, the world didn't really care as much, but inside search teams at Google, we were seeing a really good impact of the transformer technology on some of the core components of a search engine. And we saw this real promise, you know, in that day, you know, in early 2019, you know, where we could conceptually understand information. Like, you know, we didn't have to actually go build a keyword-based search system anymore, which is sort of like, you know, like quite dumb. Like if you think about like how, you know, we match information to people's questions, like it's happening at a very surface level with keywords. But Google had actually put BERT-based models in open domain in late 2018, and we were able to use them and then actually go and pre-train these, what we call now small language models on your enterprise corpus and start to sort of deeply and fundamentally understand what your business is about and what any given document is about. And then do this sort of conceptual semantic matching of information to people's questions. So interestingly, Transformers actually played a foundational role in Glitz evolution. Like, you know, we were probably the first company to use Transformers in the enterprise, you know, in that day. Because it was like really, really applicable to building a good search experience. So that's how we got started. And then, of course, as these models, in 2019, the transformers had no capability to generate information. They had none of the reasoning capabilities that models have today. But they had this embedding-based conceptual sort of mashing capability, which we were able to use. But as these new capabilities came that allowed us to evolve our product and what used to be a Google for you in your work life became more like chat GPT for work life. You can go one step further and now we don't have to just surface information to you. We can actually read that information using AI and actually give you precise answers to your questions. So sort of the evolution has been like that over the last six plus years. So you're politely telling me, I think, that you actually did see the power of Transformers, at least, in 2019 when you started the company, which is super cool. Well, I mean, I think, yes, because we saw it, actually. It was not a vision. We saw how it could really deeply, it could take two documents, documents that had no real word overlap. You look at the actual choice of words using those documents, very different. but still, you know, a bird-based model, you know, trained on your enterprise data could actually tell you that, like, hey, these two documents are very close to each other, semantically. And so that was, you know, in those, remember in 2019, when we, you know, today we expect a lot from AI, but AI was not there in 2019. And that itself was big. That was still a big step function in terms of, like, our ability to, like, you know, really conceptually match people's questions with knowledge. We saw that power and then we decided to bring it to enterprises. You're in this world where these labs are investing massive resources, constantly coming out with new models. Do you still fine-tune a custom model for every customer? Is that important to you? How do you think about what parts to buy or use and what parts to build in-house? Yeah. Well, I mean, the company, like the thing that I actually keep telling our technology team is that do not reinvent things that have been invented already. I mean, this is, we will not, like that's not doing service to us or to our customers. And so our approach always is to actually maximally use innovation that's happening outside because it's available to us. I mean, the good thing is that all of the innovation that's happening in LLM space, all of these models, companies like Google and OpenAI and Enthropic, they're all making those models available to us to use. So we actually like to use those models. And we use them quite a bit in our stack. So the way our technology works is that we actually pre-train and build some models for semantic matching. That we have to do ourselves because these models are all about your enterprise and your data and sort of building an understanding of what your business is about. And so there we actually will go and connect with your enterprise data and knowledge. We'll take all the documents, we'll actually train the model on it so that it can start to understand how your business talks and speaks, what code words you use, what's your lingo. So for that, it's actually important to actually go and train a model on your enterprise corpus. But these are very small models. We don't actually train large models. And these models allow us to actually build custom embeddings for your enterprise content. and so that when people come and ask for things, we can match their requests with your documents in your corpus in a much better way using semantic matching. So those models we build ourselves. But then when it comes time to actually do something with the information, imagine the flow of when you come and ask a question and glean, so the first step of it is, well, we see your question, we're trying to understand what it is about and we're going to use our semantic embedding models, the one that we train ourselves, to assemble the right pieces of information to get the best knowledge within your company that we think can answer those questions. And now we're going to actually take this information and ask AI to actually synthesize an answer from it. And then for that, we don't actually use our own models, you know, use out of the box models like GPT or And similarly for reasoning when you have to take a complex question and try to break it down into a multi agent workflow again we use the reasoning capabilities of the largest formation models. So that's sort of the architecture. So yeah, it's more the don't train, don't fine-tune when you don't need to. But there are certain things in the search stack where it makes sense to do that. Maybe I'll give you one or two more examples. Please, yeah. Well, what one... So in search, of course, part of it is you get a question from a user and you tend to semantically match it with that information. So that's what I described right now. But then there are smaller parts of the stack. For example, we have to expand in retrieval systems. You have to spell check users' questions. You have to actually find synonyms. You have to find acronyms. And for those models, what we realize is that you can take these sort of small open domain language models, and you can fine-tune them for those specific tasks. So, for example, for spy checking, we have a fine-tuned version of a language model that will do it for us. So for some point tasks, like small, very, very specific things, these fine-tuned models actually help us do those things faster and cheaper. they don't add more capabilities. You could actually take a large GPD model to actually do a spell check for you. It'll do a fantastic job. It's just that it's so awkward to actually use that model for that kind of a task. So there's a little bit of that fine-tuning that's happening in the search stack for us. But otherwise, we are, for reasoning, for generation, we use out-of-box models. I think another one of your kind of advantages or parts of your technology that you talk about a lot is the security model that you have. So it's like, I mean, in enterprise search, there's such complicated rules around who can access what data. And as these models get more complicated, like building an embedding model, you could imagine it somehow leaks information across embeddings. Can you talk a little bit about how that works, how you think about it? Yeah. So the way Glean works inside your enterprise is that as we connect with all of these internal systems like Google Drive or Slack or Salesforce, we are understanding content that lives in those systems, but we're also understanding the governance of that data, like who are the people who can access that information. For any given document, for example, we'll look at the list of people who are authorized to see it. And this is fundamental, because if you are going to deliver AI in the enterprise in any way. You have to do it in a safe and secure way. You can't have AI-based systems start to leak information internally to your employees who didn't have rights to see those information. So the way our system works is that we connect with these individual applications, we understand the permission model, and then we actually bake that into our core indexing technology. The permissions are baked into our indexing system. So now when you come and ask questions in Glean, we will retrieve information for you which can answer that question. But we'll actually only retrieve information that you have rights to use. We know who you are, you're assigned in, and we'll use our enterprise identity to establish that these are the documents that you have access to and these are relevant to this topic. And then we make actually models work on it. So that's the fundamental, like, you know, in a RAC-style system, you can actually, you know, make AI safely work on data that only you have, you know, that you have permissions for. But then you also mentioned this interesting question around embedding, like if you're actually building custom embeddings, what content do you actually train it on? Because if you train it on all of your enterprise corpus, it will indeed, like, start to leak information. And you won't realize, you know, in what form or shape it actually does it because ultimately you can't understand behind the scenes how AI works. So what we do is when we're actually training models, custom models, we typically would train them on content that we think is safe to train on. So there's a lot of curiousness that go on behind the scenes to understand what information is actually safe to train on. and so yeah if you use it only on a subset that is safe then your embedding models are actually not leaking information I see I guess speaking of security I think a really interesting fact about you at least from my perspective is you're also the founder of Rubrik which is like maybe not ever as heard of it but an incredibly successful company that recently IPO'd so I feel like you're like you know kind of twice like unicorn verging on like decacorn maybe uh founder and it's like so amazing like you're so like kind of humble and understated um i just wonder like if you have any insights into kind of you know what you're doing differently than other people like what what do you kind of attribute these um these two successes because i think like you know i would think like 10 billion plus exit has got to be like one in a thousand, one in 10,000, maybe where you start. And so the fact that these two are correlated is a powerful indicator that there's more than luck involved, at least in your case. Well, I mean, I think it is correlation, and I think no more than that. But I think what I would say from our experiences, we started Rubrik that was early 2014, and we picked a problem which we knew every business faced. It was really hard to protect your data especially as the attacks are on the rise with ransomware and other things and we felt that at that time, like 2013 when we were contemplating starting a rubric, we felt that all the energy and attention all the best tech people were all focused on consumer technology and enterprise was sort of lagging at the time. We didn't have a lot of great companies, like the SaaS, which was still fairly early at the time. And we felt businesses were being ignored and just helping them with this very fundamental problem that will help keep your data safe was something where there were no products that were being built that were less than two decades old. So we saw an important large problem, you know, with large market. And then after that, it's about just going to do a good job, you know, building a product, good job building a business. And I think we were fortunate that, like, you know, we were able to sort of go build a good team, you know, that then, of course, you know, generated all the success. I would say that, you know, with Glean also, similarly, we picked a problem. which is not a niche problem. You can go and talk to every single person, anybody that you know, like, you know, ask them that, hey, is it easy for you to find things inside your company? And they'll all say no. And so it was a pretty obvious problem. Again, nobody was working on it at the time. And we felt that we had to actually be the pioneers and solve this. I got a lot of advice, by the way, at the time, not to actually get into this business. Well, yeah, because, okay, sorry. I knew you were going to give me something really humble like this, but I think that what these two have in common in my mind, your two companies, is they're kind of like ideas that a lot of people have that don't work. I feel like I've sort of heard those pitches for your two companies quite a lot in my life, but then you were wildly successful doing both of these different things where I think it must be really like an execution-oriented thing. It's not like you kind of came in, it seems like, with some kind of wildly new product that nobody was expecting. It seems like you did a phenomenally good job in a kind of market that didn't seem so great. It sort of seemed broadly crowded and specifically hard to execute in. I'm not sure, but I want to draw some insight out of you. Yeah. Well, I mean, I think like, yeah, that's, that's like, maybe I'll add one thing. Look, like think about search. A lot of people actually told me not to work on search and enterprise search because there were only failures. Yeah, exactly. In the two or three decades, two or three decades of enterprise search history, including like products on Google. Like, you know, every time, you know, Google tried to actually build an enterprise search product was always, you know, always failed. so when you see it's sort of like almost people start to feel like this problem is maybe too hard to solve or maybe it's not even worth solving because if nobody builds a big business maybe there's no need to actually build a product like that so that was a mindset and that's why there was no innovation in search inside the enterprise for a long time there were no startups like we started in 2019 there was no search company that you would know of that got started like 10 years in that 10-year period. This was probably one of the worst areas where no investors wanted to invest. But this is interesting. You see a problem that everyone has. Everybody faces this pain point, yet there are no good products. And then there are also technology trends. We saw some new things that happened at that time. One of them was the whole SaaS transformation, which made the problem so much more worse, but also it actually made the problem tractable in the sense that SaaS systems actually allow you to actually go and read data inside those systems. Before, we couldn't even do that. It was just so hard to get hold of data in the enterprise. We also saw transformers as a really core new capability that would help you deeply understand content and actually build a better search experience. so it was like you know we did observe those we observed those trends which made us you know made us actually confident to actually take that contrarian bet and say that like look you know we think you know we can really now go and build a good product and build a huge business so I think part of it is that like you know I don't know like you know I'm an engineer by the way like you know by you know in my build up and how I you know who I am as a person And doubts are part of my existence Like you know like that how I think most engineers are So I was surprised at myself Like, you know, when everybody else told me not to do this, I don't know what actually was it and which actually made me have that conviction. So I was surprised at myself at that. But, you know, like, maybe that's one thing to think about is, like, large companies, if you're going to build them, like, you know, of course it will tackle like, you know, problems, you know, which are very universal in nature, like, you know, broad, broad, like, you know, which have broad impact. Do you think that you like operate differently? Like, I mean, another thing that's interesting, right, is your product is to kind of help other companies operate better, you know, more efficiently. And I assume you're, you know, kind of dogfooding your own products. Like, do you think that you, like, if I was like a new hire and I was asking you like, should I join or something like that, would you tell me, get used to a special kind of culture that we have here at Queen? That's a good question. I don't know if we have a particularly unique culture. A lot of our learnings and my learnings have come from Google, which is where I spent a lot of my work life at. And Google was a really special company. It was very different from, and I'd actually worked at three companies before Google at that time. And it was so unique in the sense that it actually put technology and innovation in front of everything else. As an engineer at Google, you were the king. Nobody gets to tell you anything. You go and build whatever technology you want to build, it doesn't matter. like you know whether you get to decide whether it's important that you want to build that system and you had all the resources to actually then go and build those things so there was a Google model which is that you hire really smart people really good engineers who are very motivated to do something special and then just let them be and they'll figure out the right things to figure out, they'll figure out how to organize, you know, with other team members and do a great job. And frankly, like, you know, for a manager, that also is a pretty good situation to be you don't ever do any work. And so, like, you know, I sort of, like, you know, I love that model and we always followed that model like now at Clean, you know, where we have a really amazing team and, like, you know, my role is to generally be not in the way, like, you know, I can't help myself. I do come in the way sometimes, but often, like, it's mostly, like, you know, it's about that, like, you know, build a innovation first culture, and success will come. Do you think that having, like, a much better kind of search, and even now you're building these, like, agentic systems, it sounds like that could be even custom to what a company he's trying to do. Do you think that is causing or will cause companies to function dramatically differently in the future? Well, I mean, the way we work is fundamentally going to change, both at an individual level and also at an overall business level. In fact, the vision for the future, I believe this is how our work lives are going to look like in the future. so for any person regardless of like how senior they are you will have an amazing team of assistants, co-workers and coaches that are going to basically help you not only do 90% of the work that you need to do but also help you get better at things that you do and like if you think about today there are certain people in the company that have that luxury. If you're a CEO of a company, you've got to have your coach, you've got to have your assistants, you have a chief of staff, you have an executive team, and you have all this really amazing sort of people around you that you're surrounded with, which actually helps you become that true multiplier for the company. And AI is going to change that. Today, only the CEO or the senior executive can actually have that luxury and that help. and AI sort of democratizes that and brings that help to every single person. So everybody's going to have this dream team around them and you get to, in fact, go and build that dream team with AI-powered agents. You can have an agent that works like your assistant, another one that acts like your coach, yet another one that acts like your co-worker and shares a workload with you. And that's the world that we're going to be in. And so you're no longer alone. You have this team, like, you know, continuously, like, around you. And similarly for, like, companies, most of our business processes that we have today, you could envision that parts of those processes, like, that can actually be easily be handled, like, you know, through some kind of an AI agent. So there's a big transformation ahead of us. It's going to reshape how individuals work. It's going to reshape how a company's, you know, how an organization looks. There may be people's roles are going to change. I think about software engineers, you know, you hear that in the future, we're going to become more code reviewers as opposed to, like, you know, people who write code. So there's, you know, that kind of, like, you know, change in, like, where we spend our time is going to happen to all of us. It's funny. I think about what's in our at Waste and Bias is in our Notion and Google Drive and all these things. We have a lot of, for example, OKRs and our quarterly plans. I keep being tempted to paste them into Gemini. I know that the context window is big enough and put in my current OKRs. I get through the quarter and just be like, which one do you think we're going to miss? What do you think about this strategy? Do you think it's a good idea? If you look back at what we've done, do you ever do stuff like that? Not like tactical coding stuff, but start to ask about broader strategic questions? Yeah. Actually, today with Glean, these are the kind of things that it does a really good job at. Because, you know, Lean is actually connected to all those systems. So you don't actually even have to cut and paste, you know, that OKRs from that Notion or Google Drive doc. Because, you know, it already is in that, you know, that overall enterprise wide search index that Lean has. So some of the things that, you know, that are my favorite activities, you know, with AI now are, number one, when I want to learn about technology or what a given team is working on inside our company or how do we actually design a particular system. So now, instead of setting meetings and asking them to give me an update and walk me through some kind of a presentation, I actually asked Glean to actually generate a latest up-to-date tutorial on that topic for me. And that's all I say. And I have this agent where we told it to actually say that when I'm looking for a tutorial on a given topic, make sure to actually go and look at all the design docs that have been written, but then design docs often get obsolete. And so read the design docs, see the results updated, but then after that actually start to look at more Jira's that got resolved, code that was committed into that component, and sort of get the latest view of how things work. So don't give me the stale doc. Give me the latest on this system and how it works. And this is actually pretty cool because now there's this concept of evergreen documentation. I don't have to deal with stale stuff anymore because AI is actually doing that human-like thing where it's reading stale information and new information and it's actually deciding what's the right and upgrade information and it gets it back to me. Similarly, like an example, similar to what you mentioned. So we're a new business and new businesses, if you build a good product, you don't have a churn problem. And so we actually didn't have any dashboards in our data data dashboards around chair. It was just simply a thing that nobody actually built any analysis on it because it's not relevant. I love how humble you are. That's amazing. I mean, you're old enough that you must have a very sticky product. If six years in or something, you're not worried about churn. Yeah. But guess what? We had some churn that happened last quarter. And then it certainly got worried. Like, okay, what's happening? What's going to happen to all the other customers? And there is the customer health. And I'm exaggerating a little bit. I mean, some teams have it. But we didn't have really good versions of it. So we wanted to do an analysis. And so one of our team members in finance, you know, she's not an engineer, hasn't built anything, like, you know, from a software system perspective. But she went and actually asked this question and told, like, you know, in green, and said that, like, go and look at every single customer of ours. So you can actually ask Salesforce, like, you know, you can look in Salesforce to get a list of all the customers. And for each one of them, go and look at, like, you know, our shared Slack channels with them. go look at what the conversations look like, you know, what sentiment there, go look into our product dashboards and look at the usage. And then, like, you know, take all of that data and based on that, like, you know, now give me a risk profile, like, you know, whether it's, you know, like a account that is healthy, like green, you know, or red or yellow. And actually, sure enough, like, you know, AI is actually pretty good at this. And, like, you know, it generated a report which basically tells us, like, you know all the top risk accounts and the ones that are not And actually frankly I would say like it was actually better better than any dashboard that we would have built because it was able to actually make use of subjective textual information which obviously dashboards can only look at usage trends and some structured information. So these are certainly new capabilities where you could tap into this 95% of your enterprise data is unstructured. And it's full of insights, like only if somebody had time to actually extract them. And that's where AI sort of comes into the end. Totally. Wow, that's such a compelling story. I mean, I want to do that as soon as we get clean and salt here. Wow. How do you... I mean, one thing that I think about with you guys is, your customers probably don't let you pull metrics on their data. Or maybe they do. But how do you know if you are shipping a new model that's better or if you're embedding it's better? How do you actually measure your progress? Yeah. So first, we have customer zero, which is clean, our own instance. And as we build as we sort of bring new models in, we have a valuation framework where there is... What we do is at any customer deployment, actually, there is plenty of information available where you can have question-answer pairs. So people have asked questions and somebody has answered them. You don't even have to actually create these golden data sets. Sometimes you can just obtain them from actual real conversations that people have. For example, in Slack, somebody's asking questions, or people are responding with answers, and there's a lot of reaction and thumbs up to that information. That's a really good example of a golden set of what a question is and how I should be answering it. And so we build these golden sets for each one of our customers, and then we bring a new model, and we will actually run all those questions through the new model. It doesn't have to be a new model. There's also any change that we make in our own systems. It's like our system, there's a lot of things happening there, including changes in the search and retrieval system and then also the models that we use. So you sort of, you measure, you know, like how the answers are now looking, you know, on the colon set, you know, with this new change that you're rolling out. And then you also use AI to sort of, because, you know, like these are subjective answers. Then you have to actually also use LLMs as judged to sort of see like, you know, how close you are to what the colon response is. So that's the fundamental way for us to measure how well our end-to-end system is working. Do you have issues with hallucinations? I feel like in almost every domain, that kind of drives us crazy. Do you take special measures about that? I would think from an enterprise perspective, they might feel more sensitive to hallucinations than even a consumer. Yeah, I mean, 100%. I mean, I think AI has to, I mean, and hallucinations are there. We have some safeguard against it. But that's not to say that Glean is going to actually always answer questions exactly the right way. Like, you know, I think it's not there because there's so many things that can go wrong in terms of, like, when somebody comes and asks you a question, like, can you answer it properly? Sometimes information is not even there to answer. where sometimes information is stale or out of date and that you're using to answer questions for people. So the failure scenario are so many, and one of them is actually the hallucination of the model itself. But for the hallucination part of the model, what we do is remember that in our system, when you come and ask questions, we are the ones who first assemble the raw materials, like the knowledge that we're going to use to answer that question, and we give all of that knowledge at the prompt time to the model along with the question and the model comes back with an answer. What we do after that is that line by line, we'll take every, you know, the response of the model and we'll try to see like, you know, if we can actually find, you know, that same piece of information and, you know, in the input that we gave to the model. And then we'll sort of like, you know, in fact, you know, we'll use that for our core citations framework and we'll cite on line-by-line basis where is that human-generated information that was used to produce, like that AI used to produce this particular line. And when we don't see it, we either suppress it or we actually don't show the citations. So that way you can actually tame the model a little bit. If you see information that's being produced which is not present in the input that we gave, then we can sort of give up and tell the user that, look, you know, like you ask a question, here's some relevant piece of information, but we actually were not able to figure out amounts of it. So that's how we suppress hallucinations a little bit through this sort of citation and reference checking. But the bigger problem, I would say, in the enterprise is not that. The bigger problem in the enterprise when people ask questions is, can you even, you know, get that right information from all these different systems that you have? and you can get the information that is actually up to date and actually correct, you know, written by a subject matter expert. Most of the losses actually happen on that side. I see. Do you imagine a world where these agents start just asking people if they can't figure something out, like just like kind of routing to it? I mean, I think I would kind of like that if I could just deploy a bunch of agents to go collect information from my employees. Yeah. Well, that's an interesting use case. in Glean, so one thing that we do is we are like, like we don't think of our product as take all the information in the company and send it and then answer questions for people. It's very much a people-oriented product. So when we go and deploy Glean inside a company, we actually build a deep understanding of the enterprise itself, like what the business is about, also who are the different people, what do they work on, And in fact, it's a common scenario. Like, you know, when somebody comes and asks questions and we can't answer, sometimes our answer will be that, like, look, you know, we don't, like, have the information, but these are the people who are working on this topic and if we want to connect with them. But then, like, agents, like, you know, sort of, you flip the question, like, you know, where it's, like, instead of humans asking agents to do work, like you're saying, agents are actually pinging, you know, people to retrieve some information from them. Actually, like, that also seems quite legitimate because that's probably the future world, like, you know, where there are these deeply intelligent agents, you know, which are sort of like they work alongside humans. So, yeah, like, you know, humans can ask agents questions and agents can also ask humans questions. Do you have, like, other thoughts on where, like, organizations go as they interact with AI more deeply? well I think I would just add one thing there's a lot of this ROI conversation that tends to happen with AI that the like you know I'm going to have AI come and like you know do like a lot of the work that we do with humans today and we'll be able to replace them Maybe I'll have a small team. Maybe we'll have a single-person company that is a billion dollars in business. And there are all these sort of viewpoints around how AI replaces humans. And I feel like that's missing the point in a big way. Because ultimately, a business and its value comes from its people. I mean that's you know that's that's where the intellect that's where like you know the capacity to do work is and it's so much better to think of AI as an enabler to all of those people so like you know as a business like you know I for example I don't actually shrink my team size I just want to grow it but I want the to be able to do 10 times more work so that we can like you know we can succeed in a much bigger way and so that's that's so I think like you know when you think about enterprises and how they change with AI, people are going to have one of the two mindsets. Either they're feeling the pressure, they want to reduce the team size significantly, or you can actually go and keep investing in people and just improve their top lines significantly. Do you think that the skills that make people successful will change in this world? I think the fundamental skills remain the same. I think what makes one successful, in my opinion, is hard work, dedication and desire to succeed. And I think that doesn't fundamentally change. But of course, like all of us have to adapt. Like, you know, one of the things like, you know, for, I think, our software engineers, like, you know, the, maybe it's no longer necessary for you to sort of remember, like, you know, what are all the different APIs and frameworks available because, you know, like AI can always sort of bring that information back to you. So you spend more time thinking, more time designing, less time writing code, maybe. So like, you know, there'll be, but I feel like, you know, those are sort of like incremental, these changes will happen on an incremental basis. And we won't realize, you know, by the time we have transformed ourselves, like it's sort of, that's my feeling. It will feel like a step function change. Like, you know, like every day there's one new task, you know, that we will start using AI for. And, and, and before you know it, like, you know, like you're fundamentally different from what you were two years back. All right. Well, it seems like a nice place to end. Thanks so much for your time, Arvind. Yeah, thank you, Lubek. It's been really fun to have this conversation. Yeah, appreciate it. Thanks. Yeah, it's a good one. Thanks so much for listening to this episode of Gradient Descent. Please stay tuned for

Share on XShare on LinkedIn

Related Episodes

Comments
?

No comments yet

Be the first to comment

AI Curator

Your AI news assistant

Ask me anything about AI

I can help you understand AI news, trends, and technologies