AIの憲章・統制・実装：Midwest AI Summit 講演集 第2部

Welcome to the Practical AI Podcast, where we break down the real-world applications of artificial intelligence and how it's shaping the way we live, work, and create. Our goal is to help make AI technology practical, productive, and accessible to everyone. Whether you're a developer, business leader, or just curious about the tech behind the buzz, you're in the right place. Be sure to connect with us on LinkedIn, X, or Blue Sky to stay up to date with episode drops, behind the scenes content, and AI insights. You can learn more at practicalai.fm. Now, on to the show. Welcome to another episode of the Practical AI Podcast. I'm Chris Benson, your co-host. And this week, we're going to bring you three more of our speakers from the Midwest AI Summit. They are three speakers who are dealing with AI issues on an everyday basis that many of us are coping with.

Those three speakers are Don Barger from the International Mission Board, Mark Lay from Noblest, and Darren Talley from AeroCore Technologies. Next week, I will be back in the usual format with a guest speaker. But in the meantime, I hope you enjoy these three. Thank you very much. I want to introduce our first speaker, Don Barger. I've known Donnie for a long time, and he's taken kind of the whole journey of Prediction Guard with us. And I'm so impressed with how he has stewarded the International Mission Board forward in terms of their AI journey and adoption. He's the Director of Innovation and Artificial Intelligence at the International Mission Board. He spent a lot of years in Latin America and different parts of the world with really deploying strategies rooted in orality and storytelling. He's going to be talking about building AI trust, what we said yes to, and what we didn't.

So welcome, Don Barger from the IMB. Appreciate it. Yes, I've known Dan for a while. Well, the first time I met Dan was at a much smaller meeting than this that he was actually leading. Again, another meeting he was leading about AI. And at that meeting, I was the dumbest person at that meeting. It was hands down. Like I walked in, I'm like, I don't think I knew what I was coming to. And I can say that if it hadn't been for Dan at that meeting and then some other meetings that we met at and we kind of sketched out some stuff in a hotel in London in the lobby. and next thing you know, we are really doing a lot of crazy stuff. So, you know, every organization that's building AI eventually runs into questions about, should we be doing this? And so that's what I want to share with you today, kind of the journey that I've been on, who I am, the oddness that I am the one that's speaking to here. I'm not your typical AI presenter. Normally you would have somebody that has a degree in computer science or data science or physics,

and I will tell you that I have an undergraduate degree in accounting and two degrees in theology. So if that surprises you, I think that it should encourage you because the reality is for the first time in a long time, maybe the first time ever, normal people can build incredible things. And I've been so excited to see how our organization has embraced using AI in ways that I wouldn't even have dreamed we could do three years ago. So the problem is what's responsible? And so the question we need to ask a lot of people is, is this the right thing to do? The responsibility question is the one that builds trust. It's the question in an organization that's 181 years old, that if you show that you're building trust and you're trying to build trust, they're a lot more open to changing things. So almost four years ago, I'll tell you how we got started. almost four years ago before I sat down with Dan in London. I had another group of people in London that I was meeting with,

and we were looking at what we believed about AI. One of the guys on my team had just finished his PhD in digital ethics and computer ethics, and AI was not really as popular of a topic then. But we sat down and said, let's talk about what we believe about AI, and what will we do and what will we not do. Before we ever write one line of code, Let's come up with a charter about what we believe AI is. And we did that. We spent a couple of days with eight people, and we hammered out this agreement to what we would do with AI. And I can look back to that time as something that's impacted everything that our team has done since that time. Because it allows us to know what is possible and what we're going to say no to without even having to think about it. And there's been multiple times that we've come across issues that people have asked us to do, and we've been able to say, no, we don't think that's what AI should be doing. So the conversation that most organizations skip is just because we can do something doesn't mean we should do something.

And so we look at really hard what we believe about AI. Let me just give you some examples of things that we said we would do and what we wouldn't do. The easy things would be drafting and editing, summarizing and research. These are things that all of us do. Translation, we use a lot of translation tools. We do a lot of really good work with translation, saving a lot of money in translation. Training materials and learning support and content. We create code and we always have humans in the loop when we write code. The timing to ship new products has gone from years with lots of contractors that don't really understand what we need to be able to ship it ourselves and prototype in a day and ship within months really good product. So the things that we said no to for us were really important was AI presenting itself as a human in ministry. But I would just say anytime somebody is expecting to be thinking they're talking to a human, we don't ever want to misrepresent AI as speaking for humans.

Final decisions where people, about people that would be made solely by AI. We don't ever want to see that. Monitoring people without their knowledge, that's happening all around us. We don't think that's something we should be involved in. Autonomous actions in high risk and high stakes context. We have people serving all over the world, and we want to make sure that we're not giving AI the autonomy to make decisions about that. But then finally, probably the one that we've said no to the most is handing what I would consider soul care over to an algorithm. And for us, from a Christian standpoint, I don't want AI to be taking on pastoral responsibilities. I don't want it to do counseling because I really think that people need to be interacting with other people. We use AI a lot to help screen things and to help encourage people. But when it comes down to counseling people and sharing with people, we really want that to be a human. I like to see AI as actually freeing me up to do more of that, not replacing me in doing that.

So I want to why I lean so hard into this is that we drew the boundaries before we wrote the code. And I think that if you've not thought through what you believe about AI, the ontology of AI, what you would believe about how you treat AI and what you let it do, I challenge you to stop writing code, stop building, and at least spend some time with some people and think about what you're going to do and what you believe about AI. I don't think a lot of people are doing that. I don't think Sam Altman spends much time thinking about this question of what we should be doing with AI. I don't think Elon Musk thinks about that. I think we need to be leading out in helping do that. So the soul care question, like the pastoral care, the prayer, the counseling, discipleship, deep personal relationships, crises, dealing with people that are dealing with a crisis. We want people to be dealing with those. AI can prepare the support, but it can't really replace the person. So we had an ask come to me and somebody, one of the teams asked me, hey, we have this urgent need.

We have people who really need counseling, but we don't have counselors. There's no way to get them counseling. Can you help us build an AI chatbot that would basically become their counselor? And I didn't have to think long and hard about it because we'd already made that decision before. No, we can't help you with that. Let me share with you the charter, our AI charter about what we believe about AI. And I shared that with them, and they still pushed back and said, you don't understand. The need is so great. Something is better than nothing. And my response was, something can be worse than nothing if you're doing the wrong things. And so we said no, disagreed. They may still be doing that, but I'm not helping them pull that off. So I would just challenge you to think through what it is that you're going to say yes and no to as you're doing that. So the no is easy because the no asked long before that person had asked me to help them. I'll just confess to you, and Dan knows me, like I am famous for hopping on an airplane and building a solution from one place to another.

And so the idea of building a chatbot as a counselor was really intriguing to me. And like it would be easy for me to go home that night while we're watching Netflix and prototype it. And then the next day send it and say, hey, we've built it. What do you think? But because we had talked about this beforehand, I was able to say, no, this is not something we're going to do. Second case study. So one of the surprising tools that we built last year, it's been out for about a year and a half. It's got around 252.5 million users in about a year and a half. We've not had any paid advertising for it. It's just word of mouth. It's something called FaithBot. You can find it at faithbot.tools. So we have several tools that we've built for the church on this website, faithbot.tools. But the faithbot was a sort of like my attempt to create a Christian, a chatbot that would always be grounded in faith. Like it's going to answer your questions from a faith perspective. And so it's going to be grounded in things that are important to Christian faith. And so we built this chatbot and it's just taken off.

Like it's amazing to see the growth. It's in 50 languages. So we've had a couple of versions of it before. The third version I built on a flight from Norita, no, from Denver to Norita. I got on the plane thinking, this is what I'm going to build. I'm not going to sleep. I landed in Norita, and before I landed, it was live. It was pushed to the droplet, live, published, and people were swapped over from the old FaithBot to the new FaithBot. The crazy thing is this thing has the ability to do so much. So we had to make some decisions early on. And one of the examples would be, I don't want Faithbot to pray for people. The first time I was playing with it on the plane, the new version that I built, and by the way, there's 51 languages, and each of them use a different LLM. They all have different markdowns. They all are built uniquely with different worldviews. So it's not just one thing that just translates. It's like 51 tools built into one tool, and it knows where you're entering from, and it automatically loads in that language and worldview. So it's really cool.

But I was playing with it, and it offered to pray for me. And I thought, well, that's cool. But then I realized that's kind of creepy. Like, AI can't pray. Why am I letting AI pray? And so then I spent the next hour trying to unprogram the ability for it to pray and making it where it couldn't do that. So AI is not really praying. It's just kind of mimicking the training data that we gave it. The decision point number two was we also don want people just to find all of their needs in an algorithm So we want to point people to people And so we intentionally when you use it it will try to connect you to someone else, particularly if you're hitting it with a crisis. So one of the things we learned early on is a lot of people who use this are people who are struggling, people who are being abused, people who are self-harm thinking about killing themselves. And it really was shocking to me to read some of the – because they were flagged, and I would go back and make sure the AI was responding well. I don't know who they are, but I get, like, the data to make sure the AI is responding well.

So we built it so that when that happens, it's always going to recommend try to get you in touch with somebody else, and it's going to try to connect you to a help. And if you don't know somebody, it can recommend services and crisis centers that it can point people to. I'm also notified of anything that is critical, and I can hop into the conversation and take over for the AI. That was very possible when we had 5,000 users. It is not possible when you have 2.5 million users. So we do a lot of checking. The other things are lines that it won't cross. So I would say this is for everything we build. We let it know that there are certain things that you can't do. So we don't want this to ever be hateful or coercive or abusive. So we want to build our tools in ways that are edifying and helping people. It doesn't also replace – it's not going to give you a medical diagnosis. I'm shocked at the number of people who go there and ask it for financial advice from certain parts of the world.

And so we don't give financial advice. We just programmed it to say, hey, stick to what you're good at. Don't try to do anything you can't or you shouldn't. And I hate to have to say this, but if your pastor is going to AI to help you write a sermon, you might want to find a better church. But there's a lot of people, as I visit with people and I teach in universities, I find that this is more common than I was realizing. So it'll actually explicitly not do that. It'll actually kind of lecture you on how you shouldn't be doing that. And then finally, it's not for sale. So we're not trying to make money off of this. We don't charge for these products. We do this through people who love it, they like it, and they say, hey, I'd like to sponsor it. That's what we've been able to do. It's also meant we had to keep rebuilding to make it more and more economical. We're careful with the data. Originally, I didn't archive anything. It was just gone once you log out. But a lot of people want to go back because they're accustomed to chat GPT, and they want to go back and they pick up their old chats.

So we built it out where you can sign in if you want to, but you don't have to sign in. You can always be anonymous. and you can just keep that anonymous by choice. So last few things here. I would say that you should evaluate against trust, not just feasibility. So when you're choosing what you're going to do, you're going to evaluate how is this going to be received by the people you're working with. So a lot of how we've been able to build trust in a very conservative, very old, very risk and change of organization is we've proven time and time again to make really good, trustworthy decisions. And so we've worked hard at building trust within our organization. So whether or not that's you working within your own organization or if you're selling AI products to other people, it's really important to build that trust with people. So I'm going to leave you with two questions. First is, as you're looking at building AI and you're asking yourself, what are the things that we're going to do and what we do and what we not do? Always start with how might we? So don't start with there's two types of people.

Some people are like 50 reasons why you can't do something. I'm probably the other guy who gives you 52 reasons why you can do something. Always be that person. Even if 49 of the 52 things don't work, be positive and try to do things. And then the last thing I would leave you with is what will we never do? So make that decision early on. These are the things that we're just not going to do. I really don't care if ChatGPT offers this new feature or if Grok offers this new. We're not going to go there because we don't believe that this is what we should be using AI for. So finally, I've actually found these limits. These clear limits are actually freeing me up to be able to do way more than I could do before. I have a guideline set that says these are the things that we will and we will not do. And as long as I stay within those boundaries, it's actually way more freeing than it was whenever I was always wondering, should we do this? Could we do this? Thanks. There we go. Thank you so much, Donnie. That was a great start to things. I can attest to the fact that, well, one, Donnie is on a lot of flights, more than I can imagine.

But every time there's a flight, I get a cool message like, hey, check out what I built, which is really exciting. So that was a great way to start out today. I'm thrilled to introduce our next speaker, Mark Lay. He's vice president and CIO at Noblis, a science and technology organization serving the federal government, where he leads IT strategy and delivery and keeps infrastructure secure and resilient. And I've had the privilege of interacting with a bunch of people at Noblis and partnering with them in certain respects. And it was great to be in a room with Mark at a whiteboard. That's my favorite thing. asks amazing questions, and he's going to talk to us about how you can orient yourself as AI is moving fast and what are the actual decisions you need to make as we're all in this environment where AI is moving faster than we can keep up. I'm thrilled that Mark's here to talk about that.

So welcome, Mark. Thank you. I really love the leash concept. It's amazing. And as a CIO, I want to take it up on the level. I have four decisions that I'd like to go through. And it really starts with when I talk to fellow CIOs and CTOs, we all say AI is moving at the speed of light. Every time we turn around, it's different. It's evolving. I meet with a vendor one month. Their pitch is different the next month. The features, they're all better, but they're different. And so from a CIO perspective, that's really difficult to manage. And I think it's difficult because traditionally we've had a business problem, process functions, and codified how to address it. And now we're in a situation where we are turning over to technology an objective. We're putting guardrails or a leash around it and having it go off and produce an outcome.

And that's a scary thing, and it totally changes, I believe, the way you need to look at it. And so what I'd like to go through is four questions or decisions that I think are really critical to sort of at a NUCs level above some of the talks earlier, just how you should look at AI across the organization. The first decision is really about what should I delegate to AI? And my premise here is this idea of turning over activities to an automated system is very similar to what we've done for many, many years, and that is manage people. We have to delegate activities to people. And my premise is when you're in front of ChatGPT or another tool or you're trying to decide to build a system, you need to decide what you're going to delegate to that AI and how much of a leash you're going to put on it.

And I really think there are two dimensions of that. There's what reward you're going to get out of it and what risk. On the reward side, you're trying to get something better or something more efficient. And if you don't get either of those, you really shouldn't put it in AI. And then the question is, on the risk side, is there a risk of an error? And what's that risk of an error? Or is there a low risk of an error? And what's the consequence of that error? And so really, if you take this a step further, you can look at it and say, if there's better outcomes and less burden to do it, that's a clear gain. But then the next question is, is there risk in getting that gain? And how do you address it? If there's no risk, like when I go online and explore things, that's down in the lower left side. The upper right side is an area where you might have to put guardrails very tightly around something. And earlier, Heidi made a point about some guardrails are deterministic guardrails.

I think deterministic guardrails fall up in that left lower quadrant where you have high risk of something happening. So the first idea is look at everything as you're delegating, just like you would delegate to one of your workers. They can make mistakes, but you want to control those mistakes and you want to keep them in bounds. You want to put a leash on what you're delegating. And I find this an easy framework to sort of look at how much of a lease you have to put on it and really where you should put the AI and where you're going to get the most value. The second, the next area is really the idea of who's accountable when AI acts. And this is an area where I think a lot of organizations have oversimplified this. And the first premise is that accountability can't be delegated. I mean, humans can delegate accountability to other humans, but with AI, we should not be delegating accountability.

So somewhere up the chain in the RACI chart for the A, there always needs to be a person. And there are four areas where I think this is important. It sort of covers the landscape. The first is protecting the data. And one of the things that we've done in our organization, because we deal with a lot of compliance from the federal government, different agencies, and security, we actually spent the last year identifying over 1,000 data sets that we thought were relevant. And I don't think we got them all, but we started. And then we identified a data steward, a person who owns that, okay, and is responsible for understanding what the data is, what the quality is, with the timeliness of it, and also how does it have to be protected, who can see it, who can't see it. If you don't have that, I mean, and one of the things that came out of this is even people that you know, they know they own the data, our HR department or their finance department,

they were so traditionally programmed to say, well, the data's in the accounting system or the data's in the HR system, it protects it, okay? As you go to agentic AI and these tools, you can't live in that type of world anymore. You have to have somebody who owns the data and knows the data. In the end, after a year of work there, we got down to about 115, 120 data sets. And these are not data files. These are data sets of information, tabular or even reports that were critical to our operations of our business. And we had about 30 data stewards of those. But that was the first lane. The second is this whole concept of validating the output. I mean, you can put guardrails on it, but depending on what you've delegated and the risk of error, there a whole activity of data output quality And I think when you look at is it a platform or it a centralized tool who checking that data quality

And sort of going back to the platform versus the enterprise system, I like to use the analogy of we've been living with this for many, many years. Somebody earlier said Lotus 1-2-3, which I'm old enough to remember. But Excel is a platform that's been deployed to all our employees and finance teams. And it really, to me, it's the equivalent of AI. It allowed before Lotus 1-2-3 and Excel, there weren't tools to manipulate numbers or math the way we can. That changed the world. I see AI doing the same thing in terms of pushing it out. But if you think about Excel as opposed to a centralized Power BI tool where you have structured reports, when you give the finance team an Excel spreadsheet, they can now build macros. They can build traditional things that they only know and only their teams know how to run.

So who controls that? Who manages the output? You can't centrally manage that decentralized resource. Second. Third, which I think Yvonne also mentioned, is monitor the behavior of the agents. You've got to watch them, which is a totally new concept. But from managing employees' work, we already know how to do that. It's like, how do you take that principle that we've used for years and think of it now in an automated fashion and where it would apply? And then the last area, which sort of comes back to my Excel example, is owning the outcomes. Okay. Who's accountable for the outcomes? Okay. We have teams where we have a contracts team where we have 20 contracts administrators. One of the early career contact administrators built an app to pre-process all. He went and got everybody's knowledge from the 20-, 30-year contracts administrator and built this system that would do a first task on the contracts.

We didn't do that in the IT department. They did it in their group. That has to be managed and maintained. How do you keep that going? So there is that business ownership side of it and keeping track of that. Again, I like to use the Excel analogy, at least with the finance department, they understand they own the macros. The last area, which is another area that I don't think a lot of people are talking about, but CIOs worry about it a lot because it's costly. The question is, what technical debt are you creating? And there's a whole new different type of technical debt. And the first layer of it comes from centralized or distributed platform. and what I was already talking about with regards to departments building their own applications. That's the first piece of it. But the second piece of it, and I've had experience this, we rolled out certain tools where we let teams go off and build apps and share apps.

Well, guess what? The vendor changed the tool such that we went to the next version of it, and none of the old apps converted. Okay, so there's a whole level of when you're rolling out a technology, whether it's a control plane layer tool, whether it's just a user interface app or whatever in the technical stack, the technical stack is still evolving. And the data you're using is evolving. So there's a lot more to worry about in terms of what dependencies related to data or other systems or other things in the value chain of delivering that. And then the second piece of it is once you know that, can you reverse it? What's the cost of reversing it? That actual cost, time, whatever. And then the last piece is, which is I think every organization is starting to hit, is do you have the capacity to evolve?

I mean, a lot of people have talked about the idea of or the cost of tokens. If I build something, is it going to be too expensive for the value I'm getting out of it? But the other point is, let's say the technology changes and I need to put something else in place, either that centralized IT team or that decentralized group that's leveraged. Do they have the capacity to keep running what they deployed? These are probably the hardest questions to answer. in a fast-moving environment. So those are the four I have. I just want to close on this. It's sort of an adaptation of Drucker's management theme, which is AI can do a lot of stuff. There's nothing there, and it can produce a lot of output. But it's really how you choose and address the four decisions before of whether it's worth producing and the cost of that.

So thank you. Thank you so much, Mark. Appreciate it. Calling an AI model or creating your first agent to do a basic task can be really easy with modern tools, but it's a whole other thing to create an agent that's tied into all of the different systems that it needs to access in order to deliver real ROI over time in your organization. That's why I love what SERP API is doing and why I've used it in the past to connect valuable resources into my agents. SERP API handles the complexity and delivers search results as structured data that your application can use, whether that's from Google, Bing, Amazon, Walmart, YouTube, or hundreds of other APIs. They also have built-in enterprise security and compliance. They're SOC 2 Type 2, SOC 3, and ISO 27001 certified, making it a strong option for organizations where security and compliance matter.

I encourage you to visit surfapi.com slash practical AI to get started with Surf API for free and get 250 searches every month. That's surfapi.com slash practical AI. So next we have, I'm excited to hear this kind of story and journey from Darren at AeroCore. Darren is a senior data analytics manager at AeroCore Technologies, which is local here to our surroundings, but has this incredible technology to perform washes on jet engines all around the world. So they do this all around the world. And so what's been interesting to see is how Darren has been able to practically roll out agents into their organization, which a lot of the organization isn't building these agents,

but certainly benefits from the insights that these agents are able to put together from this, you know, wash data and work orders and other things from all over the world. So this is super practical, and I would encourage you both to, you know, of course, listen, but connect with Darren after his talk as well to follow up on that really practical journey. So welcome, Darren. In that regard, yeah. So my name is Darren Talley. I am the Senior Data Analytics Manager for a company called AeroCore Technologies. As Daniel said, we wash jet engines. We are an incredibly analog company in a digital world. There are no digital components on the wash trucks whatsoever. So my job is to bring some of the digital world to our company. And as Daniel so eloquently said, we are all over the world, right? So we have about 11 domestic sites at the major airports here in the U.S., started deploying internationally and overseas, sites in Japan, in Riyadh, and in Brazil.

That expansion is growing every year as our technology really grows to help the big airlines, especially in a time where fuel prices are going up. No one can deny that. And so that means your airline tickets are going up to reflect that, and we're trying to counteract that as best as we can. To give a quick bit, since this is a very niche thing that we do, again, we wash jet engines. We use a patented nucleated foam technology to penetrate and clean the insides of jet engines. So you've got a lot of carbon buildup, a lot of fouling that happens inside these engines. Our job is to go in, multiple cycles go in and clean that stuff out. Where that extends to what I do, I look at the before and after engine performance data, determine the efficacy of the wash itself, but looking also into long-term trends, looking into predictive maintenance type things, trying to find the bad apples that are out there for fleets of aircraft and determine how we can help the major operators. Like Daniel said, this is a very practical presentation.

This is what I have done over the last year. I think these kind of presentations fit in this kind of format very well. You have a ton of people who talk about how and the safeguards and how important all of those things are. But I think sometimes you get left coming out of here being like, well, what can I do? How do I apply this to my company? What do I take home? So I'm going to give you two very, very clear examples of what myself and my team did in order to bring using AI to bring better information and better data to the rest of our team for decision making. So the first one we're going to talk about, so the biggest challenge we have is access to data. As I said, we are an analog company in the digital world. How do we get the information to the key decision makers in time for them to make those decisions? The other one being work order verification. I'll talk about that one a lot more. That's a very nuanced problem, but I think it's probably a little more interesting in the first part. So access to data. Does anybody still know write SQL on like a daily basis?

Yeah, that's actually more than I expected. So, again, analog company, digital world, not a lot of people know how to write SQL. Most people don't have access to the database intentionally as deletes happen a little too easily these days. But then that leads to the analyst bottleneck, as we like to call it, which is the request comes to me. I have to go in, pull the data, merge the data, analyze the data, send the results. And we do that for 40 hours a week, and we've done no long-term growth. We've done no extra tasks. We've done no research. All we've done is fill out how many washes did we do in Atlanta last night, 55 times for, you know, different employees. And then that also leads to another lovely coin term, which is workaround sprawl, which essentially is if I, because, again, as you didn't hear in my title, not developer, not SQL person. So all of this turns into pandas.readcsv with group buys and location filters, right?

So those get very complicated. You look into my Jupyter notebooks and you see cells and cells and cells of just every random pull that I had to do that week. Go through the end of the week, determine what's important, delete everything else. That just gets spread around. No one else has access to that. It's not useful to anybody. So those are kind of all the challenges that we particularly faced in our data. To mention also our data analysis group consists of me and one other employee. So here comes Daniel with his big toy And first one so Agent Forge was really a big key for us to be able to deploy some of these tools out there right So the first one all right well if I just give this file to the agent, it probably can tell me all the answers about all the work orders from the last week, month, year. Okay, well, then there's MCP servers. Okay, cool. So like now I can just query the database directly. Well, you still have to go into Agent Forge. You still have to go in and pull the information out. No one likes an extra login. You know, that's really hard. Boom. Now you can put it in Teams. And I think my phone's blown up three or four times already

with guys using the agent that we put into Teams in order to access the information. So that ability to just ask, very simple text query, you know, how many, I forget which one I put up there. Oh yeah, return me all the work orders for this particular tail number. There was an issue, something happened. All right, here's all the information. Here's the actual query, here's the results. You know, fantastic for our team to be able to go and find that history, right? History is always the most important part. We've gone through so many different systems, be able to see all the work orders that are on that aircraft, when they were, who did them, where they were, and then obviously all of the technical data that we gather from every single wash. Once they got moved into teams, very high adoption, everybody's using it all the way through and it's getting a lot, a lot of hits and seeing that in the analytics. And it's been great. It's truly relieved to, you know, burden off of our analysis team to do more things that are beneficial to the company. You know, things that used to take 30 minutes, an hour, a week for complex stuff.

The system now does in a couple of minutes. If you gave it a really complicated one, you know, this came back in about three seconds. So it does a really, really, really good job of getting the decision makers the information they need. So, yeah, the big one, self-service, right, getting our decision makers that information that they particularly needed, getting leadership to not have to ping me every time they need an answer to a question, and also kind of freeing up myself and freeing up my other analysts to be able to do these bigger projects, different types, you know, bigger reporting, bigger research projects, bigger experiments through our data and how important all of that is, but we weren't really able to find the time to do it, And now we're able to go do that. So that's kind of on the simpler end. I know that there's a lot of the ability for people just to kind of connect Claude directly to your data in all those different ways. We talk a lot about safeguards and guardrails and like that. We thought that building it through AgentForge with the MCP servers provides those guardrails, also giving you the access that is wholly required.

And now kind of the more interesting one. So this is what we call work order verification, right? So hopefully most of you have not had to spend a night in an airport. But if you ever have, that's when all the maintenance happens. So the guys are out there on every plane doing a whole ton of work, including our team who work shifts from about 7 p.m. till about 5 a.m. doing these engine washes out on an aircraft. We do anywhere between, let's say, 15 to 30 washes in a night across the very different sites. You can get about six washes in a night at a particular site. Typically, it is one technician out there doing the wash on a particular engine. he's out there with his phone trying to get all the data that we need him to gather, writing down all the information that he needs to collect, also making sure that he pumps the right amount of fluid into the engine to wash it. There's no one out there to validate his inputs, his or her inputs, sorry. So errors happen. Fat fingering happens. In the middle of the night, 5087 looks a lot like 5078. It happens.

So how can, but when that comes back at the end of the month, well, we don't have the right billing counts. We don't have the right aircraft types. We don't have the right numbers. What happened there? We have had a long system of using these pictures that the operators have to submit every night, which are the first three kind of shown there on the top picture on the right, which would be a picture of a whiteboard that we've built on the side that acts as a pseudo form that the operators have to fill out, plus taking a picture of the aircraft and then taking some other pictures of things of the – we call it effluent, but it's the crud that comes out the bottom of the engine is that last picture. I'll love to talk about that with you afterwards if you're interested. But our real key is, you know, verifying remotely the work order and the wash gut every night. So, right, our challenge, and here's what we came up with for that solution. The other side of this would be that we do have a quality control plan in place for someone else to verify the work orders based on the pictures, looking at them remotely, other site leads from other spots, other professionals.

We kind of kept noticing that the same mistakes kept being made over and over. The scenario I just described where somebody looks at a 5078 and thinks that it's 5087, logs it in. That would be flagged when 5087 comes in the next night. And you're like, well, we just did this one. Why is it coming in again? Oh, that's a repeat work order. Okay, delete. Come to the end of the month. No, that one was real. The other one was wrong. How do we fix that? So this is where a vision model came into place. luckily because of our use in Agent Forge and some of the work with Daniel and big thanks to Sharon who really helped me build this and deploy it. So the crews take those photos, they upload them in the Power App that we use for them to generate the particular work order. Every day at 8.15 in the morning, Agent goes in, scans all the pictures, verifies it against the information put into the app. You know, the best part of this would be those end numbers are, you know, are pretty big written on the side of the aircraft, Vision Model does pretty great at returning them. It also actually does pretty great at returning that little tiny number in the top of

the tail when we needed to do that as well. So Vision Model goes in, looks at the aircraft, identifies it, returns it. If it aligns with what is input in the app, checks it off. If it doesn't, it returns a flag. That flag gets sent to me and says, hey, something's wrong with this work order. Please go have somebody, a human, go look at it and see what's going on here. Sometimes it's the wrong picture. Sometimes it's a cloud or a dark night, a bad picture or something like that. Compare and flag. That's the exact prompt that I wrote. Sharon did help me with that as well. But just kind of back to the, you know, how many times I had to go through and simplify that prompt to make sure that it didn't say, well, I think this aircraft is blah, blah, blah. I said, no, you need to return N and these numbers and that's it. And how many times we went back and forth fighting about that to make sure that it was correct. But the results. So this was wildly successful. So successful that it has been, it goes out to leadership every day at noon with a list of all the work orders done in the previous 24 hours across every site,

both commercial and in the military. It also then flag, was expanded to flag other abnormal procedures, other text queer or text inputs that the operators put into the app to say, okay, well, I had this issue, we worked around it, we fixed it, but just so that everybody knows, we had this issue during the wash. Or things like we did this for a smoke and fume resolution that the operator asked us to do. Or we missed a cycle. Or we missed a wash. All that information gathered. Agent also now goes through those notes, flags them for anything related to smoke, fume events, flags them for anything related to abnormal procedures, and then goes through and does a whole bunch of other checks, right? We found that the same things were being missed over and over again by the human QC going through and checking these work orders. You know, we work across the middle of the night. There's weird things with what happens when work orders cross midnight and all of a sudden work orders start taking 36 hours and you're like, well, no, that's probably not true. It was probably four hours. It just got crossed over midnight. How did that happen? So checking

all of those type things. So again, like I said, wildly successful, expanded to all work orders. flags are currently just set and tracked, and the team goes through and resolves them as needed. We are building in a resolution system as we speak, so those will actually be flags that need resolution by a human to go through and check and either identify that they are real, fix the problem, or say that they weren't really a problem at all. The biggest result of this, I think, would be an op, as we talked about in the last presentation, which was fantastic, where your AI agents fit. And this was a big workload reduction and a better outcome in both of these cases. Typically, at the end of the month, we will have a big meeting between finance, data, operations to say, okay, let's make sure we got all the wash counts right. Let's make sure that every work order is what we thought it was. All the billing goes out to our customers correctly. We canceled that meeting this month. There were no incorrect work orders. There was no reason to have to sit and go through everything.

We were talking about five, six person hours to go through that meeting. And instead, we kind of just went off and did what we should be doing, development, strategy, looking into the future. So in summary, again, a very practical intention of what this presentation was. We built a text-to-SQL agent in order for leadership to have better access to our data, also to alleviate the load off of our analysts of answering every simple question that came through. That also allowed for, you know, better context for our leadership, also better time for the analysts to go do more strategic tasks. Our vision verification has been wildly successful in reducing the amount of work required for everybody to go in and fix work orders. And then the daily digest that comes from that, right? Here's the track. Here's every work order. Here's every issue we had last night. What needs to be fixed? What doesn't need to be fixed? What can we do better? upcoming project this is fun so obviously making the operators making sure that all the information

that they put in is correct so looking into more visual models of gathering that data by itself maybe not requiring the operator to put as many things into the app as required uh or not doing it at all and say can i gather all the information from the truck from the pictures uh from other methods and then the other one being the let me show this vision of truck valve temperature pressure settings. So again, like I said, analog in a digital world. There's no computer on that truck that washes the engines. It is a guy out there, mechanical ball valves and switches. That's it. So we want to make sure that the right settings are getting put in. But again, don't want to complicate the truck. So can we use AI vision in order to ensure that all the right settings are put into the truck every night? Thank you. Thanks, Daniel. Thanks, everybody. We've had a wonderful journey in the AI space, specifically with the Prediction Guard team and for me and my team. So just a thanks to you. Thanks, Darren. Yeah, that was great. So, so wonderful for all of

you. I mean, we've got a lot of practical folks here. You know what it takes to get something from idea into daily use. And Darren's done that and got that into daily use in their team's environment. It's being utilized. The value is realized. This is the MIT study. It's like 95% of AI pilots fail. Turns out we just need more Darren's, I think. All right. That's our show for this week. If you haven't checked out our website, head to practicalai.fm and be sure to connect with us on LinkedIn, X, or Blue Sky. You'll see us posting insights related to the latest AI developments, and we would love for you to join the conversation. Thanks to our partner, Prediction Guard, for providing operational support for the show. Check them out at predictionguard.com. Also, thanks to Breakmaster Cylinder for the beats, and to you for listening. That's all for now, but you'll hear from us again next week.