AIから「価値を生む仕組み」をつくる ― Deloitte USのAI責任者 Jim Rowan が語る企業AIの壁

Generative AI is transforming industries at an unprecedented pace. But as AI changes how you or your team work, one thing is clear. Your skills also need to evolve. At DataCamp, we offer everything you or your team need to adapt and thrive with AI. Whether it's business users looking to get the most out of ChatGPT and Copilot, or developers and data scientists looking to fine-tune models, you can learn the entire AI skills spectrum on DataCamp. Power your AI transformation today. Start learning at datacamp.com The tools are how you get the work done, but it's got to start with a mindset that I'm going to come in and we're going to change how this work gets done. And it might actually mean that we've deleted five steps. I see a lot of times solutions just hit a roadblock running really fast and then it's, whoops, we never really got the right teams involved from a governance perspective. And that slows down the whole process. Almost all business leaders are talking a big game about how AI is reinventing their business.

But privately, a lot of them worry that their company is falling behind. Enterprise AI is tricky from getting people to start using the tools to getting AI features into production and a thousand things in between. We're going to be managing teams of agents before we know it. Some people already are. Well, now the outcome is data that feeds to an agent that can actually take action with a customer. I spoke to Jim Rowan, who is head of AI at Deloitte US. Jim helps Deloitte's clients achieve their AI ambitions, and he's an author of Deloitte's recent State of AI in the Enterprise report, which provides a good representation of what's really happening in enterprise AI. So, let's find out how to get value from AI in the enterprise. Hi Jim, welcome to the show. Richie, thanks for having me, appreciate it. Yeah, great to have you here. Now, I'd like to start off by talking about the Deloitte state of AI and enterprise approach, you're one of the authors of. So, do you want to tell me, like, what was the most surprising finding from that work? Yeah, well, Richard, we ran the survey. We ran it across about 4,000 different organizations, leaders globally.

So, it gives a good breadth of kind of what we see in the marketplace. And the main headline continues to be the organizations are struggling to adopt AI. Like, they talk a lot about how they want to continue to scale AI solutions. but we're seeing that the actual implementation of those scaled AI solutions are falling behind what I think we would want to see as AI leaders in this space. And so that's one of the big challenges that keeps coming up. There's also a bunch of really interesting data, too, that focus on, you know, how organizations are thinking about their people and their workforce. And fascinatingly, a lot of folks are focusing on the tech because, right, with AI, everyone talks about the tech a lot, and they're not spending time talking about their people and the change management processes and things that need to really happen. So we're seeing a big link between getting ROI from AI, organizations struggling with that, and you see in the data kind of a focus that's being missed on sort of the people and change management aspects that we've known about for transformation for years. So those are some of the key headlines. Okay, yeah, certainly like buying licenses for AI tools is kind of much easier than changing all your processes, making sure all the people are doing what they should be doing.

So we're definitely getting to that side of things throughout this conversation. but I feel like basically every leader's gone okay we're all in on AI for like several years now but you mentioned there's still problems in a lot of enterprises with making like getting value from AI so what do you think the biggest challenges are here? Yeah I mean there are a lot of challenges and I think it's not that we're not optimistic about what AI can do it's that the excitement of AI meets the reality of enterprise systems enterprise processes large businesses with all sorts of different competing priorities and so a couple things like you really start to tip the scale. Number one is that we see organizations that are investing heavy in the technology, so they're building out a really strong kind of footprint, but they're not maybe investing enough on the data side. So AI without data, you're not really going to get governed, controlled, accurate results that you need. So that's a big gap. So when you've invested in AI, some of them are coming back and saying, well, okay, now we need to rebuild the data foundation or we need to fix the master data management issues that we have for that. So that's number one. Number two, we're seeing kind of process issues, both in the sense that they've applied AI for a

POC, let's say, to a business process. That process might have five steps in it. They've applied AI to the first two and maybe the fifth, but the middle activities still aren't getting automated. And so the end-to-end process isn't actually changing, or in worst cases, it's speeding up to find a new bottleneck in the process. So that's also slowing down implementation because maybe we skipped over the two hard steps in the process, but then you didn't really make the change that you go. And then the third piece, and it's kind of hitting on this in the survey data too, is that the people in change management aspect of this is huge. I mean, there's a lot of feelings out there about AI in the world, pro and con against what's happening. And so you've got to take your people through, like, what is your AI strategy? What does it mean for our workforce and the enterprise to get people to drive the adoption of these solutions? And so I think enterprises are realizing that's not an overnight activity. That's a time in front of tools, a time in front of the people, a consistent four-star message. So those are just a couple of them that are out there but we're still seeing on the flip side richie like organizations are being successful

with ai they're able to implement these workflows they're overcoming coming those barriers it's just that you're seeing more people stumble along the way because of some of these challenges that we're running into okay yeah i mean sounds like uh lots of uh lots of big challenges there but yeah yeah i think you're right it this is like a it's an ultra marathon not a sprint you know 100 percent an ultra iron man how about that like because you gotta do a lot of other things like it's not just the running piece, right? It's the swimming, it's the biking, all those types of things. So I think that's a good analogy. Absolutely. All right. Yeah. So I think certainly the data side of things is incredibly important. AI has had a lot of hype, but maybe the data side sometimes get left behind. So do you want to talk us through like what needs to happen there? Yeah, for sure. So we see this too, like with a lot of our clients in our deployments, and even as we looked at deploying AI internally for our people, is that AI in a chatbot by itself is one thing. And sure, you can prompt it and you can give it context. But if it's not grounded in the way you work or in the data that you have, it's really hard to get those answers out. And so what I think AI is starting to do is

provide a lot of focus and attention back in the data layer that asks the question of, well, do we have a consistent definition of customer? Do we have a consistent definition of product? If we're doing financial reporting, are we rolling up the financials in the same way across all of our businesses? Are there different unique models that are in there? And so organizations are investing heavily in building that data foundation, establishing the controls and pipelines that are associated with it. They're investing on the master data management layers that are needed to drive this too. And ultimately, the control points around as these tools move from just basically, you know, kind of chat body experiences to actual agents, now thinking about the security, governance, and controls of not just to read into those data solutions, but being able to write or log those transactions and really be more dynamic. So it's not only something that people have invested in just to get started with, but they have to keep investing in to meet the next sort of wave of demand coming from these AI agents. Who has to implement that? Because a lot of the stuff you talked about is not the analytics side of things.

It's the governance, it's data management, it's security, it's things like that. So is this a shift in what the data team does, or is it like you want different people running this? Yeah, I mean, I think it's a shift in what the data teams do. It's really interesting when you look at sort of the IT organizations in general, we're seeing a lot of shifts in just in terms of how those teams are set up now. I think you're seeing roles getting collapsed across places where you might have a data engineer and an AI engineer and a testing team member and a product owner and a product manager and like all these different roles that are coming together. And we're seeing the organization shift to be actually have like more kind of individual contributor types type swim lanes that own, you know, full data products end to end or something like that. So I do think the data team owns a lot of this, but you can't do it without the business support and governance. Like a lot of this is working with the business, understanding what their requirements are. It's classic core things that we've had to do for system design. We just can't ignore those. What I am seeing that's changing a little bit was that I would say maybe five or ten years ago, we were big into setting up enterprise data lakes, data warehouses, all those terms, right?

We've been on this journey for a while. Now I'm seeing that as we think about that, we're working more towards this data product idea where we have a very defined product that we're going to connect to our AI so that at least we can sort of narrow the scope. We're not trying to do the entire solution at once, but let's figure out what data do we need to support what AI solution that we've got out there. The other thing that we're seeing is a big investment in sort of this semantic layer, common understanding layer, context across this enterprise data. The good news is that AI can be really helpful with some of these things. AI can run through your data warehouses to help pull some of this information together, can give you feedback on it, can support the data engineers in building and writing code and testing it. So the cool thing, I think, is that now we're actually accelerating some of this data work. But maybe a couple of years ago, we were having a hard time kind of getting the business case organized. And the outcome was the dashboard that was hard to take action on. Well, now the outcome is data that feeds to an agent that can actually take action with a customer. OK, now I'm seeing this end-to-end value case come together. That's a little bit of a longer answer because I want to stitch it together that I think that focus on outcome is allowing people to invest on the data side now a little bit more than maybe what we had to do a couple of years ago.

Absolutely. It does seem incredibly important that you have this business outcome from reworking all the data because it's a tough ask to be like, okay, we just need to rebuild our entire data foundations without having that kind of business case there. Do you want to talk me through how you build the business case? What goes into that? Yeah, like we've seen a lot, like early days in some of the business cases for AI, it was really focused on, I would say, the use case or series of use cases that we were trying to get. And what we saw were a lot of companies saying, well, I need to know the hard dollars associated with this use case. And the challenge with AI over the past couple of years is, well, we actually don't know some of the answers because the technology is evolving super fast for everyone. We're not quite sure where to apply and how well it's going to work. And by the way, what we try three months from now might be better than what we tried three months ago. that makes the business case even more viable. So we try to have organizations think about in sort of two categories. What are the quantitative benefits we're going to drive from the business case? Which makes sense, which is how we've done it in the past. Then also putting in some of the qualitative things that maybe we couldn't do before in the business case

that are really important to add. So like, I might not be able to drive savings from the specific implementation, but I can get access to a new set of customer group that I couldn't do before. Or we've been able to do 100 types of analyses of these scenarios. Now we can do 1,000. And then that's going to create this capability. And I'm feeling it's taking more time for organizations to explain, like, all those different elements to go into the business case. But they're becoming more, I'd say, more rich with some of that data around the qualitative benefits. The other challenge I'm seeing is that, like, you've got to link it back to the business outcome. And some of the outcomes are just everyday efficiency. It's, like, kind of the cost of doing business now in some areas. Like, we have to make some of these investments. And it's very hard to measure the everyday efficiency piece. And so I try to focus teams on what are the bigger process areas that we're really impacting and how does it in that quantitative side fit back to the P&L and the structured actual outcomes. The other piece, too, is you've got to look at this and constantly rework that business case.

And not in a bad way, but in a way that, as I said, the technology is moving. And so you have to come back to it and kind of say, well did it move so much that this business case is different today than it was tomorrow the cost per token might be different now with the solution you had from you know i was gonna say last week to this week but you know last month to this month and that now opens up a whole new set of use cases or makes the business case profitable now so that's a little bit of what we're seeing and takes a lot of effort i think to get these right and constantly manage them absolutely so i love the idea that not everything has to be completely quantitative here you sometimes you're getting a qualitative venture from their business and as long as you can explain that then that's a good thing i think sometimes like as data people we can tend to get stuck on we must have a number here and actually you know you can be a little bit fuzzier with it especially when uh you're moving quickly and maybe one thing which is bad to that too is like the fuzzy part is like let's give some ranges like that was another big struggle for for some companies was like well i'm going to give you a number it's you know 22.6 it's like no let's give you a range from 20 to 30 and we'll refine the range as we learn more about the solution that

we're building that trait takes a little bit of risk for management to agree to some of those things but i think it's the risk we're taking in a sort of like very ambiguous space as we move forward absolutely i mean the other thing you mentioned was that quite often the business case can change within a month just because you know technology's moved on a lot of everything in a large enterprise you think of like uh much longer cycles for projects you know maybe months years is how do you deal with that rapid iteration? Yeah and it a real mindset shift across the organization Like I think this starts at the board level C level You really have to start educating folks on sort of the pace of change that we in And I think a lot of executives are feeling it But then that means that we can execute faster in some of the things that we have. And so it's both that mindset shift, but also looking at our internal processes, too. And I'm not saying we skip steps that are important steps. But there's also things that we can shrink those steps now with the power of some of our AI tools as part of those things. So we can run a series of tests way faster than we could before. So let's make sure we do that.

Let's make sure we still have humans on the loop checking the things that we're doing, but we can run those a lot faster. So I'm seeing a little bit of that from a speed perspective. The other piece is the business, I feel like, is the business and IT partnership on these things to really be agile. And getting people comfortable with these alphas and betas that you release, I think, is really important too. And that's been some of the mindset shift. I've seen some of our clients be really successful defining good-sized pilot groups that are going to release out solutions to them. And they're safe solutions that they're releasing, but they're still releasing an alpha and beta because you know you're not going to get to perfect by the time you go to release it. And so that perfect being the enemy of good kind of equation comes into play a lot in this space, too, for speed. But I've always found, too, that we're not trying to sacrifice the governance and the quality. And one of the things that we've also learned in this is that if you set up the governance before you get started on this and define places where you're comfortable with AI use, how you're going to handle the data, how you're going to secure the information, what guardrails are in place, before you get started, it gives your team a lot more speed to get running.

I see a lot of times solutions just, you know, hit a roadblock running really fast and then it's, whoops, we never really got the right teams involved from a governance perspective. And that slows down the whole process. So there's a lot of elements to kind of go quickly, but it's go quickly with purpose and not in a reckless, you know, fashion, right? Put your helmet on, wear a seatbelt, do all the things you should be doing, but you got to get moving. Absolutely. I mean, yeah, going fast seems to be incredibly important. Now, a lot of the stuff you talk about is like is different connections between teams. You mentioned like connection between IT and business, connection between governance and almost everyone else. Does this require like a different approach to organizational design then? I think it will over time. We have a paper out that we talk a little about, kind of the work chart versus the org chart. It's kind of different. You'll play on words, right? But we've been behind these same enterprise org charts since the railways were set up, I think. Like, there's got to be room for improvement from that to where we are today. The tricky thing is, like, when you try to talk about that, I think people immediately assume that that means there's, like, a jobs issue or something.

It's like, no, it just means how we configure the work differently. And maybe we're all working in a slightly different way than we should in the past. So I do think we're going to see, you know, new roles become part of the solution that we have. We're going to see new ways of working change. And I do think we sort of start to collapse some of these org charts into different areas of work. And I think as individuals working on things, we're going to also, I think, find ourselves doing more with the capabilities that we have. We're going to be managing teams of agents before we know it. Some people already are. They're off doing things while we're doing other things. And so there's time in your day now you need to carve up to check in on the AI tools you have working on your behalf. We didn't do that two years ago. Maybe some people did. I wasn't doing that two years ago. And so now it's like the block time to check in with my agents, you know, and what's actually happening. is a new way of working that's going to be different. And I even think that's going to change, you know, in a few more months, maybe shorter of how these other personal agents working on our behalf interact with someone else's personal agent on their behalf

and change how that works. So I think we have to be ready for some change in that design. I also think this isn't an excuse to have a bunch of shadow IT in the business either. Like, I think people could use this as an excuse for that. I think there's an opportunity to create new roles in different business roles around AI product ownership, new ways to think about forward-deployed engineers and different elements of the business. So those are going to be challenges to our existing org chart. And I just really encourage enterprises to be willing to pilot those in parts of your business to see what's working in your culture because that's the other big part of this too. It's got to fit your culture well. Absolutely. A lot to unpack there. Yes. It's an area that's passionate to me because I feel like we can talk about the tech all we want, but if we don't actually change the process and the org around it, for the better of this stuff, you're kind of just making some faster stuff. And I don't know if that's really the end state for AI. I don't think it is. I think some of the stuff you're talking about is working with different people. It sounds like cross-functional initiatives that have been around for decades.

So what's new here? Are you suggesting getting rid of a hierarchical org chart? Or how is it going to look in the future? Yeah, I mean, I think there's more squads and teams. We've seen a lot in software development where teams working in pod-based constructs with different responsibilities. And so do you still need a functional analyst on some of these things? Totally. But the functional analyst might actually get to a workable design solution way faster than we used to in the past, where the VA might have been doing some work, then handed off to a designer, then handed off to an engineer, then handed off to the tester. That might be one person doing that end-to-end workflow with a peer review. and that's how we check to see the design things are compliant and we're meeting sort of our coding standards, et cetera. So I think that's where some of the things collapse and I think from the business side, you know, working in with IT, you're going to bring a vibe-coded solution into the meeting and say, look, we built this demo. We're kind of thinking this could do this. What do you guys think? Oh, our systems behave a little bit this way. How about we tweak that? I mean, that compressed a month of design and requirements conversations.

It's a very different way of working and I think both sides of the equation, business and IT, you are going to get more familiar with the AI tools. And so we have to kind of meet each other there. And then as you get more senior in different enterprise roles, we might have more responsibilities with different individuals and different org charts. Like I think it changes a little bit around that. If I had all the answers of how the org charts change, you know, I'd try to go for it. But that's where the organizational cultures matter too. You know, different leaders, different areas. It's going to take a little while for that to change, I think. Absolutely. I mean, certainly if you're making drastic changes, that sort of org chart level, it's like it's going to be chaos. So you mentioned leadership here. So what's the role of leadership in terms of initiating all this? Who needs to be in charge of just making sure all your AI transformation works? Yeah. What we've seen from our clients and through conversations and survey data is that this top-level leadership, so CEO, board, C-suite, really has to be behind this. And they have to own the narrative effectively.

And not just own the narrative in the sense that we can hit the talking points. They've got to live the narrative. And I think that's some of the bigger changes that are here where you need to show that the CEO is using AI. How is the CEO using AI? What's the effectiveness there? We've done workshops. We do a lot of AI education training and learning workshops as well as board ones. And I've been part of some of those where we really, like, getting the C-suite hands-on with AI tools, building agents, and then showing their peers what they're building unlocks a ton of value. So that leadership element is key. You've got to lead by doing on some of these cases. Having the clear North Star of what our intent is with the power of this technology, how we're handling the safety and security around it. Be really clear about how we view this. What are we testing? How do we think about our data? And tell your people about it. That's been great. I've also seen the success then from that is that you've got to get out to the people and talk to your broader teams. Make sure you're communicating what the vision is, hearing their feedback. We've seen a lot of success at our clients and internally with reverse mentoring.

where we're getting folks that are in different elements of the business and are able to share what they're learning about the technology, getting leaders up to speed. Leaders are then able to educate them on the strategy and talk about how that's working and how does this all fit together. It really creates awesome opportunities for culture and team building and community building. And then I think that the final piece is just the flexibility of mindset. Like, know that this stuff is going to continue to change. And some of the folks in this field, I think, said this pushed it out really well, is that you can't design these AI solutions just for today's technology. And so as leaders, you've got to look at it and challenge your team and say, okay, now that you've showed me this, what happens in six months? And you might see a team that kind of go like deer in the headlights. Whoa, what do you mean six months from now? Like this is the solution. It's like, yeah, but AI is going to change in six months. So what's it going to do? And I think that's a really good quality in our leaders is to push some of those questions too. Absolutely. So it's interesting to say it's got to start with the CEO. Like the CEO has to be making use of AI, otherwise no one else is going to take it seriously.

I think last year, maybe the year before, there was this whole thing about like, oh, every company needs a chief AI officer. Is that role still relevant or is it like, well, AI is kind of part of everything now, so you don't need a separate CAIO? I wonder about this a lot. And I think it's one of those things where a classic consulting answer would be, Richie, it depends. And everyone's like, roll their eyes. They're like, thanks. Super useful, right? So I'll give you a take. And then I think, you know, it is going to still fall back on the depends. I think it's a role that lives for a few years. But I think if the CTO function is working effectively, that's where some of this capability can live. And that's sort of my like, you know, if I had to take a stand on one of these things, how I would design an order chart. But we all know that all these structures are based on the people that have been hired, on the relationships that have been set up. So they're well ingrained. And so I'm seeing organizations at least nominate an individual that's accountable for AI. And that's really important. Whether that leader reports the CEO, CIO, CTO, chief transformation officer depends a lot on where the organizations go.

I do worry that like, does that role stick around? Or is that role really what the CDAIO does or CDO, right? Like, you know, depending on all the acronyms we add to that. Or does it fall into the CTO function? And my pressure test for it is I'll ask organizations and say, well, what happens when quantum comes? And say, do you have a chief quantum officer? Does that seem right? I don't know. And so, like, do we have an officer for every technology that comes up? We did digital. So maybe that's a thing. But I don't, you know, digital officers have kind of come and gone in some roles, too. So that's the pressure test. And I do think because of the way AI is changing some of the org charts and sort of the shape of things, we do need to pressure test. Like, do we need more management roles? And what does that really look like? And how is that helping us achieve our goals? Maybe it's an interim thing. So I don't know. You have to tell me what others tell you on the hot take. But that's where my head is. Okay. I love that chief quantum officer test. And that feels like a whole separate podcast episode.

Like, what does a chief quantum officer do? Exactly. But if you're going to throw in a chief AI officer, like, you know, do you need the next waves of tech that comes out? Do you always need a leader for that? Or is the CTO function sensing, owning, finding the next thing and cycling through that in a way that makes the organization run effectively? That's part of my test. Absolutely. And so one of the things you mentioned a moment ago was the idea of you can't just have a solution and be done with it. Because technology is going to change. AI is going to be different in six months' time. So we talked before about like you've got to have these sort of rapid iterations. How do you deal with that and also longer term planning? it seems like these two are slaggy thoughts. Yeah, I feel like, yeah, ask me for the long-term plan, the LTP on your AI activities. You get it? Like, really? Yeah, I mean, we've experienced this. We were going through an implementation of one of our solutions, and we did a custom build for it because the tech wasn't really there in the marketplace. And literally, you know, we went live, and a month later, the tech appears out there, and you're like, oh, gosh,

was it really worth it? And we had to test ourselves and say, yeah, it was worth it because we learned a lot from that experience of what to build. But what we also took away from that was, like, We've got to make sure we're really close with our vendors, alliances, whatever, you know, the structures your organization has to understand our product roadmap. I think the talents that they're facing is the same sort of thing. Like you see a product feature out there, seven other people to go to copy that product feature by the end of the week. And you're like, OK, like how do I navigate this space? And so we're finding that it's really important to have like a lab incubation facility, something on your like broader. We see a lot of companies with AICOEs, call it what you want, but on that team has a function that's able to test out the new stuff and report back about how it's working. And that's their job. Sounds like an awesome job, and a lot of people like doing it, but doing that kind of testing is really important. And their goal isn't to get everything through to a yes. Their goal is to see what's working and sense what's coming next. And so I think organizations need to do their own homework on those things and use that. They can use their parties to help, but get that information.

And then as you designing and building your solution I think it really important to focus on what are the elements of the platform that we building that we using that are consistent and aren going to change These are the things that we believe fundamental Like we have a solution for our data Here how we going to handle it We have these kind of model options presented to us either open or you know frontier or closed, whatever. We have these available for us. We know our infrastructure. We know our way to handle data. And then you need to start making sure that you've got the flexible components so you can change. And we're seeing that a lot in some of the model categories, right? As people build solutions, they're finding out that they can switch from a, you know, a frontier class model to an open weight model in some cases, or a layering of a frontier over an open weight to solve for problems. And that kind of dynamism that we're seeing in the marketplace is sort of how you have to engineer, I think, these solutions. And I think Ghana is the day of like the rigid, you know, here's the architecture printed on the wall in the, you know, chief architect's office that thou shalt not change the architecture, right? It's going to take a lot of continued whiteboard activity to solve for it.

So that's kind of where I think we need to go. I do worry about the cost of these things. And so you do need to make sure that you're kind of making these bets around the platforms, understanding where you're making choices for locking things in and where you have flexibility in those choices. And a lot of clients I'm working with are on this build versus buy challenge because, gosh, it seems so easy when you see some things that, oh, I could build that. And then it's like, OK, but these are all the things that come with the real product set. Plus, you're buying the next X number of iterations of vision that they're creating for it too and so i think wrestling with that and figuring out like what's at the heart of the business that you really want to build versus things that you can kind of buy and plug in um is really key to having that flexible architecture model too absolutely i mean yeah certainly uh there are some technologies which are very difficult to switch like uh like any time you want to switch your data warehouse or whatever your warehouse your cloud provider yeah yeah it's a lot of effort um and so i mean you mentioned like the build versus buy thing i think uh there was a time maybe like a start of the year where I'm almost like, oh, you know what, I can just

vibe code my own version of Salesforce and whatever. And that's not quite happening. So yeah, talk me through how you decided what should you build yourself, what should you buy in, and yeah, how does that work? Yeah, I think part of that is really understanding the market. Like where are the tools today and kind of what are available for us? And are we going to drive competitive advantage from some of the solutions or not? Like is this part of the core thing that I'm building. And the other question is really around data and lock-in that I'm asking too. And so when I think about something that I want to build myself, I'm thinking about those queries of the business that really fundamentally change how I engage with the customer, how that experience is handled. If I were to buy it, what am I potentially locking in or leasing or giving away in that experience that could make me want to change in a few months? I'm seeing a lot of organizations continue to buy solutions. And I think, yeah, we saw that the news and everyone was kind of going down this path of what happens to SaaS and software in general.

And I think what we're seeing is what people projected of that we're just going to have more software. It's going to do more things for us and we might build more wrappers around these tools and capabilities. So I find it to be very helpful for figuring out the working prototypes, the ideas, and then really testing to see if the buy option is cost-effective. And a lot of people are finding the buy option to be pretty cost-effective. You know, the other tricky thing is like you got to get into the weeds with some of these solutions to like a lot of stuff still requires a lot of configuration, a lot of tailoring once it's purchased. And that's important to weigh as well, because you're going to be doing that between the build and the buy anyways. You know, it's not like you're really just taking off the shelf and plugging in, unless if you're maybe a 20 person company and you've got these, you know, standard set of templates and you don't have to differentiate. But most of the organizations we're working with are so large and complex. It's not just one SaaS part. It's like a couple versions of different things, trying to connect to different global data centers. That's well beyond a Vibe-coded solution. And then, by the way, you're also buying the security protection controls that you'd have to implement anyways from scratch.

So I really caution people on building their own on a lot of these things because it might seem easy, but once you get into it, I think it becomes far more complex. Absolutely. You get to your pilot stage, but you're not getting something inter-production. It probably goes well, and then you're down the roadmap, and you see the same thing you were going to buy advanced a year ahead of you because they have 1,000 people in R&D, and you have two. Yeah, that seems fair. So all this kind of links back to maybe the hottest topic at the moment, which is AI cost control. And so it's like, well, even if you buy stuff with AI in it, you still got to figure out what the price is and the price of maintaining that software and all that kind of stuff. Talk me through, how do you not blow your whole budget in a few weeks? Yeah, everyone had the token maxing tattoo at the start of the year to show that we were token maxing. And now it's like, whoa, wait, hold up. I'm not. I'm cautiously using the tokens I have to drive output. Don't call me a token maxer. And so I think that's been a really big shift. So organizations, it's been exciting because organizations are seeing adoption. And so you go back to like circle back to our survey data. OK, great.

So now we're seeing people using AI, which is really important to see. And now it's a question of like, are the input tokens worth the output that we're creating? And I don't think very many people understood there were input costs and output costs and even understood what a token was before, I don't know, maybe April or March in terms of large. I'm sure everyone did, but in the large scale of things, it wasn't on the radar until you started seeing these stories of organizations that were maxing out a 12-month budget in two months. And so organizations really need to think about the personas that are using these tools, thinking about budgets around the personas that are using them, measuring the token input and output for the workflows that they're looking at. And it's common cost controls, right? We've seen these cost controls enter an egress cost for cloud for years. Like we know how to compute costs for cloud. Like we know how to do this. And so organizations that have had sophisticated thin ops capabilities in cloud are perfectly set up to go ahead and measure this from a token perspective. And we're seeing tools come on the market that allow you to get into the granularity of what's actually being used and where. We're seeing the companies that are advancing AI

or adding these capabilities into the harnesses that they're selling with their models, you know, tools that are showing up so you can see exactly how much your prop costs is super helpful. So when I know that it cost me $8 to make something, I know what I spent it on, right? Versus a long-running task that might be taking longer. So I think that transparency is really important, education of the workforce, and putting in these controls and budgets for folks. I think the bad thing to do is, you know, I'm going to turn this off tomorrow because we're overspending on tokens. I think that immediately sort of sets back the transformations that are happening. And so you really want to be educating your team about how these are using. I think the other thing that's popped up and it wasn't, was on the radar, but I don't think as much was the open weight models. And they're popping up because it's not that it's like, it's free in the sense that I'm not paying for to a specific price. You're going to do the inference and, you know, costs associated with running them. But that's popping up a lot too. And so organizations are thinking about how can I use those? Or a lot of companies have model families. And so you're choosing amongst the model family. And do you always need the most advanced model for some of the tasks and questions?

Probably not. And so that's the other thing that I think has helped with the education sort of quell the noise a little bit. But, yeah, a lot of CFOs have learned about token vaccine and token economics in the past couple of months. Absolutely. So, I mean, I love the idea of having tooling in place so people can actually see how much they're spending. This is like visibility across the organization, like what are you actually doing with AI? But you mentioned CFOs there. So I think traditionally, like how much IT capacity you have, That's a chief information officer thing, not a chief financial officer thing. So is there now a new switchover of roles between finance and IT? I don't think there's a switchover of roles. I think what we saw was that CIOs were showing up in the CFO's office saying, hey, I got to tell you about a little problem we ran into. Or we forecasted X and it's looking like Y and here's the reason behind the situation. So no, I think it still stays within the CIO's control plane to look at it. But if you think about what the CIO is doing, they're running the AI factory for the business.

So much like supply chain has a series of costs of goods sold that filled into the creation of a product, they're now running an AI factory for the enterprise. And they're inbound costs associated with all the different business parts that they support. And so how you think about the allocation of those costs back to the business becomes the CFO kind of conversation, right? in the fully allocated P&Ls and how you get into that sort of management. So that's where the bridge, I think, is happening across these two organizations. And it's a very different mindset shift. We didn't think of our knowledge worker teams as having a factory thing, but now we have this factory thing that sort of supports them in the work that they're doing. And so as you add agents to a task, that process, the business function probably needs to pay for the cost of what that's doing. Is it getting allocated back? Are they being thoughtful in the model selection? All that kind of stuff. So that's kind of how we're seeing it. I'm not hearing any CFOs taking over the CIO's jobs yet, but maybe there'll be a battle of agents, you know, to see who negotiates that. Yeah, maybe like a new CIFO role or something like that.

I have no idea whether I should be picking, like, a specific model of a specific effort and whether that's going to give me, like, the best return on investment for, like, the task I'm doing. Like, how should the individual figure out if they're actually getting good value from the AI that they're using? From the selection. Yeah, just even when you think about the individual tools themselves. So a lot of people have their employees using the harnesses that are coming with the models today. Some organizations have built their own harness where they access the models. And that gives you a little bit more control. So, for example, at Deloitte, we built a solution called Sidekick. And that gives us a little bit more control over some of those choices. And so that's another interesting build versus buy decision of that gave us a choice what we can control. But we don't have all the greatest, latest features from the harnesses from other companies. Now, when you get outside of the area that you control, that becomes a little bit trickier in terms of the choices because some of those tools allow you to default selections to specific models. And most of the time, you're good, right? Use the default. You should be fine.

Other times, you need to level up in terms of the thinking, reasoning, speed with which the task needs to be executed. And I think that comes down to a lot of training. But the goal should be to kind of default to the base model that the organizations, that the frontier companies have told you is the like target sweet spot for price performance and kind of live in that zone. Only the sophisticated users, and this is where I talk about personas and why it's so important, should really have access to make some of the model selection that could get people into trouble. But even then you get into budgets where you can set the budgets. And what I've experienced with those is that most of the solutions then default you back to the lowest cost model until you resolve the budget situation. And so that's kind of how to do that. As a user, though, a very frequent user of AI, what I find that I need to do is just constantly sort of be testing and trying different things at once to see what works. And then we share that back with the team about our evals, what we're experiencing, some things better for others than not. But ultimately, like this intelligence level just continues to increase so fast that it becomes for me less about model selection and more about like what processes can I do on these tools?

What work can I offload to them that I can go verify and check and make sure it's doing correctly? Okay, yeah. I mean, certainly verifying what's doing correctly, that seems like the best thing for like getting good value for it because if it gets the wrong answer, then, you know, you'd spend money on nothing. Right, right. I mean, start with the lower end of some of the questions in the task too. I'm always surprised with some of these things because they're, and I spend a lot of time in this every day, and you still get surprised at how good the capability continues to be. And as other AI leaders, you know, are listening, I'm sure they're all kind of doing the same thing. We're seeing the trends that are coming out there. We're trying the new stuff. And I think it's really important for individuals, especially that are deeper in the space, right, have to have their own evals that they're doing to kind of test their own work against this. How do you create your own evals? Like, what kind of stuff should you be testing? Yeah, like I have a series of prompts that I typically run and I use that as a check to see like what kind of answers I get back from AI tools. And so I use that as a way for me to give a sense of like, okay, this is where we've advanced here in different tasks. So thinking about tasks that involve computer use or tasks that involve the creation of a deliverable. So, you know, hey, I'm a consultant at heart. So at some point, I'm going to build a PowerPoint. I check how well I can create some PowerPoint solutions, you know, from some of these things or how well can go through the thought process.

But that's with me as an individual user. Enterprises, we do this as well. We have thousands of prompts that we run against things as they're being released to do evals and then determine kind of how they're performing both on a price perspective, a speed perspective, answer capabilities. A lot of folks are doing this in the coding space as well. And that really gives you a good sense of sort of like where the models are behaving. And the benchmarks are great and super helpful that are out there, and it's another way to measure it. But I feel like so much of this needs to get measured against the work that you actually doing in your enterprise It good to have your own evals I absolutely agree It good to like be able to evaluate stuff Yeah testing is like one of those underrated things that everyone's like, yeah, but incredibly important. So all this kind of stuff seems to lead back to the idea of like changing your processes to incorporate AI. And you mentioned that's one of the big challenges. Like sometimes you go, okay, well, I automated easy step one and the rest of it has still got to manually change the bottleneck so do you have any tips like is there like a framework or like some kind

of process you can have for like this is how i do good uh i guess process re-engineering yeah well like we've been doing process engineering forever right in the enterprise and so there's you know six sigma frameworks and other things organizations use to re-engineer their processes so i think the frameworks exist avoid has some frameworks that we use with clients with a tool called ai navigator that we help them go through as well so we we have we have those things out there. For me, it's a mindset thing. The tools are how you get the work done, but it's got to start with a mindset that I'm going to come in and we're going to change how this work gets done. And it might actually mean that we've deleted five steps. And that's the hardest part for a lot of organizations is saying, I know I started with a five-step process. I'm going to make it a 10-step process today. And I'm going to delete five of them. Because you know what? As I look at this thing, those weren't five value-added things. And I don't think we're taking enough of this first principles approach back to process re-engineering this day. And I go back to my days of building reports, right? I used to be a report developer, data visualization. Like, why do we actually need the report? Like, who's going to look at the report I'm creating?

We need to spend the time asking that question and then figuring out why we're doing some of those things. That's really important for the process re-engineering. Otherwise, I think we fall right back into the automation step, which isn't bad and will drive efficiency, but it will not drive the endpoint, which is like, I'm doing work differently because I have these AI tools and capabilities. And I worry right now a little bit with the advent of some of the personal agents and some of those things that are coming out, that that just kind of makes the same stuff happen faster, but we didn't actually change the stuff that's happening. And maybe those tools eventually change the things that we were doing. I don't know, but that's the piece that I worry about. And so a lot of our clients we think about in sort of two different buckets, like everyday AI efficiency, like what can I drive to make my employees better and more efficient, and give them the tools they want, happier at work, great. And then the bigger ones are the ROI drivers that are going to be these reimagined processes. And I found that if you take a look at that smaller group of processes, you can really own end-to-end and start looking at what you can delete, how you can change what the work is.

Work backwards from the outcome really helps you get to the answer of doing that. And once you get one or two, I think it starts to snowball. But if you go after the most complex, hairiest, complex, awful process in the work, I think you're going to grind on that way too long. We just see organizations kind of get stuck there because it's just too much to unravel out of the gate. So I hope that helps people out there that are trying to do this because it is not easy for sure. No, I feel like if you can crack this, then, you know, that's like all the problems solved. But I do like the idea of like start with simpler processes and see what you can do. And also like deleting steps. I mean, that's brilliant, right? It's like you get to not do work after that. So it seems like the perfect outcome. And a lot of times we're just stuck doing some of these things because we've had to or we've thought it's been part of our procedure for a really long time. And so sometimes this is where it's helpful to start in where you've standardized your processes already. Go back and look at some of those places and say, okay, what else can I change? Why do we really do this? I don't know. If you don't know, it's a good reason to delete it.

Absolutely. So I feel like in general, individual workers are going to be in favor of deleting exception processes. It might be the management that says, well, I'm not sure about that. and so uh there's some kind of communication thing here like uh how do you make sure that your boss is okay with you not doing steps this is where i think you come back to why we started with leadership i think you were asking about the leadership piece which is so important like leadership needs to be on board with like we might do less work in some of these areas and i'm going to get you the same trusted outcome because your outcome was x like you wanted to go achieve that outcome let me show you a better faster way to do that i think leadership buys that if leadership is hearing we're doing steps two, four, and eight from a 10-step process, they're not going to buy in on that. That's not a leadership message. That's like a recipe for chaos because you're just stopping stuff that we might need to tapage, right? So that's where I would say it's all about sort of the leadership buy-in and messaging back to the team. And the other thing that we see a little bit in this too, I'll just sort of harp on this on the leadership side, is that the reason why we're having a hard time understanding the processes is some organizations are sort of frozen

and in the middle of this because leadership isn't talking clearly about what they view AI is. They're worried about what's going to happen to them if they figure out these processes or make some changes in them. And does that impact their job? And you've got to figure out a way that you can communicate that effectively to the team so they know and can safely and have that psychological safety when they're working on these things. And that's a leadership responsibility. Absolutely, yeah. So it just seemed like leadership needs to take ownership of all the changes. Just plant this down to a bunch of people to go figure out lock him in a room turn on the whiteboard re-imagine it get back to me by the end of the week don't think it's gonna happen uh all right so the other block you mentioned at the start was around uh people having skepticism around ai and some skepticism is a good idea because you know you can have things go wrong but uh talk me through their change management process to make sure that people are making use of ai in the way you want them to yeah i mean i think it's really like in your change management probably doesn't do it justice anymore like that's like when you hear change management, you're thinking about an old school ERP rollout.

And I think it has to be a different view of this today, which is, you know, adoption and acceleration of AI is sort of the way I probably should be saying it more because that's the positive of what we're trying to create. We're trying to show workers how the power of this can make your day better, how this can improve what you're trying to do, have more time for human interaction and engagement on the things you want to go do. And so it's about shifting the mindset with some of these things about how the tools work. It's about having that leadership message baked into the communication so people get it and bringing everyone kind of along on the journey. And we've sort of had this journey of like, oh, we want to create AI fluency. And I think AI fluency is a fine term, but it's not the answer. It's like, it's about this adoption and this continuous learning mechanism that you need to build in your organization. And I think that's exciting for a lot of people. If we position it that way, that sounds exciting. I think what we're worried about is sometimes we position it is like, we're going to go change manage you. It doesn't sound like a great thing to have happened to you. I don't know if I really wanted that. So that's the mindset shift.

And I think that's why I'll correct myself. Calling it change manage is probably a bad way to frame it. It's this sort of AI adoption and acceleration. And that allows you to really kind of change the narrative of what's happening. And even then, that sounds a little bit like, oh, we're changing narratives. We're not being deceptive. We're just trying to be clear and transparent about how this is going to work in the org. Yes, certainly. I can see how like there's definitely messaging pitfalls there that you could fall into. So, yeah, probably don't say we're going to change your job and probably get rid of you. So here's our change management at one hour meeting. Like, OK, like that's probably not the way I want to handle it. But if I'm going to go accelerate how your value gets created from your position through the use of AI. OK, that sounds like a different hour. Now let's talk about how that how that works. Absolutely, yeah. You might get better at meeting attendance for that one. You mentioned continuous learning before. So talk me through, like, what do you think the most important AI skills are that everyone should be learning? Yeah, it's like, I think we went on this, like, weird wave with prompt engineering, where

it was like, it was in vogue, and like, everyone had to be prompt engineers. And I was like, oh, no, no, that's like an old skill. Don't worry about it. I think, like, still being able to engage with AI is an important tool. And so we can call that prompt engineering if we want to. But being able to structure your thoughts in a way that you can get the best possible output from AI is a really helpful thing to continue to work on and refine. And some of the fun tricks that you've learned from others and myself is like, I'll put in a prompt and ask the AI to make it better. Or I'll put in a prompt and say like, hey, this was interesting. Ask me five questions to make the output better from what we're going to do. And so all that really is, though, is just like continually using and trying new things out with these tools, I think is like number one on continuous learning. And so giving your team's abilities to go do that and to try those things out and a community mechanism to share is really important. I've also found that if you're delivering learning in this space, figure out ways to do the learning like in micro bite sized chunks at the point of where you're trying to get the AI impacted to the system.

And so video links to the training is really important, making it short and snappy and quick so people get it with our attention spans, make sure we really understand what's going on. And that's been really helpful, too. And then it's this culture of continuous learning. And so putting that into place in an organization is really important. And, you know, we talk about leadership a lot. You give me a hard time. I'll bring it up again. But like if the leadership team is showing that they're learning and adopting, too, you know, it's going to carry through the organization. One of the other funny things I've seen is like sometimes people are sheepish to say they used AI for something. It's like, oh, like I'm going to tell you I used AI to make this PowerPoint. And I'm like, great, 100 percent. Why wouldn't you? Like, I'm sure you checked your work, right? You still got to do the work to make it forward. But like, why wouldn't you use the tools? And so getting that normalized in the organization is really important. And that comes with some of the training and adoption and usage because you'll just be sharing it more with others. There's also great programs for certifications. Those longer term learnings are great. But I think just making it part of the day-to-day habit is really, really important.

I love that. So just a lot of this really is around making sure there's a culture of learning and a culture of experimentation there. But I also like the idea of just normalizing AI usage. I mean, yeah, at this point, if you're manually writing everything in a presentation, creating slides from scratch, that's like, oh, don't do that anymore. You're wasting your time. Yeah. The great joy had aligning shapes and slide decks. You know, I can move on from that. That was a nice feature, but I can let someone else do that. Absolutely. This is big coming from someone at Deloitte. Deloitte's got quite a big PowerPoint culture, right? Yeah, exactly. Brilliant. All right. Do you have any final advice on how you can approach AI transformation? Yeah. I think the key thing is getting started. This can seem like a monumental. You listen to this conversation and go, oh, my gosh, there's a lot of things to think about. Well, just get started. This is a place to get started. I think you summarized it well. You want to start small with something that you can go after. I think that's number one. Having a strong leadership, engagement, message, connection to these programs is really, really important out of the gate.

And then the other piece is, like, be the change you want to see in these things. And so it sounds a little hokey, but, like, if you're using AI tools and you're leading the transformation and you're showing people how they work, that's going to drive a lot of change in the organization and people realize that they can do new things with the tools. So I think those would be my top three. and maybe I'll throw in a fourth and have a good time with this stuff. It's an amazing time to be using some of these things, the things you can do now that you couldn't do six months ago, two months ago. It's pretty amazing. So enjoy that and see what new things you can build. Have a good time. You had me there. Absolutely. Yeah, wonderful. All right, so just finally, I always want more people to learn from. So whose work are you most excited about at the moment? Oh, gosh. Well, Ethan Mollick, I've been following a bunch. I'm really excited about the work that he's doing. He's got his new book coming out, I think, in October. I set a chance to pre-read that, so very excited about the work he's doing there. I'd like to follow The Rundown AI. It's another kind of newsletter that I read a lot. It's a guy named Rowan Chung that puts that out. I think I've been following him since early days of AI.

We probably didn't have a lot of subscribers. And then I just really love listening to the leaders in this space. I think you learn a lot from hearing from the leaders that are talking about their companies and sharing what they're focused on. And there's so much of that out there. that like get on the podcast, listen to things, you know, listen to data framed. Exactly. But like, if you're not out there listening to this stuff, you're not going to pick it up and you're not going to understand sort of like what's going on. So I get a lot from that too. Yeah. I mean, certainly if Mollick is a legend, I don't know about that. I'm definitely going to check him out. And yes, I love that the solution to getting better at all this is listen to data framed. An important takeaway for our listeners. Yeah, exactly. I mean, it's an easy plug, Richie, right? It's a great plug. Yeah. Thank you. So on that note, I think we'll wrap it up. No more to be said. Thank you so much for your time, Jim. Thanks so much. I love this great conversation. Appreciate it.