AIモデル投資の前に決めるべきこと：「仕事」を定義してからモデルを選ぶ - Salesforce Melissa Ramey氏

Welcome, everyone, to the Emoj AI in Business podcast. Today's guest is Melissa Ramey, Director of AI Transformation, Data Insights, and User Experience at Salesforce. Melissa joins us to discuss how leaders can select AI models based on the work their teams need to do. She explains why teams should compare models on specific tasks and account for the people and processes involved before making investments. Just a quick note for our audience, that the views expressed by Melissa on today's program do not reflect those of Salesforce or its leadership. If you offer AI products or services into the enterprise, you need to find enterprise leaders with relevance. That means the right title at the right type of organization. And, of course, readiness. Emerge attracts VP and higher-ranking enterprise audiences who are already convinced that they need to move beyond traditional IT. To learn the exact strategies we use to help leading AI brands and startups connect with their ideal enterprise AI buyers,

visit emerge.com slash ad1. That's emerj.com slash ad1. Now the conversation with Melissa. Melissa, welcome to the AI and Business Podcast. Thank you so much. I am so excited to be here. So our listeners, they don't know this, but you and I, we've had conversations in the past. And just a heads up for the listeners, Melissa is super passionate about this. So you are in for a treat in today's conversation. And she has this superpower of making everything very understandable and very practical. So Melissa, no pressure today. I feel no pressure to be practical and passionate about something I love this much. Thanks so much for the intro. Great.

So let's get into it. Everyone talks about better models. Like it's obvious what that means. But when we just say, oh, this needs to be better and that needs to be better. But when you dig into it, the line between worth it and not worth it gets blurry very fast because we really don't speak the same language on that just yet. So once you dig into the actual difference in performance, what separates a model that's worth the extra costs from one that isn't? So I would actually just change that a little bit and say it's not the model that changes the value. It's the way you use the model that changes the value. All the models have value, right? It really comes down to what is the work that you're trying to do, and do you have sort of the skill to use the model to the best of its ability? I think of it a little bit like a Stradivarius or violin, right? Anybody can pick up a Stradivarius, but if you don't really know how to play it, you're not going to get the sense that it's this incredible musical instrument.

You really have to know how to use it, and you need to be able to use it for the right thing. So I would just change it a little bit and say every single model that's out there, from the least expensive to the most expensive, is a valuable model. And the real question is, what is the human making that choice, trying to do? And do they have the understanding of that model and the prompting skill in order to get the most out of it? Oh, interesting. So we're not thinking about cost, really. We're thinking about practicality and where it's going to be used, how it's going to be used and things like that. So have you seen teams pay for power that they don't actually need, a power that they've never used, and then because of that, they feel that the AI is not worth it? 100 percent and you know there is research out there that will tell you that the more expensive models are slower they cost more they tend to overthink simple tasks and they are very very

verbose so for all the people that are like why is my ai model talking so much it's because you're using an expensive model that's doing a lot of reasoning and so when you needed a simple task completed, you feel like it's failing because you're like, I just need a sentence. Why have you just brought me six paragraphs? So absolutely, I have seen teams that think, oh, I need the most expensive or I need this kind of tooling, right? There are teams that are choosing agentic models like cursor or cloud code, which are more expensive than say a typical LLM like a chat GPT or a cursor without really understanding that the work that you're trying to do must determine both the kind of tool that you choose and the model that you choose within it. And if you choose a lower model for a complex task, you will not get what you need. But the inverse is also true.

If you choose a very expensive model for a very simple task, you will walk away just as frustrated because expensive models doing easy tasks can actually be more inaccurate, according to research, than a lighter model doing that light task. As someone that loves to have the top tier of everything, I know that what you're saying is true. I just don't like hearing it because let me tell you my thinking, and we can maybe see if that's also relevant to the enterprise world. If I want to go and buy a Blender I don go to my local supermarket that sell very entry blenders and get that because you know it just for me I think okay but what if I want to become a professional smoothie maker So I going to go and order the industrial-grade blender, and now I have this super powerful blender that I most probably won't ever use for what it's built for. So my thinking is always, okay, but what if I become this professional smoothie maker? So isn't there a risk with AI that today is good enough

and this is good enough for what we need. Is that not going to become tomorrow's bottleneck as the needs grow? Well, it's a great question, and there's two things I'm going to say about it. Number one, good luck in your professional smoothie-making career. I think that sounds amazing. But there's a couple things to keep in mind. Number one, you can keep, according to research, 95% of frontier model quality while rooting 80% to 85% of those questions to cheaper models. Let's talk about what this means practically. So let's imagine that you need to run to like a home improvement store to pick up some lumber. And you have the choice. You can either take the box truck or you can take the Ferrari. The Ferrari is exciting. It's expensive. It looks like the very best thing. But is it actually the right tool for the job that you are trying to do? you're going to burn up what a hundred dollars in gas just trying to get there once you get there

you're going to realize that the the car you've brought cannot hold what you're trying to buy it's a great it's a great car a ferrari is a great car but it cannot do the job of a box truck and i think the question for us always comes down to if we're using a better thing but getting worse results, that is an output, but it's not an outcome. And all we would have to do seriously in order to test this is to go into your models and put together whatever your prompt is. Okay. And I would say, keep it for this task relatively simple. We're not talking about the most complex tasks because every single model will tell you for your hardest jobs, you need to use those most advanced models. And when they say your hardest jobs, they mean multi-step, long-track reasoning, right? That's what they mean by that. So if you've got that, you need those most expensive models.

I pay for the most expensive version of Claude, and I pay for the most expensive version of ChatGPT, and I know the value of those, so I save them, right? So if you're doing those jobs, great. Get in your Ferrari and let it do amazing things for you. But think about your everyday work, right? You are trying to write an email and you need just a little bit of editing, or you've got a call that you were in and you need to summarize it. Just do it. You don't have to trust me, right? Go in and use your Fable 5.1, and then also use one of your older models and see what comes out. And the result is often that you're looking at these two results and thinking, okay, Fable took me three times as long. It came back with three times as much information, and I feel like it's overcomplicated it. Whereas when I went in and I chose something much simpler, a much older model, what came back to me was a lot faster and tends to be a lot closer to the work that I was doing. Again,

you don't have to take anybody's word for this. You can just do a simple test. I think though, so much of prompting is a skill that is assumed because these models use natural language. It can be hard without really thinking through it, that a prompt for advanced reasoning is very, very different than a prompt for a simple task. And if you're going to do a prompt for advanced reasoning, it needs to be long and extremely detailed because the more you have to come in and make adjustments, the more you're weakening the power of that initial prompt. But lighter reasoning models were built for that. So don't take my word for it. Do a small investigation in your own models. See what comes out. And then I think that is the thing that will convince you more than me or anybody that all these models are not built the same. I think I agree with you with this. I think everyone thinks bigger is better. And I'm thinking

about the psychology behind that as well. What is driving this instinct to go big? Is it fear of being wrong or missing out on something? Or is it something else? What have you picked up? I do think it's a couple of things. But one of the things I've been looking into a lot recently is something called accountability offloading. And accountability offloading is affecting industries everywhere, big consulting firms, entire governments. It's affecting legal, medical. Every profession essentially is in a place right now where we are taking things that AI has given us and then we are just putting our names on them and releasing them out to customers and the broader public. So accountability offloading comes down to AI produces a lot of volume very quickly with a lot of polish, right? I kind of skim through that. It looks pretty good. And then I just put it out there, right? I assume that because it came to me fast, looking really polished, it's probably ready.

But as we seeing all across as I said all across these industries one entire government had to pay millions of dollars A major consulting firm had like 95 of their citations shown to be fraudulent Accountability offloading is happening to all of us So if you take the idea of accountability offloading, where we essentially put an idea into whatever model we're using, we get an output, we read the first few sentences or paragraphs, it looks great, and we think, oh, this is great, this is ready, then the correlation to that is a better model is going to do that even better, right? So if the goal I have of AI is that I put my idea in, I get an output, I never have to read the output, and then I just send that work on to other people, then the bigger model that takes longer and puts more words on a page must be the thing that's going to be even better, right? I think that's a little bit of the psychology that's going on,

that if the old model gave me some good words, the better, more expensive models are going to give me more better words. And it's just not the way the models necessarily work. Yeah, I think that you're so right there. And I'm wondering also, what eventually triggers you to think this way? Because I'm sure that, like you just said, the example with Guri and Tess, your older models and the newer ones and you for basic tasks, you'll probably still want to use older ones. When do you get to that point? What triggers you to get to that realization where you're not trying to convince yourself that bigger is better? I use AI in my work all day, every day, right? So I've had that moment where I'm sitting there with a very, very good model, getting absolutely terrible results. And I'm thinking, what's going wrong? And like so many people, right? All of us are in this boat learning AI together. Like so many people, I get to a point where I've put a really good prompt in at the beginning, right? And I've gotten like pretty good outputs, but now I'm trying to edit it. And with each new edit, my prompt is getting shorter

and oftentimes more frustrated until finally I've been working with, say, a very expensive model for an hour. And the very first results I got were like close, but needed editing to the point that I'm at now where I'm frustrated. I'm blaming the AI. And this is not hypothetical. This is real. I still go through this as well. And I'm thinking, what's wrong with the AI? And then I go over to another model, just like a Gemini or something. And I'm like, I just need you to do this simple thing, and it turns around a very simple edit or polish for me. And that's what started leading me down this road of, again, it's never the fault of the AI. The AI is doing, for the most part, what it is told to do. So we have to bring the accountability back to both the input and the output to say, okay, if the AI output is no longer what I expect it to be, what have I done?

Have I not written a good prompt? Possibly. Has my frustration caused me to start prompting increasingly poorly? Like, why can't you just do this is not a good prompt for AI, right? So I think that's a big part of it is that because I use AI so much, I started seeing a pattern emerge of the, I write a really good prompt. I get a good but needs editing output. I start prompting that in little chunks, in little pieces, and my output is going down, then I go over with that simple task to a lighter model, I get a fast, good answer. So I started with the expensive model with a great detailed, multi-stepped piece of work that was right for the model. But by the end of that, I'm prompting little things. That's not how the most expensive models work, which is why the lighter model to do what I was needing. So again, it comes back to me. Do I understand the nature of

the work I'm trying to do? And have I chosen that tool? And Yolande, that is all from feeling the frustration of my own usage of AI and trying to learn how to use it better. Yeah, and I guess that's why you get to be so practical is because these are real life things that you've had to learn and work through. So thinking like a C-level executive, when they hear this, they say, okay, this is very possible when we're speaking about LLMs and just daily use. But if we're thinking about the AI investment we made to just automate this or to do that, thinking about where to invest the money, and they're still stuck in that bigger is better because how do you taste this without it turning into a huge product on its own? What guidance can you give our leaders there. For every single person using AI that I talk to today, the question I ask is, what is the exact thing that you're using this for? So AI transformation of an org is not an answer, right?

Adding more productivity to an org is not an answer. What people have to understand is, what is the exact thing that I am hiring AI to do? You wouldn't just hire a smart person and hope that you could find them a job, but that's exactly how we're hiring AI, right? We're going out and spending and finding the best and spending a lot of money and then hoping that we'll figure out the job that it's going to be able to do. So I would say start with understanding the job that you need the AI to do Find out if the architecture of the tool you looking at can handle that job Understand what it going to take in order for that job to really work And then start thinking about what model am I going to use But the model follows from the job. The job doesn't follow from the model. So find the job you need and then hire the tool for the job. Oh, I love that. I love that. And I think it

fits in perfectly with our last question for this episode. And I think this is really the question underneath everything that you've been talking about. If you're a leader who has to make this call for your team, is that exactly where you would start? Would you start with sitting down and dissecting the job or jobs that this agent would have to do? Or how should a leader go about deciding what to deploy? A hundred percent. I would sit down and I would say, talk me through this. And I think the skill of interviewing to understand this is not as simple as like, oh, here's my job. It is challenging to really understand the job. So I don't want to make that part sound easy because I do think it is challenging. It takes time. You need people in the room who are doing the work who can say, oh, but you're not considering this yet or you haven't thought about this downstream impact. So I think this is challenging work. The question is, are you going to do this work after you've made the investment and already

spent your budget on the AI? Or are you going to do this before you've made the investment to ensure that every single output from that AI leads to an actual measurable outcome? Because you're going to have to do it. The question is, do you want to do it before you spend the money or after you've spent the money in the wrong direction? There's only one way that this needs to go unless you want to answer some very uncomfortable questions. I'm going to work in one more just because I have you here and I'm really enjoying this conversation. So how often should the team revisit the choice that they made in the beginning? Because jobs do change and requirements and needs do change. So how often should we go back and revisit the choice and see if this is still working for us? Yeah, I would say we have to think about AI just like in many ways an employee. Nobody hires an employee and then just doesn't ever talk to them again, right? There are annual check-ins. There

are often weekly check-ins. There are skip levels. There are so many ways that we make sure that our human employees understand exactly what we need them to do and that we're continuing to assess their output. Agentic AI needs to be treated similarly. This isn't like you buy it, you set up and now it just runs like Microsoft Word or whatever software you were running before. This is now a dynamic, non-deterministic tool that is making decisions based on the output that you give it. You should be just like with any other part of your organization where decisions are being made. You must go back in and say, are these the decisions I need to be made? Does this tool understand or does this human employee understand the shifts in our business? I would treat it the exact same way. Check in with it regularly. Make sure that you're continuing to update grounding knowledge to reflect changes in your business, changes in the micro or macro climates around your business.

Continue assessing the quality of the work. I never would think of these things as really being finished. that doesn't mean that you're not getting outputs that are leading to outcomes in the same way that it doesn't mean that if you have to have a one-on-one that your employee failed it means that you're continuing to check in to make sure that that employee whether digital or human continues to understand is the work i'm doing right okay what needs to change and where do i need to be looking next i love that and i think that's exactly what i want to leave our audience with today is to think on that. That's our final thought. Melissa, thank you so much for being on the show, for sharing the information with us. I think this is extremely valuable for our listeners and there is going to be a lot to take away from this. Oh, wonderful. I had a wonderful time. Always love talking with you, Andy.

Wrapping up today's episode, let's look at the three key takeaways from our conversation with Melissa. First, define the specific jobs the model needs to perform and involve the people doing the work before committing to a tool. Second, compare models on representative tasks, looking at whether the results meet the need as well as the time and cost involved. And finally, review agent performance regularly and update its grounding information as business requirements change. If you offer AI products or services into the enterprise, you need to find enterprise leaders with relevance. That means the right title as the right type of organization and, of course, readiness. Emerge attracts VP and higher-ranking enterprise audiences who are already convinced that they need to move beyond traditional IT. To learn the exact strategies we use to help leading AI brands and startups connect with their ideal enterprise AI buyers, visit Emerge.com.ad1. That's Emerge.com.ad1. for further executive level analysis and to join our network of leaders delivering workflow impact with AI visit Emerge.com

on behalf of the team at Emerge we'll see you on the next episode