
トークンを買うか、AIファクトリーを築くか:エージェント時代のトークノミクスとオンプレ推論(Sadeepa Wijesekara氏)
Buying Tokens vs Building an AI Factory with Sadeepa Wijesekara - #785

トークンを買うか、AIファクトリーを築くか:エージェント時代のトークノミクスとオンプレ推論(Sadeepa Wijesekara氏)
The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence)
要約
HPE Private Cloud AIのエンジニアリング責任者Sadeepa Wijesekara氏が、Sam Charrington氏と、エージェントで急増するトークン費用への企業の向き合い方を議論した。ワークロードの配置はコスト、遅延、コンプライアンス、データの重力といった要素で決まり、まず費用の可視化が出発点になると述べている。HPE社内のAIファクトリー、エージェントのセキュリティ、顧客事例、今後の展望にも話が及んだ。
この回のポイント
- ●エージェントはチャットボットより20〜50倍のトークンを消費するとされ、単価が下がっても総額は膨らむと述べている。
- ●最初の一歩はエージェントやユースケース単位でトークン費用を把握することで、これを「ステップゼロ」と呼ぶ。多くの組織はまだ可視化できていないという。
- ●ルーティングは、まず決定的なルールベースから始め、観測しながら調整し、その後に自動化するのが顧客の傾向だという。
- ●同氏は、オンプレミス運用ならトークン費用を最大約60%削減でき、GPU稼働率が60〜70%程度でも投資対効果が出ると説明している。
- ●エージェントのセキュリティには、短命トークン、サンドボックス、機密コンピューティング、監査など多層防御が必要だと述べている。
- ●今後はコスト抑制の議論から、エージェントが本当に価値を生むかというROIの測定へ関心が移るという見方を示した。
章立て
トークン予算の高騰と背景
AppleやUberの予算抑制の話題を受け、エージェントによるトークン消費の急増と、単価が下がっても請求額が減らない理由を語る。
トークンの価値をどう測るか
コールセンター等と比べ、社内業務プロセスには成果当たりコストの指標がないことを指摘し、Tokenomics Foundationの取り組みに触れる。
ルーティングと費用の可視化
ルールベースのルーティングから始める顧客の姿勢と、エージェント単位のテレメトリやオントロジーの必要性を論じる。
ワークロード配置とGPU最適化
コスト、遅延、データガバナンスに応じた配置、GPUの種類の選び方、夜間ジョブによる稼働率向上、損益分岐を説明する。
エージェントのセキュリティとコンプライアンス
プロンプトインジェクション対策、OpenShell、機密コンピューティング、組織の準備度や標準の不在を議論する。
HPE社内のAIファクトリー
社内チャットボット、ゲートウェイ、コーディングツールのエンドポイント一本化と、PCAIシステムの拡張について語る。
フロンティアラボとの比較と顧客事例
ROI比較、データ漏洩リスクの懸念、データ重力・主権・工場など顧客の事例を紹介する。
今後1年の展望
コスト議論から価値の測定へ移ること、業界別の分類体系作り、セキュリティの重要性の高まりを展望する。
解説記事
HPE Private Cloud AIのエンジニアリング責任者で、Tokenomics Foundationの理事も務めるSadeepa Wijesekara氏が、TWIML AI PodcastでSam Charrington氏と語った回である。テーマは、エージェントの普及でトークン費用が膨らむなか、どの推論を購入し、どれを自前で動かすかという判断だ。
単価が下がっても請求額は下がらない
Appleが社内のトークン予算を削減したという報道を受け、Wijesekara氏は顧客でもHPE社内でも同様の懸念があると述べている。開発者がAIを自由に使えるようになり、エージェント的な作業が増えたことが理由だという。チャットボットに比べ、エージェントのトークン消費は20〜50倍とされる数字が示されており、単価が下がっても総額は膨らむ。
同氏は、企業がかつてのPカード(購買カード)に統制を入れたのと同じ流れだと見ている。個人に枠を割り当てて統制点を作り、次に「同じ予算でどれだけ多くをこなせるか」という問いからハイブリッド推論に関心が向かうという。
まず可視化、次にルーティングと配置
同氏が「ステップゼロ」と呼ぶのが、費用の細かな可視化だ。総額だけでなく、エージェント、ユースケース、さらに検索や調査といったサブコール単位で把握する必要がある。そのためにはエージェント単位のテレメトリと、業務プロセスを分類する共通のオントロジーが要る。SaaSに組み込まれたAI利用料まで含めて追跡するのも難しい課題だと認めている。
ルーティングについては、顧客は慎重で、まずルールベースで決定的に振り分け、挙動を観察して調整し、そのうえでプロンプト内容に応じた自動選択へ進む傾向があるという。ルーターが賢くなるほど、そのルーター自体の監視が必要になるからだ。
可視化の後に来るのが、ワークロードの配置である。判断軸はコストだけでなく、遅延、コンプライアンス、データガバナンス、データの重力(データを動かせない事情)だ。機微なデータは、フロンティアモデルが必要でもクラウドへ出せない場合がある。オープンウェイトモデルは最先端から1〜2か月遅れの水準まで来ているという見方も紹介された。
GPUの選び方と損益分岐
HPEのPrivate Cloud AIは、電源とイーサネットをつなげば2日目から推論を始められる「ターンキー」型で、ライフサイクル管理も含むと説明される。GPUはモデルのメモリ要件に応じて、H200、RTX 6000、B300などを使い分け、同一クラスタに混在させてスケジューラが振り分ける。昼は開発者の対話、夜はコードベースの脆弱性スキャンといった時間帯の使い分けで稼働率を上げられるという。
同氏の社内計算ツールは稼働率60〜70%を前提にROIが出るとしており、損益分岐はそれより低いとも述べている。フロンティアラボの規模に勝てるのかという反論に対しては、同一モデル・同一GPUで3〜5年のコストを比べれば差が出ると答えた。ただし、最先端はフロンティアラボにあるため、戦略はオンプレ一辺倒でなくハイブリッドであるべきだとも強調している。
エージェントのセキュリティと顧客事例
セキュリティでは、チケットやメールに仕込まれた悪意ある指示(プロンプトインジェクション)への警戒が語られた。対策は多層防御で、タスク限定の短命トークン、ネットワークポリシー付きのサンドボックス(NVIDIA支援のOpenShellなど)、ハードウェア証明に基づく機密コンピューティング、監査とふるまい監視が挙げられた。エージェント向けのコンプライアンス標準は、まだ見たことがないとしている。
HPE社内では約6万5千人規模の従業員向けに社内チャットボットを運用し、コーディングツールも単一のエンドポイント経由で、フロンティア提供元とローカルのPCAIに振り分けているという。システムは1〜2台から約9台へ拡張した。顧客事例としては、アジアの自動車企業がAWS上のエージェントから機微データの問題で引き返した例、欧州の防衛分野でのソブリン要件、工場内でのオフライン推論などが挙がった。
まとめ
本エピソードの核心は、トークンの購入か自前運用かを一度きりの調達判断ではなく、可視化に基づく継続的な配置の問題として捉え直している点だ。日本企業でも、エージェント導入が進むほどコスト管理と機密データの扱いは避けられない。まず自社のトークン費用をエージェントや業務単位で分解できるかが出発点になる。なお、オンプレ有利という主張や60%削減といった数値は、ハードウェアを提供する立場の発言であり、自社条件での検証が必要だ。同氏は今後、議論の焦点がコスト抑制から、エージェントが実際に価値を生んでいるかの測定へ移ると見ている。
文字起こし(英語・自動生成)
Agents are causing token usage and costs to explode because a single task can require many model inference calls. At the same time, open-weight models are getting good enough to handle more and more of that work. That's bringing back the old build versus buy question, but in a new form. It used to be a platform decision you made once. Now it can be made call-by-call inside a single-agent workflow. A planning step that needs frontier reasoning may be worth the premium. A simple lookup or summary can often run on a smaller open-weight model for a fraction of the cost. And open-weight models give you a choice frontier models don't, where they run. At enough volume, your own hardware turns a metered bill into a fixed one, and your prompts and data stay under your control. And for some steps, that control isn't optional. The data just can't leave the building. The organizations that get that call right won't just control their token bills. They'll be the ones that make agents pay off. Sadeepo Wajaskara leads engineering for HPE Private Cloud AI and is a governing board member of the Tokenomics Foundation.
Here's how he draws the line. Bookload placement goes along with like what are the model choices as well. Which model should the work get routed to? And then it becomes already that model something that you can run on-prem. Of course, you need still a frontier model for that. That's when we decide, okay, how do we route these to different places? and sometimes it's the cost, but sometimes it's the latency, the compliance of the data governance, data gravity requirements that customers have. Some of the data you cannot send it to the public cloud, or even if the requirements say I need a front-end model reasoning, you cannot send it because of the sensitive nature of the data. I'm Sam Sherrington, and this is the TwiML AI podcast. For over a decade, I've been exploring the ideas and innovations shaping the future of AI through conversations like this one that help you understand what's real, what's next, and what matters. Let's jump in.
There was a really interesting story that broke yesterday about Apple coming out saying that they're cutting internal token budgets. and of course this raises the whole question about tokenomics, token spend. This is of course on the heels of similar stories from Uber and I think Meta earlier in the year, like in the early summer. Do you see the same kinds of concerns in the folks that you work with, the customers that you engage with? Totally. I mean, we see that as the customers open up the gates for especially the developers to do their work using AI. The token budgets have just gone crazy. And that's because of the agenting nature of the work that they do today. So that is happening in the customers that I talk to as well as internally at HPV. We are forecasting a huge, large number of token spend. And then we are very aware of that if you send all of that to the cloud, that is going to be that big number.
But is there a way to reduce it? And I think one of the ways we are looking at is the private cloud AI, the product that I'm responsible for that is running on-prem, could actually save about 60%, up to 60% of the token cost. So that's something that we are looking internally, and our customers are very well aware of that as well. And they're looking for how can they move the workloads appropriately to on-prem versus on the cloud. And when we talk about this token spend, like, if you look at, like, the graph of token costs just from, you know, on the provider's pricing page, it looks like, oh, these are going down. Like, the models are getting better. The costs are going down. Is it obvious to the folks that you talk to why that doesn't necessarily reflect in their bill? Like, do they inherently get that or do you have to explain, like, the role of agents and how agents changes that? I think we, a lot of folks are getting it now. It's because the number of tokens consumed by a chatbot versus an agent is quite different.
I think we saw the numbers from like 20x to 50x more for agents on the token consumption. So even the per token price goes down, when you have a 50x increase in the usage, the total bill is going to be out. So I think that is the reality that our customers are facing today. What about on the value side? Do you see folks starting to kind of wrap their arms around the value that these agents are bringing to the organization? How do folks grapple with that? It is a fantastic question, and I think I'm part of the Tokenomics Foundation board. So, you know, this is the question that we are trying to answer as the board, just like how do we give a good framework and ways to measure the value of these tokens. As you know, the token, each token is not the same, right? And it really should be looking at the value that each token provides. we are, I would say, not super mature as an industry, as providers of these things,
as well as the consumers of these tokens to measure that. It's only certain business flows that we have very clarity on what is the cost per work done, job done. So like, for example, most companies when they have a call center, they know what the cost of a call would be. They know the time and they can easily measure it. But when it comes to other workflows, even maybe like purchasing, you can say what is the cost for processing the purchase order or receipt or whatever the metric would be. But when it comes to some of the internal business processes that they are running, there's no clear metric measurement to do it. So I think that's where the industry is heading. Like we need to start measuring those business processes first and understand that what is the current state of the cost. And then you can say, all right, that's, you know, I understand now cost per transaction or cost per activity, cost per outcome. Now let's put agents and put the agentic workflows in there.
Can we start measuring now what is the reduction that happened because now we have agents in there? Then you equate that to what is the cost of agents when you have true financial equation on saying what is the value you're getting or the benefit you get by introducing AI. I think that's not super mature, but that's where the industry is heading to calculate those. Thinking a little bit about the Apple story and some of the other stories about, again, like companies clamping down on token budgets, I think there's like a surface level read on those stories that's primarily about, you know, spend less. But, you know, there's also like this more nuanced read that is more about be smarter about how you're using the technology and, like, don't choose the biggest model for all of your work if it doesn't really require it? Does that align with what you're seeing? And, like, how do you see folks start to grapple with that technically?
I see that all the time. And, like, when you have access to all these frontier models, you know, as an end user, if I don't know what the cost to the enterprise, I would just use the best one, right? Or just don't bother to even change it. Like, why would I change it? Like, if you have a $20 subscription to OpenAI, TechGPT, I'm not going to pick a lower model. I'm just going to use the best one I can, right? So I think that's what's happening in the enterprise. And not only that, like, I know I do this and a lot of folks do. That's like they experiment, right? We just create stuff because it's like we can. So I think that's probably one of the reasons that it's just like, It's like I'm thinking like early days of P cards, right? You can just buy anything and then you don't have to worry about it. P cards like purchasing cards in a company, yeah. And then the companies put controls in place like, hey, you cannot buy this. You have to follow these processes to control the cost. But equally, I think that that's what's happening in the AI tokens
when companies are realizing there's maybe some waste in there. How do you control that? And if you put the onus onto the individual, at least now you have a control point, right? Otherwise, you just play around and do things that may be not really valid. So I think that is the trend. And then, okay, so now the next question is, I want to give somebody more than that allocated number. How can we get more with the same dollar amount? I think that's when the interesting question of hybrid inferencing, can we do something on-prem, saving costs and giving more flexibility? And can we just have a free-for-all without predictable costs? And that's when the customers come into us for those reasons. I just want to have the flexibility to do that. And even before we get to hosted externally versus on-prem,
there's this big question about routing and like how do I make it easier for folks to get the right work to the right model? Like how are you seeing folks tackle that? So there's multiple routing technologies or routing vendors out there today that sits in front of an LLM or as a selector and you have a chatbot or something, you can just pick the router as the endpoint and then router selects where to go. So, and the router's intelligence comes in, okay, it could be, you can be deterministic or you can set up some rules essentially to say this type of work should go to cloud, this kind of work should go to the on-prem and within the cloud and on-prem, these are the types of models that you select. And we see that it's been kind of the primary way, kind of deterministic rule-based approach. It is kind of the starting point. And then there's the next level of that is like automatically selecting it based on the content of the prompt and the sophistication that it requires.
It will decide that. So our customers are cautious. They don't want so many variables. So, like, you know, if the model router is super intelligent and deciding where to go, it creates another place of, like, now I need to monitor the model router. It's doing the right decision, right? So, they start with, okay, let's do something deterministic. And then let's go with, like, how the deterministic thing. And you can watch that, right? Like, you can observe how that routing happens. And then you can fine-tune that before they introduce the fully automated routing approach. That's what I see the customers wanting to do and trying to do. Like, how do you know what to do when? Like, CIO or AI leader at a large enterprise, you see, like, your token spend forecast going this way, your CFOs saying it needs to go that way or at least flatten out a little bit or, like, where is the asymptote? And you have, you know, a handful of technical options.
you've got routing, you've got like on-prem versus a frontier lab. Like how do you wade through all of that? I know that's kind of a big open-ended question, but so you first, you need to understand at a very granular level what's happening for the token spend. You cannot just say I spend $10 million on token spend. You need to break it down by individual agent and the use case and the subgroup, like you should be able to weigh to categorize them. Then you can look at it and say, who are the biggest spenders? What are the business processes that are consuming this large number of tokens? And what are the agents doing that? So first piece is that, like understanding. And then most of the organizations that you're talking to have visibility into that spend that they can, like, answer that question easily? No. That's what it is. Step zero. Step zero. So we are building some Phenops dashboards internally and for external customers as well.
I think that means like you need to have telemetry at the agent level to say this is tagged as this business process. The user, you can say, but like this agent is doing this business process, right? And then you need to go down to the very detail of, okay, this is doing some retrieval, and that's the cost of that. and this is doing some research, this is the cost of that. So when you look at that, then you can start optimizing. Maybe for retrieval type of thing, I don't need this frontier models. I can just use an open-weight model with some fine-tuning or just not directly. So you can start breaking it down when you know that even at the agent level, at a sub-calls level, where the spend is. So that's not there today for most organizations, but that's kind of the first step if you see the span. How do you kind of surgically approach fixing that?
You can just simply do that, yeah. You can simply say send it to cloud or send it to on-prem. That's like a very high-level breakdown, but then how do you break it down even further and surgically approach this? Even the visibility strikes me as one of those things that it's like maybe harder to do than it is to say. Like, you know, so you're tagging all of these, you know, agent interactions. Is an individual developer doing that? Like, are they referencing some like standard classification or ontology or something like that? so that, you know, developer who's working on workflow A versus another developer working on the same workflow call it the same thing Or like what the you know just in terms of labeling it seems like it could get messy really quickly It could get really messy. It is not an easy job. I think this ontology business is needed. I think that's some of the things that we're talking with the economics board is how do
we create ontology that the standards, as a standard, the industry can adapt and easily say, okay, this is how we should categorize this. I think that is a needed one. And companies that are large, they have their own definition of these ontologies, and they can use that. But for a majority of the companies, they didn't spend time building those ontologies, so they need to come up with an industry standard for that. So that's also a certainly needed case. One other thing that I think is the, when I talk about this tagging agents and kind of the use cases, it's really initiated through your enterprise, like the users or custom build apps. But there's a lot of token spend happening with the SaaS applications, right? Those are already, you know, some of them are built into a SaaS pricing, but some of them are now add-on, right? They say, okay, you buy this SaaS software,
but any kind of AI usage is extra. So how do you kind of, you know, How do you route that through a tagging system so you understand that piece too? So bringing all these things together, that's something that we are doing internally at HVE. So this is very interesting work, and I think a lot of customers will need this. So the first piece is visibility and understanding what you are currently working with. What do you have to think about next? So once you have the visibility, then you can, what we call the workload placement. How do you look at the workloads and the workload placement goes along with the model choices as well? Which model should the work get routed to? And then it becomes already that model something that you can run on-prem or you need still a frontier model for that. So that's when we decide, okay, how do we route these to different places?
and sometimes it's the cost, but sometimes it's the latency, the compliance of the data governance, data gravity requirements that customers have. Some of the data you cannot send it to the public cloud for, even if the requirements say, I need a frontier model reasoning, you can't send it because of the sensitive nature of the data. So then you are kind of, you are sucked in figuring it out, how to make it work with the on-prem. And I think, you know, the on-prem models, the open-made models have came such a long way now, and it's very close to anything. Whereas I heard somebody in the open frontier model person was talking about it. It's like they're maybe like one month behind or two months behind from the frontier level of reasoning. So that means maybe companies can be okay with having an open-made model.
That's a long way from the six months that it was just a few months ago. That's right. It's hitting up quite a bit. And people figuring out different techniques to get to the same level of intelligence by different harnessing and different techniques of agent to do certain things. So that's kind of the, that's where I see how you can solve this problem. The presumption with this workload placement conversation and, you know, run some locally with open weights models assumes a lot about the organization's ability to get and stand up and use GPUs and use them well. Like, what are you seeing there? That's the fear of a lot of CIOs, CTOs, who doesn't have a large engineering bench, knowing this hardware and software and infrastructure management.
All of that stack needs to be put together. So that's one of the reasons that Prior Cloud AI is kind of positioned to solve that problem, because what we are bringing to the table is very much a turnkey appliance. engineered solution that you can essentially plug it into a power supply and give ethernet, and then you start inferencing in day two. It's like it is very quick to turn on and get some results back. And then we also manage the lifecycle. So you don't have to worry about patching and keeping things up to date. The latest versions of software can be easily installed or upgraded. And that takes the burden or the fear out of bringing something on-prem. Now the problem turned into, like, how can I, how do we, what are the use cases? How do we run those use cases? How do we implement those use cases? So those are, like, application layer, business layer questions.
And we also have a forward-diplom engineers that goes along with the purchasing of prior cloud AI. They can help those things at a different business and industry level. GPU optimization. Like, whether you buy private cloud AI or you get GPUs from wherever you get GPUs, you stand them up. Just doing that doesn't necessarily make your inference cheaper. You have to kind of optimize this system that you have against your workloads. Like, how should folks think about that? What are the main challenges? What are the main kind of architectural considerations for folks that are, you know, just starting to think about local inference because of, you know, essentially because of the tokenomics implications? So, and not only tokenomics, there's different GPUs at different prices, right? So I think one of the things that we do as a pre-sales is to look at your use cases today and then what your future use cases potentially would be
and say, here's kind of the types of GPUs you need to start with, whether it's, you know, H200 type of GPUs or RTS6000. This is all in media, by the way, 6000 GPUs or B300, right? depending on the type of models that you need, what you need to run, you say these are the types of GPUs. And then you can say, you can do a combination of those. So I think it's similar to the model of like, do we need the most frontier model? Similar to that, we need like, do we need the most powerful GPU, the most expensive one? Versus you just say for this type of use cases, these type of models, we can run this type of GPU. and then it goes up in the chain as a slightly more high parameter like the KIMI or GLM type of models. I was just going to ask, is that primarily driven by the memory requirements or the models that you want to run? Primarily, primarily. And then there's the RTX one cell for like audio video type of workloads.
So that's kind of specifically for that. So there's a low latency and kind of regular high-throughput GPUs. And then once you have that, so you can have that in essentially the same cluster, like different types of GPUs, and then you can set up rules. Actually, you don't have to set up rules. With the software stack, it will automatically route the workloads to the right GPU. You can set it up. You have to set it up with the GPU scheduling, and then it goes to the right GPU based on the type of workload that you need to run. So that's kind of one way to manage the cost and have a balanced approach on what is right for you. And so the other part is I mentioned the GPU scheduling. You can look at it as how your enterprise works. Sometimes if you have, let's say, coding that happened by engineers,
that's mostly happening during their work days, work time. But at night, maybe those GPs are idle or they may not have work to do. So you can schedule jobs, right? You can say, you know, at night you scan the whole code base for security vulnerabilities. And then you produce output and you try to fix it at night. So the agent is getting fully utilized at night and during the daytime. is where developers interact and, you know, do a lot more kind of back and forth with the interface. So that way you can optimize the GPU utilization, and then that's when you get the best ROI. Like, you know, we calculate our internal calculators that's available for our customers to use through our sales team. We calculate based on like a 60-70% utilization. we still get the ROI compared to the frontier or the public models. That strikes me as quite a bit lower than like traditional,
a traditional cloud ROI model where you would need utilization to be a bit higher to get ROI. Is that true or is that not true? Because the pricing is quite different, quite big, right? So you don't need that high of a utilization to break even. Break even, it's probably less than 60% I call out. But we are looking at those different thresholds. We are not saying that you need to have 100% utilization to get any value, right? Like, it doesn't make sense. But the price difference is so significant, you know, you can say even low utilization, you break even. So when you think about this as, you know, from the CTO's perspective, when you put your CTO hat on or engineering hat on, Like, what are the technical considerations for folks that, you know, need to put together a system like this or want to explore putting together a system like this? So if you were to put together a system from ground up, so you have two choices, right?
You can just buy parts. You can say, I need a server, I need a GPUs, I need networking gear, I need storage, and I need wiring, all these things together and the rack system and liquid tooling and the power supply, all of that together. that's a data center operation type of folks. Data center engineers who knows how to do that. And I think as a CTO, I would ask the question, is that where my value comes from for my organization? So, unless you're like a very large organization or a new cloud, you would stay away from that level of work because you give that to the specialist. It's almost like you know, you go to a Home Depot, you can buy every part for the build-a-house, but you're not going to bring everything and try to build your house on your own, right? So if you were to the specialist, it's like, okay, they know how to do it. And that's what we come in, right? Like we build, we have like 200 engineers
doing this every day. That's, you know, we are optimizing the pieces and parts together, software stack that sits on it. We run the performance tent, run evaluations, and then we say this is the best one that optimize offer we can do. I mean, if I'm a CTO, I wouldn't spend my time on doing that because my business may be very different from just building hardware. So that's a call that each CTO needs to make, like how much should they invest in building the stack that they need to produce the business value. Let's talk a little bit about the management side of things. I think one of the hot topics nowadays is agentic security. How do we manage this? If we're moving into a world of fleets of agents or swarms of agents, some people call them, what does it mean to manage all of that? It's certainly something that enterprises are not used as a new set of skills, a new set
of expertise that needs to be developed. What are you seeing there with regards to that management layer? Yeah, so the management layer is definitely needed, especially when you have agents that you don't even know what they're doing. If you don't have that visibility as a CTO or a CISO, they'll be very worried. Like, I don't know what my agents are doing, what kind of systems they are connecting to, what kind of data that they are retrieving, And more importantly, what kind of poisoning that could happen in this retrieval, right? Like agents are running autonomously. Somebody could add some poison information into a ticket or email or a calendar invite. And then agents nicely read that, take that as a system prompt or instruction and try to execute it. And then, you know, create a big issue for the organization. So as a CISO or, you know, organization leader, I should be looking at what kind of a security mechanism you have to control your agents from, you know, it's a depth and defense play.
You've got to have multiple layers of controls in place. and I mentioned about prompt injection detection. And it used to be, you say, hey, please don't read this or don't write in the prompt itself, right? Like that's the early stage, like please. And that doesn't work, right? It just bypasses all of that. So what is the catchers after that? I think that those are the pieces that our platform has those, is building into making sure that the agent has the right authentication and those authentications are short tokens You just do very short for the task only you're issuing a token so that the credentials are never with the agent. You cannot do, agent cannot just wake up and do it because you need to request a token, and it's granted by a different authority, and it knows what the task is,
and it allows only for that task, right? So it's very scope control. So that's one aspect. And then the agents are running in a secure sandbox. So when you're running on a secure sandbox, the network policy is saying you can go to the Internet or you cannot go to the Internet. And even in the private network, these are the systems, subsystems that you allow to go. So we are very controlled on kind of the network part of it. And then the, I would say another layer that a lot of public cloud agent content doesn't have is the confidential computing aspect of it. So that protects the, even the administrator can see the data or the prompts in memory. So it creates a virtual, it's a secure container that you can, you know, it's all encrypted. So even an administrator cannot see what's going on, a prompt or a request that the agent is doing.
So that's kind of another layer. And then another layer we have is like we are observing everything, auditing everything, logging everything. And then there's kind of somebody on the side watching this and saying, okay, this is something strange. This agent is doing something different than it used to do. So it's like looking at a pattern and then are they going away from a pattern? So those kind of multiple layers of security is needed to be safe with the agents. And at least some of what you described sounded a little bit like OpenShell and SecureShell, a couple of open source projects from NVIDIA. can you talk a little bit about those and what other components come together to enable you to provide all of this? Yeah, so OpenShell is the NVIDIA sponsored it's a CNCF certified project so it's an open project
and that one has a great way to keep the agents in a sandbox and it allows the network policies to say where it can go, where it cannot go, and that is monitored and controlled by policy. So that's kind of a place that we are implementing, and we have that capability in the platform. So that's one. And I think, as I mentioned, the other one is the access control on behalf of, like you connect to, so VeePage agent is doing something on Workday, for example, And if I'm doing that, how can I get my agent the right level of access? Maybe I can read some information, but not right. So that's kind of another level of control. So all this, and then I mentioned the confidential computing aspect. That is, you tie up with the real hardware attestation to say,
I'm certifying that this runtime, this execution happens in this certified hardware. So you have that full control of that as well, and it's all encrypted. So that's kind of the part of that. So we are bringing all this together. So some of them are open source, some of them are proprietary or hardware related. But at the end, I think if you are a CTO, you should look at fully, like what's your end-to-end security strategy to, and including the people side of it too, right? Like how do we make sure that the people are the ones who are triggering this at the end? You know, they are the right people to have access to the agents. So that's kind of very much a given these days. We have multi-factor authentication and things for people to log into systems, right? That should not go away. And these are in addition to what we exist today. Where are organizations in terms of their readiness to adopt all of these various technologies and new ways of thinking in order to, you know, do this smoothly without exposing them to new risks that they don't want to take on?
What I've seen is that when organizations are small, it's much easier. There's a few people who they need to train and few systems to access. So there's fast moves. I mean, that's one of the reasons the startups are very successful. It's like they can just move fast on taking any kind of a new innovation and put it into their workspace and continue to innovate. For a lot of enterprises, there are different teams responsible for each one of these things, right? You know, if they bring any new software in, there's a security team involved, so they need to get accepted. So the question is, like, is the organization ready? I think the answer is also based on the size of the organization and the industry they are in. And the type of work. Like if it is a website, updates or emails, automation or marketing type of thing, it is much easier, less riskier than if it is financial transactions,
making some purchasing or supply chain decisions, automated that. So depending on that, so the companies, what I see is like they picked on one or two use cases and then they go really deep at it and then they get good at it and then they bring the less, the more riskier stuff after that to say, okay, now I can introduce this because now I feel comfortable. So as an organization, everybody needs to go through that kind of transformation to accept these new risks and learn what it means for their organization and layering the security controls along the way. We're starting to talk about organizations like financial services. That brings to mind compliance regimes. Certainly other types of organizations, health care and others, have to deal with compliance. Like, how far along are we in thinking about compliance in a world filled with agents? Compliance and agents, I think that's also a new standard.
Probably need to be come up with that to say we are following just regular security standards and MIST standards and there's a FIP standards. There are multiple things for, you know, because we do business with a lot of federal and defense industry. So we have a very high bar to reach. So we are, you know, our platforms and technology is there to meet those. Now, when we roll out different agents, right, like we are also putting this guardrails that I mentioned, there's like seven layers in security that we do to keep it safe. Now, is there a standard or a compliance standard to say that is the right way to do it? I think that I haven't seen it. So I'm curious if, you know, one of your listeners might say there's a standard out there for agents. And I think that's what we're trying to figure out in the Tokenomics Foundation. It's more towards tokenomics versus the security.
But the security, I'm sure there are some, like, agent security aspects that people are working on to say this is the compliance standard for this industry should be. I think that's going to be very interesting to have that in hand, too. So then we can, like, you know, as an industry, we can just build, software for those compliance standards. Let's talk a little bit about HPE's own internal AI factory setup. You guys have started rolling this out to your own employees. What's the scope of your internal deployment and where are you on the journey? We have 65,000 to 7,000 or something like that globally. We stood up an internal chat bot called ChatGP, very similar to the OpenAI's ChatGP, right?
Very similar to that. And that is, we rolled it out a very long time ago, like two years ago. And then as the OpenAI progressed, we were like, you know, building those features into reasoning and, you know, different kind of capabilities that they have. So that's kind of a one way that we are protecting that, you know, our internal data is now served a similar way as if you are in a public cloud, public data. So that's kind of one way. And then we also built the gateways that I'm talking about, the cost attribution aspects with that. So we, I would say, we're not fully at the detail, as I mentioned earlier, that we need to break it down by agent, by use case, by the different terms with an agent. We are not at that level, but we are at a level to say that, hey, this is a deeper organization. Here's the cost for that. So we can do that today, right? We can call that out. And in terms of, like, serving layer or the inferencing layer,
We are using a lot of our internal services, but then we are using a lot more of the frontier and open Internet services. So now we are bringing a lot more. Our plans are to bring a lot more to internally. As we see the models get improved, the open rate models of the kidneys of the world, we know we can do a lot more internally, and that's just going to be quite savings for us. So we are building that AI factory internally to support mostly. It's going to be a hybrid world. We're not going to say it's completely going to be on-prem, but we are favoring on-prem because it's going to save cost and it's going to be much more secure. So from the end-user perspective, I would say a chat user, for example, is the idea that just like in chat GPT, you can select a model and maybe a thinking level, you'd be able to do the same thing and through the factory setup it will route to the right place.
Yeah, that's exactly the experience we have. And then not only for chat experience, we have VS Code or the coding tools. Even with the coding tools, we can point to the endpoint that we serve internally versus going straight to the frontier model outside. And that endpoint that we have internally will decide, okay, this is a request. I need to go to the frontier model. That's something that we can serve locally. So serving locally is served through a PCAI system, right? And then the frontier models is going to anthropic open AIs of the world. What are some of the biggest challenges you ran into kind of standing up the system? Well, our IT team did it. They can share a lot more of the complications. I mean, it's like everybody's learning new things, right? It's not like a, here's the playbook. So every step we are learning and improving and adding new things.
I don't know what we can call as, what I can call as the biggest learning there because I wasn't part of that implementation. I would imagine just figuring out even just figuring out the workload how much requests would you get and how do I size the system it's a challenge because you have 60,000 a lot of engineers and they are going to send a lot of requests so how do you kind of even scale the system it's a challenge Do you have a sense for how that particular challenge was approached? Yeah, so it was approached through setting up, you know, just taking a calculated guest to start with and then having a infrastructure that can extend. So I know like they port like one or two PCI systems internally to set this up and then now we have like nine. So you can just scale it out very easily. We just kind of add on to say, okay, I need more capacity, I need
more capacity and then add on to the same kind of a management layer we have. So I think that's one of the things that I tell the CIOs when they look at how do you start. It's like, look at a platform that you can extend. Don't get limited to what you need right now because the needs are going to grow. One of the criticisms I hear of pulling all this stuff locally is that the Frontier Labs, they've got a wide range of model capabilities, like, you know, price performance for models. And they're operating at such a scale, like, how are you ever going to outcompete an OpenAI or an Anthropic in inference? Like, if you can even get the GPUs, like, they do it at such scale. They've got to be able to do it cheaper. They, you know, they let me choose an expensive model or a cheap model. You know, I guess I'm curious, like, when you, you know, if you hear that,
and I'm presuming you hear that at least a little, like how do you respond to that? What have you seen concretely that might give an enterprise, like if you're talking to a CIO, the comfort that it's worth exploring? I would say like let do a cost calculation right to really see the ROI between an exact same model or exact GPU on a cloud or on and then look at the 5-year ROI or a 3-year ROI on that, and you'll see the difference. Right, like that's kind of the starting point of that. Now, yes, you just... It's just that black and white. It's just right there in the numbers. Yes, just comparing the numbers, you'll see the difference. Right, like you see the difference right there. Now, the question would be like, oh, they're always innovating. They have better models. You know, they will have a better one than what we have on-prem. I agree. I think they will be the frontiers, and that's why they're called frontiers, right?
And I think that's why the strategy should not be just on-prem. It should be a hybrid model. But, like, how do you be smart about what's put into the frontier versus on-prem? and some of them are like, okay, they have the same models. Maybe they can run it in the cloud as well, like the open-weight models, they have it in a different cloud offering. So that may be the tool. Then the question becomes, do you have a data gravity issue? Do you want to send your sensitive data? And these days, data and your internal processes, it's your IP. and once it goes to the cloud, you know, how sure you are that it is protected. So I think those are the risk questions that you should be asking, not necessarily the cost questions. Yeah, it definitely seems like it's the case around on-prem. You know, there's kind of this by-the-numbers piece
that's the first one, and maybe the right answer there is doing the ROI study, and can you get it detailed enough to capture all of the costs on either side. But then there's this whole other set of factors around privacy and IP leakage and control and seeding your competitive advantage for many organizations to a potential future competitor. A lot of things that we heard in the days of cloud, right? That's right. They're analogous. They're analogous and they're kind of real in a way because when you send a prompt out, to a GPU, that prompt is visible in clear text. Administrator can just do some commands and dump the memory and see what's going on. So that is a risk if you're willing to take that because the prompt could be a real business scenario that you might want to protect.
So if you're ready for that, then you can go to Frontier then it's not a concern for you. Yeah, it kind of strikes me that the closest analogy to this is like the, I don't know what we want to call it, the math gate stuff with OpenAI. Like, you know, there's a point in time where, you know, the frontier model provider is going to steal my data as like this big theoretical concern. But then, you know, recently with the whole Navier-Stokes thing, there. And since then, in other cases, there have been like several concrete accusations that, well, this provider trained on my data and then put the paper out before I did. And, you know, we probably don't, you know, the enterprises, you know, if this were to be happening to enterprises, we don't know what those stories are. They probably don't know. But it does strike me that it's kind of shifted from this like FUD, you know, theoretical
possibility to, well, here's a concrete case where somebody really thinks that the frontier provider, you know, took their IP. Yeah, and I think they may, I mean, in the agreements, right, with the frontier model providers, they said they are not training with that data. That may be true, but before a model gets to that data or for training, there's multiple layers it goes through, right? So there's definitely going to be logged, right? Like all the prompts are getting logged. And they may be erasing that too. And then some evaluations they run. So evaluation might have that scenario that you describe that's captured in evaluation. And then now evaluation is the one. So they are training the model, but the evaluation is the one just kicking off and say, hey, look, this should be the answer. So I think that's the level of details that it's not disclosed, right? It's not clear, right? So you're not training on my data, but are you making synthetic data based on my data that you are training on? So it's an interesting place, and I don't think anybody's giving clear answers,
but they're not completely saying no either. I saw some, I don't know, entropic or open air who just said they cannot conclusively say they didn't train it or not. They just left it out there. Yeah, so those are the risks, I think like customers are realizing, and then we have a customer recently in Asia, well-known automotive companies. They were doing some agents on AWS, and then they realized that for agents to work, they need to send some sensitive data. And then that's when they kind of pulled the plug back and said, oh, I'm not ready to go there. I need something on track. Tell us more about your experience with like actual customers. What are actual customers doing in deploying these systems? And, you know, if there are ROI examples you could share or, you know, learning challenges you can share, I think all that would be really interesting. You know, one category of customers are what I just mentioned is the data gravity concerns.
Like they don't want to send this data, so they want an ultimate in a home on-prem SINDIP situation. Another one is this is one I recall from one of the European defense, one of the countries, I don't name the names here, but a European country, and they want to make sure that the sovereignty of the software stack and kind of the inferencing stack and all the things related to AI is within that country. So they partnered with a local company that builds these things and they use our infrastructure to run it. So that's kind of another category. Like in sovereignty, it's a big deal, so they want to keep it local, the software and the dependencies they have. And that's one. And the other one that I see is kind of the large, this is with industrial manufacturing use case,
that they need to run these things locally close to the machinery. So imagine the factory, and they want to have intelligence to those factory equipment, and they don't want to send the data back and forth to the cloud, and they want to operate disconnected as well. They don't want to be connecting to the Internet or reliability of the Internet connection. So they completely run that inferencing locally so that they buy our PCI software hardware and then kind of OEM that under their product and then send it to 50 or so different installations. So that's kind of another type of customer use case. I think those are kind of the ones that I can think of. Even the earlier customers, they were doing chatbots and RAG, and this is like exposing their internal knowledge bases to the enterprise. That's kind of a very out-of-the-box situation for us now.
We have something called Instant RAG. You just get the hardware in our software unit and point it to a data repository, and then you have a chatbot in days. That's kind of a quick one. Now the companies are moving beyond that feeding autonomous agents. Just building on that, you're seeing a lot of people kind of starting with RAG and now moving more towards what use cases are you seeing today like driving growth? I'm imagining software development is a big one. What others? The customers buying HP, private cloud AI, the majority, I would say the large number is using that as a shared inferencing capability for the whole organization. So they can... So they're a version of my HPE or... Exactly. My customer, GPT, whatever. Right, right. So it's like, okay, here's the software engineering team. Here's your endpoint. You know, you do whatever you want. and here's the HR team.
You have your version of that. Different teams will have different capabilities and then guardrails or secure to run their workloads. So that's kind of a shared GPU and inferencing. So essentially, you're producing tokens internally. So you become your own token factory inside your premise. So that's kind of the one type of use case that large companies use. And then when it comes to, and we see a lot of edge use cases as well, like in the factories, like video cameras, watching cameras. And those are for smaller units. We have smaller units that you can install and then, you know, detections of fraud or detection of activities in manufacturing floors and alerting right away and figuring that out. So that's kind of another use case. And then we have some voice use cases as well. So customer service agents can be running on our devices. And the devices or the approach generally can span kind of the heavyweight inference of media to the real-time use cases of voice?
Yes. So we have a spectrum from we have something called a developer kit. It starts with the two GPUs. That's like the smallest one. and then we can go up to the rack scale up to 256 GPUs. And then that would be a mix of different types of GPUs. Where do you think this whole tokenomics conversation goes over the next year or so? People are pulling different levers. The companies are pulling different levers. One is like what you just started about Apple's story of giving a quota, right? So it's controlled cost. And then what I talked about, frontier models versus open-rate models and on-prem versus frontier. So that's another controlling factor. And I think those kind of different levers would bring this token conversation to a, I would say, more not the focus. I think that that focus will go away very soon. Once people pull these different levers, then the conversation will really quickly turn into,
are we producing the value, even with these controls in place? Is the ROI there for the company? And then to the customer, that they are finally delivering that outcome. Is the customer going to do something different? And are they getting the value? And that's where the next iteration of this AI journey would be. All the optimization we do in the context management, the KVCache management that we do internally, it's just going to benefit to have the cost equation, right? The value equation is the one that needs a lot more work on to say, is this agentic world is really going to change the ROI of an organization or the value of an organization? I think that's where people will be spending more time on that. What's the current snapshot on how HPE is thinking about the answer to that question internally? You want to solve the disability problem of where the agents are going and what works.
So that's kind of the number one. And I think the next one that I'm personally interested in working with the Togonomics Foundation is like how do we create this taxonomy type of a way to define for different industries how we should look at a business process and then map a cost structure to it. And I think that is not an infrastructure play, it is not an infrastructure workload, but it's needed to justify all these investments that's needed to make this happen. And then on the technology side, like where do you see the technology going in the same time period the next year or so? So I think the security is going to be a big one, security of these agents. Because right now we are maybe worrying too much about costs, but the security things, we see the stories, right, like the agents breaking out and all these things and the attacks that's happening intentionally, unintentionally. How do we protect from that?
It's going to be a huge topic. And from HP perspective, we are building the security from layer by layer, from all the way from hardware to software. So that's kind of our focus. That's what our contribution to the industry is like being the secures of high performing software and hardware together for our customers to use. Awesome. Lusitipa, thanks so much for jumping on to share a bit about what you're up to. Great, well, Sam. Great chatting with you, and thanks for having me. Thanks so much. and our broader series on enterprise AI economics.
番組の概要欄(原文)
An AI agent can make many model calls to finish a single job. Some need frontier reasoning. Others can run on an open-weight model, and some involve data that has to stay in-house. How do you decide which inference to buy and which to run yourself? Sadeepa Wijesekara, who leads engineering for HPE Private Cloud AI, joins Sam Charrington to discuss how enterprises make those decisions. They get into what it takes to see the cost of individual calls, how teams route work between hosted and local models, and when running your own GPUs makes economic sense. Sadeepa shares how HPE routes coding requests through a single internal endpoint that can send work to a frontier provider or serve it locally. 🗒️ Full show notes: https://twimlai.com/go/785.
出典・この記事について
- 番組
- The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence)(Sam Charrington)
- 元の回
- Buying Tokens vs Building an AI Factory with Sadeepa Wijesekara - #785
- 配信日
- 要約作成日
- 公式サイト
- twimlai.com
この要約・解説記事は、元の音声を文字起こしした内容をもとに AI(Claude)が日本語で作成したものです。 固有名詞や数値に誤りを含む場合があるため、正確な内容は元の音声でご確認ください。制作方法について
関連エピソード

企業のAI活用はどう変わっているか:トークン使用量は36%増、支出は2%増
The AI Daily Brief: Artificial Intelligence News and Analysis

ジェンスン・フアンがNvidia CEOを辞任、AnthropicのIPO、AI 2027の前倒し予測、HubSpotのAI再創業
The Artificial Intelligence Show

技術面接は壊れている:HackerRank CEO が語るAI時代の採用
Software Engineering Daily

人間はAIエージェントとどう協働するか — Atlassian CEO マイク・キャノン=ブルックス
Big Technology Podcast

マルチクラウドLLMルーティングと、コンテキストを理解するコーディングエージェント
Agentic Conversations (formally mlops.community)

ServiceNow投資からさらに運用価値を引き出すには――BigPanda CEO アサフ・レズニック氏が語る
The AI in Business Podcast