AIはローカルで動きたがっている ― Pete Wardenが語るオンデバイスAIと音声インターフェース

Why AI Wants to Be Local

The Data Exchange with Ben Lorica37分
AIはローカルで動きたがっている ― Pete Wardenが語るオンデバイスAIと音声インターフェース

AIはローカルで動きたがっている ― Pete Wardenが語るオンデバイスAIと音声インターフェース

The Data Exchange with Ben Lorica

0:0037:08

要約

Moonshine Voice の共同開発者で、Useful Sensors の Pete Warden が、ローカルAIの現状と将来を Ben Lorica と語った。オープンウェイトモデルは商用モデルに数か月差まで迫り、ノートPCでも実用になると述べる。音声は低遅延と障害を前提とした設計で自然なインターフェースになり得るとし、複数モデルを組み合わせるコンパウンドAIを支持。推論向けハードウェアではAppleの動向に注目すべきだという見方を示した。

この回のポイント

  • ●ローカルモデルは商用モデルに数か月遅れ程度の性能で、普及を阻む最大の壁はモデル品質ではなくユーザビリティとマーケティングだとWardenは述べている。
  • ●9Bクラスのモデルは量子化すれば約4GBのGPUメモリで動き、256GBメモリ構成のM5 Studioなら数百Bパラメータ級も実用速度で動かせると説明した。
  • ●音声UIは200ミリ秒以下の応答が望ましく、クラウド型では数秒かかる。トークン課金がなくなれば、アプリ設計の可能性も大きく広がると指摘した。
  • ●エンドツーエンドのマルチモーダルモデルより、音声認識・LLM・音声合成を組み合わせるコンパウンドAIを支持。汎用モデルは音声内容の質問に反応してしまうなどの失敗例があるという。
  • ●ChromeにはすでにローカルLLMが内蔵されており、モデルがOSやブラウザの一部になっていく流れを「iPhone moment」と表現した。
  • ●推論はGPUよりメモリ帯域が律速で、Appleの大容量RAM・高速メモリ・推論専用アクセラレータや高帯域フラッシュなど多様な解が出てくると予想した。

章立て

  1. ローカルAIの現状と普及の壁

    ローカルAIのツール事情を議論。マーケティング不足が課題で、Ollamaなどは意外に使いやすいという。OSS対商用の歴史との類比も語られた。

  2. 必要なハードウェアとモデル選び

    9Bモデルの量子化によるメモリ要件や、Appleの大容量メモリ構成を紹介。大型ローカルモデルはClaudeと見分けがつかない場合もあるという。

  3. 業務利用・プライバシーとコスト

    知的労働でも今すぐ使える水準だと述べる。ガードレールの制約、セラピー用途、オンプレ運用、推論コストの低さにも触れた。

  4. 音声インターフェースの設計

    悪いUIが嫌われているだけだと指摘。低遅延の重要性、トークン課金の不在がもたらす自由度、失敗を前提とした設計を論じた。

  5. LLMと音声モデル、マルチモーダル

    エンドツーエンド派とコンパウンドAI派の対立を整理。汎用モデルの限界や、極小マイコンでの音声UIの実例を紹介した。

  6. ブラウザでのAI推論

    ブラウザ上での音声処理やChrome内蔵LLMの可能性を説明し、モデルがOSの一部になる未来を語った。

  7. 今後の展望とAIハードウェア

    ローカル志向の圧力とループの自社保有、Appleに注目すべき理由、GPUとの違い、高帯域フラッシュを論じた。

解説記事

The Data Exchange の今回のゲストは、Useful Sensors で Moonshine Voice を手がける Pete Warden 氏。深層学習の黎明期から活動してきたエンジニアが、司会の Ben Lorica 氏と「なぜAIはローカルで動きたがるのか」を語った。オープンウェイトモデルの進展、ノートPCやブラウザでの推論、低遅延音声UI、そしてハードウェアの行方まで、話題は幅広い。

ローカルAIは「妥協」ではない

Warden 氏はまず、ローカルAIが普及しない理由はモデルの質ではなく、使いやすさとマーケティングにあると述べる。ローカルモデルは利用者がサーバーに接続しないため収益化しにくく、Anthropic や OpenAI のような企業は、簡単にサブスクリプション化できるSaaSを優先する。その結果、多くの人はローカルモデルの存在すら知らない、という見方だ。

ただし Ollama のようなツールを使えば、意外なほど簡単に動かせて品質にも満足できるはずだという。ベンチマーク上でも、ローカルモデルは商用の最先端モデルに数か月遅れ程度で追随していると語った。かつてウェブサーバー市場で、高価な商用製品が Apache や Nginx に取って代わられた歴史になぞらえ、同様のことが起こるかもしれないと期待を示した。Lorica 氏も、Linux や Postgres の例を挙げ、スタックの重要部分ではオープンな選択肢が選ばれるようになるとの見方に賛同した。

必要なハードウェアと、使える場面

ノートPCで動かすために必要な環境について、Warden 氏は「モデル選び次第」と答える。Qwen などの90億パラメータ級モデルは性能が高く、8ビットなら約9GB、Q4など量子化を使えば約4GBのGPUメモリ(VRAM)に収まる。古めのノートPCでも載る量であり、ハードルは思われているより低いという。

さらに Apple の M5 Studio には256GBのシステムメモリ構成があり、そのメモリをアクセラレータから利用できるため、数百億〜千億級のモデルも実用的な速度で動かせると説明した。この規模になると、Claude とオープンウェイトモデルのどちらを使っているか判別しにくいとも述べている。

知的労働に使えるかという問いには、「すでにその段階に来ている」と回答。残る障壁はモデルではなくユーザビリティだという。商用モデルの制限が、セキュリティ研究者のような正当な利用者の作業を妨げる例も挙げた。また、「ローカル」はノートPC上に限らず、オンプレミスや企業が管理するクラウドインスタンスも含むと補足した。Anthropic や OpenAI の利用料が実コストに近づきつつある中、自前ホスティングのほうが安く済む場合もあり、推論だけなら高価なNVIDIA GPUが不要な選択肢もあるという。

音声は「悪いUI」が嫌われているだけ

音声インターフェースについて Warden 氏は、Siri や Alexa の印象で懐疑的な人が多いと認めつつ、ソファでテレビ番組を選ぶ時、パートナーにテキストを送る人はほとんどいないと指摘する。嫌われているのは音声そのものではなく、出来の悪い音声UIだというわけだ。

鍵は遅延である。クラウド型では応答まで数秒かかり、相手に無視されているように感じさせる。UX研究が示す200ミリ秒以下を目指すべきで、キー入力が2秒遅れて表示されるのと同じ不自然さだと例えた。さらにローカルならトークン課金が不要になり、アプリ開発者が操作のたびに数セントを払う世界ではそもそも成立しなかったアプリが生まれる、と論じた。

設計面では、失敗を前提にする点を重視する。Google検索が上位3件とスニペットで人間に絞り込ませたように、音声でも相づちや表情などの手がかりを理解し、誤認識を前提に対話を進めるべきだという。カメラも含め、すべてローカルで処理すれば録音・録画が外に出ないことを検証でき、クラウドと違って監査が可能だとした。

コンパウンドAIとブラウザ推論

Warden 氏は、すべてをエンドツーエンドの単一モデルで処理する「LLM至上主義」より、音声認識、LLM、音声合成など専門モデルを組み合わせるコンパウンドAIの立場を取る。テキスト変換で表現力は失われるが、各モジュールを個別に訓練・差し替えできる利点が大きい。エンドツーエンドは、巨額の資金が必要なことを正当化するマーケティング用語の側面が強いとも述べた。

汎用マルチモーダルモデルについては、Qwen の約17億パラメータ級が一部言語で Whisper を上回る一方、Gemma では音声内容の質問に答えてしまうなどの失敗例があると紹介。万能ナイフより木のこぎりのほうが使いやすい、という比喩で専用モデルの利点を示した。極端な例として、RAM 500KB程度の Raspberry Pi Pico マイコンで、Wi-Fi設定までこなす音声UIを動かしたことも挙げている。

ブラウザについては、Chrome に約40億パラメータ級のモデルが組み込まれており、JavaScript から利用できるようになりつつあると説明した。ユースケースはまだ模索段階だが、無料でどこにでもある状態になれば「iPhone moment」になり得ると語る。

12〜24か月の展望とハードウェア

Warden 氏は、Claude や ChatGPT 相当のものが1台のノートPCで完結し得る点で、モデルの形状はローカル実行に向いていると述べる。Google検索のように巨大なデータベースを要するサービスとは対照的だという。コスト、プライバシー、機能の突然の変更への不安などが、ローカル回帰の圧力になる。同じバージョン名でも内部が日々変わり得るため、安定性を求める企業は自前ホストが必要になる。また、モデルを使って学習するループを他社に渡さず、自社でファインチューニングして握るべきだという議論でも、Lorica 氏と意見が一致した。

ハードウェアでは Apple に注目すべきだと言う(本人は過去に Apple で勤務したと断っている)。大容量RAM、高速なメモリ、推論専用アクセラレータの3点が揃えばGPUは不要だとし、GPUは大規模バッチ処理と学習向けで、推論には「買い物に石油タンカーを使うようなもの」だと表現した。推論はトークンごとに全重みを読み出すため、メモリ帯域が律速になる。高帯域フラッシュなど、ワークロードを理解した上での多様な解が出てくるとの予想だった。

まとめ

今回の議論の要点は、ローカルAIの課題がモデル性能から、使いやすさ、流通、OS統合へ移っているという見立てだ。日本の企業にとっても、機密データを扱う業務や、モデル更新・価格変更に左右されない運用を考えるうえで、ローカルやオンプレミスの選択肢は検討に値する。音声UIでは、低遅延と失敗前提の設計という具体的な指針が参考になる。なお、これらは Warden 氏個人の見解と予測であり、本人も予測は苦手だと断っている点には留意したい。

文字起こし(英語・自動生成)

All right, so today we have an old friend, Pete Warden of Useful Sensors, and their main website is actually at moonshine.ai. And if you go to the website, you'll see that they're very much about voice on device. And we will definitely talk to Pete about all things speech and all things edge computing. But I do want to take advantage of Pete being one of the OG deep learning engineers to get his sense of what's going on these days. And so with that, Pete, welcome back to the podcast. Nice. Thanks for having me, Ben. All right. So I would say, Pete, I don't know if this is a fair characterization, but one of the main trusts of your work is the whole idea that local AI does not have to be compromised. that in fact it's something that people should take more seriously. And as I mentioned, we'll definitely talk about your work about speech models on device.

But I guess broadly speaking, Pete, for our audience that doesn't really follow local models, where are we for someone, say, who wants to run a model on a laptop, not a phone, just a laptop? What's the state of ease of use? I guess if one were to characterize one of the obstacles, it's always easier to use a cloud API, right? Yeah. So where are we in terms of just the tooling around local AI? Yeah. So at the moment, it's like many open source projects. You know, we've been waiting for the, every year is going to be the year of the Linux desktop. And, you know, so I think that there's, you know, there's always going to be a bit of a lag in the usability for kind of the average user for this stuff but I think there's an even bigger lag in marketing because it's very hard to make money

off local models and that means that the big companies like Anthropic and OpenAI and even like just the big tech companies generally you know they aren't going to push that but if you Give them a choice between, hey, here's a service, SaaS, that you can actually make money on. And, oh, here's this other thing that you're not even going to know that people are using it because they're not contacting your servers. Every company is going to go for the one that has an easy subscription, you know, revenue model. And that means you have a massive ton of marketing around all of these tools that are kind of like, oh, let's have a, you know, a little text box on a website. So it means mostly that people have never heard of these local models. Now, if you use something like Alarma, it's probably the most well-known. I think people will be pleasantly surprised by how straightforward it is to use

and by the quality of the models. I saw some news recently that Anthropic is actually a bit concerned concerned because people are turning to more more specialized models versus always going for the kind of most sprawling expensive model that's available so local models are actually doing very well in the benchmarks they you know they continue to only be a few months behind the capabilities of you know the big commercial anthropocon open ai models but i also believe that local models have got to the point where they can do the things that people want to use them for so what i'm really hoping is that it's kind of one of these you know those of us old enough to remember the dot-com boom and people paying kind of you know thousands of dollars for spark servers and or using like

Microsoft IIS and paying massive amounts of money because web servers were so seen as such a kind of an arcane thing and then of course Apache, Nginx, all of these alternatives once people actually had time to look around and they had a little bit of time to improve they just wiped the floor with the commercial alternatives. I don't know if we're going to quite get there but that's the kind of patman saying yeah it seems like the the my sense is that open models whatever your definition of open is open weights mainly i guess will ultimately prevail i mean it just seems like that makes a lot more sense to me just like uh as you mentioned right so obviously in the early days of linux versus solaris or even postgres and mysql versus oracle it was far from clear that these open source projects will ever get to the point of us having to take them seriously. But ultimately, I think they're so important parts of the stack that the open alternatives are what people will

opt for. So speaking of which, Pete, so you mentioned OLAMA, right? So and then obviously there's other alternatives. So let's say for the sake of argument that they are in fact easy to install and use. So as far as the models are concerned, so you have things like JAMA, I suppose, and other small models, right? What kind of hardware does someone need to have in order to run these models so that they don't have to wait 20 minutes for each prompt? So a lot of it comes down to your model choice. Yeah. You can get very capable models that are like 9 billion parameters these days especially if you look around at the quen family of models they tend to be doing very well though there were a bunch of other alternatives but translate that pete so so to someone with a normal laptop it's nine billion big or small so the key thing is whether you can fit that into your graphics cards memory so with weight quantization nine billion if it was

8 bits is like nine gigabytes and you know a lot of kind of mid-end decent laptops that are shipping now have more than that but using q4 and you know there's there's all sorts of schemes that can actually get that down into a four gigabyte graphics card and that is you know that that's widely available like there are a lot of you know even older laptops that have that amount of gpu memory vram in them so really the hardware barriers are a lot lower than people think and the other piece of good news is that all of the hardware manufacturers are looking at this and they're slightly hobbled by the dram craziness the price what you know the price increases but if you look at the new M5 Studio from Apple, like they have a configuration that's 256 gigabytes of system

memory, and Apple do a great job of actually making that available to the accelerators that they build into all of their Macs now too. And with that, you're looking at something that's like a hundreds of billions of parameter model that you can actually run at a decent speed. And once you get up into those numbers you really are looking at something that it can be hard to it can be hard to tell if you're using clawed or if you're using you know an open weights model at that point you know they they have different characteristics but one isn't clearly worse than the other so so as i mentioned we'll definitely dive deep into voice and speech in a second here but sticking with this notion of local models so obviously privacy and ip security is one argument for it do you think if we look ahead like let's say 12 to 24 months do you think there will be these local models are going to be capable enough that people doing knowledge work in an enterprise might even consider

them here and there right so for example someone who travels a lot is on an airplane not connected to the internet needs to keep, I don't know, writing a document or generating images or even coding, vibe coding on a laptop. Do you think we'll get to the point where these local models are essentially, you feel like you're not compromising when you're doing your knowledge work? I think you can actually get to that point right now. And I know that's a fairly bold statement. But when you say you, does that mean Pete Warden who's going to compile all of these we want normal we want civilian speed i think the biggest barriers are the usability you know it no longer if you look at the benchmarks it really is no longer a a model thing yeah a question of like model quality you know there are still things around the edge where you might prefer some of the commercial online models but it not clear

cut and people are finding some of the models actually work better especially if you're dealing with things like you know i don't want to call it censorship because i feel like that's a loaded term but a lot of commercial models have restrictions and they may be intended to prevent a bad use case, but pretty often they can also trip you up even if you're trying to do something benign. So I know a lot of people who work in cyber security, for example, who actually, you know, they're white hats, but they're having to work with a lot of stuff that's, you know, haven't asked questions about malicious actions. And with the commercial models, you're going to keep tripping over the restrictions and the guardrails that are built in with good intentions but they can really impact your work. Right, right, right. By the way, Pete, so there's, you know, the model is one component, but then there's the things surrounding the model, so-called harness or even context, memory, knowledge. So you can imagine someone,

and this is actually something I've thought about, so you know how people are starting to use these LLMs as therapists? You know, I mean, or friends, right? Yes. But then if you're going to do that, Why not do that on a local model? Why do it on something connected to the cloud? I guess one argument against it is if you do it locally, you're responsible for model upgrades and things like that, right? So whereas if you use one of these digital or AI therapist service, then they take care of all your history and all of this. So is this something, do you think, that will become easier for consumers to use local models that are essentially something they trust 100% because they own it, right? Yeah, I mean, these should be part of the OS. Like, you know, at this point, it's really kind of an operating system capability that should be available to apps. And I feel like Apple are moving in this direction.

I see. Yeah, that makes a lot of sense. yeah, they're getting the hardware capabilities. The other thing I would say is local doesn't have to mean on your laptop, especially as an enterprise user. You can host on-prem or even spin up a cloud instance that your enterprise has control over and run these models there and have the same experience. But one thing I do want to emphasize is people are finally having to pay something approaching the real cost of running these models through Amplotic and OpenAI. And hosting locally or hosting on-prem can cost a lot less too. Although if you look at their consumer offerings, it's still insanely cheap, right? So $20 a month. I mean, you'll probably blow through that in a day. Yeah, I mean, especially on the enterprise side, You don't even have to necessarily, you know, there are a bunch of options that don't require high-end NVIDIA GPUs.

Like, you still need those for training, but for inference, for just running models, you can actually, like, as an enterprise, you might already have the capacity lying around already. So another important aspect of what you've been doing is kind of getting people excited about voice and speech, right? So this notion that speech could become a basic computing primitive, you know, not necessarily. It doesn't have to feel like some remote service. Right. So I still remember you demonstrating to me this voice transcription services in a loud coffee shop near Prasida Park. Yes. Right. Yeah. And this was several years ago. The accuracy of this thing was insane. And it was running not even on a laptop, I think. I think it was just running on a Raspberry Pi. That's right, yeah. Yeah, yeah. And so then, tell us about the state of voice as an actual interface similar to a mouse or a keyboard.

Where are we? So, I think a lot of people are skeptical of voice because they've used Siri and Alexa, and it's a very stilted way of communicating. But the question I often ask people when I'm having a kind of conversation about this stuff And I kind of encounter that resistance is You know, when you're sitting on the couch And you're trying to decide what TV show to watch With your significant other next to you Do you text them or do you talk to them? Right And apart from a few Gen Z people Almost everybody says talk to them And so following on from that it's like, okay, well, it's not that people don't like voice interfaces, it's that people don't like bad voice interfaces. So the problem becomes an engineering and a user experience problem of how do we make voice interfaces that are pleasant and efficient to use? And I think once we do that, voice can really take its place as maybe even a more frequent input medium than the keyboard or touchscreen.

Yeah, because I think the general impression of people is that voice and speech are much less deterministic than a keyboard or a touchscreen, right? So although people now interact with their LLMs and misspell things right and left, then it doesn't seem to matter, right? But the impression that people have is, you know, there'll be problems with accents, background noise, phrasing, and so on and so forth. So where are we in terms of all of these things? So we have the solutions available to most of those problems, or at least big improvements. But because there's no commercial incentive to kind of pull them all together, it really hasn't hit the mainstream yet. I feel like we're still dealing with, like, you know, most of the interfaces we use are, like, previous generation. Like, they really suffer. They haven't taken on board the LLM era fully, and they're still suffering from a lot of those problems you listed.

And that's why I think open source can be a really strong lever. So as you have been trying to solve a lot of these interface problems, right? So what are some of the lessons that you've learned? And in fact, maybe some of them might even apply to people who are not working with voice but are just building AI applications, right? So I think one of the things that you mentioned to me years ago is that with voice, maybe you need to start. You don't have to wait until someone has finished talking to start doing work, right? So what are some of these interface lessons? So I think, you know, latency is a massive deal. That is something you just don't get with cloud-based voice interfaces. You're usually waiting at least a couple of seconds, and that's long enough for people to wonder if they were actually heard and understood. Yeah, it's like being on a Zoom call or phone call with someone,

and then there's a boss. Yeah, and they're just, like, sitting there. Yeah, it feels like the other person is just, like, kind of sitting there ignoring you. and we know from like general user experience research that really you want to be in that 200 milliseconds or lower range to really feel responsive like i often ask people to imagine you like press the key on the keyboard and it took two seconds for the key to show up in the window like that would be a very awkward by the way uh this is something that uh people who work with llms also have to deal with but on the kind of the output side right so they want to make sure that the response is not too long otherwise people get impatient right so yeah no absolutely and i do want to mention another thing which i think might be helpful to people thinking just about local ai in general as well is when you're no longer paying per token when you're no longer having to pay to use a voice interface the way you can use these things like the world of

possibilities becomes so much wider you know if you imagine that every app developer had to pay a few cents every time somebody touched an icon on the touch screen in their app you would design apps very differently and a lot of apps wouldn't exist because of that tax so once both llms are kind of like built into the OS or just generally available, you know, with no cost. And also the same with voice. That is not just a kind of an incremental change going from like a cent to zero That just massively increases the number of viable things that you can use those for So I think one I imagine Pete this is one thing where you done a lot of work which is basically designing for failure, right? Because basically in voice, a lot of things can go wrong, right? So what are some of the lessons there in terms of designing your applications with kind of

failure in mind almost right so one of my favorite examples of designing for failure is the use of snippets in google search so in google search you know from the beginning they knew they couldn't always get the number one result right there's a reason that you just don't you didn't hit like you know all the wonderful yeah exactly or i feel lucky or yeah um but what was genius about their approach was they actually let you browse like just scan with your eyes they hoped to get like the right result in the top three most of the time and then you actually narrow it down by looking at the snippets which tell you enough to click on the right one and the same thing for voice the more you can actually read the body language of somebody you know as we're talking you're kind of like you know nodding a little bit to kind of indicate wait a minute uh

i'm only voiced though you can't see me now i can see no no no but in general right so in general well that's and actually that's that's a whole nother part of this that i haven't mentioned i think that having a little camera that's completely locally only i see will be a massive part of this But even without a camera, if you hear supportive noises or if you, you know, if you start giving a response and somebody is like, hmm, or, you know, kind of all of those things that we recognize. And as people, we kind of respond to that and kind of ask more questions versus assuming that we picked the right thing. So really trying to understand all of those cues that we use as people. Some of them are verbal, some of them are body language and visual, but really having them all understood locally and all of those recordings ensure that they never get recorded because you can verify that locally.

Like you don't know what happens to audio and video once you send it up to the cloud. Like it's impossible to audit basically. With local stuff, you can audit. And so I'm hoping that that opens up the chance to use cameras without increasing the risk to users around privacy. So there's the people who work on speech models, and there's been a lot of work in terms of, in fact, one thing that's I think vastly underappreciated is people talk about deep learning having an ImageNet moment. But actually, the first impressive deep learning models were in speech before ImageNet, right? So there's the people working on speech models. They've even talked about end-to-end models. But obviously, we have LLMs. So for the people who don't follow this closely, Pete, what's the relationship between the LLMs and the speech models? Or do people not use LLMs, they just use speech models? So there's been a massive kind of debate and split between what I think of as like LLM maximalists who expect everything to be an end-to-end model.

Machines of loving grace. Yes. Exactly. And I see usually Anthropik, OpenAI, the big AI companies. Well, you know, the hype machine has to be fed, right? Well, the hype machine has to be fed, and specifically, if the future is end-to-end models, then only the people with the most money can actually build and train them. Now, I'm very strongly in what Berkley calls the compound AI camp, where you actually build your applications out of multiple models, each of which has a specialized task. And so there, instead of having a model that takes speech in and outputs speech audio, you actually have a model that takes speech audio in, converts it to text, feeds that into an LLM. The LLM then outputs text, which is converted into speech.

Now you lose, like every time you convert the audio into speech, you actually lose a bunch of expressiveness. it's kind of like a bottleneck of information you can't tell how somebody's you know whether asking it in a kind of a you know particular intonation and all this other stuff um can't you can't you like as you're doing that translation can't you add metadata you kind of can but the whole point of compound ai is that you can take models from different sources and plug them together so you need some kind of like common language and at the moment the the only thing that all llms speak is text input and text output but the advantages are that you can actually train all of the models independently like with the speech end-to-end model if you have a new accent then if it's a true end-to-end model you have to do a whole bunch of training you know as a new input accent if you have new output voices or new capabilities if it's a true end-to-end model

then you're having to train the whole thing if it's not actually if end-to-end is just the marketing term then you can train kind of individual components individual modules separately but then you're always kind of agreeing on some settled interface data like some schema some language because otherwise that's kind of set so even if it's not text even if it's kind of an embedding you still have to kind of have those fixed points so yeah i see end-to-end as much more of a marketing term and something to justify the insane amounts of money that are flowing into these companies what about so for our listeners who've heard of the term multimodal right so Gemini is supposed to be multimodal. And coincidentally, I just spoke to someone who works in the visual reasoning and understanding space, and they said, actually, Gemini is quite poor in terms of understanding video.

So what is the state of these multimodal models for your domain speech? So they're actually often pretty good. Like Quen, 1.7 billion, I think it is, actually does a better job than Whisper, significantly better job on some languages like mandarin and that is a and you know the same goes for gemma the trouble is that as general purpose models like one of the failure cases with gemma is that you upload audio of some speech it transcribes it but it then kind of gets confused by the content of that speech and thinks you're asking it a question say if there was a question in the transcribed speech. And so it's one of these things where you're trying to use a Swiss army knife and it's a lot easier to use just an actual wood saw versus kind of like the thing that can do everything that has a little saw attachment with it. So that's kind of another drawback of the end trend models.

By the way, obviously you're the king on this topic. General purpose models are dead on arrival on the edge, I mean, right now? I mean, some of the really small pieces of hardware that you work on. Absolutely. Like, you know, one of the things I recently did was get a full voice interface where you could connect to Wi-Fi, set the password, and all of this sort of stuff running on a Raspberry Pi Pico microcontroller that only has 500 kilobytes of RAM. Now there, you're never going to have a general purpose model that's going to work well for you. But as always. And by the way, one of the things that you've also worked on, which is interesting, which I don't see a lot of people working on, is kind of taking the browser more seriously. Right. So tell us what's the motivation behind that and what has been the reception? Yeah. So it's tricky because I sometimes think of like, oh, we've done X in the browser.

as hacker news bait because you know that that always gets a lot of attention as kind of a novelty but it's often hard to find practical uses for a lot of these things with with this what i've you know with the voice capabilities with voice input voice output what i seen is that it makes it kind of part of the browser operating system or feel like part of the browser operating system And once you are able to transcribe fast and accurately in the browser it kind of a way for people to easily start experimenting with this stuff And there aren't any collapse for this at the moment. But, you know, I've tried to put a few examples. If you go to moonshine.ai. The whole notion of using the browser as a point of inference, right? So you can imagine other models doing that too, not just yours, which is focused on speech.

You can imagine a browser inference for text kinds of applications, right? Well, and you actually, if you're running Chrome, you've already got a very capable model installed. You've got like a 4 billion parameter model. I actually did a blog post like last year showing how you could just go through Chrome settings and actually start using it as a chatbot, just typing in the window and getting responses. And I think that that is that's why, you know, an example of the models becoming part of the operating system. Because Chrome is essentially an operating system that sits on top of other operating systems. And you can already access that built-in LLM, I think, now from any website. It used to be just extensions, but now I think it's any website from JavaScript can access that. And honestly, again, we're still figuring out what the best use cases are for that. But once it becomes free and you can rely on it being everywhere, You know, I really feel like it's an iPhone moment for a lot of these new LLM technologies.

Or even what you mentioned earlier, which is for people on a laptop, it'll be part of the OS somehow, right? So looking ahead, Pete, and again, for our listeners, you can follow Pete's work at boomshine.ai, and a lot of the work is open source, and you can take advantage of that. So looking ahead, Pete, 12 to 24 months, I hate to put you on the spot. So where are we in terms of, I guess, local or edge models, number one. And number two, just the state of open models in general. Yeah, and I'm always terrible at predictions. But one thing Pete is super good at for our listeners is to know what to work on. So that's an indication. Well, just tell us what you're working on. Yes. so so what i see from like you know from being deep in the technical side you know most most of what i end up working on is because i've been working on something and it becomes blindingly

obvious that the only reasonable technical solution is x and so of course i want to work on x and what i see and what i've seen for the last few years is the gravitational pull of running this stuff locally or on-prem or however you want to call it i often say like ai wants to be local like unlike if you think about something like google search you need massive databases you need armies of servers you need all of this stuff if you look at what claude and chat gpt are doing you can run that entirely on one laptop with no internet connectivity so the shape of these models is perfect for running locally unlike almost everything else we've seen over the last like couple of decades where the cloud has made more sense so what i expect is all of these pressures cost for enterprises especially worries about privacy worries about capabilities going away

worries about changes you know if you've built an application on top of a particular version of clawed, even if you're using the same version in kind of the dropdown, what's actually happening behind the scenes can change from day to day. So if you want stability, you actually need to host your own, at least in your own cloud infrastructure, if not locally on users' laptops. And, you know, I can keep on and on. And then the other thing, too, is I think a lot of enterprises are recognizing that the remote is really not the model it's that compounding loop around the model right so yes if you use the model more you learn more and things like that so why in the world would you send that to a commercial proprietary model right why not do fine-tuning on your own model yeah yeah i mean own that loop yeah exactly uh which and then as you point out then you you actually control your uh timing and

roadmap as well because basically you have the model and you know when it's safe to replace that model with something else no one is going to replace that model underneath you right nobody's going to start charging 10 times as much exactly overnight right right right and then also even if they keep the cost the same they might weaken the model for some reason right yeah yeah and you know they could even be you know pressure from outside so by the way one last question for you pete Because you are obviously, you work in a space where you really care about hardware, right? So right now we're in a world where Jensen Wong seems to own everything. So what's your prediction about the state of AI hardware? Are we going to get more diversification? Is AMD going to up its game? What's going on in hardware in general? So watch Apple. Disclaimer, I worked at Apple for five years. Like in the 2000s, you know, my wife worked there for 13 years.

So, you know, but if you look at what they're doing, they're getting lots of RAM, increasing the RAM speed so that you can actually run models much faster, have their own inference only accelerators. And I think with those three, you don't need a GPU. So just run everything on GPU? Well, run everything on a different kind of accelerator. Oh, okay. That's not a GPU. I see, I see. So CPU could actually be pretty fast. Obviously, the NVIDIA answer is why. We have all the HBM on high bandwidth memory right on the GPU printer. And I like to think of GPUs as, you know, they're designed for large batches and training. I like to think of a GPU as like trying to use an oil tanker to go and do your shopping. You know, they're fantastic for these large training jobs, but they're complete overkill for inference.

Inference has a very different shape to its, you know, to the workload that it places on hardware. One thing that I've kind of started to pay attention lately, and it's very, very early, is a high bandwidth flash. Absolutely. Right. So what's your take? So that's another example. Like I see all of the, I mean, the massive amount of money that NVIDIA is pulling in, that also has its own gravitational pull on all of these hardware companies where they're looking from a technical side. You know, I don't have any special insights. I've just kind of like worked with, I know what the problem workloads look like. Other people are looking at those too, and they're like, oh, I see. We have to stream, all of the weights have to be processed for every token. So if you have something, even if you're running on the CPU, if you have something that's got high enough bandwidth to pull like the 27 billion weights in a fraction of a second, then the rest of it is fairly easy

in terms of like actually doing the processing. I think it's very much memory bound, usually. So it's been awesome seeing, yeah, like the high bandwidth flash, you know, what Apple are doing, increasing kind of like the bandwidth on regular RAM. I think we're going to see a lot of really imaginative solutions now that people understand what the workload looks like. And with that, thank you, Pete. And again, our website is moonshine.ai. You can follow the work of Pete Warden online at PeteWarden.com. Please join the thousands of people who subscribe to our newsletter, which you can find at gradientflow.com slash newsletter. Thanks for joining us. If you like the show, please subscribe and rate us through Apple Podcasts or Overcast or TuneIn.com or Spotify and never miss an episode. The Data Exchange Podcast is a property of Great Inflow.

And I'll be back next week and we will do this all over again.

番組の概要欄(原文)

Pete Warden, co-creator of Moonshine Voice and a longtime deep learning engineer, joins me to discuss why AI increasingly wants to run locally rather than in the cloud. We explore the rapid progress of open-weight models, what it takes to run AI on laptops and in browsers, and how low-latency speech models could make voice a more natural interface. Subscribe to the Gradient Flow Newsletter 📩 https://gradientflow.substack.com/ Subscribe: Apple · Spotify · Overcast · Pocket Casts · Antenn...

出典・この記事について

番組
The Data Exchange with Ben Lorica(Ben Lorica)
元の回
Why AI Wants to Be Local
配信日
要約作成日
公式サイト
thedataexchange.media

この要約・解説記事は、元の音声を文字起こしした内容をもとに AI(Claude)が日本語で作成したものです。 固有名詞や数値に誤りを含む場合があるため、正確な内容は元の音声でご確認ください。制作方法について

X でシェアSpotify で聴くApple Podcasts で聴く

関連エピソード