Fine-tune LLMs and deploy them on your phone

A new technology called ExecuTorch allows users to fine-tune large language models (LLMs) and deploy them directly on their mobile devices, providing privacy, instant responses, and offline capabilities.

Details

The article discusses a new technology called ExecuTorch that enables users to fine-tune large language models (LLMs) and deploy them directly on their mobile devices. This allows for privacy-first, instant responses and offline capabilities. The technology is said to be based on the same infrastructure that powers billions of users on Meta's platforms like Instagram and WhatsApp. Specifically, the article mentions that users can fine-tune a 0.6B parameter model called Qwen3 and deploy it on devices like the Pixel 8 and iPhone 15 Pro, achieving around 40 tokens per second. Additionally, the article notes that Quantization-Aware Training (QAT) via TorchAO can be used to recover 70% of the model's original accuracy, enabling efficient deployment on mobile devices.