Building Mini Gravity: A Local, Private Voice AI Agent
The article describes the development of Mini Gravity, a private, high-performance voice agent that runs entirely on the user's local machine, handling documents and generating code.
Why it matters
This project demonstrates the potential for building private, high-performance voice AI agents that can handle a wide range of tasks entirely on the user's local machine.
Key Points
- 1Mini Gravity is designed as a sequential pipeline with three layers: STT (speech-to-text), Intent (natural language classification), and Execution (system actions)
- 2The author initially used Llama 3.2 but pivoted to DeepSeek-Coder-6.7B, realizing that a robust prompt is crucial for the LLM to produce reliable output
- 3The biggest breakthrough was building simple, robust primitives for operations like PDF parsing and file management, which form the backbone of the agent
- 4Challenges included phonetic drift in speech recognition and moving from local subprocess calls to a REST API architecture
Details
Mini Gravity is designed to be a private, local voice AI agent that can handle documents, generate code, and perform various system actions on the user's machine without any data leaving the device. The architecture consists of a three-layer pipeline: the STT (speech-to-text) layer uses Whisper-large-v3 for fast transcription, the Intent layer uses DeepSeek-Coder-6.7B to classify natural language into structured JSON intents, and the Execution layer is a Python engine that maps intents to system actions. The author initially used Llama 3.2 but found the output to be contaminated with conversational filler, leading to a pivot to DeepSeek-Coder-6.7B. The key breakthrough was building robust, simple primitives for operations like PDF parsing and file management, which form the true backbone of the agent. Challenges included phonetic drift in speech recognition and moving from local subprocess calls to a REST API architecture. The author is now exploring prompt tuning and deeper local system integration to make the agent more proactive.
No comments yet
Be the first to comment