VOXEN - A Voice-Controlled Local AI Agent

The article describes the development of VOXEN, a voice-controlled AI agent that can transcribe audio, detect user intent, and execute various tasks like writing code, creating files, and summarizing text. The author discusses the architecture, design choices, and challenges faced during the project.

Details

The VOXEN project aims to create a voice-controlled AI agent that can transcribe audio, detect user intent, and execute various tasks like writing code, creating files, and summarizing text. The author divided the project into focused modules for transcription (stt.py), intent detection (intent.py), and task execution (tools.py), with a Streamlit-based UI (app.py) to tie everything together. Due to hardware limitations on the author's local machine, they opted to use the Groq API for transcription (using the Whisper model) and language model inference (using the llama3-8b-8192 model) instead of running these models locally. This allowed for faster and more reliable performance, although it meant the solution was not entirely self-contained. One of the key challenges was parsing the LLM's output to extract clean, structured JSON that could be used to determine the user's intent and execute the appropriate task. The author had to implement a layered approach to strip markdown fences, find the JSON object, and parse it, with a fallback to treating the output as a general chat if the parsing failed. The Streamlit-based UI was designed to feel like a polished product, with custom CSS, gradient hero sections, glassmorphism-style result cards, and an animated SVG logo. The author also implemented a workaround to create custom input buttons that trigger the underlying Streamlit logic, as Streamlit does not natively support this functionality. Overall, the VOXEN project demonstrates the author's ability to tackle the challenges of building a voice-controlled AI agent, from architectural design to UI implementation and handling edge cases like inconsistent LLM output and environment-specific microphone issues.