Optimizing VRAM Management on Apple Silicon for AI Agents
The author shares their experience of crashing an AI agent system due to VRAM management issues on an Apple Silicon MacBook Pro, and the steps they took to fix the problem.
Why it matters
Effective VRAM management is crucial for running AI systems, especially on resource-constrained hardware like Apple Silicon. This article provides a practical example of how to optimize VRAM usage and prevent system crashes.
Key Points
- 1Parallel loading of multiple large language models (LLMs) caused a VRAM spike, leading to OOM kills and a non-functional agent fleet
- 2The root cause was lack of resource awareness, with models being loaded simultaneously without consideration for available VRAM
- 3The solution involved sequential loading of models with a delay in between, reducing the maximum number of loaded models, and staggering cron jobs to avoid VRAM contention
Details
The author was running an autonomous AI agent system on an Apple Silicon MacBook Pro with 36GB of unified memory. The setup involved a main agent that delegated tasks to subagents running on different LLMs. The author's warmup routine loaded four models simultaneously every 4 minutes to keep them 'hot' in memory, which resulted in a 23.5GB VRAM spike that left insufficient headroom for the operating system and other processes. This led to OOM kills, hung processes, and a non-functional agent fleet. The root cause was identified as parallel loading of models without resource awareness. The solution involved sequential loading of models with a 2-second delay in between, reducing the number of loaded models from 4 to 3, and staggering cron jobs to avoid VRAM contention. An environment variable was also set to limit the maximum number of loaded models to 3, with the least-recently-used model being automatically evicted when a 4th was requested.
No comments yet
Be the first to comment