The Local AI Delegation Problem: Why Small Models Fail and How to Fix It

This article discusses the challenges of using small AI models in an orchestrated agent framework, including cold start delays, context overhead, and model-specific issues. It provides solutions such as setting infinite keep-alive, implementing a warmup cron pattern, and optimizing context injection.

Details

The article describes the author's experience building an autonomous AI agent framework called OpenClaw, where a main agent (Claude Opus) orchestrates local Ollama models as subagents. They encountered several challenges, including the 'local AI delegation problem' where small models fail to perform as expected. The first issue is the cold start delay of 60-90 seconds when Ollama models are evicted from RAM after 5 minutes of inactivity. This is unacceptable for agent tasks expected to complete in 2-3 minutes. The solution is to set the OLLAMA_KEEP_ALIVE environment variable to -1, which keeps the models pinned in memory. However, this alone is not enough, as models will go cold again if Ollama restarts. The article recommends implementing a warmup cron job to preload the most frequently used models. Another challenge is the significant context overhead of ~100 seconds, where the agent framework injects workspace context, tool definitions, and system prompts before the model even sees the task. This overhead is non-negotiable for safety and coordination, but can be minimized by keeping AGENTS.md, TOOLS.md, and task prompts lean. The article also describes the 'Qwen3 Reasoning Trap', where the Qwen3 model has an internal chain-of-thought process that takes 21 seconds before generating the visible response, causing timeouts in the agent framework.