Reducing Bootstrap Memory Cost in LLM Agents

The article discusses a new approach to managing memory in LLM (Large Language Model) agents to reduce the bootstrap memory cost, which was previously over 3,500 tokens.

Details

LLM agents are stateless by default, so the standard approach to maintain continuity is to load everything into the system prompt, including logs, past decisions, and project state. While this works, it is very wasteful, with the authors spending over 3,500 tokens on memory before the agent could even start doing anything useful. Conversely, loading nothing results in the agent forgetting preferences and repeating mistakes every session. To address this, the authors stopped trying to tune the context window and instead changed how memory is handled. They split memory into three parts: hot (a small set of curated facts always loaded, around 625 tokens), warm (recent logs from the last 7 days, only pulled in when needed), and cold (older history stored externally and not loaded by default). This simple change made a big difference, reducing the bootstrap memory cost from around 3,500 tokens to about 125 tokens, a roughly 96% reduction.