Always On Memory for AI Agents Without Vector DBs

This article discusses a new open-source project that challenges the traditional vector database approach to providing reliable memory for AI agents, instead leveraging the LLM itself as the primary reasoning layer over stored context.

Details

The Always On Memory Agent project from a Google product manager challenges the traditional approach of using vector databases to provide reliable memory for AI agents. Instead of offloading memory retrieval to a dedicated vector store, the agent leans into the LLM itself as the primary reasoning layer over stored context. The core insight is that modern long-context models are increasingly capable of doing meaningful recall and synthesis when given well-structured, continuously updated context windows, without the operational overhead of a separate retrieval infrastructure. This matters because vector databases, while powerful, introduce real friction in terms of managing index freshness, handling embedding model versioning, tuning retrieval parameters, and dealing with semantic drift. For many production agent use cases, that complexity is simply not worth it. The timing of this open-source release is significant, as autonomous agent frameworks have struggled with the challenge of providing persistent memory that can remember prior interactions, user preferences, and task history. The Always On Memory approach signals that this problem is being taken seriously at the infrastructure level, not just as a demo feature. However, the approach does have limitations, such as relying on the LLM's context window, which may require careful curation of what gets passed into context for agents handling large amounts of data. There are also trade-offs in terms of searchability compared to vector databases.