Building an AI Assistant Taught Us to Move from RAG to a 'Memory Layer'

The article discusses the challenges the authors faced while building a proactive AI assistant and how they shifted their approach from a traditional retrieval-based (RAG) pipeline to a structured 'memory layer' that extracts meaning, connects concepts, and maintains consistency.

Details

The authors were building a proactive AI assistant that could perform tasks like replying to emails, managing calendars, and surfacing relevant information. They started with a traditional RAG (Retrieval, Annotation, Generation) pipeline, which worked to an extent. However, as the assistant became more complex, the team realized that the system could retrieve relevant information but didn't truly understand the connections between concepts. Everything was based on similarity, not structure. To address this, they built a 'brain' layer that processes data into a structured representation of entities and relationships, stored in a graph and vector model. This allows querying the data based on entities, relationships, and context, rather than just keyword similarity. The authors found this approach particularly useful when the context spans multiple sources, relationships matter more than keywords, and consistency is important. They are exploring 'polarities' - surfacing a range of possible solutions based on how concepts relate in the graph, rather than just returning a single 'best' answer. The team believes that this 'memory layer' is a crucial component for building AI systems that people can rely on, as current systems are powerful but fragile in how they represent knowledge. They have open-sourced the core of this approach as BrainAPI, which can run locally or be deployed as a managed service.