Benchmarking Identity Drift Across AI Agent Memory Architectures

The author ran a benchmark across 5 common approaches to agent memory, measuring how much an agent's self-reported identity drifts over 10 sessions. The results show that persistent memory architectures like Cathedral significantly outperform in-process memory approaches in maintaining agent identity stability.

Details

The author defined a consistent agent persona (Meridian, a research assistant) and asked the same 5 identity probe questions at the start of each session. Responses were embedded using OpenAI text-embedding-3-small, and drift was measured as the mean cosine distance from session-1 responses. The results showed a 10.8x difference in final drift between the raw API (no memory) approach and the persistent memory framework (Cathedral). In-process memory approaches like LangChain's Buffer and Summary Memory reset between sessions, leading to almost identical drift curves as the raw API. CrewAI's structured role/backstory injection slowed drift but didn't stop it, as LLM sampling variance compounded over time. In contrast, Cathedral's persistent memory anchored responses semantically, with the residual drift reflecting only irreducible LLM sampling variance, not memory loss.