Open-source Python tool to detect drift in embedding spaces

The author built an open-source Python package called 'drift-lens-monitor' to detect drift in embedding spaces, which is crucial for modern ML systems that rely on embeddings.

Details

Many modern ML systems, such as semantic search, RAG pipelines, recommenders, and classification pipelines, rely heavily on embeddings. Even when the raw system appears healthy, the underlying embedding space can start changing due to factors like new user behavior, model updates, data source changes, or gradual distribution shift. Monitoring downstream metrics alone often detects these issues late. The author built an open-source Python package called 'drift-lens-monitor' to directly compare snapshots of embeddings over time and detect drift. The package supports three drift detection approaches: FED (Fréchet Embedding Distance, a statistical distance metric), MMD (Maximum Mean Discrepancy, a non-parametric kernel-based method), and persistent homology (which looks at changes in the shape of the embedding space). The tool is designed to be practical, local-first, and easy to use, with snapshots stored as Parquet files for a lightweight and reproducible workflow. The author is interested in feedback on the usefulness of persistent homology, potential baselines or benchmark datasets, and ways to improve the package and API for real-world usage.