Building a CLI Data Quality Tool with AI-Powered Insights

The article discusses the development of SageScan, a CLI tool that goes beyond schema checks to perform statistical validation on data using a YAML config. It highlights key features like distribution drift detection, outlier identification, and categorical drift analysis.

Details

SageScan is a CLI tool that runs statistical validation on data using a YAML configuration file. Unlike traditional data quality tools that rely on manually defined rules, SageScan checks if the data behaves like it used to. It uses techniques like the Kolmogorov-Smirnov (KS) test to detect distribution drift, Z-score and IQR for outlier detection, and the Population Stability Index (PSI) to quantify changes in column distributions. The tool also includes a Chi-square test to detect changes in categorical data. The author discusses the architecture choice of using Go for the fast, portable CLI and Python for the data processing engine with libraries like pandas and scipy. This approach allowed for faster development compared to rewriting everything in a single stack. The article also mentions an optional AI layer that provides possible root causes when a check fails, but emphasizes that the AI does not replace the statistical checks and is only meant to provide additional context. The author shares some lessons learned, such as the need to use a faster data processing library like Polars and the importance of building connectors for popular data sources like Postgres and Snowflake earlier in the development process.