How Anomaly Detection Actually Works in Security Operations

This article explains the technical details behind anomaly detection in security operations, including different statistical models and their applications in analyzing authentication logs.

Details

The article discusses the mathematical underpinnings of anomaly detection, which is often described simplistically as 'learning what normal looks like and alerting on deviations'. It explains three main classes of anomaly detection models: statistical models that fit a distribution to the data and flag points beyond a threshold, isolation-based models that build decision trees to quickly isolate anomalies, and density-based models that identify low-density regions as anomalies. The article then provides a concrete example of using an Isolation Forest model to detect anomalies in authentication logs, highlighting relevant features such as login hour, day of week, source IP geolocation, failed login counts, time since last successful login, and distinct source IPs. The key point is that the choice of anomaly detection model significantly impacts the results, and picking the wrong one for the data structure is a common reason for failure in production.