Using K-means clustering to detect anomalous file removes

There are a number of ways that data can be lost from an archive, including accidental or malicious deletion of data. While there is a lot of software that can check for specific known threats or problems on a system, detecting non-specific anomalous behavior, such as unusual file removal patterns, is harder.

One approach to detecting this kind of problem is machine learning. Machine learning algorithms can build a statistical model of what constitutes normal behavior and then flag data points that are outliers. To help protect the 87 petabytes of data in the National Center for Atmospheric Research's data archive, we explored our file removal patterns and implemented a k-means clustering solution to detect anomalous file removes.