Design and Evaluation of Unsupervised Machine Learning Models for Anomaly Detection in Streaming Cybersecurity Logs

Sánchez-Zas, Carmen; Larriva-Novo, Xavier; Villagrá, Víctor A.; Rodrigo, Mario Sanz; Moreno, José Ignacio

doi:10.3390/math10214043

Open AccessArticle

Design and Evaluation of Unsupervised Machine Learning Models for Anomaly Detection in Streaming Cybersecurity Logs

by

Carmen Sánchez-Zas

^*

,

Xavier Larriva-Novo

,

Víctor A. Villagrá

,

Mario Sanz Rodrigo

and

José Ignacio Moreno

ETSI Telecomunicación, Universidad Politécnica de Madrid (UPM), Avda. Complutense 30, 28040 Madrid, Spain

^*

Author to whom correspondence should be addressed.

Mathematics 2022, 10(21), 4043; https://doi.org/10.3390/math10214043

Submission received: 29 September 2022 / Revised: 18 October 2022 / Accepted: 24 October 2022 / Published: 31 October 2022

(This article belongs to the Special Issue Models and Algorithms in Cybersecurity)

Download

Browse Figures

Versions Notes

Abstract

Companies, institutions or governments process large amounts of data for the development of their activities. This knowledge usually comes from devices that collect data from various sources. Processing them in real time is essential to ensure the flow of information about the current state of infrastructure, as this knowledge is the basis for management and decision making in the event of an attack or anomalous situations. Therefore, this article exposes three unsupervised machine learning models based on clustering techniques and threshold definitions to detect anomalies from heterogeneous streaming cybersecurity data sources. After evaluation, this paper presents a case of heterogeneous cybersecurity devices, comparing WSSSE, Silhouette and training time metrics for all models, where K-Means was defined as the optimal algorithm for anomaly detection in streaming data processing. The anomaly detection’s accuracy achieved is also significantly high. A comparison with other research studies is also performed, against which the proposed method proved its strong points.

Keywords: machine learning; clustering; real-time; data pre-processing; threshold; Spark; cybersecurity; K-means; anomaly detection; logs

Share and Cite

MDPI and ACS Style

Sánchez-Zas, C.; Larriva-Novo, X.; Villagrá, V.A.; Rodrigo, M.S.; Moreno, J.I. Design and Evaluation of Unsupervised Machine Learning Models for Anomaly Detection in Streaming Cybersecurity Logs. Mathematics 2022, 10, 4043. https://doi.org/10.3390/math10214043

AMA Style

Sánchez-Zas C, Larriva-Novo X, Villagrá VA, Rodrigo MS, Moreno JI. Design and Evaluation of Unsupervised Machine Learning Models for Anomaly Detection in Streaming Cybersecurity Logs. Mathematics. 2022; 10(21):4043. https://doi.org/10.3390/math10214043

Chicago/Turabian Style

Sánchez-Zas, Carmen, Xavier Larriva-Novo, Víctor A. Villagrá, Mario Sanz Rodrigo, and José Ignacio Moreno. 2022. "Design and Evaluation of Unsupervised Machine Learning Models for Anomaly Detection in Streaming Cybersecurity Logs" Mathematics 10, no. 21: 4043. https://doi.org/10.3390/math10214043

APA Style

Sánchez-Zas, C., Larriva-Novo, X., Villagrá, V. A., Rodrigo, M. S., & Moreno, J. I. (2022). Design and Evaluation of Unsupervised Machine Learning Models for Anomaly Detection in Streaming Cybersecurity Logs. Mathematics, 10(21), 4043. https://doi.org/10.3390/math10214043

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Menu

Design and Evaluation of Unsupervised Machine Learning Models for Anomaly Detection in Streaming Cybersecurity Logs

Abstract

Share and Cite

Article Metrics

Article Access Statistics

Further Information

Guidelines

MDPI Initiatives

Follow MDPI