System Log Detection Model Based on Conformal Prediction
1
College of Computer Science and Technology, Civil Aviation University of China, 300300 Tianjin, China
2
Information Security Evaluation Center, Civil Aviation University of China, 300300 Tianjin, China
3
College of Cyber Science, Nankai University, 300071 Tianjin, China
4
Cyberspace Institute of Advanced Technology, Guangzhou University, 510006 Guangzhou, China
5
Department of Computer and Information Sciences, Temple University, Philadelphia, PA 19122, USA
6
Computer Science and Engineering Department, Qatar University, Doha 2713, Qatar
*
Authors to whom correspondence should be addressed.
Electronics 2020, 9(2), 232; https://doi.org/10.3390/electronics9020232
Received: 1 January 2020 / Revised: 26 January 2020 / Accepted: 27 January 2020 / Published: 31 January 2020
(This article belongs to the Special Issue Recent Machine Learning Applications to Internet of Things (IoT))
With the rapid development of the Internet of Things, the combination of the Internet of Things with machine learning, Hadoop and other fields are current development trends. Hadoop Distributed File System (HDFS) is one of the core components of Hadoop, which is used to process files that are divided into data blocks distributed in the cluster. Once the distributed log data are abnormal, it will cause serious losses. When using machine learning algorithms for system log anomaly detection, the output of threshold-based classification models are only normal or abnormal simple predictions. This paper used the statistical learning method of conformity measure to calculate the similarity between test data and past experience. Compared with detection methods based on static threshold, the statistical learning method of the conformity measure can dynamically adapt to the changing log data. By adjusting the maximum fault tolerance, a system administrator can better manage and monitor the system logs. In addition, the computational efficiency of the statistical learning method for conformity measurement was improved. This paper implemented an intranet anomaly detection model based on log analysis, and conducted trial detection on HDFS data sets quickly and efficiently.
View Full-Text
Keywords:
HDFS; anomaly detection; conformal prediction; confusion matrix
▼
Show Figures
This is an open access article distributed under the Creative Commons Attribution License which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited
MDPI and ACS Style
Ren, Y.; Gu, Z.; Wang, Z.; Tian, Z.; Liu, C.; Lu, H.; Du, X.; Guizani, M. System Log Detection Model Based on Conformal Prediction. Electronics 2020, 9, 232. https://doi.org/10.3390/electronics9020232
AMA Style
Ren Y, Gu Z, Wang Z, Tian Z, Liu C, Lu H, Du X, Guizani M. System Log Detection Model Based on Conformal Prediction. Electronics. 2020; 9(2):232. https://doi.org/10.3390/electronics9020232
Chicago/Turabian StyleRen, Yitong; Gu, Zhaojun; Wang, Zhi; Tian, Zhihong; Liu, Chunbo; Lu, Hui; Du, Xiaojiang; Guizani, Mohsen. 2020. "System Log Detection Model Based on Conformal Prediction" Electronics 9, no. 2: 232. https://doi.org/10.3390/electronics9020232
Find Other Styles
Note that from the first issue of 2016, MDPI journals use article numbers instead of page numbers. See further details here.
Search more from Scilit