Latent Variable Based Anomaly Detection in Network System Logs

• OTOMO Kazuki

  Graduate School of Information Science and Technology, The University of Tokyo

• KOBAYASHI Satoru

  National Institute of Informatics

• FUKUDA Kensuke

  National Institute of Informatics Department of Informatics, Sokendai

• ESAKI Hiroshi

  Graduate School of Information Science and Technology, The University of Tokyo

Abstract

System logs are useful to understand the status of and detect faults in large scale networks. However, due to their diversity and volume of these logs, log analysis requires much time and effort. In this paper, we propose a log event anomaly detection method for large-scale networks without pre-processing and feature extraction. The key idea is to embed a large amount of diverse data into hidden states by using latent variables. We evaluate our method with 12 months of system logs obtained from a nation-wide academic network in Japan. Through comparisons with Kleinberg's univariate burst detection and a traditional multivariate analysis (i.e., PCA), we demonstrate that our proposed method achieves 14.5% higher recall and 3% higher precision than PCA. A case study shows detected anomalies are effective information for troubleshooting of network system faults.

Journal

• IEICE Transactions on Information and Systems

  IEICE Transactions on Information and Systems E102.D (9), 1644-1652, 2019-09-01

  The Institute of Electronics, Information and Communication Engineers

Keywords

• network operation
• system logs
• syslog
• anomaly detection
• latent variable analysis
• variational autoencoder

Details

• CRID: 1390845702274659328
• NII Article ID: 130007699784
• DOI: 10.1587/transinf.2018ofp0007
• ISSN: 17451361
• ISSN: 09168532
• Web Site: https://www.jstage.jst.go.jp/article/transinf/E102.D/9/E102.D_2018OFP0007/_pdf
• Text Lang: en
• Data Source:
  • JaLC
  • Crossref
  • CiNii Articles
  • KAKEN
• Abstract License Flag: Disallowed