Author(s)

Pierre Dupont, Sophie Martin^*

Affiliation(s)

School of Computer Science, University of Strasbourg, Strasbourg, France. 

Corresponding Author

Sophie Martin

ABSTRACT

This paper presents a deep learning-based approach for detecting anomalies in distributed system logs, addressing the challenges posed by the increasing complexity and volume of log data generated in modern computing environments. Distributed systems, characterized by their decentralized architecture, provide enhanced scalability and fault tolerance, yet they also complicate monitoring and diagnostics due to the sheer amount of log data produced. Traditional anomaly detection methods, including statistical and rule-based approaches, often struggle to keep pace with the dynamic nature of log data and the high dimensionality of the information contained within. In response to these limitations, we propose the use of Long Short-Term Memory networks, a type of recurrent neural network adept at capturing temporal dependencies in sequential data, to effectively identify anomalies in log entries. Our methodology involves systematic data collection from diverse sources, rigorous data preprocessing, and the application of deep learning techniques to develop a robust anomaly detection model. The experimental results demonstrate that our LSTM-based approach significantly outperforms traditional methods, achieving high accuracy, precision, and recall rates in identifying both known and unknown anomalies.

This research contributes to the field by providing a scalable and effective solution for log analysis in distributed systems, ultimately enhancing system reliability and security. Future work will explore the integration of additional deep learning architectures and dynamic thresholding techniques to further improve anomaly detection capabilities.

KEYWORDS

Anomaly detection; Deep learning; Distributed systems

CITE THIS PAPER

Pierre Dupont, Sophie Martin.A deep learning approach for detecting anomalies in distributed system logs. World Journal of Engineering Research. 2024, 2(3): 8-17. DOI: https://doi.org/10.61784/wjer3008.

REFERENCES

[1] Nama P, Pattanayak S, Meka H S. AI-driven innovations in cloud computing: Transforming scalability, resource management, and predictive analytics in distributed systems. International Research Journal of Modernization in Engineering Technology and Science, 2023, 5(12): 4165.

[2] Liu Y, Hu X, Chen S. Multi-Material 3D Printing and Computational Design in Pharmaceutical Tablet Manufacturing. Journal of Computer Science and Artificial Intelligence, 2024.

[3] Singh V K, Govindarasu M. A cyber-physical anomaly detection for wide-area protection using machine learning. IEEE Transactions on Smart Grid, 2021, 12(4): 3514-3526.

[4] Qiu L. DEEP LEARNING APPROACHES FOR BUILDING ENERGY CONSUMPTION PREDICTION. Frontiers in Environmental Research, 2024, 2(3): 11-17.

[5] NG B A, Selvakumar S. Anomaly detection framework for Internet of things traffic using vector convolutional deep learning approach in fog environment. Future Generation Computer Systems, 2020, 113: 255-265.

[6] Wang X, Wu Y C, Zhou M, et al. Beyond surveillance: privacy, ethics, and regulations in face recognition technology. Frontiers in big data, 2024, 7: 1337465.

[7] Zhang X, Li P, Han X, et al. Enhancing Time Series Product Demand Forecasting with Hybrid Attention-Based Deep Learning Models. IEEE Access, 2024.

[8] Eltanbouly S, Bashendy M, AlNaimi N, et al. Machine learning techniques for network anomaly detection: A survey. In 2020 IEEE International Conference on Informatics, IoT, and Enabling Technologies (ICIoT), 2020: 156-162.

[9] Li P, Ren S, Zhang Q, et al. Think4SCND: Reinforcement Learning with Thinking Model for Dynamic Supply Chain Network Design. IEEE Access, 2024.

[10] Rezaee K, Rezakhani S M, Khosravi M R, et al. A survey on deep learning-based real-time crowd anomaly detection for secure distributed video surveillance. Personal and Ubiquitous Computing, 2024, 28(1): 135-151.

[11] Patrikar D R, Parate M R. Anomaly detection using edge computing in video surveillance system. International Journal of Multimedia Information Retrieval, 2022, 11(2): 85-110.

[12] Hosseinzadeh M, Rahmani A M, Vo B, et al. Improving security using SVM-based anomaly detection: issues and challenges. Soft Computing, 2021, 25(4): 3195-3223.

[13] Zhang X, Chen S, Shao Z, et al. Enhanced Lithographic Hotspot Detection via Multi-Task Deep Learning with Synthetic Pattern Generation. IEEE Open Journal of the Computer Society, 2024.

[14] Wang X, Wu Y C, Ji X, et al. Algorithmic discrimination: examining its types and regulatory measures with emphasis on US legal practices. Frontiers in Artificial Intelligence, 2024, 7: 1320277.

[15] Ullah I, Mahmoud Q H. Design and development of a deep learning-based model for anomaly detection in IoT networks. IEEE Access, 2021, 9: 103906-103926.

[16] Wang M. AI Technologies in Modern Taxation: Applications, Challenges, and Strategic Directions. International Journal of Finance and Investment, 2024, 1(1): 42-46.

[17] Sater R A, Hamza A B. A federated learning approach to anomaly detection in smart buildings. ACM Transactions on Internet of Things, 2021, 2(4): 1-23.

[18] Jones R, Davies H. High-performance digital forensic framework for anomalous ransomware detection in file system log data. 2024.

[19] Huong T T, Bac T P, Long D M, et al. Detecting cyberattacks using anomaly detection in industrial control systems: A federated learning approach. Computers in Industry, 2021, 132: 103509.

[20] Mounnan O, Manad O, Boubchir L, et al. A review on deep anomaly detection in Blockchain. Blockchain: Research and Applications, 2024: 100227.

[21] Weinger B, Kim J, Sim A, et al. Enhancing IoT anomaly detection performance for federated learning. Digital Communications and Networks, 2022, 8(3): 314-323.

[22] Liu Y, Ren S, Wang X, et al. Temporal Logical Attention Network for Log-Based Anomaly Detection in Distributed Systems. Sensors, 2024, 24(24): 7949.

[23] Olateju O, Okon S U, Igwenagu U, et al. Combating the challenges of false positives in AI-driven anomaly detection systems and enhancing data security in the cloud. 2024..

[24] Gayam S R. AI-Driven Fraud Detection in E-Commerce: Advanced Techniques for Anomaly Detection, Transaction Monitoring, and Risk Mitigation. Distributed Learning and Broad Applications in Scientific Research, 2020, 6: 124-151.

[25] Soldani J, Brogi A. Anomaly detection and failure root cause analysis in (micro) service-based cloud applications: A survey. ACM Computing Surveys (CSUR), 2022, 55(3): 1-39.

[26] Reddy D K, Behera H S, Nayak J, et al. Deep neural network based anomaly detection in Internet of Things network traffic tracking for the applications of future smart cities. Transactions on Emerging Telecommunications Technologies, 2021, 32(7): e4121.