A cloud/edge computing streaming system for network traffic monitoring and threat detection. (2018)

• Record Type: Journal Article
• Authors: Chen, Zhijiang; Wei, Sixiao; Yu, Wei; Nguyen, James H.; Hatcher, William Grant
• Is Part Of: International journal of security and networks. Volume 13:Number 3(2018)
• Page Start: 169
• Page End: 186
• Publication Date: 2018
• Subjects: streaming analysis -- network traffic monitoring -- threat detection -- big network data analysis -- cloud computing -- edge computing; Computer networks -- Security measures -- Periodicals; 005.805
• Languages: English
• ISSNs: 1747-8405

Abstract

The unyielding trend of increasing cyber threats has made cyber security paramount in protecting personal and private intellectual property. To provide a highly secured network environment, network threat detection systems must handle real-time big data from varied places in enterprise networks. In this paper, we introduce a streaming-based threat detection system that can rapidly analyse highly intensive network traffic data in real-time, utilising streaming-based clustering algorithms to detect abnormal network activities. The developed system integrates the high-performance data analysis capabilities of Flume, Spark and Hadoop into a cloud-computing environment to provide network monitoring and intrusion detection. Our performance evaluation validates that the developed system can cope with a significant volume of streaming data in a high detection accuracy and good system performance. We further extend our system for edge computing and discuss some key challenges, as well as some potential solutions, aiming to improve the scalability of our system.