Improving malware detection using big data and ensemble learning. (September 2020)

• Record Type: Journal Article
• Authors: Gupta, Deepak; Rani, Rinkle
• Is Part Of: Computers & electrical engineering. Volume 86(2020)
• Publication Date: 2020-09
• Subjects: Apache Spark -- Big data -- Ensemble learning -- Malware detection -- Stacking -- Weighted voting; Computer engineering -- Periodicals; Electrical engineering -- Periodicals; Electrical engineering -- Data processing -- Periodicals; Ordinateurs -- Conception et construction -- Périodiques; Électrotechnique -- Périodiques; Électrotechnique -- Informatique -- Périodiques; Computer engineering; Electrical engineering; Electrical engineering -- Data processing; Periodicals; Electronic journals; 621.302854
• DOI: 10.1016/j.compeleceng.2020.106729
• Languages: English
• ISSNs: 0045-7906

Abstract

Highlights: Big data and ensemble learning based solution is proposed for malware detection. Weights are assigned to the base classifiers using ranking algorithms and later, used in weighted voting and selection of an optimal set of classifiers for stacking. The proposed weighted voting scheme provides better performance of malware detection at a large scale than traditional ensemble methods. Abstract: Malware detection and classification play a critical role in computer and network security. Although, many machine learning models have been used in the detection of malicious binaries, however, the performance of ensemble methods has not been investigated extensively. Besides, the massive volume of malware has established it as a big data problem forcing security researchers and practitioners to deploy big data technologies to manage, store, analyze, and visualize malware data. In this paper, the authors have designed two methods based on ensemble learning and big data for improving the performance of malware detection at a large scale. The first method is based on the weighted voting strategy of ensemble learning, and the second method chooses an optimal set of base classifiers for stacking purpose. The proposed methods are implemented using Apache Spark, a popular big data processing framework, and their performance is tested and evaluated on a dataset of 198, 350 Windows files including 100, 200 malicious and 98, 150 benign samples. The experimental results successfully validate the effectiveness of the proposed approach since it improves the generalization performance in detecting new malware. Graphical abstract: Image, graphical abstract