Impact of benign sample size on binary classification accuracy. (January 2023)

• Record Type: Journal Article
• Authors: Mimura, Mamoru
• Is Part Of: Expert systems with applications. Volume 211(2023)
• Publication Date: 2023-01
• Subjects: Malware -- Machine learning -- Binary classification -- Benign sample -- Random forest -- Support vector machine -- XGBoost; Expert systems (Computer science) -- Periodicals; Systèmes experts (Informatique) -- Périodiques; Electronic journals; 006.33
• DOI: 10.1016/j.eswa.2022.118630
• Languages: English
• ISSNs: 0957-4174

Abstract

Abstract: Recently, there has been a significant increase in malware attacks and malicious traffic. Consequently, several machine learning-based detection models have been developed to detect them. However, the detection accuracy of these models is currently evaluated using different methodologies and datasets, with some studies overstating high detection rates. The lack of a common testing approach coupled with the limited datasets used for the experiments make it challenging to compare the performances of these models to identify those that provide superior detection accuracy. A few studies have focused on benign samples and their effects on detection accuracy. The datasets used in the experiments generally consist of benign and malicious samples; hence, binary classification is used in the machine learning models. In the binary classification task, the size of a benign sample affects the classification accuracy of malicious samples, that is, it can either improve or degrade detection accuracy. In this study, we propose a novel metric for evaluating accuracy degradation by increasing benign sample size. We mainly used the FFRI dataset, which consists of 11, 243 malware samples and 250, 000 benign samples, and evaluated the classification accuracy with extracted strings from the malware. In addition, we obtained other malware samples that we used as supplementary to the main dataset. We increased the number of benign samples for testing by tenfold, while maintaining the malicious sample and benign training sample sizes, which resulted in a decrease of 0.293 in the F1 score. Furthermore, we confirmed that using a sufficiently sized benign training sample set mitigates accuracy degradation. Our metric can be beneficial for evaluating the benign sample size needed in binary classification and comparing accuracy. Highlights: We propose a metric for accuracy degradation by increasing benign samples. Increasing the test benign sample size tenfold decreased the F1 score by 0.293. Using sufficient benign training samples mitigates accuracy degradation.