Applying unsupervised system-call based software security techniques for anomaly detection. Issue 5 (4th July 2022)

• Record Type: Journal Article
• Authors: Kishore, Pushkar; Nayak, Gayatri; Barisal, Swadhin Kumar
• Is Part Of: Journal of information & optimization sciences. Volume 43:Issue 5(2022)
• Page Start: 915
• Page End: 922
• Publication Date: 2022-07-04
• Subjects: 68M25; System call -- Unsupervised machine learning -- Anomaly detection system -- Adversary attack; Electronic data processing -- Periodicals; Information science -- Periodicals; Mathematical optimization -- Periodicals; 519.6
• DOI: 10.1080/02522667.2022.2091096
• Languages: English
• ISSNs: 0252-2667

Abstract

Abstract: System call analysis is a technique intended for detecting malware. The above method helps in achieving better detection accuracy. Thus, machine learning (ML) techniques are used for this evaluation. This paper discusses unsupervised ML techniques to detect malware. Our proposed detector monitors the software and marks them anomalous or benign based on their behavior. Experimental results provide performance statistics based on the true positive rate at a low false positive rate. As we got considerable accuracy in some models, there is scope for designing an anomaly detection system centered on unsupervised learning. We illustrated how models performed against various malware samples when executed on benign hosts and testbeds. We included a case study to mitigate the adversary attack on the anomaly detection system.