A flow-based approach for Trickbot banking trojan detection. Issue 84 (July 2019)

• Record Type: Journal Article
• Authors: Gezer, Ali; Warner, Gary; Wilson, Clifford; Shrestha, Prakash
• Is Part Of: Computers & security. Issue 84(2019)
• Page Start: 179
• Page End: 192
• Publication Date: 2019-07
• Subjects: Trickbot -- Banking trojan -- Machine learning -- Anomaly traffic detection -- Dynamic analysis -- Random Forest; Computer security -- Periodicals; Electronic data processing departments -- Security measures -- Periodicals; 005.805
• DOI: 10.1016/j.cose.2019.03.013
• Languages: English
• ISSNs: 0167-4048

Abstract

Abstract: Nowadays, online banking is an attractive way of carrying out financial operations such as ecommerce, e-banking, and e-payments without much effort or the need of any physical presence. This increasing popularity in online banking services and payment systems has created motivation for financial attackers to steal customer`s credentials and money. Banking trojans have been a way of committing attacks on these financial institutions for more than a decade, and they have become one of the primary drivers of botnet traffic. However, the stealthy nature of financial botnets requires new techniques and novel systems for detection and analysis in order to prevent losses and to ultimately take the botnets down. TrickBot, which specifically threatens businesses in the financial sector and their customers, has been behind man-in-the-browser attacks since 2016. Its main goal is to steal online banking information from victims when they visit their banking websites. In this study, we utilize machine learning techniques to detect TrickBot malware infections and to identify TrickBot related traffic flows without having to analyze network packet payloads, the IP addresses, port numbers and protocol information. Since command and control server IPs are updated almost daily, identification of TrickBot related traffic flows without looking at specific IP addresses is significant. We adopt behavior-based classification that uses artifacts created by the malware during the dynamic analysis of TrickBot malware samples. We compare the performance results of four different state-of-the-art machine learning algorithms, Random Forest, Sequential Minimal Optimization, Multilayer Perceptron, and Logistic Model to identify TrickBot related flows and detect a TrickBot infection. Then, we optimize the proposed classifier via exploring the best hyperparameter and feature set selection. Looking at network packet identifiers such as packet length, packet and flag counts, and inter-arrival times, the Random Forest classifier identifies TrickBot related flows with 99.9534% accuracy, 91.7% true positive rate.