Bit‐oriented format extraction approach for automatic binary protocol reverse engineering. Issue 6 (1st April 2016)

• Record Type: Journal Article
• Authors: Tao, Siyu; Yu, Hongyi; Li, Qing
• Is Part Of: IET communications. Volume 10:Issue 6(2016)
• Page Start: 709
• Page End: 716
• Publication Date: 2016-04-01
• Subjects: reverse engineering -- Bayes methods; bit‐oriented format extraction approach -- automatic binary protocol reverse engineering -- protocol message format extraction -- target protocol specifications -- text‐based protocols -- binary‐type fields -- binary protocols -- fine grained bits -- silhouette coefficient -- optimal clustering number -- back‐tracing rules -- binary field features -- Bayes decision model -- bit‐oriented field boundaries -- optimal protocol format estimation -- binary field boundaries; Telecommunication systems -- Periodicals; Speech processing systems -- Periodicals; 621.38205
• DOI: 10.1049/iet-com.2015.0797
• Languages: English
• ISSNs: 1751-8628

Abstract

Abstract : Protocol message format extraction is a principal process of automatic network protocol reverse engineering when target protocol specifications are not available. However, binary protocol reverse engineering has been a new challenge in recent years for approaches that traditionally have dealt with text‐based protocols rather than binary protocols. In this study, the authors propose a novel approach called PRE‐Bin that automatically extracts binary‐type fields of binary protocols based on fine‐grained bits. First, a silhouette coefficient is introduced into the hierarchical clustering to confirm the optimal clustering number of binary frames. Second, a modified multiple sequence alignment algorithm, in which the matching process and back‐tracing rules are redesigned, is also proposed to analyse binary field features. Finally, a Bayes decision model is invoked to describe field features and determine bit‐oriented field boundaries. The maximum a posteriori criterion is leveraged to complete an optimal protocol format estimation of binary field boundaries. The authors implemented a prototype system of PRE‐Bin to infer the specification of binary protocols from actual traffic traces. Experimental results indicate that PRE‐Bin effectively extracts binary fields and outperforms the existing algorithms.