Two-Pathway Model for Enhancement of Protocol Reverse Engineering

Abstract

With the continuous emergence of new applications and cyberattacks and their frequent updates, the need for automatic protocol reverse engineering is gaining recognition. Although several methods for automatic protocol reverse engineering have been proposed, each method still faces major limitations in extracting clear specifications and in its universal application. In order to overcome such limitations, we propose an automatic protocol reverse engineering method using a two-pathway model based on a contiguous sequential pattern (CSP) algorithm. By using this model, the method can infer both command-oriented protocols and non-command-oriented protocols clearly and in detail. The proposed method infers all the key elements of the protocol, which are syntax, semantics, and finite state machine (FSM), and extracts clear syntax by defining fine-grained field types and three types of format: field format, message format, and flow format. We evaluated the efficacy of the proposed method over two non-command-oriented protocols and three command oriented protocols: the former are HTTP and DNS, and the latter are FTP, SMTP, and POP3. The experimental results show that this method can reverse engineer with high coverage and correctness rates, more than 98.5% and 99.1% respectively, and be general for both command-oriented and non-command-oriented protocols.