STROP: Static Approach for Detection of Return-Oriented Programming Attack in Network

Abstract

Recently, a malicious user attacks a web browser through a malicious page that exploits the vulnerability of the browser and that executes malicious code. To prevent this attack, some methods have been devised such as DEP (Data Execution Prevention) that prevents data in stack frame or heap region from being executed. However, to evade these defense techniques, return-oriented programming (ROP) is introduced. ROP executes arbitrary code indirectly using gadget, which is group of instructions including ret instruction in a module that doesn't apply ASLR (Address Space Layout Randomization). In this paper, we propose a static approach to detect ROP payload in a network irrespective of the environment of the system under attack. Most studies have tried to detect ROP attacks using dynamic analysis, because ROP has various addresses of gadgets according to loaded modules. These methods have a limitation that must consider the environment of system to operate ROP, such as the version of OS and modules including gadgets. To overcome this limitation, our method detects ROP payload using static analysis without preliminary knowledge about the environment. We extract five characteristics of ROP and then propose a novel algorithm, STROP, to detect ROP in payload without execution. Our idea is as follows: STROP makes stack frame using input payload statically. It extracts addresses suspected as indicating gadgets and makes groups using the addresses. And then, STROP determine whether the payload includes ROP based on static characteristics. We implement a prototype using snort (network-based intrusion system) and evaluate it. Experiments show that our technique can detect ROP payload with a low number of false alarms. False positive (FP) is 1.3% for 2,239 benign files and 0.05-0.51% for 1GB packet dump file. Among 68 ROP payloads, STROP detects 51 payloads. This research can be applied to existing systems that collect malicious codes, such as Honeypot.