Bypassing Anti-Analysis of Commercial Protector Methods Using DBI Tools

Abstract

As most malware is infectious, anti-analysis and packing techniques supported by commercial protectors are conventionally applied to hinder analysis. When analyzing to detect and block such protected malware, it is necessary to do so in a virtual environment to prevent infection. In terms of packing, it is necessary to analyze using dynamic binary instrumentation (DBI), a dynamic analysis tool, which is advantageous for unpacking because DBI inserts code at run time and analyzes it dynamically. However, malware terminates on its own when it detects a virtual environment or DBI due to anti-analysis techniques. Therefore, it is necessary to also bypass anti-VM and anti-DBI techniques in order to successfully analyze malware in a virtual environment using DBI. It is very difficult for analysts to bypass anti-VM and anti-DBI techniques that are used in commercial protectors because analysts generally have little information on what methods are used or how to even bypass these techniques. In this paper, we suggest guidelines to aid in easy analysis of malware protected by anti-VM and anti-DBI techniques supported by commercial protectors. We analyzed the techniques used by five of the most common commercial protectors, and herein present how to bypass anti-VM and anti-DBI techniques supported by commercial protectors via a detailed algorithm analysis. We performed a bypass experiment after applying each commercial protector to 1573 executable files containing vulnerabilities provided by the National Institute of Standards and Technology (NIST). To our knowledge, this is the first empirical study to suggest detailed bypassing algorithms for anti-VM and anti-DBI techniques used in commercial protectors.