As industries evolve and mature, the natural progression is to turn toward the automation of tasks. The computer security space is not immune to this law of business, and consequently, the prevailing trend in software vulnerability detection is toward automation. While tools for detection of vulnerabilities are not new, they have historically performed a grep-style check of the source for known unsafe functions, rather than performing any useful logical analysis of the software.
Currently, analysis is taking the leap into the next generation. We are beginning to see the first generation of auditing tools which transcend source code pattern matching, and perform legitimate logic and flow analysis. In this age when the security of software is finally beginning to become a focus of industry and government, these tools will supplant the efforts of manual auditors when the security of a product must be confirmed. Their consistency and thoroughness will be relied upon heavily.
This paper will discuss several techniques for frustrating the automated analysis and reverse engineering of binaries using techniques at the source code, assembly, and binary level. These techniques will then be demonstrated against such tools as Ida, in order to show that it is possible to hide data not only from simple pattern based tools such as virus scanners, but also from more robust decompilation tools.