Windows Defender Antivirus' mpengine.dll implements the core of Defender Antivirus' functionality in an enormous ~11 MB, 35,000+ function DLL.
This year at REcon Brussels I presented on Defender's proprietary JavaScript engine for emulation of potentially malicious JS. In this new presentation, we’ll look at Defender’s emulator for analysis of potentially malicious Windows PE binaries on the endpoint. To the best of my knowledge, there has never been a conference talk or publication on reverse engineering the internals of any antivirus binary emulator before.
I’ll cover a range of topics on emulator internals and reverse engineering, including building instrumentation to observe and debug the emulator as it runs; the virtual environment (settings, environment variables, file system, registry, etc); bytecode to intermediate language lifting and execution; memory management; usermode Windows API emulation; NT kernel emulation; file system and registry emulation; and integration with Defender's antivirus features.