CTF Reverse Engineering Quick reference for RE challenges. For detailed techniques, see supporting files. Additional Resources tools.md - Tool-specific commands (GDB, Ghidra, radare2, IDA, RISC-V with Capstone) patterns.md - Core binary patterns: custom VMs, anti-debugging, nanomites, self-modifying code, XOR ciphers, mixed-mode stagers, LLVM obfuscation, S-box/keystream, SECCOMP/BPF, exception handlers, memory dumps, byte-wise transforms, x86-64 gotchas, hidden emulator opcodes, LD_PRELOAD key extraction, SPN static extraction, image XOR smoothness, byte-at-a-time cipher, mathematical convergence bitmap, Sprague-Grundy game theory, kernel module maze solving, multi-threaded VM channels, multi-layer self-decrypting brute-force languages.md - Language/platform-specific: Python bytecode & opcode remapping, Python version-specific bytecode, DOS stubs, Unity IL2CPP, Brainfuck/esolangs, UEFI, transpilation to C, code coverage side-channel, OPAL functional reversing, non-bijective substitution Problem-Solving Workflow Start with strings extraction - many easy challenges have plaintext flags Try ltrace/strace - dynamic analysis often reveals flags without reversing Map control flow before modifying execution Automate manual processes via scripting (r2pipe, Python) Validate assumptions by comparing decompiler outputs Quick Wins (Try First!)

Plaintext flag extraction

strings binary | grep -E "flag{|CTF{|pico" strings binary | grep -iE "flag|secret|password" rabin2 -z binary | grep -i "flag"

Dynamic analysis - often captures flag directly

ltrace ./binary strace -f -s 500 ./binary

Hex dump search

xxd binary | grep -i flag

Run with test inputs

./binary AAAA echo "test" | ./binary Initial Analysis file binary

Type, architecture

checksec --file = binary

Security features (for pwn)

chmod +x binary

Make executable

Memory Dumping Strategy Key insight: Let the program compute the answer, then dump it. gdb ./binary start b *main+0x198

Break at final comparison

run

Enter any input of correct length

x/s $rsi

Dump computed flag

x/38c $rsi

As characters

Decoy Flag Detection Pattern: Multiple fake targets before real check. Identification: Look for multiple comparison targets in sequence Check for different success messages Trace which comparison is checked LAST Solution: Set breakpoint at FINAL comparison, not earlier ones. GDB PIE Debugging PIE binaries randomize base address. Use relative breakpoints: gdb ./binary start

Forces PIE base resolution

b *main+0xca

Relative to main

run Comparison Direction (Critical!) Two patterns: transform(flag) == stored_target - Reverse the transform transform(stored_target) == flag - Flag IS the transformed data! Pattern 2 solution: Don't reverse - just apply transform to stored target. Common Encryption Patterns XOR with single byte - try all 256 values XOR with known plaintext ( flag{ , CTF{ ) RC4 with hardcoded key Custom permutation + XOR XOR with position index ( ^ i or ^ (i & 0xff) ) layered with a repeating key Quick Tool Reference

Radare2

r2 -d ./binary

Debug mode

aaa

Analyze

afl

List functions

pdf @ main

Disassemble main

Ghidra (headless)

analyzeHeadless project/ tmp -import binary -postScript script.py

IDA

ida64 binary

Open in IDA64

Binary Types Python .pyc import marshal , dis with open ( 'file.pyc' , 'rb' ) as f :

Header size varies by Python version:

8 bytes (2.x), 12 (3.0-3.6), 16 (3.7+)

f . read ( 16 )

16 for Python 3.7+; adjust for older versions

code

marshal . load ( f ) dis . dis ( code ) WASM wasm2c checker.wasm -o checker.c gcc -O3 checker.c wasm-rt-impl.c -o checker

WASM patching (game challenges):

wasm2wat main.wasm -o main.wat

Binary → text

Edit WAT: flip comparisons, change constants

wat2wasm main.wat -o patched.wasm

Text → binary

WASM game patching (Tac Tic Toe, Pragyan 2026): If proof generation is independent of move quality, patch minimax (flip i64.lt_s → i64.gt_s , change bestScore sign) to make AI play badly while proofs remain valid. See ctf-misc games-and-vms.md for full pattern. Android APK apktool d app.apk -o decoded/

Best - decodes resources

jadx app.apk

Decompile to Java

grep -r "flag" decoded/res/values/strings.xml .NET dnSpy - debugging + decompilation ILSpy - decompiler Packed (UPX) upx -d packed -o unpacked Anti-Debugging Bypass Common checks: IsDebuggerPresent() (Windows) ptrace(PTRACE_TRACEME) (Linux) /proc/self/status TracerPid Timing checks Bypass: Set breakpoint at check, modify register to bypass conditional. S-Box / Keystream Patterns Xorshift32: Shifts 13, 17, 5 Xorshift64: Shifts 12, 25, 27 Magic constants: 0x2545f4914f6cdd1d , 0x9e3779b97f4a7c15 Custom VM Analysis Identify structure: registers, memory, IP Reverse executeIns for opcode meanings Write disassembler mapping opcodes to mnemonics Often easier to bruteforce than fully reverse Look for the bytecode file loaded via command-line arg See patterns.md for VM workflow, opcode tables, and state machine BFS. Python Bytecode Reversing XOR flag checkers with interleaved even/odd tables are common. See languages.md for bytecode analysis tips and reversing patterns. Signal-Based Binary Exploration Binary uses UNIX signals as binary tree navigation; hook sigaction via LD_PRELOAD , DFS by sending signals. See patterns.md . Malware Anti-Analysis Bypass via Patching Flip JNZ / JZ (0x75/0x74), change sleep values, patch environment checks in Ghidra ( Ctrl+Shift+G ). See patterns.md . Expected Values Tables Locating: objdump -s -j .rodata binary | less

Look near comparison instructions

Size matches flag length

x86-64 Gotchas Sign extension and 32-bit truncation pitfalls. See patterns.md for details and code examples. Iterative Solver Pattern for pos in range ( flag_length ) : for c in range ( 256 ) : computed = compute_output ( c , current_state ) if computed == EXPECTED [ pos ] : flag . append ( c ) update_state ( c , computed ) break Uniform transform shortcut: if changing one input byte only changes one output byte, build a 0..255 mapping by repeating a single byte across the whole input, then invert. Unicorn Emulation (Complex State) from unicorn import * from unicorn . x86_const import * mu = Uc ( UC_ARCH_X86 , UC_MODE_64 )

Map segments, set up stack

Hook to trace register changes

mu . emu_start ( start_addr , end_addr ) Mixed-mode pitfall: if a 64-bit stub jumps into 32-bit code via retf/retfq , you must switch to a UC_MODE_32 emulator and copy GPRs, EFLAGS, and XMM regs ; missing XMM state will corrupt SSE-based transforms. Multi-Stage Shellcode Loaders Nested shellcode with XOR decode loops; break at call rax , bypass ptrace with set $rax=0 , extract flag from mov instructions. See patterns.md . Timing Side-Channel Attack Validation time varies per correct character; measure elapsed time per candidate to recover flag byte-by-byte. See patterns.md . Godot Game Asset Extraction Use KeyDot to extract encryption key from executable, then gdsdecomp to extract .pck package. See languages.md . Roblox Place File Analysis Query Asset Delivery API for version history; parse .rbxlbin chunks (INST/PROP/PRNT) to diff script sources across versions. See languages.md . Unstripped Binary Information Leaks Pattern (Bad Opsec): Debug info and file paths leak author identity. Quick checks: strings binary | grep "/home/"

Home directory paths

strings binary | grep "/Users/"

macOS paths

file binary

Check if stripped

readelf -S binary | grep debug

Debug sections present?

Custom Mangle Function Reversing Binary mangles input 2 bytes at a time with running state; extract target from .rodata , write inverse function. See patterns.md . Rust serde_json Schema Recovery Disassemble serde Visitor implementations to recover expected JSON schema; field names in order reveal flag. See languages.md . Position-Based Transformation Reversing Binary adds/subtracts position index; reverse by undoing per-index offset. See patterns.md . Hex-Encoded String Comparison Input converted to hex, compared against constant. Decode with xxd -r -p . See patterns.md . Embedded ZIP + XOR License Decryption Binary with named symbols ( EMBEDDED_ZIP , ENCRYPTED_MESSAGE ) in .rodata → extract ZIP containing license, XOR encrypted message with license bytes to recover flag. No execution needed. See patterns.md . Stack String Deobfuscation (.rodata XOR Blob) Binary mmaps .rodata blob, XOR-deobfuscates, uses it to validate input. Reimplement verification loop with pyelftools to extract blob. Look for 0x9E3779B9 , 0x85EBCA6B constants and rol32() . See patterns.md . Prefix Hash Brute-Force Binary hashes every prefix independently. Recover one character at a time by matching prefix hashes. See patterns.md . Mathematical Convergence Bitmap Pattern: Binary classifies coordinate pairs by Newton's method convergence (e.g., z^3-1=0). Grid of pass/fail results renders ASCII art flag. Key: the binary is a classifier, not a checker — reverse the math and visualize. See patterns.md . RISC-V Binary Analysis Statically linked, stripped RISC-V ELF. Use Capstone with CS_MODE_RISCVC | CS_MODE_RISCV64 for mixed compressed instructions. Emulate with qemu-riscv64 . Watch for fake flags and XOR decryption with incremental keys. See tools.md . Sprague-Grundy Game Theory Binary Game binary plays bounded Nim with PRNG for losing-position moves. Identify game framework (Grundy values = pile % (k+1), XOR determines position), track PRNG state evolution through user input feedback. See patterns.md . Kernel Module Maze Solving Rust kernel module implements maze via device ioctls. Enumerate commands dynamically, build DFS solver with decoy avoidance, deploy as minimal static binary (raw syscalls, no libc). See patterns.md . Multi-Threaded VM with Channels Custom VM with 16+ threads communicating via futex channels. Trace data flow across thread boundaries, extract constants from GDB, watch for inverted validity logic, solve via BFS state space search. See patterns.md . Multi-Layer Self-Decrypting Binary N-layer binary where each layer decrypts the next using user-provided key bytes + SHA-NI. Use oracle (correct key → valid code with expected pattern). JIT execution with fork-per-candidate COW isolation for speed. See patterns.md .