SKILL: VM & Bytecode Reverse Engineering — Expert Analysis Playbook

AI LOAD INSTRUCTION

Expert techniques for reversing custom virtual machines and bytecode interpreters. Covers dispatcher identification, opcode mapping, custom ISA reconstruction, disassembler/decompiler writing, maze challenges, and real-world VM protector analysis. Base models often fail to recognize the fetch-decode-execute pattern or attempt to analyze VM bytecode as native code. 0. RELATED ROUTING code-obfuscation-deobfuscation when the VM is a commercial protector (VMProtect/Themida) symbolic-execution-tools when using angr to solve VM-based challenges anti-debugging-techniques when the VM includes anti-debug checks Quick identification Binary Pattern Likely VM Type Start With while(1) { switch(bytecode[pc]) } Switch-based dispatcher Map each case to an operation Indirect jump via table jmp [table + opcode8] Table-based dispatcher Dump jump table, analyze handlers Nested if-else chain on byte value If-chain dispatcher Same as switch, just different syntax Stack push/pop dominant operations Stack-based VM Identify push, pop, arithmetic ops reg[X] = ... array operations Register-based VM Map register indices to operations 2D grid + direction input Maze challenge Extract grid, apply BFS/DFS 1. CUSTOM VM IDENTIFICATION 1.1 Structural Indicators VM Architecture Components: ┌─────────────────────────────────┐ │ Bytecode Program (data section)│ ├─────────────────────────────────┤ │ Program Counter (pc/ip) │ │ Register File / Stack │ │ Memory / Data Area │ ├─────────────────────────────────┤ │ Dispatcher Loop │ │ ├─ Fetch: opcode = code[pc] │ │ ├─ Decode: lookup handler │ │ └─ Execute: run handler │ └─────────────────────────────────┘ 1.2 IDA/Ghidra Signatures Switch dispatcher (most common in CTF): while ( running ) { unsigned char op = bytecode [ pc ++ ] ; switch ( op ) { case 0x00 : / nop / break ; case 0x01 : / push imm / stack [ sp ++ ] = bytecode [ pc ++ ] ; break ; case 0x02 : / add / stack [ sp - 2 ] += stack [ sp - 1 ] ; sp -- ; break ; // ... case 0xFF : / halt */ running = 0 ; break ; } } Table dispatcher (more optimized): typedef void ( * handler_t ) ( vm_ctx_t * ) ; handler_t handlers [ 256 ] = { handle_nop , handle_push , handle_add , . . . } ; while ( running ) { handlers [ bytecode [ pc ++ ] ] ( & ctx ) ; } 2. ANALYSIS METHODOLOGY Step 1: Find the Dispatcher Look for: Large switch statement (many cases) in a loop Array of function pointers indexed by a byte from a data buffer Single function with high cyclomatic complexity Cross-references to a data buffer read byte-by-byte Step 2: Map Opcodes to Operations For each case/handler, determine: Property How to Identify Opcode value Case number or table index Operation type Register/stack modifications Operand count How many bytes consumed after opcode Operand type Immediate value, register index, or memory address Side effects Output, memory write, flag modification Step 3: Extract Bytecode Program

Typical extraction from binary

import struct with open ( 'challenge' , 'rb' ) as f : f . seek ( bytecode_offset ) bytecode = f . read ( bytecode_length )

Or from IDA:

bytecode = idc.get_bytes(bytecode_addr, bytecode_len)

Step 4: Write Custom Disassembler OPCODES = { 0x00 : ( "nop" , 0 ) ,

(mnemonic, operand_bytes)

0x01 : ( "push" , 1 ) ,

push immediate byte

0x02 : ( "pop" , 0 ) , 0x03 : ( "add" , 0 ) , 0x04 : ( "sub" , 0 ) , 0x05 : ( "xor" , 0 ) , 0x06 : ( "cmp" , 0 ) , 0x07 : ( "jmp" , 2 ) ,

jump to 16-bit address

0x08 : ( "je" , 2 ) , 0x09 : ( "jne" , 2 ) , 0x0A : ( "mov" , 2 ) ,

mov reg, imm

0x0B : ( "load" , 1 ) ,

load from memory[operand]

0x0C : ( "store" , 1 ) ,

store to memory[operand]

0x0D : ( "print" , 0 ) , 0x0E : ( "read" , 0 ) ,

read input

0xFF

:

(

"halt"

,

0

)

,

}

def

disassemble

(

bytecode

)

:

pc

=

0

while

pc

<

len

(

bytecode

)

:

op

=

bytecode

[

pc

]

if

op

not

in

OPCODES

:

print

(

f"

{

pc

:

04x

}

UNKNOWN { op :

04x

} " ) pc += 1 continue mnemonic , operand_size = OPCODES [ op ] operands = bytecode [ pc + 1 : pc + 1 + operand_size ] operand_str = ' ' . join ( f' { b :

04x

} ' for b in operands ) print ( f" { pc : 04x } : { mnemonic : 8s } { operand_str } " ) pc += 1 + operand_size disassemble ( bytecode ) Step 5: Analyze Disassembled Program With the custom disassembly, apply standard reverse engineering: Identify input reading (read opcode) Trace data flow from input to comparison Determine success/failure conditions Extract the check logic (often XOR/ADD transformations of input compared against constants) 3. COMMON VM PATTERNS IN CTF 3.1 Stack-Based VM Operations work on a stack (like JVM or Python bytecode). Opcode Operation Stack Effect PUSH imm Push immediate value [...] → [..., imm] POP Discard top [..., a] → [...] ADD Add top two [..., a, b] → [..., a+b] SUB Subtract [..., a, b] → [..., a-b] MUL Multiply [..., a, b] → [..., a*b] XOR Bitwise XOR [..., a, b] → [..., a^b] CMP Compare [..., a, b] → [..., (a==b)] JMP addr Unconditional jump no change JZ addr Jump if top is zero [..., a] → [...] PRINT Output top as char [..., a] → [...] READ Read char to stack [...] → [..., input] HALT Stop execution - 3.2 Register-Based VM Operations use register indices (like x86, ARM). Opcode Format Operation MOV r, imm 0x01 RR II II reg[R] = imm16 MOV r1, r2 0x02 R1 R2 reg[R1] = reg[R2] ADD r1, r2 0x03 R1 R2 reg[R1] += reg[R2] SUB r1, r2 0x04 R1 R2 reg[R1] -= reg[R2] XOR r1, r2 0x05 R1 R2 reg[R1] ^= reg[R2] CMP r1, r2 0x06 R1 R2 flags = compare(r1, r2) JMP addr 0x07 AA AA pc = addr JE addr 0x08 AA AA if equal: pc = addr LOAD r, [addr] 0x09 RR AA reg[R] = mem[addr] STORE [addr], r 0x0A AA RR mem[addr] = reg[R] SYSCALL 0x0B I/O operation based on reg[0] HALT 0xFF stop 3.3 Brainfuck-like / Esoteric VMs BF Command VM Equivalent Description

INC ptr Move data pointer right < DEC ptr Move data pointer left + INC [ptr] Increment byte at pointer - DEC [ptr] Decrement byte at pointer . OUTPUT [ptr] Output byte at pointer , INPUT [ptr] Input byte to pointer [ JZ forward Jump past ] if byte is zero ] JNZ back Jump back to [ if byte is nonzero 4. MAZE CHALLENGES 4.1 Identification Binary reads directional input (WASD, arrow keys, UDLR) 2D array in data section (walls, paths, start, end) Position tracking with x,y coordinates Win condition at specific coordinates 4.2 Map Extraction

Extract maze grid from binary data section

MAZE_ADDR

0x601060 WIDTH = 20 HEIGHT = 15

From binary dump:

maze

[ ] for row in range ( HEIGHT ) : line = "" for col in range ( WIDTH ) : cell = bytecode [ MAZE_ADDR + row * WIDTH + col - base_addr ] if cell == 0 : line += "."

path

elif cell == 1 : line += "#"

wall

elif cell == 2 : line += "S"

start

elif cell == 3 : line += "E"

end

else : line += "?" maze . append ( line ) print ( line ) 4.3 Automated Solving from collections import deque def solve_maze ( maze , start , end ) : """BFS solver returns direction string.""" rows , cols = len ( maze ) , len ( maze [ 0 ] ) directions = { 'U' : ( - 1 , 0 ) , 'D' : ( 1 , 0 ) , 'L' : ( 0 , - 1 ) , 'R' : ( 0 , 1 ) } queue = deque ( [ ( start , "" ) ] ) visited = { start } while queue : ( r , c ) , path = queue . popleft ( ) if ( r , c ) == end : return path for name , ( dr , dc ) in directions . items ( ) : nr , nc = r + dr , c + dc if ( 0 <= nr < rows and 0 <= nc < cols and maze [ nr ] [ nc ] != '#' and ( nr , nc ) not in visited ) : visited . add ( ( nr , nc ) ) queue . append ( ( ( nr , nc ) , path + name ) ) return None

Find start and end positions

for r , row in enumerate ( maze ) : for c , cell in enumerate ( row ) : if cell == 'S' : start = ( r , c ) if cell == 'E' : end = ( r , c ) solution = solve_maze ( maze , start , end ) print ( f"Path: { solution } " ) 4.4 Direction Encoding Different challenges encode directions differently: Encoding Up Down Left Right WASD W S A D UDLR U D L R Arrow keys ↑ (0x48) ↓ (0x50) ← (0x4B) → (0x4D) Numbers 1 2 3 4 Hex opcodes 0x01 0x02 0x03 0x04 5. REAL-WORLD VM PROTECTORS 5.1 VMProtect Analysis Approach 1. Find VM entry: search for pushad/pushfd sequence 2. Identify VM context structure (registers, flags, bytecode pointer) 3. Locate handler table (often obfuscated with opaque predicates) 4. For each handler: a. Remove junk code / opaque predicates b. Identify the core operation c. Document handler semantics 5. Trace bytecode execution (instruction-level trace) 6. Reconstruct original code from trace 5.2 Tigress Obfuscator Academic VM obfuscator with configurable protection layers. Feature Approach Single-dispatch VM Standard handler extraction Split handlers Handlers spread across multiple functions Nested VMs Outer VM handler invokes inner VM Encrypted bytecode Dynamic decryption before each fetch Polymorphic handlers Different code for same operation on each build 5.3 Common VM Protector Patterns Protector Dispatcher Style Difficulty VMProtect Table + opaque predicates High Themida (Code Virtualizer) CISC-like, large handler set High Tigress Configurable, academic Medium-High Custom CTF VM Simple switch Low-Medium Movfuscator All-mov computation Medium 6. TOOLS Tool Purpose Usage IDA Pro Identify dispatcher, reverse handlers F5 decompile, xref analysis Ghidra Free alternative with Sleigh processor modules Write custom processor for VM ISA angr Symbolic execution through VM Treat entire VM as constraint system Pin / DynamoRIO Dynamic instrumentation for tracing Record opcode handler execution sequence REVEN Full-system trace recording Replay and analyze VM execution Unicorn Emulate VM execution Fast handler emulation Miasm IR-based analysis Lift VM handlers to IR for analysis Custom Python Write disassembler/decompiler Per-challenge custom tooling Ghidra Sleigh Processor Module For recurring VM architectures, write a Sleigh processor specification: define space ram type=ram_space size=2 default; define space register type=register_space size=1; define register offset=0 size=1 [ R0 R1 R2 R3 FLAGS PC SP ]; define token opcode(8) op = (0,7) ; :NOP is op=0x00 { } :PUSH imm is op=0x01; imm { SP = SP - 1; [ram]:1 SP = imm; } :POP is op=0x02 { SP = SP + 1; } :ADD is op=0x03 { local a = [ram]:1 (SP+1); [ram]:1 (SP+1) = a + [ram]:1 SP; SP = SP + 1; } 7. DECISION TREE Binary contains custom bytecode interpreter? │ ├─ Can you identify the dispatcher? │ ├─ Yes (switch/table/if-chain) │ │ ├─ Few opcodes (< 20) → Simple CTF VM │ │ │ ├─ Stack-based → map push/pop/arithmetic ops │ │ │ ├─ Register-based → map mov/add/cmp ops │ │ │ └─ Write disassembler → analyze program → solve │ │ │ │ │ └─ Many opcodes (50+) → Commercial protector │ │ ├─ Known protector → use specific deprotection tools │ │ └─ Custom → trace execution, pattern-match handlers │ │ │ └─ No clear dispatcher │ ├─ All-mov instructions → movfuscator │ ├─ Encrypted bytecode → find decryption, dump after decode │ └─ Split/distributed handlers → trace execution to find them │ ├─ Is it a maze challenge? │ ├─ Extract grid from data section │ ├─ Identify direction encoding │ ├─ BFS/DFS to find shortest path │ └─ Convert path to expected input format │ ├─ Is there input validation in VM? │ ├─ Small input space → brute-force via Unicorn emulation │ ├─ Known format → constrained angr solve │ └─ Complex check → write disassembler, analyze check logic │ └─ Multiple VM layers (VM in VM)? ├─ Analyze outer VM first ├─ Extract inner bytecode ├─ Repeat analysis for inner VM └─ Consider: symbolic execution may handle nested VMs directly 8. CTF SOLVING WORKFLOW 1. Run the binary — understand I/O behavior └─ What input does it expect? What output on success/failure? 2. Open in IDA/Ghidra — find the main loop └─ Look for while/for loop with switch or indirect jump 3. Identify VM components: ├─ Bytecode location (where is the program data?) ├─ PC/IP variable (how is current position tracked?) ├─ Registers/stack (where is VM state stored?) └─ I/O handlers (which opcodes read input / write output?) 4. Map all opcodes (create the ISA specification) └─ For each case/handler: opcode number, operation, operands 5. Write disassembler in Python └─ Output readable assembly for the bytecode 6. Analyze the disassembled program: ├─ Find input reading ├─ Trace transformations applied to input ├─ Find comparison against expected values └─ Reverse the transformation to find valid input 7. Solve: ├─ If simple transforms (XOR, ADD) → reverse manually ├─ If complex → feed to Z3 as constraints └─ If maze → extract grid, run pathfinding