# Module 04 — Decompilation & reverse engineering

> Reading what compilers leave behind. Binaries are not opaque: they are structured
> files (ELF/PE/Mach-O), generated by known machines (compilers), running on known
> contracts (ABIs). Learn the generator and the contract and the binary becomes a
> *derivable* artifact — names are gone, but structure never is.

**6 chapters · one final exam · prerequisite: module 01 chapters 03 (virtual
memory), 04 (the stack), 06 (object layout/vtables), 07 (assembly reference
model); module 02 ch04-05 helps for recognizing exploit shapes.**

## How to study it

Read in order: the file formats (bytes up), then the loader (what runs before
main), then reading disassembly (compiler idioms as fingerprints), then C++
recovery (vtables/RTTI/mangling), then what decompilers actually do, then
anti-analysis and the unpacking playbook. The labs are one continuous exercise on
the same binary: build it in module 01's toolchain, then take it apart.

## The chapters

| # | Chapter | One line |
|---|---------|----------|
| 01 | [Binary formats from bytes up](chapters/01-binary-formats.md) | ELF, PE, Mach-O: headers, segments, sections, symbols — hex-level. |
| 02 | [Linking & loading](chapters/02-linking-loading.md) | What runs before main: PLT/GOT, relocations, symbol resolution, RELRO. |
| 03 | [Reading disassembly](chapters/03-reading-disassembly.md) | Compiler idioms as fingerprints: prologues, loops, switch tables, structs. |
| 04 | [Recovering C++](chapters/04-recovering-cpp.md) | vtables, RTTI, name mangling, STL signatures, exception tables. |
| 05 | [Decompilers](chapters/05-decompilers.md) | Ghidra/IDA/Binary Ninja internals — what lifts, what recovers, what is lost. |
| 06 | [Anti-analysis](chapters/06-anti-analysis.md) | Packing, obfuscation, VM protectors, anti-debug — and the unpacking playbook. |

## Mandatory content per chapter (the writer's checklist)

**Chapter 01 — binary formats.** ELF64 byte-up: magic `\x7fELF`, e_ident fields
(class, data, OSABI), program headers (PT_LOAD/PT_DYNAMIC/PT_INTERP) vs section
headers (.text/.data/.bss/.rodata/.got/.symtab/.strtab) — segments are the loader's
view, sections are the linker's view (the two-tables-one-file insight, with an
ASCII diagram mapping sections into segments). Entry point e_entry. Symbols:
binding/type/value, .dynsym vs .symtab (what `strip` removes). PE: the DOS header
(why it survives: the MZ stub), PE signature, optional header, RVA→file-offset
conversion done by hand on a real example, the import (IAT) and export (EAT)
tables. Mach-O: magic numbers (0xfeedfacf), __TEXT/__DATA/__LINKEDIT segments,
universal ("fat") binaries and their header. Lab: `readelf -h -l -S`, `objdump -f`,
`xxd | head`, converting one RVA by hand; a stripped-vs-unstripped size comparison.

**Chapter 02 — linking & loading.** The loader's memory picture: ld.so mapping
PT_LOAD segments at page-aligned addresses (module 01 ch03 payoff), zeroing BSS.
Relocations: R_X86_64_RELATIVE (base + addend — the ASLR fixup mechanism),
GLOB_DAT/JUMP_SLOT. PLT/GOT lazy binding (the call→plt→got dance, `-fno-plt`),
GOT overwrite attacks and full RELRO as the fix (module 02 ch06 connection).
Symbol resolution order: LD_PRELOAD → binary → ld.so.cache → default dirs;
interposition as both attack surface and instrumentation technique. dlopen/
dlsym (RTLD_LOCAL vs RTLD_GLOBAL), dlmopen namespaces. Static linking tradeoffs
(size, update cadence, glibc static warnings). The vDSO/vvar special mapping (the
kernel mapping code into userspace — module 07 ch04 goes deeper). Lab: `LD_DEBUG=
files,libs`, `LD_PRELOAD` interposing `malloc` with a counting shim, `ldd`,
`/proc/PID/maps` after load.

**Chapter 03 — reading disassembly.** Compiler idioms as fingerprints: function
prologues (push rbp; mov rbp,rsp — and the -O2 version: sub rsp, N, endbr64 from
CET), epilogues (leave vs add rsp, N; pop rbp). Reading struct access in
addressing modes: `mov rax, [rbx+0x18]` = field at offset 24 — accumulate offsets
into a recovered struct layout (an ASCII struct diagram reconstructed from
disassembly). Loop shapes (dec/jnz, the modern cmp/jcc + induction variable
patterns). switch: jump-table vs comparison trees — how to tell them apart and
find the table. PIC code: rip-relative lea, GOT indirection. Callee/caller-saved
register discipline as a call-boundary fingerprint (module 01 ch07 payoff).
Methodology: find main (passed to __libc_start_main), follow the argument
registers, annotate as you go. Lab: compile module 01's chapter 07 example at -O0
and -O2, `objdump -d` both, and map each C construct to its two shapes; a
stripped-binary function census with `objdump -T` fallbacks.

**Chapter 04 — recovering C++.** The vtable pattern in memory and in disassembly:
object → vptr → [RTTI ptr, offsets..., function ptrs...] (Itanium layout; ASCII
diagram); recognizing constructor writes of vptrs (two writes = base+derived
construction). RTTI: typeinfo structures and class-hierarchy-info — recovering the
inheritance graph from a binary. Name mangling: reading _ZN... by hand (namespace
components, parameter encoding), `c++filt`, the MSVC ?...@ scheme in one table.
operator new/delete call sites → allocator identification (module 01 ch19
payoff: glibc vs jemalloc vs custom signatures). STL signatures: libstdc++
std::string SSO layout on 64-bit (the 15-char inline threshold, the 32-byte object
with pointer/capacity/size union — the exact layout), vector as the begin/end/end_
of_storage triple, map/set as red-black tree nodes. Exceptions: .eh_frame,
.gcc_except_table, personality routines (__gxx_personality_v0) — what LSDA tables
encode. Lab: compile a small C++ hierarchy, dump the vtable in gdb, demangle by
hand and verify.

**Chapter 05 — decompilers.** The pipeline: disassemble → lift to IR (Ghidra's
p-code, IDA's microcode, Binary Ninja's BNIL with its IL levels) → data-flow
analysis → variable recovery → type propagation → control-flow structuring (the
structurer: turning a CFG back into if/while — why perfect structuring is
impossible in general). What is genuinely recoverable (call graph, stack frame
sizes, register-derived local variables, constants, jump tables, vtable calls) vs
lost forever (names, comments, high-level types, macros, template identity,
compile-time decisions). Type guessing heuristics: sizes at call sites, API
consumption, propagating struct field offsets. The decompiler's failure modes:
variable merge/split errors, inlined functions, optimized-away bounds, indirect
calls (the CFI-shaped hole). Scripting: Ghidra headless + Python (find all vtable
writes in 10 lines of script), function signatures in IDA/Binja. Binary diffing
(BinDiff/Diaphora) for patch analysis — the 1-day exploit economics. Lab: run
Ghidra headless on the module's sample binary, fix one wrong signature, observe
propagation.

**Chapter 06 — anti-analysis.** Packing: UPX mechanics end-to-end (entry stub
decompresses, restores imports, jumps to OEP — the packer as a mini-loader; custom
packers), import reconstruction (why packed binaries hide the IAT; Scylla-style
rebuilding). Obfuscation: control-flow flattening (the dispatcher-loop signature —
recognize it in a decompilation), opaque predicates, dead-code insertion, MBA
(constant expressions that confuse algebra simplifiers), virtualization protectors
(VMProtect/Themida: bytecode VM, the dispatch opcode loop, why devirtualization is
per-program work). Anti-debug: PTRACE self-attach (a process can't be traced
twice — the ptrace contract), INT3 detection (CRC of own code, the 0xCC
checksum), timing (rdtsc deltas). Anti-VM (cpuid hypervisor bit, MAC prefixes,
registry artifacts). Anti-dump: erasing PE headers from memory. The unpacking
playbook as methodology: run to OEP (stack-restore watchpoint / memory-write
breakpoint on entry), dump, rebuild imports, then analyze — and the economics:
obfuscation buys *time*, not safety (the "days of analyst time per layer" framing).
Lab: pack the sample with UPX, unpack by hand (breakpoint on the tail jump), compare
hashes.

## The discipline

Identical to module 01 — see [CONVENTIONS.md](../01-memory-models/CONVENTIONS.md):
30-second version, derived teaching, ASCII diagrams, *What insiders know* (≥5
specific bullets), *See it yourself* lab, *How to use this knowledge*, *Connections*
by chapter number, 8-question quiz with worked answers behind `<details>`.

## What this module buys you

- **Cold-binary competence**: from unknown file to working theory of operation,
  with a repeatable methodology.
- **Compiler fluency inverted**: you can *infer the source shape* from the output —
  which makes you better at reading your own compiler's output too.
- **Malware analysis literacy**: packers, VMs, anti-debug — the standard first-24-
  hours workflow.
- **The bridge to modules 05-07**: ART/dyld/shared-cache internals are all "binary
  formats + loaders" with the volume turned up.