deep-roots learning program · module 22
Module 22 — ML systems & training infrastructure
Every chapter: first principles, What insiders know, a See it yourself lab, connections, and a quiz with worked answers.
01
The tensor: memory's industrial form
Strides, layouts, views vs copies, the memory-format war — the array of module 01 grown an economy.
→
02
Autograd: the tape as a heap
The forward tape, the backward allocations, activation memory accounting — why training RAM obeys the 20× rule.
→
03
Distributed training: sharding the unshardable
Data parallel, ZeRO, tensor/pipeline parallel — four ways a model too big for any one HBM becomes trainable anyway.
→
04
Inference serving: the KV-cache machine
KV as a paged heap, continuous batching, quantization as the bytes-per-token lever — latency vs throughput vs cost.
→
05
Collectives and the datacenter computer
AllReduce on rings and trees, NVLink vs InfiniBand, stragglers, the synchrony trade — the fabric as a memory hierarchy.
→
the exam