The Memory Arithmetic, Derived

Mixed-precision Adam costs 16 bytes per parameter, so a 7B model needs 112 GB of model states before a single activation is allocated. This course derives that number, shows why ZeRO stages 1 and 2 eliminate the redundancy at zero extra communication, and why stage 3 costs exactly 1.5×. Every recommendation elsewhere follows from it.

Worked Examples, Honestly Documented

Fourteen runnable examples from a two-parameter linear model to a 560B omni-modal system. Including the failures: a CIFAR-10 run that produced NaN at exactly chance accuracy and how it was diagnosed to root cause, a look-ahead-bias bug in the stock example, and which examples are infrastructure tests rather than trainable models.

Verifiable Without a Cluster

Most of these examples cannot run on a laptop — so the repository ships logic tests that validate configs, data handling and reward functions with no GPU and no model download. They run in CI on every push, and have already caught a config bug that manual review missed. ./tests/run_all.sh