Installation
Setting up DeepSpeed, and understanding the one thing that makes it harder to install than an ordinary Python package: it compiles CUDA extensions against your local toolchain.
Quick Start
Your first DeepSpeed run, and how to read what it prints.
DeepSpeed ZeRO Stages
Zero Redundancy Optimizer — where it came from, why the partitioning is mathematically sound, why it splits into exactly three stages, and what each stage costs you in bandwidth.