DeepSpeed Course

Author: Yiqiao Yin · LinkedIn · YouTube · GitHub

Distributed deep learning with DeepSpeed — from a two-parameter linear model to a 560-billion-parameter omni-modal system, with the memory and communication arithmetic worked out at every step.

Why This Exists

Adding GPUs to a data-parallel job gives you more throughput and exactly the same model capacity. Every device holds a bit-identical copy of the parameters, gradients, and optimizer states, so the largest model you can train is the largest that fits on one accelerator.

For mixed-precision Adam, that budget is

$$M = \underbrace{2\Psi}_{\text{weights}} + \underbrace{2\Psi}_{\text{gradients}} + \underbrace{12\Psi}_{\text{optimizer states}} = 16\Psi \text{ bytes}$$

A 7B model therefore needs 112 GB of model states before a single activation is allocated — more than an 80 GB A100, for weights that are only 14 GB. Seven-eighths of that memory is bookkeeping, replicated $N$ times across your cluster for no reason.

DeepSpeed's ZeRO eliminates that redundancy. The first two stages do it at zero additional communication cost; the third makes model size scale with aggregate cluster memory rather than a single device. This course develops that arithmetic and then applies it, example by example.

Two Ways to Read This

As a course — work through in order. Each example builds on the last, and the theory pages are referenced from the tutorials that need them.

As a reference — jump to the page for the problem you have. Troubleshooting is organized symptom-first; the config reference is organized by config block.

The Path

Level	Pages
Foundations	Basic Neural Network · ZeRO Stages
Vision	ConvNet · CIFAR-10
Sequences	RNN / LSTM · Stock Prediction
Probabilistic	Bayesian Neural Networks
Language models	HuggingFace Integration · TRL Function Calling · GRPO · GPT-OSS
Multimodal	OCR Vision-Language · Video-Text · Video-Speech

Start Here

On a SLURM cluster (CoreWeave)

git clone https://github.com/yiqiao-yin/deepspeed-course.git
cd deepspeed-course/01_basics/01_neuralnet

sbatch run_deepspeed.sh
squeue -u $USER
tail -f logs/basic_nn_*.out

You SSH to a login node, which has no GPUs, and submit jobs to reach them. See SLURM Deployment and CoreWeave Setup.

On a single-tenant pod (RunPod)

git clone https://github.com/yiqiao-yin/deepspeed-course.git
cd deepspeed-course

uv venv myenv && source myenv/bin/activate
uv pip install torch deepspeed wandb

cd 01_basics/01_neuralnet
deepspeed --num_gpus=1 train_ds.py

GPUs are available immediately; the #SBATCH lines in the launcher scripts are inert comments. See RunPod Setup.

Full setup, including the CUDA toolchain requirements that make DeepSpeed harder to install than a normal package: Installation.

Repository Layout

Each directory is self-contained — a training script, a ds_config.json, a launcher, and a README. You can run any one without touching the others.

deepspeed-course/
├── 01_basics/01_neuralnet/                       # Linear regression — the mechanics
├── 01_basics/02_convnet/                         # CNN on synthetic MNIST
├── 01_basics/03_convnet_cifar10/        # CIFAR-10 — the 10% -> 81% case study
├── 01_basics/04_rnn/                             # LSTM time series
├── 02_intermediate/01_bayesian_neuralnet/                    # Parallel tempering MCMC
├── 02_intermediate/02_rnn_stock_data/           # Real market data with yfinance
├── 03_huggingface/01_llm_finetuning/                           # LLM fine-tuning
├── 03_huggingface/02_trl_sft/                       # TRL SFT for function calling
├── 03_huggingface/03_ocr/                       # Qwen2-VL vision-language
├── 03_huggingface/06_grpo/                      # GRPO on GSM8K
├── 03_huggingface/08_gpt_oss_lora/  # gpt-oss-20b MoE LoRA
├── 03_huggingface/09_multi_agency/          # Multi-agent GRPO (exploratory)
├── 04_video_text/                                   # Video-text training
└── 05_video_speech/                                   # LongCat-Flash-Omni 560B

A Note on Honesty

Several examples in this repository are infrastructure tests rather than trainable models — the OCR example ships 10 synthetic samples, the video-text frame extractor returns the same image repeated, the multi-agent example trains against a reward its own docstring calls a dummy. Where that is the case, the corresponding page says so plainly and explains what you would change.

Validating a pipeline at small scale is genuinely most of the engineering work at these model sizes. But it is worth knowing which you are looking at.

Reference

• DeepSpeed Configuration — every key, and the combinations that are invalid
• Troubleshooting — symptom-first diagnosis
• Hardware Requirements — sizing from parameter count

Next Steps

• Installation — set up your environment
• Quick Start — run your first job and read its output
• DeepSpeed ZeRO Stages — the theory the rest of the course rests on