Vizuara Books
5D Parallelism for Large Model Training
Free preview available. Sign in and subscribe to unlock the full book.
Vizuara AI Labs · advanced

5D Parallelism for Large Model Training

Scale a model across a cluster of GPUs.

Replicate the HuggingFace Ultra-Scale Playbook from scratch: data, tensor, sequence, context, and pipeline parallelism, plus ZeRO sharding and Mixture-of-Experts. Every axis is built by hand and profiled, then composed to distribute GPT-2 across a real GPU cluster.

advancedparallelismgputrainingdistributed
40 capsules136 figures~8 hoursby Dr. Raj Dandekar

Read on your Kindle or e-reader

Download the EPUB and read offline — perfect for the train. Works on Kindle, Kobo, Apple Books & more.

Send to Kindle →
00GPU Foundations6 capsules

GPU Foundations — 6 chapters.

01Why One GPU Is Not Enoughintuitionfree10 min02GPU Primer: Anatomy of an Acceleratorconceptfree12 min03The Five Axes of Parallelismconcept🔒13 min04The Four Buckets of GPU Memoryconcept🔒12 min05Floating Point and Mixed Precisionmath🔒12 min06Memory Math for a 1B Llamamath🔒12 min
01Data Parallelism7 capsules

Data Parallelism — 7 chapters.

07Activation Recomputationconcept🔒12 min08Gradient Accumulationconcept🔒11 min09Data Parallelism from Scratchcode🔒13 min10The Ring-All-Reduce Algorithmdeep-dive🔒11 min11Choosing the Batch Sizemath🔒12 min12Profiling with TensorBoardcode🔒11 min13Overlapping Communication and Computedeep-dive🔒13 min
02ZeRO and Sharded Data Parallelism4 capsules

ZeRO and Sharded Data Parallelism — 4 chapters.

14The Redundancy Problem in Data Parallelismintuition🔒11 min15ZeRO-1: Sharding Optimizer Statesconcept🔒11 min16ZeRO-2 and ZeRO-3concept🔒13 min17Implementing ZeRO from Scratchcode🔒13 min
03Tensor Parallelism5 capsules

Tensor Parallelism — 5 chapters.

18Why Split a Single Layerintuition🔒13 min19Column- and Row-Parallel Linear Layersmath🔒11 min20Tensor Parallelism for the Transformer Blockconcept🔒12 min21Tensor Parallelism from Scratchcode🔒12 min22TP vs ZeRO for Inferencedeep-dive🔒11 min
04Sequence and Context Parallelism5 capsules

Sequence and Context Parallelism — 5 chapters.

23The Long-Sequence Memory Wallintuition🔒12 min24Sequence Parallelism Explainedconcept🔒12 min25Sequence Parallelism from Scratchcode🔒13 min26Context Parallelism and Ring Attentionconcept🔒12 min27Naive Context Parallelism from Scratchcode🔒13 min
05Pipeline Parallelism5 capsules

Pipeline Parallelism — 5 chapters.

28Why Pipeline Parallelismintuition🔒12 min29The Pipeline Bubblemath🔒12 min30Advanced Pipeline Schedulesdeep-dive🔒13 min31Pipeline Parallelism from Scratchcode🔒14 min32Megatron 3D Parallelismdeep-dive🔒12 min
06Expert Parallelism and MoE4 capsules

Expert Parallelism and MoE — 4 chapters.

33Mixture-of-Experts Basicsconcept🔒12 min34Expert Parallelism Explainedconcept🔒13 min35Load Balancing and Token Routingdeep-dive🔒14 min36Build an MoE SLM from Scratchproject🔒11 min
07Putting It All Together4 capsules

Putting It All Together — 4 chapters.

37Composing 5D Parallelismconcept🔒12 min38Distributing GPT-2 Across a Clusterproject🔒12 min39Build an SLM with 5D Parallelismproject🔒12 min40Capstone Projects and Next Stepsproject🔒12 min

Ratings & reviews

504 readers
4.3
117 ratings
5 ★50%
4 ★29%
3 ★11%
2 ★6%
1 ★4%
Rate this book
ML
Mei Lin
last week

Dense but never confusing. Every chapter builds cleanly on the last — you can feel the care in the structure.

FZ
Fatima Zahra
4 months ago

Practical and rigorous at the same time. I went straight from reading a capsule to shipping it at work.

SR
Sofia Reyes
8 months ago

The hand-drawn diagrams make abstract ideas click instantly. Wish every technical book was written like this.

HW
Hannah Weiss
11 months ago

A few sections moved fast for me, but re-reading with the figures open made everything land. Highly recommend.