Build a Data-Efficient Image Transformer (DeiT) from Scratch

Train a Vision Transformer on ImageNet-1k alone.

Build a Vision Transformer from patches to attention to a full encoder, then make it data-efficient with knowledge distillation and a dedicated distillation token. Every layer, loss, and training trick is coded by hand and illustrated.

advancedvisiontransformersdistillationfrom-scratch

42 capsules151 figures~8 hoursby Dr. Raj Dandekar

Read on your Kindle or e-reader

Download the EPUB and read offline — perfect for the train. Works on Kindle, Kobo, Apple Books & more.

Send to Kindle →

00Why DeiT?5 capsules

Why DeiT? — 5 chapters.

01Course Introductionconceptfree12 min 02The Data Hunger of Vision Transformersintuitionfree13 min 03What Makes DeiT Data-Efficientconcept🔒12 min 04CNNs vs Transformers: The Inductive-Bias Tradeoffintuition🔒11 min 05The Build Roadmapconcept🔒15 min

01Images as Tokens5 capsules

Images as Tokens — 5 chapters.

06From Pixels to Patchesintuition🔒11 min 07Patch Embedding with a Conv2dcode🔒10 min 08The CLS Tokenconcept🔒12 min 09Positional Embeddingsmath🔒10 min 10Building the Embedding Blockcode🔒12 min

02Attention from Scratch6 capsules

Attention from Scratch — 6 chapters.

11The Intuition of Self-Attentionintuition🔒12 min 12Queries, Keys, and Valuesmath🔒13 min 13Scaled Dot-Product Attention, Codedcode🔒11 min 14Multi-Head Attentioncode🔒14 min 15The MLP Block and GELUcode🔒11 min 16LayerNorm and Residual Connectionsconcept🔒12 min

03The ViT Backbone5 capsules

The ViT Backbone — 5 chapters.

17The Transformer Encoder Blockcode🔒11 min 18Stacking the Encodercode🔒13 min 19The Classification Headcode🔒13 min 20Assembling the Full ViTproject🔒11 min 21Counting Parameters and FLOPsdeep-dive🔒12 min

04Knowledge Distillation6 capsules

Knowledge Distillation — 6 chapters.

22What Is Knowledge Distillation?concept🔒10 min 23Soft Labels and Temperaturemath🔒11 min 24The KL-Divergence Lossmath🔒13 min 25Hard vs Soft Distillationconcept🔒11 min 26Choosing the Teacherintuition🔒12 min 27Combining the Lossescode🔒13 min

05The Distillation Token5 capsules

The Distillation Token — 5 chapters.

28The Distillation-Token Ideaconcept🔒11 min 29CLS Token vs Distillation Tokenintuition🔒10 min 30Adding the Token to the Modelcode🔒14 min 31The Two-Head Forward Passcode🔒12 min 32Inference-Time Fusiondeep-dive🔒11 min

06Training DeiT6 capsules

Training DeiT — 6 chapters.

33The Training Recipeconcept🔒13 min 34Data Augmentation That Mattersintuition🔒10 min 35Regularization: Stochastic Depth and Moreconcept🔒12 min 36Writing the Training Loopcode🔒11 min 37The Distillation Loss in Codecode🔒12 min 38Evaluation and Fine-Tuningcode🔒14 min

07Putting It Together4 capsules

Putting It Together — 4 chapters.

39The DeiT Model Variantsdeep-dive🔒12 min 40Reproducing the Paper Resultsproject🔒12 min 41Visualizing Attentiondeep-dive🔒11 min 42From DeiT to the Frontierconcept🔒13 min

Ratings & reviews

1,008 readers

4.3

189 ratings

5 ★49%

4 ★30%

3 ★11%

2 ★6%

1 ★5%

Rate this book

Ananya Iyer

5 days ago

Beautifully produced and genuinely deep. The reader experience makes it easy to keep going for hours.

Lucia Rossi

3 months ago

Rare mix of depth and readability. The worked examples are the clearest I have seen on this subject.

Aarav Mehta

4 months ago

I read the free preview on a whim and ended up finishing the whole thing in two sittings. Superb pacing.

Priya Nair

5 months ago

Dense but never confusing. Every chapter builds cleanly on the last — you can feel the care in the structure.