EP
Elena Petrova
3 weeks ago
Dense but never confusing. Every chapter builds cleanly on the last — you can feel the care in the structure.
Free preview available. Sign in and subscribe to unlock the full book.
Vizuara AI Labs · advanced
Transformers for Vision & Multimodal LLMs
Build every vision and multimodal transformer by hand.
From patch embeddings to diffusion: ViT, DeiT, Swin, DETR, SAM, NanoVLM, Flamingo, LLaVA and DDPM, coded from scratch and illustrated. Start at the convolutional inductive bias and end with your own end-to-end multimodal model.
37 capsules131 figures~7 hoursby Dr. Raj Dandekar
Read on your Kindle or e-reader
Download the EPUB and read offline — perfect for the train. Works on Kindle, Kobo, Apple Books & more.
00Foundations — From CNNs to Transformers5 capsules
Foundations — From CNNs to Transformers — 5 chapters.
01Why Vision Needs Transformersintuitionfree13 min02The Limits of the Convolutional Inductive Biasconceptfree11 min03How nn.Conv2d Actually Workscode🔒12 min04Journey of a Token Through a Transformerintuition🔒14 min05The Feedforward Block and the Residual Streamconcept🔒13 min01Attention From Scratch7 capsules
Attention From Scratch — 7 chapters.
06From RNNs to Attentionintuition🔒12 min07Simplified Self-Attention, Coded From Scratchcode🔒10 min08Queries, Keys and Valuesconcept🔒12 min09Self-Attention With Trainable Weightscode🔒10 min10Masked (Causal) Self-Attentioncode🔒10 min11Multi-Head Attentioncode🔒11 min12Counting GPT-3's 175 Billion Parametersmath🔒9 min02The Vision Transformer5 capsules
The Vision Transformer — 5 chapters.
13Patch Embeddings and the CLS Tokenconcept🔒14 min14Coding the Vision Transformer From Scratchcode🔒12 min15Training ViT and Its Data Hungerdeep-dive🔒10 min16DeiT — Data-Efficient Image Transformersintuition🔒11 min17Coding DeiT From Scratchcode🔒12 min03Hierarchical & Dense Vision Transformers6 capsules
Hierarchical & Dense Vision Transformers — 6 chapters.
18Swin Transformer — Windows and Hierarchyintuition🔒12 min19Reading the Swin Paperdeep-dive🔒12 min20Coding Swin From Scratchcode🔒13 min21DETR — Detection as Set Predictionconcept🔒13 min22Coding DETR From Scratchcode🔒12 min23The Segment Anything Model (SAM)project🔒12 min04Video & Multimodal Models6 capsules
Video & Multimodal Models — 6 chapters.
24TimeSformer — Attention Over Videoconcept🔒10 min25Running TimeSformer on Videocode🔒13 min26Coding NanoVLM — Your First Vision-Language Modelproject🔒11 min27Flamingo — Few-Shot Multimodal Learningconcept🔒12 min28Building Flamingo-Style Cross-Attentioncode🔒12 min29LLaVA — Visual Instruction Tuningdeep-dive🔒11 min05Generative Vision — Autoencoders to Diffusion6 capsules
Generative Vision — Autoencoders to Diffusion — 6 chapters.
30Autoencoders From Scratchcode🔒11 min31Variational Autoencodersmath🔒12 min32Coding a VAEcode🔒12 min33Introduction to Diffusion Modelsintuition🔒11 min34Coding DDPM From Scratchcode🔒11 min35From DDPM to Stable Diffusiondeep-dive🔒12 min06Putting It All Together2 capsules
Putting It All Together — 2 chapters.
36The Vision & Multimodal Transformer Family Treeconcept🔒11 min37Capstone — Build Your Own Multimodal Modelproject🔒11 minRatings & reviews
2,154 readersRate this book
ML
Mei Lin
3 months ago
Practical and rigorous at the same time. I went straight from reading a capsule to shipping it at work.
TA
Tomás Alvarez
5 months ago
The hand-drawn diagrams make abstract ideas click instantly. Wish every technical book was written like this.