Build a Diffusion Language Model from Scratch

Generate text all-at-once, from an empty file.

Build a masked diffusion language model from scratch and see a sentence emerge from noise instead of one token at a time. From embeddings and the autoregressive baseline to masking, denoising, unmasking schedules, and a full Colab build.

advanceddiffusionllmfrom-scratchgenerative

35 capsules125 figures~7 hoursby Dr. Raj Dandekar

Read on your Kindle or e-reader

Download the EPUB and read offline — perfect for the train. Works on Kindle, Kobo, Apple Books & more.

Send to Kindle →

00Foundations: Text as a Distribution6 capsules

Foundations: Text as a Distribution — 6 chapters.

01Why a Diffusion Model for Language?intuitionfree12 min 02Discrete vs Continuous Dataconceptfree11 min 03Representing Text in a Vector Spaceconcept🔒10 min 04Word Embeddings: Word2Vec, GloVe, and BERTconcept🔒14 min 05Text Generation as Sampling from a Distributionmath🔒11 min 06Language Models as Distribution Approximatorsconcept🔒11 min

01The Diffusion Idea4 capsules

The Diffusion Idea — 4 chapters.

07Generative AI: One Probabilistic View of Everythingintuition🔒10 min 08Diffusion for Image Generationconcept🔒12 min 09The Noising and Denoising Loopintuition🔒11 min 10From Noise in Pixels to Masks in Textintuition🔒11 min

02The Autoregressive Baseline6 capsules

The Autoregressive Baseline — 6 chapters.

11How Autoregressive Models Generate Textconcept🔒12 min 12Tokenization and Embeddingscode🔒12 min 13The Attention Mechanismmath🔒12 min 14The Causal Attention Maskmath🔒12 min 15Inside the Transformer Blockcode🔒13 min 16Build an Autoregressive LLM from Scratchproject🔒12 min

03Diffusion Language Model Theory6 capsules

Diffusion Language Model Theory — 6 chapters.

17Masked Diffusion Language Modelsconcept🔒12 min 18The Forward Masking Processmath🔒12 min 19The Matrix and Vector Operations Under the Hoodmath🔒14 min 20Training vs Generation Phasesconcept🔒11 min 21Denoising as Progressive Demaskingintuition🔒13 min 22Visualizing Diffusion Generationintuition🔒13 min

04Inference and Unmasking Strategies5 capsules

Inference and Unmasking Strategies — 5 chapters.

23The Denoising Process in Detaildeep-dive🔒12 min 24Token Unmasking Schedulesconcept🔒12 min 25Why Diffusion Is Faster: Speed vs Autoregressivedeep-dive🔒11 min 26The Coherence vs Speed Tradeoffconcept🔒12 min 27Speculative Decoding and Advanced Accelerationdeep-dive🔒11 min

05Build It: Code the Diffusion LM6 capsules

Build It: Code the Diffusion LM — 6 chapters.

28Data Preprocessing and Tokenizationcode🔒13 min 29The Forward Pass: Code Walkthroughcode🔒14 min 30Training the Diffusion LMcode🔒14 min 31Evaluation and Running Generationcode🔒14 min 32Running on Colab and RunPodproject🔒14 min 33Putting It All Togetherproject🔒13 min

06Frontiers and Research Directions2 capsules

Frontiers and Research Directions — 2 chapters.

34Diffusion vs Autoregressive: Full Recapconcept🔒12 min 35Potential Areas of Researchdeep-dive🔒13 min

Ratings & reviews

428 readers

4.9

93 ratings

5 ★63%

4 ★23%

3 ★9%

2 ★3%

1 ★2%

Rate this book

Marcus Bell

3 weeks ago

The hand-drawn diagrams make abstract ideas click instantly. Wish every technical book was written like this.

Noah Kim

last month

Practical and rigorous at the same time. I went straight from reading a capsule to shipping it at work.

Tomás Alvarez

5 months ago

A few sections moved fast for me, but re-reading with the figures open made everything land. Highly recommend.

Jonas Bergström

9 months ago

This is the resource I wish I had when I started. Clear mental models, zero fluff.