HW
Hannah Weiss
2 weeks ago
Rare mix of depth and readability. The worked examples are the clearest I have seen on this subject.
Free preview available. Sign in and subscribe to unlock the full book.
Vizuara AI Labs · advanced
RLHF from Scratch
Align a language model with human preferences, by hand.
Build the full RLHF stack from first principles: reward models from human preference pairs, policy gradients and the advantage, and PPO with importance sampling and clipping. Put it to work on two projects — a Positive TinyStories SLM and a Reddit post summarizer — then see how DPO simplifies the whole pipeline.
37 capsules132 figures~7 hoursby Dr. Raj Dandekar
Read on your Kindle or e-reader
Download the EPUB and read offline — perfect for the train. Works on Kindle, Kobo, Apple Books & more.
00Why Alignment4 capsules
Why Alignment — 4 chapters.
01What is RLHF and Why It Mattersconceptfree11 min02The Three-Stage Alignment Pipelineconceptfree11 min03What You Will Buildconcept🔒12 min04Prerequisites and Notationmath🔒10 min01The LLM as an RL Agent5 capsules
The LLM as an RL Agent — 5 chapters.
05The Journey of a Token Through an LLMintuition🔒11 min06Tokenization and Embeddings Refresherconcept🔒11 min07The Agent-Environment Interfaceconcept🔒10 min08States, Actions, and the Policymath🔒13 min09Generation as a Trajectoryintuition🔒11 min02Reward Models6 capsules
Reward Models — 6 chapters.
10The Problem of Subjective Rewardsconcept🔒12 min11Preferences Instead of Scoresintuition🔒9 min12The Bradley-Terry Preference Modelmath🔒12 min13Building a Reward Head on an LLMcode🔒12 min14Training the Reward Modelcode🔒12 min15Evaluating and Trusting the Reward Modeldeep-dive🔒13 min03Policy Gradients and Advantage6 capsules
Policy Gradients and Advantage — 6 chapters.
16The Policy Gradient Objectivemath🔒11 min17REINFORCE and the Log-Derivative Trickmath🔒12 min18Baselines and Variance Reductionintuition🔒12 min19Value Functions and the Advantagemath🔒12 min20Generalized Advantage Estimation (GAE)math🔒11 min21Computing Advantages Over a Sequencecode🔒12 min04Proximal Policy Optimization6 capsules
Proximal Policy Optimization — 6 chapters.
22Why Vanilla Policy Gradients Are Unstableconcept🔒12 min23Importance Sampling for Off-Policy Updatesmath🔒11 min24The Clipped Surrogate Objectivemath🔒12 min25The KL Penalty and the Reference Modelconcept🔒15 min26The Actor-Critic Value Modelcode🔒13 min27The Full PPO Training Loopcode🔒13 min05Building It From Scratch6 capsules
Building It From Scratch — 6 chapters.
28Project A: A Positive TinyStories SLMproject🔒11 min29Training a Tiny Supervised Language Modelcode🔒12 min30Aligning the SLM with a Reward Signalproject🔒13 min31Project B: A Reddit Post Summarizerproject🔒11 min32Training the Summarizer's Reward Modelcode🔒13 min33PPO Fine-Tuning with OpenRLHFcode🔒11 min06Beyond PPO4 capsules
Beyond PPO — 4 chapters.
34Does RLHF Really Need a Reward Model?deep-dive🔒12 min35Direct Preference Optimization (DPO)math🔒13 min36PPO vs DPO: Tradeoffsdeep-dive🔒12 min37Putting It All Togetherconcept🔒10 minRatings & reviews
1,008 readersRate this book
EP
Elena Petrova
4 months ago
I read the free preview on a whim and ended up finishing the whole thing in two sittings. Superb pacing.
ML
Mei Lin
7 months ago
Finally a book that explains the hard parts without hand-waving. The figures alone are worth it.
FZ
Fatima Zahra
10 months ago
Best explanation of this topic I have found anywhere. The sidenotes fill in exactly the gaps I had.