Build Large Language Models (LLMs) from Scratch

Build a GPT-style LLM from an empty file.

From tokenizer to trained transformer: attention, GPT blocks, pretraining, and fine-tuning, built by hand and illustrated.

intermediatellmtransformersfrom-scratchdeep-learning

20 capsules115 figures~6 hoursby Dr. Raj Dandekar

Read on your Kindle

We'll send this whole book straight to your Kindle — it opens natively, so you can resize the text, read fully offline, and it remembers where you left off. Nothing to download or manage.

Subscribe to send to Kindle

Sending to Kindle is for subscribers — subscribe to read the whole library on your Kindle.

00LLM Basics6 capsules

LLM Basics — 6 chapters.

01Course Introductionconcept17 min 02What are LLMs?concept19 min 03Pretraining vs Finetuning LLMs18 min 04What are transformers?concept18 min 05How does GPT-3 really work?19 min 06Stages of building an LLM from scratchcode19 min

01LLM Attention Mechanism6 capsules

LLM Attention Mechanism — 6 chapters.

07Introduction to the Attention Mechanismconcept18 min 08Simplified Attention Mechanism - Code from Scratchcode18 min 09The self attention mechanism with keys, query and values: coded form scratchcode18 min 10Causal Self Attention - Coded from scratchcode21 min 11Multi-head Attention - Part 118 min 12Multi-head Attention - Part 219 min

02LLM Data Pre-processing2 capsules

LLM Data Pre-processing.

13The GPT Tokenizer: Byte Pair Encoding and Input-Target Pairscode14 min 14Token and Positional Embeddingsconcept14 min

03LLM Architecture3 capsules

LLM Architecture.

15Bird's-eye View of the LLM Architectureconcept14 min 16Layer Norm, GELU, and Shortcut Connectionsmath15 min 17Coding the Transformer Block and the 124M GPT-2code13 min

04LLM Pretraining2 capsules

LLM Pretraining.

18Measuring Loss and Coding the Pretraining Loopcode14 min 19Temperature, Top-k Sampling, and Loading OpenAI's Weightscode13 min

05LLM Finetuning1 capsules

LLM Finetuning.

20Finetuning: From Spam Classifier to Instruction Followingcode14 min

Ratings & reviews

No ratings yet. Yours would be the first.