Vizuara Books
Pretraining a Mini Kimi K3
One H200, five billion tokens, $252.35 — the complete worklog of a Kimi K3 replica trained from scratch.
A worklog of a pretraining run that finished. A 1.02-billion-parameter Kimi K3 replica trained on 5.00 billion decontaminated tokens for $252.35: the architecture and the arithmetic of shrinking it, the corpus and its 13-gram decontamination, four undocumented reasons the released code cannot train, a router that killed 94.5% of its experts by step 20, sixteen numbered attempts to raise MFU of which three worked, three distributed bugs that never crash, and twenty-four benchmark evaluations taken across the run.
Read on your Kindle
We'll send this whole book straight to your Kindle — it opens natively, so you can resize the text, read fully offline, and it remembers where you left off. Nothing to download or manage.
Sending to Kindle is for subscribers — subscribe to read the whole library on your Kindle.
00Start Here3 capsules
What we trained, what it cost, and the practices that separate a run you can publish from a run you cannot.
01What we actually trained$2529 min02What makes a run lab-grade11 min03How to read this book11 min01The Model5 capsules
Kimi K3 as Moonshot released it, and the arithmetic of shrinking it to something one GPU can train without turning it into a different model.
04Reading K3's config, line by line2.8T11 min05Scaling it down without lying about it145M12 min06The attention stack: nine KDA layers and three MLA3:111 min07The mixture of experts that actually routestop-611 min08Counting the parameters, and checking the counter1.02B9 min02The Data5 capsules
104.80 billion tokens from six sources, decontaminated against eight benchmarks before tokenization, in a shard format the loader can resume from mid-run.
09Assembling the corpus104.8B10 min10Decontamination, and the report that makes the numbers defensible13-gram9 min11When a filter fires on nothing88.7811 min12The tokenizer we did not train163,84010 min13The mix, and the bug that would have trained on one source11 min03Making It Train5 capsules
The released modeling code cannot train. Four undocumented blockers, a training loop built around them, and the two failure modes that do not crash.
14The released code cannot train4 bugs11 min15The training loopWSD14 min16Expert collapse, and the metric that missed it94.5%9 min17Checkpointing that survives losing the GPU12 min18Watching a run you cannot see11 min04Making It Fast7 capsules
An MFU investigation with sixteen numbered attempts. Three worked. The rest are written up in as much detail, because the reasoning behind them looked sound.
19What MFU is, and why it is the exchange rate6N10 min20The expert loop: eight times the work, the same time17.5x9 min21Six ideas that did not work12 min22The profile: only 12% of the time is arithmetic45.6%10 min23Two kernels that worked, and one that did not1.38x10 min24FP8, and the bigger GPU that scored worse0.19x10 min25The silent fallback3.2x10 min05More Than One GPU2 capsules
r1 fits on a single H200. Everything above it does not, and sharding a mixture of experts introduces bugs that produce a finished run that is quietly wrong.
26Sharding the model across GPUsFSDP210 min27Three distributed bugs that do not crash10 min06Did It Work3 capsules
The loss curve, twenty-four benchmark evaluations across the run, and what five billion tokens buys a 145M-active model.
28The loss curve, end to end12.10 → 2.6211 min29Benchmarks across tokens, not at the end24 points11 min30What $252 buys12 minRatings & reviews
No ratings yet. Yours would be the first.