Knowledge distillation for language models, from the objective function to a study you could publish. An 18-chapter textbook, 13 hands-on labs, and every line of source.
Most distillation write-ups stop at the temperature-scaled loss from 2015. This one keeps going: where teacher logits actually come from, how tokenizers get in the way, what a top-k cache costs, why the choice of divergence changes the model you end up with, and which failure modes only show up once you are training on real checkpoints.
Every claim in the labs is checked with an assertion against autograd or a closed form, so a notebook fails loudly the moment a claim stops being true. Where a result needs a real training run, the lab says so plainly rather than dressing up a toy run as evidence.
The whole course is free and open. No signup, no email gate. Read the textbook in the browser, clone the repo, run the labs.
The textbook reads in the browser with a searchable table of contents. The repository holds the labs, solutions, PDF, and reference code.
Prerequisites: you can read PyTorch and have trained a transformer before. Nothing beyond that.
Nothing is used before it is defined. Part I builds the vocabulary, Part II derives the objective, and the later parts only introduce a technique once the machinery it rests on is already on the table.
Each notebook states its tier on the first line and opens with the question it exists to settle. Solutions live in the repository, one notebook per lab, with the approach explained before the code.
The split follows one rule: the material worth asserting everywhere is the material that needs no training, and the material that needs training can only be validated on the box that trains.
Synthetic logit tensors, real tokenizers, and small cached models. No GPU, finishes in minutes, fully asserted. The saved outputs come from real executions.
This is where the silent bugs actually live. A wrong beta or an unshifted mask trains the wrong thing without throwing an error or even visibly bending the loss curve.
Distillation on real checkpoints, written against cited library APIs. Each lab has three parts: a pre-flight that runs anywhere and is fully asserted, the training run itself behind a flag, and a verdict section with expected ranges and failure signatures.
If the pre-flight passes, the run cannot fail for a silent reason. A green check from a toy stand-in would just be false confidence, so Part B does not pretend.
Hinton on the temperature objective, Kim and Rush on sequence-level KD, Agarwal et al. on GKD, MiniLLM on reverse KL, Minitron and Sheared-LLaMA on prune-then-distill, T-MTB and DistillGuard on the security surface.
The reading happens before the chapter is drafted, and the labs introspect the installed library configs at runtime rather than trusting any document, including this one.
The textbook is the narrative path through the material. The repository holds the executable version: labs, solutions, the reference implementation in code/kd_core.py, the full reading list, and the PDF edition.
Looking for the other courses? Mechanistic Interpretability of Large Language Models covers reverse-engineering transformers, from the residual stream to automated circuit discovery, or see all free courses.