Reverse-engineering transformers, from the residual stream to automated circuit discovery. Two refreshers, eleven chapters, thirteen runnable labs, and a quiz and exam for every unit.
Interpretability writing tends to split into two piles: papers that assume you already know the vocabulary, and explainers that stop before anything is reproducible. This course takes the middle path. It starts by rebuilding a transformer block by hand from the model's own weights, then works up through the logit lens, QK and OV circuits, activation and attribution patching, automated circuit discovery, probing, steering, model editing, and sparse autoencoders.
Every chapter pairs with a Jupyter lab that reproduces that chapter's key experiments on GPT-2 small using TransformerLens. The numbers quoted in the worked examples were measured from those labs' actual runs, not estimated. Every lab runs on CPU, so no GPU is required to complete the course.
The whole course is free and open. No signup, no email gate. Read the textbook in the browser, clone the repo, run the labs.
The textbook reads in the browser with a searchable table of contents. The repository holds the labs, assessments, syllabus, and PDF edition.
Prerequisites: comfortable with Python and basic deep learning. Python 3.10+, torch, transformer_lens, einops, matplotlib.
The book opens with two refresher units that run before Chapter 1. If you can already reimplement a transformer block from its weights and are fluent with einops and hooks, skim them. Otherwise work them fully, labs included, before starting Chapter 1.
Each lab is self-contained, asserts its own checkpoints as it goes, and closes with troubleshooting, stretch goals, and cleanup. All of them run on CPU against GPT-2 small.
The repetition is deliberate. Once you have worked one unit you know exactly where to look in every unit after it.
Each chapter opens with learning objectives and closes with a summary. In between: the terminology it introduces, defined at first use; a worked example carried through end to end; and a going deeper section pointing at the primary sources rather than paraphrasing them.
Numerical values in the worked examples were measured from the labs' actual runs, so the book and the notebook agree.
One Jupyter notebook per unit, in course order. Self-contained, asserting its own checkpoints as it runs, so it fails loudly rather than quietly producing a plausible wrong answer.
Each ends with troubleshooting for the failures that actually happen, stretch goals for going further, and cleanup. GPT-2 small on CPU, so a laptop is enough.
An open-notes quiz of 8 questions and a closed-book exam of 16 questions for every unit, then a comprehensive final of 32 questions after Chapter 11.
Answer keys with explanations are kept as instructor material, so the questions stay usable if you are teaching from this or working through it with someone else.
The textbook is the narrative path through the material. The repository holds the executable version: the 13 lab notebooks, the quizzes and exams, the syllabus, requirements.txt, and the PDF edition.
Looking for the other courses? Distilling Generative Models covers knowledge distillation from the objective function to a reproducible study, or see all free courses.