You have a model that works and a constraint that says you cannot ship it.

The constraint is usually one of four things. The model costs too much per token to serve at the volume you need. It does not fit on the hardware where the work has to happen, which might be a phone, a car, a laptop, or a rack you already paid for. It is too slow, because a user waiting on a response has a shorter attention span than a benchmark does. Or it belongs to someone else, and you are renting access to it through an API on terms that could change.

In every one of those cases you want a smaller model that behaves like the larger one on the work you actually care about. Distillation is the family of techniques for getting one.

The name comes from Hinton, Vinyals, and Dean’s 2015 paper, which framed it as extracting the useful structure out of a large model the way you extract a spirit from a mash.¹ The idea is older than the name. Buciluǎ, Caruana, and Niculescu-Mizil compressed an ensemble of classifiers into a single neural network in 2006 by having the small model fit the ensemble’s predictions on a large pool of unlabeled data.² What Hinton’s paper added was the observation that the large model’s probabilities, not just its decisions, carry information the small model can use, and a specific mechanism for exposing that information. That observation is still the center of the subject twenty years later, and Chapter 5 spends most of its length on it.

What has changed since 2015 is scale and stakes. When the teacher is a 70-billion-parameter language model and the student is a 1.7-billion-parameter one, the practical questions stop being about a loss function and start being about which of the two models is generating text at any given moment, how many bytes per second your memory bus can move, and whether the corpus you trained on has quietly contaminated the benchmark you are about to report. Most of this book is about those questions. But you cannot answer them without first being precise about what distillation is, so that is what this chapter does.

1.1 The setup, stated once#

There are two models. The teacher is fixed: you do not train it, and often you cannot, because you do not have its weights. The student is trained. Somewhere there is a set of inputs, and for each input the teacher produces something the student is asked to reproduce.

The whole design space of distillation falls out of three questions about that picture. Get all three in view before any of them gets an answer.

What does the teacher produce that the student has to match? The candidates run from a single sampled token at the cheap end to the full probability distribution at every position at the expensive end, with truncated distributions, whole generated sequences, and internal activations from the middle of the network in between. Where you land on that range fixes two things at once: how much information flows per training example, and how much it costs to obtain.

Where do the inputs come from? Either from a fixed corpus you already have, or from text the teacher generated, or from text the student generated and the teacher scored after the fact. The last of those is on-policy training and the other two are off-policy, a distinction borrowed from reinforcement learning that matters more than most people expect. Chapter 12 is about it.

How is the mismatch measured? Cross-entropy against a hard label is one answer. A divergence between two full distributions is another, and there is a whole family of those with genuinely different behavior. Chapters 3 and 6 handle this.

Everything else in the subject is a combination of answers to those three questions, plus engineering to make the combination affordable.

1.2 What “knowledge” means here#

The word “knowledge” in “knowledge distillation” is doing real work, and the reason distillation works at all is not obvious, so pin the word down first.

Suppose you have a corpus of text with known continuations. You could train the small model directly on that corpus with ordinary next-token cross-entropy, no teacher involved. That is just language model training. Why would routing the same corpus through a teacher first do anything?

The answer Hinton gave is that the teacher’s probabilities on the wrong continuations carry information that the correct continuation alone does not. Consider a position in a sentence where the true next token is cat. A hard label says: cat is correct, everything else is equally incorrect. The teacher says something much richer. It might put 0.62 on cat, 0.19 on dog, 0.08 on kitten, 0.03 on pet, and spread the remaining 0.08 across forty thousand other tokens including 4e-7 on refrigerator. The ratios among those wrong answers encode a similarity structure the teacher learned: that dog is a plausible substitute here and refrigerator is not, that kitten is close to cat in a way pet is not quite.

That is the intuition, and it is a good one. It is also not the whole story, and one of the things this book will keep doing is telling you where the standard intuition stops being reliable. The places it stops are complicated enough to need the rest of the book, so here they are in outline, now, rather than in a section at the end.

The dark knowledge lives in very small numbers. A probability of 4e-7 contributes almost nothing to a cross-entropy loss dominated by a term of size 0.62. Hinton’s fix is temperature, which flattens the distribution before the loss is computed and therefore amplifies the small values relative to the large ones. Temperature is not a cosmetic knob; without it the mechanism does not fire. Chapter 5 derives exactly what it does, including the factor of $T^2$ that has to accompany it and that gets left out of implementations often enough to be worth its own warning.

Distillation frequently fails to make the student match the teacher, and helps anyway. Stanton and colleagues measured this directly: students trained with distillation often have worse agreement with their teacher on held-out data than you would expect from their improved generalization, and improving the optimization does not close the gap.³ Whatever distillation is buying, “the student becomes a copy of the teacher” is an incomplete description of it. Some of the benefit looks more like a regularization effect from training against a smooth target than like information transfer.

A better teacher is not always a better teacher. Past some gap in capacity, a larger teacher produces a worse student than a smaller one would have. Cho and Hariharan showed this on image classifiers and traced it to the student being unable to fit the larger teacher’s function at all, so the distillation loss stops providing useful gradient.⁴ Chapter 5 covers the competing explanations and what the course’s own capacity-gap probe measures.

I am putting these up front rather than in a “limitations” section at the end because the sequencing matters pedagogically. If you learn the clean story first and the complications later, you will spend a period confidently applying a model of the world that is wrong in ways you cannot see. Better to know from the first chapter that the clean story is a first approximation.

1.3 Where this came from#

The subject has a lineage, and it is worth walking it once, because every step in it was solving a different problem and the shape of current practice is the residue of all of them. There is also a pattern running underneath the usual retelling. Each era assumed a particular level of access to the teacher, mostly without saying so, and the eras where the field moved fastest are the ones where that unstated assumption happened to match what practitioners actually had.

2006: an ensemble you cannot ship. The winning entry in a machine learning competition in the mid-2000s was reliably an ensemble: dozens or hundreds of separately trained models whose predictions are averaged. Ensembles win because averaging cancels errors that individual models make independently of each other. They are also close to unshippable, since serving one means running every member for every request and storing one means storing all of them.

Buciluǎ, Caruana, and Niculescu-Mizil asked whether the accuracy was intrinsic to the size. Their answer was no. The function an ensemble computes is usually not complicated enough to require hundreds of models to express; a single network of modest size has the capacity to represent it. What the single network lacks is a way to find it, because the original labeled training set is small and noisy and ordinary supervised training on it lands somewhere worse. So they used the ensemble as a labeling machine: run it over a large pool of inputs, record what it says, train the single network on that. Where no pool of unlabeled inputs was available, they manufactured one by perturbing the data they had, which is the ancestor of every synthetic corpus in this book.

Hold onto two things from that paragraph. The bottleneck they identified was the training signal rather than the model class, and that diagnosis is still the one the subject rests on. And they needed to run the ensemble as often as they liked, on inputs of their choosing. That is the access assumption the field inherited without stating it, and it took fifteen years for it to become the binding constraint.

2015: probabilities instead of decisions. What the 2006 work transferred was the ensemble’s output in the ordinary sense: the predicted label, or the predicted score. By 2015 the models had changed. Deep networks with softmax heads over hundreds or thousands of classes were standard, and Hinton, Vinyals, and Dean noticed that the output of such a model is not a decision but a distribution, and that nearly all of the distribution lives in the entries nobody looks at.

The reframing did two things. It changed the object being transferred from a function’s decisions to a distribution’s shape, which is the content of §1.2. And it supplied a mechanism for making that shape visible to a loss function, because at $T=1$ a confident classifier’s wrong-answer probabilities are so small that they contribute almost nothing to a gradient. The mechanism matters as much as the observation. Without temperature the information sits in the target and stays invisible to the optimizer, and a reader who takes away “use soft labels” and nothing else will build something that does not work.

Notice the precondition. In a two-class problem there is one wrong answer, and one wrong answer has no internal structure, because there is nothing for it to be relative to. It is the thousand-way softmax of the image classification era, and later the fifty-thousand-way softmax of a tokenizer, that makes the wrong answers worth transmitting at all.

2014 to 2019: everything else the teacher knows. Once the target was a distribution, the obvious next question was whether the output was the only thing worth copying. A trained network is a stack of representations, and the final distribution is a lossy summary of the last one. FitNets asked the student to match an intermediate activation of the teacher through a learned projection, under the name “hints,” and the reason it worked was that a thin deep student got a training signal at its middle rather than only at its end. Attention transfer changed the target again, from what the network computes to where it looks, on the argument that attention maps are indexed by position rather than by channel and therefore compare across networks of different widths.²⁰ Relational methods changed it a third time, to the geometry among examples rather than the examples themselves.

Every one of those methods requires the teacher’s weights and the ability to read out its insides. The vision era took that for granted, because vision checkpoints were files you downloaded, and it is worth registering how much of the classical literature is white-box work assuming a world where the teacher belongs to you.

2019: the recipe arrives in language. Transformer encoders got good and immediately got expensive to serve at web volume, which recreated the 2006 problem with different models. DistilBERT is the compact answer: initialize the student from every other layer of the teacher, train on the teacher’s output distribution plus a term that aligns the two models’ embeddings, and keep most of the quality at roughly half the depth.²¹ TinyBERT is the maximal answer: match embeddings, hidden states through learned projections, attention matrices, and predictions, in two stages. The pair is instructive because the simple recipe does not lose to the elaborate one by much, and Chapter 14 returns to that comparison with the machinery to make it fairly.

2016, on a separate track: sequences. Everything above concerns a model that emits one distribution per input. A translation model emits a distribution over sequences, and the set of sequences is exponentially large, so the object you would like to match cannot be written down. Kim and Rush’s move was to approximate that intractable object with a single sample: take the teacher’s most likely output sequence, treat it as if it were training data, and fit the student to it with ordinary cross-entropy. The approximation is crude, the results were good, and the method has a property nobody was optimizing for at the time. It needs nothing from the teacher but text.

2023 onward: the generative era. Three things changed at once, and their interaction is why current practice looks different from the classical literature rather than like a scaled-up version of it. Teachers got large enough that generating a corpus with one is the dominant cost of a project rather than a preprocessing step, which inverts the usual ordering of what is expensive. Teachers moved behind APIs, so access stopped being an assumption and became a constraint that varies by vendor and by quarter. And the output moved from a label to a long generation judged by how it reads, which means the quantity you optimize and the quantity you are graded on stopped being the same quantity.

The methods track those changes. MiniLLM argued that a student too small to cover everything the teacher can say should not be trained on an objective that asks it to try, and replaced the forward divergence with the reverse one, which is Chapter 6’s subject.²² Generalized knowledge distillation moved the training inputs onto the student’s own generations, which is §1.4.4 and Chapter 12. DeepSeek’s R1 report went the other way entirely and distilled a series of small models with supervised fine-tuning on teacher-generated reasoning traces and nothing else, a method Kim and Rush would recognize on sight, and the resulting models were strong enough that the simple method is now the reference point rather than the fallback. Meanwhile the cross-tokenizer problem arrived, because a practitioner who wants a specific small model and a specific large one frequently cannot have them share a vocabulary; Universal Logit Distillation is the current answer and Chapter 14 covers what it gives up to get there.²³

2026. On-policy methods are the active area, active enough to have accumulated a survey of their own, although that survey describes itself as ongoing work rather than a settled account.²⁴ The honest one-sentence summary of twenty years is that the object of study moved from “compress a classifier” to “transfer a behavior,” and the binding constraint moved from compute to access.

Read the figure against your own situation before you read another method paper. If a method sits in a lane you are not in, its results do not transfer to you, however good they are.

1.4 A taxonomy that survives contact with practice#

Textbook taxonomies of distillation usually sort methods by what kind of signal the teacher provides, following the survey literature.⁵ That axis is real and I will use it. But it is not the axis that determines what you can actually build, so I want to lead with a different one and come back to the classical split afterward.

Before the axes, one scenario to hang them on, because a taxonomy is easy to agree with and hard to use. I am going to carry a single project through all four axes and say what each one answers for it and what each answer eliminates. The project is deliberately ordinary.

The product. A team ships an assistant that currently runs in the cloud against an 8-billion-parameter instruction-tuned model. They want the same assistant on the handset: rewriting and shortening messages the user is drafting, pulling dates and tasks out of pasted text, answering questions about content already on the device. The requirement that starts the project is that it has to work with the network off.

The constraint. The handset can give a model roughly 4 GB and needs the first token back in well under a second. At half a byte per weight, which is what a 4-bit format costs, four gigabytes is an 8-billion-parameter model on paper and considerably less once the key-value cache, the runtime, and the operating system take their share. The target lands at 1.7 billion parameters: 3.4 GB in bf16 during development on the reference machine, near 1 GB once quantized for the device. Nothing about that number came from the distillation literature. It came from a memory budget, which is where student sizes usually come from.

The access. The 8B teacher is an open-weights model, downloaded, sitting on disk, with its weights readable and its activations available to anything that asks.

The corpus. 400,000 prompts logged from the cloud version over six months, de-identified. That is the real asset here: they are drawn from the distribution the student will actually be asked about, which no public instruction corpus is, and §1.8’s second limit is about the teams that do not have such a thing and use a general corpus instead.

1.4.1 Sort by what access you have#

This is the first question to ask about any distillation problem, because it eliminates most of the method space immediately.

Table 1.1 What access to the teacher buys you.

The boundaries move. APIs that once returned log-probabilities have stopped; models that were closed have been released. But at any moment, for any given teacher, you sit in exactly one of those rows, and the row determines the chapter of this book that applies to you.

The white-box row deserves one note now because it contains a method people forget exists: if you have the teacher’s weights, one of your options is to build the student out of the teacher. Delete some layers, keep the rest, and every parameter you kept is already trained. Chapter 13 covers this, and the economics are startling enough to change how you plan a project. Minitron reported matching a from-scratch model of the same size at a small fraction of its training compute by pruning and then distilling briefly.⁶

The black-box row deserves a note too, because it is where a great deal of current practice actually lives, and because the method it forces you into is the simplest one in the book. If all you can get is text, you generate a corpus with the teacher and fine-tune the student on it with ordinary cross-entropy. No divergences, no temperature, no logits. DeepSeek’s R1 report describes exactly this for its distilled model series: supervised fine-tuning on teacher-generated reasoning traces, with no reinforcement learning stage applied to the students.⁷ It works well enough to be the reference point that fancier methods have to beat, and Chapter 11 is about when they do.

The on-device assistant sits in the white-box row, and the instructive thing about that answer is how little it constrains. Every method in this book is available, so the access axis eliminates nothing and the design has to be settled on other grounds. The top row is the comfortable row to be in and the hard row to plan in.

It is also less stable than it looks. The teacher is someone else’s model under a license, and a license can be superseded. If a re-licensing or an internal legal review moved this project to the bottom row next quarter, everything built on logits would stop existing overnight while the trace fine-tuning path kept working unchanged. That asymmetry is why I now build the black-box baseline first on any project whose teacher belongs to someone else, even when I have logits. It survives every change of access, and it is the number the sophisticated methods have to beat anyway.

1.4.2 Sort by what the student is asked to match#

This is the classical axis, and the three categories the surveys use are worth having.⁵

Response-based distillation matches the teacher’s outputs. For a language model that means the next-token distribution, or a truncation of it, or a sampled token, or a whole generated sequence. This is where the field spends most of its time, and Parts II through IV of this book are almost entirely about it.

Feature-based distillation matches internal representations. Pick a layer in the teacher and a layer in the student, and add a loss term that pushes their hidden states together, usually through a learned linear projection because the dimensions differ. FitNets introduced this in 2014 under the name “hints,” and TinyBERT built a full recipe on it for transformer language models.⁸⁹ Chapter 14 covers it, including the check that tells you whether the projection you need can even exist before you spend a training budget finding out.

Relation-based distillation matches the relationships between examples rather than the examples themselves: if the teacher considers inputs A and B similar and A and C distant, the student should too, regardless of where in representation space it puts them. Park and colleagues formalized this, and Tian and colleagues gave a contrastive version.¹⁰¹¹ This branch is less developed for generative language models than for vision, and I will be honest about that in Chapter 14 rather than implying a maturity the literature does not have.

For the on-device assistant, response-based is the default and feature-based is a live option rather than an obvious one. The teacher and student differ in both width and depth, so a feature-matching term needs a learned projection between hidden states of different sizes, and Chapter 14’s ridge check tells you whether one can exist before you spend a training budget finding out that it cannot. Relation-based is eliminated, and the reason is worth stating precisely: not that the situation forbids it, but that the methods are immature for generative models and the project has a delivery date. Ruling out a branch on maturity grounds is legitimate; pretending the elimination was technical is not.

1.4.3 Sort by granularity#

For a model that produces sequences, there is a third axis with no equivalent in classification.

Token-level distillation treats each position independently. At position $t$, the teacher has a distribution over the vocabulary, the student has one, and the loss compares them. Sum over positions. This is the direct translation of Hinton’s objective into the sequence setting and it is what most people mean when they say “distillation” about a language model.

Sequence-level distillation treats the whole output as the unit. Kim and Rush’s version, which is the original, approximates the intractable sum over all possible sequences by using the teacher’s most likely output as a single sample and training the student to reproduce it with ordinary cross-entropy.¹² The approximation is crude and it works remarkably well, which is one of several places in this subject where theory and practice have an uncomfortable relationship.

The distinction matters because the two methods fail differently. A token-level student can match the teacher at every position and still produce incoherent text, because matching a conditional distribution at every position under teacher-provided context says nothing about what happens when the student has to condition on its own output. Chapter 12 is about exactly that gap.

The on-device assistant starts token-level. The corpus is prompts, the teacher scores each one in a single parallel forward pass, and the replies the product needs are short and structured: a rewritten message, an extracted date, a two-sentence answer. Length is what makes the token-level failure mode bite, and a hundred-token reply accumulates far less drift than a two-thousand-token chain of reasoning. That is an argument for starting token-level rather than a proof that it will suffice, because the acceptance test is what the generations read like while the objective is measured on conditional distributions under context the teacher supplied.

1.4.4 Sort by where the training inputs come from#

The last axis is the one that has moved the most in the last three years, and it is the reason Chapter 12 exists.

Off-policy distillation trains the student on text it did not produce: a fixed corpus, or the teacher’s generations. The name is borrowed from reinforcement learning, where a policy is the thing that chooses actions and “off-policy” means learning from data generated by some policy other than the one you are training.

On-policy distillation trains the student on text the student generated, with the teacher scoring those generations after the fact. The student produces a rollout, the teacher says what it would have done at each position of that rollout, and the loss compares them.

The argument for on-policy training is exposure bias: a model trained only on text it did not produce never sees its own mistakes, and therefore never learns to recover from them.¹³ The argument was made for sequence models long before distillation adopted it, and scheduled sampling was an early attempt at a fix.¹⁴ Agarwal and colleagues built the modern distillation version, generalized knowledge distillation, which interpolates between the two regimes with a single parameter.¹⁵

The on-device assistant goes off-policy first. Scoring 400,000 prompts once with the teacher is a prefill workload, the cheapest row of Table 1.3, and what comes out is a durable asset: the cached distributions survive every later change to the student’s learning rate, divergence, initialization, and schedule, so the teacher is paid for once. Chapter 10 is that pipeline. On-policy becomes the second phase if the first leaves a gap that shows up in generations rather than in the loss. That ordering is about more than cost, since the off-policy phase is also the cheapest way to learn whether a 1.7B student can hold this capability at all, which is §1.9’s third failure class and the one you want to meet in week one rather than week six.

There is a cost inversion sitting under that ordering, and it is the reason Chapter 9 exists in the shape it does. On-policy distillation is the cheap path on this machine and a teacher-generated corpus is the expensive one, because what decides the price is which model decodes, and in on-policy training the decoding is done by the student while the teacher only ever scores text that already exists. I had it backwards for a while, for reasons §9.5 dissects; until you have read that arithmetic, take the ordering in Table 1.3 as a claim and not a result.

1.5 Why it works, as honestly as the evidence allows#

There are four explanations in circulation for why a student trained against a teacher beats the same student trained against hard labels. They are not mutually exclusive and the evidence supports different amounts of each.

The information argument. The teacher’s full distribution contains more bits per training example than a one-hot label does. A vocabulary of 49,152 tokens with a one-hot label carries at most ${\log }_{2}49152\approx 15.6$ bits; the full distribution over the same vocabulary carries substantially more, because it specifies a value for every entry. More bits per example means fewer examples needed for the same amount of learning. This is the argument Hinton made and it is the easiest to state.

The regularization argument. Soft targets are smoother than hard ones, and training against a smooth target reduces overfitting the same way label smoothing does. If this were the whole story, label smoothing should be a substitute for distillation, and it is not: Müller, Kornblith, and Hinton showed that a teacher trained with label smoothing distills worse than one trained without, even though the smoothed teacher is itself more accurate.¹⁶ The reason is that label smoothing collapses the very structure among wrong-answer probabilities that distillation transmits. That result is one of the cleanest pieces of evidence that the information argument is doing real work, and Chapter 5 reproduces the reasoning.

The optimization argument. The teacher’s outputs define a smoother loss landscape than hard labels do, so the student’s optimization problem is easier even when the optimum is the same. Beyer and colleagues pushed this line hard, showing that treating distillation as function matching, meaning making the teacher and student see identical inputs under identical augmentation and training for a very long time, produces much better students than the usual recipe.¹⁷ Their headline finding is that patience matters more than most architectural choices, which is not what the field expected.

The curriculum argument. In the sequence setting specifically, the teacher’s generated text is easier to learn from than natural text, because it is more consistent, less noisy, and drawn from a distribution a model can actually represent. This is folklore rather than settled science, but it is the most common explanation offered for why sequence-level KD works as well as it does despite its crude approximation, and Kim and Rush’s original analysis gestures at it.¹²

1.6 What makes a good teacher#

The teacher is usually chosen in about four seconds. Somebody asks which model is the strongest one the project has access to, that model becomes the teacher, and every later decision happens downstream of a choice nobody wrote down. It is the wrong procedure, and the reasons are specific enough to list, because each one is a case where the obvious criterion and the correct one point in different directions.

Teacher accuracy is not the selection criterion. The distillation loss never reads the teacher’s accuracy. It reads the teacher’s distribution, and nearly all of that distribution sits in the entries that are not the answer. Two teachers can agree on the top-1 prediction at every position of your corpus, score identically on every benchmark you would put in a report, and differ by an order of magnitude in how much structure their tails hold. The first has something to teach; the second is an expensive way to produce hard labels. What separates them is a one-number summary of how unequal the wrong-answer probabilities are, derived in Chapter 5, costing one forward pass over a few hundred prompts.

A label-smoothed teacher is a worse teacher and a better model. §1.5 gave the result: the smoothed teacher is more accurate and better calibrated and it distills worse, because the smoothing objective rewards making every wrong answer equally probable and that equality is the absence of the thing being transferred. The rule generalizes past label smoothing to any procedure that pushes the wrong answers toward each other, and what makes it unsettling is that the damage is invisible to everything anyone reports, because the argmax does not move.

A bigger teacher can be a worse teacher. §1.2 introduced the capacity gap and Chapter 5 covers the competing explanations for it. The planning consequence is that the parameter which matters is the ratio between teacher and student rather than the teacher’s absolute size, so a 70B teacher for a 1.7B student can produce a worse student than an 8B teacher would have. Under the explanation Cho and Hariharan give, the student cannot represent the larger teacher’s function at all, so the distillation term spends its gradient pointing at an unreachable target.

An early-stopped teacher can be a better teacher. This is their remedy, and it breaks people’s intuition hardest: take the teacher’s checkpoint from partway through training, the one you would refuse to ship, and it distills better than the finished model does. If you have intermediate checkpoints of a model you trained, you have candidate teachers, and losing the internal bake-off is not evidence about how they will do here.

The teacher does not have to be larger at all. Born-again networks distill a model into a fresh copy of its own architecture, same size, same everything, and the students come out ahead of the teachers.²⁵ Whatever is happening there, “the teacher must be big” is a convention inherited from compression being the original motivation rather than a requirement of the method.

One framing ties those together. Soft labels are a trade: lower variance in the training signal against a bias toward whatever the teacher is wrong about.²⁶ You want the teacher whose trade lands well for your student on your corpus, and there is no leaderboard for that.

So, a procedure that fits inside a normal project.

1. Enumerate the candidates you can actually run: every size in the family you have access to, the base and instruction-tuned variants separately, and any intermediate checkpoints of your own.
2. Score a few hundred prompts from your own corpus with each candidate, one prefill pass. Record three numbers per candidate: top-1 agreement with whatever reference you have, mean entropy of the distribution, and the spread of log-probabilities across the non-top-1 tokens.
3. Discard the candidates whose tails are flat, however they rank on accuracy. They have nothing to transfer and you would be paying full price to receive it.
4. Among the survivors, take the two that differ most on those numbers and run the shortest distillation you can afford with each, at matched budget, one key different.
5. Decide on the student’s numbers, not the teacher’s.

Every step in that procedure ends up measuring the student. The teacher’s own scores are a proxy for the thing the project cares about, and this section is a list of four named ways that proxy has been shown to fail.

1.7 The student’s side of the design#

Four decisions define the student: its architecture family, whether it shares the teacher’s tokenizer, how its weights are initialized, and how big it is. Three of those four are settled before a single training step runs, and none of the three can be revised afterward without throwing away everything trained so far. That makes them the decisions with the most riding on them, and it puts them in an awkward relationship with a literature that spends most of its length on the divergence, the temperature, and the mixing coefficient. Those three are real and Chapters 5 and 6 treat them seriously. They are also revisitable in an afternoon.

Size comes from the deployment constraint, not from the method. Work backwards from where the model has to run, as the on-device assistant did in §1.4: available memory, minus the runtime, minus the key-value cache at the context length you need, divided by the bytes per parameter of the format you will ship in. What comes out is a parameter count, and the distillation design begins after it. Size also fixes the teacher-to-student ratio, which is the capacity-gap parameter, so §1.6 and this section are two halves of one decision rather than two.

Architecture family decides whether teacher and student are relatives. Taking the teacher’s own small sibling buys you a shared tokenizer, shared chat-template conventions, similar width-to-depth proportions, and a live option on feature matching because the layers correspond in a way you can at least argue about. Taking another vendor’s model of the same size buys you whatever that model is better at, which for on-device work is often a genuinely different thing: kernels hand-tuned for the target chip, or a smaller vocabulary. The vocabulary point is not cosmetic at this scale, since the embedding and output matrices are a large fraction of a 1.7B model’s parameters. Note also that “a small model” is itself two different objects: one obtained by training at the target size on a carefully constructed corpus, and one obtained by pruning a larger model down to the target size and continuing to train it.²⁷²⁸ Both hand you a checkpoint. They hand you different checkpoints.

Tokenizer sharing is not an implementation detail. If the two models tokenize differently, then teacher position 7 and student position 7 are not the same span of text, and a position-wise loss is comparing distributions over different things while reporting a number that looks fine. Chapter 7 proves that no general position-wise alignment exists across tokenizer families, and Chapter 14 covers the methods that recover a usable signal anyway and what each one gives up. Everything in Parts II and III of this book assumes a shared tokenizer, so this single decision determines whether two of the book’s five parts apply to you. It is also a serving decision, because the tokenizer sets how many tokens a given reply costs, and tokens are what decoding time is measured in.

Initialization is the one you can revisit. Random, a published checkpoint, or the teacher with layers removed. It changes the starting weights and nothing about the interfaces, so it is the one of the four you can change on Tuesday without invalidating Monday. Chapter 13 measures what each is worth in training steps, and the answer is large enough that treating initialization as a default rather than a decision is the most expensive habit in this list.

For the on-device assistant, three of the four fell out of decisions already made. The size came from the handset. The family came from the teacher, because the 8B model has a 1.7B sibling and sharing the tokenizer keeps the whole book available. That leaves initialization as the only one still open, which is the arm worth running: the published sibling checkpoint against the teacher pruned to the same size, at matched budget. Three decisions made before training and one experiment worth doing is the typical distribution, and it is close to the inverse of where a first-time reader of the literature would expect the consequence to sit.

1.8 What distillation cannot do#

A book that only says what a technique achieves is not useful for planning, so here are the three limits, stated plainly.

It does not create capability the student’s architecture cannot hold. If a task requires more computation per token than the student can perform, no amount of teacher signal supplies it. The capacity gap is the mild version of this; the hard version is that some behaviors seem to require scale and do not survive compression at any ratio. The field does not have a crisp characterization of which behaviors those are, which is an honest gap rather than a failure of this book to look.

It does not transfer what the teacher does not express on your inputs. Distillation is supervised by teacher outputs on the corpus you chose. Teacher behavior that your corpus never elicits does not transfer. This has a practical consequence people discover late: a student distilled on a general instruction corpus can be excellent on general instructions and terrible on your domain, not because the teacher was bad at your domain but because you never asked it.

It does not launder provenance. Whatever is in the teacher can arrive in the student, including things you did not want. Chapter 17 covers the measured version of this: ordinary backdoors mostly do not survive distillation, but triggers built from tokens that are common in distillation corpora do, and the difference is measurable on your own pipeline before you ship.¹⁸ The same chapter covers the other direction, where distillation is the attack rather than the pipeline, and someone is extracting a model you serve.

1.9 The four ways a distillation project fails#

Those are limits of the technique. Failures of projects are a different list and a more useful one, because a limit is something you plan around and a failure is something you diagnose at 11pm with a training run three days in. Across the work I have done and the work I have watched, distillation projects go wrong in one of four ways. The value of having exactly four is that they are diagnosed differently and in a definite order, and each has a chapter that handles it.

The pipeline is silently wrong. Nothing crashes, the loss descends, the curve looks like a curve. Meanwhile the labels are shifted by one position relative to the logits, or the prompt tokens are in the loss when they should be masked out, or the teacher and the student tokenize the same string into different numbers of tokens and the loss is comparing position 7 to position 8, or the cache was built from a corpus revision that is not the one being trained on, or the temperature is applied to one side of the divergence and not the other. The signature is a run that trains to a plausible loss and produces a student no better than the hard-label baseline, or one that looks better on the loss and worse on anything you generate from it. You will not find these by reading the code, because the code looks correct, which is why it shipped. You find them by assertion: recompute the framework’s loss externally and compare to float precision, fingerprint the corpus and check it at load time, compute the set of configuration keys by which two arms differ and assert it has exactly one element. Chapter 7 is where these bugs live, Chapter 8 is the discipline that catches them before a run, and Chapter 10 covers the cache-specific ones.

The method does not suit the access you have. You designed around output distributions and the vendor removed the log-probability endpoint. You designed around weights and the legal review came back saying generated text only. You designed around a shared tokenizer and the model the device team ships uses a different one. It presents as a project that stalls with a plan nobody can execute, at the point where the plan meets a constraint that was decided somewhere else in the organization. The defense is to settle the access row first, as §1.4.1 argues, and to build the black-box baseline early because it is the one method available in every row. Chapter 11 is that baseline, and Chapter 12 is what query access buys on top of it.

The student cannot hold the capability. The capacity gap is the mild form; the hard form is a behavior that does not survive compression at any ratio you can afford. The signature is distinctive once you have seen it. The student closes most of the gap quickly and then stops, and the residual does not move for more steps, more data, a different divergence, or a better teacher. It also concentrates rather than spreading, so the remaining errors pile up on a recognizable subset of the evaluation, usually the items needing the most computation per token: long chains of arithmetic, multi-step retrieval across a long context, anything where an intermediate result has to be held and reused. The defense is a small-budget probe before the budget is committed, and the honest response when it fires is to change the size, the initialization, or what you are promising. Chapter 5 covers the mechanism and Chapter 13 covers the two responses that are about the student.

The evaluation cannot see the difference. The subtlest one, the most common in published comparisons, and the reason Part V exists. The loss went down and top-1 agreement with the teacher went up, and neither is evidence, because the loss is computed on the distribution the student was trained toward and agreement counts argmax matches, which is close to the statistic the objective optimized. Both numbers can improve while the model gets worse in ways nobody measured: calibration degrades, so the model’s confidence stops meaning anything;²⁹ output diversity collapses; the eval set overlaps the distillation corpus and the score is partly a memorization score. The defense is to build the evaluation before the run and decide in advance what would count as a win, which sounds like bureaucracy and is the difference between a result and an anecdote. Chapter 16 is the evaluation, Chapter 18 is the practice of registering the decision beforehand, and Chapter 17 is the version where the thing you cannot see is a security property rather than a quality one.

Table 1.2 The four failure classes, in the order to check them.

The table’s ordering is diagnostic and differs from the order I introduced them in. When a run disappoints, check the pipeline first, because it is cheap and it is the most likely explanation. Check the evaluation second, because it is also cheap and because being wrong about it wastes everything downstream. Check the method against your access third, and capacity last, because ruling capacity in or out costs a training run and it is the only one of the four whose remedy is to change what you promised.

1.10 The economics, in one page#

Everything in Part III and Part V of this book rests on a small number of facts about hardware, so here is the shape of the argument in advance. Chapter 9 derives it.

Running a transformer has two very different modes. Prefill processes text that already exists, which means every position can be computed at once, which means the arithmetic units stay busy. Decode generates one token at a time, and each step has to read the entire weight matrix out of memory to produce a single token, which means the memory bus is the bottleneck and the arithmetic units idle.

The ratio between them on real hardware is large. Measurements on a 20-billion-parameter model in a 4-bit format on this class of machine put prefill near 2,053 tokens per second and decode near 49.7, a ratio of roughly 40 to 1. That single number reorganizes the whole method space:

Table 1.3 Distillation workloads sorted by who generates.

Read the table as a planning tool. Two of those rows are nearly free, one is affordable, and one costs two orders of magnitude more than the others. If your plan involves the expensive row, you should know it before you start, price it explicitly, and then decide whether to buy the corpus once and keep it forever, use someone else’s, or rent different hardware for that stage alone.

The memory side is simpler and just as binding. A parameter in bf16 costs 2 bytes to hold for inference. Full fine-tuning costs about 16 bytes per parameter once you count bf16 weights, bf16 gradients, and the fp32 optimizer moments and master weights that a standard Adam setup keeps. Low-rank adaptation collapses the optimizer share to nearly nothing, which is why it shows up whenever a configuration would otherwise not fit.¹⁹ Appendix B has the tables.

1.11 The reference machine#

Every number in this book that depends on hardware refers to one specific machine unless it says otherwise: a single workstation with 128 GB of unified CPU-GPU memory at roughly 273 GB/s of bandwidth, on arm64, CUDA compute capability sm_121.

I am being specific on purpose, for the reason the preface gives: the same method is cheap on one machine and impossible on another, and the ordering of methods by cost inverts between hardware classes. The machine above has generous capacity and modest bandwidth, which is a combination that makes large models fit and makes them slow to generate from. On a machine with the opposite profile you would make different choices, and Chapter 9 gives you the arithmetic to redo the comparison for whatever you have rather than inheriting mine.

1.12 How this book is organized#

The table of contents lists the parts and I am not going to restate it here. Two things about the arrangement are not visible from a contents page, and both change how you should read.

The first is that Part III is the part I would put in a practitioner’s hands if I could only give them one. It covers tokenizers and alignment, where the silent bugs live; the first complete run and the discipline around it; the hardware economics; and then the cached-logit pipeline, which is the cheapest correct thing you can build on this machine and the one I would reach for first on a new project. Parts I and II exist to make Part III readable rather than as ends in themselves, and Parts IV and V are what you reach for once the cheap thing is working and you want to know whether it is good.

The second is that the book has a companion: thirteen laboratory notebooks that execute everything described here. The relationship is deliberate. The book explains and derives; the labs assert. Almost every claim in Part I and Part II is checked mechanically in a notebook against autograd or a closed form, so if a claim ever stops being true the notebook fails loudly. Where a chapter states a measured number, that number came out of one of those executions, and the chapter says which.

1.13 Where this lands in the labs#

Lab 00 is the entry point and it presumes nothing from this chapter beyond the vocabulary. The taxonomy here is not itself asserted anywhere, because taxonomies are not the kind of thing you can assert; what the labs check is every specific claim the taxonomy organizes, and they are scattered across the whole course rather than gathered into a Lab 01. The access rows of Table 1.1 turn into concrete pipelines in Labs 04, 06 and 07, one per row, and running any two of them back to back teaches the ordering in Table 1.3 faster than the arithmetic in Chapter 9 does. If you want the fastest possible confirmation that the subject is real before committing to any of that, Lab 01’s temperature-limit cell takes under a second and shows the $T^2$ factor doing its job on a fixed logit vector you can read in one line.

1.14 Exercises#

1. For each of the four constraints in this chapter’s opening paragraph, name a distillation method from Table 1.1 that addresses it and one that does not. Where a method addresses the constraint only partially, say what remains.

2. A colleague proposes distilling a 70-billion-parameter teacher into a 1-billion-parameter student by having the teacher generate 10 million tokens of training corpus. Using only Table 1.3 and the bandwidth figure from §1.11, estimate the wall-clock cost of the generation step to within a factor of two, and state every assumption you made. Chapter 9 will give you the tools to do it properly; do it roughly now and compare later.

3. The regularization argument and the information argument make different predictions about what happens when you distill from a teacher trained with heavy label smoothing. State both predictions before reading §1.5’s answer, then say which prediction the measured result supports.

4. You have API access to a teacher that returns the top 20 log-probabilities per position. Which row of Table 1.1 are you in, and which methods from §1.4.2 and §1.4.3 become available or unavailable? Name one piece of information you are missing and one way to bound its size.

5. Stanton and colleagues found that students often generalize better while agreeing with their teacher less than expected. Propose two experiments that would distinguish “the student learned a different function that happens to be better” from “the student learned the teacher’s function badly in a way that happens to help.” Say what each experiment would measure.

6. Give a task where you would expect distillation to fail outright, using the three limits in §1.8. Be specific enough that someone could test your prediction.

7. You are handed four candidate teachers for the on-device assistant in §1.4: the 8B instruction-tuned model, the same model’s base checkpoint, a 32B instruction-tuned model from the same family, and an 8B checkpoint saved 40 percent of the way through instruction tuning. Rank them using only the five facts in §1.6, saying for each which fact applies and in which direction. Then name the one measurement that could overturn your ranking and say what it costs.

8. A run of the §1.4 scenario finishes. The distillation loss is a third of where it started, top-1 agreement with the teacher on held-out prompts is 71 percent against 44 percent for the hard-label baseline, and the product team reports that the replies read worse than the baseline’s. Assign this to one or more of the four classes in §1.9, say which you would test first and why, and give the cheapest test for each class you named.

Every method in this book bottoms out in the same operation. A model emits a vector of raw scores, one per vocabulary entry. That vector becomes a probability distribution over the next token. Some other distribution, obtained the same way from another model, is compared against it, and the comparison produces a number that gets differentiated. That pattern is what the surveys call response-based distillation.¹⁴¹⁵ Chapters 3 and 6 are about which comparison to use. This chapter is about what comes first: the distribution itself, and the floating-point arithmetic that has to hold it.

The second half of that sentence is the part people skip, and the part that costs them a week. A next-token distribution has between 50,000 and 256,000 entries spanning forty orders of magnitude: the token the model is confident about sits near 0.9, and one it has ruled out sits at $10^{-30}$ or lower. Distillation is the business of caring about the small entries, because that is where the teacher’s learned structure lives. So it is the business of doing arithmetic near the floor of whatever number format you are in, and the format has a floor, and things below the floor become exactly zero, and the logarithm of exactly zero is negative infinity.

Here is what that looks like from the outside. You launch a run and the first loss prints as nan. Or worse, a thousand losses print as ordinary numbers, then one prints as inf, and everything after it is nan. The natural reading of that dashboard is “the optimizer diverged,” and the natural response is to lower the learning rate, add gradient clipping, and try again. That response is wrong, it costs hours, and the cause is one function call that took a logarithm of something which had underflowed to zero. I have watched people, myself included, chase the wrong hypothesis here.

So this chapter builds the object, then the number system under it, and tries to leave you able to predict which arithmetic survives before you run it.

2.1 What a next-token distribution is#

A language model, given a sequence of tokens, produces at each position a vector of real numbers with one entry per vocabulary item. Call the vocabulary size $V$ and the vector $z \in \mathbb{R}^V$. These numbers are not probabilities. They are unnormalized scores, and they are called logits.

The map from logits to probabilities is the softmax:

One property of the softmax determines most of what follows, and it is worth deriving rather than accepting.

The softmax is shift invariant. Add the same constant $c$ to every logit and the distribution does not change:

$$\mathrm{softmax}(z + c)_i = \frac{e^{z_i + c}}{\sum_j e^{z_j + c}} = \frac{e^{c}\,e^{z_i}}{e^{c}\sum_j e^{z_j}} = \frac{e^{z_i}}{\sum_j e^{z_j}} = \mathrm{softmax}(z)_i.$$

The factor $e^c$ appears once in the numerator and once in every term of the denominator, so it factors out and cancels. Nothing about this is deep, and it has three consequences that are.

First, only logit differences carry information. If someone tells you a model assigned a logit of 14.2 to a token, you have learned nothing, because 14.2 relative to what is the only question. Comparing two models’ raw logits entry by entry is meaningless; comparing their log-probabilities is fine.

Second, the softmax is not injective. A whole line of logit vectors, ${z + c : c \in \mathbb{R}}$, maps to the same distribution, so “match the teacher’s distribution” and “match the teacher’s logits” are different objectives and the second is strictly stronger. Chapter 5 shows they converge at high temperature, which is a fact about the limit rather than the general case.

Third, and this is the one that saves the computation, you are free to subtract anything you like from all the logits before exponentiating. Section 2.2 uses that freedom.

A note on notation, fixed here for the rest of the book: $p$ is always the teacher’s distribution and $q$ is always the student’s. The course’s code follows the same convention, while the divergence functions in kd_core take arguments in the order (student, teacher), the opposite of the order they appear in the mathematics. Chapter 3 puts a warning on that.

The vocabulary sizes involved are large enough to change the arithmetic. The course prices three: Llama 3 at 128,256 entries, Qwen2.5 and Qwen3 at 151,936, Gemma 2 and 3 at 256,000.¹ The 360M-parameter student the labs train has 49,152.² A dense distribution over 151,936 entries in a 16-bit format is about 302 KB per token position, the number that makes Chapter 10 exist. That figure rounds the vocabulary to 151k, which is the form the labs carry it in; multiply the exact 151,936 out and you get 303.9 kB, and the difference has never changed a decision made with it.

2.2 Overflow, and why the textbook formula does not run#

Write out the softmax exactly as it appears above, in code, and it will fail on real inputs.

Every floating-point format has a largest finite value. A calculation whose true result exceeds it does not wrap around or saturate; it produces inf, and inf divided by inf produces nan, and nan propagates through every later operation until the whole tensor is nan.

The single-precision format, fp32, tops out at about $3.4\times 10^{38}$. So the question “which logits overflow when you exponentiate them” has an exact answer:

That is the whole derivation. No rule of thumb is involved: 88.7 is the natural logarithm of fp32’s ceiling, so exp of any logit above roughly 88 overflows and every logit below it is fine.

Half precision, fp16, has a much lower ceiling of 65,504, and the same derivation gives

so exp of any logit above about 11 overflows fp16. Eleven. A trained language model routinely emits logits well past that.

Now put realistic numbers in. Logits from a trained checkpoint sit around 10 to 40 in magnitude, comfortable in fp32 and already dangerous in fp16. Then divide by a temperature, an operation Chapter 5 performs constantly and Section 2.4 defines. A temperature of 0.25 multiplies every logit by 4, so logits near 30 become logits near 120, well past 88.7, and the naive softmax overflows in fp32. Lab 00 runs this case on the vector $[31.0,29.5,27.0,12.0]$ at $T=0.25$ and asserts that the naive implementation produces nan while the stable one matches PyTorch’s F.softmax to $10^{-7}$.

The fix is the third consequence of shift invariance. Let $m={\max }_j{z}_j$ and compute

By shift invariance with $c=-m$, this is the same distribution, exactly rather than approximately, and the largest exponent is now $e^0=1$, so no term can overflow. Terms can still underflow, which is Section 2.3’s problem, but they cannot blow up. The transformation costs one pass to find the maximum, and it is what every library does inside F.softmax.

The listing below is the shape of the idea. Look at the single line of difference between the two functions and at what it does to the exponent.

import torch

def naive_softmax(z):
    e = z.exp()                                  # e^124 has nowhere to go in fp32
    return e / e.sum(-1, keepdim=True)

def stable_softmax(z):
    m = z.max(-1, keepdim=True).values           # shift invariance says this is free
    e = (z - m).exp()                            # largest exponent is now exactly 0
    return e / e.sum(-1, keepdim=True)

z = torch.tensor([31.0, 29.5, 27.0, 12.0]) / 0.25   # temperature 0.25 -> logits near 120
print(naive_softmax(z))    # tensor([nan, nan, nan, nan])
print(stable_softmax(z))   # tensor([9.9753e-01, 2.4726e-03, 1.1226e-07, 9.8298e-34])

The two functions implement the same formula and differ in whether the intermediate quantities fit in the format. That distinction, between the mathematics being right and the arithmetic being survivable, is the subject of this chapter.

You will not write naive_softmax on purpose. You will meet it when you implement anything below the level of the framework, and in a nastier form when someone builds a logit cache and stores $e^z$ instead of $z$ to save an exponential at training time. The stored values carry no record of what was subtracted, the protection is gone, and nothing warns you. Chapter 10 specifies what a cache record should contain, and log-probabilities are the answer, for this reason among others.

2.3 logsumexp, and the rule about never taking the log of a softmax#

The denominator of the softmax has a name and a life of its own. The log-partition function, called logsumexp in code, is

Take the logarithm of the softmax formula and you get, with no approximation,

That identity is the definition of log_softmax. A log-probability is a logit minus a single scalar shared by the whole vector.

LSE has the same overflow problem as the softmax, and the same fix. With $m={\max }_j{z}_j$,

$$\mathrm{LSE}(z) = \log \sum_j e^{z_j} = \log \left( e^{m} \sum_j e^{z_j - m} \right) = m + \log \sum_j e^{z_j - m}.$$

The first step pulls $e^m$ out of the sum, valid because it is constant with respect to $j$; the second uses $\log (ab)=\log a+\log b$. The right-hand side never exponentiates anything above zero, and the remaining term $\log {\sum }_j{e}^{z_j-m}$ is bounded between 0 and $\log V$. Same quantity, computed in an order the number format can survive.

Now the rule, which is the single most useful sentence in Lab 00:

Never take the logarithm of a probability you obtained from a softmax. Get log-probabilities directly from log_softmax.

The reason is underflow, which is overflow’s mirror image and much less widely understood.

Here is the argument in full, because it is the load-bearing one.

The fp32 format’s smallest positive value is about $1.4\times 10^{-45}$. (That figure is the smallest subnormal, which Section 2.7 defines; the smallest value with full precision is much larger, at $1.18\times 10^{-38}$.) Anything below the floor is stored as exactly 0.0, with no warning and no exception.

Take logits $z=[0.0,-110.0]$. This is not an exotic input. It is what a trained model looks like at a position where it is confident: one plausible continuation and one it has ruled out, separated by a large margin. The true probability of the second token is

That is below $1.4\times 10^{-45}$ by a factor of about a thousand. Run the composed path, softmax and then log: the softmax stores $p_2$ as 0.0, the logarithm of 0.0 is $-\infty$, and you have a negative infinity sitting in your log-probability tensor.

Run the direct path, log_softmax, and the identity $\log p_i=z_i-\mathrm{LSE}(z)$ is evaluated instead. Here $\mathrm{LSE}(z)=0+\log (1+e^{-110})\approx 0$, so the answer is $-110-0=-110$, a subtraction of two ordinary-sized numbers. The tiny probability is never materialized in any intermediate at any point, and the result is a normal float that behaves correctly downstream.

Same mathematics. Different order of operations. Only one order fits inside the number format.

What happens next is worth memorizing, because it is how you will encounter this. The $-\infty$ flows into a divergence. Forward KL contains a term $p_i(\log p_i-\log q_i)$; if the student’s log-probability is $-\infty$ where the teacher’s probability is positive, that term is $+\infty$ and the loss is inf. Subtract inf from inf downstream and you get nan. One backward pass later every parameter is nan, and every subsequent loss is nan regardless of the data.

On a dashboard that is a curve which behaves and then goes vertical, and the label your brain attaches to it is “the run became unstable.” Everything you know about unstable training tells you to reduce the learning rate, add warmup, clip the gradient norm. None of it helps. The tell is that the failure reproduces at the same step on the same data regardless of the learning rate, which is not how genuine optimizer divergence behaves.

The listing below is the smallest complete demonstration. Watch the second entry of each output.

import torch
import torch.nn.functional as F

z = torch.tensor([0.0, -110.0])          # a token the model has ruled out

composed = F.softmax(z, -1).log()        # materialises exp(-110) ~ 1.7e-48, stores 0.0
direct   = F.log_softmax(z, -1)          # evaluates z_i - logsumexp(z), all ordinary sizes

print(composed)   # tensor([0., -inf])
print(direct)     # tensor([   0., -110.])

assert torch.isinf(composed[1])
assert torch.isfinite(direct[1])

The composed path is wrong by an infinite amount on an input a real model produces at a real position, which is the justification for a rule that otherwise sounds like style preference.

Both of those have a consequence past this one function. If you need $\log (1-x)$ for small $x$, use log1p(-x) rather than composing, for the same class of reason. And if you are handed probabilities rather than log-probabilities, by an API or a cache or a colleague, you have lost information you cannot recover, because every value that underflowed is now indistinguishable from one that was genuinely zero. Serving stacks return log-probabilities for this reason, and Chapter 15 covers the shape of that response.¹⁶

2.4 Temperature#

Temperature is one scalar divided into every logit before the softmax.

The name is borrowed from statistical physics, where the same expression describes the occupancy of energy states at temperature $T$, and the analogy carries: high temperature means the system spends time in states it would otherwise never visit.

What the knob does is pinned down by its two limits, both derivable in a few lines.

As $T\to 0$, the distribution converges to a one-hot vector on the largest logit. Write the softened probability with the max subtracted, which shift invariance permits. Let $m={\max }_j{z}_j$, attained at a unique index $k$. Then

Every exponent is zero or negative. For $i=k$ the numerator is $e^0=1$. For every other $i$, $z_i-m<0$, so $(z_i-m)/T\to -\infty$ as $T\to 0^{+}$ and the numerator goes to zero while the denominator goes to 1. So $p_k\to 1$ and everything else goes to 0. Ties for the maximum spread the limit uniformly over the tied set, by the same argument.

As $T\to \infty$, the distribution converges to uniform. Now every exponent $(z_i - m)/T \to 0$,soeverynumeratorgoesto1andthedenominatorgoesto$V$.Every$p_i \to 1/V$, and the distribution has forgotten the logits.

Between those limits it moves smoothly, and the way it moves is why temperature is in this book at all. Take the seven-token vector Lab 01 uses, with logits $[8.0,5.5,5.0,1.0,0.5,-2.0,-3.0]$ standing for the, a, an, cat, dog, xylophone, qq.

Table 2.1 One logit vector, six temperatures. Probabilities rounded; entropy in nats.

Read the top row. At $T=0.25$ the distribution is one-hot to five decimal places, so a student trained to match it learns which token was correct and nothing else, which is what a hard label would have taught it. Everything distillation is supposed to transfer has been annihilated by the temperature.

Read the bottom row. At $T=10$, xylophone is at 0.0878 against the at 0.2387, a ratio of about one to three. The teacher does not believe xylophone is a third as likely as the; it assigned them logits of $-2.0$ and $8.0$. What the student is being asked to reproduce, with a substantial share of the loss, is the teacher’s noise floor, the region where it has ruled everything out and the remaining differences are close to arbitrary.

The middle rows are where the mechanism operates. Dark knowledge, defined in Chapter 1, is carried by the ratios among the teacher’s rejected tokens. In the $T=1$ row, everything below the top token amounts to 0.118 of the mass, and the informative part of it, the difference between cat at 0.0008 and xylophone at 0.00004, contributes a cross-entropy term four orders of magnitude below the top token’s. The gradient will not notice. Raise the temperature and those ratios survive while their absolute sizes grow: at $T=2$ the same two tokens sit at 0.0192 and 0.0043, large enough to move a loss.

That is the mechanism: temperature does not create the information in the teacher’s tail, it rescales the tail into the range where a gradient-based optimizer can respond to it. Hinton, Vinyals, and Dean introduced this move, and it is the one hyperparameter their method could not do without.³ A factor of $T^2$ has to accompany the softened loss term to keep its gradient scale comparable across settings of $T$, and Chapter 5 derives it.

One clarification, because the same word covers three jobs. The temperature on an inference API applies this formula to change what gets sampled. The temperature in a distillation loss changes what the target looks like, with no sampling involved. And temperature scaling as post-hoc calibration fits one scalar on held-out data so that a model’s confidences match its accuracy, which makes it measured rather than chosen; Chapter 16 uses that version.⁴

The useful range of $T$ depends on how peaked your teacher is on your data. Peaked domains, where the continuation is nearly forced (code, structured output, arithmetic), tolerate low temperatures because there is little tail to expose. Diffuse domains, where many continuations are reasonable, need more. That is a property of your corpus and your teacher, measurable in an afternoon, and not a constant to copy out of someone else’s configuration file.

2.5 Entropy, in nats#

Temperature moves a distribution between “certain” and “uniform.” Entropy says where on that axis a distribution currently sits.

Both extremes are worth confirming. For a one-hot distribution one term is $1\cdot \log 1=0$ and every other is $0\cdot \log 0$, taken to be zero by the limit $x\log x\to 0$. So $H=0$: a distribution that always gives the same answer carries no surprise. For the uniform distribution every $p_i=1/V$, so

and a standard argument by concavity shows no distribution does better. Over a 151,936-token vocabulary that ceiling is $\log V=11.93$ nats.

The teacher’s entropy is one of the two terms in the identity connecting entropy, cross-entropy, and KL, and Chapter 3 lives inside that identity. What I want here is the shape of $H$ as a function of temperature. The limits are established: $H\to 0$ as $T\to 0$ and $H\to \log V$ as $T\to \infty$. In between, $H$ rises monotonically, and the rate is worth knowing at both ends.

At high temperature, expand $e^x\approx 1+x$ for small $x=(z_i-\bar{z})/T$, where $\bar{z}$ is the mean logit. Then $p_i\approx \frac{1}{V}\left(1+\frac{z_i-\bar{z}}{T}\right)$, and carrying the expansion of $-\sum p_i\log p_i$ to second order gives

where ${\sigma }^2=\frac{1}{V}{\sum }_i(z_i-\bar{z}{)}^2$ is the variance of the logits. Entropy approaches its ceiling like $1/T^2$, at a rate set by how spread out the logits were. On Table 2.1’s seven-token vector that variance is 14.48, so the prediction at $T=10$ is $1.9459-14.48/200=1.8735$ nats against an exact 1.8748. It lands that close because $T=10$ is already deep in the asymptotic regime for logits of this size.

At low temperature, let ${\Delta }_i$ be the gap between the largest logit and logit $i$. Each rejected token contributes about $e^{-{\Delta }_i/T}(1+{\Delta }_i/T)$ to the entropy, so

which goes to zero exponentially fast in $1/T$. Entropy does not decline gently as you cool a distribution; it falls off a cliff. On Table 2.1, dropping $T$ from 1.0 to 0.5 cuts entropy by a factor of eight, and dropping to 0.25 cuts it by another factor of ninety.

Entropy earns its place here for a reason beyond Chapter 3’s identity: it is the cheapest useful thing you can monitor during a run that is not the loss. The loss tells you the objective is going down, which it does whether or not anything good is happening. Mean entropy tells you whether the student is becoming more decisive, and how fast.

A healthy student’s entropy declines somewhat and then plateaus, because training on a teacher’s targets concentrates its mass. A student in trouble keeps going, and the failure has a name. Entropy collapse is the runaway version, where the distributions narrow until the output is effectively deterministic and the text loses all variety. It is well documented in reinforcement-learning post-training of reasoning models, characterized there as an exchange between entropy and short-term performance, and the same mechanism appears in on-policy distillation, where the entropy worth watching is measured on the student’s own rollouts.⁵⁶¹² Chapter 12 covers the monitors that catch it and the distinction that makes one useful: a healthy decline of ten to thirty percent is not collapse, and a monitor that fires on that is worthless. The course’s own pairs a floor of 0.15 nats with a rule about entropy lost inside a trailing window.

Entropy also has a diagnostic use that costs nothing. If your student’s mean entropy is larger than your teacher’s on the same inputs, the student is hedging, which is the signature of a mode-covering objective (Chapter 6). If it is much smaller, the student has committed, which is the signature of a mode-seeking one. That one number, read before any sample text, usually tells you which situation you are in.

2.6 The tail#

A next-token distribution has a head and a tail, and much of distillation depends on which one you are paying attention to.

The head is easy to describe. At most positions in ordinary text a trained model is confident: the top token takes most of the mass, a handful of alternatives take most of what is left, and tens of thousands of remaining entries share a remainder that rounds to nothing. On Table 2.1 at $T=1$, the top token holds 0.8823 and the other six share 0.1177.

The tail is the rest of the vocabulary, and it has two properties that pull in opposite directions.

The tail holds almost no probability mass. The course measures this on real data: Lab 02 runs the teacher over the corpus Lab 03 trains on and computes how much of each position’s mass the top $k$ entries capture. At $k=64$, mean retained mass exceeds 0.99. Ninety-nine percent of the teacher’s belief lives in 64 out of 49,152 entries, 0.13 percent of the vocabulary. Chapter 10 turns that measurement into a storage decision.

The tail holds most of the entries, and all of the information about what the model considers absurd. Dark knowledge is a statement about ratios among rejected tokens, and rejected tokens are by definition in the tail. A student that matches the head perfectly and ignores the tail has copied the teacher’s decision and learned nothing about its similarity structure.

Both statements hold at once, which is why the tail is the interesting part of the object and the hardest to work with. A quantity can be numerically negligible and informationally significant at the same time, and temperature is the device that converts the second property into something the first does not suppress.

The tail’s third property is the one this chapter cares about most. The tail is where the underflow lives: a probability of $10^{-30}$ is a fine fp32 number and a hard zero in fp16, and $10^{-48}$ is a hard zero in both. Every claim about the tail is therefore also a claim about your number format, and a pipeline that computes the head correctly can be silently wrong about the tail on the same batch.

One more thing the tail does will surprise you if you have not seen it, and Chapter 3 makes it precise. A tiny amount of mass in the wrong place can produce an unbounded KL divergence, because the divergence weights each token by the logarithm of a probability ratio and a logarithm has no ceiling. Lab 00’s second solution exercise builds distribution pairs whose total variation distance goes to zero while their KL goes to infinity, using nothing but a shrinking mass on a token where the other distribution is astronomically small. So “the tail holds one percent of the mass” does not license “the tail contributes one percent of the loss.” It usually does, it can fail arbitrarily badly, and the case where it fails is the one where the student puts near zero probability on something the teacher likes, which is what a freshly initialized student does everywhere.

Carry three things out of this section and into Chapter 3. The tail exists, its mass is measurable on your own corpus with one prefill pass, and you should measure it. When the only thing you have is the token that was sampled, the tail becomes something you estimate rather than compute, and Chapter 4 is about how badly that can go.¹³ And the split between the top-1 token and everything else is the split between what a hard label tells you and what distillation is for. Work on generation quality has circled the same object from the sampling side for years: nucleus sampling exists because the shape of the tail decides whether sampled text degenerates.⁷

2.7 Floating point, in enough detail to predict failures#

Everything above assumed the numbers behave. This section says what they actually do.

2.7.1 The anatomy of a float#

A floating-point number is stored as three bit fields: a sign bit $s$, an exponent field $e$, and a mantissa field $f$ with $m$ bits. In the normal range, the value is

The factor in parentheses is the significand. It lies in $[1,2)$, and the leading 1 is not stored because it is always there, which buys one bit of precision for free. That implicit bit is the source of the documentation conflict this section resolves later.

That layout has two consequences, and between them they explain every property of every format.

The exponent field buys range. With $b$ exponent bits the exponent covers roughly $2^b$ powers of two, so each additional exponent bit squares the ratio between the largest and smallest representable magnitude. Range is cheap.

The mantissa field buys precision, and it buys the same relative precision everywhere. With $m$ stored mantissa bits the significand is quantized into $2^m$ steps across $[1,2)$, so within any binade (any interval $[2^k,2^{k+1})$) the representable values are spaced $2^{k-m}$ apart. Spacing scales with magnitude: near 1.0 the gap is $2^{-m}$, near 1000 it is a thousand times larger, near $10^{-6}$ a million times smaller. A float has constant relative precision and wildly varying absolute precision, and most floating-point surprises reduce to expecting the opposite.

That constant has a name.

Machine epsilon is measurable without knowing anything about the bit layout, which makes it the honest way to settle an argument about a format’s precision. If it comes out to $7.8\times 10^{-3}$, then $m=7$, whatever the documentation says.

2.7.2 The three formats you will actually use#

Table 2.2 The formats, with everything that follows from the bit split.

Read the table as one decision made twice with opposite answers. Both 16-bit formats have 16 bits to spend. fp16 puts 10 into mantissa and 5 into exponent. bf16 puts 7 into mantissa and 8 into exponent, which is exactly fp32’s exponent field.

The consequences follow from the table. fp16 overflows at 65,504, which Section 2.2 showed is exp of a logit above 11, and it flushes any probability below about $6\times 10^{-8}$ to exact zero. Both thresholds sit inside a language model’s operating range, which is why fp16 needs loss scaling to train at all and why it produces exact zeros where a distribution should have small positive numbers. bf16 cannot overflow on anything fp32 could hold and cannot underflow before fp32 does. What it does instead is round: every bf16 result is good to about three decimal digits and noise after that.

The failure signatures differ accordingly. fp16 fails loudly, with inf, nan, and a dead run. bf16 fails quietly: the curve is smooth, the numbers are plausible, and the third digit is garbage. On a dashboard that looks like a plateau, or like a run that gives slightly different numbers each time you repeat it with the same seed. You conclude your intervention did not work, when your instrument could not resolve the effect you were measuring.

2.7.3 Subnormals and flush-to-zero#

Section 2.7.1’s formula has a lower limit: once the exponent field hits its minimum you cannot go smaller by decrementing it. IEEE 754 handles that with a second regime.

The gap between the two regimes is larger than people expect. fp32’s smallest normal value is $1.18\times 10^{-38}$ and its smallest subnormal is $1.4\times 10^{-45}$, seven orders of magnitude further down, obtained by giving up the 23 mantissa bits one at a time. Below $1.4\times 10^{-45}$ there is nothing but zero.

Operationally, subnormals matter in two places. First, they are why Section 2.3’s underflow threshold is $1.4\times 10^{-45}$ rather than $1.18\times 10^{-38}$, seven orders of magnitude in how far a probability can fall before becoming a hard zero. Second, they are not always enabled. Many accelerators and some compiler settings implement flush-to-zero, replacing any subnormal result with zero because subnormal arithmetic is expensive in hardware. The effective floor then rises to the smallest normal, and probabilities between $10^{-45}$ and $10^{-38}$ that survived on one device become exact zeros on another. That is one mechanism by which a loss that was finite on your laptop becomes inf on your accelerator with no change to the code. The defense is the one this chapter keeps giving: stay in log space.

2.7.4 The 7-versus-8 question, resolved#

The course’s own materials disagree with themselves here, and rather than quietly fixing one side I want to state the resolution, because this is the kind of disagreement that looks like a contradiction and is not.

The habit worth building is to say which count you mean. “bf16 has 8 mantissa bits” will make a reader compute the wrong machine epsilon by a factor of two, and a factor of two in a precision budget is the difference between a clamp that works and a clamp that does nothing.

2.7.5 The dtype policy: model in bf16, loss in fp32#

The rule the course follows, and the one Hugging Face and TRL implement internally, is one line:

Run the model in bf16. Compute the loss in fp32.

The first half is forced by memory. A parameter in bf16 costs 2 bytes instead of fp32’s 4, so a 32-billion-parameter teacher occupies 64 GB rather than 128 GB, which is the difference between fitting on the reference machine alongside a student and not fitting at all. Chapter 8 does that arithmetic in full.

The second half is forced by precision, and the argument is the digit budget. bf16’s machine epsilon of $7.8\times 10^{-3}$ means every stored value carries a relative error of up to about 0.4 percent, and a loss accumulates over tens of thousands of terms. Lab 00 measures the effect: a KL computed through fp32 log_softmax has relative error under $10^{-5}$ against an fp64 reference, while the same KL through bf16 log_softmax is more than ten times worse, and the lab asserts both bounds.

Now ask what you were going to do with that loss. An ablation compares arms whose difference late in training is a few percent of a loss value, and a run’s improvement over its own previous checkpoint is smaller still. If the third digit is noise you cannot resolve either. The policy is not about correctness of the forward pass, which bf16 handles fine. It is about whether your loss is a usable instrument.

The policy costs one fp32 copy of the logit tensor at the loss site, which at batch 8, sequence 512, and vocabulary 151,936 is 2.5 GB. That cost is why people skip the upcast. When the copy does not fit, the move is not to compute the loss in bf16; it is to reduce the batch or chunk the vocabulary dimension, both of which preserve the answer.

One more consequence, because it returns in Chapter 10. If you cache teacher outputs in bf16, which you should for storage reasons, you are storing three good digits. Three digits is plenty for log-probabilities, which are ordinary-sized negative numbers, and nothing at all for probabilities near 1. The format is identical; what changed is where on the number line you put the quantity.

2.7.6 A prediction that failed#

Lab 00’s first exercise asks the reader to break bf16 on purpose: sweep the logit scale and find where bf16’s loss arithmetic first disagrees with fp32’s in the second significant digit. The premise is plausible, since bf16 keeps two to three significant digits and larger logits produce larger absolute rounding errors.

The solution notebook ran the sweep at scales 1, 2, 4, 8, 12, 20, and 40 against an fp64 reference, and the premise did not survive. bf16’s relative error stayed between about $10^{-4}$ and $2\times 10^{-3}$ throughout, peaking at scale 12 and getting better at scale 40. The third digit went bad at scales 2 and 12, with no monotone trend between. The second digit was never corrupted at any scale tested, and the notebook asserts it: maximum bf16 relative error below $10^{-2}$.

The reason is a cancellation the premise did not account for. Scaling the logits up grows the absolute rounding error, because float spacing is proportional to magnitude. It also grows the KL itself, from about 1 nat at scale 1 to about 115 nats at scale 40. Relative error is the ratio of those two growing quantities, and they grow at comparable rates, so it wanders inside the digit budget instead of climbing out. Averaging over the test problem’s 64 rows claws back a little more, which is why the measured error sits below machine epsilon rather than at it.

2.8 A clamp that did nothing#

This is a real bug from the course’s own history, and the cleanest illustration in the book of why the previous section’s arithmetic is worth carrying around.

Chapter 10 covers top-k logit caching properly. The piece needed here: when you keep only the teacher’s top $k$ log-probabilities, you have to decide what to do about the missing mass, and one standard answer is to keep the total discarded mass as a single extra bucket. The loss then gains a term comparing the teacher’s tail mass against the student’s, where the student’s tail mass is one minus its mass on the retained tokens:

That needs care in floating point, because the sum can be very close to 1 and can exceed it by a rounding error. The obvious defense is to clamp the sum below 1 before subtracting, with a bound like $1-10^{-6}$, and to use log1p rather than composing a subtraction with a logarithm. That is what the implementation did.

The listing below is the whole bug in six lines. Look at the comparison in the last two prints.

import torch

CLAMP = 1.0 - 1e-6                       # intended as "strictly below 1"
student_mass = torch.tensor([0.999])     # student's probability on the teacher's top-k

for dtype in (torch.float32, torch.bfloat16):
    s = student_mass.to(dtype)
    bound = torch.tensor([CLAMP], dtype=dtype)
    guarded = torch.minimum(s, bound)
    print(dtype, float(guarded), float(torch.log1p(-guarded.float())))

# torch.float32   0.99900001   -6.90777
# torch.bfloat16  1.0               -inf

The same expression, with the same guard, on the same input, is finite in one dtype and infinite in the other. Nothing in the source distinguishes the two cases, which is why this class of bug survives review.

2.9 What all of this costs#

A short accounting, in the spirit the rest of the book keeps.

Max-subtraction costs one reduction over the vocabulary dimension and is already inside your framework. Log space costs nothing either, since log_softmax and softmax do the same work; its price is conceptual, because the identities you know for probabilities (they sum to one, they multiply) become different ones (they logsumexp to zero, they add). The fp32 loss policy costs one copy of the logit tensor, 2.5 GB at [8, 512, 151936], which is one more reason Chapter 10’s top-k pipeline is attractive: it shrinks the tensor that has to be upcast.

What the policy gives up is the last few digits. fp32 relative error here lands near $10^{-6}$, fine for a training loss and not fine for a claim about a difference of $10^{-7}$ between two arms. If you make that kind of claim, compute the measurement in fp64 and say so, which is what the labs do when they need a ground-truth reference.

2.10 What to be suspicious of#

The presentation above makes three things sound more settled than they are.

The set of 16-bit formats is not stable, and neither format was designed for this. fp16 came from graphics and bf16 from accelerator design, and both were compromises that happened to be available when large-model training needed them. Formats below 16 bits are already in serious use, including 8-bit floating-point variants and 4-bit schemes that are not IEEE floats at all: QLoRA’s NF4 assumes the stored values are approximately normally distributed rather than spread over a uniform exponent grid, and post-training quantization methods choose for themselves which values deserve precision.⁸⁹¹⁰ Chapter 15 covers what those cost in quality. Section 2.7’s analysis applies to them in structure, and none of its numbers do.

“Compute the loss in fp32” rests on a threshold nobody has characterized properly. I have given you the digit-budget argument and one measurement. The argument is sound, and nobody has systematically mapped which distillation objectives at which scales degrade in bf16 and by how much. The rule is cheap enough that following it without that map is the right call, and you should know you are following an argument rather than evidence.

The relationship between measured entropy and anything you care about is loose. Entropy is cheap to compute and it is the standard early-warning signal for collapse. A run can also have healthy entropy and produce bad text, or declining entropy and produce good text. The work on when diversity actually collapses during post-training is active and does not agree with itself yet.¹¹

2.11 Where this lands in the labs#

Lab 00 is where every claim in this chapter gets asserted rather than argued, on no GPU and with no training: the naive softmax on temperature-scaled logits, asserted nan; both paths to a log-probability, asserted -inf on one and $-110.0$ on the other; a KL computed in fp64, fp32, bf16, and fp16 with the error ordering asserted; fp16 turning a probability of $2\times 10^{-16}$ into an exact zero. What the lab does that this chapter cannot is let you change one number and watch the failure move. Set the temperature to 0.3 and find where the overflow starts, or change the $-110$ to $-100$ and watch the composed path start working. Lab 00’s first solution exercise is the bf16 sweep whose premise failed in Section 2.7.6, and the fifteen minutes you spend expecting the second digit to break are the point of it. Lab 01 §1 is Table 2.1, and it runs in under a second.

2.12 Exercises#

1. A checkpoint’s largest logit at some position is 34.2, and you are about to compute a softmax at temperature 0.2 in fp16 without max-subtraction. Write the arithmetic out, say whether it overflows, and give the smallest temperature at which it would not. Repeat for fp32. Then say what happens to both answers under max-subtraction, and prove it from shift invariance.

2. Using $H(T)\approx \log V-{\sigma }^2/(2T^2)$, predict the entropy of Table 2.1’s logit vector at $T=5$ before looking at Figure 2.3. State the condition under which the two-term expansion should be trusted, in terms of $\sigma$ and $T$, and say whether $T=2$ satisfies it.

3. A colleague reports a run whose loss is stable at 2.417 for a hundred steps, reads inf at step 101, and is nan afterward, and that cutting the learning rate by 10x reproduces the failure at step 101 exactly. Give the most likely cause, name the line of code you would read first, and say what single measurement would confirm or refute your hypothesis before you change anything.

4. Section 2.8’s clamp bound of $1-10^{-6}$ fails in bf16. Someone proposes changing the bound to $1-10^{-2}$, which bf16 can represent. Say what that does to the tail term when the student’s true tail mass is $10^{-4}$, whether the resulting loss is biased upward or downward, and which of the two fixes you would ship.

5. You inherit a logit cache whose README says it stores probabilities in fp16, top 32 entries per position. List what you can no longer determine from it that a cache of fp32 log-probabilities would have given you, and say for each item whether the loss is recoverable.

6. Argue both sides of this: “since the top 64 tokens hold more than 99 percent of the teacher’s probability mass, discarding the rest costs at most one percent of the training signal.” Use Section 2.6 for the case against, and be specific about the quantity that makes the claim fail. Then say what you would measure on your own corpus to settle it there.

At one position in one sequence, the teacher has a probability distribution over its vocabulary and the student has another one. Two vectors of 151,936 numbers each, both summing to one. Your job is to collapse the difference between them into a single scalar, because a scalar is what an optimizer can descend.

There are a lot of ways to do that, and the choice is not a formality. Two students trained on the same corpus, from the same initialization, against the same teacher, for the same number of steps, with the same learning rate, will produce visibly different text depending on which scalar you chose. One will hedge. One will commit. One will occasionally produce something the teacher would never have said. Chapter 6 runs that comparison as a controlled experiment and reports what happens. This chapter builds the objects the comparison is made of.

I want to be careful about the order here. Almost everything written about divergence choice in distillation starts from behavior: forward KL is mode covering, reverse KL is mode seeking, use reverse KL if you want crisp output. Those statements are true and I will get to them. But they are consequences of arithmetic, and if you learn the consequences without the arithmetic you will not be able to predict what a new objective does when someone publishes one next month. So this chapter stays with the arithmetic. Every behavioral claim I make here is derived from the shape of a function, and every one of them is checkable by hand on a four-token vocabulary.

Throughout, and throughout the rest of the book, $p$ is the teacher and $q$ is the student. That convention comes from the labs and it is worth fixing now, because half the confusion in this subject comes from people writing $\mathrm{KL}(p\Vert q)$ and not saying which letter is which model.

3.1 One identity, and everything that follows from it#

Chapter 2 defined entropy. Here are the other two quantities in the same family, all three measured in the same unit.

Every divergence below is in nats, as Chapter 2 defined them, because that is what log returns in every framework you will use and because it is the unit the entropy in the identity is already in.

For two distributions $p$ and $q$ over the same finite set of outcomes indexed by $i$:

The three are related by one identity, and it is the single most useful line in this chapter:

Read it as an accounting statement. Cross-entropy is the total bill. Entropy is the part of the bill you were always going to pay, because $p$ is genuinely uncertain and no model can remove uncertainty that lives in the target. KL is the remainder, the part that is your model’s fault. Drive KL to zero and you have paid exactly the irreducible price and not one nat more.

Now the consequence that makes the identity load-bearing for distillation. During training the teacher is fixed. Its parameters do not move, so $H(p)$ is a constant with respect to everything the optimizer touches. Which means:

Three corollaries fall out of that identity immediately, and I want all three on the table before the next section.

Hard-label training is distillation with a one-hot teacher. A one-hot distribution puts probability 1 on a single outcome. Its entropy is $-1\cdot \log 1=0$, because there is nothing uncertain about it. So the identity collapses to $H(p,q)=\mathrm{KL}(p\Vert q)$, and the ordinary next-token cross-entropy loss you have been using since your first language model is forward KL against a degenerate teacher. Distillation with a soft teacher is not a different objective. It is the same objective with a better reference distribution, and every piece of machinery in this chapter applies to both.¹ What makes a reference distribution better is itself a variable rather than a given: a teacher whose training procedure flattened the structure among its wrong-answer probabilities transmits less through this channel than one that did not, which is why label smoothing in the teacher measurably degrades the student it produces.² Chapter 5 reproduces that argument in full.

Perplexity is a KL statement wearing different clothes. Per-token perplexity is $e^{H(p,q)}$, read as “the model is as uncertain as if it were choosing uniformly among this many options.” Substituting the identity gives $\mathrm{PPL}=e^{H(p)}\cdot e^{\mathrm{KL}(p\Vert q)}$. When someone reports that a distilled student has 1.4x the perplexity of its teacher on a corpus, they have reported $e^{\mathrm{KL}}$ averaged in a particular way, and the KL in question is about $\log 1.4\approx 0.34$ nats. The perplexity ratio is the interpretable quantity; the absolute perplexity mixes in $H(p)$, which is a property of the data and not of your model.

KL is never negative. This is Gibbs’ inequality, and it follows from concavity of the logarithm:

3.2 The two conventions that make the sum well defined#

The formula ${\sum }_i{p}_i\log (p_i/q_i)$ has two edge cases, and both of them show up in real training runs rather than in textbook footnotes.

Where $p_i=0$, the term is 0. The convention is $0\log 0=0$, which is not arbitrary: the limit ${\lim }_{t\to 0^{+}}t\log t=0$, so defining the term this way makes $t\log t$ continuous at the origin. Operationally it means forward KL charges the student nothing at all for its behavior on tokens the teacher has ruled out. The student can put whatever it likes there, and the forward-KL loss will not notice. Hold onto that; it is half of §3.4.

Where $p_i>0$ and $q_i=0$, the term is $+\infty$. There is no convention that rescues this one. The ratio $p_i/q_i$ diverges, the logarithm of it diverges, and the whole sum is infinite. The extreme case has a name.

I used to think of the infinity as a mathematical corner case, a thing that happens on the whiteboard and never in float32. It happens constantly. Two mechanisms produce it. The first is cold start: a freshly initialized student, or a student that has been pruned down from its teacher and not yet recovered, will assign genuinely negligible probability to tokens the teacher is confident about. The second is the number format. Chapter 2 established that fp16 stores nothing smaller than about 6e-8, so in fp16 a probability of 2e-16 is exactly zero rather than a small number. A distribution that was perfectly healthy in fp32 arrives at the loss site in fp16 with holes in it, and the KL that was 14 nats becomes inf, and the gradient becomes NaN, and the run dies. That is the most common way a distillation run fails on the first day.

Lab 00 makes the infinity concrete by clamping the zeros rather than pretending they are not there. On the four-token pair $p=[0.5,0.5,0,0]$ and $q=[0,0,0.5,0.5]$ with zeros replaced by $\epsilon =10^{-12}$, the KL comes out at $\log (0.5/10^{-12})=\log (5\times 10^{11})=26.94$ nats. Halve $\epsilon$ and the answer grows by $\log 2$. There is no value it converges to. The clamp chooses how large you would like your infinity to be, and its size is set by a constant nobody in the room chose deliberately.

3.3 KL is not a distance, and both failures matter#

A distance in the mathematical sense is a metric, and a metric has to satisfy four conditions: non-negativity, zero exactly on identical arguments, symmetry, and the triangle inequality.

KL satisfies the first two and fails the last two. Both failures have operational consequences and both are demonstrable on distributions you can hold in your head.

3.3.1 Asymmetry, with numbers#

Take a four-token vocabulary. Let $p$ be the softmax of the logits $[8,0,0,0]$, which is a teacher that is confident about token 0, and let $q$ be the softmax of $[0,0,0,0]$, which is a student that has no opinion at all. Writing them out:

3.3.2 The triangle inequality, with a triple you can verify in your head#

Asymmetry is the failure everyone knows about. The triangle failure gets less attention and it matters more for how you plan a project, because it invalidates a chain of reasoning that multi-stage pipelines invite you to make.

Here is the cleanest counterexample I know. Take a two-token vocabulary and three distributions:

The factor of exactly two is not a coincidence and the derivation is short enough to do here. Let $p=[x,1-x]$, $q=[\frac{1}{2},\frac{1}{2}]$, and $r=[1-x,x]$ for any $x\in (0,1)$. Then

The consequence is a piece of reasoning you cannot use. Suppose you distill a 70B teacher into a 14B intermediate model, then distill the intermediate into a 1.7B student. Both measured KLs come out small, and you would like to conclude that $\mathrm{KL}(\text{teacher}\Vert \text{student})$ is small. You cannot. Nothing supports that step, and the family above shows the error can be a clean factor of two in the friendliest case. If you want the conclusion you have to measure it directly against the teacher, which costs one forward pass of the teacher over your evaluation set and is worth paying for. Teacher-assistant setups that insert a mid-sized model between an enormous teacher and a small student are a standard recipe, and the prune-then-distill pipeline of Chapter 13 has the same shape.

3.4 Forward and reverse KL, stated as facts about the integrand#

Since KL is asymmetric, there are two of it, and both are used.

Now the mechanical statement, which I want on the page before any word about training behavior.

Forward KL sums $p_i\log (p_i/q_i)$ over $i$. Look at what each term does at the boundaries.

Where the teacher has mass and the student does not, $p_i>0$ and $q_i\to 0$, the term is $p_i\log (p_i/q_i)\to +\infty$. The student is charged without limit.

Where the student has mass and the teacher does not, $p_i=0$ and $q_i>0$, the term is $0\cdot \log 0=0$ by the convention in §3.2. The student is charged nothing.

Reverse KL sums $q_i\log (q_i/p_i)$, and the same two checks come out the other way around.

Where the teacher has mass and the student does not, $q_i\to 0$, the term is $q_i\log (q_i/p_i)\to 0$, because $t\log t\to 0$. Charged nothing.

Where the student has mass and the teacher does not, $p_i\to 0$ with $q_i>0$ fixed, the term is $q_i\log (q_i/p_i)\to +\infty$. Charged without limit.

That is the entire asymmetry, stated as two sentences with no metaphor in them:

Forward KL penalizes the student for putting near-zero mass where the teacher has mass. Reverse KL penalizes the student for putting mass where the teacher has none.

Everything anyone has ever written about mode covering and mode seeking is a restatement of those two sentences in the presence of a student too small to satisfy both constraints. If the student can represent the teacher exactly, both directions are minimized at the same place, namely $q=p$, and the choice does not matter. The choice matters exactly when the student cannot, which is the situation you are always in when distilling.