A.1 One map, thirteen numbers#

Everything in this course is stated in terms of one concrete model, GPT-2 small, and everything in this refresher is in service of one skill: being able to say, at any point inside that model, exactly what tensor exists there, what its shape is, and which learned matrix produced it. Interpretability work is tensor bookkeeping before it is anything else. When a later chapter says "patch the residual stream at layer 7, position 4", you must already know that this names a vector of 768 numbers and know precisely which computations wrote it.

The table below fixes the dimensions of GPT-2 small. Commit it to memory; the course uses these symbols constantly.

Note the ratios: d_head × n_heads = d_model, and d_mlp = 4 × d_model. Neither is forced by the mathematics, but both are conventions GPT-2 follows and both matter for shape bookkeeping. A forward pass on a prompt of n_tokens tokens carries a tensor of shape (batch, n_tokens, d_model) through the network; for a single prompt the batch dimension is 1 and we often suppress it.

A.2 Tokens and embeddings#

Text enters the model as integers. GPT-2's tokenizer uses byte pair encoding (BPE): starting from raw bytes, it repeatedly merges the most frequent adjacent pair into a new vocabulary entry, ending with 50257 tokens that include whole common words (usually with a leading space, such as " Paris"), word fragments, and single bytes as a fallback. Tokenization is deterministic and lossless, but its boundaries are an artifact of training-corpus statistics, not linguistics. This matters for interpretability: the model's atomic units are BPE tokens, so a circuit that appears to process "words" is really processing token sequences, and rare words split into several tokens the model must reassemble internally.

The embedding step is a lookup. The embedding matrix W_E has shape (d_vocab, d_model) = (50257, 768); token ID t selects row W_E[t]. In parallel, the learned positional embedding W_pos, of shape (n_ctx, d_model) = (1024, 768), supplies one vector per position. The initial hidden state at position p holding token t is

Two observations set up the rest of the course. First, token identity and position enter by addition into the same 768-dimensional space; from here on the model cannot cleanly separate them except by whatever directions it has learned to use for each. Second, this sum x₀ is the first state of the running sum that every subsequent component will add to. Chapter 1 names that running sum the residual stream and makes it the central object of the course; in this refresher we simply keep track of it as "the hidden state, which only ever gets added to".

A.3 LayerNorm before every block#

GPT-2 is a pre-LN transformer: layer normalization is applied to the input of each attention block and each MLP block, and once more before the unembedding, rather than after each block as in the original architecture of Vaswani et al. Pre-LN placement is why the hidden state is a clean running sum: normalization happens on a copy that feeds the block, while the addition back into the stream is untouched.

LayerNorm acts independently at each position. Given a vector x ∈ R⁷⁶⁸, it computes the mean μ and standard deviation σ over the 768 coordinates and returns

with learned elementwise scale γ and bias β. The centering and the affine parameters are linear or affine maps and can be folded into the adjacent weight matrices: multiply the block's input weights by γ and absorb β into the block's biases. TransformerLens does this by default when loading a model (fold_ln=True), after which each LayerNorm reduces to the purely normalizing core, center and divide by σ. That leftover division is the one operation in the whole architecture that is not linear in its input, a fact with consequences that Chapter 1 and Chapter 2 treat carefully. For this refresher, remember the operational form you will implement in the lab: subtract the mean, divide by the root mean square of the centered coordinates (plus a small ε = 10⁻⁵ for numerical safety), and pass the result to the block.

A.4 Attention, one head at a time#

Each of the 12 blocks contains an attention layer of 12 heads. Treat the heads as 12 independent units that operate in parallel and whose outputs are added together; this per-head view, rather than the fused matrix view common in implementations, is the interpretability convention, because individual heads turn out to have individual functions.

Head h of layer l owns four matrices: W_Q, W_K, W_V, each of shape (d_model, d_head) = (768, 64), and W_O of shape (d_head, d_model) = (64, 768), plus bias vectors. Let X ∈ R^{n_tokens × 768} be the LayerNormed block input. The head computes queries, keys, and values as

Attention scores compare every query position i against every key position j:

The division by √d_head = 8 keeps score magnitudes stable: if query and key coordinates are order-one, a dot product over 64 coordinates has standard deviation on the order of √64, and without the rescaling the subsequent softmax would saturate. The causal mask then sets S[i, j] = −∞ for all j > i, so that no position can read from its future; this is what makes the model autoregressive and valid as a next-token predictor. A row-wise softmax converts scores to the attention pattern A, where each row is a probability distribution over source positions:

Z[i] is a weighted average of value vectors, weighted by where position i attends. Finally the head projects back to model width and its output is added to the hidden state:

Implementations usually concatenate the 12 per-head Z tensors into one (n_tokens, 768) tensor and multiply by a single fused (768, 768) output matrix. The two views are identical: the fused product decomposes exactly into the sum of the 12 per-head Z_h W_O_h terms. The per-head form is the one to hold in your head, because it exposes the structure interpretability exploits. A head factors into two coupled but separable computations: the pattern A, built from W_Q and W_K, decides where information moves; the value pathway, built from W_V and W_O, decides what information moves. Chapter 3 develops this factorization into QK and OV circuits. Note also the rank constraint: everything a head moves passes through a 64-dimensional bottleneck, so each head is a low-rank operator on the 768-dimensional state.

A.5 The MLP block#

After attention, the block applies its MLP to the LayerNormed intermediate state. The MLP acts on each position independently; it cannot move information between positions. It expands to d_mlp = 3072 dimensions, applies a nonlinearity, and contracts back:

with W_in of shape (768, 3072) and W_out of shape (3072, 768). GPT-2 uses the Gaussian error linear unit (GELU), a smooth relative of the rectified linear unit; in practice it uses the tanh approximation gelu_new, and your manual implementation must use the same variant or the outputs will diverge by more than rounding error. The 3072 post-GELU activations are the "neurons" of the layer, the only elementwise nonlinear coordinates in the block, and the natural first place to look for interpretable units. Much of the later course (superposition in Chapter 10, key-value memories in Chapter 9) is about what these neurons do and why individual neurons usually resist interpretation. The MLP output is added to the state exactly as attention outputs are:

A useful contrast to carry forward: attention moves information between positions but applies only a low-rank linear map to what it moves, while the MLP computes nonlinearly but only within a position. Circuits, when we find them in later chapters, are typically alternations of the two.

A.6 Unembedding and weight tying#

After block 11, a final LayerNorm is applied and the result is mapped to logits by the unembedding matrix W_U of shape (d_model, d_vocab) = (768, 50257), plus a bias b_U:

The logit at position i for token t is the model's unnormalized score for t being the next token after position i; a softmax over the vocabulary axis turns logits into next-token probabilities. Because softmax is invariant to adding a constant to all logits, only logit differences carry meaning, a point Chapter 2 turns into the standard attribution metric.

GPT-2 ties its weights: W_U is the transpose of W_E, one (50257, 768) matrix serving as both the input lookup and the output readout. Tying saves 38 million parameters and imposes a soft symmetry between how a token is written into the state and how it is read out. Interpretability leans on this readout constantly: any 768-dimensional vector, from any point in the network, can be multiplied by W_U and inspected as a distribution of token scores. Note one library detail now, because it will surface in every lab: TransformerLens centers W_U when loading (softmax invariance makes this harmless) and stores a nonzero b_U, so any exercise that reconstructs logits from components must include b_U to match the model.

A.7 The whole pass, and the view the course takes#

Assemble the pieces. Token IDs of shape (n_tokens,) become x₀ of shape (n_tokens, 768) via W_E and W_pos. Twelve identical blocks follow. Block l computes LN(x_l), runs 12 attention heads on it, and adds their outputs to the state; then computes LN of that intermediate state, runs the MLP, and adds its output. A final LayerNorm and W_U produce logits of shape (n_tokens, 50257). Counting components: 1 embedding step, 144 attention heads, 12 MLPs, 1 unembedding.

The interpretability way of holding this in mind is not as a pipeline of 12 transformations but as 157 components that all read from and write into one shared, additive hidden state. No component ever overwrites the state; each adds a vector to it. The final state is exactly the embedding plus the sum of 144 head outputs plus the sum of 12 MLP outputs (plus biases). Every technique in this course, direct logit attribution, activation patching, circuit discovery, sparse autoencoders, is a way of asking which of those additive contributions carries a behavior and how the components communicate through the shared state. Chapter 1 gives that shared state its proper name, the residual stream, and develops its geometry. Your job in this refresher is narrower and foundational: know the shapes, know the matrices, and be able to rebuild one block by hand, which is exactly what the lab has you do.

B.1 Vectors as directions in activation space#

Every intermediate quantity this course manipulates is a vector or a stack of vectors. The residual stream at one token position in GPT-2 small is a vector in R⁷⁶⁸; a full prompt's stream is a (position, 768) matrix; a batch of prompts is a (batch, position, 768) tensor. Interpretability's working ontology is that meaning in these spaces is carried by directions, not by individual coordinates. The claim "the model represents sentiment linearly" means there is some unit vector d such that the dot product of an activation with d tracks sentiment, whatever coordinates d happens to have.

Two habits follow. First, separate magnitude from direction. Any nonzero vector v factors as v = ‖v‖ · v̂, where ‖v‖ = √(Σ_i v_i²) is the Euclidean norm and v̂ is the unit vector v / ‖v‖. When we ask "is the model writing the same thing, only more strongly, at layer 8 as at layer 4", we are asking whether direction is stable while magnitude grows, and the two questions need different measurements. Second, be suspicious of intuitions imported from two or three dimensions. In R⁷⁶⁸, two vectors drawn at random are nearly orthogonal with overwhelming probability, so a cosine similarity of 0.5 between two activations is not "halfway related"; it is a strong signal of shared structure. High dimensional spaces have room for enormous numbers of nearly orthogonal directions, which is the geometric fact behind superposition in Chapter 10.

B.2 Dot products and cosine similarity as alignment#

The dot product u · v = Σ_i u_iv_i is the fundamental measurement. Geometrically, u · v = ‖u‖ ‖v‖ cos θ, where θ is the angle between the vectors. It mixes three things: the length of u, the length of v, and their alignment. Cosine similarity strips the lengths away:

and lands in [-1, 1]: 1 for parallel, 0 for orthogonal, -1 for opposed. Use the raw dot product when magnitude matters, for instance when a readout is literally a linear map applied to the stream, so a component writing twice as hard contributes twice the logits. Use cosine similarity when you want to compare directions across contexts where magnitudes differ for irrelevant reasons, for instance across layers of GPT-2, where the stream norm grows roughly tenfold with depth.

Every linear readout in this course is a dot product in disguise. Projecting the residual stream through one unembedding column to get one logit is a dot product with that column. A linear probe is a learned direction whose dot product with activations predicts a label. A steering vector works because adding it changes downstream dot products. When Chapter 2 computes direct logit attribution, the operation is: take each component's additive write, dot it with a readout direction, report the scalar. If dot products are second nature, most formulas in this course reduce to bookkeeping.

B.3 Orthogonal projection: onto a direction, onto a subspace#

Projection answers "how much of v points along d, and what is left over". For a unit direction d̂, the orthogonal projection of v onto d̂ is

a vector of length |v · d̂| along d̂. The residual r = v - proj_d̂(v) is orthogonal to d̂ by construction, and the Pythagorean identity ‖v‖² = (v · d̂)² + ‖r‖² holds exactly. Projection is the mathematical content of many interpretability operations: "how much of this head's output is about the answer token" projects the output onto an unembedding direction; "remove the gender direction from this activation" subtracts the projection, keeping only r; "ablate this feature" is projection removal along a learned direction.

Projection onto a k dimensional subspace generalizes this. Collect an orthonormal basis of the subspace as the columns of a matrix Q (of shape d by k, with Q^TQ = I). Then proj(v) = QQ^Tv: read out k coordinates with Q^T, then rebuild the vector with Q. If the basis is not orthonormal, say the columns of a full-rank matrix A span the subspace, the projector is A(A^TA)^-1A^T, the least squares formula. Subspace projection appears whenever a claim involves more than one direction: an attention head writes into a 64 dimensional subspace of the stream, and "what the head wrote" at a position is the projection of the stream update onto that subspace. Concept erasure methods project activations onto the orthogonal complement of a concept subspace. In all cases the decomposition v = (part inside) + (part outside) is exact and the two parts are orthogonal.

B.4 Basis, change of basis, and why the stream has no privileged one#

A basis of R^d is a set of d linearly independent vectors; every vector then has unique coordinates with respect to that basis. The standard basis, e₁ through e_d, is what your tensors are stored in: coordinate i of an activation is its dot product with e_i. A change of basis is multiplication by an invertible matrix; when the new basis is orthonormal, that matrix is a rotation (orthogonal matrix R with R^TR = I), and it preserves all norms, dot products, and angles.

Here is the point that matters for interpretability. Some activation spaces in a transformer have a privileged basis and some do not. The MLP hidden layer applies an elementwise nonlinearity, GELU, and elementwise means coordinate by coordinate: the architecture itself singles out the standard basis, so it is at least coherent to ask what neuron 1337 does. The residual stream has no such operation. Every interaction with the stream is through matrix multiplication, and if you rotate the stream by R while replacing every write matrix W by WR and every read matrix W by R^TW (reading Refresher A's conventions), the model computes exactly the same function. Nothing distinguishes the stored coordinates from any rotation of them. Consequently, questions about the stream must be basis independent: ask about directions, subspaces, dot products, and norms, which survive rotation, not about individual coordinates, which do not. The empirical caveat from Chapter 1 stands: training artifacts such as outlier dimensions can make the standard basis statistically special in practice even though the architecture does not privilege it.

B.5 Rank, low rank factorization, and the SVD#

The rank of a matrix M is the dimension of its image: how many dimensions survive the map. A d by d matrix of rank k < d destroys d - k dimensions of input and reaches only a k dimensional subspace of output. Rank interacts with factorization: if M = AB with A of shape (m, k) and B of shape (k, n), then rank(M) ≤ k, because everything must squeeze through the k dimensional middle. Transformers are full of exactly this structure. An attention head's effective value-output map W_VW_O is (768, 64) times (64, 768): a 768 by 768 matrix of rank at most 64. The head can only move a 64 dimensional slice of the stream, and knowing which slice is knowing what the head does.

The singular value decomposition is the tool that names the slice. Any real matrix M of shape (m, n) factors as

where U and V have orthonormal columns (the left and right singular vectors u_i and v_i) and Σ is diagonal with nonnegative entries σ₁ ≥ σ₂ ≥ ... ≥ 0, the singular values. Geometrically, applying M is: rotate the input to align with the right singular vectors (V^T), scale each resulting coordinate i by σ_i (Σ), then rotate into the output space along the left singular vectors (U). Rotate, scale, rotate. Every matrix, however messy, is a diagonal scaling wearing two rotations.

The sum form is the interpretability-relevant reading: M is a sum of rank one pieces σ_iu_iv_i^T, ordered by importance. Each piece detects one input direction (v_i), and writes one output direction (u_i), with strength σ_i. The rank of M is the number of nonzero singular values, and truncating the sum after k terms gives the best rank k approximation of M in the least squares sense (the Eckart-Young theorem). When singular values decay fast, a few rank one pieces tell most of the story. This is why SVD is the standard instrument for reading transformer weight matrices: Chapter 3 runs SVD on a head's W_OV, pushes the top singular vectors through the embedding and unembedding to see them as token distributions, and often finds that a head's entire function is legible from three or four directions. Low rank is not an accident there; the architecture builds it in through the d_head bottleneck, and training appears to concentrate meaning in the top of the spectrum.

B.6 einops: shape manipulation you can read#

Interpretability code is mostly plumbing between tensors of shape (batch, position, d_model), (batch, head, position, position), (layer, head, d_model, d_head), and friends. Raw PyTorch reshape and transpose calls make that plumbing unreadable and error prone, because the meaning of each axis lives only in your head. The einops library makes the axes part of the code. Three functions cover nearly everything.

rearrange reorders, splits, and merges axes according to a pattern string. To flatten per-head outputs of shape (batch, pos, head, d_head) into (batch, pos, head*d_head), write rearrange(z, "b p h d -> b p (h d)"). To split it back, write rearrange(zf, "b p (h d) -> b p h d", h=12); einops needs the size of one of the merged axes to undo the merge, and it checks consistency of every named axis. reduce collapses axes with an operation: reduce(x, "b p d -> p d", "mean") averages over the batch; reduce(attn, "b h q k -> h", "max") finds each head's largest attention weight. repeat is the inverse of reduce, broadcasting a tensor along new axes: repeat(direction, "d -> b p d", b=4, p=10) tiles a single direction across a batch. The companion einops.einsum writes contractions with the same named-axis style: multiplying per-head z of shape (b, p, h, d_head) by W_O of shape (h, d_head, d_model) into per-head stream writes is einsum(z, W_O, "b p h d, h d m -> b p h m"), and summing out h afterward reproduces the attention block's output. The pattern string documents the operation and the library verifies it, which converts silent shape bugs into loud errors. Use it everywhere.

B.7 PyTorch hooks: intercepting the forward pass#

A PyTorch module can carry forward hooks: callbacks registered with module.register_forward_hook(fn) that fire every time the module computes its output. The callback receives the module, its inputs, and its output, and may record the output somewhere or return a modified tensor to replace it. Hooks are the mechanism beneath every intervention in this course: caching activations is a hook that copies its tensor; ablation is a hook that zeroes it; activation patching is a hook that substitutes a tensor saved from another run; steering is a hook that adds a vector. Two disciplines matter. Hooks must be removed after use (each registration returns a handle with a remove method), or they silently accumulate and every later experiment inherits your leftover interventions. And a hook that modifies its output changes everything downstream, which is precisely the point but means you must be deliberate about which run is instrumented.

B.8 The TransformerLens working set#

TransformerLens wraps GPT-style models so that every internal tensor is exposed at a named hook point, with hook registration, caching, and cleanup managed for you. The objects and methods below are the working set the entire course relies on; learn them now and every later lab gets shorter.

HookedTransformer.from_pretrained("gpt2") loads GPT-2 small with interpretability-friendly preprocessing (LayerNorm folding, unembedding centering) applied by default. Token utilities: model.to_tokens(text) returns a (batch, pos) tensor of token ids, prepending the beginning-of-sequence token by default; model.to_string(ids) inverts it; model.to_str_tokens(text) shows the tokenization piece by piece, which you should print whenever position indices matter, because tokenization rarely splits where intuition says; model.to_single_token(" Paris") returns the id of a string that must be exactly one token, and raises otherwise. The leading space matters: " Paris" and "Paris" are different tokens in GPT-2's vocabulary.

logits, cache = model.run_with_cache(tokens) runs a forward pass and stores every activation in an ActivationCache. The cache is indexed by hook name. Full names follow the scheme blocks.<layer>.<module>.hook_<tensor>: for layer 5, blocks.5.attn.hook_z is the per-head attention output before W_O of shape (batch, pos, head, d_head), blocks.5.attn.hook_pattern is the attention pattern (batch, head, query, key), blocks.5.mlp.hook_post is the MLP hidden activation, and blocks.5.hook_resid_pre, hook_resid_mid, hook_resid_post are the residual stream before attention, between attention and MLP, and after the MLP. The cache accepts a shorthand: cache["resid_post", 5] is cache["blocks.5.hook_resid_post"], and cache["z", 5], cache["pattern", 5], cache["attn_out", 5], cache["mlp_out", 5] work the same way. Weights are exposed as stacked tensors: model.W_Q has shape (layer, head, d_model, d_head), model.W_O (layer, head, d_head, d_model), model.W_E (vocab, d_model), model.W_U (d_model, vocab), with biases b_Q, b_O, and so on alongside.

For interventions, model.run_with_hooks(tokens, fwd_hooks=[(name, fn)]) runs one forward pass with your hook functions attached to the named hook points and removes them afterward. A hook function receives the tensor and a HookPoint object whose name attribute tells you where you are; returning a tensor replaces the activation. Because registration and cleanup are handled per call, the leftover-hook failure mode of raw PyTorch largely disappears, though hooks added with model.add_hook persist until model.reset_hooks(). The lab has you verify your understanding in both directions: a recording hook must reproduce what run_with_cache captured, and a cached tensor must be reproducible by hand from other cached tensors using the residual arithmetic of Refresher A.

1.1 The residual stream is the central object#

Most introductions to the transformer present it as a stack of layers, each transforming a hidden state, as if the network were a pipeline where each stage replaces the output of the previous one. That picture is accurate for older architectures but misleading for transformers. In a GPT-style model no layer ever replaces the hidden state. Each attention head and each MLP block reads from the hidden state, computes something, and adds its result back. The hidden state at a token position is therefore a running sum, and mechanistic interpretability calls this running sum the residual stream.

Formally, let x₀ = Embed(t) + PosEmbed(p) be the initial state for token t at position p, with x₀ in R^d_model. In GPT-2 small, d_model = 768. Layer l applies

where the sum over h runs over the heads of layer l. Unrolling the recurrence over all L layers gives the identity that powers most of this course:

The final state is exactly the sum of the embedding and every component's output. Nothing is lost, nothing is overwritten. Because the unembedding is (up to a final LayerNorm) a linear map from x_final to logits, every component's effect on the model's prediction is, to a controlled approximation, its own additive term. This is why interpretability treats the residual stream as a communication channel: components at early layers write information into it, components at later layers read that information out, and the stream itself performs no computation.

Two consequences deserve emphasis. First, the residual stream has no privileged basis. Nothing in the architecture ties meaning to individual coordinates of the 768-dimensional vector; any rotation of the stream, compensated by rotating the reading and writing matrices, leaves the network's function unchanged. Meaningful structure lives in directions and subspaces, not in coordinates. This is why the course speaks of "the direction representing X" rather than "the neuron representing X" when discussing the stream. Second, the stream is a bottleneck. Hundreds of components in GPT-2 small share 768 dimensions, so components must share, reuse, and overwrite subspaces, a fact that becomes central when Chapter 10 discusses superposition.

1.2 Reading and writing: components as low-rank operators on the stream#

Every component interacts with the stream through linear maps at its boundary. An attention head with head dimension d_head = 64 reads through W_V (a 768 by 64 matrix, in the reading direction) and writes through W_O (64 by 768). Whatever nonlinear computation attention performs, the information a head can move is confined to the 64-dimensional subspace image of its W_V, and the information it can write lies in the 64-dimensional row space of its W_O. A head is thus a rank-at-most-64 operator on the stream. The MLP reads through W_in (768 by 3072) and writes through W_out (3072 by 768), a much wider interface mediated by an elementwise nonlinearity.

This boundary-linearity means we can ask a precise question: how much does the output of component A feed into the input of component B? Multiply A's write matrix by B's read matrix and you get a single effective linear map, the virtual weight matrix from A to B. For head h1 writing with W_O^h1 and head h2 reading values with W_V^h2, the virtual value weights are W_O^h1 W_V^h2, a 64 by 64 map from h1's internal value space to h2's. Large virtual weights (relative to the components' typical activations) are evidence of composition, the phenomenon Chapter 3 develops into Q-, K-, and V-composition and uses to explain induction heads. The residual stream never stores "which component wrote this"; virtual weights are how we recover the effective wiring diagram anyway.

1.3 LayerNorm, the annoying denominator#

The decomposition above is exact for the residual stream itself but the phrase "linear attribution to logits" comes with a caveat: before the unembedding, and before every block, GPT-2 applies LayerNorm. LayerNorm centers its input, divides by the standard deviation of its coordinates, then applies a learned elementwise scale and bias. The centering, scaling, and bias are linear or affine and can be folded into adjacent weight matrices; TransformerLens does this automatically when loading models (fold_ln=True), which is why we can treat LN as "just" a projection followed by a fixed normalization. The division by the standard deviation is the genuinely nonlinear part: it couples all coordinates of its input.

The standard working approximation treats the final LayerNorm's scale as a constant when attributing logits to components: freeze the normalization denominator at its actual value from the forward pass, and the map from any residual stream contribution to logits becomes linear. This is exact for the question "holding the total norm fixed, how did this component move the logits" and only approximate for counterfactual questions like "what if this component were absent", since removing a component would change the denominator. The approximation is good when any single contribution is small relative to the whole stream, which is typical, and Chapter 2 quantifies attribution error cases where it is not. When you meet a paper that reports per-component logit contributions, it is using exactly this frozen-LayerNorm linearization, and you should ask how large the neglected denominator shift is.

1.4 The geometry of depth: norms grow, directions rotate#

Empirically, the norm of the residual stream grows roughly monotonically with depth, and in GPT-2 the growth is substantial: late-layer streams are an order of magnitude larger than the embedding. One driver is simple accumulation: each layer adds nonnegatively correlated contributions. A practical consequence is that when you compare "how much did layer 3 contribute" against "how much did layer 11 contribute" by raw norm, you are comparing against different baselines; contributions are best normalized by the stream norm at the point of writing. A second empirical regularity is that the direction of the stream stabilizes: cosine similarity between x_l and x_l+1 approaches 1 in later layers, partly because each layer's addition is small relative to an already-large stream. Neither regularity is a law; both are diagnostics you will compute in the lab, and deviations from them (a sudden rotation at a particular layer for a particular token) are often exactly where the interesting computation lives.

One more geometric fact matters for later chapters. A handful of residual-stream dimensions in GPT-2 have anomalously large magnitude across nearly all tokens and positions. These outlier dimensions inflate norm-based measurements, distort cosine similarities, and make naive per-dimension analysis misleading. Robust practice either works with the stream after LayerNorm (which suppresses them) or checks findings against median-based statistics.

1.5 Tracing a prompt through GPT-2 small#

The lab instrument for everything above is TransformerLens. Loading HookedTransformer.from_pretrained("gpt2") gives a GPT-2 small with every intermediate tensor exposed at a named hook point, such as blocks.5.attn.hook_z (per-head attention outputs before W_O at layer 5) or blocks.5.hook_resid_post (the stream after layer 5). Calling model.run_with_cache(tokens) returns logits plus an ActivationCache holding every one of them. Three cache methods do the accounting for you and you should understand precisely what each returns. cache.accumulated_resid(layer, incl_mid, apply_ln) returns the partial sums x₀, x₁, ..., the stream as it stands after each layer. cache.decompose_resid(layer) returns the individual addends: embedding, each layer's attention-block output, each layer's MLP output. cache.stack_head_results() splits attention-block outputs one step finer, into per-head contributions, using the fact that concatenating heads then applying W_O equals summing per-head outputs through per-head slices of W_O.

1.6 What tracing can and cannot tell you#

Residual stream tracing is descriptive, not causal. Decomposition tells you what each component wrote and how aligned that writing is with a readout direction on this particular input. It does not tell you what would happen if the component were removed, whether the information it wrote is used downstream, or whether another component would compensate. A component can write a large vector that later components ignore; a component can write a small vector that a downstream head amplifies enormously through virtual weights. Treat tracing as hypothesis generation. The causal tools that test such hypotheses, ablation and activation patching, arrive in Chapter 4, and the honest interpretation of the gap between correlational and causal evidence is a running theme of the whole course.

2.1 From decomposition to prediction: closing the accounting chain#

Chapter 1 established that the final residual state is an exact sum: embedding plus positional embedding plus one term per attention head, one per multilayer perceptron (MLP) block, plus the fixed biases each block adds. That identity lives entirely in residual space. The model, however, is judged in logit space, on which token it predicts. This chapter closes the gap between the two spaces. The map from the final residual state x_final to logits is

where W_U is the unembedding matrix (768 by 50257 in GPT-2 small) and b_U is the unembedding bias. If LN_final were linear, we could push the whole map through the residual decomposition and read off, for every component, exactly how many logits it contributed to any token. LayerNorm's division by the input's standard deviation is the one nonlinearity in the way, and Chapter 1 already introduced the standard remedy: freeze the denominator at its value from the actual forward pass. With the denominator frozen, LN_final becomes a fixed centering followed by a fixed scaling, both linear, so they distribute over the addends of the residual stream. Every result in this chapter, the logit lens and DLA alike, is an application of this one move: decompose the stream, apply the frozen LayerNorm to each piece, multiply by W_U.

Applied to partial sums of the decomposition, the move yields the logit lens: what would the model predict if it stopped after layer l. Applied to individual addends, it yields direct logit attribution: how much did this particular head or MLP push the prediction. The two are complementary views of the same linear accounting, one cumulative, one marginal, and the lab implements both on the same cached forward pass.

2.2 The logit lens: watching a prediction form#

Let x_l denote the accumulated residual stream at the position of interest after layer l, the partial sum of the embedding and every component output through layer l. The logit lens computes

for every l, using the final LayerNorm even for early layers. Each intermediate state is treated as if it were a finished answer, and the sequence of lens distributions shows the model's best guess forming with depth. In TransformerLens this is three lines: cache.accumulated_resid(layer=-1, incl_mid=True, apply_ln=True, pos_slice=-1) returns the stack of partial sums at the chosen position with the frozen final LayerNorm already applied, and a matrix multiply by model.W_U plus model.b_U converts each to logits. The natural summaries to plot are the logit of the correct token and its rank among all vocabulary tokens, layer by layer.

Run on a factual prompt such as "The Eiffel Tower is in the city of", the trajectory is striking. At the embedding the correct token " Paris" sits deep in the vocabulary, at a rank in the tens of thousands. Through the early and middle layers its rank falls steadily as attention moves the subject's identity to the final position and mid-network MLPs contribute the association, and by layer 9 or 10 " Paris" is at or near the top and stays there. The overall picture is one of iterative refinement: the network does not defer prediction to a final readout stage; it maintains a running guess in the residual stream, in a basis close enough to the output vocabulary that the unembedding can read it, and successive layers refine that guess. This picture is why the residual stream is sometimes described as carrying the model's current belief about the next token, with layers acting as updates to that belief.

Two quantitative habits make lens plots more trustworthy. Plot ranks on a logarithmic scale, since the interesting motion covers four orders of magnitude. And plot the lens logit of the target alongside its rank: the raw logit typically rises through the network and then can fall in the last layer or two, because the final layers calibrate the whole distribution rather than pushing the winner further, and a rank plot alone hides this.

2.3 Where the raw lens fails, and the tuned lens fix#

The logit lens is an approximation with a specific, well-understood failure mode: it assumes intermediate residual states already live in the basis the unembedding reads. Nothing forces this. The unembedding was trained against x_final only; earlier states merely need to be useful to later layers. Three symptoms follow. First, at very early layers the lens output is often dominated by the current input token rather than any prediction, because the embedding of the input is the largest term in the stream and W_U is correlated with the embedding. Second, representations drift: a direction that means "the next token is a city name" at layer 4 may be rotated or rescaled relative to the direction the unembedding decodes, so the lens systematically misreads mid-network states even when the information is present and linearly recoverable. Third, the failure is model dependent. On GPT-2 the raw lens gives readable trajectories; on other model families, GPT-Neo and BLOOM being the documented examples, the raw lens outputs near-garbage until the last few layers, not because those models compute nothing early but because their intermediate bases differ more from their output basis. A tool that silently works on one architecture and fails on another is dangerous if you interpret its output as ground truth about what the model knows.

The tuned lens repairs exactly this basis mismatch. For each layer l it introduces a translator, a learned affine map A_lx + c_l, inserted before the frozen final LayerNorm and unembedding. The translators are trained, with the model itself frozen, to minimize the Kullback-Leibler divergence between the translated lens distribution at layer l and the model's actual final distribution. A translator can undo rotation, rescaling, and mean shift between the layer-l basis and the output basis, but being affine it cannot add computation the model has not yet performed. The tuned lens therefore answers a sharper question than the raw lens: not "what does the unembedding say about this state" but "what does this state linearly encode about the final prediction". Empirically tuned-lens trajectories are lower perplexity, far more consistent across model families, and less biased toward the input token. The cost is that the lens is no longer parameter-free: you must train and validate translators per model, and conclusions inherit whatever the translators learned. The lab implements only the raw lens, which is adequate for GPT-2; you should reach for the tuned lens whenever the raw lens looks pathological or the model is not from the GPT-2 family.

2.4 Direct logit attribution#

Where the lens applies the frozen readout to partial sums, direct logit attribution (DLA) applies it to the individual addends. Write the final residual state at the answer position as x_final = Σ_c r_c, the sum over components c of their residual writes. With the final LayerNorm frozen, the logit vector decomposes as

and each term LN_frozen(r_c) W_U is component c's direct contribution to every logit. Dotting with a readout direction (next section) gives one scalar per component, and those scalars sum, exactly, to the model's actual output on that direction. DLA is the marginal version of the lens, and it inherits the same caveat: it is descriptive. It reports what each component wrote in the direction of the answer, on this input, holding the normalization fixed. It says nothing about what would happen if the component were removed. The exactness of the sum is what makes DLA more than a heuristic: it is a complete accounting, and any assertion built on it can be checked by reconstruction.

Granularity is a free choice. At block level, cache.decompose_resid(layer=-1) returns the embedding, positional embedding, and per-layer attention and MLP outputs. At head level, cache.stack_head_results() splits each attention block into its individual heads, using the fact from Chapter 1 that concatenating heads and applying W_O equals summing per-head outputs through per-head slices of W_O. One can go finer still, to individual MLP neurons or even to the source positions a head copied from, and the same frozen-LayerNorm projection applies at every granularity.

Three bookkeeping details decide whether your reconstruction check passes or fails, and all three cost real debugging time when ignored. First, LayerNorm handling: slice the component stack to the answer position before applying the frozen normalization, and use the cache's stored scale for that same position (cache.apply_ln_to_stack(stack, layer=-1, pos_slice=-1) in TransformerLens; note that this call slices the stored scale, not your stack, so the stack must already be at the position). Second, the unembedding bias b_U: GPT-2 has a nonzero b_U, it is added after the residual stream, and it belongs to no component. Its contribution to a logit difference is input independent but not zero, so the per-component scalars sum to the logit difference minus the b_U difference. Third, block biases: decompose_resid folds each attention block's output bias b_O into that block's term, but per-head results exclude it, since a bias belongs to no individual head. A head-level accounting therefore needs an explicit bias term, the frozen-LayerNorm projection of the summed b_O vectors, to close the books. The lab asserts the full identity: per-head contributions plus MLP and embedding contributions plus the b_O term plus the b_U difference equals the actual logit difference to numerical precision.

2.5 Logit difference: the right scalar#

DLA needs a scalar readout, and the standard choice is the logit difference between two candidate completions: logit(" Mary") minus logit(" John"), logit(" Paris") minus logit(" Rome"). The reasons are worth internalizing because the choice of metric quietly determines what every attribution method in this course reports.

The softmax is invariant to adding a constant to all logits, so a single raw logit is not meaningful on its own: a component that uniformly raises every logit changes nothing about behavior yet shows a large single-logit DLA. A difference of logits cancels any such common mode. Concretely, the readout becomes a single fixed direction in residual space, the difference of two unembedding columns, W_U[:, a] minus W_U[:, b], and DLA reduces to dotting each component's normalized write with that one vector. Second, a difference isolates the specific behavior under study from everything else the model is doing. In the IOI sentence the model also assigns mass to " the", " a", and other continuations; the Mary-versus-John difference asks only about the discrimination we care about, choosing the indirect object over the repeated subject, and is insensitive to how much probability leaks elsewhere. Third, logit differences behave better than probabilities under intervention: probabilities saturate, so a component pushing an already-confident prediction shows near-zero probability change while its logit-space effect is large and additive. The cost of the choice is symmetric blindness: a component that raises both candidates equally is invisible to the difference, which is exactly the invariance we asked for, but worth remembering when a head that "does nothing" by this metric turns out to matter for a differently posed question.

2.6 The IOI task and the heads that answer it#

Indirect object identification (IOI) is the running causal example of this course, introduced here and dissected further in Chapters 4 through 6. The template: "When John and Mary went to the store, John gave a drink to". Completing it correctly requires noticing that " John" is repeated, inferring that the recipient is the other name, and producing " Mary". GPT-2 small does this reliably, with a healthy positive logit difference for " Mary" over " John". The task earns its central role for three reasons. It is a crisp nontrivial computation, requiring the model to bind names to roles rather than echo recency. It comes with a natural metric, the Mary-versus-John logit difference. And it comes with a natural family of corrupted prompts (swap or replace the names) that later chapters use for patching.

DLA gives the first cut of the IOI circuit. Decompose the final residual state at the last position, project every head's write onto the logit difference direction, and a distinctive pattern appears: nearly all of the direct effect concentrates in a handful of heads in the last four layers, with mid-network attention and MLPs contributing modestly and early layers almost nothing. The dominant positive contributors in GPT-2 small are heads in layers 9 and 10, with L9H9 the single largest, and the original circuit analysis named them name mover heads: they attend from the final position to the position of the correct name and copy its identity into the output direction. Alongside them sit heads with substantial negative contributions, negative name movers such as L10H7 and L11H10, which write against " Mary". Their existence is an early warning that circuits are not teams of uniformly helpful parts; later chapters connect them to calibration and to the self-repair phenomena that complicate patching.

Keep the epistemic status precise. DLA on one prompt shows that these heads wrote the answer direction on this input. It does not show they computed the answer: a name mover could be copying a decision made earlier in the network (in fact it is; the S-inhibition heads that steer the name movers' attention are found by patching in Chapter 4, not by DLA, precisely because their effect on the logits is indirect). DLA finds the last movers in the chain, the components whose writes the unembedding reads directly. Everything upstream is invisible to it, which is the structural limitation the causal methods of Chapters 4 and 5 exist to overcome.

2.7 Reading attribution honestly#

The lens and DLA are the cheapest tools in the interpretability kit: one forward pass, no gradients, no interventions, exact accounting. Used well they generate strong, checkable hypotheses, and their reconstruction identities catch bookkeeping errors that would silently corrupt fancier methods. Their shared blind spot is indirection. A component whose output is consumed by later components, rather than by the unembedding, has zero direct effect no matter how essential it is; a component with a large direct write may be cancelled downstream or may be echoing a decision made elsewhere. Frozen-LayerNorm linearity is an approximation whose error grows with the size of the contribution being considered, since a genuinely absent component would have changed the normalization denominator. And single-prompt results can reflect idiosyncrasies of one sentence; serious work averages DLA over a dataset of templated prompts. Treat this chapter's methods as the descriptive first pass of every investigation: form the hypothesis with the lens and DLA, then take it to the causal tools of Chapter 4.

3.1 Factoring a head: pattern times movement#

Chapter 1 treated an attention head as a rank-at-most-64 writer to the residual stream. This chapter opens the head up. The key structural fact, from the transformer circuits framework, is that a head does exactly two separable jobs. It decides where to attend, and it decides what to move. These two jobs are carried out by two disjoint sets of parameters, and neither job sees the other's parameters at all.

Write x_i for the (LayerNorm-normalized) residual stream at position i. The attention score between destination i and source j is

followed by a causal-masked softmax over j to give the attention pattern A_ij. Here W_Q and W_K are each d_model by d_head (768 by 64 in GPT-2 small), so W_QK = W_Q W_K^T is a 768 by 768 bilinear form of rank at most 64. The individual matrices W_Q and W_K are not meaningful on their own: any invertible 64 by 64 matrix R can be inserted as W_Q R, R^-1 W_K^T without changing the model. Only the product W_QK is an invariant of the head. This is the QK circuit: a low-rank bilinear map that scores query-key pairs.

Given the pattern, the head's write to the stream at position i is

where W_V is 768 by 64, W_O is 64 by 768, and W_OV = W_V W_O is a 768 by 768 linear map, again of rank at most 64 and again invariant only as a product. This is the OV circuit: the pattern chooses a convex combination of source residual vectors, and W_OV transforms that combination into the vector written at the destination. The factorization "head = pattern applied to W_OV-transformed inputs" is exact, not an approximation. Its interpretive payoff is that the two circuits can be studied independently: the QK circuit answers "under what conditions does this head fire from i to j", and the OV circuit answers "when it fires, what function of x_j does it add at i", and the second question does not depend on the attention pattern at all. The OV circuit is a property of the weights alone, so conclusions about it hold for every input, unlike the activation-based analyses of Chapters 1 and 2 which hold for one prompt at a time.

3.2 SVD: reading a low-rank map direction by direction#

A 768 by 768 matrix of rank 64 is best understood through singular value decomposition (SVD). Writing W_OV = U S V^T, the columns of U are orthonormal input directions, S is a diagonal matrix of nonnegative singular values of which at most 64 are nonzero, and the columns of V (rows of V^T) are orthonormal output directions. The action of W_OV is: measure how much of the input lies along U's k-th column, multiply by the k-th singular value, and write that amount along V's k-th direction. A head is therefore a bank of at most 64 independent channels, each reading one direction of the stream and writing another, with a gain.

Directions in the residual stream mean nothing by inspection, but Chapter 2 gave us two dictionaries: the embedding W_E maps tokens into the stream, and the unembedding W_U maps stream directions to logits. To interpret channel k, project its input direction through the embedding (which tokens' embeddings have high inner product with U_k, i.e. what the channel reads) and its output direction through the unembedding (which token logits V_k promotes, i.e. what the channel writes). One caveat: SVD fixes each pair (U_k, V_k) only up to a joint sign flip, so always examine both the top and bottom tokens of a projection, and treat the two ends symmetrically.

Doing this on real GPT-2 heads is often strikingly informative. Head L10H9 devotes its top three channels to third-person pronoun families: one channel writes the "his/He/himself" cluster, the next writes "their/They", the next writes "her/She/hers". Head L9H10 has a channel for temporal expressions (month names, "yesterday", "shortly") and, separately, a channel for closing code delimiters. This is the observation of Millidge and Black (2022): singular vectors of transformer weight matrices, rendered in the vocabulary basis, frequently correspond to coherent semantic clusters, which is evidence that heads carry structured, human-describable functions rather than arbitrary linear mixing. It is also a practical technique you will use whenever you meet an unfamiliar head: an SVD plus two projections costs seconds and yields a first hypothesis about the head's role.

3.3 The full OV circuit and copying heads#

The OV circuit becomes fully token-level if we compose it with both dictionaries at once. Define the full OV circuit as the vocabulary-to-vocabulary matrix

Entry M[s, t] answers: if the head attends to a pure occurrence of token s, how much does its write increase the logit of token t? The most important structural question about M is whether it is diagonal-heavy. A head whose M has a strongly positive diagonal pushes up the logit of whatever token it attends to. Such a head copies. Copying is the single most load-bearing head behavior in small transformers, and detecting it from weights alone is a standard move.

M is 50257 by 50257, far too large to materialize. Two cheap surrogates do the job. First, the diagonal itself is cheap: with A = W_E W_V (50257 by 64) and B = W_O W_U (64 by 50257), the diagonal is the rowwise dot product of A with B^T, no large matrix required. For the top induction head of GPT-2 small, 96 percent of diagonal entries are positive. Second, eigenvalues. A matrix with positive eigenvalues maps vectors to outputs positively aligned with the input, which for M means "attend to s, promote s". The eigenvalues of the huge M can be obtained through a classical identity: the nonzero eigenvalues of a product AB equal the nonzero eigenvalues of BA. Since every factor of M has inner dimension 64, M has at most 64 nonzero eigenvalues and they equal the eigenvalues of a 64 by 64 matrix. The same identity applies one level down: the nonzero eigenvalues of W_OV = W_V W_O (768 by 768) are exactly the eigenvalues of W_O W_V (64 by 64), so a full-model scan over all 144 heads of GPT-2 small costs a fraction of a second. The summary statistic is the copying score Σ_k Re(λ_k) / Σ_k |λ_k|, which lies in [-1, 1]; note W_OV is not symmetric, so eigenvalues are complex in general and the real part carries the aligned component. A score near +1 means essentially all eigenvalue mass is positive real: the head writes back what it reads, in the same directions. In GPT-2 small a band of late heads (L9 through L11) score above 0.99 while the all-head mean is about 0.25, a clean bimodal signature of a dedicated copying population.

3.4 Composition: how heads become circuits#

A single head is a fixed linear rule gated by a pattern. Nontrivial algorithms arise when heads compose across layers, and the residual stream permits exactly three composition routes into a head. The output of an earlier head, x W_OV⁽¹⁾, sits in the stream when a later head reads it. If it enters through the later head's query input it shapes where that head attends from the destination side (Q-composition). If it enters through the key input it shapes which sources look attractive (K-composition). If it enters through the value input it becomes part of what is moved (V-composition), and the two heads' OV circuits chain into the virtual weight W_OV⁽¹⁾ W_OV⁽²⁾ of Chapter 1.

Whether a route is live is a property of the weights. The Chapter 1 heuristic (Frobenius norm of the product over the product of norms) becomes, in this chapter's notation, the composition score family of Elhage et al. For heads h1 (earlier) and h2 (later):

The intuition: two random 64-dimensional subspaces of a 768-dimensional space overlap little, so the norm of the product of two unrelated low-rank maps is small relative to the factors. A score well above the random baseline means h1's write subspace is preferentially aligned with the relevant read subspace of h2, so the wiring exists for h1's output to steer h2. The score is evidence of a channel, not of use: it says nothing about whether typical inputs actually send signal down the channel, which is why weight-based composition scores are always paired with an activation-based or causal check.

3.5 Induction heads: the canonical two-head circuit#

The flagship application of composition is the induction circuit, which implements in-context pattern completion: having seen ... A B ... earlier in the context, on encountering A again, predict B. No single head can do this. The QK circuit of a head in layer 0 can only compare raw token and position embeddings, so it can match tokens or fixed offsets but cannot express "the token after the previous occurrence of the current token". The circuit needs two heads.

The first ingredient is a previous-token head: a head whose attention pattern robustly attends from position j to position j-1, and whose OV circuit therefore writes information about token j-1 into the stream at position j. In GPT-2 small, L4H11 attends to the previous token with pattern weight 0.99 on random text. The second ingredient is an induction head in a later layer whose QK circuit K-composes with the previous-token head: its query at the current position encodes the current token A, and its key at position j reads, through the previous-token head's write, the identity of token j-1. The bilinear form then scores position j highly exactly when token j-1 equals A, that is, when position j sits immediately after an earlier occurrence of A. Attention lands on B, and the induction head's OV circuit, which is a copying circuit in the sense of section 3.3, promotes B's logit. Prefix matching by K-composition, then copying by OV.

Detection is refreshingly operational. Feed the model a sequence of random tokens repeated twice, say 25 random tokens then the same 25 again. Random tokens kill every confound: no natural-language statistics, no memorized n-grams, so any head that systematically attends from position i in the second half to position i-24 (the token after its own earlier copy, one period minus one back) can only be doing prefix matching. The mean attention weight on that offset is the induction score, also called the prefix-matching score, and the induction stripe it produces in the attention pattern is unmistakable. In GPT-2 small, L5H5 scores about 0.94 on such sequences, several other heads in layers 5 through 7 score above 0.5, and the mean over all 144 heads is under 0.1. The weight-level story closes the loop: among all heads in layers 0 through 4, the K-composition score into L5H5 is maximized by exactly L4H11, at about twice the mean. Olsson et al. (2022) argue these heads are not a curiosity: their emergence during training coincides with a sharp drop in in-context loss, and they appear to be a substantial mechanism of in-context learning in models of every size.

3.6 What attention patterns alone cannot tell you#

Attention pattern analysis, staring at A_ij heatmaps, is the oldest interpretability method for transformers and remains useful as a fast screen; the induction stripe above is an attention-pattern observation. But a pattern is half a head. First, a pattern shows where information is gathered from, not what is done with it: two heads with identical patterns, one with a copying OV circuit and one with W_OV mapping tokens to unrelated logits, do entirely different things. Without the OV analysis of this chapter, "attends to the subject" licenses no conclusion about what the head contributes to the output. Second, attention weight is not importance: a large A_ij moving a vector that downstream components ignore, or that W_OV nearly annihilates (the value vector's norm matters as much as the pattern weight), changes nothing. Third, patterns invite narrative overreading; the literature contains heads whose patterns suggested syntax but whose OV circuits revealed positional bookkeeping. The discipline this chapter installs: every claim "head h does X" should name both circuits, where it attends (QK) and what it moves (OV), and should ultimately be tested causally with the tools of the next chapter.

4.1 The interventionist turn#

Chapters 1 and 2 built a toolkit for describing what a model wrote: decompose the residual stream, project each component's contribution onto a readout direction, rank the components. Every one of those numbers is correlational. Direct logit attribution (DLA) tells you that head L9H9's output, as written on this input, points strongly along the logit difference direction. It does not tell you what the model would do if that output were different, whether the information originated in that head or merely passed through it, or whether the rest of the network depends on it at all. A component can have zero DLA and still be load bearing, for example a head that moves information between positions in the middle of the network without writing anything the unembedding can read. The name mover heads of the IOI circuit are visible to DLA; the duplicate token heads and subject inhibition heads that feed them are nearly invisible.

To move from "correlated with the output" to "causes the output" you must intervene: change something inside the network and observe the consequence. The crudest intervention is ablation, setting a component's output to zero or to a dataset mean. Ablation answers a blunt question, "what happens without this component", and the answer is often distorted because zero is far outside the distribution of activations the downstream network was trained to consume. Activation patching is the sharper instrument. Instead of deleting an activation, you replace it with the same activation from a counterfactual input. Both values are ones the network produces naturally; the intervention stays on the data manifold, and the difference between the two inputs is under your control. Patching is the core causal method of this course. Chapter 5 approximates it with gradients, Chapter 6 automates it into circuit discovery, and Chapter 9's causal tracing for model editing is a special case of it.

4.2 The clean/corrupted paradigm#

Every patching experiment starts with two runs. The clean run is the prompt on which the model does the thing you care about. The corrupted run is a modified prompt on which it does something else. The experiment then asks, for each internal site: if I transplant the activation at this site from one run into the other, how much does the output move between the two behaviors?

The running example for this chapter, and for much of the interpretability literature, is the IOI minimal pair:

One token differs, the second occurrence of the subject name, and the correct completion flips. Because both names tokenize to a single token, the two prompts have identical length and identical tokens at every position except one. This matters more than it looks. Patching transplants the activation at a specific (layer, position) coordinate, so the two runs must be positionally aligned: position 10 in the clean run must mean the same thing as position 10 in the corrupted run. A corruption that changes the token count, or shifts the sentence structure, destroys that alignment and makes the results uninterpretable. The first cell of any patching experiment should assert that the clean and corrupted token tensors have the same shape and differ only where intended.

Minimality serves a second purpose: it controls what the experiment can find. The only information that differs between the two runs is "which name is the repeated subject", so any site whose patch moves the output must carry exactly that information. A sloppier corruption, say replacing the whole sentence with unrelated text, changes hundreds of features at once, and a positive patching result then tells you only that the site carries something relevant, not what.

4.3 Denoising and noising answer different questions#

There are two directions to patch, and they are not interchangeable.

Denoising copies a clean activation into the corrupted run. If the output moves toward the clean behavior, the patched activation is sufficient, in the context of an otherwise corrupted forward pass, to restore the behavior. Causal tracing and most localization work use this direction. Denoising is a demanding test: a single site only scores highly if it carries enough of the clean signal, on its own, to overcome everything else still being corrupted. Sites that pass are strong candidates for being the routes through which the critical information flows.

Noising copies a corrupted activation into the clean run. If the output degrades toward the corrupted behavior, the clean activation at that site was necessary: the rest of the clean run cannot maintain the behavior without it. Noising is closer in spirit to ablation, but with an on-distribution replacement instead of zeros.

The two directions give different circuit-level conclusions, and conflating them is a standard error. Consider a model that computes the answer through two redundant pathways, an OR gate. Noising either pathway alone changes little, because the other still delivers the answer, so noising reports that neither is necessary. Denoising either pathway alone restores the behavior, so denoising reports that each is sufficient. Both reports are true; neither alone describes the circuit. Conversely, in an AND-like structure where several components must all be intact, denoising one component of the conjunction restores nothing, understating its importance, while noising it is devastating. A careful investigation runs both directions and treats disagreement between them as information about circuit structure, not as noise. When you read a paper, check which direction was run before accepting a claim of the form "component X is (not) important"; sufficiency evidence cannot support a necessity claim, and the reverse.

4.4 Choosing the corruption#

Two corruption styles dominate practice. The token swap, as in the IOI pair, exchanges a small number of tokens for alternatives of the same length and syntactic role. Its strength is specificity: the difference between runs is a single controlled feature, so results localize that feature. Its cost is coverage. You need a natural minimal pair, which not every behavior admits, and the corrupted prompt is itself a well-formed input on which the model runs a related computation, so patches can interact with that computation in ways that need thought.

Gaussian noising, the choice of ROME-style causal tracing, adds noise to the input embeddings of the tokens naming the critical entity, for example the subject of a factual statement. The noise scale is set to a few times the standard deviation of embedding components, enough to destroy the identity of the entity. Its strength is generality: it needs no matched counterfactual, so it applies to any prompt. Its costs are that noised embeddings are off-distribution inputs the model never saw in training, that the noise destroys all information in those positions rather than one controlled feature, and that results can be sensitive to the noise scale. Zhang and Nanda (2023) show that the two corruption styles can disagree materially about which layers matter, notably in causal tracing for factual recall, where symmetric token-swap corruptions dampen the apparent importance of mid-layer MLP sites that Gaussian noising highlights. The corruption is part of the experimental design, not a nuisance parameter; report it, vary it, and distrust conclusions that survive only one choice.

4.5 What to patch: the granularity ladder#

A transformer offers a hierarchy of patchable sites, and the right choice depends on the question. The coarsest useful site is the residual stream itself, hook_resid_pre at layer l and position p. Patching it transplants everything the first l layers wrote at that position. A layer by position heatmap of residual patching is the standard first map of a behavior: it shows where in the sequence the critical information lives and at which depth it moves. In the IOI pair you will see in the lab, the information sits at the corrupted name position through the early and middle layers, then jumps to the final position around layers 8 to 10, the signature of attention moving it where the unembedding can use it.

One structural fact makes these heatmaps easy to sanity check. Causal attention means position p at layer l depends only on positions up to p of the input. Any patch at a position strictly before the corruption site transplants an activation identical to the one already there, so its effect is exactly zero. Nonzero readings there mean a bug, and the lab asserts this.

Finer sites decompose a layer. Patching hook_attn_out or hook_mlp_out separates what attention moved from what the MLP computed. Patching hook_z for a single head, the per-head output before the output projection, localizes to one of the 144 heads of GPT-2 small. Finer still, you can patch a head's attention pattern (did it look at the right place?) separately from its value vectors (did it carry the right content?), or its query, key, and value inputs individually. Each step down the ladder multiplies the number of experiments and sharpens the mechanistic claim: patching z tells you a head matters; patching pattern versus v tells you whether it matters because of where it attends or what it moves. Wang et al. (2022) used exactly this dissection to separate the IOI circuit's name movers, which matter through their values and attention, from the subject inhibition heads, which matter by modifying the queries of downstream heads.

4.6 Metrics, and how they change conclusions#

Patching produces a modified output; you must summarize it as a scalar. The default for two-candidate tasks is the logit difference between the correct and the competing completion, for IOI logit(" Mary") minus logit(" John") at the final position. Chapter 2 argued for logit differences on attribution grounds; for patching they have the further virtue of being linear in the final residual stream and symmetric under the swap that defines the corruption.

Raw logit differences are awkward to compare across prompts and models, so patching results are usually normalized. Define LD_clean and LD_corr as the logit differences of the unpatched clean and corrupted runs, and LD_patch as the patched run's value. The normalized metric for denoising is

so m = 0 means the patch did nothing (output still fully corrupted) and m = 1 means full restoration of clean behavior. Values slightly outside [0, 1] are real and meaningful: m > 1 means the patch overshoots the clean logit difference. The same formula serves noising experiments, where success is movement toward 0 instead of 1.

The main alternative is the KL divergence from the clean run's full next-token distribution to the patched run's distribution. KL sees the whole vocabulary, so it catches effects a two-token logit difference misses, and it is the metric of choice when no natural answer pair exists or when you want to detect any behavioral change at all, as automated circuit discovery does in Chapter 6. But metric choice is not neutral. The logit difference is signed and rewards movement along one axis, so a component that damages both candidate logits equally scores zero on it while scoring large on KL. Probability-based metrics saturate: once the correct token has probability near 1, large logit movements barely register, so a probability metric can rank a head as unimportant that a logit metric ranks first. Zhang and Nanda (2023) document cases where the identified "important layers" change substantially with the metric alone, holding the corruption and sites fixed. State your metric, justify it against the claim you are making, and rerun the ranking under a second metric before believing it.

4.7 Pitfalls: self-repair, OR gates, and small effects#

The clean logic of 4.3 meets a complication in real models: the network reacts to your intervention. When a name mover head is ablated or noised, other heads, called backup name movers, increase their contribution and recover much of the lost logit difference. This self-repair means noising a single component understates its importance, sometimes drastically: the measured necessity of a head is not a property of the head but of the head plus the compensation machinery behind it. The IOI paper found this directly, and later work has shown self-repair is widespread. Practical consequences: single-component noising results are lower bounds on importance; knocking out sets of components (the primary plus its backups) gives a truer picture; and a small noising effect must never be read as "this component is not involved".

Denoising has the dual failure. In OR-gated structures, restoring one pathway suffices, so denoising happily assigns high scores to several pathways whose joint story only makes sense once you know the gate structure. And both directions share a subtler trap: over-interpreting small patches. A site with m = 0.05 moved the logit difference by a third of a logit; with enough sites, many will show such effects through LayerNorm interactions, norm changes, and generic disruption rather than through the mechanism you care about. Effects of a few percent are where motivated reasoning lives. Set an effect-size threshold in advance, check sign consistency across a set of prompt pairs rather than one, and treat the heatmap as a hypothesis generator whose winners must survive finer-grained patching before they enter your circuit diagram.

Finally, remember what a perfect patching result licenses. Finding that a site restores m = 1 tells you the needed information passes through that site on this task distribution. It does not tell you how the information is encoded, what computation produced it, or whether the model uses the same route on other inputs. Localization is the beginning of a mechanistic account, not the end.

4.8 The cost of exhaustiveness#

Patching is one forward pass per site. A single layer by position residual map for a 15 token prompt in a 12 layer model is 180 runs; all heads at all positions is 2160; every pattern/q/k/v variant multiplies again, and every number should be averaged over a dataset of pairs, not one. For GPT-2 small on a short prompt this is minutes; for a frontier model over a benchmark it is prohibitive. The lab keeps within its time budget only by patching a handful of informative positions and one band of layers for heads, and that discipline is the honest version of what every real investigation does: choose sites by hypothesis, not exhaustively. Chapter 5 removes the constraint a different way, approximating the effect of every patch simultaneously from one forward and one backward pass. The approximation is imperfect in instructive ways, and understanding this chapter's exact method is what makes the next chapter's fast method auditable.

5.1 The cost problem#

Activation patching, as practiced in Chapter 4, is honest but expensive. Every patch is one forward pass: restore one head's clean activation into the corrupted run, read the metric, repeat. GPT-2 small has 144 attention heads; patching each head at each of, say, 15 token positions is over two thousand forward passes for one prompt pair. That is tolerable for a small model and a single experiment. It stops being tolerable when you scale any of the three axes that real investigations scale: model size (a 70B model has thousands of heads and each forward pass is expensive), dataset size (averaging over hundreds of prompt pairs to suppress noise), or granularity.

Granularity is the axis that explodes. Chapter 4 patched nodes, whole component outputs. Circuit discovery, the subject of Chapter 6, needs edges: it must ask not just "does head 9.9 matter" but "does head 8.10's output matter specifically as an input to head 9.9's query". The number of edges between component outputs and downstream component inputs grows quadratically in the number of components. In GPT-2 small a reasonable edge-level graph has tens of thousands of edges; in a frontier model, billions. A method that costs one forward pass per question cannot answer quadratically many questions. We need a method whose cost is independent of how many questions we ask.

5.2 Attribution patching: the derivation#

Fix a prompt pair (clean, corrupted) and a scalar patching metric M, for instance the logit difference between the two candidate answers, computed from the model's output. Consider a single activation tensor a somewhere in the network (one head's output, one residual position, anything with a hook). Write a_clean and a_corr for its values on the two runs. A denoising patch replaces a_corr with a_clean inside the corrupted run and measures the change in the metric:

Now treat M, on the corrupted run, as a function of the value of this one activation, holding every activation upstream of it fixed. Taylor-expand around the corrupted value:

Substituting a = a_clean gives the attribution patching estimate:

The dot product runs over every element of the activation tensor, so for a head output you sum over positions and head dimensions. Three facts make this formula powerful rather than merely cute. First, a_clean comes from one forward pass on the clean prompt, cached for every component simultaneously. Second, a_corr comes from one forward pass on the corrupted prompt, likewise cached everywhere. Third, and decisively, backpropagation computes ∇_aM at every intermediate activation in one backward pass from the metric on the corrupted run. That is what backpropagation is: the reverse-mode sweep that delivers the gradient of one scalar with respect to all tensors that produced it. So the full recipe is: run the clean forward and cache activations; run the corrupted forward with gradients enabled and cache activations; backpropagate the metric once; then every component's attribution score, all 144 heads, all MLP layers, every position, every edge we define in Section 5.5, is a cheap elementwise multiply-and-sum over cached tensors. Two forwards, one backward, then arithmetic. The cost no longer depends on the number of questions.

The same derivation with the roles reversed (expand around the clean run, evaluate at the corrupted activation) gives the noising estimate. The two are not equal, because the gradient is evaluated at different points; the convention in this chapter and the lab is the denoising form, with the gradient taken on the corrupted run.

5.3 Where the linearization fails#

Attribution patching is exact only if the metric is a linear function of the activation between a_corr and a_clean. Everything downstream of the patched activation intervenes in that function: attention softmaxes, GELU nonlinearities, LayerNorm denominators, and finally whatever nonlinearity the metric itself applies. The estimate degrades in predictable places.

Saturated attention is the classic failure. Suppose a downstream head places attention probability 0.99 on one token in both the clean and corrupted runs. The softmax is nearly flat there: an infinitesimal change in its logits changes probabilities almost not at all, so the gradient through it is tiny. But the true patch may move the attention logits by many units, enough to travel out of the flat region and redistribute attention massively. The first-order estimate, built from the tiny local slope, reports almost nothing while the true effect is large. Kramar et al. found exactly this pattern dominating attribution error in practice: queries and keys of saturated attention are where AtP is weakest.

The limiting case deserves its own name: the zero-gradient blind spot. If ∇_aM happens to vanish at the corrupted point, the attribution estimate for that component is exactly zero regardless of how large its true effect is, since the formula multiplies the activation difference by the gradient. A gradient can vanish at a point for many reasons: the component sits at a local extremum of the metric with respect to that activation, a downstream softmax is saturated, or two downstream paths cancel at first order. None of these imply the component is unimportant. This is a systematic false-negative mode, and it is why validation against true patching (Section 5.7) is not optional.

Two further failure sources are LayerNorm, whose division by the input's standard deviation makes even the residual-to-logit map nonlinear (Chapter 1 discussed why we routinely freeze it), and sheer distance: the Taylor expansion is local, so the larger ‖a_clean − a_corr‖ is, the less the tangent at one endpoint says about the function's total change. Corruptions that alter many tokens or move activations far, such as Gaussian noising of embeddings, strain the approximation more than a minimal one-token swap does. This is one more reason the field prefers minimal pairs.

5.4 Nodes and edges: the computational graph#

To go beyond per-component scores we need to say precisely what an edge is. Define a directed graph whose nodes are component outputs and component inputs: each attention head contributes an output node (its result written to the residual stream) and three input nodes (the residual streams feeding its query, key, and value projections); each MLP contributes an output node and an input node; the token embedding is a source; the final residual stream feeding the unembedding is the sink. An edge (u → v) connects the output of upstream component u to one input of downstream component v.

What makes this graph more than a metaphor is the linearity of the residual stream. The input that component v reads is the stream at its layer, and Chapter 1 established that this stream is exactly the sum of the embedding and every upstream component's output. So "u's contribution to v's input" is not an approximation or an allocation rule; it is the addend u wrote, unchanged, sitting inside the sum v reads. The stream is a wire, and edges are well defined because addition is. If components were connected through nonlinearities, asking "how much of v's input came from u" would have no canonical answer. In the residual stream it has an exact one, up to the LayerNorm applied at v's input, which we handle with the usual frozen-scale linearization.

Node patching and edge patching answer different questions, and the difference matters for circuits. Patching head u's output tests u's total effect through all paths. Patching the edge (u → v) tests one path: it asks what happens if v alone reads u's clean output while every other reader still sees the corrupted one. Chapter 4's IOI analysis needed this distinction constantly: the S-inhibition heads matter almost entirely through the query input of the name-mover heads, not through their values or through the direct path to the logits. Edges carry the circuit structure.

5.5 Edge attribution patching#

Combining the last two sections gives edge attribution patching. The true edge patch would replace, in v's input only, u's corrupted contribution with its clean contribution, changing v's input by Δu = u_clean − u_corr (the difference of u's output written to the stream). First-order in that change:

The score couples u's activation difference with the gradient of the metric at v's input. Everything needed is already in hand after the standard recipe: u's output difference comes from the two cached forwards, and the backward pass delivered the metric gradient at every component input (in TransformerLens, hooks like hook_q_input, hook_k_input, hook_v_input expose the per-head input streams once split-input mode is enabled). One backward pass prices every edge in the graph. The edge to the sink is the special case v = logits: score u's output difference against the gradient at the final residual stream, which measures u's direct effect on the metric, a gradient-based sibling of the direct logit attribution (DLA) of Chapter 2.

Syed et al. showed that ranking edges by |EAP| and keeping the top ones recovers circuits that match or beat greedy causal discovery methods at a small fraction of the cost, which is why EAP scores are the standard first pass of automated circuit discovery in Chapter 6.

5.6 EAP-IG: integrating the gradient#

The failures of Section 5.3 share a root cause: the gradient is evaluated at a single point, and the function between the corrupted and clean activations may bend. Integrated gradients (IG) addresses exactly this. Instead of the slope at one endpoint, average the gradient along the straight-line path between the two activation settings. For m steps, evaluate the gradient at the interpolated points a_corr + (k/m)(a_clean − a_corr) for k = 0, ..., m−1, average the m gradients, and use that average in place of the single-point gradient:

In practice the interpolation is applied at the model inputs or at a chosen activation site, and each of the m points costs one forward and one backward pass. The averaged gradient sees the whole path: if a downstream softmax is flat at the corrupted endpoint but steep in the middle of the path, the average captures the steep region that the endpoint gradient misses, which directly repairs saturation blind spots. Hanna et al. found that circuits selected by EAP-IG scores are consistently more faithful (Chapter 6 defines faithfulness precisely) than circuits selected by plain EAP, at the cost of m times the compute, with m between 5 and 30 typically sufficient. The method interpolates between the two regimes you know: m = 1 at the corrupted endpoint is plain attribution patching, and the exact line integral (m → ∞) of a gradient recovers the true difference of the metric along that path by the fundamental theorem of calculus, though not the true patch effect itself, since the real intervention does not move activations along this straight line jointly.

5.7 Validation: never trust an estimate you have not spot-checked#

Attribution patching earns its keep only if it agrees with the ground truth it approximates, and agreement is an empirical question that must be re-answered for each model, task, metric, and corruption. The standard protocol is cheap because it only needs ground truth for a subset. Compute attribution scores for all components; select a validation set that mixes the components with the largest |attribution| (where errors matter most) with a random sample (to catch blind spots among components the estimate claims are unimportant); run true activation patching for just that subset; scatter true effect against estimate; report the correlation; then read the outliers one by one.

The outliers are where the science is. A point far below the diagonal with near-zero estimate and large true effect is a blind spot, and the first suspects are saturated attention downstream or gradient cancellation. A point with correct sign but shrunken magnitude usually reflects curvature over a large activation difference. Systematic sign flips suggest the metric is nonmonotone in the activation over the patch distance. On IOI with GPT-2 small and a logit-difference metric, node attribution for head outputs correlates with true patching above r = 0.95, and you will reproduce this in the lab; on harder metrics (KL divergence, which is minimized at the corrupted run, making its gradient structure less friendly) and on queries and keys, agreement is visibly worse.

5.8 AtP*: patching the approximation#

Kramar et al. studied attribution patching's failures at scale and proposed AtP*, two targeted fixes. First, the QK fix: for attention queries and keys, do not linearize through the softmax. Instead, recompute the attention probabilities exactly with the patched queries or keys (cheap, since it does not require rerunning the rest of the model) and linearize only downstream of the attention pattern. This removes the saturation failure at its source. Second, gradient dropout: gradient cancellation between downstream paths causes blind spots, so AtP* recomputes attributions several times with random subsets of gradient paths dropped, making it unlikely that cancellation hides a component in every run. Both corrections keep the cost far below exhaustive patching. The conceptual lesson generalizes: when a cheap approximation fails in a structured, diagnosable way, fix the structure rather than abandoning the approximation.

5.9 What attribution patching is for#

Keep the division of labor clear. Attribution patching is a screening instrument: it prices every node and edge cheaply, with known failure modes, so that expensive exact methods can be spent where they matter. It is not a replacement for causal evidence. A finding of the form "head X matters for behavior Y" should rest on true patching of X, with attribution having told you to look at X in the first place. The workflow that Chapter 6 automates is exactly this: score all edges with EAP or EAP-IG, keep the top-k subgraph, then evaluate that subgraph with real interventions and faithfulness metrics. Gradient estimates nominate; interventions confirm.

6.1 Circuits as subgraphs#

Chapters 4 and 5 localized behavior one component or one edge at a time. This chapter assembles those pieces into an object with a definition. Fix a model and unroll it into a component-level computational graph G. The nodes are the embedding, every attention head, every MLP block, and the logits; edges connect node u to node v whenever u's output can reach v's input directly through the residual stream, without passing through another component. Because the stream is a shared additive channel, this graph is dense: every head feeds every later head's query, key, and value inputs, every MLP, and the logits. GPT-2 small has 158 component nodes and, at the granularity of separate query, key, and value inputs, around 32,000 edges.

A circuit C is a subgraph of G proposed as the mechanism for a specific task. The claim "C is the circuit for task T" is a compressed causal hypothesis: computation relevant to T flows along the edges of C, and everything outside C is irrelevant to T. The indirect object identification (IOI) circuit of Chapter 4 is the canonical example: 26 heads in seven functional classes (duplicate token, induction, S-inhibition, name mover, negative name mover, backup name mover, previous token) wired in a specific pattern. Note what the formalism buys. Without it, "we found the circuit" is a narrative about interesting heads. With it, the claim has a precise complement (everything not in C), and the complement is what evaluation interrogates.

Circuits can be defined at several granularities: full edge level (which specific connections matter), node level (which components matter, keeping all edges among them), or coarser still, layer level. Finer granularity makes stronger claims and costs more to search and evaluate. The lab works at the node level over attention heads, which preserves the logic of the pipeline at a fraction of the cost.

6.2 The discovery pipeline#

Every circuit discovery method, manual or automated, instantiates the same four-stage pipeline. First, fix a task: a distribution of prompts on which the model exhibits the behavior, such as IOI templates with names and objects varied. Second, fix a metric: a scalar functional of the output that quantifies the behavior, such as the logit difference between the correct and the distractor name, or the KL divergence from the model's clean output distribution. Third, fix a corruption: a paired distribution of counterfactual prompts that removes the behavior while disturbing as little else as possible, such as swapping the repeated name. Fourth, run a search: some procedure that uses interventions or their approximations to decide which nodes and edges to keep.

The first three choices are not preprocessing; they define what "the circuit" means. A circuit for IOI under logit difference need not equal the circuit under KL divergence, because logit difference only tracks two logits while KL tracks the whole distribution. A circuit discovered against name-swap corruptions answers "what processes the name structure", while one discovered against Gaussian-noise corruptions answers "what depends on these embeddings at all". When two papers disagree about a circuit, check these three choices before comparing search algorithms.

6.3 ACDC: greedy recursive edge pruning#

Automated circuit discovery (ACDC) automates what the IOI authors did by hand. It operates on the edge-level graph and asks, for each edge, whether the model's behavior survives that edge's removal. "Removing" an edge (u, v) means running the model with v's input receiving u's corrupted-run output instead of its clean-run output, exactly the path-level patching of Chapter 5, while all other edges into v stay clean. The criterion is distributional: remove the edge, measure the KL divergence from the full model's clean output distribution, and if the divergence increase is below a threshold τ, discard the edge permanently.

The algorithm iterates backward from the output. Starting at the logits, it considers every incoming edge in turn, prunes those that pass the threshold test, then recurses into the surviving parents, layer by layer toward the embeddings. Greedy backward ordering has a rationale: an edge matters only if its information reaches the output through surviving downstream edges, so pruning downstream first prevents wasted work upstream. The threshold τ is the sensitivity dial. Small τ keeps many edges and yields large, conservative circuits; large τ prunes aggressively and yields small circuits that miss weak but real pathways, and there is no principled way to choose τ other than sweeping it and reporting the tradeoff curve.

The cost is the method's main liability. Every edge test is a forward pass, edges are tested repeatedly as the frontier moves, and realistic runs on GPT-2 small require tens of thousands of forward passes, hours of computation for one task on one small model. The cost scales with the number of edges, which grows roughly quadratically with model size. ACDC is also greedy: once an edge is pruned it never returns, so an edge that matters only in combination with another (a backup pathway dormant until its partner is removed) can be pruned early on the basis of a small solo effect. Greedy pruning commits exactly the independence error that self-repair phenomena punish.

6.4 Attribution-based discovery, and a third paradigm#

Chapter 5 built the alternative. Edge attribution patching (EAP) prices every edge with two forward passes and one backward pass, using the first-order approximation of the patching effect. Attribution-based discovery is then a one-liner: rank all edges (or nodes) by absolute EAP score and keep the top k, sweeping k instead of τ. There is no recursion and no sequential dependence; the whole ranking comes from one gradient computation.

The empirical finding that made this the default is that attribution ranking rivals or beats ACDC's greedy search at recovering known circuits, measured by area under the receiver operating characteristic curve against manually verified ground truth edges, while costing three model passes instead of tens of thousands. The result deserves a moment of surprise: a linearized estimate outperforms the method that runs true interventions. The explanation is that ACDC's exactness per test does not compensate for the pathologies of greedy sequential commitment, while EAP's noisy scores are computed for all edges simultaneously and independently, so a top-k cut degrades gracefully. Refinements such as EAP with integrated gradients tighten the estimates further, and hybrid schemes use EAP to shortlist edges and true patching to verify the shortlist.

A third search paradigm learns the circuit instead of searching for it: subnetwork probing places a continuous mask parameter on every component or edge, runs the model with masked components ablated, and optimizes the masks by gradient descent to preserve task behavior while an L0-style penalty pushes masks toward zero. The surviving unmasked subnetwork is the circuit. This turns discovery into differentiable optimization, with its own benefits (joint rather than per-edge decisions) and costs (nonconvexity, seed dependence). Chapter 7 develops it fully; here it completes the taxonomy: exhaustive greedy intervention (ACDC), linearized scoring (EAP), and learned masking (subnetwork probing).

6.5 Evaluating a candidate circuit#

A search procedure outputs a candidate. Whether the candidate deserves the name "circuit" is a separate question with three parts.

Faithfulness asks whether the circuit is sufficient. Run the model with only the circuit intact: every component (or edge) outside C is ablated, and the metric is compared to the full model's. A common form is F(C) = m(C) / m(M), the fraction of the full-model metric the circuit retains; a stricter form normalizes against the empty circuit, F(C) = (m(C) - m(∅)) / (m(M) - m(∅)), so that a circuit scores zero when it does no better than ablating everything. The normalized form matters when the ablated skeleton already produces some metric, as the lab's empty-circuit baseline shows.

Completeness asks whether the complement is as unimportant as claimed. The direct test knocks the circuit out of the full model and checks that the behavior collapses, and knocks out components from the complement and checks that the behavior does not move. The IOI paper's incompleteness score sharpens this: over random subsets K of the proposed circuit, compare the effect of removing K from the full model against removing K from the circuit; if the circuit is complete these should match, and the worst-case discrepancy over K is the score. A circuit can be highly faithful yet incomplete: sufficient pathways can coexist with important components the search missed, especially backup components that activate only under ablation.

Minimality asks whether every element earns its place: for each node or edge in C, is there a measurable drop from removing it (in some context, possibly in combination)? Without minimality, faithfulness is trivially gameable by inflation: the full model is a perfectly faithful "circuit". Reported circuits should come with all three numbers and the ablation scheme under which they were computed, which brings us to the load-bearing detail.

6.6 Ablation semantics decide the answer#

"Ablate everything outside the circuit" hides the most consequential design choice in the pipeline: what value replaces an ablated activation. Zero ablation writes zeros. It is simple and reference-free, but it is a strong off-distribution intervention: no head outputs the zero vector on real text, so zeroing shifts the residual stream's mean and norm, disturbs downstream LayerNorm denominators, and perturbs every downstream component in ways unrelated to the task. Mean ablation replaces the activation with its mean over a reference distribution, typically computed per position over prompts from the task template. It removes the prompt-specific information a component carries while keeping its output at a typical operating point. Resample ablation goes one step further: replace the activation with the same component's activation on a different prompt drawn from the task distribution. Every replacing value is one the component actually produced, so nothing is off-distribution; what is removed is precisely the information that varies across the task distribution.

These choices are not interchangeable, and conclusions flip between them. A circuit that retains 70 percent of behavior under mean ablation can score below zero under zero ablation of the same complement, not because the circuit changed but because the zeroed complement corrupts the circuit's own inputs. The lab demonstrates exactly this flip on the IOI top-8 circuit. The direction of the flip is not even consistent: zero ablation can also inflate a score when the zeroed components were suppressing the behavior. Mean ablation has its own subtlety, the choice of reference distribution: means over the task template preserve template-level structure (positions of names, syntax), while means over generic text remove it, and the two test different hypotheses. Resample ablation is generally the most defensible default for task-scoped claims, at the cost of variance from the resampling draw. Whatever the choice, a faithfulness number reported without its ablation semantics and reference distribution is uninterpretable, and comparisons across methods are valid only at fixed semantics.

6.7 Ground truth, benchmarks, and critiques#

Evaluating discovery methods requires tasks where the circuit is already known with reasonable confidence. Three serve as standard ground truth: IOI in GPT-2 small, the 26-head name-tracking circuit; Greater-Than in GPT-2 small, predicting that a year like "1742" in "from 1732 to 17.." must end above 32, carried by a small set of heads and mid-layer MLPs; and Docstring in a 4-layer attention-only model, predicting Python docstring argument names. Benchmark suites assemble such tasks with annotated edges and score discovery methods by how well their rankings retrieve the annotated circuit; later suites add semi-synthetic models trained to contain known circuits by construction, removing the circularity of "ground truth" that was itself produced by interpretability methods.

The critiques are serious and you should internalize them before trusting any discovered circuit, including your own. First, faithfulness is gameable. It is a single scalar over a narrow distribution under one ablation scheme; circuits can score well by exploiting the ablation (for instance, when mean-ablated components leak template structure that does the work) rather than by capturing mechanism, and different faithfulness definitions rank the same candidate circuits differently. Second, instability: rerunning discovery across corruption seeds, metric choices, or thresholds yields visibly different circuits for the same task and model, with overlap sometimes far below what the confident presentation of a single circuit diagram suggests. Sweeping k or τ and reporting curves, as the lab does, is the minimum honest practice. Third, the field has moved toward explicit hypothesis testing: rather than reporting a faithfulness score, state the circuit hypothesis as a formal claim (equivalence of the circuit and model on the task distribution, independence of the complement, minimality of each part) and test each claim with controls, such as comparing the discovered circuit against size-matched random circuits. Known circuits pass such tests only partially, which is evidence about the circuits, the tests, and the underlying claim that discrete subgraph mechanisms exist at all. Hold all three possibilities open.

7.1 Two different questions#

Every method so far in this course has asked some version of one question: which components cause a behavior? Activation patching (Chapter 4), attribution patching (Chapter 5), and automated circuit discovery (Chapter 6) all intervene on the computation and watch the output move. This chapter starts from a different question: what information is present in an activation at all? The two questions come apart in both directions, and confusing them is one of the most common errors in interpretability practice.

Call the first the representation question: given the residual stream at layer 8 above some token, can the grammatical number of the sentence's subject be decoded from it? Call the second the use question: does the model's own downstream computation rely on that information, encoded that way, when it predicts the verb? A property can be decodable but ignored. The stream is a 768-dimensional vector carrying everything the model might need; plenty of linearly present information is never read by any downstream head or MLP, in the same way a component can write a large vector into the stream that nothing downstream consumes (Chapter 1). Conversely, a property can be used but hard for your probe to see, if it is encoded nonlinearly or distributed in a basis your probe family cannot express.

Probing answers the representation question. It is cheap, differentiable-free at test time, and scales to any property you can label. But it is correlational, exactly as residual stream tracing was in Chapter 1, and it must be paired with causal checks before you conclude anything about mechanism. The chapter builds up the probing toolkit, its main confound and the control for it, then two ways to make probing causal: amnesic-style direction removal and subnetwork probing.

7.2 Linear probes and layerwise probing curves#

A linear probe is logistic regression on frozen activations. Collect activations x in R^d_model at a chosen hook point and position for a labeled dataset, freeze them, and train a weight vector w and bias b to minimize binary cross entropy on the prediction σ(w·x + b). Nothing about the model changes; the probe is a measurement instrument. Accuracy is evaluated on a held-out split, and the split must respect the structure of the data: if the same lexical item appears in train and test, the probe can succeed by memorizing vocabulary rather than reading the property, so hold out item types, not just examples.

Training the same probe at every layer produces a probing curve, accuracy as a function of depth, and the shape of the curve is the finding. For a property that requires contextual computation, the curve starts near chance at the embedding and rises through the layers where the model computes and routes the property. Where the curve jumps is a hypothesis about where the computation happens. For a lexical property probed at the word's own position, the curve starts near ceiling at layer 0, because the embedding itself determines the answer; such a curve tells you nothing about the model's computation, only about the tokenizer. The lab exploits this contrast deliberately: it probes the subject's grammatical number at a fixed final token that is identical across classes, so any decodable signal must have been moved there by attention from the subject position, and the probing curve measures that routing.

One design choice deserves emphasis: the readout position. Probing "the model's representation of X" is ill-posed until you say where. The stream above the subject noun, the stream above the final token, and the stream averaged over positions are three different measurements with three different meanings. Choose the position that the downstream behavior actually reads. If you care about verb agreement, probe where the verb will be predicted.

7.3 The expressivity confound and control tasks#

Here is the confound that nearly sank early probing literature. Suppose you use a two-layer multilayer perceptron (MLP) probe instead of a linear one, and accuracy rises from 85 to 97 percent. Did you discover that the property is represented nonlinearly? Or did you hand a more powerful function approximator enough capacity to carve arbitrary structure out of a rich, high-dimensional input? A sufficiently expressive probe can reach high accuracy on activations even when the property is not represented in any meaningful sense, because the activations retain enough information about the raw input to let the probe recompute the property itself. At the extreme, a powerful probe on the embedding layer can learn the task from scratch, using the activations as little more than a fancy input encoding.

The control-task methodology resolves this by measuring what the probe's capacity alone can achieve. Construct a control task with the same form as the real task but with labels that carry no linguistic content: assign each word type a random label, held fixed across all its occurrences. Train an identical probe, same architecture, same capacity, same training budget, on the control labels. Any held-out accuracy the probe achieves on the control task is manufactured by probe capacity plus memorization of type identity, since the labels are random. Define selectivity as real-task accuracy minus control-task accuracy. A high-accuracy, high-selectivity probe is reading a real representation. A high-accuracy, low-selectivity probe is doing the task itself, and its accuracy says little about the model.

Selectivity reframes probe design: you want the least expressive probe that can read the property, not the most accurate one. Linear probes are the standard not because they win benchmarks but because their low capacity keeps control accuracy near chance, making selectivity interpretable. This is also why the subfield tolerates the awkwardness that "linearly decodable" is a probe-family-relative notion: relative to the linear family, the control task pins down how much accuracy is real.

7.4 The linear representation hypothesis and its scope#

Why expect linear probes to work at all? The working ontology of this course, directions in activation space carry meaning, has a strong form: the linear representation hypothesis, the claim that models encode high-level features as directions, with the feature's value given by the projection of the activation onto the direction. Several independent lines of evidence support it. Linear probes succeed at high selectivity across a wide range of syntactic and semantic properties. Arithmetic on directions works: adding a direction to the stream shifts behavior in the corresponding way, which is the entire basis of steering vectors in Chapter 8. Sparse autoencoders (Chapter 10) recover directions whose activations align with human-interpretable features. And the architecture itself gives a reason: every component reads the stream through a linear map before its nonlinearity, so features that downstream components consume must be accessible to linear reads, which pressures the model to store them linearly.

The hypothesis has scope limits you should keep loaded. Some features appear to be encoded in more than one dimension with meaningful geometry, such as circular arrangements for periodic quantities like days of the week. Features in superposition share directions with interference, so a single probe direction may mix several features. Magnitude and direction can carry separate information. And the hypothesis is about features, not computations: how features are transformed between layers is not a linear story. Treat linearity as a strong default prior, verified per feature, not as a law.

7.5 From decodable to used: amnesic-style checks#

A selective probe establishes representation. To establish use, intervene. The amnesic probing idea: identify the subspace the probe reads, remove it from the activation during a forward pass, and measure whether the behavior that supposedly depends on the property degrades. Removing a single direction d (unit norm) from an activation x is a projection: x' = x minus (x·d)d. Run the model with a hook that applies this projection at the probed layer and compare a behavioral metric before and after. If the model's verb-agreement preference collapses when the number direction is removed, and does not move when a random direction of equal norm is removed, you have causal evidence that the model reads that direction, not merely that a probe can.

Both controls matter. The random-direction control guards against the objection that any intervention on the stream damages behavior; removing one random direction out of 768 should be nearly free, and if it is not, your metric is fragile. The magnitude of the drop matters too: behavior rarely collapses to chance, because information is redundantly encoded and later layers partially rebuild what you removed, the same self-repair phenomenon that complicates patching in Chapter 4. Iterative variants remove direction after direction until the probe can no longer decode the property, giving a cleaner "amnesia," at the cost of removing a growing subspace whose side effects are harder to bound. Note the family resemblance to other interventions in the course: amnesic removal is a rank-one ablation in activation space, mean ablation (Chapter 6) is removal of a component's information content, and steering (Chapter 8) is the constructive inverse, adding a direction instead of deleting it. All three are bets on the linear representation hypothesis.

7.6 Subnetwork probing: masks instead of readouts#

Probing so far reads activations. Subnetwork probing asks a component-level question instead: which small subset of the model's components is sufficient, by itself, to perform a behavior? Formally, attach a binary gate m_i in {0, 1} to each component (attention heads, or individual weights in the finest-grained variant), run the model with each component's output multiplied by its gate, and search for a mask that preserves task performance while an L0 penalty, the count of open gates, drives the mask sparse. The surviving components are the discovered subnetwork.

The obstacle is that binary masks are not differentiable. The standard fix is the hard-concrete relaxation: each gate is a stretched, clamped sample from a concrete (relaxed Bernoulli) distribution controlled by a learnable parameter, so gates take values exactly 0 or exactly 1 with nonzero probability while remaining differentiable in expectation, and the expected L0 norm has a closed form that can be penalized directly. The lab uses a simpler relaxation to fit the compute budget, a plain sigmoid gate per head with an L1 penalty on total mask mass, trained with a hinge objective that preserves the indirect object identification (IOI) logit difference from Chapter 4. The training dynamic is the instructive part: once the masked model matches clean performance, the hinge contributes zero gradient and only the sparsity pressure remains, so every head not needed for the behavior is pushed toward zero, and the heads that resist are the ones the behavior needs. After training, binarize at 0.5 and evaluate the binary subnetwork. Expect a gap: the soft mask was optimized with partially open gates, and snapping to 0/1 moves the network off the optimum. The size of that gap is precisely what the hard-concrete machinery exists to reduce, by training with near-binary samples so binarization at the end is a small step.

Contrast this with the discovery methods of Chapters 5 and 6. Automated circuit discovery (ACDC) greedily prunes edges one at a time, testing each removal; edge attribution patching (EAP) scores all edges in a constant number of passes using gradients. Both are search or scoring procedures over a fixed graph. Subnetwork probing is continuous optimization over the whole mask jointly, so it can find component sets whose members matter only together, a combination greedy pruning can miss; the price is a nonconvex optimization with its own hyperparameters (penalty weight, learning rate, initialization) whose failure modes look like science. On circuit benchmarks the three methods recover substantially overlapping but not identical subnetworks, and head-to-head evaluations have at times ranked subnetwork probing and EAP above ACDC on ground-truth circuits. No method dominates; agreement among them is the strong signal.

7.7 Probing and patching as complementary evidence#

Probing and patching measure different things, and their combination is more informative than either alone. Probing (with controls) tells you whether a property is represented at a location. Patching or amnesic removal tells you whether the location or direction is used for a behavior. Crossing the two answers gives four cells, and each cell is a distinct scientific situation.

The off-diagonal cells are where methodology earns its keep. Represented-but-unused findings are the standing rebuttal to probing papers that stop at accuracy: without an intervention, the top row cannot be told apart. Used-but-not-represented findings are the rebuttal to overconfident negative claims: a chance-level probe does not show the information is absent, only that your probe family cannot see it. When you read an interpretability paper, placing each experiment into this table is a fast way to audit whether its causal language is licensed by its evidence.

8.1 From reading to writing#

Chapter 7 established that many properties of the input are linearly readable from the residual stream: a probe vector w exists such that w^Tx predicts the property. It also established the gap between represented and used. This chapter closes the loop from the other side. If a behavior is associated with a direction, then adding that direction back into the stream should shift the behavior. Where probing reads representations, activation engineering writes them.

The logic rests on two facts you have already verified. First, the residual stream is additive: every component contributes by vector addition, so an externally added vector is architecturally indistinguishable from one more component writing to the stream. Downstream layers process it exactly as they process any other contribution. Second, under the linear representation hypothesis, behaviorally relevant features are directions, so a well-chosen constant vector can represent "be positive" or "perform the antonym task" the same way the model's own components would. Steering is therefore both a control technique and an experiment: if adding direction v causes behavior B, that is causal evidence that v encodes something the model treats as B, evidence of a kind that probing alone can never provide.

8.2 Activation addition from a contrastive pair#

The simplest recipe, activation addition (ActAdd), needs one pair of prompts that differ in the property you want to control. Run the model on a positive prompt p⁺ ("The movie was wonderful") and a negative prompt p^- ("The movie was terrible"), cache the residual stream at a chosen layer l and position, and take the difference:

During generation on a new prompt, add c · v to the residual stream at layer l, where c is the steering coefficient. The subtraction is what makes this work. Each activation individually is dominated by content that has nothing to do with sentiment: token identity, position, syntax, the shared words of the template. Because the two prompts share everything except the contrastive word, the difference cancels the shared content and leaves, approximately, the direction along which the model separates the two conditions. The same cancellation argument justified logit differences in Chapter 2 and contrast pairs in patching in Chapter 4; steering reuses it in activation space.

A single pair cancels only what the two prompts share. Everything idiosyncratic to that template survives into v: the topic (movies), the register, the specific tokens chosen. Contrastive activation addition (CAA) fixes this by averaging over a dataset of N contrastive pairs built from varied templates:

Idiosyncratic components point in different directions across pairs and shrink under averaging; the shared sentiment component points the same way in every pair and survives. Even eight pairs help noticeably, and published CAA work uses hundreds. CAA implementations typically extract at one middle-to-late layer, at the final token position of each prompt (where the contrastive content has been integrated) or averaged over positions, and the resulting vector is often normalized so that the coefficient has a consistent meaning across layers and datasets. In the lab you will normalize v to unit norm, which makes the coefficient an absolute norm in residual units and comparable against the stream norm at the injection layer.

8.3 Design choices: layer, positions, coefficient#

Three knobs determine whether steering works, and each has a mechanistic rationale rather than being a free hyperparameter.

Layer. Middle layers usually steer best, and the failure modes on either side are instructive. Inject too early and the vector is one small addend at the bottom of a deep stack; a dozen layers of attention and MLPs process, dilute, and effectively overwrite it before it reaches the unembedding, and early streams also encode mostly token-level rather than concept-level structure, so the direction may not yet mean anything. Inject too late and the opposite problem appears: the vector skips the downstream processing that would integrate it into the generation. Adding a sentiment direction at the last layer nudges next-token logits toward a few sentiment-loaded tokens, but no later component can propagate the shift into coherent continuation. Middle layers hit the compromise: abstract enough to carry the concept, early enough that remaining layers elaborate it. In GPT-2 small this means roughly layers 5 through 9, and a layer sweep is cheap enough that you should always run one.

Positions. The vector can be added at every token position, at only the final position, or only at newly generated positions. Adding at all positions gives the strongest and most persistent effect, since every step of generation sees the shifted context, and it is the common default for CAA. Adding only at the final position steers the immediate next-token distribution but fades as generation proceeds. All-position injection also perturbs the model's reading of the existing context, which is part of why large coefficients derail fluency.

Coefficient. The scale c controls a tradeoff you will measure directly in the lab: the steering/coherence tradeoff. Small coefficients shift behavior gently; large ones push the stream far off the data manifold the model was trained on, and fluency collapses into repetition or incoherent text. The relevant comparison is the norm of the addition against the norm of the stream at the injection layer. In GPT-2 small the residual norm at a middle layer is on the order of 100 in raw units, so adding a unit-norm vector with coefficient 8 is a few percent perturbation, usually enough to shift sentiment while leaving perplexity on neutral text nearly untouched. Coefficients approaching the stream norm produce word salad. Negative coefficients steer toward the opposite pole, which is a useful symmetry check: a real sentiment direction should make text more negative when subtracted.

8.4 Function vectors: steering with a task, not a trait#

Steering is not limited to traits like sentiment. Transformers running in-context learning (ICL), the ability to induce a task from examples in the prompt, form compact internal representations of the demonstrated task, and those representations can be extracted and reinjected. Given prompts of the form "hot -> cold, big -> small, fast -> slow, ..., wet ->", a specific set of mid-layer attention heads writes a representation of the antonym task into the residual stream at the positions where the answer is due. Averaging those head outputs (or, in the simplified form used in the lab, the residual state at the arrow positions) over many examples yields a function vector: a single activation that encodes "produce the antonym".

The striking property is zero-shot transfer. Inject the function vector into a forward pass on the bare prompt "wet ->", with no examples at all, at a middle layer, and the probability of " cold" rises sharply; the model behaves as if the demonstrations were present. Averaging plays the same role as in CAA: any single arrow position carries both the task and the content of its example word, and averaging over eight examples cancels the content while preserving the shared task component. Function vectors show that ICL partially factors into an explicit task variable carried in the stream, which is a claim about mechanism, not just a control trick, and they transfer with some success across prompt formats and even across models of the same family.

8.5 Case study: refusal as a single direction#

The most consequential steering result to date concerns refusal, the behavior in which a chat model declines a request. Across many open-weight chat models, a single direction in the residual stream mediates refusal. The direction is found with exactly the CAA recipe: average the residual difference between harmful and harmless instructions at a middle layer, over a dataset of each. Two interventions then provide converging causal evidence. Ablating the direction, meaning projecting it out of the stream at every layer, sharply reduces refusal of requests the model would normally decline. Adding the direction to the stream causes the model to refuse innocuous requests.

The scientific content of this result is worth separating from its security implications, which this course does not develop. As science, it is the strongest known instance of the linear representation hypothesis holding for a high-level behavior: a safety-relevant, reinforcement-trained behavior compressed into one direction of a 4096-dimensional stream, discovered with a difference of means. It also illustrates the dual character of interpretability results: the same finding that explains a mechanism also exposes how brittle a safety behavior built on a single direction is. Behaviors trained by fine-tuning may occupy far less of the representation space than their importance suggests, and defenses that assume otherwise inherit that fragility.

8.6 Evaluating steering: shift versus damage#

A steering claim has two parts: the behavior moved, and the model still works. Each part needs its own metric, and reporting only the first is the most common way steering results overstate themselves.

Behavioral shift. The cheapest reliable metric is a probability readout: fix a set of target tokens and measure how their next-token probability moves under steering. For sentiment, sum the probability of a small positive word set minus a negative word set at a fixed position of a neutral prompt, and sweep the coefficient; a real steering vector produces a monotone dose-response curve, and the negative-coefficient side should move symmetrically. For open-ended generation, score continuations with a sentiment classifier or keyword counts over a prompt set. Probability readouts are deterministic and cheap; classifier scoring is closer to what a user experiences but noisier.

Capability damage. Measure the model's quality on text unrelated to the steered behavior, with the steering active. The standard instrument is perplexity (or mean log probability) of a fixed neutral continuation, computed with and without the vector. A well-scaled steering intervention leaves neutral log probability nearly unchanged while moving the behavioral metric; a badly scaled one buys behavioral shift by damaging everything. Plotting behavioral shift against capability damage across coefficients gives a frontier, and honest comparisons between steering methods compare frontiers, not single points.

The known limits deserve equal billing. Steering is brittle across prompts: a vector tuned on one template family can fail on another phrasing of the same idea. It is coefficient sensitive, with the useful range varying by layer, dataset, and model, so published coefficients rarely transfer. And steering vectors are entangled: a "sentiment" vector built from restaurant and movie reviews carries topic, register, and lexical correlates of its training pairs, so steering positive may also steer toward review-like text. These are the same confound problems probing faced in Chapter 7, now appearing on the write side, and the same remedy applies: varied contrast sets, control evaluations, and suspicion of any result demonstrated on a single prompt.

8.7 Steering in context#

Steering completes a progression that the course has been building. Probing (Chapter 7) shows a direction correlates with a property. Patching (Chapter 4) shows an activation carries causal weight on one prompt. Steering shows a direction suffices to induce behavior across prompts, which is the strongest single-direction causal claim available without touching weights. Chapter 9 takes the final step, editing the weights themselves, and inherits every evaluation lesson from this chapter: measure the intended change, measure the collateral damage, and expect both.

9.1 MLPs as key-value memories#

Chapter 1 treated the MLP as a wide reader and writer of the residual stream. This chapter needs the finer-grained view introduced by Geva et al.: an MLP layer is a key-value memory. Write the block as

with x in R⁷⁶⁸, W_in mapping 768 to 3072, σ the GELU nonlinearity, and W_out mapping 3072 back to 768. Read this in two halves. Each of the 3072 hidden units owns one input-side weight vector, a column of W_in: a direction in the residual stream that the unit detects. These are the keys. Each unit also owns one output-side weight vector, the corresponding row of W_out (in TransformerLens shape conventions, W_out is [d_mlp, d_model], so unit i owns row i): a direction the unit writes into the stream when active. These are the values. The GELU sits between them as a soft gate: a unit whose key matches the current residual state produces a large pre-activation, passes through the gate, and contributes its value scaled by the activation. The MLP output is a sparse-ish weighted sum of value vectors, selected by key matches.

Under this reading, factual recall has a natural mechanism. When the residual stream at the subject's position encodes "Eiffel Tower", some set of keys in mid-layer MLPs match, and the corresponding values write attribute information (is-a-tower, located-in-Paris, built-in-1889) into the stream, where later attention heads can fetch it to the position that needs it. Geva et al. supported this by showing that individual keys respond to interpretable input patterns and that values, projected through the unembedding, often favor tokens completing those patterns. The picture is an approximation: units are polysemantic (Chapter 10 explains why), and recall is distributed over many units and several layers. But it is accurate enough to support a working technology, which is what the rest of this chapter builds.

9.2 Causal tracing: where is a fact recalled?#

To edit a fact you must first choose where to intervene. Causal tracing, introduced alongside ROME, is activation patching (Chapter 4) specialized to factual recall. Run the model on a clean prompt such as "The Eiffel Tower is located in the city of" and record all activations. Run it again with Gaussian noise added to the embeddings of the subject tokens ("The Eiffel Tower"), destroying the model's knowledge of what the prompt is about; the probability of " Paris" collapses. Then, in a third family of runs, keep the noised embeddings but restore a single clean activation, one (layer, position) site at a time, and measure how much of the clean-minus-corrupted probability gap that single restoration recovers. Sites whose restoration recovers a large fraction of the gap are causally implicated in carrying the fact.

On large models, this procedure produced a now-famous result: restoring hidden states at the subject's last token in early-to-middle layers recovers most of the effect, and severing attention while restoring MLPs (but not the reverse) preserves the recovery. The interpretation offered was that mid-layer MLPs at the subject's last token perform the recall, and late attention moves the answer to the output position. Two cautions apply. First, this is a denoising experiment (Chapter 4): it shows restored sites are sufficient to recover the behavior on top of a specific corruption, not that they are the unique storage location. Second, the localization is model-dependent and coarser in small models. In the lab you will run tracing on GPT-2 small and find that the strongest single-layer MLP restoration site is the very first layer, whose MLP acts as an extended embedding, rather than a clean mid-layer bump. Keep that result in mind when Section 9.6 discusses whether tracing tells you where to edit.

9.3 The ROME update, derived#

ROME (Rank-One Model Editing) treats the output matrix of one chosen MLP layer as a linear associative memory. Abstract the layer as W k ≈ v: keys k are post-GELU hidden activations (3072-dimensional), values v are the MLP outputs written to the stream (768-dimensional), and W is the (transposed) W_out. If W were trained by least squares to store many pairs (k_i, v_i), it satisfies the normal equations W C = V K^T, where C = E[k k^T] is the second-moment (covariance) matrix of keys. We want to insert a new association (k*, v*), meaning the constraint W' k* = v*, while disturbing the stored associations as little as possible. Formally: minimize ||W' - W|| subject to W' k* = v*, where the norm weights directions by how much key mass C places on them. The solution is a rank-one update:

Verify the constraint by multiplying: W' k* = W k* + (v* - W k*) · 1 = v*. The correction is the outer product of the residual error (v* - W k*) with the direction C^-1 k*. The C^-1 factor is the important subtlety. Keys that occur frequently in ordinary text dominate C, so C^-1 shrinks the update along common key directions and lets it act mostly along directions where k* is distinctive. That is what "minimal disturbance of existing associations" means operationally. In practice C is estimated by averaging k k^T over tens of thousands of Wikipedia tokens. If you approximate C by the identity, as the lab does for speed, the update becomes plain least-squares insertion, W' = W + (v* - W k*) k*^T / (k*^T k*): still exact on k*, but no longer steered away from common key directions, so you should expect more collateral change on other inputs. The lab measures exactly that.

Two quantities remain to be chosen. The key k* should be the activation the model actually produces when reading the subject, so ROME averages the post-GELU MLP activation at the subject's last token over the edit prompt embedded in several sampled contexts; averaging makes the key robust to surrounding text and directly serves generalization. The value v* is not read off anywhere; it is optimized. Holding the model fixed, treat the chosen MLP's output at the subject's last token as a free vector and run gradient descent on it so that the model's final-position prediction becomes the new target ("Rome"), with a regularizer keeping v* close to the original output so the subject's other properties survive. Around 20 to 30 optimizer steps suffice. Then apply the rank-one formula. Note what was and was not learned: no model weights were trained; a single closed-form update was applied after a small optimization over one 768-dimensional vector.

9.4 Evaluating an edit#

A single number cannot certify an edit, because the failure modes trade off against one another. The ROME evaluation protocol uses four axes, and you should internalize them as the standard for any editing claim. Efficacy: on the edit prompt itself, does the model now produce the new object? Trivial to satisfy in isolation; a lookup table patched onto the output would pass. Generalization: does the edit hold on paraphrases ("Where is the Eiffel Tower? It is in the city of")? This is what separates editing the fact from memorizing the prompt string. Specificity: are neighboring facts unchanged? The neighborhood is deliberately adversarial: prompts about similar subjects ("The Louvre is located in the city of") that share vocabulary and structure with the edit prompt. An edit that drags the Louvre to Rome has overwritten a region of key space, not one association. Fluency and consistency: does generation from edited prompts remain non-degenerate (measured by n-gram entropy) and on-topic (measured by similarity to reference text about the new object)? Aggressive updates can satisfy the first three axes while collapsing generation into repetition.

Fine-tuning on the single new fact tends to ace efficacy, damage specificity through drift, and generalize erratically. Constrained fine-tuning protects specificity but undershoots generalization. ROME's headline result was strong performance on all axes at once on GPT-2 XL, which is evidence for, though not proof of, the claim that it intervenes near where the association is computed.

9.5 MEMIT: many edits, many layers#

ROME edits one fact in one layer. Asking it to edit thousands of facts sequentially degrades the model: each rank-one update slightly disturbs the key space the next update depends on, errors compound, and cramming every new association into one matrix concentrates a large total weight change at a single site. MEMIT (Mass-Editing Memory in a Transformer) makes two changes. First, it spreads each edit over a contiguous range of mid layers (the causal-tracing window) rather than one layer, assigning each layer a fraction of the residual-stream change needed; each layer's update is smaller, and the total perturbation is distributed. Second, it solves for all edits jointly in closed form: with thousands of target pairs (k_i*, v_i*), each layer receives a batched least-squares update generalizing the rank-one formula, again with C^-1 protecting existing keys. MEMIT holds efficacy, generalization, and specificity roughly flat out to about 10,000 edits on GPT-J, where sequential ROME and fine-tuning baselines have long since collapsed. The scaling result matters for practice, and it also sharpens the conceptual point that factual associations are distributed across several layers rather than resident in one.

9.6 What editing results do not show#

Three critiques define the current understanding of editing, and each is a general lesson in interpretability methodology.

Ripple effects. A fact is not an isolated triple; it entails other facts. After editing "the Eiffel Tower is in Rome", a consistent model should answer "Italy" for the tower's country and adjust everything downstream of location. Cohen et al. built a benchmark of such entailed questions and found that ROME and MEMIT mostly fail them: the edited triple changes, its logical neighborhood does not. The model now asserts a fact whose consequences it does not believe. This is the editing-versus-knowing distinction: the methods rewrite an association, not the web of inferences that constitutes knowing. Any deployment that needs consistency under reasoning cannot rely on association-level edits alone.

Localization does not dictate editability. Since causal tracing motivated editing mid-layer MLPs, one would predict that edit success tracks tracing results layer by layer. Hase et al. tested this directly and found it false: ROME-style edits succeed about equally well at many layers, including layers where tracing shows no restoration effect, and tracing peaks correlate near zero with editing success. Your own lab results will instantiate this: tracing on GPT-2 small points at layer 0, yet the mid-layer edit works. The resolution is that tracing answers "where does restored clean information suffice to recover the output" while editing answers "where can a targeted perturbation redirect the output", and these are different causal questions about the same network. Sufficiency-style localization does not imply that intervention elsewhere fails.

Bleedover and paraphrase fragility. Specificity scores measure a small neighborhood; broader probes show edits leaking into unrelated completions that share surface features with the edit prompt, and edits weakening or vanishing under aggressive paraphrase or when the fact is elicited in another language or format. Efficacy on one template is weak evidence about the model's behavior distribution.

These critiques position editing among its alternatives. Fine-tuning on curated data updates knowledge with consequences intact, at the cost of compute and the risk of broad drift. Retrieval-augmented generation sidesteps weight surgery entirely by placing current facts in context, with perfect revocability but no change to the model's parametric beliefs. Rank-one editing is fast, local, and reversible (store the outer product and subtract it), which makes it an excellent experimental probe of where associations live, and a risky production tool. For this course, the experimental-probe role is the important one: editing is the strongest available causal test of the key-value memory hypothesis, and its partial failures are measurements of where that hypothesis stops being true.

10.1 Polysemanticity: why neurons are the wrong unit#

The natural first move in interpreting an MLP is to ask what each neuron does: find the inputs that maximally activate neuron 2378, read them, and name the neuron. For a minority of neurons in language models this works. For most it fails in a characteristic way: the top activating inputs form several unrelated clusters. A single GPT-2 neuron might respond to academic citations, the token "though" in contrastive clauses, and Korean text. This is polysemanticity, and it is the rule rather than the exception. It is not an artifact of bad luck in training. There is a structural reason to expect it, and understanding that reason tells you what to do about it.

The reason is counting. A model the size of GPT-2 small plausibly tracks far more properties of its input than it has dimensions: tens of thousands of tokens, plus syntax, entities, topics, sentiment, languages, formatting, and countless conjunctions of these. The residual stream at any layer offers 768 dimensions, an MLP layer offers 3072 neurons. If the model wants more features than dimensions, and it does, then features cannot each get a private orthogonal direction, let alone a private neuron. Something has to share.

10.2 The superposition hypothesis#

The superposition hypothesis says the sharing is systematic: the model represents n features in a d-dimensional space with n much larger than d by assigning each feature a direction, with the directions nearly but not exactly orthogonal. Reading feature i by projecting onto its direction then picks up small amounts of every other active feature, an error called interference. The scheme works because of two facts. First, high-dimensional geometry is permissive: while only d directions can be exactly orthogonal, exponentially many can be pairwise nearly orthogonal, so per-pair interference can be kept small. Second, and critically, natural features are sparse. Most features are absent from most inputs. Interference is only incurred between features that are simultaneously active, so if few features are active at once, total interference stays small and the compression is worth it. Superposition is a bet the model places on sparsity.

Toy models make this precise. Train a small autoencoder to reconstruct n sparse features through a bottleneck of d < n dimensions, with reconstruction error weighted by per-feature importance. The result is a phase diagram in importance and sparsity. When features are dense (usually active), the model represents only the d most important features, orthogonally, and drops the rest. As sparsity increases, there is a phase transition: the model starts packing extra features in, accepting interference in exchange for coverage. At high sparsity the learned directions arrange into striking geometric configurations, antipodal pairs, pentagons of five directions in two dimensions, tetrahedra, that spread interference as evenly as possible. The lab reproduces the pentagon: five features, two dimensions, and a learned arrangement in which every direction has large negative inner product with two others. Interference in these toy models is not noise; it is a negotiated compromise with visible structure.

For interpretability the hypothesis has a sharp consequence. If features are directions and directions are shared across neurons, then the neuron basis is simply the wrong coordinate system, and no amount of staring at neurons will yield clean concepts. What you want is a change of basis into the feature directions. Since the model does not hand you those directions, you must learn them, and sparsity, the same property that makes superposition viable, is the handle that makes them learnable.

10.3 Sparse autoencoders: dictionary learning on activations#

A sparse autoencoder (SAE) is sparse dictionary learning applied to a model's activations. Fix a hook point, typically the residual stream after one layer, and collect activation vectors x in R^d over a large token corpus. The SAE posits that each x is approximately a nonnegative sparse combination of dictionary directions, and learns both the dictionary and the code:

Here f in R^m is the latent vector with m = (expansion factor) times d, commonly 8x to 64x. Each row of W_dec is one dictionary direction, constrained to unit norm so that the magnitude of a feature lives in the latent f rather than in the direction. Subtracting b_dec before encoding centers the data around the decoder's bias, which is a small but consistently helpful detail. The training loss is

reconstruction error plus an L1 penalty on the latents. Why L1 and not L0, which is what we actually want? L0 is not differentiable, and L1 is its standard convex surrogate. The mechanics of why L1 produces exact zeros matter: the gradient of λ|f_i| with respect to a positive latent is the constant λ, regardless of how small f_i is. A latent whose contribution to reconstruction is worth less than λ is pushed all the way to zero, where the ReLU keeps it. A squared (L2) penalty instead has gradient proportional to f_i, which vanishes near zero and merely makes latents small, never zero.

The same constant pull causes the known defect of L1 training, shrinkage: latents that should be active are systematically smaller than the value that best reconstructs x, because at the optimum the reconstruction gradient must balance the constant λ. Every active feature is underestimated by an amount that does not vanish with training. Shrinkage degrades reconstruction at a given sparsity and biases downstream measurements of feature magnitude.

Two architectural variants attack sparsity without L1. TopK SAEs replace the penalty with a hard constraint: after the encoder's linear map, keep only the k largest latents and zero the rest. L0 is then exactly k by construction, there is no shrinkage on the surviving latents, and the sparsity level becomes an architectural dial rather than an emergent property of a tuned λ. The cost is a fixed per-token budget: every token gets exactly k features whether it needs 3 or 40. JumpReLU SAEs instead learn a per-latent threshold θ_i: the activation function passes f_i unchanged if it exceeds θ_i and outputs zero otherwise. Training penalizes L0 directly, and since both the threshold step and L0 have zero gradient almost everywhere, gradients are estimated with straight-through estimators, which pretend on the backward pass that the step function has a finite-width slope. JumpReLU keeps variable per-token sparsity, avoids shrinkage above threshold, and currently sits at or near the best reconstruction-sparsity frontier, at the price of a more delicate training setup.

10.4 The training recipe#

Which activations to train on is the first decision. The common choice is the residual stream after one layer (resid_post), because the stream is the model's communication channel and a dictionary there captures what all upstream components have written. MLP outputs and attention outputs are also used when the question concerns a specific component. Activations are gathered over a large corpus, hundreds of millions to billions of tokens at research scale, because the dictionary can only learn features the data exhibits, many features are rare, and each latent needs many examples.

Because activations from neighboring tokens of the same document are highly correlated, feeding documents in order gives correlated minibatches and noisy training. Production pipelines therefore run the model, dump activations into a large buffer, shuffle the buffer, and draw training minibatches from it, refilling as it drains. After each optimizer step the decoder rows are renormalized to unit norm; without this the model games the L1 penalty by shrinking latents while growing decoder rows, making the penalty meaningless.

The characteristic pathology of SAE training is dead features: latents that stop activating on any input. Once dead, a latent receives no gradient through the ReLU and cannot recover; at scale, uncared-for training can kill a third of the dictionary, wasting capacity. Two mitigations are standard. Resampling periodically finds dead latents and reinitializes them toward inputs the SAE currently reconstructs worst, pointing spare capacity at the residual error. Auxiliary losses such as ghost grads instead add a term in which dead latents are forced to help predict the current reconstruction residual, giving them a gradient path back to life without disturbing live features.

10.5 Evaluation: fidelity, sparsity, interpretability#

An SAE is evaluated on three axes, and reporting only one is a red flag. Sparsity is measured by L0, the mean number of active latents per token; useful dictionaries land roughly between 10 and 100. Fidelity has a weak and a strong form. The weak form is variance explained, 1 − MSE divided by the variance of the activations. The strong form asks what the model thinks of the reconstruction: run the model, replace the activation at the hook point with the SAE's reconstruction, and measure the CE loss on next-token prediction. Compare three numbers: clean loss, spliced loss, and the loss with the activation destroyed entirely (zero or mean ablation). The fraction of the gap between destroyed and clean that splicing closes is loss recovered. This is the metric that matters, because it prices reconstruction error in the units the model cares about. A reconstruction can have 95 percent variance explained and still lose meaningful CE, which tells you the missing 5 percent was not noise.

Interpretability is assessed per feature. A feature dashboard collects the feature's maximum activating examples over a corpus, its activation histogram, and its logit weights: project the feature's decoder direction through the unembedding W_U to see which output tokens the feature promotes and suppresses. A feature that fires on golden-gate-bridge contexts and promotes bridge-related tokens is doing what it appears to do. At dictionary scale, manual inspection does not cover tens of thousands of features, so auto-interpretation pipelines have a language model read each feature's activating examples, propose an explanation, and score the explanation by predicting held-out activations from it. Auto-interp scores are noisy but allow whole-dictionary comparisons.

Two phenomenological findings recur. Feature splitting: train dictionaries of increasing size on the same activations and coarse features fracture into finer ones; a "chemistry" feature becomes separate features for elements, reactions, and lab equipment. This suggests the dictionary size you choose sets the granularity of description rather than discovering a unique true feature set. Universality: similar features (curve detectors in vision, base64 detectors and syntax features in language models) appear across independently trained models, evidence that features reflect the data distribution as much as the particular network.

10.6 Limitations, stated plainly#

SAEs are the current workhorse of representation-level interpretability, and every one of the following caveats is established in the literature. First, reconstruction error is not semantically small. The few percent of variance an SAE misses is not random noise; spliced models change behavior in structured ways, and the missing component (the "dark matter") plausibly contains dense or non-linearly-encoded information the sparse dictionary cannot express. Second, feature absorption: when a general feature (starts with S) and a specific feature (the token "short") co-occur, training can teach the specific latent to absorb the general one's contribution, so the general feature mysteriously fails to fire exactly where its concept is present. Absorption means a feature's silence is not evidence of a concept's absence. Third, SAEs describe representations, not computation. A dictionary tells you which directions are present at a layer; it does not tell you how the MLP transforms them or which downstream component reads them. Connecting features causally requires the patching tools of Chapters 4 to 6 applied in feature space, or the transcoder and attribution-graph machinery of Chapter 11. Fourth, an SAE is a function of its training distribution. Features are whatever was frequent enough in the corpus to justify a latent; deploy the same SAE on a different distribution and it reconstructs poorly and silently, with no warning that its ontology no longer fits.

Tooling makes all of this practical. SAELens is the standard open-source library for training SAEs and for loading published dictionaries (for example the residual-stream SAEs for GPT-2 small) into TransformerLens models. Neuronpedia hosts browsable dashboards, max-activating examples, and auto-interp explanations for thousands of published SAE features; browsing it for an hour is the fastest way to calibrate your expectations about what real features look like, including the messy ones.

11.1 From describing activations to replacing computation#

Chapter 10 left you with a tension. An SAE gives interpretable features of an activation vector, but it tells you nothing about how those features are computed or used. The SAE sees one point in the network and reconstructs it from itself; the MLP nonlinearity between an SAE at the MLP input and an SAE at the MLP output remains a black box. To trace a circuit through an MLP you were forced back to input-dependent methods: run a prompt, take gradients or patches, and accept that the answer holds only for that prompt.

The frontier work of the last two years attacks exactly this gap, and the move that unlocks it is small enough to state in one sentence. Instead of training a dictionary to reconstruct an activation from itself, train it to predict one activation from an earlier one, so that the dictionary becomes a replacement for the computation in between. Everything else in this chapter, cross-layer transcoders, crosscoders, attribution graphs, follows from that move plus engineering.

11.2 Transcoders: sparse stand-ins for the MLP#

A transcoder for MLP layer l is trained on paired activations. Let x be the MLP's input (the residual stream after the layer's second LayerNorm, LN2) and y = MLP(x) its output. The transcoder computes

and is trained to minimize ||ŷ - y||² plus a sparsity penalty on f (L1 in the original work; TopK and JumpReLU variants apply here just as they did for SAEs). The architecture is identical to an SAE. The training pairs are not: the input is x and the target is y. An SAE on the MLP output learns "what directions make up y"; a transcoder learns "what function maps x to y", approximated as a sparse sum of learned input-output rules. Dunefsky et al. (2024) found that at matched sparsity, transcoders equal or beat SAEs on the fidelity frontier for GPT-2, an encouraging sign that the harder task costs little.

The payoff is structural. Suppose every MLP in the model has been replaced by its transcoder. Feature i of the layer-l transcoder fires when W_enc,i has high inner product with the incoming stream, and when it fires it writes the fixed vector W_dec,i into the stream. Both of these are weights, not activations. The strength with which feature i of layer l feeds feature j of layer m > l through the direct residual path is the scalar W_dec,i · W_enc,j (up to LayerNorm scaling), computable without running the model on anything. This is the same virtual-weights logic as Chapter 1, but now the endpoints are sparse, mostly monosemantic features rather than 64-dimensional head subspaces. Circuit analysis between MLP features becomes input-invariant: a claim about the weights of the replacement model, true across all prompts, rather than a gradient measured on one prompt. The prompt-dependent part of a transcoder circuit is cleanly quarantined in which features are active and in the attention patterns; the feature-to-feature connections themselves are fixed.

Contrast this with an SAE latent on the MLP output. Its decoder direction tells you what the latent writes, and projecting through the unembedding tells you what it pushes the model to say. But to ask "what in the MLP's input causes this latent to fire" you must differentiate through the MLP nonlinearity, and the answer depends on where you linearize: a different prompt gives a different gradient. The SAE describes the output space; the transcoder replaces the map. That asymmetry is the entire argument, and the lab makes you verify both halves of it numerically.

One refinement matters in practice. MLPs have a sizable "roughly linear" component, and forcing sparse features to spend capacity on it is wasteful. A skip transcoder adds a learned dense linear term: ŷ = W_dec^T f + b_dec + W_skip x. The skip absorbs the linear bulk, the sparse features specialize in the nonlinear residue, and reconstruction improves at fixed sparsity. The skip is itself a fixed linear map, so input-invariant analysis survives intact.

11.3 Cross-layer transcoders and crosscoders#

Per-layer transcoders inherit a problem you have met repeatedly: adjacent layers are redundant. The same feature is often computed incrementally across several MLPs, so per-layer dictionaries learn several copies of it, and paths in a feature circuit zigzag through duplicates. The cross-layer transcoder (CLT) addresses this by changing the wiring: each latent reads from the residual stream at one layer, but owns a separate decoder vector for the MLP output of that layer and every later layer. Training jointly, a feature that the model builds up over layers 4 through 7 becomes one latent that writes to all four MLP outputs, instead of four near-duplicate latents. Anthropic's circuit-tracing work found CLTs give markedly simpler attribution graphs than per-layer transcoders at matched total dictionary size, precisely because they absorb inter-layer redundancy.

Crosscoders generalize the idea from "one reading point, many writing points" to "one shared latent, many places". A crosscoder assigns each latent a shared activation but distinct encoder and decoder vectors per site, where the sites can be several layers of one model or the same layer of two different models. Trained on both models' activations at once, the dictionary is forced to explain both with one set of latents. The decoder norms then carry a diff signal: a latent whose decoder norm is large in model B but near zero in model A is a feature that exists only in B. Train a crosscoder on a base model and its fine-tuned chat variant and you can enumerate what fine-tuning added (persona features, refusal-adjacent features) and what it left untouched (the bulk of the shared representation). This is model diffing, and it turns "how did fine-tuning change the model" from a vague question about weight deltas into a list of named features with examples. The same construction across checkpoints tracks feature formation during training, and across model scales probes universality.

11.4 Attribution graphs and circuit tracing#

Transcoders make feature-level connections computable; attribution graphs assemble them into an account of one prompt. The recipe, from Ameisen et al. (2025), has four stages. First, build a local replacement model: substitute a CLT (or per-layer transcoders) for every MLP, keep attention patterns frozen at their values from the real model's forward pass on this prompt, and freeze LayerNorm denominators likewise. Attention with frozen patterns is linear in the stream, so the whole replacement model is linear in the features that are active. Second, add error nodes: at each MLP output, the difference between the true output and the transcoder's prediction is inserted as a fixed extra input, so the replacement model reproduces the real model's activations on this prompt exactly. Third, compute the graph: nodes are active features, token embeddings, error nodes, and output logits; the edge weight from node u to node v is u's direct linear contribution to v's pre-activation, computable in closed form because everything between them is frozen-linear. Fourth, prune: keep only the paths that carry significant influence from tokens to the chosen logit, typically reducing millions of potential edges to a subgraph small enough to read.

Read correctly, an attribution graph is a hypothesis generator with a built-in honesty meter. The hypotheses are the pruned paths: chains like "feature for Texas capital-related text, activated by 'Austin'-adjacent context, feeds a say-Austin feature, which writes the Austin logit". These are then validated the way this course has always validated claims, by intervention: clamp or ablate the feature in the real model and check the downstream effect. Multi-step factual recall ("the capital of the state containing Dallas") shows up as two hops through an intermediate Texas feature; poetry continuation shows planning-like structure, with features for a rhyme word active at the line break before the line that ends with it, and steering those features redirects the line. The honesty meter is the error nodes. Their share of each activation is unexplained variance, dark matter the dictionary missed; if the paths through error nodes dominate, the graph is telling you not to trust its story. And the local-replacement caveat must be said plainly: the graph is faithful to a linearization around one prompt with attention and LayerNorm frozen, so it cannot explain why the attention patterns are what they are, and a graph on one prompt licenses no claim about another prompt until you test it.

11.5 What has held up, and what has not#

A field moving this fast needs a scorecard. Robust so far: induction heads exist in essentially every transformer trained on natural language, verified across scales, seeds, and architectures, with clean weights-level and behavioral signatures. The indirect object identification (IOI) circuit replicates, and its head classes (name movers, S-inhibition, backup name movers) reappear under re-analysis with newer tools, though the exact head sets and the completeness of the story vary. Superposition as a phenomenon, and sparse dictionaries as a useful microscope, have survived heavy use. The core causal toolkit, patching with careful metric choice, remains the standard of evidence.

More fragile: claims of the form "concept X is represented by a single direction". Some such claims replicate (refusal-adjacent directions steer robustly); others weaken under probing with better baselines, turn out to be dataset artifacts, or fragment into feature families under a larger dictionary (feature splitting, Chapter 10). Faithfulness of discovered circuits is contested territory: circuits that pass one ablation regime fail another, self-repair inflates completeness claims, and papers have shown you can game faithfulness metrics with subgraphs that no one would call the mechanism. SAE-specific worries (absorption, dark matter, sensitivity of interpretations to dictionary size and seed) carry over to transcoders largely unresolved. Hold both lists in mind whenever you read a new paper: ask which prior claims it depends on, and which regime of evidence it offers.

11.6 Open problems#

Six clusters of open problems structure most current research. Scalability: dictionaries and attribution graphs are demonstrated on models up to the tens of billions of parameters, but cost scales with model size times dictionary expansion, and frontier-model coverage remains partial; automated interpretation of millions of features is itself an unsolved evaluation problem. Faithfulness guarantees: we lack a principled account of when a replacement model's story is licensed to stand in for the real mechanism, and error nodes quantify but do not close the gap. Universality: whether features and circuits recur across seeds, scales, and architectures determines whether interpretability findings are science or model-specific natural history; crosscoders give a tool, not yet an answer. Features versus computation: dictionaries describe representations, but the dark matter problem includes computation no current decomposition captures, and some argue parameter-space decomposition, not activation-space, is the right ontology. Benchmarks: MIB (the Mechanistic Interpretability Benchmark) standardizes circuit-localization and causal-variable evaluation across tasks and models, and InterpBench provides semi-synthetic models with known ground-truth circuits; both exist because self-reported wins had become uninterpretable. Safety auditing: turning these tools into procedures that can check a frontier model for deception, hidden goals, or sabotage-relevant computation is the field's load-bearing ambition, and pilot audits of model organisms are only the first step. Each cluster is an entry point: pick one, find its benchmark, and the open questions are explicit.

11.7 Entering the field#

The reading flow that keeps you current has three tiers. The transformer-circuits.pub thread publishes Anthropic's interpretability work, including the framework, superposition, monosemanticity, crosscoder, and circuit-tracing papers, with interactive artifacts; read new entries as they appear and mine their related-work sections. Peer-reviewed venues carry the broader field: ICLR, NeurIPS, and ICML main tracks and their interpretability workshops, plus ACL for NLP-flavored analysis work. The Alignment Forum and LessWrong host early-stage results, critiques, and negative results that never reach a conference, and much of the field's error correction happens there.

The on-ramp that works is reproduction. Pick a paper with public code and a small reference model, reproduce its headline result, then push one variation: a different model, a different task, an ablation the authors skipped. Every chapter of this course has already made you do this in miniature (IOI in Chapters 2 and 4, induction heads in Chapter 3, ACDC in Chapter 6, ROME in Chapter 9, SAEs in Chapter 10). A reproduction with one novel twist is a workshop paper; several are a research program. The tooling landscape you already partly know: TransformerLens for hook-level work on small models, SAELens for training and loading dictionaries, nnsight for intervention on larger models with remote execution, and Neuronpedia for browsing existing SAE and transcoder features with auto-interp labels and steering interfaces. The ARENA curriculum provides guided exercises covering most of this course's topics with solutions. Compute is not the barrier at entry level: everything in this course ran on a CPU, and a single consumer GPU covers most published small-model work.