Tools & calculators

Takeaway. Backprop never writes down a global formula for dL/dw. It seeds dL/dL = 1 at the loss and then, at every edge, multiplies the gradient arriving from downstream by that edge's local derivative — the only thing each node has to know. Because every leaf here is reached by exactly one path, the products collapse to dL/dw = 2e a′(s) x, dL/dx = 2e a′(s) w, dL/db = 2e a′(s), dL/dy = −2e; when a node feeds several consumers the arriving gradients are summed instead, which is exactly why PyTorch accumulates into .grad and why you must call zero_grad(). Choose ReLU and push s below zero to watch the gate close: a′(s) = 0 annihilates every gradient upstream of it, while dL/dy — which never crosses that edge — survives. The finite-difference column is the same check torch.autograd.gradcheck runs.

What is exact here. The schema is compiled into a real character-level DFA (states = start, expect-key, in-key, expect-colon, expect-value, in-value, after-value, done, product-ed with the field index, the position inside the key literal, and the string/number length counters). The token-level mask is the genuine Outlines-style index: for every token the DFA is stepped one character at a time from the current state, and the token is allowed if and only if it never reaches the dead state. Masking is l~_v = l_v if allowed else −∞, followed by a max-subtracted softmax, so the surviving probabilities are exactly p_v = p_raw,v / Σ_{u allowed} p_raw,u — look at "mass kept" in the readout: that denominator is the whole story. The logits themselves come from a small hand-written scorer plus seeded Gaussian noise (a stand-in for a real network); it is deterministic, and its only job is to be JSON-ish but fallible. The guarantee check enforces exactly the language the DFA accepts, bounds included — key order, value types, string lengths and digit counts — so "40 / 40" is the grammar's guarantee and not a weaker one.

Three things worth pausing on. (1) Tokens, not characters. With name, na and me all in the vocabulary, the legal set changes with every partial match: at the start of the key both name and na are legal and me is not; one token later it is the other way round. That is why the index must simulate multi-character tokens instead of checking single characters. (2) Alive, not accepting. A token is legal if it leaves the machine in any live state; only EOS requires an accepting state, which is exactly why constrained output can never be truncated mid-object. (3) Masking is local, not conditioning. The mask renormalises over legal next tokens, which is not the same as sampling from the model conditioned on the whole string being valid: mass the model wanted to spend on an illegal continuation gets redistributed over its legal siblings, so constrained sampling is a genuine distribution shift, not a filter. Watch the guarantee bar: constrained decoding is valid 40/40 by construction, while the same model at the same seeds, unconstrained, loses runs to an unquoted value ({"name":Oslo,), a hallucinated key ("adok"), a colon swallowed into a key (": null"), a boolean slot filled with a number, or an integer with more digits than the schema allows — and the failure rate climbs with σ and T.

And the catch. Validity is not correctness. Because the mask only ever asks "can the machine survive this token?", the sampler is free to spend the redistributed mass on rubbish that happens to be well-formed: set the seed to 20 and the constrained run happily emits {"name":", ","age": 12, ...} — perfectly schema-valid, perfectly useless. A grammar buys you syntax; it never buys you semantics, and a model that had to be dragged into the format is telling you something about the prompt. In production the index is precomputed once per grammar (|Q| × |V| bits, well under a megabyte), so the per-step cost is a bitmask lookup that XGrammar overlaps with the model forward pass.

Both systems see the identical seeded arrival stream and the same device model, so every difference is scheduling. Continuous batching re-derives the running set each forward pass, so a slot freed by an EOS is refilled on the next iteration and the active batch stays pinned near max_num_seqs. Static/request-level batching (given the benefit of a 1 s dynamic-batching window to fill its batch) freezes membership: short requests turn into dead slots that still cost a row in every GEMM until the longest request in the batch finishes, and prompts are padded to the batch maximum — the three leaks, visible as hatching. Turn chunked prefill off (try the RAG workload) and a long prompt seizes a whole prefill-only iteration: τ = 20 + 0.2×2400 = 500 ms during which every decode stalls, drawn as warn-hatched cells. Turn it on and the same prompt is spread over ⌈prompt / leftover budget⌉ iterations that ride along with the decodes, and the longest forward pass row collapses to the budget cap τ₀ + β · max_num_batched_tokens — a bound the widget holds exactly. That bound is the point: it is a hard ceiling on inter-token latency. Whether it also costs raw throughput depends on the mix (compare the two output tok/s numbers with the box on and off): chopping prefill into more, thinner passes pays τ₀ more times, but keeping decodes alive alongside it raises the mean active batch, and on chat-shaped traffic the second effect usually wins. That trade — ITL ceiling versus τ₀ amortization — is the knob Sarathi-Serve turns. Shrink KV blocks until the scheduler cannot reserve the next block for a running request: it evicts the newest request (LIFO), whose materialized KV must be recomputed on resume — that redundant work is what the “useful-compute share” row charges continuous batching for, and it is why you keep headroom instead of admitting to the last block. Caveats: τ₀ and β are a linear stand-in for a real kernel timing curve, attention cost grows with context length (ignored here), and the mock model emits a fixed number of tokens per request instead of a real EOS.

The loop is exactly the chapter's sampler: start all-[MASK], run one full bidirectional pass, softmax the logits at every masked position (with max-subtraction), and unmask the top-n_keep by confidence, where n_keep = (#masked) − round(n_gen × (1 − (k+1)/T)) follows the linear schedule. Unmasking is one-way: a committed token is never revised, which is why an early mistake is permanent. Serial depth is what diffusion buys: NFE = T regardless of length, versus L for AR. But each diffusion pass is more expensive — every position is recomputed, so there is no KV cache and attention costs O(L²) per step instead of O(L) per AR decode step; L/T is an upper bound on wall-clock speedup, not the speedup itself. Why the order matters: within one step the model factorizes p(x₀ | x_t) as a product of independent per-position predictions, which is wrong for language — set T = 1 and watch half the block come out individually plausible but jointly wrong. Committing only the peaked positions and re-running defers the flat ones until their neighbours exist — that is the repair, and it is why the confidence curve dominates random and left-to-right. Push the order to left-to-right at T = n_gen (with nothing clamped on the right) and masked diffusion is an autoregressive model: same match rate, same NFE, no win. Infilling is free here because prefix and suffix are clamped from step 0, so masked positions have context on both sides immediately; AR has to regenerate everything to the right of the edit and nothing constrains it to land on the required suffix.

What is real and what is scripted. The softmax, the linear schedule, the top-n_keep selection, the one-way commit and the NFE accounting are the real algorithm. The denoiser is a seeded toy: each position has 4 candidates, the reference token scores 3.0·a + 3.4·(0.6·left + 0.4·right) where a is its base predictability and left/right are the fractions of its exponentially-weighted neighbourhood already committed, while the three rivals sit on a fixed plateau scaled by (1 − a). That is calibrated on purpose: with no context every wrong prediction has p ≤ 0.39 and every right one p ≥ 0.51, so confidence really does identify the positions worth committing — the assumption the whole remasking rule rests on. It also makes the AR path a perfect oracle by construction, so its reference line sits at 100%; the question this chart answers is how few serial passes diffusion needs to match AR, not whether AR can be beaten.

Soft targets carry more information than hard labels. A one-hot label says only "class 7, probability 1, everything else 0." The teacher's full distribution instead says "class 7, but also a bit like class 1" — that relative ranking of the wrong classes is Hinton et al.'s "dark knowledge," and it is exactly what a hard label throws away. Raising T divides every logit before the softmax, which shrinks the gaps between logits and pulls probability mass off the top class onto its runners-up: watch the confusable digit's bar grow, and watch the thin reference tick (its T=1 height) get left behind. Push T far enough and every bar collapses onto the dashed 1/K line — at that point the ordering information is gone too, which is why τ ∈ [2, 5] is the usual range. The student is trained on a blend: α weights ordinary cross-entropy against the true label (computed at T=1), (1−α) weights the KD term. Because that term is T² · KL(q_T ‖ p_T) rather than raw KL, its gradient is T·(pT − qT): the extra T cancels the 1/T that differentiating a T-scaled softmax introduces, and since pT − qT itself shrinks like 1/T, the gradient norm stays roughly flat instead of dying like 1/T². The last stat tile shows both norms — slide T from 1 to 8 and compare. Click "Auto-train" and watch the student's bars migrate onto the teacher's, including onto the small dark-knowledge bars, while the loss curve shrinks. (With α > 0 the loss plateaus above zero: the hard label pulls toward one-hot while the teacher pulls toward its soft target, and the student settles at the compromise. The curve also rescales whenever you move T or α, because that changes the loss function itself.)

Two operations, one guarantee. DP-SGD replaces the batch-mean gradient with ĝ = (1/B)(Σ_i g̃_i + N(0, σ²C²I)), where g̃_i = g_i · min(1, C/‖g_i‖₂). Clipping is what makes the guarantee possible: after it, adding or removing any single example changes the summed gradient by at most C, so the mechanism's L2 sensitivity is exactly C — a number you control instead of a number the data chooses. Noise is what buys the guarantee: Gaussian noise of standard deviation σC on the sum hides that one-example change. Note what the picture shows when you drag the canary slider with a large C: one weird example visibly drags the mean. Shrink C below its norm and the mean locks in place — that is bounded sensitivity, drawn. The accountant. ε here is not a fudge factor: it is the Renyi-DP bound for the Poisson-subsampled Gaussian mechanism (Mironov, Talwar & Zhang 2019) at integer orders α, ε_α = (1/(α−1)) log Σ_k C(α,k)(1−q)^α−kq^ke^{k(k−1)/2σ2}, composed over t steps (RDP adds) and converted to (ε,δ) with the standard bound, minimized over α. It is an upper bound; a PRV accountant is slightly tighter. Sampling rate q = B/N assumes Poisson sampling — shuffle-and-chunk batching silently invalidates it. Three things to discover. (1) Once C is smaller than the gradients, C cancels: signal and noise both scale with C, so C sets the step size, and σ alone sets the privacy. That is why tuning C is a learning-rate problem, not a privacy problem. (2) The noise is isotropic in all d trained parameters, so its norm grows like σC√d/B while the signal is capped at C — switch d from a LoRA adapter to a 7B full fine-tune and watch the signal-to-noise collapse. That single fact is why DP fine-tuning is done on few parameters. (3) Bigger batches at fixed epochs shrink the noise per step but raise q and cut t, so ε goes up: privacy is not free compute. The canary. Exposure here uses an illustrative extraction model, not a simulated attack: over t steps a canary present in r copies is sampled rqt times, so its imprint grows like rqt · min(C,‖g_canary‖) while the injected noise in that direction is a random walk of size σC√t, giving a discrimination score z = rq√t · min(C,‖g_canary‖)/(σC); the canary's rank among 10⁸ candidate secrets is then 10⁸Φ(−z), and exposure is log₂(10⁸) − log₂(rank), the Secret Sharer definition used earlier in this chapter. Two things fall out. The "clipping only" bar sits at full exposure: bounding sensitivity without noise hides nothing, so clipping alone is not a privacy mechanism. And a secret that appears once is buried at any reasonable σ, while dragging r up to 100 lights it back up — because DP-SGD's guarantee is per example, and r duplicates degrade it to group privacy of about rε. Deduplication is not an alternative to DP; it is what makes the ε you report mean what you think it means.

DPO needs exactly these four numbers per pair — two policy log-probs, two frozen reference log-probs — and turns preference learning into ordinary binary classification: is the implicit reward r = β·log(π_θ/π_ref) higher for the chosen response than for the rejected one? Only the difference of the two log-ratios matters, so anything that shifts both equally (a longer prompt, a globally sharper model) cancels out. The gradient's weight σ(−Δ) is large exactly when the model still gets the pair wrong, so training automatically focuses on unsolved pairs and fades out solved ones — run the 10-step button and watch the point slide right while the weight decays: that is the self-curriculum. β is the trust-region dial: push it down and the sigmoid needs a much larger log-ratio swing before the loss (and the gradient weight) relaxes, so the policy is free to drift far from π_ref — sharper fit, weaker KL control. Push β up and a tiny swing already saturates the curve, so the policy barely has to move — strong KL control, potentially underfit preferences. Two honest caveats about this toy: the step here moves the two log-probs as free scalars, while in a real model they share weights and a softmax normalizer, so raising log π_θ(y_w) usually drags log π_θ(y_l) along — which is why real DPO runs often see both log-probs fall while the margin still grows; and nothing here caps the margin, which is exactly the unbounded-drift failure IPO and the margin-based variants were invented to fix. Unlike PPO-RLHF, none of this needs a trained reward model, sampled rollouts, or a value network — just a sigmoid over four log-probs, backed by the same reward-minus-β-KL objective PPO optimizes explicitly.

Each point is a 2-D stand-in for a row of the embedding matrix W_E: a learned direction and magnitude, not a lookup key. Cosine similarity is cos θ = (A·B) / (|A|·|B|) – the dot product with both lengths divided out, so it depends only on the angle. Drag the Scale A by k slider: A slides along its own ray, so A·B and |A−B| both change while cos θ does not move at all. That invariance is exactly why embeddings are compared by cosine – a token whose vector happens to have a large norm (frequent tokens usually do) should not automatically look "more similar" to everything. The construction overlay shows the two unit vectors on the circle |v| = 1 (cos θ is literally their dot product) and the projection of B onto A, whose signed length is |B| cos θ, so that A·B = |A| × (|B| cos θ). The analogy button draws king − man + woman as vector arithmetic: the offset from man to king transplanted onto woman, with the nearest stored embedding to the landing point highlighted – and, because the two metrics rank neighbours differently, cosine and Euclidean answers are reported separately.

The takeaway: FlashAttention never forms the S×S score matrix. It walks tiles of it, and for each tile updates only three per-row statistics that live in registers/SRAM: running max m_i, running denominator ℓ_i, and unnormalized accumulator O_i. Before adding a tile's contribution it rescales the old state by exp(m_old − m_new) — that one multiply re-bases every earlier term onto the new max, so the recurrence is exact, not an approximation: watch the max|diff| column fall to ~1e-16 the moment a row's last KV block lands. The single divide by ℓ_i happens once, at the end. Raise B_r/B_c and the SRAM tile grows as B_rB_c; lower B_c and K/V get re-read more times (more HBM traffic) — that tension is the entire kernel-design tradeoff, and the naive path stays stuck at O(S²) either way.

GQA interpolates between MHA and MQA. Group size s = 1 gives every query its own KV head (MHA: best quality, biggest cache); s = h collapses all queries onto one shared KV head (MQA: smallest cache among the head-sharing schemes, largest quality risk); any divisor of h in between trades quality for cache linearly in the KV-head count g = h / s, since the cache is 2 · g · d_h elements per token per layer (the 2 is K and V). MLA takes a different axis: instead of sharing whole KV heads it caches one low-rank latent vector per token (d_c numbers, no factor of 2) and reconstructs full per-head keys/values from it on the fly – so it can beat GQA's cache while keeping every head distinct. Real MLA also caches a single shared decoupled-RoPE key of d_r = 64 (toggle it above); the text's simplified forward pass is the content-only version. Note the “total” column: the same number is both the memory one sequence occupies and the bytes that must be streamed from HBM on every decode step – which is why shrinking the cache speeds up decoding, not just fits more of it.

Split the stack into k segments, store only each segment's input, and recompute the rest during backward: M(k) ≈ k·(2BTd) + (L/k)·a_layer — stored checkpoints grow with k, the one segment brought back to life shrinks with k. (Exactly, since backward pops segments last→first and only checkpoints 1…i survive while segment i is re-materialized, the peak is max_i [ i·2BTd + size_i·a_layer ] — which is what the chart plots, hence the sawtooth.) The textbook answer k = √L assumes the two tensors cost the same; in a real block they do not. Per Korthikanti et al. the stored bytes per layer are a_layer = 34·B·T·d (+ 5·a·B·T² when the T×T scores are materialized) — 17× the bf16 block input — so the optimum sits at k* ≈ √(17L), far to the right of √L. For any stack shallower than 17 blocks that is past L, i.e. k* = L exactly; at L = 32 it is 24 of 32, and even at L = 80 (≈37) the curve around the minimum is shallow enough that checkpointing every block costs only tens of percent more activation memory than the true optimum. That is why frameworks expose “checkpoint every block” and stop thinking about it. (The 34 counts every tensor a block stashes, dropout masks included; the chapter’s back-of-envelope 12 elements × 2 bytes = 24 B/token/layer counts only the big ones, which is why it says ≈6.4 GB where this widget says 8.5 GiB for the same 7B config — same story, one constant apart.) The compute price is one extra forward pass, i.e. +33% on a fwd+bwd step, for every k > 0 — it does not grow with k, so there is no reason to stop short of the memory optimum. Note also what checkpointing never touches: the 16P of weights, gradients and Adam state, and the B×T×V fp32 logits of the loss head — which is why a small model with a big vocabulary OOMs on the head, not on the blocks. Everything here is closed form: no randomness, same inputs → same numbers.

The surface is the exact convex bowl L(θ) = ½(λ₁θ₁² + λ₂θ₂²) with λ₁ = 1/√κ and λ₂ = √κ, so the Hessian is exactly diag(λ₁, λ₂), the condition number is κ = λ₂/λ₁ by construction, and the eigenvalues keep geometric mean 1 for every κ. Each update is the real rule applied to the analytic gradient ∇L = (λ₁θ₁, λ₂θ₂) plus optional noise σ·N(0,1) per coordinate — SGD: θ ← θ − ηĝ; momentum: v ← βv − ηĝ, θ ← θ + v; Adam: bias-corrected first/second moments with β₂ = 0.999, ε = 1e−8. Three things worth doing. (1) With σ = 0, GD on a quadratic is a linear recursion, so its per-step contraction is exactly ρ = max_i|1 − ηλ_i| (heavy ball: the larger root modulus of z² − (1+β−ηλ)z + β, which is √β whenever the roots are complex). The dashed line on the loss chart is that prediction, L₀ρ^2t, and the measured curve settles onto its slope (with momentum it oscillates around it, because the roots are complex). Divergence is not at η = 1/λ_max — that is the guaranteed-descent bound; the real ceiling is η = 2/λ_max (2(1+β)/λ_max with momentum), and just below it you get the classic zig-zag rather than an explosion. (2) The gradient ratio |g₂/g₁| = κ·|θ₂/θ₁| is enormous, and SGD's step ratio equals it exactly (the step is proportional to the gradient) — that is the zig-zag. Switch to Adam and the step ratio sits at ≈1 no matter how large the gradient ratio: that is what "Adam preconditions the ill-conditioned directions" means in numbers. (3) Raise σ: SGD stops converging and hovers at the stationary noise floor E[L_∞] = ½∑_i λ_iη²σ²/(1 − (1−ηλ_i)²) ≈ ησ²d/4 for small η — a floor proportional to η, which is exactly why the learning rate has to decay. The noise draws come from a seeded PRNG, so Reset replays a bit-identical run.

The group mean is a Monte-Carlo estimate of the same baseline that PPO trains a whole separate critic network to predict — GRPO gets it for free from the G samples you already generated. Subtracting a baseline removes the common-mode reward level, so gradients point toward relatively better responses; dividing by the group's own std (GRPO) further rescales, so an easy prompt with big reward swings and a hard prompt with small ones contribute comparably-sized updates. Note that RLOO is not quite “mean-only”: its leave-one-out baseline gives A_i = (G/(G−1))·(R_i − mean), i.e. Dr. GRPO's advantage rescaled by a constant — switch between them and watch only the axis numbers change, never the shape. Every mean-based option makes ΣA_i = 0 exactly; “no baseline” does not, which is why it pushes every response up. The failure mode to remember: when every response in the group earns (nearly) the same reward — all correct, or all wrong — the group carries no information about which completion was better, every advantage collapses toward 0, and the rollout's gradient vanishes. That is the empirical case for dynamic sampling (DAPO): detect zero-variance groups and resample instead of wasting compute on them. Pick the worked example reward source to check the widget against the ±0.992 / −0.811 / −1.172 numbers derived in the text.

A brute-force scan is O(N) distance computations and always exact. HNSW replaces the scan with a walk: each node keeps up to M links to nearby vectors, and a node's top layer is drawn from an exponentially decaying distribution (level = floor(−ln U · m_L), m_L = 1/ln M), so layer sizes shrink geometrically and the top layer holds a handful of nodes with very long links. Search descends that hierarchy with a beam of width 1 — pure greedy hill-descent, a coarse-to-fine zoom — and only in layer 0 opens the beam to ef. Each hop evaluates just the current node's neighbors, so the total work grows like O(M · log N), not N. Three experiments, each reproducible in the panels above:

1. The scaling gap is the whole point. Set ef = 6 and sweep N from 8 to 60, watching the mean-cost dot on the chart. The scan cost is the N slider's own value; the HNSW cost goes from roughly 12 to roughly 26. Seven and a half times the data, barely double the work — and note the crossover: at N = 8 the walk costs more than the scan. A hierarchy only pays for itself once N is large, which is why nobody indexes a thousand vectors.

2. “Approximate” means measurably wrong, sometimes. Two different failures hide behind a low recall number. With ef < k the beam cannot even hold k results, so recall is capped at ef/k for arithmetic reasons — always set ef ≥ k. The real failure is the one the true nearest neighbor MISSED card reports: drop ef to 1 or 2 and drag the query into the gap between two clusters, and the greedy walk settles into a local minimum and returns a neighbor that is not the nearest. That is the tax you pay for the speedup, and it is why recall must be measured against brute force, never assumed.

3. Diverse links beat close links. Push N to 60, then flip neighbor selection to M closest and watch layer 0 thicken: the link counter jumps from 142 to 284. Twice the links, all of them spent pointing back into the same cluster, because the nearest M candidates to a node inside a dense blob are all in that blob. Look at the sweep chart now: the recall curve stops at about 0.79 and stays there however far you push ef — some blobs have lost the bridge the walk needs to cross, and a wider beam cannot reach a component it can never enter. The paper's relative-neighborhood rule spends each link on a distinct direction out of the node instead, and reaches recall 1.0 on the same data with half the links. That is why Algorithm 4 exists.

Real indexes are this same code in 768 dimensions, where the picture is impossible to draw but the mechanics are identical.

This is a K-composition circuit, not a Q-composition one: the induction head’s key at position j is built from what the previous-token head wrote there (the identity of token j−1), while its query at the final position just reads the current token directly — no composition needed on the query side. The score spikes exactly where token[j−1] equals the query token, which is precisely “the position right after the earlier occurrence.” Three things to try. (1) Make the last token something that never appeared before: the induction row goes exactly uniform over the causal history — a fan of faint arcs, no confident copy — because the circuit has no signal to key off. (2) Drag sharpness to S = 0: matched and unmatched keys score identically and the head degenerates to uniform averaging; that is the softmax-saturation trade-off from the self-attention explorer, and it is why the QK circuit has to learn a large effective score gap. (3) Hit Repeat as [X X]: every position in the second copy becomes predictable from the first, the prefix-matching score jumps toward 1, and you are looking at exactly the synthetic probe used to find induction heads in a real model in the code below. This deterministic toy is the mechanism real induction heads implement (Olsson et al., 2022); its emergence during training is one of the most reproducible phase changes in mechanistic interpretability, and it is the textbook explanation for why in-context learning works at all.

Locate. Corrupting the two subject-token embeddings destroys the fact. Restoring one cached clean activation at a time and re-running tells you where the fact is causally mediated: the MLP trace lights up at the last subject token in the middle layers (the memory band), and the attention trace lights up at the last token in the late layers (the head that copies the retrieved attribute to where it is read out). That two-site pattern is the empirical result ROME is built on, and here it is a genuine consequence of the circuit rather than a drawing. Widening the restore window recovers more of the effect because no single layer holds the whole association — which is exactly why MEMIT spreads its update over a band of layers. Edit. The MLP down-projection is read as a linear associative memory W k = v. We capture the key k_* at the edit site, optimize the value v_* by Adam so the model emits the new object, and then add the closed-form rank-one Δ = r (C⁻¹k_*)^T / ((C⁻¹k_*)^Tk_*) with r = v_* − W₀k_*. The grid makes the structure visible: every column of Δ is the same vector r, rescaled — that is what "rank one" means, and it is a consequence of the least-squares objective, not an assumption. The one knob that matters. Because the constraint W k_* = v_* is enforced exactly, reliability is free at any site with a live key and a causal path: P(o_*) stays pinned near 1 for every λ. Generalization is free here too, for a reason worth naming — the paraphrase shares the entire prefix "The Space Needle", so at the last subject token it produces the identical key k_* and inherits the edit automatically. (Real ROME is not so lucky: paraphrases that change the subject's surface form produce a different key, which is why generalization is a scored metric and not a theorem.) The only thing you can still lose is locality. Drag λ up (C → a multiple of I — "ROME without statistics") and watch the two numbers move in opposite directions: ‖Δ‖_F goes down, because that limit is the minimum-norm solution, while the collateral mean ‖Δk_i‖² on the layer's ordinary keys goes up, and at the default site two of the three unrelated landmarks start answering with the injected city. That is the whole argument for estimating C: the smallest edit is not the least damaging one. The least damaging one is the edit that C⁻¹ steers into the directions the layer's real keys barely use. Site matters as much as λ. Three failures worth reproducing. Keep layer 4 but move the edit to the last token, "downtown": all three unrelated facts flip, because before the mover heads have run that position looks identical in every prompt — you are overwriting a key the whole corpus fires. Move to layer 0 instead: that MLP is not a memory, so its key is generic and whatever it writes has to survive nine more layers — the solve pays for it with a ‖Δ‖_F over ten times the norm of the entire matrix it is editing, and a bystander flips anyway. Try layer 8 or 9 with the edit token back on "Needle": the mover heads have already read that position, so nothing downstream reads what this MLP writes, the v_* objective is perfectly flat, and Δ = 0 exactly (v_* never moves off W₀k_*, so the residual r is zero). The traced peak is the site where a small, specific update suffices. (The peak button follows whichever trace is on screen, so sending it to the attention peak lands you on the last token in a late layer — the failure above. Both peaks are real; only the MLP one is editable. That asymmetry is why ROME locates with attention and MLP traces but writes only to MLPs.) Two honest caveats. This W₀ is only 16×10 and holds four facts, so it is a tiny target: ‖Δ‖_F lands around two to three times ‖W₀‖_F even at the good site — read the ratio as "this layer is mostly the new fact now", not as a bug. In GPT-J's 4096×16384 down-projection the identical construction is a sub-percent perturbation, because the denominator is 67M real parameters instead of 160. What carries over unchanged is the rank: one, stored as d + d_mlp numbers instead of d · d_mlp, and trivially undone by subtracting it. And the edit installs an association, not a belief: nothing here propagates to "which country is it in?" — the ripple-effect failure, which no setting of λ or the edit site can fix.

KV bytes = 2 × layers × KV-heads × head-dim × tokens × bytes (the 2 is K and V). With grouped-query attention only the KV heads count — that's the whole memory point of GQA/MLA. Total tokens = context × concurrent requests.

KV bytes = 2 × L × H_kv × d_h × seq_len × bytes/elem — the leading 2 stores both K and V, and batch size is 1 here (one sequence). Only the number of KV heads enters the formula; query heads do not. That is exactly why GQA shrinks the cache: several query heads read the same stored KV head, so fewer bytes per token, at every layer, for the whole sequence. Prefill writes the entire prompt's KV in one parallel pass (the solid block appears at once); each decode step appends exactly one token's worth of KV, so the cache grows linearly forever — which is why a long, many-turn conversation can end up costing more memory than the model's own weights.

Each line is the angle m · theta'_k that the scaled scheme feeds to dimension k, divided by the largest angle that dimension ever saw in training (L_train · theta_k, original base). RoPE's angle is linear in position, so every line is exactly straight, and the shaded band 0–1 is "angles this dimension actually trained on". None: every dimension ends at ratio = s, so nothing past L_train is in distribution. PI: every dimension ends at exactly 1 – safe, but adjacent tokens are now sx closer in angle (watch the local-resolution tile), which is why PI needs finetuning. NTK-aware: raising the base pins only the lowest-frequency dimension at 1 and leaves k=0 untouched, so the middle, under-rotated dimensions still overshoot. YaRN decides per dimension from its wavelength: leave it alone if it completes ≥ beta_fast rotations inside L_train (its angles already wrap the circle many times, so "new" angles are not new), interpolate it fully (theta_k/s) if it completes ≤ beta_slow – so every genuinely novel dimension lands at exactly 1 and local resolution stays 1.00x. The tile to watch is worst ratio among under-rotated dims: 8.00 for None, 1.75 for NTK-aware, 1.00 for PI and YaRN at the default 8x – but only YaRN gets there while keeping local resolution at 1.00x.

Compare cosine against WSD at the same T: the mean LR stat shows WSD spends far more of the budget at full speed (cosine's average is only about ½(η_max+η_min), because the cosine is decaying from step T_w onward), yet Hägele et al. (2024) find they reach the same final loss at equal compute. That is the whole argument for WSD: the long stable phase commits to no total step count, and the short decay leg (shaded; typically ~10% of T) is where the characteristic extra loss drop lives. Set the min-LR ratio to 0 to reproduce MiniCPM's practice of decaying all the way down. Because the lowest-LR tokens are also the most influential, the decay window is the natural seam for a data-mixture switch — this is exactly why mid-training/annealing is scheduled inside the WSD decay phase in the capstone build (see Optimizer & Schedule: Muon + MuonClip and WSD and Mid-Training).

Every arrow in this diagram is an API call, and every API call re-sends its whole context. That is the entire lesson. A call is charged input = S + goal + R·(packets it reads) + (forwarded messages) and output = O for an intermediate call or N·O for a call that writes the whole report, priced at the chapter's illustrative gpt-4o-mini tier (USD 0.15 / 1M input, USD 0.60 / 1M output — the same tier as the 5-agent pipeline worked example); latency is 0.5 s + out/200 tok/s per call, summed over waves (calls in the same parallel wave overlap, so the dispatch toggle changes wall-clock but never the token bill). Watch the meters as you switch topology: the supervisor pays N extra system prompts plus a plan and a synthesis; the pipeline forwards every earlier stage's output into every later stage, so context grows linearly and the bill grows quadratically; the debate is all-to-all — each of N debaters re-reads all N opening statements, which is N² message-copies before the judge has read a word. The single agent is almost always the cheapest, which is why the chapter tells you to benchmark it first. Press Next message (or Replay) to walk the run one call at a time and watch the token meter climb; Show whole run puts the totals back.

The reliability panel is a toy model, stated exactly so you can check it. A task is systematically hard for this model with probability ρ (then every sample fails); otherwise each subtask-sized operation succeeds independently with q = p/(1-ρ), so the marginal per-subtask rate is the p you set. Chains multiply (q^k), redundancy uses the chapter's 1-(1-x)^N, and everything is capped by 1-ρ. Single: q^N (it must get all N subtasks right). Supervisor: q^N·q^2 — the plan and the synthesis are model calls too, and they can drop or garble a result. Pipeline: q^(2N-1), because each of the N−1 handoffs is one more place an error is inherited as ground truth. Debate: q·[1-(1-q^N)^N] — genuine redundancy, but the judge must still pick the right answer, and no N ever beats the 1-ρ ceiling. Turn the critic gate on: a failed worker is caught with probability 0.7 and retried, so q -> q + (1-q)·0.7·q. That is usually the biggest single win available, and it costs two calls — the MAST result that most multi-agent failures are organisational, not capability, in one number.

The honest verdict is the last column, cost per successful run = cost / P. At the defaults the single agent wins it. Now drag R up: at some point one context can no longer hold all N source packets, the single agent's call turns amber with OVERFLOW, its success probability drops to zero, and decomposition stops being a luxury. That crossing — not the vibe of "more agents, more intelligence" — is the real argument for multi-agent systems.

Activation memory for one pipeline stage can roughly double the model-state figure at steady state (chapter Worked Example: 11 GB state + 3.2 GB activations + 2 GB buffers). Treat green (<50%) as the safe target; amber means budget activations carefully. This models full ZeRO-3 / FSDP sharding across DP × TP × PP; ZeRO-1 (chapter Worked Example) shards only across TP × PP and gives a larger per-GPU figure.

The takeaway. Training state is 16 bytes/param (2 bf16 weights + 2 bf16 grads + 12 bytes of fp32 Adam master/momentum/variance), and activations are a separate, often larger, pile. DP replicates all 16 bytes on every GPU and splits only the batch — it buys throughput, never memory, and pays one gradient all-reduce per step. FSDP/ZeRO-3 divides those 16 bytes by G and pays ~1.5× DP's volume per microbatch (all-gather forward, all-gather backward, reduce-scatter grads) — cheap memory, expensive wire. TP also divides model state by G and is the only axis that shrinks activations, but it all-reduces an s×b×h activation tensor 4× per layer, so it must stay inside one NVLink node. PP divides state by the number of stages with almost no bandwidth cost (point-to-point activations at stage boundaries only) but pays the bubble, (p−1)/(m+p−1), and under 1F1B stage 0 still holds p microbatches in flight × L/p layers ≈ a full model's worth of activations. That is why real runs compose them: TP inside the node, PP across nodes, FSDP/DP outermost. Memory model: parameter counts are exact for the Llama-shaped presets (GQA-aware; 7B/13B/70B reproduce 6.74B / 13.02B / 68.98B). Activation bytes use the Megatron formula s·b·h(10 + 24/t + 5as/(ht)) with the published sequence-parallel (everything /t) and selective (drop the 5as²b term) / full (2sbh only) recompute variants. The microbatch is fixed at 1 sequence, all-reduce traffic is costed at the ring value 2(G−1)/G × payload, and the output-logit tensor (s·b·V) plus the tiny replicated norm weights are excluded.

Approximate: weight matrices only (ignores biases, LayerNorm, tied vs. untied LM head). 1 MAC = 2 FLOPs. FFN assumed d_ff = 4d (vanilla); gated SwiGLU adds ~50% to MLP. Matches the chapter's 24d² + 4sd per-token-per-layer accounting.

Every stage of RAG is a filter, and the answer is only as good as what survives all of them. Retrieve: chunks and query share one embedding space, and cos θ = (q·c) / (|q||c|) ranks by direction alone – drag the query straight out along its own ray and not one number in the table moves, exactly the length-invariance from the embeddings tool. Two independent gates then apply: top-k caps how many candidates the ANN index hands back, and the similarity gate throws away candidates that are not actually close (drag the query into the empty space between clusters and retrieved chunks vanish although k never changed). Rerank: each question has one chunk that genuinely answers it (ringed ◎), and for two of the four questions the bi-encoder does not rank it first – turn on the reranker and Reciprocal Rank Fusion, RRF(d) = Σ_r 1/(60 + rank_r(d)), promotes it, because a chunk both retrievers like beats a chunk only one of them likes; a chunk that shares no query term is simply absent from the lexical list and contributes nothing from that term, and the large constant 60 keeps any single first place from running away with the fusion. On the INT8 question this decides the answer outright: the bi-encoder ranks the right chunk third, the budget then evicts it and the model refuses – switch the reranker on and the same settings produce a correctly cited answer. Augment: the budget is a hard wall – system prompt and question are paid first, chunks are appended in final rank order, and the tail that no longer fits is dropped before the model ever sees it. Generate: the answer cites only chunks that reached the prompt, so when the answering chunk is filtered out the model is forced into "I don't know" – the readout names which gate dropped it. That failure chain (good corpus, bad k / threshold / budget → refusal) is why chunk size, top-k and context budget must be tuned together, not one at a time.

Each click reveals one phase — a Thought, an Action, or the Observation the harness injects — and lights up the node doing the work: the LLM thinks and emits Action: tool(args), the tool layer parses and dispatches it, the environment runs it and hands back an observation that is appended to the context for the next Thought. The context bar only ever grows: after k steps it is O(k·s) tokens, estimated the same crude way the chapter does it (total_chars / 4). That gives the loop exactly three exits, and you can trigger all three here: the model calls finish(...); the step counter hits T_max; or context_full() trips at 85% of the window. On the defaults the agent just finishes — drop T_max to 3 to make the step ceiling bite, or shrink the window to about 200 tokens to make the token guard fire first. Toggle “something goes wrong” to watch a call fail exactly the way this chapter's own calculator, search and _parse_action really would — a character-class rejection, an empty fake_db hit, and a TypeError from the regex splitting "299,792,458 / 1000" on its thousands separators (the exact bug native tool-calling removes). Each failure costs a full extra iteration, and the next Thought can only recover because the failed observation is sitting in the context: the context window is the agent's only memory of its own mistakes.

Each vector is one RoPE dimension pair rotating by angle m·θ_k. Drag m: high-frequency pairs (left) spin many times; low-frequency pairs (right) barely move – the same geometric frequency ladder as sinusoidal encodings. Hover / focus a cell for its exact numbers.

Fit-then-extrapolate is the whole scaling-law workflow. You cannot afford to train at the target scale, so you train a cheap ladder of small models, fit L(C) = E + A·C−α to their losses, and read the curve forward to a budget you have not paid for yet. With E held fixed the fit is linear in log space — ordinary least squares of ln(L−E) on ln C, closed form, no local minima. The solid segment is the range your ladder actually covers; the dashed segment is extrapolation, and the further right it runs the more you are trusting the shape of the curve rather than data.

Three lessons are built into the controls. (1) The model is misspecified even with zero noise. The ladder is generated by the real Chinchilla surface L = E + A/Nα + B/Dβ, which along a fixed ratio D = rN is a sum of two power laws in C, not one — so a single-term fit can only approximate it. The "α the ladder implies" stat is the exact slope of that true curve across your ladder's compute range: your fitted α should land near it, and how far it drifts is your bias budget. (2) E is barely identifiable. Slide the assumed floor and watch the prediction at the target swing by hundreds of a nat while R² stays above 0.99. "Auto-fit E" scans the floor for maximum R²: at σ = 0 it recovers ≈1.70 against a true 1.69, but at σ = 0.05 it drifts to ≈1.4 and at σ = 0.3 it collapses to 0 — hit Reseed a few times to see the spread. That is the same fragility Epoch AI reported when replicating Chinchilla (Besiroglu et al., 2024). (3) One bad run rotates the line. Drag a point far off the curve and watch the residual column and the extrapolated loss move; this is why real ladders drop under-converged and loss-spiked runs before fitting.

The "exact compute-optimal" stat does not use the fit at all: it minimises the true surface subject to C = 6ND by Lagrange multipliers, giving N* = (αA/βB)1/(α+β)(C/6)β/(α+β) — the same closed form as the book's optimal_alloc. Because α (0.34) ≠ β (0.28), the implied D*/N* is not a constant: these published constants give ≈32 tok/param at 10¹⁹ FLOPs, ≈62 at 10²², and ≈122 at 10²⁵. The famous "~20 tokens per parameter" is the ratio Chinchilla reported, and the fact that its own fitted exponents do not reproduce it is precisely the inconsistency the replication flagged — so treat 20 as a regime-local rule of thumb, never a law. Finally, raising "your plan" ratio above the optimum trains a smaller, cheaper-to-serve model for a loss penalty you can now read off exactly — and the valley is flat: at 10²⁴ FLOPs, going from the optimal ≈100:1 out to 500:1 costs under 0.01 nats, which the last stat restates as the honest currency, "the optimal split would have reached this loss on ~19% less compute". A fifth of your compute for a 5× smaller model to serve is exactly the over-training trade every inference-heavy lab now makes deliberately.

Compute-optimal means splitting a fixed budget C = 6·N·D between parameters N and tokens D. Chinchilla found the split scales as N,D ∝ √C at a ratio of ~20 tokens per parameter (adjust the ratio for over-/under-training, e.g. inference-aware training uses more). GPU-hours convert via C = GPU-hrs × 3600 × effective FLOP/s.

The heatmap is the transformer: each row is one query's probability distribution over all keys (random here, learned in a trained model), and the context vector below is literally that distribution used as mixing weights over the value vectors — so the output can never leave the convex hull of V. Watch the row sum stay pinned at 1.000: softmax normalizes over keys, never over queries. Turn the scale off and the score spread jumps by exactly √8 ≈ 2.83×, the rows sharpen toward one-hot, and the effective-key count collapses — that is softmax saturating into its low-gradient regime, which is the whole reason for the 1/√d_k factor. This toy has no positional encoding, so two copies of the same word get identical embeddings; only the causal mask makes their rows differ. New to this? Press Start guided tour at the top for a seven-step walkthrough that drives these same controls.

Verification is exact, not approximate: a drafted token x ~ q is accepted with probability min(1, p(x)/q(x)); on rejection the replacement is resampled from the residual (p−q)⁺. Summed over both paths a token is emitted with probability min(p,q) + (p−q)⁺ = p(x) — the target's own distribution — no matter how bad the drafter q is. α changes how many tokens come out per parallel pass, never which distribution they come from. With i.i.d. acceptance, E[tokens per target call] = (1−α^γ+1)/(1−α) and cost = 1+γc target-equivalent passes, so speedup = E/(1+γc). The γ-th extra draft slot buys only α^γ expected tokens but always costs c, so once α^γ+1/c drops below the current speedup, growing γ hurts — that is the γ* marked on the chart. Caveats this idealization hides: acceptance is not really i.i.d. (it decays along the chain), single-chain drafting is weaker than tree drafting, and at large batch sizes decode is compute-bound so the “free” verification FLOPs stop being free.

The takeaway: an SSM layer never builds a T×T score matrix. It carries N numbers per channel and folds each token in with one multiply-add: h_t = Ā_th_t-1 + B̄_tx_t, y_t = C_t·h_t, with Ā_t = exp(Δ_tA) on a diagonal, negative-real A. In S4 the parameters are constant, so the layer is a fixed convolution: it decays at one rate forever and cannot be told to drop or hold anything — switch to "Constant" and watch <sep> and <pad> stop resetting and stop coasting, becoming ordinary decay steps. In Mamba, Δ_t, B_t, C_t are linear functions of the token, and Δ_t becomes a gate: large Δ drives Ā = exp(ΔA) → 0 and erases the state (a reset at <sep>); Δ → 0 drives Ā → 1 and B̄ → 0, so the token is skipped and the state coasts (<pad>). Sequential order is not a training bottleneck, because the recurrence is an associative scan — the tree above reproduces the loop to the last bit in ceil(log₂T) rounds. And the punchline is the memory chart: a transformer's KV cache is O(T) and never stops growing, while the recurrent state is O(1) — a fixed 304 KiB/layer here, past the crossover at only ~38 tokens and ~3450× smaller at 128k. The bill for that constant state is that everything the model still needs must fit in N numbers, which is exactly why selectivity — deciding what to keep — had to be invented.

Every rule starts at the same place: at N = 1 best-of-N, majority vote and weighted vote all equal pass@1. Everything above that line is selection, and the dashed pass@N curve is the ceiling selection could ever reach — coverage keeps climbing roughly log-linearly in N (Brown et al., Large Language Monkeys, 2024) while every selection rule plateaus below it. The gap between them is exactly what a better verifier buys you, which is why pass@k and maj@k must be reported separately. The verifier is a real model here, not a fudge factor: a candidate's score is v = ±d/2 + N(0,1) (plus for correct, minus for wrong), so the two score distributions are unit Gaussians separated by d and the verifier's ROC AUC is exactly Φ(d/√2) — the slider inverts that, d = √2·Φ-1(AUC). The calibrated posterior follows from the likelihood ratio: log-odds = logit(p) + d·v, so P(correct | v) = σ(logit(p) + d·v). Best-of-N takes argmax v; weighted vote sums those posteriors per distinct answer. Drag AUC to 0.50: d = 0, the score is pure noise, best-of-N degenerates into picking a random one of the N samples and its curve goes flat at pass@1 no matter how much you spend. That is the generator-vs-verifier split in one drag — the samples contained the answer, nothing could find it. Majority vote needs no verifier but assumes errors are diverse. Push wrong-answer agreement up and the wrong chains pile onto one attractor answer; once p < 0.5 with high agreement, self-consistency converges to the wrong answer and accuracy falls as N grows. Exact check: set agreement = 1.00, σ = 0, p = 0.60 — voting is now binary and maj@15 is exactly the binomial tail P(Bin(15, 0.6) ≥ 8) = 0.787; the widget reproduces that to within a Monte-Carlo standard error or two (2000 seeded problems, s.e. √(.787·.213/2000) ≈ ±0.009 — hit Resample a few times to watch the sampling noise move it around). Note that maj@16 gives the same 0.787: an 8–8 tie is broken by earliest occurrence, which under exchangeability is a fair coin, so an even N buys nothing over the odd N below it. Beam search spends the same budget differently. Both allocations generate exactly N × D step-completions; flat sampling puts all of it in width (N independent chains, never pruned), beam search keeps only w prefixes and prunes with the step verifier at every one of the D depths, respawning N children per depth spread over the w survivors (⌊N/w⌋ each, with the first N mod w getting one extra, so the cost matches flat even when w does not divide N). A weak step verifier applied D times beats a strong one applied once — this is the whole PRM-over-ORM argument, and you can read it straight off the widget. Set N = 32, D = 6, w = 2, p = 0.35, highlight best-of-N, and sweep AUC: at 0.50 beam and flat are both pinned at pass@1 ≈ 34% (uniform pruning keeps live prefixes at exactly rate q per step, so nothing is gained); at 0.52 — a verifier barely distinguishable from a coin — beam already reaches ~43% against best-of-32's ~37%; by 0.60 it is ~68% against ~49%. Mild selection pressure compounds across depth. The bill arrives as coverage: only w prefixes survive, so beam coverage collapses far below flat coverage (99% at N = 32), and the diversity you destroyed is unrecoverable — a large part of why DeepSeek-R1 abandoned PRM-guided search in favour of outcome-only RL, and why narrow beams with dull verifiers (w = 1, AUC near 0.5) fall all the way back to a single random-walk chain. Set w ≥ N (e.g. N = 4, w = 8) and nothing is ever pruned: beam search reduces exactly to flat sampling and the two curves coincide up to seed noise. The reported α is a least-squares fit of log(1 - accuracy) = log ε - α log N over the highlighted curve, i.e. the test-time scaling law acc ≈ 1 - εN-α. Model caveats: samples are i.i.d. (no self-correction or sequential revision), verifier scores are Gaussian and calibrated against the population prior p rather than each problem's own pi (a real reward model is neither Gaussian nor calibrated, and RL will hack it), and a chain is correct only if all D steps are — no lucky recoveries.

The takeaway: byte-level BPE guarantees every possible input is encodable — it bottoms out at raw bytes, so there is no <unk> token — while learned merges compress frequent substrings into single tokens. Push the merge slider and watch bytes / token climb: that ratio is your context-window and inference-cost multiplier. Merges are chosen purely by adjacent-pair frequency in the training corpus, so a tokenizer inherits its corpus's biases (English prose compresses well; unseen scripts fall back toward one token per byte). Decoding is just concatenating each token's stored bytes, which is why encode → decode reproduces the input's bytes exactly, even though a single merged token's bytes need not be valid UTF-8 on their own.

Training FLOPs use the standard 6ND rule (forward+backward, dense). Memory uses the Adam mixed-precision 16 bytes/parameter optimizer-state budget (fp16 param+grad + fp32 master+m+v), sharded evenly across GPUs (ZeRO-3/FSDP) — activation memory is extra and not shown.

The residual stream is a width-d highway: attention and the MLP only ever add to it (never overwrite it), so its shape [T,d] is identical at every checkpoint no matter how many layers you stack. Everything you see is computed from genuinely fixed, seeded (untrained) weights — the shapes and the arithmetic at every stage are exactly what a real GPT computes, but since nothing has been trained, the final next-token distribution is linguistically arbitrary, not a meaningful prediction. New to this? Start guided tour at the top walks the whole pass in 15 steps, naming every shape; exit at any point to keep exploring on your own.

An image becomes a sequence — that is the whole trick. Patchify cuts H×W into N = (H/P) · (W/P) non-overlapping P×P patches; each is flattened to P²C numbers and hit with one shared matrix E to become a D-dimensional token — arithmetically the same thing as a Conv2d(C, D, kernel_size=P, stride=P), up to the order in which each patch's pixels are flattened (this widget uses the chapter's (P, P, C) einops order; Conv2d uses (C, P, P), a column permutation of E). Prepend a learned [CLS] token, add a learned positional embedding to every position, and an unmodified transformer encoder does the rest. E, x_cls and E_pos are parameters: they depend on the seed and nothing else, so they hold perfectly still while you repaint — only the patch rows move. Patch size is the compute dial: halving P quadruples N and multiplies attention's (N+1)² score matrix by about 16 — which is why ViT-B/16 (196 patch tokens) is cheap while a 1024 px input at P = 16 (4096 tokens) needs FlashAttention. Positional embeddings are not decoration: switch them off and two identical patches produce byte-identical tokens; self-attention is permutation-invariant over its inputs, so the model then literally cannot tell top-left from bottom-right (watch the duplicate-token counter on the Checker or Stripes preset). Read the attention honestly: these weights are seeded-random, not trained. LayerNorm (Pre-LN, exactly as in the chapter's ViTBlock) puts every token on the sphere of radius √D, so with the content-similarity head (W_K = W_Q) the score q·k/√D is an unbiased estimate of √D · cos(ẑ_i, ẑ_j): a patch attends to patches that look like it, which is the texture/locality behaviour real ViT heads do learn. The weight vs. looks-like-the-query chip measures exactly that, and typically lands around r ≈ 0.6–0.8 for this head (median 0.64 over the first 60 seeds). Untick the box for independent random W_Q, W_K and that correlation collapses to about zero on average (roll the seed a few times to see past the noise — any single seed can land anywhere): the head still paints a confident-looking map, it just has nothing to do with what the query looks like. The architecture supplies the mechanism; training supplies the meaning. Query the [CLS] token and you see the same lesson from the other side: its row is usually sharper than a patch's (lower entropy on about two seeds in three), yet [CLS] has no pixels at all — it is one learned constant vector, so at initialisation its “favourite” patches are simply whichever ones happen to align with a random direction. Object-centric CLS attention — the DINO maps later in this chapter — is a product of training, not of the architecture.

The mechanism. At every step the previous token is hashed (with a secret key) into a pseudo-random split of the vocabulary: a γ-fraction green list gets its logits raised by +δ before the softmax, so the model samples green tokens more often than chance with no change to its weights. The grid shows one step: each cell is a vocabulary entry with its probability before the boost (grey bar) next to its probability after (coloured bar) — δ visibly pushes mass from the red cells onto the green ones. Detection replays the same hash on a candidate text, counts observed green tokens g out of T, and forms z = (g − γT) / √(T γ(1−γ)). Under the null each token is green independently with probability γ, so human z wanders around 0 like a random walk, while watermarked text with green rate p > γ gives z ≈ (p − γ) √T / √(γ(1−γ)) — it grows like √T, not linearly, which is why detection needs a few hundred tokens rather than a few dozen. Three tradeoffs to play with. (1) Raising δ raises p and reaches z* = 4 in fewer tokens, but the KL(boosted ‖ original) statistic below shows exactly how much distribution you paid for it. (2) σ controls how peaked the model's own distribution is: crank it up (low entropy, near-greedy text like code or a quoted fact) and a fixed δ can no longer move the argmax — the green rate collapses toward γ and z stalls below threshold. That entropy ceiling, not the statistics, is the real limit of logit-bias watermarks. (3) γ is not monotone: a small green list makes each green hit more surprising (more z per green token) but also leaves δ fewer good tokens to promote, so for a fixed δ the z-score peaks at an intermediate γ (try γ ≈ 0.25, the KGW paper's recommendation) and collapses as γ → 1, where green is nearly free and the test is powerless. Note that the null variance γ(1−γ) is largest at γ = 0.5, so γ trades surprise-per-token against the width of the null — it is not a "more green is better" dial.