13.6 LLM-Specific Deep-Dive Questions¶

The breadth round (ML Breadth: Rapid-Fire Concepts & Model Answers) tests whether you know machine learning. The LLM deep-dive tests whether you actually understand the modern stack — the part that didn’t exist when most ML textbooks were written. At Google, this often arrives mid-conversation: the interviewer name-drops “KV cache” or “GQA” or “DPO” and watches you either light up or stall. A strong candidate doesn’t recite a definition; they explain the mechanism, the failure mode it fixes, the cost, and the tradeoff.

This chapter is a question bank with model answers, organized by topic. Each answer is written the way you should say it out loud: lead with the one-sentence essence, then the mechanism, then a number or a tradeoff. We cross-link to the full chapters so you can go deeper, but everything you need to survive the question is here. Treat the code blocks as the reconstructions you should be able to whiteboard from memory under pressure — if you can write the eight-line attention and the online-softmax recurrence, you have already beaten most candidates.

A meta-note on style: when you don’t know something, say what you do know and reason toward the rest. “I haven’t read the EAGLE paper closely, but speculative decoding generally works by…” is a strong answer. Bluffing a fabricated number is the single fastest way to lose an LLM interviewer’s trust.

Attention, RoPE, KV-Cache & FlashAttention¶

These four are the load-bearing wall. If you can’t explain them crisply, the interviewer learns everything they need in five minutes.

Explain self-attention from scratch¶

Essence. Attention lets every token mix in information from every other token, weighted by learned relevance. Each token emits a query, every token exposes a key and a value; the query dots against all keys to produce weights, and the output is the weighted sum of values.

The \(\sqrt{d_k}\) divisor is not cosmetic: if \(q,k\) have unit-variance components, \(q\cdot k\) has variance \(d_k\), so without scaling the logits grow with head dimension, the softmax saturates, and gradients vanish. You should be able to reconstruct this in a few lines:

import torch, torch.nn.functional as F

def attention(q, k, v, causal=True):
    # q,k,v: (batch, heads, seq, d_head)
    d_head = q.shape[-1]
    scores = (q @ k.transpose(-2, -1)) / d_head ** 0.5   # (b,h,T,T)
    if causal:
        T = q.shape[-2]
        mask = torch.triu(torch.ones(T, T, device=q.device), diagonal=1).bool()
        scores = scores.masked_fill(mask, float("-inf"))  # no peeking ahead
    weights = F.softmax(scores, dim=-1)                   # rows sum to 1
    return weights @ v                                    # (b,h,T,d_head)

The causal mask is what makes a decoder-only LM autoregressive: position \(t\) may attend to \(\le t\) only. The full development — why dot-product, why multi-head, the geometry — is in The Attention Mechanism From Scratch.

Cost. The \(QK^\top\) matrix is \(T\times T\), so attention is \(O(T^2 d)\) compute and, naively, \(O(T^2)\) memory. That quadratic is the root cause of half the engineering in this book.

Explain RoPE and why it beats learned positional embeddings¶

Essence. Rotary Position Embedding encodes position by rotating the query and key vectors by an angle proportional to their absolute position, in 2D subspaces. Because a dot product of two rotated vectors depends only on the difference of their rotation angles, the attention score between positions \(m\) and \(n\) becomes a function of \(m-n\) — RoPE injects relative position while applying an absolute rotation.

For a 2D pair \((x_1,x_2)\) at position \(m\) with frequency \(\theta\):

The head dimension is split into \(d/2\) such pairs, each with its own frequency \(\theta_i=\text{base}^{-2i/d}\) (base typically 10000). Low-index pairs rotate fast (capture local order); high-index pairs rotate slowly (capture long-range position).

def build_rope_cache(seq_len, d_head, base=10000.0, device="cpu"):
    # inverse frequencies: one per dimension-pair
    inv_freq = 1.0 / (base ** (torch.arange(0, d_head, 2, device=device).float() / d_head))
    t = torch.arange(seq_len, device=device).float()
    freqs = torch.outer(t, inv_freq)            # (seq, d_head/2)
    return torch.cos(freqs), torch.sin(freqs)

def apply_rope(x, cos, sin):
    # x: (b, h, seq, d_head); rotate consecutive pairs (x_even, x_odd)
    x1, x2 = x[..., 0::2], x[..., 1::2]
    cos, sin = cos[None, None], sin[None, None]
    rx1 = x1 * cos - x2 * sin
    rx2 = x1 * sin + x2 * cos
    out = torch.empty_like(x)
    out[..., 0::2], out[..., 1::2] = rx1, rx2
    return out

Why it wins. (1) It’s relative, which generalizes better than absolute position. (2) It’s applied to Q and K only — no extra parameters, no added embedding to the residual stream. (3) It extends to longer contexts via frequency tricks (NTK scaling, YaRN) without retraining from scratch — you just stretch or interpolate the angles. This is the foundation of context extension; see Positional Encodings: Sinusoidal, Learned, RoPE & ALiBi and Long-Context Pretraining & Context Extension.

What is the KV cache and why does it dominate inference memory?¶

Essence. During autoregressive decoding, generating token \(t+1\) requires attending over all previous keys and values. Those K and V tensors depend only on past tokens, not the current one, so we cache them and reuse them — turning per-step attention from \(O(t^2)\) recomputation into \(O(t)\).

The cache size is the number every LLM systems interviewer wants you to derive on the spot:

where 2 is for K and V, \(L\) is layers, \(n_{\text{kv}}\) is KV heads, \(T\) is sequence length, \(B\) is batch. The KV cache, not the weights, is what limits how many concurrent requests you can serve — see the worked example below and The Anatomy of LLM Inference: Prefill, Decode & The KV Cache. Its fragmentation problem is solved by PagedAttention (PagedAttention & KV-Cache Memory Management).

Explain FlashAttention without hand-waving¶

Essence. FlashAttention computes exact attention without ever materializing the \(T\times T\) score matrix in slow GPU memory (HBM). It tiles Q, K, V into blocks, keeps a running softmax in fast on-chip SRAM, and is therefore IO-bound-aware: it trades a little extra compute for a massive reduction in HBM traffic, which is the actual bottleneck.

The trick is the online softmax: you can compute a numerically-stable softmax-weighted sum in one pass if you carry a running max \(m\) and running denominator \(\ell\), rescaling the accumulator when a new block reveals a larger max.

def online_softmax_attention(q, K_blocks, V_blocks):
    # q: (d,)  one query; K_blocks/V_blocks: list of (block, d) tiles
    import math
    m = float("-inf")      # running max of logits
    l = 0.0                # running sum of exp(logit - m)
    acc = torch.zeros_like(q)   # running output (un-normalized)
    for Kb, Vb in zip(K_blocks, V_blocks):
        s = (Kb @ q) / q.shape[-1] ** 0.5      # logits for this tile
        m_new = max(m, s.max().item())
        # rescale old accumulator + denom to the new max
        scale = math.exp(m - m_new)
        p = torch.exp(s - m_new)               # tile probabilities
        l = l * scale + p.sum().item()
        acc = acc * scale + (p[:, None] * Vb).sum(0)
        m = m_new
    return acc / l

That recurrence is the whole idea — FlashAttention 1 (Dao et al.) is this, tiled and fused into a CUDA kernel; FA2 improves work partitioning across warps; FA3 adds FP8 and Hopper async copies. The reason it’s faster despite recomputation in the backward pass: attention is memory-bandwidth bound, not compute bound, so cutting HBM reads/writes from \(O(T^2)\) to \(O(T)\) wins. Full treatment: FlashAttention I: IO-Awareness & The Online Softmax and FlashAttention 2 & 3.

Why GQA, MQA & MLA¶

Walk through MHA → MQA → GQA¶

Essence. These are points on a spectrum trading attention quality for KV-cache size. Multi-Head Attention (MHA) gives every query head its own K and V. Multi-Query Attention (MQA) shares one K/V across all query heads — minimal cache, but a quality hit and training instability. Grouped-Query Attention (GQA) is the compromise: split query heads into \(g\) groups, each group shares one K/V head.

MHA:  Q heads: [q0 q1 q2 q3 q4 q5 q6 q7]
      K/V:     [k0 k1 k2 k3 k4 k5 k6 k7]    (8 KV heads — big cache)

GQA:  Q heads: [q0 q1 q2 q3 | q4 q5 q6 q7]
      K/V:     [   k0       |    k1     ]    (2 KV heads — 4x smaller)

MQA:  Q heads: [q0 q1 q2 q3   q4 q5 q6 q7]
      K/V:     [          k0            ]    (1 KV head — smallest)

The KV cache shrinks by the factor \(n_{\text{head}}/n_{\text{kv}}\). Since the cache, not FLOPs, bounds concurrency during decode, GQA directly multiplies your serving throughput — that’s the answer to “why GQA”: it cuts KV-cache memory by the group factor with negligible quality loss, raising the batch size you can serve and thus tokens/sec/GPU.

What is MLA and how does it differ?¶

Essence. Mult-head Latent Attention (DeepSeek) goes further: instead of caching K and V directly, it caches a low-rank latent that is up-projected to K and V on the fly. The cache stores a small compressed vector per token, decoupling cache size from head count almost entirely, while preserving more expressive per-head attention than GQA. It also folds RoPE in via a separate decoupled key path. Tradeoff: more compute and more architectural complexity at train time. Full comparison: Multi-Head Attention, MQA, GQA & MLA.

Preference Optimization: DPO vs PPO vs GRPO¶

This cluster separates people who’ve read the RLHF papers from people who’ve trained the models. The full pipeline is in The RLHF Pipeline & Reward Modeling.

Explain the classic RLHF/PPO pipeline¶

Essence. Three stages: (1) SFT — supervised fine-tune on demonstrations. (2) Reward model — collect pairwise human preferences and train a model \(r_\phi(x,y)\) to score responses, using the Bradley–Terry loss. (3) PPO — optimize the policy \(\pi_\theta\) to maximize reward while a KL penalty keeps it near the SFT reference \(\pi_{\text{ref}}\):

PPO needs four models in memory simultaneously: policy, reference, reward, and a value/critic network for advantage estimation. It’s powerful but heavy, sensitive to hyperparameters, and notoriously fiddly. See Policy Gradients & PPO for Language Models.

Why DPO, and what is it really doing?¶

Essence. Direct Preference Optimization removes the reward model and the RL loop entirely. The key insight: the RLHF objective has a closed-form optimal policy, and you can algebraically invert it so the reward is expressed in terms of the policy itself. Substituting into the Bradley–Terry preference likelihood yields a simple supervised classification loss on preference pairs \((y_w \succ y_l)\):

The model implicitly becomes its own reward model. Here is the loss in full — note it’s just two forward passes and a log-sigmoid:

import torch.nn.functional as F

def dpo_loss(policy_logp_w, policy_logp_l,    # sum log-prob of chosen/rejected under policy
             ref_logp_w, ref_logp_l,          # ...under frozen reference
             beta=0.1):
    # "implicit reward" = beta * log-ratio of policy to reference
    pi_logratio  = policy_logp_w - policy_logp_l     # policy prefers chosen by this much
    ref_logratio = ref_logp_w   - ref_logp_l         # reference's baseline preference
    logits = beta * (pi_logratio - ref_logratio)     # how much we *increase* the margin
    loss = -F.logsigmoid(logits).mean()              # push chosen above rejected
    # implicit rewards (for logging margins / accuracy):
    chosen_reward   = beta * (policy_logp_w - ref_logp_w).detach()
    rejected_reward = beta * (policy_logp_l - ref_logp_l).detach()
    return loss, chosen_reward, rejected_reward

Tradeoff. DPO is far simpler, more stable, and cheaper (two models, no rollouts). Its weaknesses: it learns only from the offline preference pairs you collected — no exploration — so it can over-fit to the dataset’s quirks and is prone to length bias and pushing down the probability of both responses if not regularized. On-policy methods (PPO, GRPO) can keep improving by generating fresh samples. See Direct Preference Optimization & Its Variants.

What is GRPO and why did it take over for reasoning?¶

Essence. Group Relative Policy Optimization (DeepSeek) is PPO without the critic. Instead of a learned value network to estimate the baseline, it samples a group of \(G\) responses per prompt, scores each, and uses the group’s mean reward as the baseline. The advantage of a response is just its reward standardized within the group:

def grpo_advantages(rewards):
    # rewards: (group_size,) scalar reward per sampled completion for ONE prompt
    mean = rewards.mean()
    std  = rewards.std() + 1e-6
    return (rewards - mean) / std        # within-group standardized advantage

Why it took over. (1) It drops the value network — halving memory and removing the hardest-to-tune component. (2) The group baseline is well-suited to verifiable rewards (math/code where reward is 0/1 correctness), the engine behind reasoning models. A correct answer in a group of mostly-wrong answers gets a big positive advantage; that’s a clean learning signal. See GRPO, RLOO & Critic-Free RL and RL with Verifiable Rewards. The KL term and advantage normalization details matter a lot in practice — Advantage Estimation, KL Control & Stability Tricks.

Quantization¶

The interviewer wants to know you understand what you’re quantizing (weights? activations? KV cache?) and why it works without destroying the model.

How does quantization work and why doesn’t it break the model?¶

Essence. Quantization maps high-precision weights (fp16/bf16) to low-bit integers (int8, int4) using a scale (and optional zero-point), reconstructing approximate values at compute time. It works because (1) neural nets are over-parameterized and robust to weight noise, and (2) most of the cost of LLM inference is memory bandwidth — moving weights from HBM — so storing weights in 4 bits means 4× fewer bytes to move, a near-4× decode speedup, even if you up-cast to do the matmul.

Symmetric per-group quantization in a few lines:

def quantize_int4_symmetric(w, group_size=128):
    # w: (out, in) weight matrix; quantize along input dim in groups
    out, inn = w.shape
    w = w.reshape(out, inn // group_size, group_size)
    absmax = w.abs().amax(dim=-1, keepdim=True)          # per-group scale
    scale = absmax / 7.0                                  # int4 range [-8,7] -> use 7
    q = torch.clamp(torch.round(w / scale), -8, 7)        # quantized integers
    return q.to(torch.int8), scale                        # store 4-bit packed + scale

def dequantize(q, scale, group_size=128):
    out = q.shape[0]
    return (q.float() * scale).reshape(out, -1)           # approximate original

PTQ vs QAT, and what’s the hard part?¶

Essence. Post-Training Quantization (PTQ) quantizes a finished model with a small calibration set — fast, no retraining (GPTQ, AWQ, SmoothQuant). Quantization-Aware Training (QAT) simulates quantization during training so the model learns to be robust — more accurate at very low bits, but expensive. The hard part is outlier activations: a few channels have huge magnitudes that blow up the quantization range and destroy precision for everyone else. SmoothQuant migrates the difficulty from activations into weights via a per-channel scale; AWQ protects the salient weight channels identified by activation magnitude; GPTQ uses second-order (Hessian) information to quantize weights while compensating for the error. KV-cache quantization is a separate axis that directly extends context length. Full treatment: Quantization I: PTQ and Quantization II: Formats & QAT; the float side connects to Mixed Precision, bf16 & FP8 Training.

Scaling Laws, MoE & Architecture¶

State the Chinchilla result and its practical consequence¶

Essence. Kaplan et al. (2020) found loss falls as a power law in parameters \(N\), data \(D\), and compute \(C\). Chinchilla (Hoffmann et al., 2022) refined the compute-optimal allocation: for a fixed compute budget, you should scale parameters and tokens roughly equally — about 20 tokens per parameter — whereas earlier models like GPT-3 were badly under-trained (too big, too little data). The loss model:

with \(\alpha,\beta\approx 0.34\). Given compute \(C\approx 6ND\) FLOPs, you minimize \(L\) subject to that constraint.

The practical twist interviewers love. Chinchilla optimizes training compute. But in production, inference compute dominates over a model’s lifetime, so it’s rational to “over-train” a smaller model on far more than 20 tokens/param (Llama-3 used trillions of tokens for 8B). You trade extra training cost for a permanently cheaper, faster model to serve. Knowing this distinction — compute-optimal training vs inference-optimal deployment — is the mark of someone who’s actually shipped. See Scaling Laws: Kaplan, Chinchilla & Beyond.

Explain Mixture-of-Experts and the routing problem¶

Essence. An MoE replaces the dense feed-forward layer with many expert FFNs and a router that sends each token to a small subset (top-\(k\), often \(k=2\)). The result: you scale total parameters (knowledge capacity) while keeping active parameters per token — and thus FLOPs — roughly constant. A model can have 600B total but activate only ~37B per token.

def moe_layer(x, experts, router, k=2):
    # x: (tokens, d_model); experts: list of FFN modules; router: Linear(d_model -> n_experts)
    logits = router(x)                          # (tokens, n_experts)
    weights, idx = torch.topk(logits.softmax(-1), k, dim=-1)   # pick top-k experts
    out = torch.zeros_like(x)
    for slot in range(k):
        for e in range(len(experts)):
            mask = idx[:, slot] == e            # tokens routed to expert e in this slot
            if mask.any():
                out[mask] += weights[mask, slot:slot+1] * experts[e](x[mask])
    return out

The hard parts. (1) Load balancing — without an auxiliary loss, the router collapses to a few favorite experts; you add a balancing loss (or use loss-free bias tricks) to spread tokens. (2) All-to-all communication — experts are sharded across GPUs (expert parallelism), so routing means shuffling tokens across the network, which can bottleneck. (3) Training instability and the discrete, non-differentiable top-\(k\). The tradeoff: huge memory footprint (all experts must be resident) for lower FLOPs/token. Deep dive: Mixture-of-Experts (MoE) Architectures; the parallelism in Distributed Training II.

Name three modern architecture choices and why¶

Quick-fire, the way you’d rattle them off: Pre-norm (LayerNorm/RMSNorm before the sublayer) for stable gradients in deep stacks; RMSNorm over LayerNorm (drops the mean-centering, cheaper, works as well); SwiGLU activation (gated GLU variant, better than ReLU/GELU at equal params); RoPE for relative position; GQA for cheap KV cache; no bias terms in linears (free stability). See Modern Architecture Improvements & Design Choices.

Inference Acceleration & Hallucination¶

Explain speculative decoding¶

Essence. Decode is memory-bandwidth bound: each step loads the entire model to produce one token, wasting compute. Speculative decoding uses a small, fast draft model to propose \(k\) tokens cheaply, then the big target model verifies all \(k\) in a single parallel forward pass. A rejection-sampling acceptance rule guarantees the output distribution is identical to the target model’s — it’s exact, not approximate. You get a speedup equal to the average number of tokens accepted per target pass.

def speculative_step(target, draft, prefix, k=4):
    # 1) draft proposes k tokens autoregressively (cheap)
    draft_tokens, draft_probs = draft.sample(prefix, n=k)
    # 2) target scores all k+1 positions in ONE forward pass (parallel)
    target_probs = target.forward(prefix + draft_tokens)   # (k+1, vocab)
    # 3) accept/reject each drafted token to preserve target's distribution
    accepted = []
    for i, tok in enumerate(draft_tokens):
        p, q = target_probs[i][tok], draft_probs[i][tok]
        if torch.rand(1) < min(1.0, p / q):     # accept with prob min(1, p/q)
            accepted.append(tok)
        else:
            # reject: resample from the adjusted residual distribution (p - q)_+
            accepted.append(sample_residual(target_probs[i], draft_probs[i]))
            return accepted                      # stop at first rejection
    accepted.append(sample(target_probs[k]))     # bonus token: all k accepted
    return accepted

Why it’s free quality-wise: the math guarantees the marginal output distribution equals the target’s. Variants: Medusa adds extra decoding heads to the target itself (no separate draft); EAGLE drafts in feature space; lookahead uses n-gram jacobi iteration. The speedup depends entirely on draft acceptance rate — a well-aligned draft on predictable text can accept 3–4 tokens/pass. Full coverage: Speculative Decoding: Draft Models, Medusa, EAGLE & Lookahead. Why decode is memory-bound in the first place: The Anatomy of LLM Inference.

What causes hallucination, mechanistically?¶

Essence. Hallucination — confident, fluent, false output — isn’t a bug in one place; it’s the product of how LLMs are trained and decoded. Give a layered answer:

1. The objective rewards plausibility, not truth. Next-token prediction (and the MLE/cross-entropy loss) optimizes for likely continuations given the training distribution. The model has no built-in notion of factual grounding; it interpolates over patterns. See The Pretraining Objective & Loss.
2. No epistemic uncertainty signal. The model outputs a probability distribution over tokens, but a high-probability token is not a calibrated “I’m sure this is true.” After RLHF, models often become worse-calibrated — confidently wrong — because preference data rewards confident, helpful-sounding answers.
3. Training-time incentives to guess. If the SFT/RL data punishes “I don’t know” and rewards attempts, the model learns that guessing beats abstaining — a direct cause of fabrication on the long tail.
4. Knowledge gaps & staleness. Facts absent from (or rare in) pretraining, or events after the cutoff, force the model to extrapolate. Closed-book recall of rare entities is where hallucination spikes.
5. Decoding pressure. Sampling (temperature, top-p) injects randomness that can tip a borderline-correct answer into a wrong one; the model must emit some token even when no continuation is well-supported.

Mitigations map onto each cause: RAG grounds answers in retrieved evidence (Retrieval-Augmented Generation Architectures); calibration / abstention training rewards “I don’t know”; verifiers and tool use check claims (Tool Use & Function Calling); LLM-as-judge / self-consistency catches inconsistency (LLM-as-a-Judge); lower temperature reduces stochastic errors. The honest framing for an interview: hallucination is intrinsic to a likelihood model with no grounding; we manage it with retrieval, verification, and calibration rather than “fixing” it.

Putting It Together: How to Answer in the Room¶

A meta-pattern for every question above:

1. One-sentence essence first. “GQA shares K/V across groups of query heads to shrink the KV cache.” Don’t make them wait.
2. Mechanism. The formula or the data flow — show you understand how, not just that.
3. A number or a tradeoff. “8× smaller cache → 8× more concurrent requests.” Magnitudes prove you’ve touched the system.
4. The failure mode it addresses. Every technique exists because something broke. Naming the break shows depth.
5. Honest edges. “DPO is simpler but can’t explore; that’s why reasoning models use GRPO.” Knowing the limits is more impressive than evangelizing.

Practice saying these out loud. The difference between a hire and a no-hire on the LLM deep-dive is rarely knowledge — it’s fluency: the ability to go from “explain flash attention” to a clear, structured, three-layer answer in twenty seconds without a script. The chapters cross-linked throughout give you the depth; this chapter gives you the compression.

Further reading¶

• Vaswani et al., Attention Is All You Need (2017) — the transformer and scaled dot-product attention.
• Su et al., RoFormer: Enhanced Transformer with Rotary Position Embedding — RoPE.
• Dao et al., FlashAttention and FlashAttention-2 — IO-aware exact attention.
• Ainslie et al., GQA: Training Generalized Multi-Query Transformer Models — grouped-query attention and uptraining.
• Ouyang et al., Training Language Models to Follow Instructions with Human Feedback (InstructGPT) — the RLHF/PPO pipeline.
• Rafailov et al., Direct Preference Optimization — DPO and the implicit reward derivation.
• Shao et al. / DeepSeek, DeepSeekMath (GRPO) and DeepSeek-V3 / R1 — GRPO, MLA, and the reasoning recipe.
• Hoffmann et al., Training Compute-Optimal Large Language Models (Chinchilla); Kaplan et al., Scaling Laws for Neural Language Models.
• Shazeer et al., Outrageously Large Neural Networks (Sparsely-Gated MoE); Fedus et al., Switch Transformers.
• Leviathan et al., Fast Inference from Transformers via Speculative Decoding; Chen et al., Accelerating LLM Decoding with Speculative Sampling.
• Frantar et al., GPTQ; Lin et al., AWQ; Xiao et al., SmoothQuant — post-training quantization.