12.7 Online Evaluation: A/B Testing, Canaries & Guardrail Metrics¶

Offline benchmarks are necessary but not sufficient. A model that tops your held-out evaluation set may still lose you users, inflate costs, or silently regress on edge cases that only emerge at production scale. The gap between an offline MMLU score and real user satisfaction is not a failure of offline evaluation — it is a structural property: users are not IID draws from a benchmark corpus, and the distribution of inputs, intents, and behaviors that matter commercially is only observable in production.

This chapter closes the loop. We cover how to design statistically sound A/B and interleaving experiments for LLM features, which guardrail metrics to instrument and how to prevent them from being gamed, how to validate that the experiment itself is trustworthy (sample ratio mismatch, clustered standard errors for per-event metrics), how to make decisions faster with CUPED variance reduction and sequential testing, how to roll out safely via shadow deployments and canaries, and how to keep live judges scoring sampled traffic so you stay honest about quality even after launch. We also examine the systematic biases — novelty effects, feedback loops, position bias — that corrupt online signals if left unaddressed.

Related chapters that provide useful context: The Evaluation Problem & Benchmark Landscape, LLM-as-a-Judge & Automated Evaluation, Statistical Rigor in Evaluation: Confidence Intervals & Significance, Data Flywheels & Continuous Improvement, and Observability, Logging & LLMOps.

The Offline–Online Gap and Why It Exists¶

Before designing experiments, we need to understand why offline metrics so often disagree with online outcomes.

Distribution shift. A benchmark is a frozen snapshot. Real users evolve: they learn to phrase queries that exploit a new model’s strengths, or they hit topics that weren’t in your eval set. A model fine-tuned on last quarter’s support tickets may degrade subtly on the new product surface launched this quarter.

Task proxies are imperfect. ROUGE-L measures n-gram overlap; users care about whether the answer resolved their problem. Perplexity on a held-out set predicts coherence but not helpfulness. Even carefully designed human evals may not reflect actual user intent distributions.

Unobservable label noise. Offline evals often rely on human annotations, which have inter-annotator disagreement, recency bias, and annotator fatigue. The “ground truth” is fuzzier than it appears.

Survivor bias in retrieval. In RAG systems, offline retrieval evals are run on queries that have known relevant documents. Live traffic contains queries for which no document exists — a silent failure mode invisible in offline settings.

The practical implication: offline improvements should be treated as evidence for online experiments, not as proof. The question is never “did our offline eval improve?” but “does our offline eval predict online outcomes?”

A useful diagnostic is the offline–online correlation coefficient: track, across many launches, whether a 1% offline improvement predicts a positive online signal. If this correlation is weak (below ~0.5), your offline eval is not a reliable proxy and you are flying blind between offline and live.

Guardrail Metrics: What to Measure and Why¶

Before running experiments, you must decide what you are measuring. For LLM products, metrics fall into three tiers.

Primary metrics (goal metrics)¶

These are the business outcomes you ultimately care about:

• Resolution rate: the fraction of user sessions in which the user’s problem was solved without escalation, re-query, or abandonment. For a customer support bot, this is the north star.
• Task success rate: for agent or coding assistant products, automated verification that the task was completed (tests pass, form submitted, booking confirmed).
• Session engagement: downstream engagement signals (clicks, purchases, return visits) in contexts where helpfulness precedes a downstream action.

Guardrail metrics (do-no-harm metrics)¶

These must not regress below a defined threshold, even if the primary metric improves. Violating a guardrail halts a rollout regardless of primary metric performance.

The regeneration rate deserves special attention: it is a behavioral signal that does not require users to explicitly rate anything, making it much higher-coverage than explicit thumbs-up/down while still directionally reliable.

Sensitivity metrics (canary signals)¶

These are leading indicators with lower latency than the primary metric. Because resolution rate may take days or weeks of data to reach significance (a user may not return to signal resolution), you need sensitive proxies that respond within hours:

• Thumbs-up/thumbs-down ratio: available immediately after each session.
• Session depth: number of follow-up turns (more turns often indicates confusion or failure).
• Copy rate: users copying model output — a strong positive behavioral signal.
• Share rate (where applicable): users sharing the response externally.

Designing A/B Experiments for LLM Features¶

Randomization unit¶

For LLM features, randomization is almost always at the user-session level (randomly assign each user to a variant, then serve that variant consistently throughout the experiment). Randomizing at the request level within a user session violates the stable unit treatment value assumption (SUTVA): a model swap mid-conversation changes the distribution of subsequent turns, introducing contamination.

For products without user identity (anonymous API access), randomize on a stable session token or a hashed combination of IP + user agent. The key property is that the same user sees the same variant across multiple requests in a session.

Control and treatment specification¶

An A/B experiment compares a control (the current production model) against one or more treatments (candidate models or features). Common structures:

For a new product with no established baseline, use an A/A test first: split traffic into two identical control arms and verify that your metrics show no significant difference. An A/A test failing is a red flag for systematic logging errors, selection bias, or a broken randomization layer.

Sample size and statistical power¶

For a two-sided test at significance level \(\alpha\) and power \(1 - \beta\), the required number of users per arm is:

where \(\delta\) is the minimum detectable effect (MDE) you care about, \(\sigma^2\) is the within-arm variance of the metric, and \(z_{\alpha/2}\), \(z_\beta\) are the corresponding normal quantiles (\(z_{0.025} \approx 1.96\), \(z_{0.2} \approx 0.84\) for 80% power).

Running the test¶

import hashlib
import math
from dataclasses import dataclass
from typing import Literal

# -------------------------------------------------------------------------
# Deterministic user assignment: same user → same variant every call.
# We use a SHA-256 hash of (experiment_id + user_id) for uniform assignment.
# -------------------------------------------------------------------------

@dataclass
class Experiment:
    experiment_id: str
    traffic_fraction: float = 1.0   # fraction of total traffic to enroll
    treatment_fraction: float = 0.5  # of enrolled users, fraction to treatment

def assign_variant(
    user_id: str,
    experiment: Experiment,
) -> Literal["control", "treatment", "holdout"]:
    """
    Returns the arm assignment for a given user in a given experiment.
    'holdout' means the user is not enrolled (outside traffic_fraction).
    """
    # Hash to [0, 1) using experiment_id as salt so different experiments
    # produce independent assignments for the same user.
    digest = hashlib.sha256(
        f"{experiment.experiment_id}:{user_id}".encode()
    ).hexdigest()
    bucket = int(digest[:8], 16) / 0xFFFFFFFF  # uniform [0, 1)

    if bucket >= experiment.traffic_fraction:
        return "holdout"

    # Re-hash to assign within enrolled users (avoids correlation between
    # enrollment and treatment assignment).
    digest2 = hashlib.sha256(
        f"{experiment.experiment_id}:assign:{user_id}".encode()
    ).hexdigest()
    bucket2 = int(digest2[:8], 16) / 0xFFFFFFFF

    return "treatment" if bucket2 < experiment.treatment_fraction else "control"


# -------------------------------------------------------------------------
# Simple two-proportion z-test for a Bernoulli metric (e.g. thumbs-up rate).
# -------------------------------------------------------------------------

from scipy import stats
import numpy as np

def two_proportion_z_test(
    n_control: int,
    k_control: int,   # successes in control
    n_treatment: int,
    k_treatment: int,
) -> dict:
    """
    Returns p-value, confidence interval, and relative lift.
    Uses the pooled proportion for the null hypothesis.
    """
    p_c = k_control / n_control
    p_t = k_treatment / n_treatment
    p_pool = (k_control + k_treatment) / (n_control + n_treatment)

    se = math.sqrt(p_pool * (1 - p_pool) * (1/n_control + 1/n_treatment))
    z = (p_t - p_c) / se if se > 0 else 0.0
    p_value = 2 * (1 - stats.norm.cdf(abs(z)))

    # 95% CI on absolute difference using unpooled SE
    se_unpooled = math.sqrt(
        p_c * (1 - p_c) / n_control + p_t * (1 - p_t) / n_treatment
    )
    diff = p_t - p_c
    ci_lo = diff - 1.96 * se_unpooled
    ci_hi = diff + 1.96 * se_unpooled

    return {
        "p_control": p_c,
        "p_treatment": p_t,
        "absolute_lift": diff,
        "relative_lift": diff / p_c if p_c > 0 else float("nan"),
        "z_statistic": z,
        "p_value": p_value,
        "ci_95": (ci_lo, ci_hi),
        "significant": p_value < 0.05,
    }

# Example usage:
result = two_proportion_z_test(
    n_control=10_000, k_control=4_000,
    n_treatment=10_000, k_treatment=4_200,
)
# Expected: ~+5% relative lift, p ≈ 0.001 → significant
print(result)

In production you do not hand-roll the assignment layer, because the hash is the easy half — the hard half is emitting a durable exposure event (user, experiment, variant, timestamp) at the moment the variant is actually served, since that log defines the analysis population. Open-source feature-flag/experiment SDKs do both: GrowthBook (its SDK hashes hashAttribute + seed exactly as above and logs exposures via a trackingCallback), Unleash, and Flagsmith, all behind OpenFeature — the CNCF vendor-neutral flag API — so the evaluation call site does not change when you swap providers. Keep the from-scratch version anyway: it is what you use in a load test or a notebook replay, and it makes the failure modes below legible.

Sanity check first: sample ratio mismatch (SRM)¶

Before reading a single metric, check that the arms received the traffic they were supposed to receive. A sample ratio mismatch — an observed split that deviates from the designed split by more than chance — means the assignment or logging layer is broken, and every downstream number is untrustworthy. Typical causes are very LLM-specific: the treatment model times out on long prompts and those sessions never reach the metrics table; a retry loop re-buckets a user; a bot filter matches one arm’s latency profile more often. Kohavi, Tang & Xu call SRM the single highest-yield trustworthiness check, and it is one chi-square test:

import numpy as np
from scipy import stats

def srm_check(
    n_control: int,
    n_treatment: int,
    expected_treatment_fraction: float = 0.5,
    alarm_p: float = 0.001,   # deliberately strict: SRM is a bug, not an effect
) -> dict:
    """Chi-square goodness-of-fit test on the observed traffic split."""
    total = n_control + n_treatment
    expected = [total * (1 - expected_treatment_fraction),
                total * expected_treatment_fraction]
    observed = [n_control, n_treatment]
    chi2 = sum((o - e) ** 2 / e for o, e in zip(observed, expected))
    p_value = float(stats.chi2.sf(chi2, df=1))
    return {"observed_treatment_fraction": n_treatment / total,
            "chi2": chi2, "p_value": p_value, "srm": p_value < alarm_p}

print(srm_check(100_000, 98_000))  # 1% shortfall at 100k → p ≈ 7e-6, SRM: True
print(srm_check(1_000, 980))       # same 1% shortfall at 1k → p ≈ 0.65, fine

The two calls make the key point: a 1% imbalance is invisible noise at a thousand users and a screaming alarm at a hundred thousand. When SRM fires, do not “adjust for it” — find the bug, fix it, and restart the experiment.

Ratio metrics: the analysis unit is not the randomization unit¶

The \(z\)-test above assumes one Bernoulli observation per randomized unit. Most LLM metrics are not shaped like that: you randomize users but measure per message (thumbs-up per assistant message, tool-call error rate per call, regeneration rate per response). A chatty user contributes dozens of correlated events, so treating events as independent understates the standard error and overstates significance. This is the same clustering problem as in offline evaluation (Statistical Rigor in Evaluation), now at the experiment layer.

The metric is a ratio of sums, \(M = \sum_i Y_i / \sum_i D_i\), over users \(i\) with numerator \(Y_i\) (thumbs-up events) and denominator \(D_i\) (messages). Its variance follows from the delta method:

with \(K\) users and \(\bar{D}\) the mean denominator per user. (The two cheaper alternatives are equally acceptable: collapse to one value per user and run the ordinary test, losing a little power; or bootstrap over users. The delta method is what production stats engines use because it is closed-form and streams.)

def ratio_metric_delta_method(y: np.ndarray, d: np.ndarray) -> dict:
    """
    SE of a ratio-of-sums metric when the randomisation unit is the user:
    y[i] = user i's numerator events, d[i] = user i's denominator events.
    """
    k = len(y)
    d_bar = d.mean()
    m = y.sum() / d.sum()
    var = (np.var(y, ddof=1)
           - 2 * m * np.cov(y, d, ddof=1)[0, 1]
           + m ** 2 * np.var(d, ddof=1)) / (k * d_bar ** 2)
    return {"metric": m, "se": float(np.sqrt(var)), "n_users": k}

# 2,000 users with heterogeneous per-user thumbs-up propensities.
rng = np.random.default_rng(0)
k_users = 2_000
d = rng.poisson(8, k_users) + 1          # messages per user
p_user = rng.beta(4, 6, k_users)         # user-level propensity (the cluster)
y = rng.binomial(d, p_user)              # thumbs-up per user

clustered = ratio_metric_delta_method(y, d)
m = clustered["metric"]
naive_se = np.sqrt(m * (1 - m) / d.sum())  # pretend messages are i.i.d.
print(f"metric={m:.4f}  clustered SE={clustered['se']:.4f}  "
      f"naive SE={naive_se:.4f}  ratio={clustered['se'] / naive_se:.2f}x")
# metric=0.3960  clustered SE=0.0049  naive SE=0.0036  ratio=1.34x

The naive per-message SE is understated by 1.34× here, which inflates the \(z\)-statistic by the same factor — enough to turn a null result into a “significant” one. The gap grows with messages per user: in this simulation it is about 1.2× at 3 messages per user, 2.0× at 30, and over 3× at 100. Any agentic or coding-assistant product, where one session emits hundreds of events, sits at the dangerous end of that range.

Interleaving: A Faster Alternative to A/B for Preference Signals¶

Traditional A/B tests require large samples to detect small effects because the between-user variance dominates. Interleaving (also called interleaved comparison) sidesteps this by showing outputs from both models within the same user session, then measuring which model’s outputs users prefer.

How interleaving works for LLM outputs¶

In search/recommendation, interleaving mixes ranked lists. For LLM chat products, one variant is: present two completions side-by-side (a “compare” UI) and record which the user acts on. A more subtle variant records which completion a user copies, continues the conversation from, or clicks “insert” on in a coding assistant.

For document editing or summarization, you can show two alternative completions and ask the user to select or edit one. The fraction of users who prefer treatment over control — the win rate — is the primary signal.

The statistical efficiency gain is substantial. Because each user sees both models, within-user variance is eliminated. Empirically, interleaving experiments have been reported to require on the order of 100x fewer user-sessions to detect the same effect size as a parallel A/B test for ranking systems (Radlinski & Craswell, “Optimized Interleaving for Online Retrieval Evaluation,” WSDM 2013). The gain for LLM completions is product-dependent but typically a factor of 10–30x.

from collections import defaultdict
from typing import NamedTuple
from scipy import stats

class InterleavingSession(NamedTuple):
    user_id: str
    control_response: str
    treatment_response: str
    # Which response did the user take a positive action on?
    # 'control', 'treatment', or 'none'
    preferred: str

def compute_interleaving_win_rate(
    sessions: list[InterleavingSession],
) -> dict:
    """
    Compute treatment win rate and a two-sided binomial test.
    Only sessions with a preference (not 'none') are counted.
    """
    decisive = [s for s in sessions if s.preferred != "none"]
    n = len(decisive)
    if n == 0:
        return {"win_rate": float("nan"), "n_decisive": 0}

    wins_treatment = sum(1 for s in decisive if s.preferred == "treatment")
    win_rate = wins_treatment / n

    # Under H0: win_rate = 0.5; use binomial test
    # (scipy.stats.binom_test was deprecated in SciPy 1.7 and removed in 1.12+;
    # use the modern binomtest API, which returns a result object.)
    p_value = stats.binomtest(wins_treatment, n, p=0.5, alternative="two-sided").pvalue

    return {
        "win_rate": win_rate,
        "n_decisive": n,
        "p_value": p_value,
        "significant": p_value < 0.05,
    }

Interleaving is best suited for preference signals (which response is better?) rather than outcome signals (did the user’s problem get resolved?). For resolution rate and similar task-completion metrics, you still need an A/B test since both models cannot solve the same problem simultaneously in a meaningful way.

CUPED and Sequential Testing for Low-Traffic Products¶

Many LLM products do not have the luxury of millions of daily users. A specialized enterprise copilot may have a few thousand active users. The standard A/B test becomes impractically slow — waiting weeks for significance means slow iteration velocity.

CUPED: Controlled-experiment Using Pre-Experiment Data¶

CUPED (Deng et al., “Improving the Sensitivity of Online Controlled Experiments by Utilizing Pre-Experiment Data,” WSDM 2013) exploits pre-experiment covariate information to reduce variance in the metric estimator.

The idea: compute the user’s metric value in the period before the experiment. This pre-period value is strongly correlated with the in-experiment value (users who frequently gave thumbs-up before the experiment are likely to do so again). By subtracting the projection of the metric onto this covariate, we obtain a lower-variance estimator of the treatment effect.

The CUPED-adjusted estimator for user \(i\) in arm \(a\) is:

where \(Y_i\) is the in-experiment metric, \(X_i\) is the pre-experiment covariate, \(\bar{X}\) is the covariate mean (pooled across arms), and the optimal coefficient is:

The variance of the adjusted estimator is:

where \(\rho\) is the Pearson correlation between \(Y\) and \(X\). If \(\rho = 0.7\) (typical for behavioral metrics), variance drops by \(1 - 0.49 = 51\%\), and required sample size drops by half.

import numpy as np
from scipy import stats

def cuped_estimate(
    y_control: np.ndarray,
    y_treatment: np.ndarray,
    x_control: np.ndarray,    # pre-experiment covariate, control arm
    x_treatment: np.ndarray,  # pre-experiment covariate, treatment arm
) -> dict:
    """
    Compute CUPED-adjusted treatment effect and t-test p-value.

    y_*: in-experiment metric values per user
    x_*: pre-experiment metric values for the same users
    """
    # Pool covariate mean and compute theta using pooled data
    x_all = np.concatenate([x_control, x_treatment])
    y_all = np.concatenate([y_control, y_treatment])
    x_bar = x_all.mean()

    theta = np.cov(y_all, x_all, ddof=1)[0, 1] / np.var(x_all, ddof=1)

    # Adjust each user's metric
    y_control_adj = y_control - theta * (x_control - x_bar)
    y_treatment_adj = y_treatment - theta * (x_treatment - x_bar)

    # Two-sample t-test on adjusted values
    t_stat, p_value = stats.ttest_ind(y_treatment_adj, y_control_adj)

    delta = y_treatment_adj.mean() - y_control_adj.mean()
    se = np.sqrt(
        np.var(y_treatment_adj, ddof=1) / len(y_treatment_adj)
        + np.var(y_control_adj, ddof=1) / len(y_control_adj)
    )

    # Variance reduction achieved
    var_unadjusted = np.var(np.concatenate([y_control, y_treatment]), ddof=1)
    var_adjusted = np.var(np.concatenate([y_control_adj, y_treatment_adj]), ddof=1)
    rho_sq = 1 - var_adjusted / var_unadjusted

    return {
        "delta": delta,
        "p_value": p_value,
        "ci_95": (delta - 1.96 * se, delta + 1.96 * se),
        "theta": theta,
        "variance_reduction_fraction": rho_sq,
        "significant": p_value < 0.05,
    }

# Simulate: 500 users per arm, thumbs-up rate 0.40 control / 0.42 treatment
rng = np.random.default_rng(42)
n = 500
x_c = rng.binomial(1, 0.40, n).astype(float)  # pre-exp covariate
x_t = rng.binomial(1, 0.40, n).astype(float)
# In-experiment: add treatment effect + correlation with pre-exp
y_c = np.clip(x_c * 0.7 + rng.binomial(1, 0.12, n), 0, 1)
y_t = np.clip(x_t * 0.7 + rng.binomial(1, 0.14, n), 0, 1)

result = cuped_estimate(y_c, y_t, x_c, x_t)
print(f"Delta: {result['delta']:.4f}, p={result['p_value']:.4f}, "
      f"variance reduction: {result['variance_reduction_fraction']:.1%}")

Sequential testing (always-valid p-values)¶

The temptation with slow experiments is to peek repeatedly at the p-value and stop early if \(p < 0.05\). This inflates the false positive rate dramatically: peeking daily for 14 days at \(\alpha = 0.05\) yields an actual false positive rate of roughly 20–25%.

Sequential testing (also called always-valid inference) provides a test statistic that can be evaluated at any time without type-I error inflation. The mSPRT (mixture Sequential Probability Ratio Test, Johari et al., 2017) and e-values framework guarantee that at any stopping time \(\tau\):

The concrete object you compute is a confidence sequence: a sequence of intervals \(\{C_n\}\) such that \(P(\forall n \geq 1:\ p \in C_n) \geq 1 - \alpha\). Note where the quantifier sits — the coverage guarantee is over all sample sizes at once, not one pre-committed sample size, which is exactly what licenses looking after every event. For Bernoulli (or any \([0,1]\)-bounded) metric, the simplest correct construction is the sub-Gaussian normal-mixture boundary of Howard, Ramdas, McAuliffe & Sekhon (2021). Many experimentation platforms (Statsig, Optimizely, GrowthBook) now implement a sequence like this by default, often alongside CUPED and bandit-based adaptive traffic allocation in the same stats engine. (Statsig itself was acquired by OpenAI in a ~USD 1.1B all-stock deal announced in September 2025 — its founder became OpenAI’s CTO of Applications — but the platform continues operating and serving outside customers.)

import math

# -------------------------------------------------------------------------
# Sub-Gaussian normal-mixture confidence sequence (Howard, Ramdas, McAuliffe
# & Sekhon, 2021). A metric bounded in [0, 1] is (1/2)-sub-Gaussian by
# Hoeffding's lemma, and the normal-mixture boundary with tuning parameter
# rho holds SIMULTANEOUSLY for all n >= 1 with probability >= 1 - alpha:
#
#   |p_hat_n - p| <= sigma * sqrt( 2 (n rho^2 + 1) / (n^2 rho^2)
#                                  * log( sqrt(n rho^2 + 1) / alpha ) )
# -------------------------------------------------------------------------

def cs_halfwidth(n: int, rho: float, alpha: float = 0.05,
                 sigma: float = 0.5) -> float:
    """Half-width of the normal-mixture confidence sequence at sample size n."""
    return sigma * math.sqrt(
        2.0 * (n * rho ** 2 + 1.0) / (n ** 2 * rho ** 2)
        * math.log(math.sqrt(n * rho ** 2 + 1.0) / alpha)
    )

def tune_rho(n_target: int, alpha: float = 0.05) -> float:
    """
    A confidence sequence cannot be tightest everywhere; rho chooses *where*.
    Pick the sample size you realistically expect to stop at and minimise the
    boundary there (it is unimodal in rho, so a ternary search suffices).
    """
    lo, hi = 1e-5, 10.0
    for _ in range(200):
        m1, m2 = lo + (hi - lo) / 3, hi - (hi - lo) / 3
        if cs_halfwidth(n_target, m1, alpha) < cs_halfwidth(n_target, m2, alpha):
            hi = m2
        else:
            lo = m1
    return 0.5 * (lo + hi)

def always_valid_ci(
    n: int,
    k: int,                    # successes so far
    alpha: float = 0.05,
    n_target: int = 10_000,    # sample size to optimise the boundary for
) -> tuple[float, float]:
    """Anytime-valid CI for a Bernoulli proportion; valid at every n >= 1."""
    p_hat = k / n
    w = cs_halfwidth(n, tune_rho(n_target, alpha), alpha)
    return (max(0.0, p_hat - w), min(1.0, p_hat + w))

# The price of anytime validity: compare with the fixed-n 95% interval, which
# is only honest if you look exactly once, at a pre-committed sample size.
for n, k in [(100, 40), (1_000, 400), (5_000, 2_000), (20_000, 8_000)]:
    lo, hi = always_valid_ci(n, k, n_target=10_000)
    p = k / n
    fixed_width = 2 * 1.96 * math.sqrt(p * (1 - p) / n)
    print(f"n={n:6d}: anytime {lo:.3f}-{hi:.3f} (width {hi - lo:.3f})"
          f"   fixed-n width {fixed_width:.3f}")
# n=  1000: anytime 0.340-0.460 (width 0.121)   fixed-n width 0.061
# n= 20000: anytime 0.389-0.411 (width 0.022)   fixed-n width 0.014

There is no free lunch, and the numbers say so plainly: near the tuning point the anytime interval is roughly 1.5–2× wider than the fixed-\(n\) interval, and very early (n = 100) it is vacuous. That width is the peeking correction, paid up front and honestly, instead of being silently spent by stopping the moment \(p\) dips under 0.05.

Shadow Deployments and Canary Rollouts¶

Statistical significance tells you the effect is real; it does not tell you the system is safe to deploy at scale. Canary and shadow deployments are the operational complement to experiment design.

Shadow mode¶

In shadow mode, you run the candidate model on all production traffic but discard its responses — users never see them. Shadow mode lets you:

1. Profile latency and cost at realistic load without user impact. You discover that the new model has 40% higher P99 latency under batch pressure before any user is affected.
2. Run offline judges on shadow outputs. Sample shadow responses at, say, 1% and run your LLM-as-judge pipeline (see LLM-as-a-Judge & Automated Evaluation) to get a quality distribution before traffic exposure.
3. Test infrastructure integration — does the new model endpoint return the expected JSON schema? Are there new failure modes (timeout patterns, empty responses on edge inputs)?

You do not have to build the duplication yourself: the service mesh does it. Envoy (and therefore Istio) supports request mirroring, which fire-and-forgets a copy of each request to a second cluster and discards the response, so mirrored latency and mirrored errors can never touch the user. The mirrored Host header is suffixed with -shadow so downstream logs and traces are trivially separable:

# Istio VirtualService: 100% of live traffic served by v1, 10% mirrored to v2.
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: llm-api
spec:
  hosts: ["llm-api"]
  http:
    - route:
        - destination: {host: llm-api, subset: v1}
          weight: 100
      mirror:
        host: llm-api
        subset: v2                 # candidate model — responses discarded
      mirrorPercentage:
        value: 10.0                # start low: shadowing doubles GPU cost

Two caveats specific to LLM shadowing. First, mirrored traffic is real GPU work — at 100% mirroring you are paying for two fleets, which is why mirrorPercentage starts low and why shadow runs are time-boxed. Second, mirror only idempotent requests: if your handler writes to a database, sends an email, or calls a paid third-party tool, the shadow copy will do it twice. Route shadow traffic to a sandboxed tool layer (see Security: Prompt Injection, Jailbreaks & Defenses for the sandbox pattern), or restrict mirroring to read-only endpoints. For pure throughput/latency profiling without any mesh at all, replay a captured prompt-length distribution against the candidate server with vllm bench serve (vLLM’s serving benchmark, formerly benchmarks/benchmark_serving.py) — cheaper than mirroring and reproducible, at the cost of not exercising real request arrival patterns.

Canary rollout¶

A canary routes a small, controllable fraction of real traffic to the new model, with automatic rollback triggers tied to guardrail metrics.

# Example: canary rollout configuration (Kubernetes + Argo Rollouts style)
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: llm-api-server
spec:
  strategy:
    canary:
      steps:
        - setWeight: 1       # 1% canary
          pause: {duration: 30m}
        - analysis:
            templates:
              - templateName: guardrail-check
        - setWeight: 10      # 10% canary
          pause: {duration: 2h}
        - analysis:
            templates:
              - templateName: guardrail-check
        - setWeight: 50      # 50% — effective A/B
          pause: {duration: 24h}
        - setWeight: 100     # full rollout
      # Automatic rollback if any guardrail fires
      autoPromotionEnabled: false

# Guardrail check logic that would back the AnalysisTemplate above
import statistics

def should_rollback(
    canary_metrics: dict,
    baseline_metrics: dict,
    thresholds: dict,
) -> tuple[bool, str]:
    """
    Returns (rollback, reason) based on guardrail metric comparisons.
    Thresholds define maximum *relative* degradation allowed.

    Example thresholds:
      {
        "thumb_down_rate": 0.20,     # allow up to 20% increase
        "latency_p99_ms": 0.30,      # allow up to 30% increase
        "cost_per_session_usd": 0.15,
        "safety_violation_rate": 0.0, # zero tolerance
      }
    """
    for metric, max_relative_increase in thresholds.items():
        baseline_val = baseline_metrics.get(metric)
        canary_val = canary_metrics.get(metric)
        if baseline_val is None or canary_val is None:
            continue
        if baseline_val == 0:
            if canary_val > 0:
                return True, f"{metric}: baseline=0, canary={canary_val} (zero tolerance)"
            continue

        relative_change = (canary_val - baseline_val) / baseline_val
        if relative_change > max_relative_increase:
            return True, (
                f"{metric}: baseline={baseline_val:.4f}, canary={canary_val:.4f}, "
                f"relative increase={relative_change:.1%} > threshold={max_relative_increase:.1%}"
            )

    return False, "all guardrails passed"


# Example call
rollback, reason = should_rollback(
    canary_metrics={
        "thumb_down_rate": 0.062,
        "latency_p99_ms": 1845,
        "cost_per_session_usd": 0.041,
        "safety_violation_rate": 0.0,
    },
    baseline_metrics={
        "thumb_down_rate": 0.055,
        "latency_p99_ms": 1420,
        "cost_per_session_usd": 0.038,
        "safety_violation_rate": 0.0,
    },
    thresholds={
        "thumb_down_rate": 0.20,
        "latency_p99_ms": 0.30,
        "cost_per_session_usd": 0.15,
        "safety_violation_rate": 0.0,
    },
)
print(f"Rollback: {rollback}, reason: {reason}")
# Rollback: False, reason: all guardrails passed
# (P99 latency increased 29.9% — just barely passes; in practice, tighten to 0.25)

A well-run canary pipeline can catch regressions within minutes. The 1% initial stage exists specifically for catastrophic failures (model returns empty strings, crashes with certain inputs). The 10% stage provides the first statistically meaningful read on behavioral metrics. The 50% stage is where you run the full hypothesis test.

Live Judges: Scoring Sampled Production Traffic¶

Even with good behavioral metrics, they are proxies. The only way to know whether output quality has changed is to evaluate outputs directly. Live judges do this continuously.

The pipeline¶

The LLM judge rates each sampled response on a rubric (helpfulness, accuracy, safety, format). Human raters review a smaller fraction (on the order of 0.05–0.1% of traffic) to calibrate judge accuracy and detect drift in judge behavior. See LLM-as-a-Judge & Automated Evaluation for the judge design.

The open-source layer that runs this loop is your tracing backend: Langfuse ships online evaluators that attach a judge prompt to a sampled fraction of live traces and write the result back as a numeric score on the trace, and Arize Phoenix and Evidently do the equivalent. Because the traces already carry the experiment variant (log it as a span attribute — see Observability, Logging & LLMOps), the judge score becomes just another metric you can slice by arm, which is what lets a quality regression show up in the same dashboard as latency and cost. Build the sampler and the rolling monitor below yourself only when you need routing logic the platform does not express — the stratified sampling policy is usually exactly that case.

Stratified sampling¶

Uniform sampling misses tail behaviors. A stratified sampler over-represents:

• New users (first-session queries are often unusual)
• Long sessions (users who are stuck)
• Queries with low confidence (model’s own log-probability is low)
• Queries triggering guardrails (safety classifier near-misses)
• Queries in underrepresented categories (rare intents)

import random
from typing import Callable

def stratified_sampler(
    request: dict,
    base_rate: float = 0.005,  # 0.5% baseline
    boost_rules: list[tuple[Callable[[dict], bool], float]] | None = None,
) -> bool:
    """
    Returns True if this request should be sampled for judging.

    boost_rules: list of (predicate, multiplier) pairs. The highest
    applicable multiplier is used (not additive, to avoid double-counting).
    """
    if boost_rules is None:
        boost_rules = []

    effective_rate = base_rate
    for predicate, multiplier in boost_rules:
        if predicate(request):
            effective_rate = max(effective_rate, base_rate * multiplier)

    return random.random() < effective_rate

# Example configuration
boost_rules = [
    (lambda r: r.get("session_turn_count", 0) == 1, 5.0),    # first turn
    (lambda r: r.get("session_length", 0) > 10, 4.0),         # long session
    (lambda r: r.get("model_log_prob", 0.0) < -2.5, 8.0),     # low-confidence
    (lambda r: r.get("safety_score", 0.0) > 0.4, 20.0),       # near-miss safety
    (lambda r: r.get("user_is_new", False), 3.0),              # new user
]

# Simulate over 1M requests
sample_count = sum(
    1 for _ in range(1_000_000)
    if stratified_sampler(
        {"session_turn_count": random.randint(1, 15),
         "model_log_prob": random.gauss(-1.0, 1.5)},
        boost_rules=boost_rules,
    )
)
print(f"Estimated sample rate: {sample_count / 1_000_000:.2%}")

Monitoring quality metrics over time¶

Rather than a single pass/fail, maintain rolling statistics so regressions are visible as trends:

from collections import deque
import numpy as np

class RollingQualityMonitor:
    """
    Maintains a sliding window of judge scores and raises an alert
    when the mean score drops below a configurable threshold.
    """

    def __init__(self, window_size: int = 1000, alert_threshold: float = 0.05):
        self.scores = deque(maxlen=window_size)
        self.alert_threshold = alert_threshold  # max allowed drop from baseline
        self.baseline_mean: float | None = None

    def set_baseline(self, scores: list[float]) -> None:
        """Call once with initial production scores to establish baseline."""
        self.baseline_mean = float(np.mean(scores))

    def add_score(self, score: float) -> dict:
        """Add a new judge score; returns alert status."""
        self.scores.append(score)
        current_mean = float(np.mean(self.scores))
        alert = False
        reason = None

        if self.baseline_mean is not None and len(self.scores) >= 50:
            drop = (self.baseline_mean - current_mean) / self.baseline_mean
            if drop > self.alert_threshold:
                alert = True
                reason = (
                    f"Mean quality dropped {drop:.1%} below baseline "
                    f"(current={current_mean:.3f}, baseline={self.baseline_mean:.3f})"
                )

        return {
            "current_mean": current_mean,
            "n_samples": len(self.scores),
            "alert": alert,
            "reason": reason,
        }

Biases That Corrupt Online Signals¶

Online experiments are not automatically reliable. Several systematic biases can lead to wrong conclusions.

Novelty effect¶

Users often engage more positively with any change, simply because it is new. A new UI for the LLM chat box may generate elevated thumbs-up rates for the first few days regardless of underlying quality. The novelty effect typically decays over one to two weeks.

Defense: Run experiments for at least two weeks, and segment by user tenure (days since first use). If new users show higher lift than returning users, novelty is likely the confounder. Report the effect separately for users who have been in the experiment for more than one week.

Feedback-loop bias¶

In many LLM applications, model outputs become inputs to future sessions: documents users compose with an AI, code they commit, answers they cite. Over time, the model trained on data from production begins to see its own outputs in training data, creating a feedback loop. Metrics may improve because the world has shifted to match the model, not because the model has improved.

Defense: Maintain a holdout cohort — a randomly selected 1–5% of users who are never exposed to any treatment variant. Compare the holdout’s metric trajectory over months to detect secular trends in the overall user base.

Position and presentation bias¶

If your experiment changes how responses are presented (length, formatting, bullet points vs. prose), behavioral metrics like copy rate and click-through can change for presentation reasons unrelated to quality.

Defense: Separate model quality experiments from UI experiments. Run them sequentially, not simultaneously, unless your experiment platform supports factorial designs and you have sufficient traffic for interaction terms.

Selection bias from opt-in feedback¶

Thumb ratings are provided by a self-selected subset of users — typically those with strong opinions (very positive or very negative). The non-response majority may have different preferences. A treatment that increases the thumbs-up count may simply be eliciting more feedback, not improving quality.

Defense: Track the feedback rate (what fraction of sessions result in any rating) alongside the conditional thumbs-up rate. A rising feedback rate with stable conditional thumbs-up rate signals more engagement, not more satisfaction.

Peeking and multiple testing¶

See the sequential testing discussion above. Additionally, if you are running 10 simultaneous A/B experiments (common in fast-moving teams), the probability that at least one shows a false positive at \(\alpha = 0.05\) is \(1 - 0.95^{10} \approx 40\%\). Apply Bonferroni correction or control the False Discovery Rate (Benjamini-Hochberg) when reporting across many simultaneous tests.

Connecting the Pieces: The Evaluation Lifecycle¶

A mature evaluation stack is not a collection of ad-hoc tests — it is a pipeline that continuously validates every change.

The lifecycle makes explicit that offline evaluation, shadow testing, canaries, A/B experiments, and live judging are not alternatives — they are sequential filters. Each stage catches different failure modes, and bypassing any stage (usually in the name of speed) shifts the risk onto downstream stages or onto users.

The same lifecycle at Stack-100M scale¶

None of this requires a million users. When you deploy the capstone model from Evaluation & Serving: Honest Benchmarks, int4 Quantization, and Running on a Laptop, the same five filters collapse into a weekend-sized version: (1) offline bits-per-byte and task accuracy on the held-out split gate the checkpoint; (2) shadow the new checkpoint behind the old one on a replayed prompt log to compare tokens/s and P99 latency at int4; (3) canary it to one of two server replicas with the should_rollback guardrail on refusal rate, latency, and empty-output rate; (4) run interleaving rather than A/B — with a handful of users, the 10–30× efficiency gain is not a nicety, it is the difference between a readable signal and none, and the win rate can be scored by an LLM judge when human raters are scarce; (5) keep a live judge on 1–5% of traffic. For the narrow research agent in A Narrow Auto-Research Agent the primary metric is unambiguous and cheap — automated task success (did the tool call parse, did the retrieved citation actually support the claim) — which is exactly the regime where a small product can run a trustworthy online experiment: verifiable outcomes need far fewer sessions than noisy preference proxies.

Further Reading¶

• Deng, A., Xu, Y., Kohavi, R., Walker, T. — “Improving the Sensitivity of Online Controlled Experiments by Utilizing Pre-Experiment Data” (CUPED), WSDM 2013.
• Johari, R., Koomen, P., Pekelis, L., Walsh, D. — “Peeking at A/B Tests: Why It Matters, and What to Do About It” (mSPRT), KDD 2017.
• Howard, S. R., Ramdas, A., McAuliffe, J., Sekhon, J. — “Time-uniform, nonparametric, nonasymptotic confidence sequences,” Annals of Statistics 2021.
• Radlinski, F., Craswell, N. — “Optimized Interleaving for Online Retrieval Evaluation,” WSDM 2013.
• Kohavi, R., Tang, D., Xu, Y. — Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing, Cambridge University Press, 2020.
• Benjamini, Y., Hochberg, Y. — “Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing,” Journal of the Royal Statistical Society B, 1995.
• Gu, J. et al. — “A Survey on LLM-as-a-Judge,” arXiv, 2024 (the judge design that live scoring of production traffic depends on).
• Fabijan, A., Gupchup, J., Gupta, S., Omhover, J., Qin, W., Vermeer, L., Dmitriev, P. — “Diagnosing Sample Ratio Mismatch in Online Controlled Experiments,” KDD 2019.

Exercises¶

1. The chapter insists on randomizing at the user-session level rather than the request level. Suppose a colleague proposes randomizing per request instead — for each incoming request, flip a coin and route it to control or treatment — arguing this doubles the effective sample size since every request becomes an independent data point. Explain what goes wrong, referencing the specific assumption this violates and what kind of contamination it introduces.

2. You run an A/B test on thumbs-up rate. Control gets \(k=4{,}000\) thumbs-up out of \(n=10{,}000\) sessions; treatment gets \(k=4{,}200\) out of \(n=10{,}000\). Using the chapter’s pooled two-proportion \(z\)-test, compute the \(z\)-statistic and state whether the result is significant at \(\alpha = 0.05\). Also report the relative lift.

3. Your thumbs-up rate is \(0.40\) with Bernoulli standard deviation \(\sigma = 0.49\). You want to detect an MDE of \(\delta = 0.02\) (2 percentage points) with 80% power at \(\alpha = 0.05\), so \(z_{\alpha/2} = 1.96\) and \(z_\beta = 0.84\). (a) Compute the required users per arm with a plain A/B test. (b) Now suppose you have a pre-experiment covariate correlated with the metric at \(\rho = 0.7\) and apply CUPED. Compute the new required sample size per arm and the number of days to reach it at 500 users/arm/day.

4. After launching a new model, your treatment arm’s thumbs-up count rises by 12% versus control, and your team celebrates. A skeptical analyst points out that this could be selection bias from opt-in feedback rather than a real quality gain. Explain the mechanism, and describe the specific diagnostic the chapter recommends to distinguish “more feedback” from “more satisfaction.”

5. A fast-moving team runs 10 independent A/B experiments simultaneously, each tested at \(\alpha = 0.05\). (a) What is the probability that at least one experiment shows a false positive under the null? (b) What per-test threshold does the Bonferroni correction prescribe to hold the family-wise error rate at 0.05, and what is the resulting family-wise false-positive probability?

6. The chapter’s should_rollback function only fires when a guardrail metric increases beyond a relative threshold (higher = worse: latency, cost, thumb-down rate). But some metrics are higher-is-better — e.g. resolution rate or thumbs-up rate — and a canary should also roll back if one of those drops too far. Modify should_rollback to accept a second threshold dict for higher-is-better metrics and roll back on an excessive relative decrease. Keep it consistent with the chapter’s style, including the zero-baseline guard.

def should_rollback(
    canary_metrics: dict,
    baseline_metrics: dict,
    thresholds: dict,                       # higher = worse (max relative INCREASE)
    higher_is_better_thresholds: dict | None = None,  # max relative DECREASE
) -> tuple[bool, str]:
    """
    Returns (rollback, reason) based on guardrail metric comparisons.

    `thresholds`: metrics where an increase is bad (latency, cost, thumb-down).
    `higher_is_better_thresholds`: metrics where a decrease is bad
        (resolution rate, thumbs-up rate); values are the max tolerated
        *relative decrease*, e.g. {"resolution_rate": 0.05} allows up to a
        5% relative drop before rollback.
    """
    # --- metrics where an INCREASE is a regression (unchanged) ---
    for metric, max_relative_increase in thresholds.items():
        baseline_val = baseline_metrics.get(metric)
        canary_val = canary_metrics.get(metric)
        if baseline_val is None or canary_val is None:
            continue
        if baseline_val == 0:
            if canary_val > 0:
                return True, f"{metric}: baseline=0, canary={canary_val} (zero tolerance)"
            continue
        relative_change = (canary_val - baseline_val) / baseline_val
        if relative_change > max_relative_increase:
            return True, (
                f"{metric}: baseline={baseline_val:.4f}, canary={canary_val:.4f}, "
                f"relative increase={relative_change:.1%} > threshold={max_relative_increase:.1%}"
            )

    # --- metrics where a DECREASE is a regression (new) ---
    for metric, max_relative_decrease in (higher_is_better_thresholds or {}).items():
        baseline_val = baseline_metrics.get(metric)
        canary_val = canary_metrics.get(metric)
        if baseline_val is None or canary_val is None:
            continue
        if baseline_val == 0:
            # No positive baseline to fall from; nothing to guard.
            continue
        # Relative DROP is positive when canary < baseline.
        relative_drop = (baseline_val - canary_val) / baseline_val
        if relative_drop > max_relative_decrease:
            return True, (
                f"{metric}: baseline={baseline_val:.4f}, canary={canary_val:.4f}, "
                f"relative drop={relative_drop:.1%} > threshold={max_relative_decrease:.1%}"
            )

    return False, "all guardrails passed"


# Example: thumbs-up rate falls from 0.410 to 0.380 (a 7.3% relative drop),
# which exceeds a 5% tolerance and triggers rollback.
rollback, reason = should_rollback(
    canary_metrics={"latency_p99_ms": 1500, "resolution_rate": 0.380},
    baseline_metrics={"latency_p99_ms": 1420, "resolution_rate": 0.410},
    thresholds={"latency_p99_ms": 0.30},
    higher_is_better_thresholds={"resolution_rate": 0.05},
)
print(f"Rollback: {rollback}, reason: {reason}")
# Rollback: True, reason: resolution_rate: baseline=0.4100, canary=0.3800,
#                         relative drop=7.3% > threshold=5.0%

7. A coding-assistant team reports a win: over a two-week experiment, control logged 100,000 sessions and treatment 98,000, and the per-suggestion accept rate rose from 0.300 to 0.309, with a naive two-proportion \(z\)-test over all suggestions giving \(z = 2.60\) (\(p \approx 0.009\)). (a) Run the SRM check on the session counts and say what it implies. (b) Assuming the SRM is explained and fixed, the metric is still suggestions-per-user, with a delta-method standard error 1.34× the naive per-suggestion one. Recompute \(z\) and the two-sided \(p\)-value. © What should the team report?