11.2 LLM-as-a-Judge & Automated Evaluation¶

Human evaluation of LLM outputs is the gold standard — but it is slow, expensive, and hard to reproduce. A single preference study across a few thousand examples might cost weeks of annotator time and thousands of dollars. When you are running ablations, A/B testing prompts, or iterating on a fine-tuned model daily, that cadence is untenable.

The insight behind LLM-as-a-Judge (LLMaaJ) is simple: a sufficiently capable language model can replicate many of the judgments a human makes about quality, correctness, and helpfulness — and it can do so at the speed of inference. Since roughly 2023, this approach has become the backbone of nearly every large-scale automated evaluation pipeline, including the reward modeling loop described in The RLHF Pipeline & Reward Modeling.

This chapter is a rigorous treatment of the machinery: how to prompt a judge, which biases infect its scores, how to calibrate and audit it against human labels, how to serve an open judge locally with vLLM, and how to build the Chatbot Arena ranking — both the online Elo update and the Bradley-Terry maximum-likelihood fit the live leaderboard actually computes — from scratch. In the capstone, this machinery does two jobs: it mines the on-policy preference pairs that DPO consumes (Post-Training: SFT, DPO, and Narrow RLVR (GRPO) That Works at 100M) and it grades the open-ended half of the final capability report (Evaluation & Serving: Honest Benchmarks, int4 Quantization, and Running on a Laptop).

Why Human Evaluation Alone Does Not Scale¶

Before diving into the mechanics, it is worth being precise about why we need automation at all.

Throughput. A single annotator, given a pair of model responses, might take 2–3 minutes to read, compare, and label. At that rate, 10,000 comparisons require roughly 300–500 annotator-hours. With a team of 10, that is multiple weeks. Modern development cycles need daily evaluation loops.

Consistency. Even trained annotators disagree. Inter-annotator agreement on open-ended quality often sits at Cohen’s κ in the range of 0.4–0.6 — barely moderate. Annotator mood, fatigue, and framing effects introduce noise that is hard to control.

Coverage. Benchmarks like MMLU or HumanEval cover well-defined task categories with right/wrong answers — see The Evaluation Problem & Benchmark Landscape. Open-ended generation (summaries, essays, conversational responses) cannot be evaluated this way. LLMaaJ fills that gap.

Reproducibility. A judge prompt is code. It can be versioned, diffed, and re-run. Human panels cannot.

The tradeoff is that LLM judges have their own systematic biases — and the chapter’s central skill is understanding and mitigating those biases so the judge is trustworthy.

The Two Evaluation Paradigms: Pointwise vs Pairwise¶

All LLMaaJ setups fall into one of two modes, and the choice has deep implications for reliability and use case.

Pointwise (Absolute) Scoring¶

A single response is scored on an absolute scale — commonly 1–5 or 1–10 — against a rubric.

Judge prompt (pointwise):

  [System]
  You are a rigorous and calibrated evaluator. Score the following response
  on a scale of 1 to 5 for the criterion "helpfulness", where:
    1 = completely unhelpful or harmful
    2 = partially addresses the request but has major gaps
    3 = adequate response with some flaws
    4 = good, nearly complete response
    5 = excellent, comprehensive, directly useful response

  [User]
  Question: {question}
  Response: {response}

  Reply with ONLY a JSON object: {"score": <int 1-5>, "rationale": "<1-2 sentences>"}

Advantages: can compare any number of models without re-running pairs; scores are additive (you can average across criteria); efficient at O(N) queries per N responses.

Disadvantages: the absolute scale is hard to calibrate across prompt types. A model might give a score of 4 for responses that a human would rate quite differently, because the judge has no anchor.

Pairwise (Comparative) Scoring¶

Two responses A and B are shown simultaneously. The judge picks a winner (or declares a tie).

Judge prompt (pairwise):

  [System]
  You are a fair, expert evaluator. Given a question and two responses (A and B),
  decide which response is more helpful, accurate, and well-written.
  Do not let response length, formatting style, or order influence your decision.

  [User]
  Question: {question}

  Response A:
  {response_a}

  Response B:
  {response_b}

  Output ONLY JSON: {"winner": "A" | "B" | "tie", "rationale": "<2-3 sentences>"}

Advantages: relative judgments are cognitively easier and better calibrated than absolute ones — the judge (like a human) can often tell which of two things is better even when scoring each on an absolute scale is hard. Consistency within a comparison is high.

Disadvantages: requires O(N²) queries to compare all pairs; does not yield an absolute quality estimate; susceptible to position bias (discussed next).

When to Use Each¶

In practice, production eval pipelines often use pointwise scoring for monitoring (cheap, daily runs) and pairwise comparison for model selection (milestone decisions).

Rubric Design: Decomposing Quality¶

Open-ended “quality” is multidimensional. A response can be accurate but terse, or verbose but partially wrong. Collapsing everything into a single score loses signal. The solution is rubric-based multi-criteria evaluation.

The G-Eval Framework (Liu et al.)¶

G-Eval (Liu et al., 2023) pioneered using chain-of-thought (CoT) to generate evaluation criteria and score steps, then computing a weighted score. The key idea: ask the judge to first produce a rationale, then emit a score — this dramatically improves calibration vs. asking for only a score.

A production rubric typically includes several dimensions:

RUBRIC_CRITERIA = {
    "correctness": (
        "Is every factual claim in the response accurate? "
        "Does it avoid hallucinations and unsupported assertions?"
    ),
    "helpfulness": (
        "Does the response directly address the user's question? "
        "Is the answer complete?"
    ),
    "safety": (
        "Does the response avoid harmful, toxic, or policy-violating content?"
    ),
    "conciseness": (
        "Is the response appropriately concise without omitting important information?"
    ),
    "instruction_following": (
        "Did the response follow all explicit instructions (format, length, language, etc.)?"
    ),
}

The judge evaluates each criterion independently (or in one pass with separate scores), then we aggregate. Separating criteria also makes it easier to debug regressions: if a new model drop has lower helpfulness but stable safety, you know exactly where to look.

Weighted Aggregation¶

If criteria have different business importance, use a weighted sum:

where \(w_c\) is the weight for criterion \(c\) and \(s_c \in [1, 5]\) is the score. Weights should be set based on product requirements and validated against human preference labels, not chosen arbitrarily.

Biases in LLM Judges: The Menagerie¶

The central challenge of LLMaaJ is systematic bias — the judge deviates from human ground truth in predictable, correctable ways. You must know these biases to build a trustworthy system.

Position Bias¶

In pairwise evaluation, the judge systematically favors whichever response appears first (or second, depending on the model). This has been empirically documented across multiple judge models. The effect can be as large as 10–15 percentage points on win rate depending on task difficulty.

Mitigation: always run both orderings. Present (A, B) and (B, A), then aggregate:

def pairwise_judge_debiased(judge_fn, question, response_a, response_b):
    """
    Run both orderings and combine results to cancel position bias.
    Returns: "A", "B", or "tie"
    """
    # First ordering: A before B
    result_ab = judge_fn(question, response_a, response_b)  # "A","B","tie"

    # Second ordering: B before A (we swap and interpret the result)
    result_ba = judge_fn(question, response_b, response_a)  # returns in terms of B,A
    # Flip result_ba back to A/B space
    flip = {"A": "B", "B": "A", "tie": "tie"}
    result_ba_flipped = flip[result_ba]

    # Resolve
    if result_ab == result_ba_flipped:
        return result_ab          # Both orderings agree
    else:
        return "tie"              # Disagree → call it a tie

This doubles inference cost but nearly eliminates position bias.

Verbosity Bias (Length Bias)¶

LLM judges tend to prefer longer responses, independent of quality. A response that elaborates at length — even if it contains more hedging and filler — often beats a tight, accurate response. This has been measured in MT-Bench and other benchmarks.

Mitigation strategies:

1. Explicitly instruct the judge: “Do not let response length or formatting influence your decision. A concise, accurate answer is better than a verbose one with the same information.”
2. Include a “conciseness” criterion in the rubric to counterbalance length preference.
3. Normalize scores by response length in your analysis pipeline to detect length artifacts.
4. Test your judge’s calibration by sampling response pairs where one is a padded version of the other.

Public leaderboards now correct for this at the aggregation level: LMArena’s style control adds answer length and markdown structure (headers, bold, lists) as covariates in its Bradley-Terry regression, and length is the dominant style factor — applying it demonstrably reshuffles rankings, pulling verbose-but-weaker models down.

Self-Preference Bias¶

When the judge model is the same as (or closely related to) one of the evaluated models, it tends to prefer its own outputs. GPT-4 as judge overrates GPT-4 responses. This is sometimes called the “narcissism” problem.

Mitigation: Use a different model family for judging than for generation when possible. If you are evaluating GPT-4-class models, use Claude or Gemini as judge, and vice versa. Track self-preference rate in your calibration set.

Sycophancy Bias¶

If the judge is shown a hint about which response a human preferred (or if one response includes flattery or authoritative-sounding language), it may defer to that cue rather than evaluating on merit. This extends to the judge agreeing with whatever opinion is asserted confidently in a response.

Mitigation: Never leak labels into the judge context. Use neutral prompt framing.

Instruction-Following Bias¶

Some judges over-weight instruction following and under-weight factual accuracy. A response that perfectly follows format instructions but contains factual errors may score higher than one with a minor format deviation but accurate content.

Mitigation: Use separate rubric criteria. Weight them explicitly.

The Bias Calibration Protocol¶

Build a gold calibration set of ~200–500 examples with trusted human labels (e.g., majority vote from 3+ annotators). Regularly run your judge on this set and compute:

• Agreement rate (what fraction match the human majority label?)
• Bias profile per ordering (does position A win more than expected?)
• False positive / false negative rates per criterion

If calibration drops, it usually signals that your judge prompt needs updating for the new model family or that the task distribution has drifted.

Agreement with Humans: Measuring Judge Quality¶

A judge is only trustworthy if it agrees with humans at a high enough rate. The standard metric is agreement rate on a held-out human annotation set.

Cohen’s Kappa¶

Raw agreement is confounded by chance. Cohen’s kappa adjusts for it:

where \(p_o\) is observed agreement and \(p_e\) is expected agreement under the null hypothesis that both raters assign labels randomly according to their marginals.

For a three-way (A wins / B wins / tie) pairwise task, if the judge labels A=60%, B=30%, tie=10% and humans label A=50%, B=35%, tie=15%, the expected agreement \(p_e\) is the dot product of the marginals, and \(\kappa\) will typically be in the 0.5–0.7 range for a good judge.

Spearman’s Rank Correlation¶

For pointwise scores, compare judge scores to human quality ratings using Spearman’s \(\rho\) (rank correlation, preferred to Pearson because scores are ordinal):

where \(d_i\) is the rank difference for the \(i\)-th example. A good LLM judge achieves \(\rho\) in the range of 0.7–0.9 on well-defined criteria.

Practical Calibration Pipeline¶

import json
from collections import Counter
from scipy.stats import spearmanr

def calibrate_judge(judge_fn, calibration_set):
    """
    calibration_set: list of dicts with keys:
      - "question": str
      - "response": str
      - "human_score": int (1-5)

    Returns calibration report dict.
    """
    judge_scores = []
    human_scores = []

    for ex in calibration_set:
        result = judge_fn(ex["question"], ex["response"])
        judge_scores.append(result["score"])
        human_scores.append(ex["human_score"])

    # Spearman correlation
    rho, pvalue = spearmanr(judge_scores, human_scores)

    # Score distribution
    judge_dist = Counter(judge_scores)
    human_dist = Counter(human_scores)

    # Mean absolute error
    mae = sum(abs(j - h) for j, h in zip(judge_scores, human_scores)) / len(judge_scores)

    return {
        "spearman_rho": round(rho, 3),
        "p_value": round(pvalue, 4),
        "mae": round(mae, 3),
        "judge_score_distribution": dict(sorted(judge_dist.items())),
        "human_score_distribution": dict(sorted(human_dist.items())),
        "n_examples": len(calibration_set),
    }

A Complete Judge Implementation¶

Here is a production-ready pointwise judge with CoT-before-score, retries, and structured output parsing.

import json
import re
import time
from typing import Optional
from openai import OpenAI  # or any chat completion API

client = OpenAI()  # set OPENAI_API_KEY in environment

JUDGE_SYSTEM_PROMPT = """\
You are a rigorous, calibrated evaluator of AI assistant responses.

Your task: evaluate the RESPONSE to the QUESTION on the following criteria.
For each criterion, think step-by-step before scoring.

Criteria (each scored 1-5):
  correctness    – factual accuracy; 1=major errors, 5=fully accurate
  helpfulness    – directly addresses the user's need; 1=unhelpful, 5=maximally helpful
  conciseness    – appropriate length; 1=bloated/terse to detriment, 5=perfectly sized
  safety         – avoids harm; 1=harmful content, 5=completely safe

Rules:
- Score based on content quality, NOT on length, formatting polish, or style.
- A short accurate answer is better than a long inaccurate one.
- Provide a brief rationale (1 sentence per criterion) before each score.

Output a JSON object with this exact structure:
{
  "correctness":   {"rationale": "...", "score": <1-5>},
  "helpfulness":   {"rationale": "...", "score": <1-5>},
  "conciseness":   {"rationale": "...", "score": <1-5>},
  "safety":        {"rationale": "...", "score": <1-5>}
}
Do not output anything outside the JSON block.
"""

def judge_response(
    question: str,
    response: str,
    model: str = "gpt-4o",
    max_retries: int = 3,
) -> Optional[dict]:
    """
    Evaluate a single (question, response) pair.
    Returns a dict with per-criterion {rationale, score} and an overall score,
    or None if parsing fails after retries.
    """
    user_message = f"QUESTION:\n{question}\n\nRESPONSE:\n{response}"

    for attempt in range(max_retries):
        try:
            completion = client.chat.completions.create(
                model=model,
                messages=[
                    {"role": "system", "content": JUDGE_SYSTEM_PROMPT},
                    {"role": "user",   "content": user_message},
                ],
                temperature=0.0,   # Deterministic for reproducibility
                max_tokens=512,
                response_format={"type": "json_object"},  # GPT-4o feature
            )
            raw = completion.choices[0].message.content
            result = json.loads(raw)

            # Validate structure
            criteria = ["correctness", "helpfulness", "conciseness", "safety"]
            for c in criteria:
                assert c in result
                assert "score" in result[c]
                assert 1 <= int(result[c]["score"]) <= 5

            # Compute a weighted overall score
            weights = {"correctness": 0.4, "helpfulness": 0.4,
                       "conciseness": 0.1, "safety": 0.1}
            overall = sum(weights[c] * result[c]["score"] for c in criteria)
            result["overall"] = round(overall, 2)
            return result

        except (json.JSONDecodeError, KeyError, AssertionError) as e:
            if attempt < max_retries - 1:
                time.sleep(2 ** attempt)  # Exponential backoff
            else:
                print(f"Judge parsing failed after {max_retries} attempts: {e}")
                return None


# ---------- Usage example ----------
if __name__ == "__main__":
    q = "What is the capital of France?"
    r_good = "Paris is the capital and largest city of France."
    r_verbose = (
        "Great question! France, a beautiful country in Western Europe, "
        "has many wonderful cities. The capital city is Paris, which is "
        "located in the north of the country along the Seine River. "
        "Paris is famous for the Eiffel Tower and amazing cuisine!"
    )

    score_good    = judge_response(q, r_good)
    score_verbose = judge_response(q, r_verbose)

    print(f"Concise response overall:  {score_good['overall']}")
    print(f"Verbose response overall:  {score_verbose['overall']}")
    # A well-calibrated judge should penalize verbosity here

Running the Judge Locally: vLLM + an Open Judge Model + Constrained JSON¶

Nothing above requires a proprietary API. The standard open-source deployment is to serve an open judge model behind vLLM’s OpenAI-compatible HTTP server (see vLLM: Architecture, PagedAttention & Internals) and point the exact same client at localhost. For a from-scratch build this matters twice over: judging is the single highest-volume LLM call in an eval pipeline, and it is the one you least want metered.

pip install vllm
# Prometheus-2 is a purpose-trained open judge (absolute + pairwise grading).
# Any strong instruct model of the same size works too; what matters is that
# the judge is NOT the model under test (self-preference bias).
vllm serve prometheus-eval/prometheus-7b-v2.0 --port 8000 --max-model-len 4096

The second win is structured decoding. Rather than sampling freely and retrying on JSONDecodeError, we constrain the decode to a JSON schema, so a syntactically valid object is guaranteed by construction. vLLM implements this with a grammar backend (xgrammar by default; outlines and lm-format-enforcer are also selectable) that masks the logits of any token which cannot continue a legal parse — see Structured & Constrained Generation for the mechanism.

from openai import OpenAI

# Same client class, different base_url. Nothing else in the judge changes.
local_judge = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")

CRITERIA = ["correctness", "helpfulness", "conciseness", "safety"]
CRITERION_SCHEMA = {
    "type": "object",
    "properties": {
        "rationale": {"type": "string"},
        "score": {"type": "integer", "minimum": 1, "maximum": 5},
    },
    "required": ["rationale", "score"],
}
JUDGE_SCHEMA = {
    "type": "object",
    "properties": {c: CRITERION_SCHEMA for c in CRITERIA},
    "required": CRITERIA,
}

def judge_local(question: str, response: str, model: str) -> dict:
    completion = local_judge.chat.completions.create(
        model=model,
        messages=[
            {"role": "system", "content": JUDGE_SYSTEM_PROMPT},
            {"role": "user",   "content": f"QUESTION:\n{question}\n\nRESPONSE:\n{response}"},
        ],
        temperature=0.0,
        max_tokens=512,
        # vLLM-specific knob. Recent vLLM also accepts the OpenAI-style
        # response_format={"type": "json_schema", "json_schema": {...}}.
        extra_body={"guided_json": JUDGE_SCHEMA},
    )
    return json.loads(completion.choices[0].message.content)

Chatbot Arena and the Elo Rating System¶

Chatbot Arena (Zheng et al., LMSYS, 2023 — now operated as LMArena at arena.ai) is the crowd-sourced human preference platform that pits two anonymous models against each other on a user-provided prompt. Millions of such pairwise battles are aggregated into a single ranking via the Elo rating system, originally designed for chess. (The live leaderboard now fits a static Bradley-Terry MLE with style-control covariates rather than the streaming online-Elo update taught below; the online algorithm remains the clearest way to build intuition, and the two agree in the large-sample limit.)

The Elo Model¶

Each model \(i\) has a latent skill rating \(\theta_i\). Given two models with ratings \(\theta_A\) and \(\theta_B\), the probability that A beats B follows the logistic function:

The constant 400 (from chess convention) sets the scale so that a 400-point rating gap corresponds to a 10:1 win-rate advantage.

After each game with outcome \(s_A \in \{1, 0.5, 0\}\) (win, tie, loss for A):

where \(K\) is a step size (typically 32–64).

From LLMaaJ to Arena Elo¶

We can run the same arena protocol with an LLM judge instead of humans. Each evaluation prompt becomes a “battle”: run two model responses through the judge, record the outcome, and update Elo ratings. This is sometimes called automated arena or judge-based ranking.

import math
import random
from collections import defaultdict

class EloRanker:
    """
    Maintains Elo ratings for a set of LLM models.
    Ratings start at 1000 (the Arena convention; chess itself starts new
    players near 1500 — the offset is arbitrary, only differences matter).
    """

    def __init__(self, k: float = 32.0, base: float = 10.0, scale: float = 400.0):
        self.k = k
        self.base = base
        self.scale = scale
        self.ratings: dict[str, float] = defaultdict(lambda: 1000.0)
        self.game_counts: dict[str, int] = defaultdict(int)

    def expected_score(self, rating_a: float, rating_b: float) -> float:
        """P(A wins) under Elo model."""
        return 1.0 / (1.0 + self.base ** ((rating_b - rating_a) / self.scale))

    def update(self, model_a: str, model_b: str, outcome: str) -> None:
        """
        outcome: "A" (A wins), "B" (B wins), or "tie"
        Updates ratings in place.
        """
        score_a = {"A": 1.0, "B": 0.0, "tie": 0.5}[outcome]
        score_b = 1.0 - score_a

        ra, rb = self.ratings[model_a], self.ratings[model_b]
        ea = self.expected_score(ra, rb)
        eb = 1.0 - ea

        self.ratings[model_a] += self.k * (score_a - ea)
        self.ratings[model_b] += self.k * (score_b - eb)
        self.game_counts[model_a] += 1
        self.game_counts[model_b] += 1

    def leaderboard(self) -> list[tuple[str, float, int]]:
        """Returns [(model_name, rating, n_games)] sorted by rating desc."""
        return sorted(
            [(m, round(r, 1), self.game_counts[m]) for m, r in self.ratings.items()],
            key=lambda x: -x[1],
        )


def run_automated_arena(
    judge_fn,       # callable(question, resp_a, resp_b) -> "A"|"B"|"tie"
    prompts: list[str],
    models: dict[str, callable],  # name -> generate_fn(prompt) -> str
    n_battles: int = 500,
    k: float = 32.0,
) -> EloRanker:
    """
    Run an automated arena: sample random prompt/model pairs,
    call judge, update Elo.
    """
    ranker = EloRanker(k=k)
    model_names = list(models.keys())

    for battle_idx in range(n_battles):
        # Sample a random prompt and two distinct models
        prompt = random.choice(prompts)
        a, b = random.sample(model_names, 2)

        # Generate responses
        resp_a = models[a](prompt)
        resp_b = models[b](prompt)

        # Judge (debiased: run both orderings)
        out_ab = judge_fn(prompt, resp_a, resp_b)
        out_ba_raw = judge_fn(prompt, resp_b, resp_a)
        flip = {"A": "B", "B": "A", "tie": "tie"}
        out_ba = flip[out_ba_raw]

        outcome = out_ab if out_ab == out_ba else "tie"

        ranker.update(a, b, outcome)

        if (battle_idx + 1) % 100 == 0:
            print(f"\n--- Battle {battle_idx+1} ---")
            for name, rating, n in ranker.leaderboard():
                print(f"  {name:<25} {rating:>7.1f}  ({n} games)")

    return ranker

From Online Elo to the Bradley-Terry MLE¶

Online Elo has a defect that matters once you publish a leaderboard: it depends on the order the battles arrive in. Shuffle the same battle log and you get different final ratings, because each update uses only the ratings known so far. A model whose first hundred opponents happened to be weak carries an inflated rating for a long time, and \(K\) trades responsiveness against variance with no principled setting.

The fix is to stop treating the ratings as a running average and instead fit them all at once by maximum likelihood. Elo’s logistic curve is the Bradley-Terry (BT) model: give each model a latent strength \(w_i\) and let

Over a battle log \(\mathcal{B}\) of triples \((i, j, y)\) with \(y = 1\) if \(i\) won, the log-likelihood is

This is exactly the log-likelihood of a logistic regression with no intercept whose design row for a battle is \(+1\) in column \(i\), \(-1\) in column \(j\), and \(0\) elsewhere. It is concave in \(w\), so the optimum is unique up to the one unidentifiable degree of freedom (adding a constant to every \(w_i\) changes nothing), which we fix by centering. Ratings are then mapped onto the familiar Elo scale by matching the two parameterizations — \(10^{\Delta\theta/400} = e^{\Delta w}\) gives \(\theta = \frac{400}{\ln 10} w + 1000 \approx 173.7\,w + 1000\).

This is what LMArena publishes, and it is also where style control enters: append extra columns to the design matrix holding per-battle style differences (A minus B) — answer length, number of markdown headers, bold spans, list items — and fit them jointly. The style coefficients soak up the part of the preference explained by presentation, and the model coefficients that remain are style-controlled ratings.

import numpy as np
from sklearn.linear_model import LogisticRegression

def fit_bradley_terry(battle_log, style_features=None,
                      scale=400.0, base=10.0, init_rating=1000.0):
    """
    Order-invariant MLE alternative to the online EloRanker above.

    battle_log:     [{"model_a": str, "model_b": str, "outcome": "A"|"B"|"tie"}, ...]
    style_features: optional list (same length/order as battle_log) of dicts of
                    A-minus-B style covariates, e.g.
                    {"len_diff": (len_a - len_b) / 1000, "header_diff": ...}.
                    Pass them to get STYLE-CONTROLLED ratings; omit for raw ones.

    Returns {model_name: elo_rating}.
    """
    models = sorted({m for b in battle_log for m in (b["model_a"], b["model_b"])})
    idx = {m: i for i, m in enumerate(models)}
    style_keys = sorted(style_features[0]) if style_features else []
    n_col = len(models) + len(style_keys)

    rows, ys, ws = [], [], []
    for t, b in enumerate(battle_log):
        x = np.zeros(n_col)
        x[idx[b["model_a"]]] = +1.0      # the +1 / -1 contrast IS the BT model
        x[idx[b["model_b"]]] = -1.0
        for k, key in enumerate(style_keys):
            x[len(models) + k] = style_features[t][key]

        if b["outcome"] == "tie":
            # A tie is half a win each way: duplicate the row with weight 0.5.
            rows += [x, x]; ys += [1, 0]; ws += [0.5, 0.5]
        else:
            rows.append(x); ys.append(1 if b["outcome"] == "A" else 0); ws.append(1.0)

    # fit_intercept=False: only rating DIFFERENCES are identifiable.
    # The default C=1.0 is a mild L2 prior — not cosmetic: it keeps ratings
    # finite for a model that is undefeated (perfect separation => w -> inf).
    clf = LogisticRegression(fit_intercept=False, max_iter=1000)
    clf.fit(np.vstack(rows), np.array(ys), sample_weight=np.array(ws))

    w = clf.coef_[0][:len(models)]
    w = w - w.mean()                                  # fix the gauge freedom
    theta = w * (scale / np.log(base)) + init_rating  # -> Elo scale
    return {m: round(float(theta[i]), 1) for m, i in idx.items()}

Two practical notes. First, the bootstrap in bootstrap_elo_ci below carries over directly: replace the inner EloRanker loop with a single fit_bradley_terry(sample) call and accumulate the returned {model: rating} dict. Resampling the battle log this way is the standard route to the confidence intervals LMArena reports. Second, you do not have to write this yourself: LMArena open-sourced its leaderboard methodology as arena-rank (pip install arena-rank), whose BradleyTerry(...).compute_ratings_and_cis(dataset, significance_level=0.05) reproduces the published pipeline including the CI machinery.

Reward Models as Judges¶

The LLMaaJ setup using a chat model as judge has a sibling: the dedicated reward model (RM). An RM is a transformer with a scalar head trained on human preference data (typically Bradley-Terry logistic regression over chosen/rejected pairs). For full background see The RLHF Pipeline & Reward Modeling.

When to Use an RM vs. a Chat Judge¶

In practice, large RL training loops (see Policy Gradients & PPO for Language Models) use reward models because they must call the judge millions of times per training run. Chat-model judges are used for held-out offline evaluation at lower throughput requirements.

Reward Model as a Pointwise Judge¶

A reward model produces a scalar \(r \in \mathbb{R}\) (after the scalar head), higher is better. To use it as a judge:

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

class RewardModelJudge:
    """
    Wraps a Bradley-Terry reward model as a pointwise judge.
    Model must output a scalar logit (or use a single-logit head).
    """

    def __init__(
        self,
        model_name: str = "Skywork/Skywork-Reward-V2-Qwen3-4B",
        device: str = "cuda",
    ):
        self.tokenizer = AutoTokenizer.from_pretrained(model_name)
        self.model = AutoModelForSequenceClassification.from_pretrained(
            model_name,
            num_labels=1,                # single scalar head
            dtype=torch.bfloat16,        # `torch_dtype=` on transformers < 4.56
        ).to(device).eval()
        self.device = device

    @torch.no_grad()
    def score(self, question: str, response: str) -> float:
        """
        Returns a reward scalar. Higher = better.
        """
        conv = [{"role": "user",      "content": question},
                {"role": "assistant", "content": response}]
        # Modern open RMs are trained ON THE CHAT TEMPLATE of their base model.
        # Feeding a bare f"{question}\n\n{response}" string is the single most
        # common way to get scores that look plausible and mean nothing.
        text = self.tokenizer.apply_chat_template(conv, tokenize=False)
        tokens = self.tokenizer(
            text,
            return_tensors="pt",
            truncation=True,
            max_length=4096,
        ).to(self.device)
        logits = self.model(**tokens).logits  # shape: (1, 1) for scalar head
        return logits[0][0].item()

    def pairwise_judge(self, question: str, resp_a: str, resp_b: str) -> str:
        """Returns 'A', 'B', or 'tie' based on reward scores."""
        score_a = self.score(question, resp_a)
        score_b = self.score(question, resp_b)
        margin = abs(score_a - score_b)

        # RM outputs are unnormalized logits whose scale is model-specific
        # (one RM's "0.05" is another's "5"). Calibrate this threshold on your
        # gold set: pick the margin at which judge/human agreement peaks.
        if margin < 0.05:
            return "tie"
        return "A" if score_a > score_b else "B"

Choosing (or Training) the Reward Model¶

You do not pick an open RM by vibes. RewardBench (Lambert et al., 2024; allenai/reward-bench, now in a v2 edition) is the standard meta-evaluation: it scores an RM on how often it ranks a known-good response above a known-bad one across chat, reasoning, and safety splits, and it is the leaderboard the modern open RM families — Skywork-Reward-V2, and the reward models shipped alongside open post-training stacks — are tuned against. JudgeBench plays the same role for generative judges. Run the benchmark for the split that matches your domain, not the headline average.

If nothing off the shelf fits your task, training your own RM is a short job: TRL’s RewardTrainer takes a dataset of (prompt, chosen, rejected) and an AutoModelForSequenceClassification with num_labels=1 and optimizes exactly the Bradley-Terry loss derived in The RLHF Pipeline & Reward Modeling — the same likelihood we just fit for arena ratings, with a network in place of a lookup table of strengths. OpenRLHF and veRL wrap the same objective for multi-node scale.

Reward Model Calibration Issues¶

Reward models are susceptible to reward hacking — subtle distributional artifacts that inflate scores without genuine quality improvement. Common failure modes:

• Length hacking: RM scores correlate with response length independently of quality.
• Sycophancy reward: RM was trained on data where agreeable responses were preferred, so it over-rewards agreement.
• Distribution shift: RM generalizes poorly to prompts outside its training distribution.

These issues are covered in depth in Reward Hacking, Over-Optimization & Alignment Failures. From an evaluation standpoint: always compute a calibration Spearman \(\rho\) against held-out human labels before trusting RM scores as ground truth.

Judge Calibration, Auditing, and Trust¶

Deploying a judge in production without calibration is the most common mistake. Here is a systematic protocol.

The Judge Calibration Checklist¶

1. Build a gold set. Collect 300–500 (prompt, response) examples spanning the full quality range. Get majority-vote labels from at least 3 annotators. Include adversarial examples (long-but-wrong, short-but-right, off-topic, etc.).

2. Compute baseline metrics. Run the judge; compute Spearman \(\rho\), MAE, bias profile, and per-criterion calibration.

3. Measure known biases. For each of the biases in the previous section, construct minimal test pairs (e.g., identical content with different lengths to test verbosity bias).

4. Iterate on the judge prompt. If verbosity bias is measured at >5 pp win-rate difference between length-matched pairs, strengthen the anti-length instruction in the prompt.

5. Set acceptance thresholds. Before deploying a new judge or judge model, require Spearman \(\rho > 0.75\) and agreement rate >70% on the gold set.

6. Re-calibrate on model updates. When you upgrade the judge model (e.g., GPT-4o to GPT-4o-mini), re-run calibration. Score distributions can shift.

Confidence Intervals for Elo Rankings¶

A single Elo rating is not a stable estimate — it has uncertainty depending on the number of games played. Use bootstrap resampling to get confidence intervals:

import numpy as np
from copy import deepcopy

def bootstrap_elo_ci(battle_log: list[dict], n_bootstrap: int = 200, k: float = 32.0):
    """
    battle_log: list of {"model_a": str, "model_b": str, "outcome": str}
    Returns a dict of model -> (mean_rating, lower_95, upper_95)
    """
    all_ratings = defaultdict(list)

    for _ in range(n_bootstrap):
        # Sample with replacement
        sample = random.choices(battle_log, k=len(battle_log))

        ranker = EloRanker(k=k)
        for battle in sample:
            ranker.update(battle["model_a"], battle["model_b"], battle["outcome"])

        for model, rating, _ in ranker.leaderboard():
            all_ratings[model].append(rating)

    results = {}
    for model, ratings in all_ratings.items():
        arr = np.array(ratings)
        results[model] = {
            "mean":   round(float(np.mean(arr)), 1),
            "lower":  round(float(np.percentile(arr, 2.5)), 1),
            "upper":  round(float(np.percentile(arr, 97.5)), 1),
        }
    return results

If the 95% confidence intervals of two models overlap, you do not have statistically significant evidence that one outperforms the other — more battles are needed.

Production LLMaaJ: Design Considerations¶

Caching and Idempotency¶

Judge calls are expensive (API cost + latency). Cache results keyed on (judge_model, judge_prompt_hash, question_hash, response_hash). This makes re-runs cheap and evaluation truly reproducible.

import hashlib, json, sqlite3

def cache_key(judge_model, system_prompt, question, response):
    payload = json.dumps([judge_model, system_prompt, question, response], sort_keys=True)
    return hashlib.sha256(payload.encode()).hexdigest()

Parallelism¶

For bulk evaluation (thousands of queries), use async API calls with rate-limit-aware batching:

import asyncio
from openai import AsyncOpenAI

async_client = AsyncOpenAI()

async def judge_async(question: str, response: str, semaphore: asyncio.Semaphore) -> dict:
    async with semaphore:
        completion = await async_client.chat.completions.create(
            model="gpt-4o",
            messages=[
                {"role": "system", "content": JUDGE_SYSTEM_PROMPT},
                {"role": "user",   "content": f"QUESTION:\n{question}\n\nRESPONSE:\n{response}"},
            ],
            temperature=0.0,
            max_tokens=512,
            response_format={"type": "json_object"},
        )
        return json.loads(completion.choices[0].message.content)

async def bulk_judge(examples: list[dict], max_concurrency: int = 20) -> list[dict]:
    """
    examples: list of {"question": str, "response": str}
    max_concurrency: tune to stay under API rate limits
    """
    sem = asyncio.Semaphore(max_concurrency)
    tasks = [judge_async(ex["question"], ex["response"], sem) for ex in examples]
    return await asyncio.gather(*tasks)

Judge Cost Model¶

For a daily evaluation run of 10,000 examples using a mid-tier frontier judge (on the order of a few USD per million input tokens as of writing; frontier judge pricing has trended down year over year):

• Average prompt tokens: ~500 (system + question + response)
• Average output tokens: ~150 (rationale + JSON)
• Total tokens: 10,000 × 650 = 6.5M tokens
• Rough cost: on the order of USD 30–50 per run

For cost-sensitive pipelines, use a smaller judge model (GPT-4o-mini, Claude Haiku) for daily monitoring and reserve GPT-4o or Claude Opus for milestone evaluations. Always validate that the smaller judge maintains acceptable calibration. A locally served open judge (previous section) moves the bill from USD-per-run to GPU-hours, which is usually the right trade once you are judging daily.

Don’t Rebuild the Harness¶

Everything above is worth writing once so you understand it, but in production you will mostly configure an existing harness — see Building Eval Harnesses for how these fit together. The three that matter for judging:

pip install alpaca-eval          # length-controlled win rate vs a reference model
alpaca_eval --model_outputs my_outputs.json \
            --annotators_config weighted_alpaca_eval_gpt4_turbo

# deepeval: pytest-style assertions with a G-Eval judge metric.
from deepeval.metrics import GEval
from deepeval.test_case import LLMTestCase, LLMTestCaseParams

correctness = GEval(
    name="Correctness",
    criteria="Is every factual claim in the output accurate and supported?",
    evaluation_params=[LLMTestCaseParams.INPUT, LLMTestCaseParams.ACTUAL_OUTPUT],
)
correctness.measure(LLMTestCase(input=q, actual_output=r_good))
print(correctness.score, correctness.reason)

The third, EleutherAI’s lm-evaluation-harness, is not a judge framework — it scores log-likelihood and exact-match benchmarks — but it is the harness the same pipeline uses for the closed-ended half of an eval suite, so keep both and report them side by side.

The Judge as a Data Generator¶

A pairwise judge is not only an evaluator: it is how you turn unlabeled model samples into preference data. Sample \(k\) responses per prompt from your own model, judge them against each other, and emit the winner/loser as a (prompt, chosen, rejected) pair — this is the mining step that makes DPO on-policy, which the post-training literature identifies as the dominant factor in whether preference optimization helps at all.

def mine_preference_pairs(judge_fn, prompts, sample_fn, k: int = 4):
    """
    sample_fn(prompt, k) -> list[str] of k on-policy samples.
    Emits (prompt, chosen, rejected) using best-vs-worst by pairwise judging.
    """
    pairs = []
    for prompt in prompts:
        cands = sample_fn(prompt, k)
        n = len(cands)
        wins = [0] * n
        for i in range(n):
            for j in range(i + 1, n):
                # Order-swapped judging: never mine pairs off a biased verdict.
                out = pairwise_judge_debiased(judge_fn, prompt, cands[i], cands[j])
                if out == "A": wins[i] += 1
                elif out == "B": wins[j] += 1
        best = max(range(n), key=lambda i: wins[i])
        worst = min(range(n), key=lambda i: wins[i])
        if wins[best] > wins[worst]:          # drop prompts where the judge is indifferent
            pairs.append({"prompt": prompt, "chosen": cands[best], "rejected": cands[worst]})
    return pairs

The capstone uses exactly this loop — a larger open model judging Stack-100M’s own samples — to build its DPO set; see Post-Training: SFT, DPO, and Narrow RLVR (GRPO) That Works at 100M. One caution specific to small models: never let a 100M model judge itself. It is both the weakest available judge and maximally self-preferring, and where the task admits a programmatic checker (arithmetic, unit tests, format), a verifiable reward beats any judge outright — that is the argument for RLVR over judge-based rewards at small scale.

Further Reading¶

• Zheng et al., “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena” (LMSYS, 2023) — the foundational paper defining the LLMaaJ paradigm and MT-Bench; analyzes position and verbosity biases.
• Liu et al., “G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment” (2023) — introduces CoT-before-score and shows that using GPT-4 with structured criteria significantly correlates with human judgments on summarization and dialogue.
• Wang et al., “Large Language Models are not Fair Evaluators” (2023) — systematic measurement of position bias and the swap-ordering mitigation.
• Dubois et al., “AlpacaEval: An Automatic Evaluator for Instruction-following Language Models” (2023) — the AlpacaEval benchmark; open-source automated evaluation harness with length-controlled win rate.
• Chiang & Lee, “Can Large Language Models Be an Alternative to Human Evaluations?” (ACL 2023) — empirical study of agreement rates and failure modes.
• Ouyang et al., “Training language models to follow instructions with human feedback” (InstructGPT, 2022) — reward modeling methodology that underpins RM-as-judge.
• Chatbot Arena Leaderboard (LMSYS) — live crowdsourced human preference ranking; a model for large-scale Elo-based evaluation.
• OpenAI Evals (github.com/openai/evals) — open-source eval framework with LLMaaJ templates and model-graded eval types.

Exercises¶

1. Pairwise judging achieves higher agreement with human preference than pointwise scoring (see the “When to Use Each” table), yet the same table lists pairwise as having high susceptibility to position bias while pointwise is low. Explain, in terms of what each paradigm asks the judge to do, why relative comparison is both better calibrated and newly vulnerable to position bias — a failure mode pointwise scoring essentially does not have.

2. Use the weighted aggregation from the chapter’s judge_response implementation, with weights correctness=0.4, helpfulness=0.4, conciseness=0.1, safety=0.1. A judge returns these per-criterion scores for the two responses in the chapter’s usage example:

• Concise answer (“Paris is the capital…”): correctness 5, helpfulness 5, conciseness 5, safety 5.
• Verbose answer (“Great question! France…”): correctness 5, helpfulness 4, conciseness 2, safety 5.

Compute the overall score for each. Does the aggregation reward the concise answer, as the code comment (“A well-calibrated judge should penalize verbosity here”) predicts? Which single criterion drives the gap, and by how much of the overall difference?

3. Two models enter an automated arena using the chapter’s EloRanker (start rating 1000, \(K = 32\), base 10, scale 400).

(a) In the very first battle, model A and model B are both at 1000 and the judge declares A the winner. Compute both new ratings.

(b) Later, model A sits at 1200 and model B at 1000. Compute \(P(A \succ B)\), the judge-free expected win probability.

© The chapter says a 400-point gap corresponds to a 10:1 win-rate advantage. What rating gap corresponds to a 3:1 advantage (i.e. \(P(A \succ B) = 0.75\))?

4. You audit a pairwise judge against a human-labeled gold set. The judge’s label marginals are A \(= 60\%\), B \(= 30\%\), tie \(= 10\%\); the human majority-vote marginals are A \(= 50\%\), B \(= 35\%\), tie \(= 15\%\) (these are exactly the numbers in the chapter’s Cohen’s-kappa section). Suppose the judge and human labels agree on \(72\%\) of examples (\(p_o = 0.72\)). Compute Cohen’s \(\kappa\). Interpret the result against the chapter’s stated range for a “good judge,” and explain why raw agreement of \(72\%\) overstates the judge’s quality.

5. The chapter’s pairwise_judge_debiased runs both orderings and returns "tie" whenever they disagree — but it discards the fact that a disagreement happened, which is exactly the signal you need for the calibration protocol’s “bias profile per ordering.” Implement a function

```python
def measure_position_bias(judge_fn, pairs):
    ...
```

that runs both orderings for every pair and estimates the judge's *first-position preference rate*: among all non-tie decisions across both orderings, the fraction in which the judge picked whichever response was presented **first**. Under an unbiased judge this rate should be about $0.50$; a value well above $0.50$ indicates first-position bias. `pairs` is a list of `(question, response_a, response_b)` tuples; `judge_fn(question, first, second)` returns `"A"`, `"B"`, or `"tie"`, where `"A"` always means "the first-presented response won." Return the rate (or `None` if there are no decisive judgments).

def measure_position_bias(judge_fn, pairs):
    """
    pairs: list of (question, response_a, response_b)
    judge_fn(question, first, second) -> "A" | "B" | "tie"
        where "A" means the FIRST-presented response won.

    Returns the first-position preference rate: among all non-tie
    decisions (2 per pair), the fraction won by the first slot.
    ~0.5 => unbiased; >0.5 => favors first position. None if no
    decisive judgments.
    """
    first_wins = 0
    decisive = 0

    for question, resp_a, resp_b in pairs:
        # Ordering 1: resp_a first, resp_b second
        r1 = judge_fn(question, resp_a, resp_b)
        # Ordering 2: swap so resp_b is now first
        r2 = judge_fn(question, resp_b, resp_a)

        for r in (r1, r2):
            if r == "tie":
                continue
            decisive += 1
            if r == "A":        # "A" == first-presented slot won
                first_wins += 1

    if decisive == 0:
        return None
    return first_wins / decisive