11.4 Reasoning, Coding & Agentic Evals¶

Evaluating a model’s raw language modeling performance — perplexity, BLEU, or even static multiple-choice benchmarks — tells us almost nothing about whether it can actually solve problems. Reasoning, coding, and agentic tasks are the canonical hard cases: the answer is either right or wrong (or can be verified cheaply), the failure modes are subtle, and the evaluation methodology itself can be gamed in ways that corrupt the signal entirely. This chapter covers the techniques that make these evals trustworthy.

We build from first principles: why exact-match is insufficient, how Pass@k turns stochastic generation into a reliable probability estimate, how sandboxed execution verifies correctness without trusting the model, how reasoning traces are evaluated rather than just answers, and how modern agentic harnesses turn long multi-step tasks into reproducible benchmarks. We close with test-time-compute-aware evaluation and a rigorous treatment of reward hacking in eval contexts.

Related background: the broader benchmark landscape is covered in The Evaluation Problem & Benchmark Landscape, LLM-as-a-Judge methodology lives in LLM-as-a-Judge & Automated Evaluation, and how to build a general eval harness is in Building Eval Harnesses. The RL machinery that trains reasoning models — and why its reward signal must be trustworthy — is covered in RL with Verifiable Rewards (RLVR) & The Reasoning Recipe and Reward Engineering, Verifiers & Sandboxes. Agent-specific benchmarks and trajectory-scoring methodology get a full treatment in Agent Evaluation & Benchmarks; the confidence intervals and significance tests you need before claiming one Pass@k beats another are in Statistical Rigor in Evaluation: Confidence Intervals & Significance. Everything here is applied at small scale, on a model you built yourself, in Evaluation & Serving: Honest Benchmarks, int4 Quantization, and Running on a Laptop.

Pass@k: Measuring Coding Ability as a Probability¶

Why a Single Sample Is Not Enough¶

A model that generates correct code 30% of the time and a model that generates correct code 80% of the time both produce some correct outputs if you generate enough samples. Evaluating coding ability on a single generation per problem conflates two things: the model’s intrinsic ability and sampling luck. Pass@1 (the fraction of problems solved on the first try) is what matters for single-shot user interactions, but it underestimates capability when \(k > 1\) generations are allowed — which is the norm in production coding agents.

Pass@k is the probability that at least one of \(k\) independent samples passes all tests for a given problem. If we generate \(n\) samples per problem (with \(n \geq k\)) and \(c\) of them pass, the unbiased estimator is:

This avoids the naive estimator \(1 - (1 - c/n)^k\), which is biased when \(c/n\) is estimated from finite samples. The combinatorial form is mathematically exact: it computes the probability that a random draw of \(k\) samples from the \(n\) generated contains zero passing samples, then subtracts from one.

It is worth being precise about what is unbiased and under what assumption. Model the \(n\) samples as i.i.d. draws that each pass with unknown probability \(p\), so \(c \sim \text{Binomial}(n, p)\) and the estimand is \(\text{Pass@}k = 1 - (1-p)^k\). The ratio \(\binom{n-c}{k}/\binom{n}{k}\) satisfies \(\mathbb{E}\!\left[\binom{n-c}{k}/\binom{n}{k}\right] = (1-p)^k\) exactly, so the combinatorial form is an unbiased estimator of \(\text{Pass@}k\) for every finite \(n \geq k\). The plug-in form is not: because \(x \mapsto (1-x)^k\) is convex, Jensen’s inequality gives \(\mathbb{E}[(1-\hat{p})^k] \geq (1-p)^k\), so \(1 - (1-\hat{p})^k\) is biased downward — it systematically understates Pass@k, exactly the direction you see in the worked example below. The i.i.d. assumption is not free: it holds when you sample independently at a fixed temperature, and breaks if you use beam search, diverse-decoding heuristics, or a rejection loop that conditions later samples on earlier failures. Unbiasedness also says nothing about variance: a Pass@k estimate from \(n=20\) samples on 164 problems still carries a standard error of a few percentage points, which is why comparisons need the interval machinery of Statistical Rigor in Evaluation: Confidence Intervals & Significance.

Implementation¶

import math
from typing import List


def pass_at_k(n: int, c: int, k: int) -> float:
    """
    Unbiased Pass@k estimator from Chen et al. (HumanEval, 2021).

    Args:
        n: total number of generated samples for this problem
        c: number of samples that pass all tests
        k: the k in Pass@k (e.g. 1, 10, 100)

    Returns:
        estimated probability that at least one of k random draws passes.

    Note: uses log-space arithmetic to avoid combinatorial overflow.
    """
    if n < k:
        raise ValueError(f"n={n} must be >= k={k}")
    if c > n:
        raise ValueError(f"c={c} cannot exceed n={n}")
    if c == n:
        # All samples pass — all possible draws of k pass.
        return 1.0
    if n - c < k:
        # Fewer failing samples than k: any draw of k must include a
        # passing sample. C(n-c, k) is 0 by definition (can't choose k
        # items from a pool of n-c), so the failure probability is 0.
        return 1.0
    # Numerically stable: compute log(C(n-c, k) / C(n, k))
    # = sum(log((n-c-i)/(n-i)) for i in range(k))
    log_prob_fail = sum(
        math.log(n - c - i) - math.log(n - i)
        for i in range(k)
    )
    return 1.0 - math.exp(log_prob_fail)


def aggregate_pass_at_k(results: List[dict], k: int) -> float:
    """
    Average Pass@k across problems.

    Each entry in results: {'n': int, 'c': int}
    """
    scores = [pass_at_k(r['n'], r['c'], k) for r in results]
    return sum(scores) / len(scores)


# Example: HumanEval-style evaluation
problems = [
    {'n': 20, 'c': 6},    # moderate difficulty
    {'n': 20, 'c': 19},   # easy
    {'n': 20, 'c': 0},    # failed completely
    {'n': 20, 'c': 2},    # hard
]

for k in [1, 10, 20]:
    score = aggregate_pass_at_k(problems, k)
    print(f"Pass@{k:>2d}: {score:.3f}")

Pass@ 1: 0.338
Pass@10: 0.689
Pass@20: 0.750

Pass@k is monotonically non-decreasing in \(k\) for a fixed \(n\): drawing more samples can only increase the chance that at least one of them passes. Pass@20 here equals \(k=n=20\) — drawing all the generated samples — so it collapses to a simple rule: the score is 1.0 for every problem with at least one passing sample (\(c > 0\)) and 0.0 for the one problem with \(c=0\), giving \((1+1+0+1)/4 = 0.750\). That one \(c=0\) problem is also what caps Pass@20 below 1.0: no amount of additional draws can produce a pass when zero of the generated samples passed.

Code Execution Sandboxes¶

Why Execution Is Non-Negotiable¶

String matching against expected outputs fails for several reasons: (1) there are many syntactically distinct but semantically equivalent programs; (2) floating-point outputs differ across platforms; (3) the model may produce correct code with wrong formatting. Ground truth for code is behavior, not text. This means we must run the generated code.

Running untrusted model-generated code on the host machine is a security catastrophe: the code might delete files, exfiltrate secrets, or crash the evaluator. A proper sandbox provides:

• Isolation: the evaluated process cannot reach the host filesystem or network.
• Resource limits: CPU time, memory, and file descriptor caps prevent runaway resource consumption.
• Reproducibility: identical OS environment for every run.

Sandbox Architectures¶

Docker + seccomp is the most common approach for research harnesses — SWE-bench, for instance, ships one Docker image per benchmark instance so that a repo’s dependency pins can never leak across tasks. For stronger isolation, gVisor (Google’s user-space kernel, which intercepts syscalls before they reach the host kernel) and Firecracker microVMs (the substrate under AWS Lambda, which boots a microVM in on the order of 100 ms) give VM-grade boundaries without a full VM’s boot cost.

Below the container layer, the real open-source primitives are worth naming because you will reach for them directly when Docker-per-execution is too slow: nsjail (Google) and bubblewrap (the sandbox under Flatpak) both wrap Linux namespaces, cgroups, and seccomp-BPF filters behind a single command line, so you can spawn a locked-down python3 in a few milliseconds rather than a few hundred; firejail is a lighter-weight alternative. For hosted execution — the usual choice when eval or RL rollouts need thousands of concurrent sandboxes and you do not want to operate the fleet — E2B, Modal, and Daytona expose per-execution microVM sandboxes behind a Python SDK. The same sandbox pool typically serves both eval and RLVR training, which is precisely why its correctness matters twice over; see Reward Engineering, Verifiers & Sandboxes.

A Minimal Sandbox in Python¶

import subprocess
import tempfile
import os
from dataclasses import dataclass
from typing import Optional


@dataclass
class ExecutionResult:
    passed: bool
    exit_code: int
    stdout: str
    stderr: str
    timed_out: bool
    oom_killed: bool


def run_in_sandbox(
    code: str,
    test_cases: list[dict],
    timeout_seconds: float = 5.0,
    memory_mb: int = 256,
) -> ExecutionResult:
    """
    Executes model-generated Python code against unit tests in a subprocess
    with resource limits.

    In production, replace the subprocess call with a Docker/gVisor invocation.
    Here we use resource limits via `resource` module for demonstration.
    """
    # Build a self-contained test script: solution + assertions
    test_body = "\n".join(
        f"assert {tc['input']} == {tc['expected']!r}, "
        f"'Failed: {tc['input']}'"
        for tc in test_cases
    )
    # Note: interpolate at column 0, don't dedent afterwards — `code` is
    # itself unindented, and textwrap.dedent() looks at the *combined*
    # string's common leading whitespace. Since {code} contributes
    # zero-indent lines, dedent would find no common prefix and leave the
    # template lines (e.g. "import resource, sys") wrongly indented,
    # causing an IndentationError before the script even runs.
    full_script = f"""
import resource, sys
# Hard memory limit (Linux only)
resource.setrlimit(
    resource.RLIMIT_AS,
    ({memory_mb * 1024 * 1024}, {memory_mb * 1024 * 1024})
)
# --- Model-generated solution below ---
{code}
# --- Test assertions ---
{test_body}
print("PASS")
"""

    with tempfile.NamedTemporaryFile(
        mode='w', suffix='.py', delete=False
    ) as f:
        f.write(full_script)
        script_path = f.name

    try:
        result = subprocess.run(
            ["python3", script_path],
            capture_output=True,
            text=True,
            timeout=timeout_seconds,
            # Additional isolation: no inherited env vars
            env={"PATH": "/usr/bin:/bin"},
        )
        passed = (result.returncode == 0 and "PASS" in result.stdout)
        return ExecutionResult(
            passed=passed,
            exit_code=result.returncode,
            stdout=result.stdout,
            stderr=result.stderr,
            timed_out=False,
            oom_killed="MemoryError" in result.stderr or result.returncode == -9,
        )
    except subprocess.TimeoutExpired:
        return ExecutionResult(
            passed=False, exit_code=-1,
            stdout="", stderr="Timeout",
            timed_out=True, oom_killed=False,
        )
    finally:
        os.unlink(script_path)


# Example usage
solution_code = """
def two_sum(nums, target):
    seen = {}
    for i, n in enumerate(nums):
        diff = target - n
        if diff in seen:
            return [seen[diff], i]
        seen[n] = i
    return []
"""

test_cases = [
    {"input": "two_sum([2,7,11,15], 9)", "expected": [0, 1]},
    {"input": "two_sum([3,2,4], 6)",     "expected": [1, 2]},
    {"input": "two_sum([3,3], 6)",        "expected": [0, 1]},
]

result = run_in_sandbox(solution_code, test_cases)
print(f"Passed: {result.passed}, exit={result.exit_code}")

HumanEval, MBPP, and LiveCodeBench¶

The seminal HumanEval benchmark (Chen et al., 2021) contains 164 Python programming problems with unit tests, designed so solutions cannot be looked up from training data. MBPP (Mostly Basic Python Problems, Austin et al., 2021) covers 374 crowd-sourced problems. Both are now considered partially contaminated — frontier models were trained on code from the web that likely contains solutions.

Contamination is not their only weakness. EvalPlus (Liu et al., 2023) showed that HumanEval’s hand-written test suites are so thin that many “passing” solutions are simply wrong on untested inputs; it augments each problem with automatically generated inputs (roughly two orders of magnitude more tests) to produce HumanEval+ and MBPP+, on which reported pass rates typically drop by several points. If you report HumanEval at all in 2026, report the plus variants — they are the same problems with an honest oracle.

You should not hand-roll the runner for any of these. The standard open-source tooling:

# EvalPlus: generate + execute against the augmented test suites.
pip install "evalplus[vllm]"
evalplus.evaluate --model "Qwen/Qwen2.5-Coder-7B-Instruct" \
  --dataset humaneval --backend vllm --greedy   # reports pass@1 for HumanEval and HumanEval+

# BigCode's harness: many code benchmarks (HumanEval, MBPP, MultiPL-E,
# DS-1000) behind one CLI, with containerized execution.
# `--n_samples`/`--temperature` are what let it report pass@k for k > 1.
accelerate launch bigcode-evaluation-harness/main.py \
  --model bigcode/starcoder2-7b --tasks humaneval \
  --n_samples 20 --temperature 0.2 --allow_code_execution

EleutherAI’s lm-evaluation-harness also carries code tasks and is the right choice when you want code numbers in the same run as everything else (see Building Eval Harnesses); note that all of these require an explicit opt-in flag before they will execute model output, which is the correct default.

LiveCodeBench (Jain et al., 2024) addresses contamination by continuously adding new problems from competitive programming platforms (Leetcode, Codeforces, AtCoder) after each model’s training cutoff. This makes it a living benchmark where contamination is structurally impossible for recent additions.

SWE-bench (Jimenez et al., 2023) is qualitatively different: it presents real GitHub issues from open-source Python repositories and asks the model to produce a patch that makes the failing CI tests pass. The sandbox here is the project’s own test suite, and success is measured by the fraction of tests that flip from red to green. This is far harder than HumanEval and requires navigating real codebases, reading documentation, and multi-file edits.

Math Verification and Reasoning-Trace Evaluation¶

Exact-Match vs. Symbolic Equivalence¶

Math problems have a special property: the final answer is often a number, expression, or set that can be verified cheaply. But string exact-match fails embarrassingly: “½” and “0.5” are the same answer, as are “\(x = 3\)” and “\(x=3\)” and “3”.

The standard approach is normalization + symbolic equivalence:

1. Strip LaTeX formatting and whitespace.
2. Parse into a symbolic expression (using Python’s sympy).
3. Check symbolic equality, not string equality.

from sympy import sympify, simplify, Eq, solve, latex
from sympy.parsing.latex import parse_latex


def math_answers_equivalent(model_answer: str, gold_answer: str) -> bool:
    """
    Check if two math answer strings are symbolically equivalent.
    Handles: numbers, fractions, expressions, simple equations.

    Caveats: does not handle set answers, multiple solutions, or
    answers requiring context (e.g., 'x > 0').
    """
    # Try numeric equality first (fast path)
    try:
        a = float(sympify(model_answer))
        b = float(sympify(gold_answer))
        return abs(a - b) < 1e-6
    except Exception:
        pass

    # Symbolic equality (slower path)
    try:
        expr_a = parse_latex(model_answer)
        expr_b = parse_latex(gold_answer)
        # simplify(A - B) == 0 iff A == B
        diff = simplify(expr_a - expr_b)
        return diff == 0
    except Exception:
        pass

    # Fallback: normalized string match
    def normalize(s):
        return s.strip().replace(" ", "").replace("\\,", "").lower()

    return normalize(model_answer) == normalize(gold_answer)


# Test
pairs = [
    ("\\frac{1}{2}", "0.5"),             # should be True
    ("x^2 + 2x + 1", "(x+1)^2"),         # should be True
    ("3.14159", "\\pi"),                   # should be False
    ("\\sqrt{4}", "2"),                    # should be True
]

for a, b in pairs:
    print(f"  '{a}' == '{b}': {math_answers_equivalent(a, b)}")

Two practical notes on that function. First, sympy.parsing.latex.parse_latex needs the optional antlr4-python3-runtime package installed; without it the LaTeX branch raises and control falls through to the string-normalization fallback, which silently turns real matches into false negatives. Second — and more important — do not ship this. Hand-rolled graders are where math evals quietly lose several points of accuracy: they mishandle \boxed{} extraction, units, intervals, sets, matrices, tuples, and \%, and every research group’s version differs, which is a large part of why published MATH/AIME numbers are hard to reproduce.

The 2026 default is HuggingFace’s math-verify, a dedicated answer-checking library that grew out of the Open-R1 effort and is the grader behind lighteval. It parses both LaTeX and plain expressions, extracts the final answer, and compares with sympy under the hood, with the edge cases already handled:

# pip install math-verify
from math_verify import parse, verify

# `parse` returns a list of candidate interpretations (most-specific first);
# `verify(gold, answer)` is True if ANY gold form matches ANY answer form.
gold = parse("${1,3} \\cup {2,4}$")
answer = parse("${1,2,3,4}$")
print(verify(gold, answer))          # True — set-valued answers just work

print(verify(parse("$\\frac{1}{2}$"), parse("$0.5$")))       # True
print(verify(parse("$\\boxed{2}$"),   parse("$\\sqrt{4}$")))  # True

The same library is the natural reward function for math RLVR, which is the point made in Post-Training: SFT, DPO, and Narrow RLVR (GRPO) That Works at 100M: eval verifier and training verifier should be the same code path, so a grading bug shows up as a flat reward curve rather than as a phantom capability. Whatever grader you use, log the parse-failure rate separately from accuracy — “wrong answer” and “unparseable output” demand opposite fixes.

Benchmarks: MATH, GSM8K, AMC/AIME, MMLU-Pro¶

GSM8K (Cobbe et al., 2021) contains 8,500 grade-school math word problems. Every problem has a step-by-step solution and a final numeric answer. It was saturated by frontier models around 2024 — many models exceed 90% accuracy, making it a poor discriminator at the top.

MATH (Hendrycks et al., 2021) is substantially harder: 12,500 competition math problems across algebra, geometry, number theory, and calculus, at levels AMC 8 through AIME. Difficulty is labeled 1–5; level 5 problems (olympiad-style) remain challenging even for top models.

AIME (American Invitational Mathematics Examination) is now commonly used as a frontier discriminator. It is scored out of 15 (each problem worth 1 point), with no partial credit. The short integer answer format makes verification trivial while the mathematical depth is substantial. By 2025–2026 the strongest reasoning models score near the top of the scale on AIME — approaching saturation, a level previously only top human competitors reached — so each year’s fresh contest (e.g., AIME 2025) is used as a contamination-controlled discriminator, and evaluators are increasingly moving to newer, harder olympiad sets as headroom on AIME disappears.

Olympiad-level benchmarks (FrontierMath, OlympiadBench) push further, collecting unpublished research-level problems to prevent data contamination. Beyond math specifically, Humanity’s Last Exam (Phan et al., 2025) assembles a few thousand expert-written questions across many disciplines with the explicit design goal of resisting saturation; it, FrontierMath, and fresh contest sets are the current generation of frontier discriminators, and each of them will in turn be saturated — the honest lesson is that a benchmark’s useful life is now measured in months, so how you rotate benchmarks matters more than which one you pick today.

Formal-proof benchmarks are a categorically different kind of math eval. Every benchmark above (GSM8K, MATH, AIME, FrontierMath, OlympiadBench) is informal: the model writes natural-language reasoning and the grader checks only the final answer, which is why the whole symbolic-equivalence machinery earlier in this chapter is necessary — and why it fails on set-valued answers, multiple solutions, or expressions sympy cannot parse, producing false negatives even when the model is correct. miniF2F (Zheng et al., 2021; ~488 olympiad and textbook problems), ProofNet (Azerbayev et al., 2023; undergraduate pure-math theorems), and PutnamBench (Tsoukalas et al., 2024; Putnam competition problems) sidestep that problem entirely. Here the “answer” is not a number but a complete formal proof written in a proof assistant such as Lean 4 (Isabelle and Coq variants also exist), and the verifier is the assistant’s own type-checker/kernel: a proof either type-checks against the formal statement or it does not. There is no equivalence heuristic and therefore no false-negative equivalence-checking problem — the same property that makes formally verified rewards attractive for RLVR (see the cross-links above). The tradeoff is upfront cost: each problem’s statement must first be formalized into the proof assistant’s language, which is labor-intensive and can itself introduce statement-level errors, so these benchmarks are far smaller than their informal counterparts and measure a narrower, harder skill (autoformalization plus proof search) rather than everyday numeric reasoning.

Evaluating the Reasoning Trace, Not Just the Answer¶

A model can get the right answer via a flawed reasoning chain (lucky coincidence, “sycophancy” in the trace, or incorrect steps that happen to cancel). Conversely, a model can write a perfect chain but make an arithmetic error at the last step. Evaluating only the final answer misses both cases.

Process reward models (PRMs) assign a score to each step in a chain-of-thought solution, trained on human annotations of step correctness. PRMs can be used at eval time to:

1. Rank samples: among \(n\) rollouts with the correct final answer, prefer those where all intermediate steps are also scored highly.
2. Identify failure modes: a model might consistently produce correct answers via flawed steps — an important red flag for reasoning training.

Outcome reward models (ORMs) score only the final answer. They are cheaper to train but cannot distinguish lucky correct answers from reliably correct reasoning.

The interplay between PRMs and ORMs in training is covered in RL with Verifiable Rewards (RLVR) & The Reasoning Recipe. For eval purposes, running both a final-answer check and a PRM on a sample of traces gives the most complete picture of reasoning quality.

Agentic Eval Harnesses¶

What Makes Agentic Eval Hard¶

A single-turn QA benchmark has one input and one output. An agentic task might involve 10–50 model calls, tool invocations, file reads, web searches, and code executions. The challenges are:

• Partial credit: the task may be 80% complete but fail the final assertion. Binary pass/fail discards useful signal.
• Non-determinism: different runs of the same model on the same task may branch differently, requiring multiple rollouts.
• Environment statefulness: a web browsing task that clicks the wrong button early may be unrecoverable, masking the model’s ability to handle later steps correctly.
• Evaluation cost: running a 30-step agent trajectory on 500 tasks, each with 5 rollouts, is expensive.

SWE-bench Verified¶

SWE-bench (Jimenez et al., 2023) and its Verified variant (where human judges confirmed the task specification is unambiguous) are the primary coding-agent benchmarks. Each instance is:

Repo:        django/django
Issue:       "QuerySet.delete() crashes with an F() expression in filter"
Gold patch:  a real commit from the repo history that fixes the issue
Test:        the CI test suite; success = failing tests now pass

The harness must: 1. Check out the repo at the commit before the fix. 2. Confirm the relevant tests fail on the un-patched repo. 3. Apply the model’s patch. 4. Run the test suite. 5. Confirm the previously-failing tests now pass and no new failures were introduced.

This pipeline is containerized per-instance to avoid cross-contamination. The Docker image for each repo is built once and cached. The resolved rate (fraction of issues fixed) is the primary metric.

You run the official harness rather than reimplementing it, because the environment setup (exact dependency pins, which tests count as FAIL_TO_PASS vs. PASS_TO_PASS) is the benchmark. Your agent’s only job is to emit a JSONL file of predictions:

pip install swebench
# predictions.jsonl: one object per instance —
#   {"instance_id": "django__django-11099",
#    "model_name_or_path": "my-agent-v3",
#    "model_patch": "diff --git a/... "}
python -m swebench.harness.run_evaluation \
  --dataset_name princeton-nlp/SWE-bench_Verified \
  --predictions_path predictions.jsonl \
  --max_workers 8 --run_id my-agent-v3
# -> my-agent-v3.json with resolved_ids / unresolved_ids and per-instance logs

Note the split of responsibilities: the harness scores patches, it does not produce them. Generating model_patch is the agent’s problem, and the agent-computer interface you wrap around the model dominates the resulting score — the same model can move by tens of points depending on its edit format and search tools, which is the argument developed in Harness Engineering: Building a Coding Agent. Open-source reference agents (SWE-agent, OpenHands, Aider) exist precisely so that a model comparison is not silently a scaffold comparison; always report which scaffold produced a SWE-bench number.

WebArena, OSWorld, and τ-Bench¶

WebArena (Zhou et al., 2023) evaluates agents on realistic web tasks: booking a flight, submitting a form, finding information in a CMS. The agent controls a real browser (Playwright) inside a sandbox, and success is measured by final page state or database content — not by what the agent said it did.

OSWorld extends this to full desktop environments: the agent interacts with a virtual machine via screenshot + keyboard/mouse, performing tasks in real applications (LibreOffice, Chrome, terminal).

τ-bench (Yao et al., 2024) focuses on tool-augmented agents in retail and airline domains, with a simulated database and user simulator. The key metric is task completion rate across multi-turn conversations rather than single-step accuracy; τ-bench also popularized pass^k (pass-hat-k: the fraction of tasks solved in all \(k\) independent trials), which measures reliability rather than coverage and is the strictly harder sibling of Pass@k — an agent you would actually deploy needs pass^8 to be high, not just Pass@8.

Two more families are worth knowing. GAIA (Mialon et al., 2023) tests general assistants on questions that are easy for humans but require multi-step tool use, web browsing, and file handling; terminal-bench (2025) scores agents on end-to-end tasks inside a real terminal container. SWE-bench itself has grown variants — Lite (a cheap subset), Multimodal, and Multilingual — so “SWE-bench score” without a split name is not a number. The full landscape, including trajectory-level scoring and harness-effect analysis, is in Agent Evaluation & Benchmarks.

On the tooling side, the framework most worth adopting for agentic evals is Inspect AI (UK AI Security Institute): it models an eval as a Task with a dataset, a solver (which may be a full tool-using agent loop), and a scorer, and it ships first-class sandboxing (sandbox="docker") so tool calls execute inside a per-sample container without your writing any of the plumbing in the next section by hand. Build the harness below once to understand the mechanism; use Inspect AI (or the benchmark’s own official harness) when you need numbers other people will believe.

Building a Custom Agentic Eval Harness¶

"""
Minimal agentic eval harness illustrating the core loop.
Separates: task loading, trajectory collection, and scoring.
"""

import json
import asyncio
from dataclasses import dataclass, field
from typing import Callable, Any
from abc import ABC, abstractmethod


@dataclass
class AgentTask:
    task_id: str
    description: str           # Natural-language task specification
    initial_state: dict        # Starting environment state
    success_fn: Callable       # fn(final_state) -> bool
    max_steps: int = 30


@dataclass
class TrajectoryStep:
    step_idx: int
    action: str               # e.g., "bash('ls -la')"
    observation: str          # environment response
    success_so_far: bool      # did this step crash?


@dataclass
class Trajectory:
    task_id: str
    steps: list[TrajectoryStep] = field(default_factory=list)
    final_state: dict = field(default_factory=dict)
    completed: bool = False
    error: str = ""


class AgentEnvironment(ABC):
    @abstractmethod
    async def reset(self, initial_state: dict) -> str:
        """Reset to initial state, return initial observation."""

    @abstractmethod
    async def step(self, action: str) -> tuple[str, bool, dict]:
        """Execute action; return (observation, done, final_state)."""


class AgentModel(ABC):
    @abstractmethod
    async def act(self, task: str, history: list[TrajectoryStep]) -> str:
        """Given task description and history, return next action string."""


async def collect_trajectory(
    task: AgentTask,
    agent: AgentModel,
    env: AgentEnvironment,
) -> Trajectory:
    """Run one agent episode and collect the full trajectory."""
    traj = Trajectory(task_id=task.task_id)
    obs = await env.reset(task.initial_state)
    history = []

    for step_idx in range(task.max_steps):
        try:
            # Model chooses next action
            action = await agent.act(task.description, history)
            # Environment executes it
            obs, done, final_state = await env.step(action)

            step = TrajectoryStep(
                step_idx=step_idx,
                action=action,
                observation=obs,
                success_so_far=True,
            )
            history.append(step)
            traj.steps.append(step)

            if done:
                traj.final_state = final_state
                break
        except Exception as e:
            traj.error = str(e)
            break

    # Score the trajectory
    traj.completed = task.success_fn(traj.final_state)
    return traj


async def evaluate_agent(
    tasks: list[AgentTask],
    agent: AgentModel,
    env_factory: Callable[[], AgentEnvironment],
    rollouts_per_task: int = 3,
) -> dict[str, Any]:
    """
    Evaluate an agent across all tasks with multiple rollouts.
    Returns summary statistics.
    """
    results = {t.task_id: [] for t in tasks}

    for task in tasks:
        for rollout in range(rollouts_per_task):
            env = env_factory()
            traj = await collect_trajectory(task, agent, env)
            results[task.task_id].append(traj.completed)

    # Task is "solved" if at least one rollout succeeded (Pass@rollouts_per_task)
    solved = sum(any(r) for r in results.values())
    total = len(tasks)
    pass_at_1 = sum(r[0] for r in results.values()) / total
    pass_at_k = solved / total

    return {
        "pass_at_1": pass_at_1,
        f"pass_at_{rollouts_per_task}": pass_at_k,
        "per_task_results": results,
        "mean_steps": None,  # compute from trajectories
    }

Partial Credit and Step-Level Metrics¶

For long-horizon tasks, pass/fail is too coarse. A 20-step task where the model completes 18 steps correctly but fails on step 19 should score higher than one where it fails on step 2. Useful step-level metrics:

• Progress rate: fraction of subtasks completed before failure.
• Recovery rate: given an injected error at step \(i\), how often does the agent recover?
• Efficiency: number of steps taken divided by the oracle minimum (lower is better).

These metrics require decomposing the task into checkable substeps, which demands additional annotation effort but produces far richer signals than binary success.

Test-Time-Compute-Aware Evaluation¶

The Problem with Fixed-Compute Evals¶

Before “thinking” models (o1, DeepSeek-R1, QwQ, Gemini 2.0 Flash Thinking), all models used roughly the same compute per token at inference: one forward pass per generated token. Test-time compute is now a dial: a model can spend one token or 10,000 tokens of chain-of-thought before producing an answer.

This creates a fundamental incomparability problem: a model scoring 80% on AIME with 1,000 reasoning tokens may be less capable than one scoring 75% with 100 tokens, or may be far more capable if it can reach 95% with 10,000 tokens. Standard benchmarks that report a single accuracy number without specifying the inference budget are comparing apples to oranges.

Compute-Normalized Comparison¶

The right framework is to compare models at the same token budget or the same wall-clock time. We define:

and plot accuracy vs. token budget \(B\) as a curve. A model that dominates everywhere on this curve is unambiguously better. When curves cross, the comparison is budget-dependent.

Majority voting (self-consistency, Wang et al., 2022): generate \(k\) independent solutions and take the majority answer. Accuracy improves with \(k\), but at the cost of \(k \times\) compute. The marginal return diminishes — this is a compute-accuracy tradeoff curve.

Best-of-N with a verifier: generate \(N\) solutions and return the one scored highest by a reward model or PRM. This is more compute-efficient than majority voting for hard problems because it can identify the single correct solution among \(N\), rather than requiring a majority to be correct.

Tree-of-thought / MCTS with PRM: explore a tree of reasoning steps, pruned by a PRM at each node. This can solve problems that are unsolvable with linear CoT under the same token budget.

import statistics
from typing import Callable


def majority_vote_accuracy(
    generate_fn: Callable[[str, int], list[str]],
    verify_fn: Callable[[str, str], bool],
    problems: list[dict],
    k_values: list[int] = [1, 4, 16, 64],
) -> dict[int, float]:
    """
    Measure accuracy under majority voting for different sample budgets k.

    Args:
        generate_fn(problem, n) -> list of n answer strings
        verify_fn(model_answer, gold_answer) -> bool
        problems: list of {'input': str, 'gold': str}
        k_values: list of k's to evaluate
    """
    max_k = max(k_values)
    per_problem_samples = []

    # Generate max_k samples per problem once; reuse for smaller k
    for prob in problems:
        samples = generate_fn(prob['input'], max_k)
        per_problem_samples.append({
            'gold': prob['gold'],
            'samples': samples,
        })

    results = {}
    for k in k_values:
        correct = 0
        for entry in per_problem_samples:
            # Take first k samples
            answers = entry['samples'][:k]
            gold = entry['gold']

            # Majority vote: pick the most frequent answer
            from collections import Counter
            vote_counts = Counter(answers)
            majority_answer = vote_counts.most_common(1)[0][0]

            if verify_fn(majority_answer, gold):
                correct += 1

        results[k] = correct / len(problems)

    return results


# Example output (illustrative, not real benchmark numbers):
# {1: 0.42, 4: 0.61, 16: 0.74, 64: 0.81}
# The curve should be plotted to compare models at equal k.

Scaling Curves as Eval Outputs¶

Rather than reporting a single number, a test-time-compute-aware eval report should include:

This framing makes the cost-performance tradeoff explicit. For most production use cases, the sweet spot is around \(k = 4 \text{–} 8\) for hard reasoning tasks.

Avoiding Reward Hacking in Evals¶

Reward hacking in training (covered in Reward Hacking, Over-Optimization & Alignment Failures) is about a policy exploiting the reward model rather than solving the real task. In evaluation, the analogous failure is a model (or evaluator) exploiting the benchmark metric rather than solving the real problem. The distinctions are subtle but consequential.

Forms of Eval Reward Hacking¶

1. Test set contamination. The model was trained on data that contains the benchmark questions or their answers. This inflates scores without improving real capability. Mitigation: use held-out test sets with release dates after the model’s training cutoff; use procedurally generated problems.

2. Format exploitation. The answer parser accepts “the answer is 42” but not “42”. A model trained with RLVR learns to prepend boilerplate that matches the parser. Mitigation: normalize before parsing; use multiple parsers; do symbolic equivalence.

3. Spurious correlation shortcuts. Multiple-choice benchmarks may have positional biases (option C is correct more often) or stylistic tells. Models fine-tuned on the benchmark distribution learn these shortcuts. Mitigation: debiased sampling; few-shot calibration; ablate by position permutation.

4. Unit test hacking for coding evals. A model can pass unit tests without solving the problem: hardcode outputs for the specific test inputs. Mitigation: use private test cases not visible at generation time; test behavioral properties, not just I/O pairs.

5. Self-report inflation in agentic evals. A model might output “Task completed successfully” without actually completing the task, if the success detector is a language model rather than an oracle. Mitigation: always use ground-truth oracles (database state, file contents, test suite results) — never ask the model to self-report success.

6. Overfitting to public benchmarks. When organizations tune hyperparameters, prompts, or fine-tuning data to maximize scores on public benchmarks, those benchmarks lose generalizability. Mitigation: treat public benchmarks as the val set; maintain a private eval suite as the test set; rotate benchmarks regularly.

"""
Detecting unit test hacking: check if a submitted solution
hardcodes outputs for the visible test cases.
"""

import ast
import re
from typing import Optional


def detects_hardcoded_solution(code: str, test_inputs: list[str]) -> bool:
    """
    Heuristic check for hardcoded solutions.
    Flags:
      1. If return statements contain only literals (no computation).
      2. If test input values appear as literals in if-conditions.
    """
    try:
        tree = ast.parse(code)
    except SyntaxError:
        return False  # Can't parse, not our problem here

    # Check 1: only-literal returns (no variable references)
    for node in ast.walk(tree):
        if isinstance(node, ast.Return):
            if node.value and isinstance(node.value, ast.Constant):
                # A single constant return in a function is suspicious
                return True

    # Check 2: test input values hardcoded in conditions
    for test_input in test_inputs:
        # Extract numeric literals from the test input
        nums = re.findall(r'\b\d+\b', test_input)
        for num in nums:
            # If this exact number appears in an if-condition return structure
            if_hardcode_pattern = rf'if.*\b{num}\b.*:\s*return'
            if re.search(if_hardcode_pattern, code):
                return True

    return False


# Example of a hardcoded (hacked) solution
hacked_solution = """
def two_sum(nums, target):
    if nums == [2, 7, 11, 15] and target == 9:
        return [0, 1]
    if nums == [3, 2, 4] and target == 6:
        return [1, 2]
    return []
"""

test_inputs = ["two_sum([2, 7, 11, 15], 9)", "two_sum([3, 2, 4], 6)"]
print(detects_hardcoded_solution(hacked_solution, test_inputs))
# True — suspicious pattern detected

The practical defense is to always include private test cases alongside the public ones that models can see. The model might hack the public tests, but private tests exercise general behavior.

How Frontier Reasoning Is Measured in Practice¶

The Measurement Stack for Frontier Labs¶

When Anthropic, Google DeepMind, or OpenAI report reasoning benchmark scores, the evaluation pipeline involves several layers:

Every decision in this stack affects the reported number. Running a reasoning model at temperature 0 without a thinking budget may report a different number than running it with extended thinking enabled — neither is “wrong,” but they’re measuring different things.

Calibration and Human Baselines¶

A benchmark is only interpretable when we have a human baseline. For AIME, the human baseline is well-established: qualification requires scoring in roughly the top few percent of AMC 10/12 takers, and typical qualifier scores sit in the low-to-middle single digits out of 15. For SWE-bench Verified, the human baseline (a skilled software engineer given the same issue and repo, without context on the fix) is high — professional developers routinely resolve such issues, suggesting the benchmark is not at the boundary of human capability but is a meaningful proxy for junior-developer-level coding.

Contamination Detection¶

Contamination detection methods include:

1. N-gram overlap: compute Jaccard similarity between benchmark test cases and training data. High overlap suggests contamination.
2. Perplexity test: a contaminated model will have anomalously low perplexity on benchmark inputs compared to a reference model.
3. Canary insertion: insert synthetic problems into training data that are superficially similar to benchmark problems but with different answers. If the model answers the canary correctly and the benchmark incorrectly, contamination from a specific source is likely.
4. Temporal holdout: use only benchmark problems with release dates after the model’s stated training cutoff. LiveCodeBench and OlympiadBench are designed around this principle.

The Evaluation Loop: Connecting Evals to Training¶

Evals are not just a reporting mechanism — they are an active signal in the development loop. The connection between eval methodology and training integrity is direct:

• RLVR training (covered in RL with Verifiable Rewards (RLVR) & The Reasoning Recipe) uses the same execution-based verifiers as eval — if those verifiers are buggy or gameable, the trained model learns to game them.
• Benchmark saturation: when a model achieves near-ceiling performance on a benchmark, it disappears as a gradient signal. New harder benchmarks must be introduced continuously.
• Eval–train separation: using the same benchmark for training reward and for final reporting violates the train/test split principle. Keep a held-out test set that no RLVR iteration has ever optimized against.

The data flywheel for reasoning models is: hard evals reveal failure modes → new training data targets those modes → improved model → harder evals needed → repeat. This is described in the context of production systems in Data Flywheels & Continuous Improvement.

Further Reading¶

• Chen et al., “Evaluating Large Language Models Trained on Code” (HumanEval), 2021 — introduced Pass@k with the unbiased estimator and the HumanEval benchmark.
• Austin et al., “Program Synthesis with Large Language Models” (MBPP), 2021 — the Mostly Basic Python Problems benchmark and few-shot synthesis evaluation.
• Liu et al., “Is Your Code Generated by ChatGPT Really Correct?” (EvalPlus), 2023 — test-suite augmentation exposing false positives in HumanEval/MBPP; source of HumanEval+ and MBPP+.
• Yao et al., “τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains”, 2024 — multi-turn tool-agent evaluation and the pass^k reliability metric.
• Hendrycks et al., “Measuring Mathematical Problem Solving with the MATH Dataset”, 2021 — the MATH benchmark and analysis of difficulty tiers.
• Cobbe et al., “Training Verifiers to Solve Math Word Problems” (GSM8K), 2021 — grade-school math benchmark and the use of verifier models.
• Wang et al., “Self-Consistency Improves Chain of Thought Reasoning in Language Models”, 2022 — majority voting as a test-time compute strategy.
• Jimenez et al., “SWE-bench: Can Language Models Resolve Real-World GitHub Issues?”, 2023 — the real-repo issue-fixing benchmark and its eval harness.
• Zhou et al., “WebArena: A Realistic Web Environment for Building Autonomous Agents”, 2023 — the web-task agentic eval benchmark.
• Jain et al., “LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code”, 2024 — living benchmark with temporal contamination control.
• Lightman et al., “Let’s Verify Step by Step”, 2023 — process reward models for evaluating and training mathematical reasoning chains.
• OpenAI, “Learning to Reason with LLMs” (o1 technical report), 2024 — frontier reasoning evaluation methodology and compute-scaling eval curves.

Exercises¶

1. (Conceptual.) Two code models are each evaluated by generating a single sample per problem. Model A is correct 30% of the time and Model B 80% of the time on their respective intrinsic pass rates. Explain why a single-sample evaluation “conflates the model’s intrinsic ability and sampling luck,” and describe one deployment scenario where Pass@1 is the metric you actually care about and one where Pass@k with \(k > 1\) is more appropriate.

2. (Quantitative.) You generate \(n = 10\) samples for a problem and exactly \(c = 3\) pass all tests. Using the chapter’s unbiased estimator, compute Pass@2 by hand. Then compute the naive estimator \(1-(1-c/n)^k\) for the same \(k=2\) and state which is larger. (\(\binom{7}{2}=21\), \(\binom{10}{2}=45\).)

3. (Quantitative.) Using the compute-scaling table in the chapter (MATH L5: Pass@k \(= 0.31, 0.52, 0.68, 0.79\) at \(k = 1, 4, 16, 64\); cost/problem \(= \$0.002, \$0.008, \$0.032, \$0.128\)), compute the cost per solved problem at \(k=1\) and \(k=16\), and state the multiplicative factor between them. What does this say about the “sweet spot” argument for \(k \approx 4\text{--}8\)?

4. (Conceptual.) A model scores 90% final-answer accuracy on a set of math problems, but when you run a process reward model (PRM) over its correct-answer traces, many are scored poorly at intermediate steps. Explain, using the chapter’s PRM-vs-ORM distinction, (a) how a trace can reach the right final answer through low-quality steps, and (b) why an outcome reward model (ORM) alone cannot surface this, and © why it is a red flag for reasoning training, not just reporting.

5. (Implementation.) The “Practitioner tip” argues you should always stratify eval results by difficulty tier rather than reporting a single mean. Building directly on the chapter’s pass_at_k and aggregate_pass_at_k, implement stratified_pass_at_k(results, k) where each entry in results is {'n': int, 'c': int, 'tier': str}, returning a dict mapping each tier to its aggregated Pass@k. Demonstrate that a model can tie on overall mean yet differ sharply across tiers.

from collections import defaultdict


def stratified_pass_at_k(results: list[dict], k: int) -> dict[str, float]:
    """
    Aggregate Pass@k separately within each difficulty tier.

    Each entry: {'n': int, 'c': int, 'tier': str}
    Returns: {tier -> aggregated Pass@k}, plus an 'overall' key.
    """
    buckets: dict[str, list[dict]] = defaultdict(list)
    for r in results:
        buckets[r['tier']].append(r)

    report = {
        tier: aggregate_pass_at_k(items, k)
        for tier, items in buckets.items()
    }
    # Reuse the chapter's aggregator across all problems for the mean.
    report['overall'] = aggregate_pass_at_k(results, k)
    return report


# Two models with (nearly) equal overall Pass@1 but different shapes.
model_flat = [
    {'n': 10, 'c': 7, 'tier': 'easy'},
    {'n': 10, 'c': 7, 'tier': 'medium'},
    {'n': 10, 'c': 7, 'tier': 'hard'},
]
model_skewed = [
    {'n': 10, 'c': 10, 'tier': 'easy'},
    {'n': 10, 'c': 10, 'tier': 'medium'},
    {'n': 10, 'c': 1,  'tier': 'hard'},
]

for name, res in [('flat', model_flat), ('skewed', model_skewed)]:
    print(name, {t: round(v, 3)
                 for t, v in stratified_pass_at_k(res, 1).items()})

flat   {'easy': 0.7, 'medium': 0.7, 'hard': 0.7, 'overall': 0.7}
skewed {'easy': 1.0, 'medium': 1.0, 'hard': 0.1, 'overall': 0.7}

6. (Implementation, hardest.) The agentic harness leaves "mean_steps": None and scores trajectories with a binary success_fn, which the “Partial Credit” section calls “too coarse.” (a) Implement a progress_rate(subtask_results) function computing the chapter’s progress-rate metric (fraction of ordered subtasks completed before the first failure). (b) Explain why this metric must be driven by ground-truth subtask oracles rather than by asking the model whether each subtask is done, referencing the relevant form of eval reward hacking.

def progress_rate(subtask_results: list[bool]) -> float:
    """
    Fraction of ordered subtasks completed before the first failure.

    subtask_results[i] is True iff subtask i was verified complete by a
    ground-truth oracle. Order matters: progress stops at the first False.
    """
    if not subtask_results:
        return 0.0
    completed = 0
    for ok in subtask_results:
        if not ok:
            break
        completed += 1
    return completed / len(subtask_results)


# A 20-step task that fails at step 19 (index 18):
results = [True] * 18 + [False, False]
print(f"{progress_rate(results):.2f}")   # 0.90

# Same binary outcome (failed), but collapsed early:
print(f"{progress_rate([True, False] + [False] * 18):.2f}")  # 0.05