5.8 GRPO, RLOO & Critic-Free RL¶

In Policy Gradients & PPO for Language Models we built the canonical RLHF optimizer: a policy network, a frozen reference model for the KL penalty, a reward model, and a separate value network — the critic — trained to predict the expected future reward of a partial generation so we can compute low-variance advantages via Generalized Advantage Estimation (GAE). That critic is expensive. It is a second transformer, often the same size as the policy, with its own optimizer state, its own forward and backward passes, and its own failure modes. On a 7B policy, the critic roughly doubles the trainable-parameter memory footprint and adds a whole training subsystem that can silently diverge.

This chapter is about a simple, radical question that the field answered emphatically in 2024–2025: what if we just delete the critic? It turns out that for the LLM setting — where we generate a complete response, score it once at the end, and care about the whole trajectory — we can estimate the advantage we need using nothing but a small group of sampled responses to the same prompt. The group itself becomes the baseline. No value network, no GAE, no bootstrapping. This idea, in two closely related forms — RLOO (REINFORCE Leave-One-Out) and GRPO (Group Relative Policy Optimization) — is what drove DeepSeek-R1 and most of the open reasoning-model wave. By the end you will be able to derive both, implement GRPO from scratch, reason about its now-famous bugs, and apply the 2025 fixes (Dr. GRPO, token-level loss, clip-higher).

This chapter pairs tightly with RL with Verifiable Rewards (RLVR) & The Reasoning Recipe (where the reward becomes a binary correctness checker) and with Advantage Estimation, KL Control & Stability Tricks (the infrastructure-level view).

Why the critic is optional: the baseline as variance reduction¶

The REINFORCE skeleton¶

Strip RLHF down to its core. The policy \(\pi_\theta\) is an autoregressive language model; a “trajectory” is a sampled response \(o\) to a prompt \(q\). We observe a single scalar reward \(R(q,o)\) at the end (from a reward model, a verifier, or a rule). We want to maximize expected reward,

The REINFORCE / score-function estimator (see Policy Gradients & PPO for Language Models) gives the gradient

This is an unbiased estimator, but its variance is brutal. If every reward is positive (say rewards live in \([0,5]\)), then every sampled response gets pushed up — good ones harder, bad ones less hard, but all in the “more likely” direction. The signal we actually care about — which responses were better than typical — is buried in a large common-mode offset.

Baselines: subtract anything that doesn’t depend on the action¶

The classic fix is a baseline \(b(q)\) subtracted from the reward:

This remains unbiased for any \(b\) that does not depend on the sampled action \(o\), because \(\mathbb{E}_{o}\big[b(q)\,\nabla_\theta \log \pi_\theta(o\mid q)\big] = b(q)\,\nabla_\theta \underbrace{\textstyle\sum_o \pi_\theta(o\mid q)}_{=1} = 0\). The variance, however, changes a lot. The variance-minimizing baseline is (approximately) the expected reward \(\mathbb{E}_o[R(q,o)]\) — exactly what PPO’s value function tries to learn. The quantity \(A = R - b\) is the advantage: how much better this response was than the baseline.

The critic-free insight is that we do not need a learned estimate of \(\mathbb{E}_o[R(q,o)]\). We can get a Monte Carlo estimate of it for free by sampling several responses to the same prompt and averaging their rewards. The group of samples is the baseline. That is the whole trick — everything below is engineering around it.

Why this is especially natural for LLMs¶

In classic RL (robotics, Atari) the reward is dense and per-step, the horizon is long, and bootstrapping a value function is essential to assign credit across thousands of steps. The LLM-with-verifiable-reward setting is different in three ways that make the critic almost vestigial:

1. The reward is terminal. You generate the whole answer, then score it once. There is no intermediate reward to bootstrap from. GAE with \(\lambda\to 1\) collapses toward plain Monte Carlo returns anyway.
2. You can cheaply resample. Unlike a robot, you can draw \(G=8\) or \(16\) completions for the same prompt in one batched generation call. A Monte Carlo baseline is therefore essentially free relative to the cost you already pay for generation.
3. The value function is hard and unstable here. Training a per-token value head on sparse terminal rewards through a 7B transformer is finicky; it is one of the main sources of RLHF instability. Removing it removes a whole class of bugs.

So we trade a learned, low-variance, biased-if-wrong baseline (the critic) for a Monte Carlo, slightly-higher-variance, unbiased baseline (the group). For terminal-reward LLM RL, that trade is almost always a win.

RLOO: REINFORCE with a Leave-One-Out baseline¶

The estimator¶

REINFORCE Leave-One-Out (RLOO; popularized for LLMs by Ahmadian et al., Back to Basics, 2024, building on earlier LOO baselines) sharpens the Monte Carlo baseline with a subtle correctness fix. Suppose for prompt \(q\) we sample \(G\) responses \(o_1,\dots,o_G\) i.i.d. from \(\pi_\theta\) with rewards \(R_1,\dots,R_G\). The naive baseline is the full group mean \(\bar R = \frac1G\sum_j R_j\). But \(\bar R\) includes \(R_i\), so the baseline for sample \(i\) is correlated with \(R_i\) — and a baseline correlated with the action you are evaluating introduces bias. RLOO removes exactly the offending term: the baseline for response \(i\) is the mean of the other \(G-1\) responses:

The second equality is a tidy identity worth knowing: leave-one-out is just the full-group-centered advantage rescaled by \(G/(G-1)\). The RLOO gradient estimate over the group is

This is unbiased (each \(b_i\) is independent of \(o_i\)), low-variance for modest \(G\), and — crucially — uses no value network and no clipping. It is “back to basics”: plain policy gradient with a clever, free baseline. RLOO treats the whole response as a single action; the per-token log-probs are simply summed.

A tiny RLOO implementation¶

import torch
import torch.nn.functional as F

def rloo_loss(logprobs_per_token, response_mask, rewards, group_size):
    """
    Minimal RLOO loss for one batch of prompts, each with `group_size` responses.

    logprobs_per_token : (B, T) log pi_theta(o_t | q, o_<t) for the SAMPLED tokens.
                         B = num_prompts * group_size, laid out so that every
                         contiguous block of `group_size` rows shares a prompt.
    response_mask      : (B, T) 1.0 for response tokens, 0.0 for prompt/padding.
    rewards            : (B,)   scalar terminal reward per response.
    group_size         : G, number of responses per prompt.
    Returns a scalar loss to call .backward() on.
    """
    B, T = logprobs_per_token.shape
    G = group_size
    n_prompts = B // G

    # Sequence log-prob = sum of per-token log-probs over response tokens.
    seq_logprob = (logprobs_per_token * response_mask).sum(dim=-1)      # (B,)

    # Group rewards: (n_prompts, G)
    r = rewards.view(n_prompts, G)
    group_sum = r.sum(dim=1, keepdim=True)                             # (n_prompts, 1)
    # Leave-one-out baseline: (sum of others) / (G - 1)
    loo_baseline = (group_sum - r) / (G - 1)                           # (n_prompts, G)
    advantage = (r - loo_baseline).view(B)                            # (B,)

    # Policy-gradient loss. We MAXIMIZE advantage * logprob, so MINIMIZE its negative.
    # advantage is detached: it is a weight, not something we differentiate through.
    loss = -(advantage.detach() * seq_logprob).mean()
    return loss

Two things to internalize. First, the advantage is detach()-ed: gradients flow only through seq_logprob, never through the reward. Second, RLOO as written is a single-step estimator — it assumes the data was generated by the current \(\pi_\theta\). If you reuse the same rollouts for several gradient steps (which you do for efficiency), the estimator becomes off-policy and you need importance weighting and clipping. That is exactly the bridge to GRPO.

GRPO: Group Relative Policy Optimization¶

GRPO (introduced by Shao et al. in DeepSeekMath, 2024, and made famous by DeepSeek-R1, 2025) is the workhorse of modern reasoning-model training. It keeps PPO’s machinery you actually want — the clipped surrogate objective that makes multi-epoch updates safe — and throws away the part you don’t — the value network. The baseline becomes the standardized group reward.

The advantage¶

For a prompt \(q\), sample a group of \(G\) outputs \(\{o_1,\dots,o_G\}\) from the behavior policy \(\pi_{\theta_{\text{old}}}\) (the weights used to generate). Compute rewards \(\{R_1,\dots,R_G\}\). The GRPO advantage assigned to every token of response \(i\) is the group-standardized reward:

Note what this is: a z-score of the reward within its group. Subtracting the mean is the baseline (variance reduction, exactly as above). Dividing by the std is a normalization that makes the advantage scale-free, so a prompt where rewards happen to be large and a prompt where they are small contribute comparably. Every token in \(o_i\) shares the same scalar \(\hat A_i\) — GRPO does no per-token credit assignment, because with a terminal reward there is nothing to assign per token. (We will see in the “fixes” section that the std-normalization is the most controversial design choice in the whole method.)

The clipped surrogate objective¶

Let \(\pi_\theta\) be the current policy and \(\pi_{\theta_{\text{old}}}\) the policy that generated the rollouts. Define the per-token importance ratio

The GRPO objective (in its original DeepSeekMath form) is the PPO clipped surrogate, averaged over tokens within a response and then over responses, plus a KL penalty to a reference model:

We maximize \(\mathcal J\), equivalently minimize \(-\mathcal J\). Three pieces:

• The clipped term \(\min(r\hat A,\ \operatorname{clip}(r,1-\epsilon,1+\epsilon)\hat A)\) is the PPO trust region (see Policy Gradients & PPO for Language Models). When the new policy has moved too far on a token (ratio outside \([1-\epsilon,1+\epsilon]\)) in a direction that would keep increasing the surrogate, the gradient is clipped off. This is what makes it safe to take several gradient epochs on one batch of rollouts. Typical \(\epsilon \approx 0.2\).
• The KL penalty \(-\beta\,\mathbb{D}_{\text{KL}}[\pi_\theta\|\pi_{\text{ref}}]\) keeps the policy from drifting too far from the SFT model \(\pi_{\text{ref}}\), preserving fluency and preventing reward hacking (Reward Hacking, Over-Optimization & Alignment Failures). Note that GRPO puts the KL as a penalty term in the loss, not folded into the reward as PPO often does.
• The double average — mean over tokens within a response (\(\frac{1}{|o_i|}\sum_t\)), then mean over responses (\(\frac1G\sum_i\)). This per-response length normalization is, as we’ll see, the source of GRPO’s length bias.

The k3 KL estimator¶

GRPO does not compute the exact KL (you would need the full vocabulary distribution from the reference at every position, which is expensive to log). It uses the k3 unbiased, always-positive, low-variance estimator from John Schulman’s well-known note on approximating KL. For a token where \(\rho = \dfrac{\pi_{\text{ref}}(o_t\mid\cdot)}{\pi_\theta(o_t\mid\cdot)}\),

This is computed only on the sampled tokens (so just one log-prob from each of the policy and reference). It is non-negative for all \(\rho>0\) (since \(x-\log x-1\ge 0\)), which is a nice property for a penalty.

def k3_kl(logprob_policy, logprob_ref):
    """
    Schulman's k3 estimator of KL(pi_theta || pi_ref), per token, always >= 0.
    Inputs are log pi(o_t | ...) for the SAMPLED token o_t under each model.
    """
    log_ratio = logprob_ref - logprob_policy        # log(pi_ref / pi_theta)
    ratio = log_ratio.exp()                         # pi_ref / pi_theta
    return ratio - log_ratio - 1.0                  # rho - log rho - 1

GRPO from scratch: a complete, runnable training step¶

Below is a self-contained GRPO trainer for a small Hugging Face causal LM. It is deliberately explicit: rollout, reward, advantage, the clipped loss with KL, and a multi-epoch update over the same rollouts. It runs on a single GPU with a tiny model and a toy verifiable reward. This is the code an interviewer might ask you to sketch on a whiteboard, fleshed out to actually run.

import torch
import torch.nn.functional as F
from transformers import AutoModelForCausalLM, AutoTokenizer

# ---------------------------------------------------------------------------
# 0. Setup. Use a tiny instruct model so this runs on a laptop GPU.
# ---------------------------------------------------------------------------
MODEL = "Qwen/Qwen2.5-0.5B-Instruct"
device = "cuda" if torch.cuda.is_available() else "cpu"
tok = AutoTokenizer.from_pretrained(MODEL)
policy = AutoModelForCausalLM.from_pretrained(MODEL, torch_dtype=torch.bfloat16).to(device)
ref    = AutoModelForCausalLM.from_pretrained(MODEL, torch_dtype=torch.bfloat16).to(device)
ref.eval()
for p in ref.parameters():
    p.requires_grad_(False)
opt = torch.optim.AdamW(policy.parameters(), lr=1e-6)

GROUP_SIZE   = 8       # G: responses sampled per prompt
CLIP_EPS_LOW = 0.2     # lower clip (1 - eps_low)
CLIP_EPS_HIGH= 0.28    # upper clip (1 + eps_high); "clip-higher", see below
KL_BETA      = 0.0     # many R1-style recipes drop KL entirely; set >0 to use it
PPO_EPOCHS   = 2       # gradient epochs over one batch of rollouts
MAX_NEW      = 200

# ---------------------------------------------------------------------------
# 1. A toy verifiable reward: 1.0 if the response contains the right number,
#    plus a small format reward for using <answer>...</answer>. In a real
#    RLVR run this is a math/code/unit-test checker (see chapter 5.9).
# ---------------------------------------------------------------------------
def reward_fn(question, response, gold):
    r = 0.0
    if f"<answer>{gold}</answer>" in response:
        r += 1.0                                   # correctness
    if "<answer>" in response and "</answer>" in response:
        r += 0.2                                   # format bonus
    return r

# ---------------------------------------------------------------------------
# 2. Rollout: sample GROUP_SIZE responses per prompt with the OLD policy.
#    In production this is a vLLM/SGLang server, not model.generate; see
#    chapter 6.2 (the generation-training loop).
# ---------------------------------------------------------------------------
@torch.no_grad()
def rollout(prompts, golds):
    seqs, masks, prompt_lens, rewards = [], [], [], []
    policy.eval()
    for q, gold in zip(prompts, golds):
        chat = [{"role": "user", "content": q}]
        ids = tok.apply_chat_template(chat, add_generation_prompt=True,
                                      return_tensors="pt").to(device)
        plen = ids.shape[1]
        # Sample G completions in one batched call (num_return_sequences=G).
        out = policy.generate(ids, do_sample=True, temperature=1.0, top_p=1.0,
                              max_new_tokens=MAX_NEW, num_return_sequences=GROUP_SIZE,
                              pad_token_id=tok.eos_token_id)
        for g in range(GROUP_SIZE):
            full = out[g]                          # (plen + gen_len,)
            text = tok.decode(full[plen:], skip_special_tokens=True)
            rewards.append(reward_fn(q, text, gold))
            seqs.append(full)
            prompt_lens.append(plen)
    policy.train()
    # Pad to a common length and build a response mask (1 on generated tokens).
    maxlen = max(s.shape[0] for s in seqs)
    B = len(seqs)
    input_ids = torch.full((B, maxlen), tok.eos_token_id, dtype=torch.long, device=device)
    resp_mask = torch.zeros((B, maxlen), dtype=torch.float, device=device)
    for i, (s, plen) in enumerate(zip(seqs, prompt_lens)):
        input_ids[i, :s.shape[0]] = s
        resp_mask[i, plen:s.shape[0]] = 1.0        # 1.0 on generated tokens
        # generate(num_return_sequences=G) RIGHT-PADS early-finishing completions
        # with pad_token_id (== eos here) up to the group's longest sequence.
        # Those trailing pad-eos tokens were NEVER sampled by the policy; leaving
        # them in resp_mask feeds advantage-weighted gradient and KL into
        # positions the policy never chose. Keep exactly ONE eos (the true stop
        # token the policy did emit) and mask everything after it.
        gen = s[plen:]
        eos_hits = (gen == tok.eos_token_id).nonzero(as_tuple=True)[0]
        if eos_hits.numel() > 0:
            first_eos = plen + int(eos_hits[0])
            resp_mask[i, first_eos + 1:] = 0.0     # drop pad-eos after true stop
    rewards = torch.tensor(rewards, dtype=torch.float, device=device)
    return input_ids, resp_mask, rewards

# ---------------------------------------------------------------------------
# 3. Per-token log-prob of the SAMPLED tokens under a given model.
# ---------------------------------------------------------------------------
def token_logprobs(model, input_ids, resp_mask):
    out = model(input_ids).logits[:, :-1, :]       # predict token t+1 from t
    logprobs = F.log_softmax(out.float(), dim=-1)
    targets = input_ids[:, 1:]                     # the actually-sampled next tokens
    tok_lp = logprobs.gather(-1, targets.unsqueeze(-1)).squeeze(-1)  # (B, T-1)
    return tok_lp, resp_mask[:, 1:]                # shift mask to align with targets

# ---------------------------------------------------------------------------
# 4. GRPO advantage: z-score the reward within each group of GROUP_SIZE.
# ---------------------------------------------------------------------------
def grpo_advantage(rewards, group_size, normalize_std=True):
    g = rewards.view(-1, group_size)               # (n_prompts, G)
    adv = g - g.mean(dim=1, keepdim=True)          # subtract group mean (baseline)
    if normalize_std:
        adv = adv / (g.std(dim=1, keepdim=True, correction=0) + 1e-4)  # population std (matches worked example); torch default is SAMPLE std
    return adv.reshape(-1)                          # (B,) one scalar per response

# ---------------------------------------------------------------------------
# 5. The GRPO loss for ONE gradient epoch over a fixed batch of rollouts.
# ---------------------------------------------------------------------------
def grpo_loss(input_ids, resp_mask, advantage, old_lp, ref_lp,
              token_level=True):
    new_lp, mask = token_logprobs(policy, input_ids, resp_mask)   # (B, T-1)
    A = advantage.unsqueeze(1)                     # (B, 1) -> broadcast over tokens

    ratio = (new_lp - old_lp).exp()                # pi_theta / pi_old, per token
    unclipped = ratio * A
    clipped   = torch.clamp(ratio, 1 - CLIP_EPS_LOW, 1 + CLIP_EPS_HIGH) * A
    pg = -torch.min(unclipped, clipped)            # negative surrogate (we minimize)

    if KL_BETA > 0:
        kl = k3_kl(new_lp, ref_lp)                 # per-token KL(theta || ref)
        pg = pg + KL_BETA * kl

    if token_level:
        # Dr.GRPO / token-level: sum over ALL tokens, divide by ALL tokens.
        loss = (pg * mask).sum() / mask.sum().clamp(min=1.0)
    else:
        # Original GRPO: mean over tokens WITHIN a response, then mean over responses.
        per_resp = (pg * mask).sum(dim=1) / mask.sum(dim=1).clamp(min=1.0)
        loss = per_resp.mean()
    return loss

# ---------------------------------------------------------------------------
# 6. One full GRPO outer step: rollout -> advantage -> PPO_EPOCHS updates.
# ---------------------------------------------------------------------------
def grpo_step(prompts, golds):
    input_ids, resp_mask, rewards = rollout(prompts, golds)
    advantage = grpo_advantage(rewards, GROUP_SIZE, normalize_std=True)

    # Cache OLD (behavior) and REF log-probs once; they are fixed for this batch.
    with torch.no_grad():
        old_lp, _ = token_logprobs(policy, input_ids, resp_mask)
        ref_lp, _ = token_logprobs(ref,    input_ids, resp_mask) if KL_BETA > 0 \
                    else (torch.zeros_like(old_lp), None)

    for _ in range(PPO_EPOCHS):
        loss = grpo_loss(input_ids, resp_mask, advantage, old_lp, ref_lp)
        opt.zero_grad()
        loss.backward()
        torch.nn.utils.clip_grad_norm_(policy.parameters(), 1.0)
        opt.step()
    return rewards.mean().item(), loss.item()

# Example driver (toy data):
# prompts = ["What is 17 + 26? Put the final number in <answer></answer>."]
# golds   = ["43"]
# for it in range(100):
#     mean_r, l = grpo_step(prompts * 4, golds * 4)
#     print(it, "mean_reward", round(mean_r, 3), "loss", round(l, 4))

A few engineering notes that matter in practice:

• generate pads with pad_token_id, so the response mask must stop at the true EOS. With num_return_sequences=G, completions that finish early are right-padded with pad_token_id (here the EOS id) up to the group’s longest sequence. The rollout loop above therefore masks everything after the first EOS, so the ratio, KL, and advantage-weighted loss are computed only on tokens the policy actually sampled. Forgetting this silently injects advantage-weighted gradient on repeated pad-EOS positions — a classic, hard-to-spot GRPO bug.
• old_lp is recomputed, not reused from generation. In the toy code we recompute log-probs with a forward pass. In production, generation happens on a separate inference engine (vLLM/SGLang) and you must be careful that the log-probs used for the ratio come from a consistent policy. Mismatch between the sampler’s numerics and the trainer’s numerics is a real, subtle source of bias — see The Generation–Training Loop & Rollout Engines.
• KL is often set to zero in R1-style recipes. DeepSeek-R1-Zero used essentially no KL/penalty and let the model drift far from the base, which is desired when you want emergent long reasoning. Keep \(\beta>0\) for chat alignment where you must preserve the SFT persona.
• PPO_EPOCHS > 1 is why we need the ratio and clip at all. If you only ever take one gradient step on each rollout batch (\(\pi_\theta=\pi_{\text{old}}\), ratio \(=1\)), GRPO collapses into RLOO-with-std-normalization. The clip earns its keep only when you reuse rollouts.

The same algorithm in TRL and veRL¶

You will write this loss from scratch once, to understand it; after that you configure it. Every algorithmic choice made above is a flag in the two libraries that dominate open GRPO training — HuggingFace TRL (GRPOTrainer, single-node, the standard entry point; see TRL: HuggingFace’s RL Library) and veRL (multi-node HybridFlow, vLLM/SGLang rollouts + FSDP/Megatron training; see veRL: HybridFlow & The Single-Controller Architecture). The mapping is one-to-one:

So the 2025-fixed GRPO of the next section — Dr. GRPO advantage plus DAPO’s clip-higher — is a handful of flags:

from datasets import Dataset
from trl import GRPOConfig, GRPOTrainer

# TRL wants a "prompt" column; any extra column is forwarded to the reward fns.
train_ds = Dataset.from_dict({
    "prompt": ["What is 17 + 26? Put the final number in <answer></answer>."] * 64,
    "gold":   ["43"] * 64,
})

cfg = GRPOConfig(
    output_dir="grpo-run",
    num_generations=8,              # G
    num_iterations=2,               # gradient epochs per rollout batch
    epsilon=0.2, epsilon_high=0.28, # clip-higher (DAPO)
    loss_type="dr_grpo",            # token-level aggregation, no per-response 1/|o_i|
    scale_rewards=False,            # drop the /std normalization (Dr. GRPO)
    beta=0.0,                       # R1-style: no KL anchor
    learning_rate=1e-6,
    use_vllm=True,                  # generate on vLLM instead of .generate()
)
# reward_funcs are plain Python callables returning one float per completion --
# exactly our reward_fn. Extra dataset columns (here "gold") arrive as kwargs.
# Pass a list and TRL sums them (optionally weighted via `reward_weights`).
def correctness_reward(completions, gold, **kwargs):
    return [1.0 if f"<answer>{g}</answer>" in c else 0.0
            for c, g in zip(completions, gold)]

def format_reward(completions, **kwargs):
    return [0.2 if ("<answer>" in c and "</answer>" in c) else 0.0
            for c in completions]

trainer = GRPOTrainer(model="Qwen/Qwen2.5-0.5B-Instruct",
                      reward_funcs=[correctness_reward, format_reward],
                      args=cfg, train_dataset=train_ds)
trainer.train()

Flag names drift across releases (scale_rewards, for instance, became a string like "none"/"group" in later TRL versions), so pin a version and read GRPOConfig’s docstring or veRL’s ppo_trainer.yaml — but the concepts in the table are stable, and knowing which flag corresponds to which term in the loss is the whole point of having derived it. For an end-to-end run at the scale of this book’s capstone model — where \(G\), the prompt set, and the verifier are all sized to a single GPU — see Post-Training: SFT, DPO, and Narrow RLVR (GRPO) That Works at 100M.

A worked numerical example¶

Let’s make the magnitudes concrete. Take one prompt, a group of \(G=4\) sampled responses, and a verifiable reward in \(\{0, 1\}\) (wrong / right), plus the format bonus from our reward_fn.

The DeepSeek-R1 recipe¶

GRPO is the optimizer; DeepSeek-R1 is the recipe that made it iconic. Two variants matter, and the contrast is instructive.

R1-Zero: pure RL from a base model¶

R1-Zero starts from the base (pretrained, not even SFT’d) DeepSeek-V3 model and applies GRPO directly with a rule-based verifiable reward — no reward model, no human preferences. The reward has two parts:

1. Accuracy reward: for math, did the boxed final answer match the gold? For code, did it pass the unit tests in a sandbox (Reward Engineering, Verifiers & Sandboxes)? This is binary and uncheatable in the usual reward-model sense.
2. Format reward: did the model put its reasoning between <think>...</think> and its answer between <answer>...</answer>? A small shaping reward that teaches the structure, not the content.

   R1-Zero reward = accuracy_reward(answer, gold)        # 0 or 1 (the real signal)
                  + format_reward(uses <think></think>)  # small, e.g. +0.x
   (no reward model, no learned value model, no human labels)

The striking empirical result was that, with nothing but GRPO and this rule reward, the model spontaneously learned to think longer: response length grew over training, and behaviors like self-verification and backtracking (“wait, let me reconsider…”) emerged without ever being demonstrated. The famous “aha moment” — the model catching its own error mid-reasoning — appeared from pure RL pressure on correctness. This is the central evidence that long chain-of-thought reasoning (Reasoning, Chain-of-Thought & Test-Time Compute) can be grown by RL rather than only imitated from data.

R1-Zero had warts: poor readability, language mixing (switching between languages mid-thought), and ugly formatting — precisely because there was no KL anchor to a polished SFT model and the reward only cared about correctness.

R1: the multi-stage pipeline¶

To get a deployable model, DeepSeek-R1 wraps the same GRPO core in a multi-stage pipeline (described qualitatively; we do not cite exact figures):

The takeaways for an engineer: (1) a tiny cold-start SFT dramatically stabilizes RL and improves the final model’s polish; (2) rejection sampling + SFT (“STaR-like” self-distillation) is a cheap, stable way to bank RL gains and broaden coverage; (3) you eventually need a reward model again for the non-verifiable parts (helpfulness, safety) — GRPO is reward-source-agnostic, it just needs some scalar per response. The DeepSeek team also showed that distilling R1’s outputs into smaller dense models often beats running GRPO on those small models directly — a result echoed in Distillation, Model Compression & Knowledge Transfer.

The 2025 fixes: Dr. GRPO, token-level loss, and clip-higher¶

GRPO as originally written has two now-well-documented optimization biases — places where the loss does not faithfully estimate the policy gradient and instead silently rewards or punishes length. The 2025 literature (notably Liu et al., Understanding R1-Zero-like Training / “Dr. GRPO”, and the Qwen team’s DAPO) diagnosed and fixed them. These are favorite interview topics because they require you to actually look at the loss algebra.

Bias 1: the response-level length normalization¶

Recall the inner term \(\frac{1}{|o_i|}\sum_{t=1}^{|o_i|}(\cdot)\). Dividing each response’s summed token loss by its own length \(|o_i|\) means each response contributes equally regardless of length, which sounds fair but isn’t, gradient-wise. Consider two responses with the same positive advantage. The gradient signal per token is scaled by \(1/|o_i|\), so a long correct response gets a smaller per-token push than a short correct one. Conversely, for negative advantage, long wrong responses are penalized less per token than short wrong ones. The net effect of this asymmetry is a systematic pressure that, combined with std-normalization, inflates response length — the model learns that rambling is cheap when wrong and reinforced when right. This is a major driver of the “GRPO models get longer and longer” phenomenon, separate from genuine reasoning gains.

The fix (Dr. GRPO / token-level loss): drop the per-response division and instead sum the loss over all tokens in the batch and divide by a constant (or by the total token count). Every token gets equal weight; length no longer modulates the per-token gradient:

In the code above, this is the token_level=True branch: (pg * mask).sum() / mask.sum(). DAPO frames it as “token-level policy-gradient loss”; Dr. GRPO removes the length term entirely (dividing by a constant max length). Both kill the length bias.

Bias 2: the standard-deviation normalization¶

The other half of “Dr. GRPO” (the name puns on “Done Right” / removing the bias) targets the \(\div\,\operatorname{std}(\{R\})\) in the advantage. Dividing by the group std means prompts with low reward variance get their advantages amplified. A prompt where the group rewards are \(\{1,1,1,0\}\) (std small) produces larger-magnitude advantages than a prompt with \(\{1,1,0,0\}\) (std large), even though the latter is arguably more informative. This re-weights prompts by the inverse of their reward spread — an artifact, not a feature. It particularly distorts very easy and very hard prompts (where std is tiny) and interacts badly with the length bias.

The fix: simply don’t divide by the std. Keep the mean-subtraction (the legitimate baseline) and drop the normalization:

This is exactly the group-mean baseline of RLOO (up to the \(G/(G-1)\) factor). In the code, set normalize_std=False. The empirical claim from the Dr. GRPO work is that removing both biases yields more token-efficient models — they reason just as well with shorter chains, because the optimizer is no longer secretly paying them to be verbose.

Fix 3: clip-higher (decoupled clipping, from DAPO)¶

PPO’s symmetric clip \([1-\epsilon, 1+\epsilon]\) has a subtle pathology for exploration noted by the DAPO authors. The upper clip \(1+\epsilon\) caps how much a low-probability but good token can be boosted in one update, and because the cap is multiplicative it bites hardest exactly where exploration lives. With \(\epsilon=0.2\), a currently-rare token at \(\pi_{\theta_{\text{old}}}=0.01\) can reach at most \(1.2\times 0.01 = 0.012\) in one update — an absolute gain of \(0.002\) — whereas a token already at \(0.9\) has a ceiling of \(1.08\), which is no binding constraint at all. Promising rare tokens therefore climb far too slowly while the policy doubles down on what it already likes, and entropy collapses. DAPO’s fix is clip-higher: decouple the lower and upper clip ranges and raise the ceiling,

For example \(\epsilon_{\text{low}}=0.2,\ \epsilon_{\text{high}}=0.28\). This gives more headroom to promote promising rare tokens (better exploration / sustained entropy) while keeping the lower clip tight to avoid catastrophic suppression. In the code, that is CLIP_EPS_LOW vs CLIP_EPS_HIGH. DAPO pairs this with two more ideas worth knowing:

• Dynamic sampling: discard prompts whose entire group is all-correct or all-wrong (zero advantage, zero gradient) and resample, so every gradient step has informative groups. This directly addresses the “mixed-outcome” requirement from the practitioner tip above.
• Overlong filtering / soft length penalty: mask the loss (or apply a graded penalty) on responses that hit the generation length cap, so truncated garbage doesn’t get treated as a real (and mis-scored) sample.

Fix 4: GSPO — put the importance ratio where the reward is¶

The fixes above leave one structural mismatch untouched, and it is the deepest of the three. The advantage is per-sequence, but the importance ratio is per-token. \(\hat A_i\) is a single scalar for the whole response — that is the granularity at which the reward was actually measured — yet the surrogate re-weights and clips each token independently by \(r_{i,t}\). Those per-token ratios are not an importance weight for anything the estimator is measuring; over a 4k-token chain of thought they are thousands of individually noisy corrections applied to one shared signal, and the noise compounds. The failure is worst for Mixture-of-Experts policies (Mixture-of-Experts (MoE) Architectures), where a small weight update can reroute a token to a different expert and swing that token’s ratio wildly even though the sequence-level likelihood barely moved — which is why early MoE RL runs needed the “routing replay” workaround (cache the old routing so the ratios stay comparable).

GSPO (Group Sequence Policy Optimization, Zheng et al., 2025 — the algorithm behind the Qwen3 models) resolves the mismatch by defining a length-normalized sequence ratio and clipping that:

The \(1/|o_i|\) exponent makes \(s_i\) the geometric mean of the per-token ratios, which is what keeps it from being astronomically large or small for long responses (a raw product of 4000 ratios is numerically hopeless). It also means GSPO’s clip range must be far tighter than PPO’s, because \(s_i\) concentrates very close to \(1\): clip widths on the order of \(10^{-3}\) rather than \(0.2\). In our code it is a two-line change:

# GSPO: ONE ratio per sequence (geometric mean of the per-token ratios).
seq_logratio = ((new_lp - old_lp) * mask).sum(1) / mask.sum(1).clamp(min=1.0)  # (B,)
s = seq_logratio.exp().unsqueeze(1)                    # (B, 1), broadcasts over tokens
pg = -torch.min(s * A, torch.clamp(s, 1 - 3e-4, 1 + 4e-4) * A)   # e.g. tiny eps

Note the consequence for clipping granularity: GSPO discards or keeps an entire response, never individual tokens. That is a blunter instrument, but it is the honest one when the reward itself was sequence-level. TRL exposes it as importance_sampling_level="sequence" on GRPOConfig.

Putting the fixes together¶

The arc is clear: each “fix” removes an artificial scaling from the loss until what remains is, essentially, a clean token-level REINFORCE with a group-mean baseline and a PPO clip for off-policy safety — RLOO’s spirit with PPO’s trust region. GSPO is the one branch that runs the other way: it does not simplify the aggregation, it corrects the unit of the importance weight, which is what starts to matter once the policy is an MoE or the responses run to thousands of tokens. If you remember one thing: the legitimate parts of GRPO are the group-mean baseline and the clipped ratio; the std-normalization and per-response length normalization are the parts that caused trouble.

Length, format, and reward shaping¶

GRPO’s reward can be anything that returns a scalar per response. In practice for reasoning models it is a sum of shaped components, and getting the shaping right is most of the art.

• Correctness (the real signal). Binary or graded, from a verifier: exact-match on a boxed answer, a math equivalence checker (e.g., symbolic comparison so \(\frac12\) and \(0.5\) both count), or unit-test pass-rate for code. This is the only component you truly trust; everything else is a guardrail. Covered in depth in RL with Verifiable Rewards (RLVR) & The Reasoning Recipe.
• Format reward. A small bonus for emitting the required structure (<think>/<answer> tags, a \boxed{}). Keep it small relative to correctness, or the model learns to nail the format on wrong answers — a cheap local optimum. A common pattern: format reward only if the answer is also parseable, so it cannot be farmed independently of attempting the task.
• Length control. Because uncorrected GRPO inflates length, you may add an explicit length term: a mild penalty above a target budget, or DAPO’s overlong-soft-punishment that ramps a penalty as you approach the max-token cap. The cleaner solution, per Dr. GRPO, is to remove the biases that cause length inflation in the first place rather than counter-shaping against them.
• Language consistency. R1 added a reward for keeping the whole response in the target language, trading a small amount of raw accuracy for readability — a deliberate, documented tradeoff.

A subtle but important point: shaped rewards are where reward hacking enters GRPO. The model optimizes the sum, and any component that is easier to satisfy than “be correct” becomes a target. Format bonuses get farmed; length bonuses produce padding; an LLM-judge helpfulness reward gets sycophancy. The mean-subtraction in the advantage does not protect you here — if hacking the format raises every response’s reward, the baseline rises too and you’re fine, but if it raises some responses (the ones that stumble into the exploit), those get positive advantage and the exploit is reinforced. Treat every non-correctness reward component as a liability to be minimized. See Reward Hacking, Over-Optimization & Alignment Failures.

Where GRPO/RLOO sit relative to DPO and PPO¶

It helps to place these methods on a single map (full PPO treatment in Policy Gradients & PPO for Language Models; DPO in Direct Preference Optimization & Its Variants).

The mental model: DPO is the cheapest (offline, no generation) but is limited to pairwise preferences and can’t exploit a programmatic reward. PPO is the most general and lowest-variance but carries the critic. GRPO/RLOO hit the sweet spot for verifiable-reward reasoning RL: online (so the policy explores its own current behavior), critic-free (cheap), and able to consume any scalar reward — which is exactly what a unit-test runner or math checker provides. That is why the 2025–2026 reasoning-model wave runs on GRPO and its descendants rather than PPO or DPO — including the sequence-level variant GSPO derived above, which is GRPO with the importance ratio moved to the granularity of the reward.

Further reading¶

• Shao, Wang, Zhu, et al., DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (2024) — introduces GRPO.
• DeepSeek-AI, DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning (2025) — R1-Zero, the multi-stage R1 recipe, distillation results.
• Ahmadian, Cremer, Gallé, et al., Back to Basics: Revisiting REINFORCE-Style Optimization for Learning from Human Feedback in LLMs (2024) — RLOO for LLMs.
• Liu, Chen, et al., Understanding R1-Zero-Like Training: A Critical Perspective (2025) — the “Dr. GRPO” analysis of GRPO’s length and std biases.
• Yu, et al. (Qwen / ByteDance Seed), DAPO: An Open-Source LLM Reinforcement Learning System at Scale (2025) — token-level loss, clip-higher, dynamic sampling, overlong filtering.
• Zheng, et al. (Qwen), Group Sequence Policy Optimization (2025) — sequence-level importance ratio and clipping; the optimizer behind Qwen3.
• John Schulman, Approximating KL Divergence (blog note) — the k1/k2/k3 estimators used for the GRPO KL term.
• Williams, Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning (1992) — the original REINFORCE.
• HuggingFace TRL (GRPOTrainer, RLOOTrainer) and veRL repositories — production implementations of everything in this chapter; see TRL: HuggingFace’s RL Library and veRL: HybridFlow & The Single-Controller Architecture.

Exercises¶

1. (Conceptual) The chapter claims that subtracting a baseline \(b(q)\) from the reward leaves the policy-gradient estimator unbiased for any \(b\) that does not depend on the sampled action \(o\), yet RLOO goes to the trouble of excluding \(R_i\) from the baseline of sample \(i\). If any action-independent baseline is unbiased, why is the naive full-group mean \(\bar R = \frac1G\sum_j R_j\) a biased baseline for sample \(i\), while the leave-one-out mean \(b_i=\frac1{G-1}\sum_{j\ne i}R_j\) is not? State the exact property the baseline must have and explain which one has it.

2. (Quantitative) For one prompt you sample \(G=4\) responses with rewards \(R=\{1.2,\;0.2,\;1.2,\;0.0\}\). (a) Compute the leave-one-out baseline \(b_i\) and advantage \(A_i=R_i-b_i\) for each response directly from the definition. (b) Verify each \(A_i\) against the identity \(A_i=\frac{G}{G-1}(R_i-\bar R)\). © What is \(\sum_i A_i\), and why does this value hold for RLOO in general?

3. (Quantitative) A prompt yields \(G=4\) binary rewards \(R=\{1,1,0,0\}\). (a) Compute the GRPO advantages \(\hat A_i\) using the population std (the convention pinned by grpo_advantage in the chapter, correction=0), taking \(\varepsilon\to 0\). (b) Recompute using PyTorch’s default sample (Bessel-corrected) std and give the resulting advantages. © Take a token in a correct response (\(\hat A>0\), use your part-(a) value) whose importance ratio after one update is \(r_{i,t}=1.4\), with \(\epsilon_{\text{low}}=0.2,\ \epsilon_{\text{high}}=0.28\). Is its surrogate clipped? (d) A different prompt yields \(R=\{1,1,1,1\}\). What advantage does every token get, and what does this group contribute to the gradient?

4. (Conceptual) GRPO’s original loss has two “optimization biases” that Dr. GRPO removes. (a) Explain how the per-response factor \(\frac1{|o_i|}\) biases the gradient with respect to response length, and state the token-level fix. (b) Explain how dividing the advantage by \(\operatorname{std}(\{R\})\) re-weights prompts, and state the fix. © Separately, DAPO’s clip-higher raises only the upper clip. What failure mode does the symmetric upper clip cause, and why does raising it help?

5. (Implementation) DAPO’s dynamic sampling discards groups whose rewards are all equal (dead groups contribute zero gradient) so every update sees only informative groups. Building on the chapter’s code, (a) write a function dynamic_sampling_mask(rewards, group_size) that returns a per-response boolean mask (True for responses whose group is non-degenerate) and the fraction of non-degenerate groups (the diagnostic the chapter says to monitor). (b) Show how to use it in grpo_loss’s token-level branch so dead-group tokens are excluded from the loss. Keep it consistent with the chapter’s tensor conventions.

import torch

def dynamic_sampling_mask(rewards, group_size, tol=1e-8):
    """
    DAPO dynamic sampling: flag responses in NON-degenerate groups.

    rewards    : (B,) scalar terminal reward per response; every contiguous
                 block of `group_size` rows shares a prompt (chapter layout).
    group_size : G.
    Returns
      keep_resp : (B,) bool, True where the response's group has mixed rewards.
      frac      : float, fraction of groups that are non-degenerate.
    """
    G = group_size
    g = rewards.view(-1, G)                                   # (n_prompts, G)
    # Non-degenerate <=> rewards are not all identical in the group.
    alive = (g.max(dim=1).values - g.min(dim=1).values) > tol  # (n_prompts,) bool
    frac = alive.float().mean().item()
    keep_resp = alive.unsqueeze(1).expand(-1, G).reshape(-1)   # (B,) bool
    return keep_resp, frac

def grpo_loss_dyn(input_ids, resp_mask, advantage, old_lp, ref_lp,
                  rewards, group_size):
    new_lp, mask = token_logprobs(policy, input_ids, resp_mask)  # (B, T-1)
    A = advantage.unsqueeze(1)                                   # (B, 1)

    ratio = (new_lp - old_lp).exp()
    unclipped = ratio * A
    clipped   = torch.clamp(ratio, 1 - CLIP_EPS_LOW, 1 + CLIP_EPS_HIGH) * A
    pg = -torch.min(unclipped, clipped)
    if KL_BETA > 0:
        pg = pg + KL_BETA * k3_kl(new_lp, ref_lp)

    # Zero out tokens belonging to dead (all-equal-reward) groups.
    keep_resp, frac = dynamic_sampling_mask(rewards, group_size)  # (B,), float
    mask = mask * keep_resp.unsqueeze(1).float()                  # (B, T-1)

    loss = (pg * mask).sum() / mask.sum().clamp(min=1.0)          # token-level
    return loss, frac                                            # frac = diagnostic