6.11 Scaling RL: Throughput, Load Balancing & The Latest Tricks¶

By now you have seen the anatomy of an RL-for-LLM system (The Anatomy of an RL-for-LLM System), the generation–training loop (The Generation–Training Loop & Rollout Engines), and the controllers that orchestrate it (veRL: HybridFlow & The Single-Controller Architecture). This chapter is about a brutal, unglamorous fact: most of the wall-clock time in an RL run is spent waiting, and most of that waiting is self-inflicted. A GRPO step that should take 90 seconds takes 240 because seven of your eight rollout workers finished early and sat idle while one chewed through a 16,000-token chain-of-thought. Your H100s, which cost real money, idle at 30% MFU not because the math is hard but because the schedule is bad.

Scaling RL is therefore overwhelmingly a problem of filling bubbles: keeping the generation engines saturated, keeping the gradient device busy, and not throwing away samples that cost you a forward pass to produce. On top of that scheduling substrate sits a second layer — a fast-moving body of algorithmic tricks (DAPO’s dynamic sampling and clip-higher, Dr. GRPO’s bias fixes, VAPO’s value-model stabilization, length penalties and over-long filtering) that change which samples you keep and how much each token contributes to the loss. These two layers interact: dynamic sampling, for instance, is simultaneously a statistics fix and a throughput disaster if you implement it naively.

We will treat them together because in practice you tune them together. We assume you know GRPO and the RLVR recipe (GRPO, RLOO & Critic-Free RL; RL with Verifiable Rewards (RLVR) & The Reasoning Recipe) and continuous batching from the serving side (Continuous Batching & Request Scheduling).

Where the time actually goes: a throughput model¶

Before optimizing anything, write down where the seconds go. A synchronous (on-policy) RL step has four phases that run serially in the simplest design:

For reasoning workloads the generation phase dominates — frequently 70–85% of step time — because you generate long sequences autoregressively (memory-bandwidth bound, one token per forward pass) but train on them in a few large, compute-bound matmuls. So the single highest-leverage question in scaling RL is: what fraction of generation wall-clock is useful work?

Write the whole budget down before you optimize any one term:

The last term is the one people forget. After the optimizer step, the updated parameters must reach the rollout engines — and because the trainer’s layout (FSDP/Megatron shards) differs from the engine’s (inference tensor parallelism), that is a resharding plus broadcast, not a memcpy (Colocated vs Disaggregated RL & Weight Synchronization). A colocated hybrid engine doing in-place NCCL/CUDA-IPC updates keeps \(T_{\text{sync}}\) down to roughly a second for a few-billion-parameter model over NVLink; a naive “write a checkpoint, have vLLM reload it” sync can cost tens of seconds every step and quietly dominate everything else in this chapter. Measure it once before you tune anything else.

Define, for a single generation phase processing a batch of \(N\) prompts each sampled \(G\) times (so \(B = NG\) sequences):

The \(\max_i t_i\) term is the killer. If sequence lengths \(L_i\) are heavy-tailed (and reasoning-trace lengths absolutely are), the longest sequence can be \(5\)–\(10\times\) the median. Every other worker that drains its queue before the tail finishes contributes a bubble. Define generation efficiency as useful token-time over total token-time:

This is just \(\bar{L}/L_{\max}\) — mean over max. If your mean trace is 2,000 tokens and one outlier hits 16,000, then \(\eta_{\text{gen}} \approx 0.125\) for a batch that runs to completion as one unit on one engine. That is an 8× slowdown from one sequence. Everything in the load-balancing section below is an attack on this single ratio.

Generation load imbalance & continuous batching in RL¶

The problem: static batching wastes the tail¶

The naive rollout loop hands each engine a fixed shard of prompts and waits for all of them. Inside a single engine, if it uses static batching — pad every sequence to the longest in the batch, run them lock-step — you pay \(B \cdot L_{\max}\) token-forwards even though only \(\sum_i L_i\) are real. With reasoning traces this is catastrophic.

Continuous batching (a.k.a. in-flight / iteration-level batching), the standard in vLLM and SGLang, fixes the intra-engine part: as soon as one sequence emits its EOS, its slot is freed and the next queued prompt is admitted mid-step. No padding to a global max; the engine processes a rolling population of active sequences sized to fit the KV-cache budget. Within one engine, \(\eta\) recovers to near 1 as long as there is always more work queued.

# Sketch of why continuous batching helps. Two regimes, same 8 prompts.
import numpy as np

lengths = np.array([200, 220, 240, 260, 300, 1800, 320, 280])  # one outlier

# Static batching: every step processes ALL live slots padded to max length.
static_token_forwards = len(lengths) * lengths.max()           # 8 * 1800

# Continuous batching: a slot is freed at its own EOS; total real work only.
# (Assume the engine always has the 8 in flight; freed slots stay idle here
#  because there is nothing else queued -> tail still hurts, but no padding.)
continuous_token_forwards = lengths.sum()

print(static_token_forwards)      # 14400
print(continuous_token_forwards)  # 3620  -> ~4x less compute on the same batch

But note the comment: continuous batching removes padding waste, not tail waste. If those 8 prompts are the entire batch and nothing else is queued, the engine still runs until the 1,800-token outlier finishes, with 7 idle slots. To kill the tail you must either (a) feed more prompts than slots so freed slots get refilled, or (b) stop the tail early, or © overlap the tail with training. We will do all three.

Lever 1: oversubscribe the engines¶

The cleanest fix: make the generation batch larger than the number of concurrent slots and let continuous batching pull from the queue. If an engine holds 64 concurrent sequences but you give it 512 prompts to produce, the 7 freed slots from short sequences immediately admit new prompts, and the engine stays full until the queue drains. Now the only bubble is the final tail — the last few long sequences with no replacements. With a deep queue, \(\eta_{\text{gen}}\) climbs from \(\bar L/L_{\max}\) toward \(\approx 1 - (\text{slots}\cdot L_{\max})/\sum_i L_i\).

In veRL/OpenRLHF this is just: don’t shard prompts 1:1 to slots; pour the whole rollout batch into the inference pool and let the scheduler balance. The practical constraint is the KV-cache budget (PagedAttention & KV-Cache Memory Management): more concurrent long sequences need more cache pages.

Lever 2: length-aware dispatch across engines¶

When you have \(K\) inference engines (data-parallel replicas of the policy), how you partition prompts across them matters. If you round-robin, one engine may by bad luck receive several outliers and become the straggler that gates the whole step. You cannot know lengths in advance, but you can balance adaptively: pull from a shared queue rather than pre-partitioning, so a fast engine that finishes its short sequences immediately grabs the next prompt. A shared global queue is strictly better than static per-engine assignment for heavy-tailed work — it is the classic “single queue, many servers” result from queueing theory.

# Length-balanced greedy dispatch (LPT, "longest processing time first") when
# you DO have a length estimate (e.g., from a cheap length-predictor or from a
# previous epoch's trace for the same prompt). Minimizes makespan = max load.
import heapq

def lpt_dispatch(prompt_len_estimates, num_engines):
    """Assign prompts to engines to minimize the max per-engine total length.
    prompt_len_estimates: list of (prompt_id, estimated_gen_len)."""
    # Sort longest-first so big jobs are placed before small ones can imbalance.
    jobs = sorted(prompt_len_estimates, key=lambda x: -x[1])
    # Min-heap of (current_load, engine_id); always feed the least-loaded engine.
    heap = [(0, e) for e in range(num_engines)]
    heapq.heapify(heap)
    assignment = {e: [] for e in range(num_engines)}
    for pid, est in jobs:
        load, eng = heapq.heappop(heap)
        assignment[eng].append(pid)
        heapq.heappush(heap, (load + est, eng))
    return assignment

# LPT is a 4/3-approximation to optimal makespan -- in practice near-perfect
# balance when you have many small jobs and a few large ones.

In on-policy RL you usually do not have length estimates, so the shared-queue (pull) model is the workhorse and LPT shows up only when you cache per-prompt length statistics across epochs.

Lever 3: overlap generation with training¶

Even a perfectly balanced generation phase leaves the training GPUs idle while it runs, and vice versa. The fix is to pipeline: while the trainer consumes batch \(t\), the rollout engines already generate batch \(t+1\). This is one-step-off-policy (sometimes “staleness-1”) RL — the data the trainer sees was produced by weights one update old. Empirically a staleness of 1–2 is harmless for GRPO-style objectives if you apply the importance-sampling correction (PPO’s ratio handles exactly this; see Policy Gradients & PPO for Language Models). This is the gateway to fully async RL.

Fully async / disaggregated designs push this further — separate the inference and training fleets entirely and let them run continuously, reconciling via periodic weight broadcast. That is the subject of Prime-RL, Async RL & Decentralized Training and Colocated vs Disaggregated RL & Weight Synchronization; here we just note that overlap is the third and largest lever against the bubble, and it is the one that turns a 50%-idle system into a ~90%-utilized one.

Partial rollout: don’t pay for the tail twice¶

Overlap helps the fleet but a single 16k-token sequence still takes a long time to produce, and in a synchronous step it gates everything. Partial rollout (popularized by the Kimi k1.5 report and adopted widely) addresses the tail directly: cap how long any generation phase runs, truncate the unfinished sequences mid-generation, save their KV state and partial text, train on what completed, and resume the truncated sequences in the next rollout phase from where they left off.

The trick is that a partially generated sequence is not wasted — it is a checkpoint. You amortize one very long trajectory across multiple RL steps instead of letting it stall a single step.

The subtlety is off-policyness: a trajectory resumed at step \(k+3\) has a prefix sampled by the weights at step \(k\). The continuation is on newer weights. This is exactly the partial-trajectory staleness async RL must handle, and you correct for it with the per-token importance ratio just as PPO does — each token is reweighted by \(\pi_{\text{new}}(a_t)/\pi_{\text{behavior}}(a_t)\). In practice you store, per resumed token, the log-prob under the behavior policy that produced it, and use that as the denominator. Partial rollout without that correction silently biases the gradient. One extra trap: those stored log-probs come from the inference engine (vLLM/SGLang, possibly FP8, fused kernels), whose numerics differ from the trainer’s, so there is a second, engine-level ratio sitting underneath the policy ratio — bound it with truncated importance sampling and monitor the sampler-vs-trainer log-prob gap, as detailed in Prime-RL, Async RL & Decentralized Training.

from dataclasses import dataclass

GLOBAL_MAX = 16384  # absolute cap on generated tokens across all phases

@dataclass
class PartialTrajectory:
    prompt_ids: list[int]
    generated_ids: list[int]          # tokens emitted so far across phases
    behavior_logprobs: list[float]    # per-token logprob under the policy that
                                      # actually sampled each token (may differ
                                      # across phases -> needed for IS correction)
    kv_handle: object | None          # opaque engine-side KV cache handle, or
                                      # None if KV was dropped and must be re-prefilled
    done: bool

def run_phase(engine, trajs, token_budget):
    """Generate up to `token_budget` new tokens per trajectory, then truncate."""
    for tr in trajs:
        if tr.done:
            continue
        # Resume from saved KV if the engine kept it; else re-prefill the prefix.
        out = engine.generate(
            prefix_ids=tr.prompt_ids + tr.generated_ids,
            kv_handle=tr.kv_handle,
            max_new_tokens=token_budget,
            return_logprobs=True,
        )
        tr.generated_ids.extend(out.token_ids)
        tr.behavior_logprobs.extend(out.logprobs)  # tag with THIS phase's policy
        tr.kv_handle = out.kv_handle
        tr.done = out.hit_eos or len(tr.generated_ids) >= GLOBAL_MAX
    return trajs

KV trade-off. You can either keep the truncated sequence’s KV cache resident on the inference engine between phases (fast resume, but the KV pages sit pinned and reduce the slots available for fresh prompts) or drop it and re-prefill the prefix on resume (frees memory, costs a prefill). Prefix caching (Prefix Caching & KV-Cache Reuse) softens the re-prefill cost. The right choice depends on your KV pressure; under heavy oversubscription, dropping + re-prefill often wins because slots are the scarce resource.

Dynamic sampling & filtering: DAPO and the zero-gradient problem¶

Now we shift from infrastructure throughput to statistical throughput — making each produced sample actually contribute signal. The flagship example is DAPO (Decoupled clip and Dynamic sAmpling Policy Optimization), an open recipe that stacks four ideas on GRPO. Two of them — dynamic sampling and over-long filtering — are squarely about the rollout/throughput layer, so we treat DAPO here.

Why GRPO wastes whole prompts¶

Recall GRPO computes a group-relative advantage by standardizing rewards within the \(G\) samples for one prompt:

Consider a verifiable-reward task where \(r \in \{0, 1\}\). If a prompt is too easy, all \(G\) samples are correct: \(r = (1,1,\dots,1)\), so \(\operatorname{mean}=1\), every \(\hat A_i = 0\). If too hard, all are wrong: again every \(\hat A_i = 0\). A prompt with all-same rewards contributes exactly zero gradient. You paid \(G\) full generations for it and got nothing. As training proceeds and the model masters the easy prompts, the fraction of all-correct groups grows — so the fraction of your rollout budget that is pure waste increases over training. This is the zero-gradient (or “vanishing-advantage”) problem, and it is a throughput problem dressed as a statistics problem.

DAPO’s dynamic sampling: oversample, then filter to a full effective batch¶

DAPO’s fix: keep sampling new prompts and discarding all-same-reward groups until you have a full batch of groups that actually have non-zero advantage variance. Concretely, set a target of \(N\) “useful” prompt-groups per step; generate groups, drop any with \(\operatorname{std}(r)=0\) (all correct or all wrong), and keep going until \(N\) survive.

def dynamic_sampling_step(engine, reward_fn, prompts_iter,
                          target_groups, G, max_oversample_factor=4):
    """Return `target_groups` prompt-groups, each with non-degenerate rewards.
    Oversamples to compensate for filtered-out all-same groups (DAPO-style)."""
    kept = []
    generated_groups = 0
    while len(kept) < target_groups:
        if generated_groups >= max_oversample_factor * target_groups:
            # Safety valve: if too many groups are degenerate, the curriculum
            # is mismatched (model too strong/weak). Stop to avoid runaway cost.
            break
        prompt = next(prompts_iter)
        completions = engine.generate(prompt, n=G)            # G samples
        rewards = [reward_fn(prompt, c) for c in completions]
        generated_groups += 1
        # Degenerate group => every group-relative advantage is 0 => no gradient.
        if min(rewards) == max(rewards):
            continue                                          # discard, don't train
        kept.append((prompt, completions, rewards))
    return kept

The throughput consequence is double-edged. Statistically it is a huge win: every group in the batch now moves the policy. Mechanically it is dangerous — the oversampling factor is data-dependent and grows over training, so a naive synchronous implementation generates a variable, ballooning number of sequences per step and reintroduces tail bubbles plus unpredictable step times. The production fix is to run dynamic sampling on top of a continuous-batching, oversubscribed, async generation pool: keep a deep queue of prompts, generate continuously, filter on the fly, and snapshot a training batch the moment \(N\) useful groups have accumulated — never blocking the generators. Filtering and throughput are thus coupled: dynamic sampling is only cheap if your generation layer is already async and oversubscribed.

Over-long filtering & length control¶

The second DAPO throughput idea concerns truncated generations. If you cap generation at, say, 8k tokens and a sequence hits the cap without emitting EOS, what reward does it get? If you assign it the task reward (often 0 for an unfinished answer), you punish the model for a length-budget artifact and inject noise — the trajectory might have been on track to a correct answer. DAPO’s over-long filtering simply masks the loss on sequences that were truncated by the length cap, so they neither reward nor penalize. A gentler variant is an over-long soft punishment: a small, length-graded penalty that ramps up as the sequence approaches the cap, nudging the model toward concision without the cliff of a hard zero.

def length_shaped_reward(task_reward, gen_len, soft_start, hard_cap, lam=1.0):
    """DAPO-style soft overlong penalty.
    No penalty below soft_start; linearly ramps to -lam at hard_cap.
    Truncated (>= hard_cap, no EOS) sequences are handled by masking elsewhere."""
    if gen_len <= soft_start:
        return task_reward
    if gen_len >= hard_cap:
        # over-long: mask this trajectory's loss instead of training on noise
        return None   # sentinel -> caller drops/masks it
    frac = (gen_len - soft_start) / (hard_cap - soft_start)
    return task_reward - lam * frac

Length control matters beyond throughput: unfiltered RLVR tends to make traces longer and longer (a known reward-coupled length bias), inflating your generation cost every step in a vicious loop — longer traces, slower rollouts, more tail, lower MFU. Explicit length shaping is partly a cost-control mechanism. We return to the length-bias cause in the next section, because Dr. GRPO argues part of it is an artifact of the loss normalization itself.

The latest algorithmic tricks: Dr. GRPO, VAPO, and the clip family¶

DAPO’s other two: clip-higher and token-level loss¶

The two DAPO ingredients we deferred are pure objective changes:

• Clip-Higher (decoupled clipping). PPO/GRPO clip the importance ratio symmetrically to \([1-\varepsilon, 1+\varepsilon]\). DAPO observes this suppresses exploration: for a low-probability but promising token, the upside is capped at \(1+\varepsilon\), so the model can never aggressively up-weight a rare good token, and entropy collapses. DAPO decouples the bounds into \(\varepsilon_{\text{low}}\) and \(\varepsilon_{\text{high}}\) with \(\varepsilon_{\text{high}} > \varepsilon_{\text{low}}\) (e.g. raise the upper clip), giving rare-but-good tokens more room to grow while keeping the downside tight. The clipped surrogate becomes

• Token-level policy-gradient loss. GRPO as originally written averages the loss per sequence first, then over sequences — so every sequence gets equal weight regardless of length, which means tokens in long sequences are down-weighted. DAPO computes the loss as a flat mean over all tokens in the batch, giving each token equal weight. This couples directly to the length-normalization bug that Dr. GRPO formalizes next.

A newer member of the clip family, GSPO (Group Sequence Policy Optimization, the recipe behind the Qwen3 models), moves the importance ratio and clipping from the token level to a length-normalized sequence level. This markedly stabilizes large-scale and especially MoE RL, where compounding per-token ratios can explode and blow up training — it is increasingly the default for MoE reasoning runs.

Dr. GRPO: removing GRPO’s two normalization biases¶

Dr. GRPO (“GRPO Done Right”) makes a sharp, surgical claim: vanilla GRPO contains two normalization terms that introduce optimization bias, and one of them directly causes the runaway-length pathology.

1. Length normalization in the loss. GRPO divides each sequence’s summed token loss by its length \(|o_i|\). Combined with the way advantages are shared across tokens, this makes the per-token gradient magnitude depend on sequence length: for a negative-advantage (wrong) trajectory, dividing by a larger length shrinks the penalty per token — so the model learns that making wrong answers longer reduces the per-token penalty. That is a direct gradient incentive to ramble when wrong. Dr. GRPO removes the length division, using a constant normalizer (or token-level mean) so per-token gradients are length-independent.

2. Standard-deviation normalization in the advantage. Dividing the group-relative advantage by \(\operatorname{std}(\{r_j\})\) over-weights prompts that happen to have low reward variance (a small denominator inflates the advantage). This is a question-level difficulty bias. Dr. GRPO drops the std division, keeping only the mean-subtraction baseline:

import torch

def grpo_advantage(rewards):                       # rewards: (G,)
    mean = rewards.mean()
    std = rewards.std() + 1e-6
    return (rewards - mean) / std                  # vanilla GRPO

def dr_grpo_advantage(rewards):                    # Dr. GRPO: mean-only baseline
    return rewards - rewards.mean()

def sequence_loss_vanilla(token_logp_ratio, adv, mask):
    # per-sequence mean over tokens (divides by length) -> length bias
    per_tok = -(token_logp_ratio * adv.unsqueeze(-1)) * mask
    return (per_tok.sum(-1) / mask.sum(-1).clamp(min=1)).mean()

def token_loss_drgrpo(token_logp_ratio, adv, mask):
    # flat mean over ALL valid tokens (no per-length division) -> length-neutral
    per_tok = -(token_logp_ratio * adv.unsqueeze(-1)) * mask
    return per_tok.sum() / mask.sum().clamp(min=1)

The two fixes interact with DAPO’s token-level loss: both camps converge on “don’t normalize per-sequence length; treat tokens uniformly.” If you take one practical lesson from this section, it is be deliberate about your loss normalizer — it is not a cosmetic constant, it encodes a length policy and a difficulty policy, and getting it wrong is a common silent cause of length blow-up and instability (Reward Hacking, Over-Optimization & Alignment Failures).

VAPO: making the value model work for long CoT¶

GRPO/DAPO/Dr. GRPO are critic-free — they replace the learned value function with a group baseline (Advantage Estimation, KL Control & Stability Tricks). VAPO (Value-model-based Augmented PPO) argues the opposite: a well-tuned value model gives lower-variance, per-token advantages that beat group baselines on long-chain reasoning — if you fix the things that usually make value models fail there. Its main ingredients are engineering hardening for the critic: a length-adaptive GAE \(\lambda\) (the bias–variance knob of generalized advantage estimation must scale with sequence length, because a fixed \(\lambda\) that works at 500 tokens over-smooths at 8,000), value-pretraining/warmup so the critic is not garbage at step 0 when it would otherwise inject huge advantage noise, and DAPO-style clip-higher and token-level losses layered on top. VAPO is the strongest evidence that critic-free is a convenience, not a law: with enough care the value model wins, at the cost of training and hosting a second network (more memory, another forward/backward) — a classic throughput-vs-sample-efficiency trade.

The practical takeaway for scaling: critic-free (GRPO family) is cheaper per step and far simpler to host (no second model on the training GPUs), which is why it dominates open infra; value-based (VAPO/PPO) can be more sample-efficient but costs you memory and a second model to keep in sync. Pick based on whether your bottleneck is GPUs-per-step or samples-to-converge.

Replay, curriculum & sample reuse¶

Two more levers raise sample efficiency — getting more learning per generated token, which is the ultimate throughput because generation is the expensive part.

Replay / sample reuse. On-policy purists generate fresh samples every step and throw them away. But each sample cost a generation; reusing it for \(k\) minibatch passes (PPO epochs) amortizes that cost. The catch is staleness: after one gradient step the data is off-policy, so you must use the clipped importance ratio (which is exactly what PPO/GRPO already provide). A modest replay — 1–4 inner epochs over each rollout batch — is nearly free sample-efficiency. Go too far and the ratios drift outside the clip range, gradients get clipped to zero, and you waste compute on dead samples. A prioritized replay can preferentially revisit high-advantage-magnitude (most informative) trajectories.

Curriculum. Dynamic sampling already implements an implicit curriculum: it discards prompts that are all-correct (mastered) or all-wrong (hopeless), so the model trains on prompts in its zone of proximal development — those it gets right sometimes. You can make this explicit: bin prompts by historical pass-rate, schedule from easy to hard, or up-sample prompts near the 50% success boundary where group-relative advantage variance (hence gradient signal) is maximized. For a Bernoulli group with success probability \(p\), the reward variance is \(p(1-p)\), maximized at \(p=0.5\) — so a prompt the model solves half the time gives the strongest learning signal, and one it never or always solves gives none. Curriculum is just steering the rollout budget toward \(p \approx 0.5\).

def informativeness(pass_rate):
    """Expected per-group reward variance for Bernoulli reward; gradient signal
    peaks at pass_rate = 0.5 and vanishes at 0 or 1 (the DAPO-filtered cases)."""
    return pass_rate * (1.0 - pass_rate)

# Schedule rollout budget proportional to informativeness (a soft curriculum).
def allocate_budget(prompt_pass_rates, total_budget):
    w = {pid: informativeness(pr) + 1e-3 for pid, pr in prompt_pass_rates.items()}
    z = sum(w.values())
    return {pid: int(total_budget * wi / z) for pid, wi in w.items()}

This unifies the chapter’s two halves: the infra trick (dynamic sampling) and the learning trick (curriculum) are the same idea — spend generation on prompts that move the policy — viewed from throughput and from statistics respectively.

A worked throughput example¶

Putting it together: a state-of-the-art recipe¶

The 2024–2026 open-source consensus stack for scaling RL on reasoning models looks roughly like this — note how throughput tricks and algorithmic tricks interleave:

These are flags, not rewrites. Every lever above is a config key in the two frameworks that dominate open RL training: veRL for multi-node runs (veRL: HybridFlow & The Single-Controller Architecture) and HuggingFace TRL for single-node ones (TRL: HuggingFace’s RL Library). The objective knobs (clip-higher, token-level loss, dropping \(/\operatorname{std}\)) are tabulated in GRPO, RLOO & Critic-Free RL; these are the throughput ones:

Key names drift between releases, so pin a version and read veRL’s ppo_trainer.yaml (plus its recipe/dapo/ configs) and TRL’s GRPOConfig docstring — but the concepts, and which number to move when a given metric misbehaves, are stable. Scaled all the way down, most of this collapses: at the capstone’s ~100M-parameter, single-GPU scale the tail bubble is mild (short verifiable answers), so the levers that actually matter are \(G\), degenerate-group filtering, and colocated vLLM generation — see Post-Training: SFT, DPO, and Narrow RLVR (GRPO) That Works at 100M.

The meta-point: throughput in RL is won by overlap and by not wasting samples, not by faster kernels. Kernel-level speedups (FlashAttention, FP8, CUDA graphs — Part IV) matter, but they shave the useful work; the order-of-magnitude wins come from eliminating bubbles (overlap, oversubscription, partial rollout) and from eliminating useless work (dynamic sampling, over-long filtering, curriculum). Get the scheduling right first.

Further reading¶

• Yu et al., DAPO: An Open-Source LLM Reinforcement Learning System at Scale — dynamic sampling, clip-higher, token-level loss, over-long filtering.
• Liu et al., Understanding R1-Zero-Like Training: A Critical Perspective (Dr. GRPO) — the length- and std-normalization bias analysis and fixes.
• Yue et al., VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks — value-model-based long-CoT RL with length-adaptive GAE.
• Kimi Team, Kimi k1.5: Scaling Reinforcement Learning with LLMs — partial rollout and long-context RL infrastructure.
• DeepSeek-AI, DeepSeekMath (GRPO) and DeepSeek-R1 — the group-relative baseline and the RLVR reasoning recipe these tricks build on.
• Schulman et al., Proximal Policy Optimization Algorithms and High-Dimensional Continuous Control Using Generalized Advantage Estimation (GAE) — the clipping and advantage machinery underneath.
• The veRL (HybridFlow) and OpenRLHF repositories — production implementations of oversubscription, continuous batching, and async overlap for RL.

Exercises¶

1. (Conceptual) In GRPO with a verifiable binary reward \(r \in \{0,1\}\), explain why a prompt whose \(G\) samples all receive the same reward (all correct, or all wrong) contributes exactly zero gradient. Then explain why the fraction of your rollout budget wasted this way tends to increase as training proceeds, and name the DAPO mechanism that fixes it.

2. (Quantitative) Take the chapter’s 8-sequence batch with generated lengths $$ L = [200, 220, 240, 260, 300, 1800, 320, 280]. $$ (a) Compute the generation efficiency \(\eta_{\text{gen}} = \bar L / L_{\max}\) for running this batch to completion as one unit. (b) Compute the static-batching token-forwards \(B\cdot L_{\max}\) and the continuous-batching token-forwards \(\sum_i L_i\), and the ratio between them. © In one sentence, say which kind of waste (padding vs tail) the ratio in (b) measures, and which it does not.

3. (Quantitative) Rewards are Bernoulli: for a prompt the model solves with probability \(p\), each of the \(G\) independent samples is correct w.p. \(p\). (a) Using the chapter’s informativeness(p) = p(1-p), compute the per-group reward variance at \(p = 0.1,\ 0.5,\ 0.9\) and state where signal is maximal. (b) A group is degenerate (filtered by dynamic sampling) exactly when all \(G\) samples are correct or all wrong. For \(G=8\), compute the degenerate probability at \(p=0.5\) and at \(p=0.9\), and interpret. © If a fraction \(f=0.35\) of generated groups turn out degenerate, what oversampling factor do you need to still deliver \(N\) useful groups?

4. (Implementation) The chapter’s run_phase stores, per resumed token, the log-prob under the behavior policy in behavior_logprobs. Implement a function is_corrected_surrogate(new_logprobs, behavior_logprobs, advantages, eps_low, eps_high) that computes DAPO’s decoupled clip-higher surrogate loss (a scalar to minimize) for one partial trajectory, using the per-token importance ratio \(\rho_t = \pi_\theta(a_t)/\pi_{\text{behavior}}(a_t)\). Use the chapter’s asymmetric clip bounds \([1-\varepsilon_{\text{low}},\ 1+\varepsilon_{\text{high}}]\) and a flat mean over tokens (token-level loss). Then briefly state why the behavior log-prob — not the current-policy log-prob — must go in the denominator.

import torch

def is_corrected_surrogate(new_logprobs, behavior_logprobs, advantages,
                           eps_low=0.2, eps_high=0.28):
    """Clip-higher, token-level surrogate LOSS for one partial trajectory.
    new_logprobs, behavior_logprobs, advantages: 1-D tensors, per token.
    Returns a scalar to MINIMIZE."""
    # rho_t = pi_theta(a_t) / pi_behavior(a_t), computed in log space.
    ratio = torch.exp(new_logprobs - behavior_logprobs)
    unclipped = ratio * advantages
    clipped = torch.clamp(ratio, 1.0 - eps_low, 1.0 + eps_high) * advantages
    # min() = PPO's pessimistic bound; asymmetric bounds = DAPO clip-higher.
    surrogate = torch.minimum(unclipped, clipped)
    # Flat mean over all tokens (token-level loss, no per-length division).
    return -surrogate.mean()

5. (Conceptual + short calculation) Dr. GRPO claims vanilla GRPO’s per-sequence length normalization gives a direct gradient incentive to make wrong answers longer. Make this concrete: two wrong trajectories share the same negative advantage \(\hat A = -1\), but have lengths \(|o_1| = 100\) and \(|o_2| = 1000\) tokens. Assuming the summed per-token loss magnitude for a trajectory is proportional to \(|\hat A| \cdot |o_i|\) before normalization, compute the per-token penalty each trajectory receives (a) under vanilla GRPO’s divide-by-length normalizer and (b) under Dr. GRPO’s token-level (flat) mean. Explain what the model learns from (a).