6.1 The Anatomy of an RL-for-LLM System¶

In Part V you learned the algorithms of post-training: SFT, reward modeling, PPO, DPO, GRPO, RLVR. Those chapters answered “what loss do we optimize, and why?” This part answers a different, harder question: “what machine actually computes that loss, at scale, without falling over?” That machine — the distributed system that runs reinforcement learning on a large language model — is one of the most demanding pieces of infrastructure in all of ML. It is the only common training workload that must, inside a single training step, run a high-throughput autoregressive inference engine and a memory-hungry training engine, keep their weights in lockstep, and pipe gigabytes of freshly generated tokens between them every few seconds.

This chapter is the map for Part VI. Before we dive into specific frameworks (TRL, veRL, OpenRLHF, Prime-RL) and specific problems (weight sync, colocation, advantage estimation, reward sandboxes), we need a shared mental model: the components of an RL-for-LLM system, how data flows between them, and the structural reasons this is uniquely hard. If you internalize this one chapter, every later chapter becomes “here is a better way to build component X” or “here is how framework Y wires the components together.”

We will assume you already know the math of policy-gradient RL for LLMs from Policy Gradients & PPO for Language Models and GRPO, RLOO & Critic-Free RL. Here we treat those algorithms as a fixed contract and ask how to serve them. We will end this chapter able to draw the full system diagram from memory, account for every GPU in a cluster, and explain to an interviewer exactly why “RL is just SFT with a weird loss” is wrong.

The one-loop problem: why RL infra is its own discipline¶

Start by contrasting RL post-training with the two workloads you already understand.

Pretraining and SFT are one-engine workloads. You have a fixed dataset on disk. Each step: load a batch of tokens, forward, backward, optimizer step. The data never changes in response to the model. The entire system is a single dataflow — a training engine plus a data loader — and the hard problems are parallelism and throughput (Distributed Training I: Data Parallelism, DDP, ZeRO & FSDP) and stability (Training Stability, Loss Spikes & Debugging Large Runs). DPO and offline preference methods inherit this simplicity: the preference pairs are fixed on disk, so DPO is “SFT with a contrastive loss” and runs on a vanilla trainer.

Inference serving is also one-engine. You have a fixed set of weights and a stream of requests. The hard problems are batching, KV-cache memory, and latency (The Anatomy of LLM Inference: Prefill, Decode & The KV Cache). The weights never change.

Online RL is two engines in one loop. The training data does not exist on disk — it is generated by the model being trained, then scored, then consumed to update that same model, which changes what data it generates next. The loop is:

   ┌──────────────────────────────────────────────────────────────────┐
   │                                                                    │
   │   prompts ──► [GENERATE responses]                                 │
   │                    │  (autoregressive decode: an INFERENCE engine) │
   │                    ▼                                               │
   │              [SCORE responses]  ──► rewards                        │
   │                    │  (reward model / verifier / sandbox)          │
   │                    ▼                                               │
   │              [LEARN from (response, reward)]                       │
   │                    │  (forward+backward+optimizer: a TRAIN engine) │
   │                    ▼                                               │
   │              new weights ──────────────► (sync back to generator)  │
   │                    │                                               │
   └────────────────────┘   repeat                                     │

This single fact — generation and training live in the same loop, on the same (or freshly-copied) weights — is the source of essentially every hard problem in Part VI. It forces four things that no other ML workload forces simultaneously:

1. Two fundamentally different compute profiles share a cluster. Generation is memory-bandwidth-bound, latency-sensitive autoregressive decode: tiny matmuls, huge KV cache, terrible arithmetic intensity. Training is compute-bound, throughput-oriented: big fused matmuls, full activation graphs, optimizer state. The same GPUs (or two pools of GPUs) must do both well. We analyze this tension formally with the roofline model in The Roofline Model & Performance Engineering; the practical upshot is that the best inference stack (vLLM/SGLang) and the best training stack (FSDP/Megatron) are different code with incompatible memory layouts, and an RL system must run both.

2. Weights must be synchronized between the two engines, fast, every step. After each optimizer step the generator is stale. Re-syncing tens to hundreds of GB of parameters over the network, possibly reshaping from a training parallelism layout to an inference parallelism layout, is a first-class system problem with its own chapter (Colocated vs Disaggregated RL & Weight Synchronization).

3. The workload is bursty and heterogeneous in time. A step is “generate for 40 seconds, then train for 8 seconds.” Naively, half your expensive training GPUs sit idle during generation and your inference GPUs sit idle during training. Squeezing out that idle time — via colocation, async pipelines, or disaggregation — is most of what “scaling RL” means (Scaling RL: Throughput, Load Balancing & The Latest Tricks).

4. Up to four copies of the model may coexist. Policy (trained), reference (frozen, for KL), reward model (frozen, if learned), and a separate inference replica of the policy. On a 70B model that is potentially 280B+ parameters resident across the cluster before you count optimizer state. Memory accounting is not an afterthought; it is the design.

The six components¶

Every RL-for-LLM system, from a 100-line TRL script to a production cluster, is built from the same six logical components. Frameworks differ in where these run (same process? same GPU? same node? separate clusters?) and who coordinates them (a single Python driver? a Ray controller? a message queue?), but the components are invariant. Learn them once.

Let us take them one at a time. For each, we give the job, the interface (what goes in, what comes out), and the systems characteristics that make it interesting.

1. The actor (policy)¶

The actor, or policy, is the model you are training: the parameters \(\theta\) of the language model \(\pi_\theta\). It is the only component whose weights change. Conceptually it is one thing, but physically it appears in two places at once that you must keep distinct in your head:

• as the generation weights inside the rollout engine (used to sample responses), and
• as the training weights inside the learner (used to compute gradients and apply the optimizer step).

In the colocated design these are the same bytes time-sliced on the same GPUs; in the disaggregated design they are two separate copies on two pools of GPUs that must be synchronized. Either way, the policy is the hub: generation reads it, training writes it, and the gap between “the weights that generated this data” (\(\theta_{\text{old}}\), the behavior policy) and “the weights we are updating now” (\(\theta\)) is what makes the math off-policy and forces importance ratios and clipping (recall the ratio \(r_{i,t}=\pi_\theta/\pi_{\theta_{\text{old}}}\) from GRPO, RLOO & Critic-Free RL).

2. The rollout / generation engine¶

The rollout engine turns prompts into experience: it samples one or more responses per prompt by autoregressive decoding under the current policy. This is the inference half of the loop, and it is almost always the dominant cost of an RL step — typically 60–80% of wall-clock — because long-CoT responses are thousands of decode steps each, and decode is memory-bandwidth-bound.

Its interface: in come prompts (and a sampling config: temperature, top-p, max tokens, group size \(G\)); out come, per response, the token ids, the per-token log-probs under the behavior policy \(\log \pi_{\theta_{\text{old}}}(o_t\mid\cdot)\), and stop/length metadata. Those behavior log-probs are not optional decoration — they are the denominator of the importance ratio and must come from the same forward pass that sampled the tokens, or you introduce subtle bias (more on this below and in The Generation–Training Loop & Rollout Engines).

In a toy script the rollout engine is model.generate(). In a real system it is a dedicated inference server — vLLM (vLLM: Architecture, PagedAttention & Internals) or SGLang (SGLang: RadixAttention & Structured Programs) — because those give you continuous batching (Continuous Batching & Request Scheduling), PagedAttention KV management (PagedAttention & KV-Cache Memory Management), and 10–30× the throughput of naive HuggingFace generation. The price you pay is that this engine has its own copy of the weights in its own memory layout, which is exactly the weight-sync problem.

3. The reward / verifier¶

The reward model assigns a scalar (or vector) score to each response. This is the component with the most variety in the whole system, because “reward” can be almost anything that returns a number:

• A learned reward model (RM): a transformer with a scalar head trained on human preferences, run as a frozen forward pass over each response. This is classic RLHF (The RLHF Pipeline & Reward Modeling). It is a fourth large model on the cluster.
• A rule-based verifier: parse the boxed answer, compare to gold, return 0/1. Cheap, CPU-bound, uncheatable in the usual RM sense. This is the heart of RLVR (RL with Verifiable Rewards (RLVR) & The Reasoning Recipe). In practice you rarely write the parser yourself: HuggingFace’s math-verify handles LaTeX/answer-equivalence checking for math, and environment libraries such as verifiers package prompt sets, parsers and rubrics into reusable, framework-agnostic reward environments.
• A code sandbox: actually execute the generated program against unit tests in an isolated container and return pass-rate. Now your reward path includes process isolation, timeouts, and security — a whole subsystem (Reward Engineering, Verifiers & Sandboxes).
• An LLM-as-judge: another model grading the response (LLM-as-a-Judge & Automated Evaluation).

From the system’s point of view the reward is a function reward(prompt, response) -> float that may be a GPU forward pass, a CPU regex, or a 5-second Docker run. Its latency and placement vary by 4 orders of magnitude across these cases, which is why a good RL framework treats the reward as a pluggable, possibly-remote, possibly-batched, possibly-async stage rather than baking it in.

4. The learner / trainer¶

The learner consumes scored experience and updates \(\theta\). It is a standard distributed training engine — FSDP or Megatron, mixed precision (Mixed Precision, bf16 & FP8 Training), gradient clipping, an AdamW step — with one twist: its loss is the policy-gradient surrogate, not next-token cross-entropy. Per step it:

1. recomputes the current-policy per-token log-probs \(\log\pi_\theta(o_t\mid\cdot)\) with a forward pass (this is the numerator of the ratio),
2. queries the reference model for \(\log\pi_{\text{ref}}(o_t\mid\cdot)\) if a KL term is used,
3. forms the advantage (from rewards + the chosen estimator: GAE with a critic, or group-relative for GRPO/RLOO),
4. computes the clipped surrogate loss, backpropagates, and steps the optimizer.

Crucially, the learner is compute-bound and uses training parallelism (sharded params, sharded optimizer state, activation checkpointing — Memory-Efficient Training: Checkpointing, Offloading & LoRA Math). If the algorithm uses a value network/critic (PPO), the learner also trains that — a second transformer with its own forward/backward — which is a big reason critic-free methods (GRPO/RLOO) took over for reasoning RL.

5. The reference model¶

The reference model \(\pi_{\text{ref}}\) is a frozen copy of the policy (usually the SFT checkpoint you started RL from). Its only job is to provide \(\log\pi_{\text{ref}}(o_t\mid\cdot)\) so the learner can compute the KL-divergence penalty that keeps the policy from drifting too far and reward-hacking (Reward Hacking, Over-Optimization & Alignment Failures). It is forward-only (no gradients, no optimizer state) so it is cheap in compute but still costs a full set of weights in memory. Some recipes — notably R1-Zero — drop the KL term entirely and therefore drop the reference model, saving that memory. When present, it is usually colocated with the learner (it needs the same tokenized batch and the same log-prob machinery) and is a prime candidate for offload-to-CPU or quantization since it is never updated.

6. The experience / replay buffer¶

The experience buffer is the dataset that does not exist on disk. It holds the rollouts produced this iteration: for each response, the token ids, behavior log-probs, reward, computed advantage, prompt/response masks. The learner draws minibatches from it.

In LLM RL the buffer is usually small and short-lived compared to classic deep-RL replay buffers (DQN-style). Most algorithms are “near-on-policy”: you generate a batch, take a handful of gradient epochs on that batch (PPO’s ppo_epochs, GRPO’s reuse), then throw it away and generate fresh, because the data goes stale as \(\theta\) moves away from \(\theta_{\text{old}}\). The buffer is more of a staging area than a long-term memory. The interesting design choices are: how many gradient steps before data is too off-policy to trust (the “staleness budget”); whether to keep a small mix of older data (off-policy RL, async RL — Prime-RL, Async RL & Decentralized Training); and how to lay the buffer out across the cluster so the learner’s data-parallel ranks each get a balanced shard.

These six roles are not merely pedagogical: they are, almost one-for-one, the objects you configure in real frameworks. Reading a strange RL config becomes easy once you can point at which component each key belongs to (names below are for the current major releases — always check against your installed version):

Notice the shape of the table: TRL keeps everything inside one trainer object (multi-controller, one process per GPU running the same script); veRL gives each component its own named config subtree because each is a separately-placed Ray worker pool; OpenRLHF exposes them as CLI flags over Ray actors. Same six nouns, three different placements. We dissect each framework in TRL: HuggingFace’s RL Library, veRL: HybridFlow & The Single-Controller Architecture, and OpenRLHF, NeMo-Aligner & Ray-Based Systems.

The data flow, end to end¶

Components are nouns; the system is the verb that connects them. Here is the canonical synchronous on-policy step, the one every framework implements as its baseline before adding async/colocation tricks. We trace one outer iteration for a GRPO-style critic-free setup (the most common modern case).

Two arrows in this loop are the entire reason Part VI exists, and neither has any analogue in SFT or serving:

• (b)→©→(f): the data is manufactured, scored, and consumed in one pass. There is no dataset.parquet. If generation is slow, the whole step is slow; if the reward sandbox flakes, your training signal is corrupted. The data pipeline is the model.
• (g): the weight-sync arrow. After every update, the inference engine holds stale weights. You must copy the new \(\theta\) from the training layout into the inference layout, possibly across the network, possibly across a parallelism-layout change (FSDP shards ≠ tensor-parallel shards). This arrow is invisible in the math and dominant in the engineering.

Here is a single, runnable, deliberately-minimal driver that makes the six components and the dataflow concrete. It is colocated and synchronous — the simplest real design — and uses HuggingFace generation as a stand-in for a real rollout engine so it runs on one GPU. Every later framework in Part VI is an elaboration of this skeleton.

import torch
import torch.nn.functional as F
from transformers import AutoModelForCausalLM, AutoTokenizer

# ===========================================================================
# A minimal, COLOCATED, SYNCHRONOUS RL-for-LLM loop that exhibits all six
# components and the full dataflow. Toy model + rule reward so it runs on a
# laptop GPU. This is the mental-model reference for the whole of Part VI.
# ===========================================================================
MODEL  = "Qwen/Qwen2.5-0.5B-Instruct"
device = "cuda" if torch.cuda.is_available() else "cpu"
tok    = AutoTokenizer.from_pretrained(MODEL)

# --- COMPONENT 2: the ACTOR/POLICY (trainable θ). In a colocated design the
#     SAME object serves both generation and training (time-sliced). ---------
policy = AutoModelForCausalLM.from_pretrained(MODEL, torch_dtype=torch.bfloat16).to(device)

# --- COMPONENT 6: the REFERENCE model (frozen θ_ref, for the KL term). -------
reference = AutoModelForCausalLM.from_pretrained(MODEL, torch_dtype=torch.bfloat16).to(device)
reference.eval()
for p in reference.parameters():
    p.requires_grad_(False)

# --- COMPONENT 4 belongs to the LEARNER: the optimizer over θ. ---------------
opt = torch.optim.AdamW(policy.parameters(), lr=1e-6)

G            = 8       # group size: responses sampled per prompt
KL_BETA      = 0.02    # KL penalty weight (0.0 to drop the reference entirely)
CLIP_EPS     = 0.2     # PPO/GRPO clip
PPO_EPOCHS   = 2       # gradient epochs reusing the SAME rollouts (off-policy reuse)
MAX_NEW      = 256

# ------------------- COMPONENT 3: the REWARD / VERIFIER ---------------------
def reward_fn(response_text: str, gold: str) -> float:
    """Rule-based verifier: 1.0 if correct, +0.2 format bonus. CPU, no model."""
    r = 0.0
    if f"<answer>{gold}</answer>" in response_text:
        r += 1.0
    if "<answer>" in response_text and "</answer>" in response_text:
        r += 0.2
    return r

# ------------- COMPONENT 1: the ROLLOUT / GENERATION ENGINE -----------------
@torch.no_grad()
def rollout(prompts, golds):
    """Sample G responses per prompt under θ_old. Returns the EXPERIENCE BUFFER
    fields: padded token ids, response mask, behavior log-probs, rewards."""
    policy.eval()
    seqs, plens, rewards = [], [], []
    for q, gold in zip(prompts, golds):
        ids = tok.apply_chat_template([{"role": "user", "content": q}],
                                      add_generation_prompt=True,
                                      return_tensors="pt").to(device)
        plen = ids.shape[1]
        # In production this call is a vLLM/SGLang server, not model.generate.
        out = policy.generate(ids, do_sample=True, temperature=1.0, top_p=1.0,
                              max_new_tokens=MAX_NEW, num_return_sequences=G,
                              pad_token_id=tok.eos_token_id)
        for g in range(G):
            full = out[g]
            text = tok.decode(full[plen:], skip_special_tokens=True)
            rewards.append(reward_fn(text, gold))     # <-- COMPONENT 3 invoked
            seqs.append(full); plens.append(plen)
    policy.train()

    # Pad into a rectangular batch and build the response mask.
    B, maxlen = len(seqs), max(s.shape[0] for s in seqs)
    input_ids = torch.full((B, maxlen), tok.eos_token_id, dtype=torch.long, device=device)
    resp_mask = torch.zeros((B, maxlen), device=device)
    for i, (s, plen) in enumerate(zip(seqs, plens)):
        input_ids[i, :s.shape[0]] = s
        resp_mask[i, plen:s.shape[0]] = 1.0           # 1 on generated tokens only
    rewards = torch.tensor(rewards, device=device)

    # Behavior log-probs logπ_old: in a colocated design we recompute them with
    # the current weights (which == θ_old until the first optimizer step).
    old_lp, mask = token_logprobs(policy, input_ids, resp_mask)
    return input_ids, mask, old_lp.detach(), rewards

def token_logprobs(model, input_ids, resp_mask=None):
    """Per-token logπ of the SAMPLED tokens under `model`, plus the shifted mask.
    Pass resp_mask=None when the caller already holds the shifted mask."""
    logits = model(input_ids).logits[:, :-1, :]                 # predict t+1 from t
    lp = F.log_softmax(logits.float(), dim=-1)
    tgt = input_ids[:, 1:]
    tok_lp = lp.gather(-1, tgt.unsqueeze(-1)).squeeze(-1)        # (B, T-1)
    return tok_lp, (None if resp_mask is None else resp_mask[:, 1:])

# ------------------- ADVANTAGE: group-relative (GRPO) ----------------------
def grpo_advantage(rewards, group_size):
    g = rewards.view(-1, group_size)
    adv = (g - g.mean(dim=1, keepdim=True)) / (g.std(dim=1, keepdim=True) + 1e-4)
    return adv.reshape(-1)                                        # (B,) one per response

# --------------------- COMPONENT 4: the LEARNER ----------------------------
def learner_step(input_ids, mask, old_lp, advantage, ref_lp):
    new_lp, _ = token_logprobs(policy, input_ids)                 # current θ: ratio numerator
    A = advantage.unsqueeze(1)                                    # broadcast over tokens
    ratio   = (new_lp - old_lp).exp()                            # π_θ / π_old
    surr    = -torch.min(ratio * A,
                         torch.clamp(ratio, 1 - CLIP_EPS, 1 + CLIP_EPS) * A)
    if KL_BETA > 0:                                              # COMPONENT 6 used here
        log_r = ref_lp - new_lp
        kl    = log_r.exp() - log_r - 1.0                        # Schulman k3, >= 0
        surr  = surr + KL_BETA * kl
    loss = (surr * mask).sum() / mask.sum().clamp(min=1.0)       # token-level mean
    opt.zero_grad(); loss.backward()
    torch.nn.utils.clip_grad_norm_(policy.parameters(), 1.0)
    opt.step()
    return loss.item()

# --------------------- THE OUTER LOOP (the CONTROLLER) ----------------------
def rl_iteration(prompts, golds):
    # (b) rollout + (c) reward -> experience buffer
    input_ids, mask, old_lp, rewards = rollout(prompts, golds)
    # (d) advantage
    advantage = grpo_advantage(rewards, G)
    # reference log-probs are fixed for this batch; cache once
    with torch.no_grad():
        ref_lp, _ = token_logprobs(reference, input_ids) if KL_BETA > 0 \
                    else (torch.zeros_like(old_lp), None)
    # (f) learn: E epochs over the same rollouts
    for _ in range(PPO_EPOCHS):
        loss = learner_step(input_ids, mask, old_lp, advantage, ref_lp)
    # (g) weight sync is a NO-OP here because generation and training share θ
    #     (colocated). In a disaggregated system this is where you'd push θ
    #     into the vLLM engine. See chapter 6.7.
    return rewards.mean().item(), loss

# prompts = ["What is 17 + 26? Put the final number in <answer></answer>."] * 4
# golds   = ["43"] * 4
# for it in range(200):
#     mean_r, loss = rl_iteration(prompts, golds)
#     print(it, "mean_reward", round(mean_r, 3), "loss", round(loss, 4))

Read that loop until the six components and the seven dataflow stages are obvious. The single most important thing the toy hides is stage (g): here weight sync is free because generation and training share one set of weights on one GPU. That is the colocated, synchronous design — perfect for understanding, and genuinely used for small models. At the ~100M scale of this book’s capstone it is not a simplification at all but the correct production choice: policy + reference + inference replica + optimizer state together fit in a couple of GB, so one GPU holds all four copies and the weight-sync arrow really is a no-op — see Post-Training: SFT, DPO, and Narrow RLVR (GRPO) That Works at 100M, which runs essentially this loop with a real verifier and a real prompt set. Everything in Part VI past this chapter is about what happens to this loop when you can no longer afford that simplicity: when the model is too big to colocate, when generation idle time is too expensive, when the reward is a remote sandbox, when you have 512 GPUs and a Ray cluster.

Three batch sizes, and the staleness they buy¶

Newcomers reading a real RL config are ambushed by the fact that there is no single “batch size.” There are three nested ones, and conflating them is how people accidentally train far more off-policy than they intended.

1. Rollout batch — how many prompts the controller hands to the generation engine per outer iteration (veRL data.train_batch_size, OpenRLHF --rollout_batch_size, TRL generation_batch_size). Multiplied by the group size \(G\) (veRL actor_rollout_ref.rollout.n, OpenRLHF --n_samples_per_prompt, TRL num_generations) it gives the number of responses generated before any weight update. Larger is better for generation throughput (more concurrency for continuous batching) and lowers advantage variance, but stretches the interval between updates.
2. Mini-batch — how much of that rollout batch is consumed per optimizer step (veRL actor_rollout_ref.actor.ppo_mini_batch_size, OpenRLHF --train_batch_size). This is the knob that decides on-policyness. If mini-batch equals rollout batch you take exactly one optimizer step per rollout and the update is fully on-policy (\(r_{i,t}\equiv 1\), clipping never fires). If it is smaller you take several steps on data generated by weights that are already stale by the time you reach the last one.
3. Micro-batch — how much fits in GPU memory at once (veRL ppo_micro_batch_size_per_gpu, OpenRLHF --micro_train_batch_size, TRL per_device_train_batch_size with gradient_accumulation_steps). Micro-batches are gradient-accumulated into one mini-batch, so this is a pure memory knob with no effect on the math — changing it must not change your loss curve, which makes it a good sanity check on an implementation.

Combine these with the number of gradient epochs over the same rollouts (PPO’s ppo_epochs, TRL’s num_iterations, PPO_EPOCHS in the toy code above) and you get an explicit staleness budget: the final gradient step of an iteration is

optimizer steps removed from the policy that produced its data. When \(n_{\text{stale}}=0\) the importance ratio and clip are decoration; when it is 8 or 16 they are the only things holding the run together. That number — not the algorithm’s name — tells you how off-policy your “on-policy” method really is.

Why this is uniquely hard: three structural tensions¶

We can now state precisely why RL infra is its own discipline. Three tensions fall directly out of the one-loop structure. Every framework in Part VI is a different point in the trade-off space these tensions define.

Tension 1: generation and training want opposite things¶

Generation (decode) and training have nearly opposite hardware appetites. Make this quantitative with the roofline lens (The Roofline Model & Performance Engineering):

These columns are so different that the best code for each is mutually incompatible: vLLM lays weights out for paged tensor-parallel inference; FSDP shards them for sharded-optimizer training. You cannot, in general, hand vLLM an FSDP-sharded parameter and have it serve. So an RL system runs both stacks and bridges them — and the bridge is the weight-sync machinery. This is why “just call model.generate() inside your training loop” works for a 0.5B toy and collapses for a 70B model: naive generation is 10–30× too slow, so you need vLLM, which means you need two weight layouts, which means you need sync.

Tension 2: idle time — the loop is a relay race, not a team sport¶

In the synchronous loop, stages run in series: generate, then score, then learn. While the inference engine generates, the training engine’s GPUs (with their fat optimizer state) sit idle. While the learner trains, the inference engine sits idle. If generation is 70% of the step and training 20%, then under a strict disaggregated split your training GPUs are idle 70% of the time and your inference GPUs idle 20–30% of the time. On a cluster costing thousands of USD/hour, that idle time is the headline cost of RL.

There are three escapes, each a later chapter:

• Colocation — put generation and training on the same GPUs, time-sliced, so there is no second idle pool. You pay by stopping training to free memory for the KV cache and back, plus the local weight reshape. This is the default for small/medium models (Colocated vs Disaggregated RL & Weight Synchronization).
• Asynchrony — let generation for step \(k{+}1\) run while training for step \(k\) proceeds, accepting that the data is now slightly off-policy (generated by weights a step or two stale). This trades a little algorithmic correctness for a lot of hardware utilization and is the basis of async RL (Prime-RL, Async RL & Decentralized Training).
• Disaggregation done right — separate inference and training pools but pipeline them and size each pool so neither starves (Scaling RL: Throughput, Load Balancing & The Latest Tricks).

A second, sneakier source of idle time is the long-tail of generation: in a batch of \(N{\times}G\) responses, a few will run to the full max_tokens while most finish early. Synchronous designs must wait for the slowest response before scoring and training. This “straggler” problem motivates continuous batching in the rollout engine and partial/streaming rollout consumption — see The Generation–Training Loop & Rollout Engines.

Tension 3: weight sync — keeping two copies of a moving target consistent¶

After every optimizer step, the policy moves. The inference engine now holds stale weights. To keep the loop on-policy you must propagate the new \(\theta\) to the generator before the next rollout. For a 70B model in bf16 that is ~140 GB of parameters to move, every step. The hard parts:

1. The layout mismatch. Training weights are FSDP-sharded across data-parallel ranks (or tensor/pipeline-sharded under Megatron). Inference weights are tensor-parallel-sharded for vLLM. Syncing means gathering the full parameter (or the right shard) from the training layout and scattering it into the inference layout — a collective-communication problem (Parallel Computing & Collective Communication).
2. The transport. Same node? Use NVLink / CUDA IPC and stay on-GPU. Different nodes? You are on the InfiniBand/RoCE fabric, and a 140 GB broadcast at, say, 100 GB/s is over a second of pure communication per step — often comparable to the training time itself.
3. The freshness/throughput trade-off. Sync every step → maximally on-policy but maximally stalling. Sync every few steps → faster but more off-policy drift, which the importance ratio and clip must absorb (and which, past a point, destabilizes training).

The cleanest production approaches use a collective broadcast directly from training GPUs into the inference engine’s GPU buffers (no CPU round-trip, no disk), overlapped with the next generation where the algorithm tolerates a step of staleness. The whole of Colocated vs Disaggregated RL & Weight Synchronization is devoted to doing this well.

A worked example: sizing a 7B GRPO run¶

Abstract tensions become concrete the moment you try to put a real run on real GPUs. Let us size a synchronous GRPO run for a 7B model and see where the time and memory go. We will use round, illustrative numbers (the point is the method and the magnitudes, not vendor-exact figures).

The example crystallizes the part’s whole thesis: in RL-for-LLM, memory is dominated by the multiplicity of model copies, and time is dominated by generation. Optimize for those two facts and you have understood 80% of RL infrastructure.

The mental model for Part VI¶

You now have the scaffolding for everything that follows. Here is how the rest of Part VI hangs off this chapter, so you can read it as a coherent argument rather than a list of tools:

Three load-bearing ideas to carry forward:

1. The six components are invariant; the architecture is where they run and who coordinates them. When you meet a new framework, your first questions are: where does generation run relative to training (colocated/disaggregated)? Who holds the outer loop (single- vs multi-controller)? Is the loop synchronous or async? How does weight sync happen? Those four questions fully locate any system in the design space.

2. Generation is the cost center. Memory is dominated by model-copy multiplicity; wall-clock is dominated by autoregressive decode. Every serious optimization in Part VI is ultimately about making generation cheaper, hiding it behind training, or reusing the GPUs it leaves idle.

3. The off-policy gap is the price of the loop. The instant you let \(\theta\) (trainer) drift from \(\theta_{\text{old}}\) (generator) — via multi-epoch reuse, async generation, or stale weight sync — you are off-policy, and the importance ratio + clip are what keep you honest. Most stability tricks in Advantage Estimation, KL Control & Stability Tricks are about managing exactly this gap.

Further reading¶

• Ouyang, Wu, Jiang, et al., Training Language Models to Follow Instructions with Human Feedback (InstructGPT, 2022) — the canonical RLHF pipeline that defines the policy/reward/reference component structure.
• Schulman, Wolski, Dhariwal, Radford, Klimov, Proximal Policy Optimization Algorithms (2017) — the clipped surrogate and the actor-critic structure the learner implements.
• Shao, Wang, Zhu, et al., DeepSeekMath (2024) and DeepSeek-AI, DeepSeek-R1 (2025) — the critic-free GRPO loop and the rule-reward verifier that reshaped modern RL infra.
• Sheng, Zhang, Ye, et al., HybridFlow: A Flexible and Efficient RLHF Framework (veRL, 2024) — the single-controller architecture and the placement abstractions that motivate much of this part; see veRL: HybridFlow & The Single-Controller Architecture.
• Hu, et al., OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Framework — a Ray-native reference design for the components in this chapter.
• Kwon, Li, Zhuang, et al., Efficient Memory Management for Large Language Model Serving with PagedAttention (vLLM, 2023) — the inference engine that powers most rollout components; see vLLM: Architecture, PagedAttention & Internals.
• HuggingFace TRL and the veRL, OpenRLHF, and NeMo-Aligner repositories — production implementations of the six components, studied in TRL: HuggingFace’s RL Library through OpenRLHF, NeMo-Aligner & Ray-Based Systems.

Exercises¶

1. (Conceptual.) The chapter’s “four-copies rule” says up to four copies of the model may coexist on the cluster: the trainable policy, its inference replica, the reference, and a learned reward model. For each of the first three, state (a) whether it carries gradients and optimizer state, (b) whether its weights ever change during the run, and © roughly how many bytes-per-parameter of resident state it costs. Then explain why the R1-Zero recipe can delete one of these copies entirely, and which one.

2. (Quantitative.) Redo the chapter’s memory accounting for a 13B policy instead of 7B: KL on (reference resident), critic-free GRPO (no critic), rule-based reward (no reward model). Use bf16 = 2 B/param for weights and gradients, and AdamW = 12 B/param for optimizer state. Ignore activations and KV cache. (a) Give the size of each resident copy and the total. (b) Which single line item is largest? © What is the minimum number of 80 GB GPUs just to hold this static footprint?

3. (Quantitative.) On some cluster a synchronous GRPO step measures: generate \(= 45\) s, reward \(= 2\) s, train \(= 12\) s, weight sync \(= 3\) s. (a) What fraction of the step is generation? (b) In a disaggregated layout (separate training pool), what fraction of the step are the training GPUs idle, if they are busy only during train + sync? © If you go fully async and overlap generation of step \(k{+}1\) with the train+sync of step \(k\), estimate the pipelined per-step wall-clock and the resulting speedup over the synchronous step.

4. (Conceptual.) On a fresh batch, before any optimizer step, the importance ratio \(r = \exp(\log\pi_\theta - \log\pi_{\theta_{\text{old}}})\) should equal exactly \(1.0\) for every token. In a real disaggregated system it often does not – it sits systematically at, say, \(0.9\) or \(1.1\). Explain the mechanism, why it biases the gradient, and give three ways to handle it.

5. (Implementation.) In GRPO a group of \(G\) responses whose rewards are all identical (all solved, or all failed) produces a zero advantage for every token in the group, and therefore no learning signal – pure wasted generation. Modify the chapter’s grpo_advantage into a version that (a) still returns the per-response advantages, and (b) also returns the fraction of groups that were degenerate (zero reward variance), so you can log it as a health/curriculum metric. Show the code and explain, using the surrogate loss, why such groups contribute no gradient.

def grpo_advantage_monitored(rewards, group_size, var_eps=1e-8):
    """Group-relative advantage plus a diagnostic: the fraction of groups
    with no reward variance (all responses scored identically). Those groups
    yield zero advantage -> zero gradient -> wasted rollout compute."""
    g   = rewards.view(-1, group_size)
    std = g.std(dim=1, keepdim=True)
    adv = (g - g.mean(dim=1, keepdim=True)) / (std + 1e-4)   # same as chapter
    degenerate = (std <= var_eps).float()                   # 1.0 per dead group
    frac_degenerate = degenerate.mean().item()
    return adv.reshape(-1), frac_degenerate

advantage, frac_dead = grpo_advantage_monitored(rewards, G)
# ... learner epochs as before ...
print("frac_degenerate_groups", round(frac_dead, 3))