5.7 Direct Preference Optimization & Its Variants¶

In The RLHF Pipeline & Reward Modeling and Policy Gradients & PPO for Language Models we built the classical alignment machine: collect human preference pairs, fit a reward model \(r_\phi\) to them, then run PPO to maximize that reward minus a KL penalty that keeps the policy near a frozen reference. It works — it is how InstructGPT and the first ChatGPT were aligned — but it is a lot of moving parts. You train a reward model, you spin up online generation, you carry a value network (the critic), you tune clip ranges and KL coefficients, and you pray the whole feedback loop stays stable. A single misbehaving component can quietly wreck the run.

In 2023, Direct Preference Optimization (DPO) (Rafailov, Sharma, Mitchell, Manning, Ermon, Finn, Direct Preference Optimization: Your Language Model is Secretly a Reward Model) collapsed that pipeline into a single supervised-style loss. The headline result is almost too clean to believe: the constrained RLHF objective has a closed-form optimal policy, and if you substitute that closed form back into the reward model’s own loss, the reward model disappears. What’s left is a simple classification loss over preference pairs that you train with ordinary backprop — no reward model, no sampling, no critic, no rollouts. DPO turned alignment from an RL problem into a fine-tuning problem.

This chapter derives DPO from first principles, builds the intuition for its implicit reward and the temperature \(\beta\), implements the loss from scratch, and then tours the now-large family of descendants — IPO, KTO, ORPO, SimPO, CPO — each of which fixes a specific weakness. We close with the DPO-vs-PPO debate, which is one of the hottest topics an interviewer can hand you. This is among the most interview-dense chapters in the book; read it with a pen.

From RLHF to a closed-form policy¶

The objective we are actually optimizing¶

The RLHF objective (developed in The RLHF Pipeline & Reward Modeling) is: maximize reward while staying close, in KL divergence, to a reference policy \(\pi_{\text{ref}}\) (usually the SFT model). For a prompt \(x\) and a response \(y\) sampled from policy \(\pi_\theta\):

The KL term is not a nicety — without it the policy collapses onto whatever degenerate output maximizes the reward model, exploiting every crack in \(r\) (see Reward Hacking, Over-Optimization & Alignment Failures). The coefficient \(\beta\) controls how hard we are willing to move away from \(\pi_{\text{ref}}\).

The key move: this has a closed-form solution¶

Here is the pivot the DPO authors exploit. For a fixed reward function \(r\), this KL-regularized objective is solved exactly — we do not need RL at all. Write the objective per prompt \(x\) and treat \(\pi_\theta(\cdot\mid x)\) as a probability distribution we optimize freely (one number per possible response \(y\)). Expanding the KL:

Divide by \(\beta\) and rearrange the bracket: maximizing \(\tfrac{1}{\beta}r - \log\frac{\pi}{\pi_{\text{ref}}}\) is the same as minimizing the KL between \(\pi\) and a distribution proportional to \(\pi_{\text{ref}}(y\mid x)\exp\!\big(\tfrac1\beta r(x,y)\big)\). A KL is minimized (to zero) when the two distributions are equal, so the optimum is:

This is a Gibbs / Boltzmann distribution: tilt the reference policy by the exponentiated reward, then renormalize. It is intuitive — the optimal aligned policy is the reference reweighted toward high reward, with \(\beta\) as a temperature that controls how aggressively. The only problem is \(Z(x)\): it sums over all possible responses, an astronomically large set, so we cannot compute \(\pi^*\) directly. That intractable partition function is exactly why classical RLHF resorts to sampling-based RL (PPO) instead of just writing down the answer.

Inverting the relationship: the implicit reward¶

DPO’s trick is to refuse to compute \(Z(x)\) and instead solve for the reward in terms of the policy. Take the log of the boxed equation and rearrange:

Read that carefully. It says: any reward function induces an optimal policy, and conversely, given the optimal policy we can recover the reward — up to the prompt-dependent constant \(\beta\log Z(x)\). So instead of parameterizing a reward model \(r_\phi\) and then training a policy to match it, we can parameterize the reward implicitly through the policy \(\pi_\theta\) that we ultimately want:

This \(r_\theta\) is the implicit reward (sometimes “DPO reward”). It is just \(\beta\) times the log-ratio of the trained policy to the reference — a quantity we can compute from two forward passes, with no separate reward network at all. The term \(\beta\log Z(x)\) is still intractable, but in a moment it will cancel.

Deriving the DPO loss¶

The Bradley–Terry bridge¶

We need one more ingredient from reward modeling: how preferences relate to rewards. The Bradley–Terry model says the probability that a response \(y_w\) (“winner”) is preferred over \(y_l\) (“loser”) for prompt \(x\) is a logistic function of their reward difference:

A reward model is normally trained by maximizing the likelihood of the observed human preferences under this model — a binary cross-entropy on reward differences. The crucial structural fact: only the reward difference matters, never the absolute reward.

Now substitute the implicit reward into the Bradley–Terry difference. Watch the magic:

The intractable \(\beta\log Z(x)\) depends only on \(x\), so it is identical for the winner and loser and cancels in the difference. This is the whole reason DPO works. We are left with a reward difference expressed purely in policy log-probabilities:

The loss¶

Plug this into the Bradley–Terry negative log-likelihood and we get the DPO loss. For a dataset \(\mathcal{D}\) of preference triples \((x, y_w, y_l)\):

That is it. No reward model, no sampling, no RL. You need exactly four log-probabilities per training example — \(\pi_\theta\) and \(\pi_{\text{ref}}\) on both the chosen and rejected responses — and a single sigmoid/cross-entropy. The reference log-probs are constant, so in practice you precompute them once or run a frozen reference in a second (no-grad) forward pass. Define the convenient shorthand

so \(\mathcal{L}_{\text{DPO}} = -\log\sigma(\Delta)\). The loss is small when \(\Delta\) is large and positive — i.e., when the policy has raised the chosen response’s log-ratio above the rejected one’s, relative to the reference.

The fine print: what the derivation assumes¶

The derivation is exact, but exactness is conditional, and the conditions are where real runs go wrong. (1) Bradley–Terry must hold: preferences are assumed to be generated by a single scalar reward, so they are transitive and consistent — an assumption human annotators routinely violate (IPO, below, is a direct response). (2) The closed-form optimum is over the whole response space, but the loss is only ever evaluated on the pairs in your dataset. Those pairs are usually sampled from some other model, so DPO is off-policy; the equivalence to RLHF degrades exactly as far as the data distribution sits from \(\pi_\theta\)’s own support. This is not a footnote — the Tülu 3 report and much of the 2024–2026 open-recipe literature find that preference-optimization quality is dominated by whether the pairs are on-policy (generated by the very model being tuned), which is also why iterative/online DPO works so well. (3) \(\pi_{\text{ref}}\) should assign non-negligible probability to the chosen responses. If the winners came from a much stronger model, they are near-impossible strings under \(\pi_{\text{ref}}\), the log-ratios start deeply negative, and the easiest way to lower the loss is to crush the rejected response — dragging the chosen down with it (the degeneration failure below). The standard mitigation is the Zephyr recipe: SFT the reference on the chosen responses first, so the preference pairs live inside \(\pi_{\text{ref}}\)’s support. At small scale this effect is severe; Post-Training: SFT, DPO, and Narrow RLVR (GRPO) That Works at 100M shows it dominating a 100M-parameter DPO run and mines on-policy pairs to fix it.

What the gradient does (and why the reference matters)¶

Differentiate \(-\log\sigma(\Delta)\). Using \(\frac{d}{dz}\log\sigma(z)=\sigma(-z)\):

This is beautifully interpretable. The gradient increases the log-probability of the chosen response and decreases that of the rejected one — a contrastive push. The scalar weight \(\sigma(-\Delta)\) is large precisely when the model is wrong about a pair (the implicit reward of the loser exceeds the winner, so \(\Delta<0\) and \(\sigma(-\Delta)\to 1\)) and small when the model already gets the pair right (\(\Delta\gg0\), weight \(\to 0\)). So DPO automatically focuses its gradient on the examples it currently mis-ranks. The reference appears only inside \(\Delta\) (and hence inside the weight), gating how much to push but not the direction of the push.

DPO from scratch¶

Here is the canonical implementation. The only non-obvious part is computing the sequence log-probability — the sum of per-token log-probabilities over the response tokens (masking out the prompt). Everything else is the boxed loss verbatim.

import torch
import torch.nn.functional as F

def sequence_logprob(logits, labels, loss_mask):
    """
    Sum of per-token log p(label_t | context) over response tokens.

    logits    : (B, T, V) raw model logits for positions 0..T-1.
    labels    : (B, T)    token ids; for autoregressive LMs label at position t
                          is the token that should be predicted AT t (already shifted
                          by the caller, see below).
    loss_mask : (B, T)    1.0 for response tokens we score, 0.0 for prompt/padding.
    Returns   : (B,)      summed log-prob of each response.
    """
    logprobs = F.log_softmax(logits, dim=-1)                      # (B, T, V)
    # Gather the log-prob of the actual next token at each position.
    token_logp = torch.gather(
        logprobs, dim=2, index=labels.unsqueeze(-1)
    ).squeeze(-1)                                                 # (B, T)
    return (token_logp * loss_mask).sum(dim=-1)                   # (B,)


def dpo_loss(policy_chosen_logps, policy_rejected_logps,
             ref_chosen_logps,   ref_rejected_logps,
             beta=0.1):
    """
    The DPO loss. Inputs are SEQUENCE log-probs (already summed over tokens),
    one scalar per example, for the policy and frozen reference on both responses.

    Returns
      loss          : scalar to call .backward() on
      chosen_reward : β·(logπθ(yw) − logπref(yw))   — the implicit reward of the winner
      reject_reward : β·(logπθ(yl) − logπref(yl))   — the implicit reward of the loser
    """
    # Log-ratios π_θ / π_ref for chosen and rejected responses.
    chosen_logratio   = policy_chosen_logps   - ref_chosen_logps      # (B,)
    rejected_logratio = policy_rejected_logps - ref_rejected_logps    # (B,)

    # Δ = β · (chosen_logratio − rejected_logratio).
    logits = beta * (chosen_logratio - rejected_logratio)            # (B,)

    # −log σ(Δ) == softplus(−Δ); softplus is numerically stable.
    loss = F.softplus(-logits).mean()

    # Implicit rewards, handy to LOG (detached — diagnostics only).
    chosen_reward = (beta * chosen_logratio).detach()
    reject_reward = (beta * rejected_logratio).detach()
    return loss, chosen_reward, reject_reward

And the training step that wires the reference model in. The reference is frozen, so its forward pass runs under torch.no_grad(); in production you usually precompute ref_*_logps once over the whole dataset and cache them, which removes the reference model from the hot loop entirely and halves memory.

def dpo_training_step(policy, ref_model, batch, beta=0.1):
    """
    batch contains, for chosen and rejected responses:
      *_input_ids (B, T), *_labels (B, T) [shifted], *_loss_mask (B, T).
    """
    def logps(model, ids, labels, mask, grad):
        ctx = torch.enable_grad() if grad else torch.no_grad()
        with ctx:
            logits = model(ids).logits[:, :-1, :]    # predict next token
        return sequence_logprob(logits, labels[:, 1:], mask[:, 1:])

    # Policy: needs gradients. Two forward passes (chosen, rejected).
    pol_chosen = logps(policy, batch["chosen_input_ids"],
                       batch["chosen_labels"], batch["chosen_loss_mask"], grad=True)
    pol_reject = logps(policy, batch["rejected_input_ids"],
                       batch["rejected_labels"], batch["rejected_loss_mask"], grad=True)

    # Reference: frozen, no gradients (or precomputed & cached offline).
    ref_chosen = logps(ref_model, batch["chosen_input_ids"],
                       batch["chosen_labels"], batch["chosen_loss_mask"], grad=False)
    ref_reject = logps(ref_model, batch["rejected_input_ids"],
                       batch["rejected_labels"], batch["rejected_loss_mask"], grad=False)

    loss, r_chosen, r_reject = dpo_loss(pol_chosen, pol_reject,
                                        ref_chosen, ref_reject, beta=beta)

    # The single most useful training metric: preference accuracy =
    # fraction of pairs where implicit reward of chosen > rejected.
    reward_acc = (r_chosen > r_reject).float().mean()
    margin = (r_chosen - r_reject).mean()
    return loss, {"reward_acc": reward_acc.item(), "reward_margin": margin.item()}

The role of \(\beta\)¶

\(\beta\) is the same KL temperature from the original objective, and it controls the strength of the trust region around \(\pi_{\text{ref}}\). In the loss it scales \(\Delta\), so it sets how hard the policy is allowed to push the chosen/rejected log-ratios apart before the sigmoid saturates.

• Small \(\beta\) (e.g. 0.01): weak regularization. The policy can drift far from \(\pi_{\text{ref}}\); you fit preferences hard but risk degeneration, reward over-optimization, and the implicit-reward magnitude exploding.
• Large \(\beta\) (e.g. 0.5): strong regularization. The policy stays close to \(\pi_{\text{ref}}\); safer but may underfit the preferences and barely move.
• Typical: \(\beta \in [0.1, 0.5]\), with 0.1 the de-facto default from the paper and TRL. Tune it like a learning rate; it is the most important DPO hyperparameter after the LR itself.

The variants: fixing what DPO gets wrong¶

DPO is elegant but not perfect. Three weaknesses drive the entire variant zoo: (1) it overfits / pushes margins to infinity on deterministic preferences (no explicit margin cap); (2) it needs paired data \((y_w,y_l)\) for the same prompt, which is expensive; and (3) it needs a frozen reference model, doubling memory and adding a forward pass. Each variant attacks one or more of these.

IPO — capping the margin¶

Identity Preference Optimization (IPO) (Azar et al., A General Theoretical Paradigm to Understand Learning from Human Preferences) diagnoses a subtle DPO flaw. Bradley–Terry assumes preferences come from underlying real-valued rewards, but human preference data is often near-deterministic: for a clearly better \(y_w\), the empirical preference probability is essentially 1. When \(P(y_w\succ y_l)\to 1\), the BT log-likelihood is minimized by driving the reward difference to \(+\infty\) — i.e., DPO keeps pushing \(\Delta\) ever larger with no natural stopping point, and the \(\sigma\) saturates so the KL regularization stops biting. The policy overfits, collapsing \(\pi_\theta(y_l)\to 0\) and drifting arbitrarily far from \(\pi_{\text{ref}}\).

IPO replaces the log-sigmoid with a squared-error loss around a target margin \(\tfrac{1}{2\beta}\), which has a finite optimum:

def ipo_loss(pol_chosen, pol_reject, ref_chosen, ref_reject, beta=0.1):
    # h = (chosen log-ratio) − (rejected log-ratio); note: NO β multiplier inside.
    h = (pol_chosen - ref_chosen) - (pol_reject - ref_reject)
    # Regress h toward the finite target 1/(2β) instead of pushing it to +∞.
    return ((h - 1.0 / (2.0 * beta)) ** 2).mean()

Because the target is finite, IPO cannot push the margin to infinity; here \(\beta\) literally sets the target gap between chosen and rejected log-ratios. IPO is more robust to label noise and to repeated/over-represented pairs, at some cost in peak performance on clean data.

A cheaper fix for the noise half of that problem — one that never leaves the DPO loss — is conservative DPO (cDPO), a.k.a. DPO with label smoothing: assume each preference label is flipped with probability \(\epsilon\) and smooth the binary cross-entropy target accordingly,

Setting the derivative to zero gives \(\sigma(\Delta)=1-\epsilon\), i.e. a finite optimal margin \(\Delta^\star=\log\frac{1-\epsilon}{\epsilon}\) (with \(\epsilon=0.1\), \(\Delta^\star\approx 2.2\)) instead of \(+\infty\) — so a mislabeled pair can no longer drag the margin without bound. In TRL this is one argument: DPOConfig(label_smoothing=0.1). Choose \(\epsilon\) to match your estimated annotator error rate; it should be \(0\) for programmatically verified labels and clearly positive for crowd-sourced ones.

KTO — unpaired, prospect-theory alignment¶

Kahneman–Tversky Optimization (KTO) (Ethayarajh, Xu, Muennighoff, Jurafsky, Kiela, KTO: Model Alignment as Prospect Theory Optimization) removes the pairing requirement entirely. Instead of \((x,y_w,y_l)\) triples, KTO consumes individual examples each labeled simply desirable or undesirable (a thumbs-up/thumbs-down dataset). This is enormously practical: production feedback is almost always unpaired — users rate single completions, they don’t rank pairs.

KTO is grounded in prospect theory: humans are loss-averse (a loss hurts more than an equivalent gain pleases) and evaluate outcomes relative to a reference point. KTO builds a value function \(v\) on the implicit reward \(r_\theta(x,y)=\beta\log\frac{\pi_\theta(y\mid x)}{\pi_{\text{ref}}(y\mid x)}\), measured against a reference point \(z_0\) (a running estimate of the average KL/reward, computed in-batch). Desirable examples should have reward above \(z_0\); undesirable ones below it, with separate weights \(\lambda_D,\lambda_U\) that let you encode loss-aversion and correct for class imbalance:

def kto_loss(policy_logps, ref_logps, labels_desirable, kl_z0,
             beta=0.1, lam_desirable=1.0, lam_undesirable=1.0):
    """
    policy_logps, ref_logps : (B,) sequence log-probs (single, UNPAIRED responses).
    labels_desirable        : (B,) bool — True = thumbs up, False = thumbs down.
    kl_z0                    : scalar reference point z0 on the IMPLICIT-REWARD scale,
                              i.e. beta times the detached running-mean in-batch KL
                              (same units as r = beta * log(pi/pi_ref)). beta is
                              already folded into r and z0, so it is NOT reapplied
                              inside the sigmoid below.
    """
    r = beta * (policy_logps - ref_logps)                 # implicit reward, (B,)
    margin = r - kl_z0
    # Desirable: want margin > 0  → value = σ(margin); loss = λ_D · (1 − value).
    # Undesirable: want margin < 0 → value = σ(−margin); loss = λ_U · (1 − value).
    desirable_loss   = lam_desirable   * (1.0 - torch.sigmoid(margin))
    undesirable_loss = lam_undesirable * (1.0 - torch.sigmoid(-margin))
    loss = torch.where(labels_desirable, desirable_loss, undesirable_loss)
    return loss.mean()

KTO often matches or beats DPO despite using strictly weaker (unpaired) supervision, and it degrades gracefully under extreme class imbalance — making it a strong default when your feedback is a stream of single-completion ratings rather than curated pairs.

ORPO — alignment with no reference model¶

Odds Ratio Preference Optimization (ORPO) (Hong, Lee, Thorne, ORPO: Monolithic Preference Optimization without Reference Model) goes after the reference model itself. DPO needs \(\pi_{\text{ref}}\) both to define the implicit reward and to anchor the KL. ORPO observes that if you fold preference learning directly into SFT, the SFT term anchors the model and a reference becomes unnecessary. ORPO’s loss is the standard SFT negative log-likelihood on the chosen response plus an odds-ratio penalty that pushes the odds of the chosen above the rejected:

The odds ratio (rather than a probability ratio) is the deliberate choice: \(\log\)-odds penalizes the rejected response more gently as it gets unlikely, avoiding the runaway suppression that a raw probability ratio would cause. The win is operational: one model, one stage. You merge SFT and alignment into a single pass — no separate preference-tuning phase, no reference forward pass, roughly half the memory of DPO.

def orpo_loss(pol_chosen_logps, pol_rejected_logps,
              chosen_len, rejected_len, chosen_nll, lam=0.1):
    """
    pol_chosen_logps / pol_rejected_logps : (B,) SEQUENCE log-probs (sum over tokens)
        of chosen/rejected under the policy -- e.g. the output of sequence_logprob.
    chosen_len / rejected_len : (B,) number of scored response tokens in each
        response (the per-example sum of the loss_mask). Used to length-normalize
        the log-probs BEFORE the odds so exp(mean_logp) is a valid per-token
        probability in (0, 1); matches TRL ORPOTrainer's average_log_prob=True.
    chosen_nll : (B,) standard SFT negative log-likelihood on the chosen response.
    """
    # Length-normalize to a per-token MEAN log-prob. Without this, a summed logp
    # is very negative (e.g. -12), exp(-12) ~ 6e-6, log1p(-exp) ~ 0, and the odds
    # term collapses to a plain (mean_chosen - mean_rejected) log-prob difference.
    mean_chosen   = pol_chosen_logps   / chosen_len       # (B,)
    mean_rejected = pol_rejected_logps / rejected_len     # (B,)

    # log-odds(y) = log[ p/(1-p) ] = logp - log(1 - exp(logp))  (log1mexp form).
    def log_odds(logp):
        return logp - torch.log1p(-torch.exp(logp.clamp(max=-1e-6)))

    log_or = log_odds(mean_chosen) - log_odds(mean_rejected)   # (B,)
    or_term = -torch.nn.functional.logsigmoid(log_or)          # push chosen odds up
    return (chosen_nll + lam * or_term).mean()

ORPO is the method of choice when you want a single, simple training stage and cannot afford a reference model in memory — common for from-scratch fine-tunes on modest hardware.

SimPO — reference-free, length-normalized¶

SimPO (Meng, Xia, Chen, SimPO: Simple Preference Optimization with a Reference-Free Reward) keeps DPO’s contrastive log-sigmoid shape but makes two changes. First, drop the reference model: define the reward as the length-normalized average log-probability, \(r(x,y)=\tfrac{\beta}{|y|}\log\pi_\theta(y\mid x)\). Length normalization is the key insight — DPO’s summed log-prob is biased by length (longer sequences have more negative summed log-prob), which SimPO argues is a major cause of DPO’s tendency to inflate response length. Second, add a target reward margin \(\gamma\) that the chosen must beat the rejected by:

def simpo_loss(pol_chosen_logps, pol_rejected_logps,
               chosen_len, rejected_len, beta=2.0, gamma=1.0):
    # Length-normalized (per-token average) log-prob = the SimPO reward / β.
    r_chosen   = beta * (pol_chosen_logps   / chosen_len)
    r_rejected = beta * (pol_rejected_logps / rejected_len)
    # Require the chosen reward to exceed rejected by at least the margin γ.
    return -torch.nn.functional.logsigmoid(r_chosen - r_rejected - gamma).mean()

SimPO removes the reference forward pass and the length bias, often matching or beating DPO with less compute. The cost: \(\gamma\) and \(\beta\) now interact and need tuning, and without the reference anchor you lean entirely on the data and \(\gamma\) to prevent degeneration. (SimPO typically uses a larger \(\beta\), e.g. 2.0–2.5, because the reward is now an average rather than a sum.)

CPO — contrastive, memory-light, from machine translation¶

Contrastive Preference Optimization (CPO) (Xu et al., Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation) was developed for MT but is general. CPO approximates DPO with a uniform reference (equivalently, drops \(\pi_{\text{ref}}\)) — saving the reference memory and forward pass — and adds a behavior-cloning (SFT) regularizer on the chosen response so the policy doesn’t drift while it contrasts:

CPO is essentially “DPO without the reference, plus an explicit SFT term to compensate” — the same intuition as ORPO and CPO converging from different directions: fold the anchor into the loss so you can delete the reference model.

A unified view¶

Almost every method here is a choice of (a) what reward to read off the policy, (b) what loss shape contrasts winner vs. loser, and © whether/how to anchor to a reference.

The practitioner’s decision tree: paired data + can afford a reference → start with DPO (\(\beta=0.1\)). Noisy or near-deterministic labels → IPO or cDPO. Only unpaired ratings → KTO. Want one stage / no reference model / tight memory → ORPO or SimPO. There is no universal winner; results are dataset- and base-model-dependent, and you should treat the choice as a hyperparameter to sweep.

Running the whole family in TRL¶

You now know every loss in the table from the inside, which is exactly what you need to use the production implementation intelligently. HuggingFace TRL is the reference open-source library for all of them, and its DPO family takes preference datasets with prompt / chosen / rejected columns (either plain strings or chat-message lists, which TRL renders with the tokenizer’s chat template):

# pip install "trl" "peft" "datasets" "transformers"
from datasets import load_dataset
from transformers import AutoModelForCausalLM, AutoTokenizer
from trl import DPOConfig, DPOTrainer

MODEL = "Qwen/Qwen2.5-0.5B-Instruct"          # your SFT checkpoint in a real run
policy    = AutoModelForCausalLM.from_pretrained(MODEL, torch_dtype="bfloat16")
tokenizer = AutoTokenizer.from_pretrained(MODEL)
pref_ds   = load_dataset("HuggingFaceH4/ultrafeedback_binarized", split="train_prefs")

cfg = DPOConfig(
    output_dir="dpo-out",
    beta=0.1,                       # the KL temperature β from the derivation
    loss_type="sigmoid",            # "sigmoid" = vanilla DPO; "ipo" = the IPO squared loss
    label_smoothing=0.0,            # >0 → conservative DPO (cDPO) for noisy labels
    precompute_ref_log_probs=True,  # cache log π_ref once, then free the reference model
    learning_rate=5e-7,             # DPO LRs are 10–100× smaller than SFT LRs
    num_train_epochs=1,
    per_device_train_batch_size=2,
    gradient_accumulation_steps=16, # effective batch ≈ 32–128 pairs is typical
    max_length=1024, max_prompt_length=512,
    bf16=True,
)

trainer = DPOTrainer(
    model=policy,          # an AutoModelForCausalLM, optionally PEFT-wrapped
    ref_model=None,        # None + a LoRA policy → TRL uses the adapter-disabled base as π_ref
    args=cfg,
    train_dataset=pref_ds, # columns: prompt, chosen, rejected
    processing_class=tokenizer,
)
trainer.train()

Two implementation details worth knowing, because they are exactly the optimizations our from-scratch code left on the table: TRL runs a concatenated forward — chosen and rejected are packed into one batch so each model sees a single forward per step instead of two — and precompute_ref_log_probs implements the caching trick described above. The rest of the family maps onto sibling trainers: KTO → KTOTrainer / KTOConfig on an unpaired dataset (prompt, completion, label), with desirable_weight / undesirable_weight playing the roles of \(\lambda_D,\lambda_U\); ORPO → ORPOTrainer / ORPOConfig, whose beta is the odds-ratio weight \(\lambda\) (there is no reference model to temper); CPO and SimPO → both live in CPOTrainer, with CPOConfig(loss_type="simpo", simpo_gamma=…) selecting the length-normalized SimPO reward and cpo_alpha controlling the SFT anchor (set it to 0.0 for pure SimPO). Online/iterative DPO — regenerate pairs from the current policy, label them with a reward model or an LLM judge, re-run the loss — is OnlineDPOTrainer. Check the trainer docs for the version you install; TRL’s loss_type menu grows every release. Beyond TRL, the same losses ship in OpenRLHF, Axolotl, LLaMA-Factory, and HuggingFace’s alignment-handbook (the Zephyr recipe); the mechanics you just implemented are identical in all of them. See TRL: HuggingFace’s RL Library for the library end to end.

The DPO-vs-PPO debate¶

This is the single most likely “deep” alignment question in an interview, so let’s make it crisp.

Why DPO is attractive¶

DPO is offline, stable, and cheap. No reward model to train, no online generation, no critic, no reward-hacking feedback loop, no sampling hyperparameters. It is a supervised loss — it trains like SFT. For most teams, that operational simplicity is decisive, and DPO + good preference data gets you most of the way to a well-aligned chat model. Open recipes like Zephyr and many Llama/Mistral fine-tunes used DPO precisely because it is so much easier to get right than PPO.

Why PPO can still win¶

The catch is that DPO is off-policy and offline. It learns only from the fixed preference pairs in your dataset — responses generated by some other policy. It never sees its own current outputs. This has real consequences:

• Distribution shift / out-of-distribution behavior. DPO’s loss only constrains behavior on \((y_w,y_l)\) pairs in the data. The mechanism that lowers \(\pi_\theta(y_l)\) can, as a side effect, raise probability mass on completely unseen responses — including bad ones not represented in the data. Several analyses (notably Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study, Xu et al.) show DPO can find policies that exploit out-of-distribution regions, and that on the hardest benchmarks well-tuned PPO outperforms DPO.
• No exploration. PPO generates fresh samples and scores them with a reward model, so it can discover and reinforce new high-reward behaviors. DPO can only re-rank what’s already in the dataset. For tasks where the best behavior isn’t well-represented in the preference data — competitive coding, math — online RL’s exploration matters.
• A reward model generalizes; a dataset doesn’t. PPO’s reward model can score any generated response, giving dense signal across the whole output space. DPO’s “reward” is implicit and only ever evaluated on the dataset pairs.

The honest synthesis¶

The consensus that solidified across 2024–2026: DPO is the better default — simpler, cheaper, stable, and good enough for most alignment (style, format, safety) — while well-tuned online RL (PPO, and increasingly GRPO) achieves the highest ceiling, especially on hard, verifiable tasks. The gap narrows or vanishes if you make DPO more online: iterative / online DPO regenerates fresh pairs from the current policy (labeled by a reward model or judge) and re-runs DPO each round, recovering much of PPO’s exploration benefit while keeping DPO’s simple loss. This is why the frontier moved toward online and verifiable-reward RL (GRPO, RLOO & Critic-Free RL, RL with Verifiable Rewards (RLVR) & The Reasoning Recipe) rather than abandoning either family.

Where this sits in the stack¶

DPO and its variants are the offline preference-tuning layer of post-training. They consume the preference data and reward-modeling concepts from The RLHF Pipeline & Reward Modeling, they are the lightweight alternative to Policy Gradients & PPO for Language Models, and they sit upstream of the online critic-free methods in GRPO, RLOO & Critic-Free RL. In practice DPO almost always runs after Supervised Fine-Tuning & Instruction Tuning (the SFT model is your \(\pi_{\text{ref}}\)), is frequently combined with PEFT I: LoRA, QLoRA, DoRA & The Adapter Family to fit on a single GPU, and shares its reward-hacking failure modes with everything in Reward Hacking, Over-Optimization & Alignment Failures. Production implementations of every loss in this chapter live in TRL: HuggingFace’s RL Library. To see this chapter’s loss actually trained end to end — on-policy pairs mined from the model itself, cached reference log-probs, length-normalized variant, and the honest verdict on what DPO can and cannot buy at that scale — build it in Post-Training: SFT, DPO, and Narrow RLVR (GRPO) That Works at 100M.

Further reading¶

• Rafailov, Sharma, Mitchell, Manning, Ermon, Finn, Direct Preference Optimization: Your Language Model is Secretly a Reward Model (2023) — the original DPO derivation and experiments.
• Azar, Guo, Piot, Munos, et al., A General Theoretical Paradigm to Understand Learning from Human Preferences (2023) — the \(\Psi\)PO framework and IPO.
• Ethayarajh, Xu, Muennighoff, Jurafsky, Kiela, KTO: Model Alignment as Prospect Theory Optimization (2024) — unpaired, prospect-theory alignment.
• Hong, Lee, Thorne, ORPO: Monolithic Preference Optimization without Reference Model (2024) — odds-ratio, single-stage, reference-free.
• Meng, Xia, Chen, SimPO: Simple Preference Optimization with a Reference-Free Reward (2024) — length-normalized reward with a target margin.
• Xu, Sharaf, Chen, et al., Contrastive Preference Optimization (CPO, 2024) — reference-free contrastive with an SFT anchor (developed for machine translation).
• Xu, Fu, Gao, et al., Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study (2024) — the empirical core of the DPO-vs-PPO debate.
• Bradley & Terry, Rank Analysis of Incomplete Block Designs (1952) — the preference model underlying all of the above.
• HuggingFace TRL (DPOTrainer, KTOTrainer, ORPOTrainer, CPOTrainer) — production implementations; see TRL: HuggingFace’s RL Library.

Exercises¶

1. (Conceptual) DPO never trains, evaluates, or even stores a reward model, yet the chapter insists it is implicitly optimizing the same Bradley–Terry objective a reward model would. Explain precisely (a) what the “implicit reward” \(r_\theta(x,y)\) is in terms of the policy and reference, (b) why the intractable partition term \(\beta\log Z(x)\) never has to be computed, and © what would break if the loss compared absolute implicit rewards \(r_\theta(x,y)\) instead of the difference \(r_\theta(x,y_w)-r_\theta(x,y_l)\).

2. (Quantitative) Take \(\beta=0.1\) and one preference pair. The policy assigns summed log-probs \(\log\pi_\theta(y_w)=-8.0\) and \(\log\pi_\theta(y_l)=-7.0\); the reference assigns \(\log\pi_{\text{ref}}(y_w)=-9.0\) and \(\log\pi_{\text{ref}}(y_l)=-7.2\). Compute (a) the two log-ratios, (b) the implicit rewards \(r_w,r_l\) and whether the model currently ranks the pair correctly, © the margin \(\Delta\) and the DPO loss \(-\log\sigma(\Delta)\) in nats, and (d) the gradient weight \(\sigma(-\Delta)\). Is the loss above or below the coin-flip value \(\log 2\)?

3. (Conceptual) During a DPO run your reward_acc climbs steadily past 0.7, but generations get shorter and more repetitive, and you notice the absolute chosen log-prob \(\log\pi_\theta(y_w)\) is falling throughout training. (a) Explain how reward accuracy can improve while the chosen response’s own probability drops. (b) Where is the displaced probability mass going, and why does the loss not punish it? © Give two fixes drawn from this chapter’s variants and say what each one changes.

4. (Quantitative) SimPO argues DPO’s summed log-prob reward is length-biased. Consider two responses to the same prompt that are equally fluent per token — both have an average log-prob of \(-1.0\) per token — but response \(A\) has length \(10\) (summed log-prob \(-10\)) and response \(B\) has length \(30\) (summed log-prob \(-30\)). (a) Using a summed-log-prob reward \(r=\beta\sum_t\log\pi_\theta\) with \(\beta=0.1\) (take the reference term as identical for both, so it drops out), compute \(r_A\) and \(r_B\) and say which the summed reward prefers. (b) Using SimPO’s length-normalized reward \(r=\frac{\beta}{|y|}\log\pi_\theta\) with \(\beta=2.0\), compute \(r_A\) and \(r_B\). © In one sentence, state what this shows about a length-blind summed reward.

5. (Implementation) The chapter gives the CPO loss as a formula but no code. CPO is “reference-free DPO plus an SFT anchor”:

$$
\mathcal{L}_{\text{CPO}} = -\log\sigma\big(\beta\log\pi_\theta(y_w\mid x) - \beta\log\pi_\theta(y_l\mid x)\big)\;-\;\lambda\log\pi_\theta(y_w\mid x).
$$

Implement `cpo_loss` in the chapter's style. It takes the policy sequence log-probs for the chosen and rejected responses (outputs of `sequence_logprob`) and the chosen SFT negative log-likelihood `chosen_nll` (as `orpo_loss` does), plus `beta` and `lam`. Return the mean loss. Note that no reference log-probs appear anywhere — that is the point.

import torch
import torch.nn.functional as F

def cpo_loss(pol_chosen_logps, pol_rejected_logps, chosen_nll,
             beta=0.1, lam=1.0):
    """
    pol_chosen_logps / pol_rejected_logps : (B,) SEQUENCE log-probs (sum over
        tokens) of chosen/rejected under the policy -- output of sequence_logprob.
        NOTE: no reference log-probs -- CPO is reference-free.
    chosen_nll : (B,) standard SFT negative log-likelihood on the chosen response
        (= -pol_chosen_logps if you score the full chosen sequence).
    """
    # Reference-free contrastive term: DPO's log-sigmoid on RAW policy log-probs.
    logits = beta * (pol_chosen_logps - pol_rejected_logps)      # (B,)
    contrastive = -F.logsigmoid(logits)                         # -log sigma(.)

    # SFT anchor -lambda * log pi(yw) == lambda * chosen_nll, holds chosen up.
    return (contrastive + lam * chosen_nll).mean()