3.15 Synthetic Data for Pre- and Post-Training¶

For most of deep learning’s history, “data” meant something you found: ImageNet was scraped and hand-labeled, Common Crawl was harvested from the open web, and the implicit contract was that the internet would always be bigger than your model’s appetite. That contract is breaking. Frontier models are now trained on tens of trillions of tokens, and the supply of high-quality human-written text — clean prose, correct code, well-structured explanations — is finite and largely exhausted. The Chinchilla math from Scaling Laws: Kaplan, Chinchilla & Beyond says we want roughly 20 tokens per parameter; a 1-trillion-parameter model wants ~20T tokens, and the clean, deduplicated, English subset of the web is smaller than that. We are running into a data wall.

Synthetic data is the response. Instead of (or in addition to) scraping tokens, we generate them with a model — rephrasing messy web text into clean prose, writing textbook-style explanations of concepts, manufacturing instruction–answer pairs, or distilling the step-by-step reasoning of a strong teacher into a smaller student. Done well, this is one of the highest-leverage techniques in the modern stack: it converts compute (cheap, scalable) into high-quality tokens (scarce), and it lets you target exactly the distribution you want. Done badly, it is a slow-motion catastrophe — model collapse, where models trained on their own outputs progressively forget the tails of the distribution and converge to bland, narrowed mush.

This chapter is about doing it well. We will cover the five dominant recipes — web rephrasing (WRAP), textbook-style generation (Phi, Cosmopedia, FineWeb-Edu), instruction-augmented pretraining, reasoning-trace distillation, and instruction/preference synthesis for post-training — and the open-source stack that implements them (vLLM for generation, distilabel for orchestration, datatrove for filtering and decontamination, math-verify and sandboxes for verification). We will derive when mixing synthetic with natural data helps versus hurts, build the math of model collapse from first principles, write runnable generation-and-verification pipelines, and catalog the failure modes — contamination, distributional narrowing, fact drift — that separate a useful synthetic corpus from a poisoned one.

This chapter builds on Pretraining Data: Sources, Crawling & The Data Pipeline (where natural tokens come from), Data Cleaning, Deduplication & Quality Filtering (the filters synthetic data must also pass), and Data Mixing, Domain Weighting & Curriculum (how to weight a natural/synthetic blend). The post-training half connects to Supervised Fine-Tuning & Instruction Tuning, Distillation, Model Compression & Knowledge Transfer, and RL with Verifiable Rewards (RLVR) & The Reasoning Recipe. Both halves are exercised concretely in the capstone: synthetic textbooks are 15% of Stack-100M’s pretraining mix (Data: Sourcing, Filtering, Dedup, Tokenize & Pack ~20B Tokens), and its SFT/DPO/RLVR data is synthesized (Post-Training: SFT, DPO, and Narrow RLVR (GRPO)).

Why Generate Tokens At All?¶

The data wall, quantified¶

Let us make the scarcity concrete. Estimates of the high-quality public text stock vary, but a useful order-of-magnitude figure is that the clean, deduplicated, English web is on the order of \(10^{13}\) tokens — call it tens of trillions, with a long tail of lower-quality text beyond that. Meanwhile training token counts have grown roughly an order of magnitude every couple of years. When the model’s appetite catches up to the supply, you have three moves:

1. Train more epochs over the same data. Repetition gives diminishing returns and eventually hurts — re-seeing the same tokens four-plus times yields little new signal and risks memorization (see Privacy, Memorization & Differential Privacy for LLMs). The Muennighoff et al. “data-constrained scaling” work quantifies this: up to ~4 epochs is roughly as good as fresh data, after which value decays fast.
2. Lower the quality bar and ingest more of the noisy tail. This dilutes the average token’s value and can degrade the model.
3. Manufacture new high-quality tokens with a generator model. This is synthetic data.

The third option is attractive because of an asymmetry: verifying or restyling text is easier than writing it from scratch, and a model that is mediocre at open-ended generation can still be excellent at transforming existing good text. That asymmetry — generation-easier-than-from-scratch, verification-easier-than-generation — is the engine behind every recipe in this chapter.

A taxonomy of synthetic data¶

It helps to classify by how far the synthetic text departs from a human-written source. The further you go, the more value you can add — and the more risk you take on.

• Source-grounded transformation (WRAP-style rephrasing). Take a real document; rewrite it in a target style (clean, formal, Q&A). Facts are inherited from the source, so hallucination risk is bounded by the source’s accuracy plus paraphrase drift.
• Knowledge-augmented generation. Seed with a real passage but add structure the model knows — worked examples, contrasting cases, follow-up questions. Risk rises because added content is not grounded.
• Prompt-seeded generation (Cosmopedia, Phi textbooks). Seed only with a topic, audience, and format (“write a textbook section on photosynthesis for a high-schooler”). The model writes from its parametric knowledge. Highest density of clean pedagogical text, highest hallucination risk.
• Reasoning-trace distillation (post-training). Have a strong teacher solve problems with chain-of-thought, then train a student on those traces — ideally keeping only traces whose final answer is verified correct.

Recipe 1 — Rephrasing the Web (WRAP)¶

The lowest-risk, highest-ROI recipe is Web Rephrasing Augmented Pretraining (WRAP), introduced by Maini et al. The insight: raw web text is information-rich but stylistically poor — boilerplate, broken HTML remnants, SEO spam, inconsistent formatting. Loss on such text is dominated by predicting noise. If you take each web document and ask an instruction-tuned model to rewrite it in one of a few clean styles, you get tokens that carry the same information at much higher “signal per token.”

WRAP uses a small set of rewrite styles, for example:

• Easy / Wikipedia-style: clear expository prose, like an encyclopedia entry.
• Q&A: turn the passage into question–answer pairs.
• Concise / formal: tighten and formalize.

The reported result is striking: pretraining on a mix of original web text plus its rephrasings reaches a target perplexity with roughly 3× fewer tokens than web text alone, and improves downstream zero-shot accuracy — without needing any human-written corpus beyond the original web. Crucially, you keep the original document too; the rephrasing augments rather than replaces, which preserves the factual grounding and the natural diversity of the source.

The recipe has since been validated at production scale. NVIDIA’s Nemotron-CC (2024) rebuilt Common Crawl by rephrasing borderline documents into cleaner form instead of discarding them — the explicit shift from “filter away 90% of the crawl” to “rewrite what you would have thrown out” — and reports retaining several times more unique high-quality tokens than pure filtering at equal quality (see Pretraining Data: Sources, Crawling & The Data Pipeline). If you only ever adopt one recipe from this chapter, adopt this one: it is the cheapest, the most grounded, and the best-evidenced.

The generator: vLLM in offline-batch mode¶

Every recipe in this chapter is, mechanically, offline batch inference over a corpus. The standard open-source engine for that job is vLLM (see vLLM: Architecture, PagedAttention & Internals): its LLM class runs continuous batching and PagedAttention over an arbitrarily long list of prompts with no server in the loop. SGLang’s Engine and TensorRT-LLM’s Python runtime are drop-in alternatives; the economics matter because you are running inference over billions of tokens, so throughput — not latency — is the objective.

"""
The generator behind every recipe in this chapter. One vLLM engine, batched
offline. `pip install vllm`; needs a GPU.
"""
from vllm import LLM, SamplingParams

class BatchGenerator:
    """Thin adapter so the recipes below can pass sampling params as a dict."""
    def __init__(self, model="meta-llama/Llama-3.2-3B-Instruct",
                 tensor_parallel_size=1, quantization=None):
        self.llm = LLM(
            model=model,                       # any 1-8B instruct model will do
            tensor_parallel_size=tensor_parallel_size,
            quantization=quantization,         # e.g. "fp8" / "awq" -> ~2x throughput
            gpu_memory_utilization=0.90,       # leave headroom for the KV cache
            max_model_len=8192,
        )
        self.tokenizer = self.llm.get_tokenizer()

    def generate(self, prompts, sampling_params):
        # vLLM's real signature takes a SamplingParams object; accept a dict too.
        sp = (SamplingParams(**sampling_params)
              if isinstance(sampling_params, dict) else sampling_params)
        return self.llm.generate(prompts, sp)   # -> [RequestOutput]; .outputs[0].text

    def chat_prompt(self, user_msg: str) -> str:
        """Render the model's chat template -- instruct models need it."""
        return self.tokenizer.apply_chat_template(
            [{"role": "user", "content": user_msg}],
            tokenize=False, add_generation_prompt=True)

# llm = BatchGenerator()   # every `llm` below is this object

A runnable WRAP pipeline¶

With that generator in hand, here is the core loop.

"""
WRAP-style web rephrasing. For each web doc, sample one of a few rewrite
styles and generate a clean version. Keep BOTH original and rephrasing.
"""
import random
from dataclasses import dataclass

# A small, fixed menu of rewrite styles. Diversity of styles is what keeps
# the synthetic distribution from collapsing onto one bland mode.
STYLE_PROMPTS = {
    "wiki":    "Rewrite the following text as a clear, encyclopedic passage. "
               "Preserve all factual content; remove boilerplate and noise.\n\n{doc}",
    "qa":      "Convert the following text into 2-4 self-contained "
               "question-and-answer pairs covering its key facts.\n\n{doc}",
    "concise": "Rewrite the following text concisely and formally, keeping "
               "every fact but removing redundancy.\n\n{doc}",
}

@dataclass
class WrapConfig:
    max_input_chars: int = 6000      # truncate very long docs before rewriting
    max_new_tokens: int = 1024
    temperature: float = 0.7         # some entropy -> stylistic diversity
    styles_per_doc: int = 1          # how many rephrasings per source doc

def build_prompts(docs, cfg: WrapConfig):
    """Yield (doc_id, style, prompt) for batched generation."""
    for doc_id, doc in docs:
        doc = doc[: cfg.max_input_chars]
        for _ in range(cfg.styles_per_doc):
            style = random.choice(list(STYLE_PROMPTS))
            yield doc_id, style, STYLE_PROMPTS[style].format(doc=doc)

def rephrase_corpus(docs, llm, cfg: WrapConfig):
    """
    `llm.generate(prompts, ...)` is the vLLM batched API. We process the whole
    shard at once so the GPU stays saturated -- throughput, not latency, is
    what dominates cost when rewriting trillions of tokens.
    """
    items = list(build_prompts(docs, cfg))
    prompts = [p for (_, _, p) in items]
    outputs = llm.generate(
        prompts,
        sampling_params=dict(temperature=cfg.temperature,
                             max_tokens=cfg.max_new_tokens),
    )
    for (doc_id, style, _), out in zip(items, outputs):
        text = out.outputs[0].text.strip()
        # Emit BOTH the original (once) and the rephrasing. Keeping the
        # original preserves grounding and natural diversity.
        yield {"doc_id": doc_id, "kind": "synthetic",
               "style": style, "text": text}

Why keep the original — and how much to mix¶

If you trained only on rephrasings, you would inherit every quirk of the rewriter: its preferred sentence rhythms, its vocabulary, its tendency to “tidy away” rare facts. Keeping the original web text anchors the model to the true data distribution. The empirical sweet spot in WRAP-style work is roughly a 1:1 mix of natural and synthetic — enough synthetic to lift quality, enough natural to preserve the tails. We will formalize this trade-off in the model-collapse section below.

Recipe 2 — Textbook-Style Generation (Phi, Cosmopedia, FineWeb-Edu)¶

The second recipe goes further: generate new expository content rather than rewriting a specific source. The thesis, from Microsoft’s Phi series (“Textbooks Are All You Need,” Gunasekar et al.), is that data quality can substitute for quantity and model size. A small model trained on a curated diet of “textbook-quality” text — clear, pedagogically structured, low-noise — can match much larger models trained on raw web scrapings. Phi-1 (1.3B) reached strong code performance on a tiny, curated corpus; later Phi models extended the recipe to general reasoning. By Phi-4 (14B, late 2024) synthetic data had become a central pillar of the pretraining mix rather than a garnish — and, tellingly, phi-4 surpasses its own teacher on STEM question-answering, evidence that careful data generation can go beyond plain distillation.

There are two ways to get textbook-quality tokens:

1. Filter the web for them (FineWeb-Edu). Train a lightweight classifier to score how “educational” a web page is, then keep only the high-scoring pages. This is selection, not generation — but it is the natural-data analogue of textbook generation and pairs with it. See Data Cleaning, Deduplication & Quality Filtering.
2. Generate them (Cosmopedia, the open reproduction of Phi’s synthetic data). Prompt a strong model with a seed (a topic, an audience, a format) and have it write the passage. The art is in the prompt diversity — without it, you generate the same fifty essays a million times. Cosmopedia v2 (the regeneration SmolLM2 actually trained on, published under HuggingFaceTB/smollm-corpus) is exactly the 15% synthetic slice of our capstone’s pretraining mix — see Data: Sourcing, Filtering, Dedup, Tokenize & Pack ~20B Tokens.

The FineWeb-Edu educational classifier¶

The filter half is cheap and worth always doing. You score the web with a small regression head on top of frozen embeddings, trained on a few hundred thousand LLM-labeled examples.

"""
FineWeb-Edu-style educational quality filter. A strong LLM labels a seed set
with an 'educational value' score 0-5; we distill that into a fast linear head
on frozen sentence embeddings, then score the whole web with the cheap head.
"""
import numpy as np
from sklearn.linear_model import Ridge

def make_labeling_prompt(text: str) -> str:
    # The rubric is the product. Be explicit about what 'educational' means
    # so the labels are consistent across millions of pages.
    return (
        "Rate the educational value of the text for teaching a student, "
        "on an additive 0-5 scale:\n"
        "+1 if it contains some educational content,\n"
        "+1 if it addresses a topic relevant to school/college curricula,\n"
        "+1 if it is coherent and well-written,\n"
        "+1 if it is comprehensive and self-contained,\n"
        "+1 if it is outstanding, like a high-quality textbook.\n"
        "Reply with ONLY the integer.\n\nTEXT:\n" + text[:4000]
    )

def train_edu_head(embeds: np.ndarray, scores: np.ndarray) -> Ridge:
    """embeds: (N, d) frozen embeddings; scores: (N,) LLM labels in [0,5]."""
    head = Ridge(alpha=1.0)
    head.fit(embeds, scores)
    return head

def keep_mask(head: Ridge, embeds: np.ndarray, threshold: float = 3.0):
    """Keep only pages the head scores >= threshold (e.g. 3/5)."""
    preds = head.predict(embeds)
    return preds >= threshold

A threshold around ⅗ typically keeps a small single-digit percentage of the web but yields a corpus on which models learn dramatically faster per token. The lesson generalizes: a cheap classifier distilled from expensive LLM labels is one of the best dollar-for-dollar quality levers in pretraining.

You do not have to build this from scratch. The real artifact is on the Hub as HuggingFaceFW/fineweb-edu-classifier — a regression head over a frozen sentence-embedding backbone, trained on a few hundred thousand pages annotated 0–5 by a large instruct model with essentially the rubric above — and it loads with transformers’ AutoModelForSequenceClassification like any other model. The scoring pass itself belongs in a sharded data pipeline: datatrove (the library that built FineWeb) and NVIDIA’s NeMo-Curator both ship classifier-filter stages that run the same logic over thousands of shards on SLURM or Ray.

Cosmopedia-style generation with prompt diversity¶

For generation, the central engineering problem is avoiding repetition at scale. If you prompt “write a textbook section about biology” a million times, deduplication will throw away 99% of it. The fix is a combinatorial prompt template seeded by web snippets (so topics track real-world distribution) crossed with audiences, styles, and formats.

"""
Cosmopedia-style synthetic textbook generation. The key is PROMPT DIVERSITY:
seed each prompt with a real web snippet (for topical grounding) and sample an
audience + format. This combinatorial explosion keeps near-duplicates rare.
"""
import random, hashlib

AUDIENCES = ["a curious middle-school student", "an undergraduate",
             "a professional in an unrelated field", "a young child"]
FORMATS   = ["a textbook section with a worked example",
             "a Socratic dialogue", "a how-to tutorial with steps",
             "a blog post that builds intuition first"]
STYLES    = ["rigorous and precise", "warm and intuitive",
             "concise and example-driven"]

def make_generation_prompt(seed_snippet: str) -> str:
    aud, fmt, sty = (random.choice(AUDIENCES),
                     random.choice(FORMATS), random.choice(STYLES))
    return (
        f"Using the topic implied by the snippet below, write {fmt} aimed at "
        f"{aud}. Be {sty}. Do not mention the snippet; teach the underlying "
        f"concept thoroughly and accurately. If you state a fact, make sure it "
        f"is correct.\n\nSNIPPET:\n{seed_snippet[:1500]}\n\nPASSAGE:\n"
    )

def dedup_key(text: str, n: int = 8) -> str:
    """MinHash-lite: hash the first n normalized words as a near-dup bucket.
    In production use real MinHash/LSH (see the dedup chapter)."""
    words = " ".join(text.lower().split()[:n])
    return hashlib.md5(words.encode()).hexdigest()

def generate_textbook(seeds, llm, seen=None):
    seen = seen if seen is not None else set()
    prompts = [make_generation_prompt(s) for s in seeds]
    outs = llm.generate(prompts, sampling_params=dict(
        temperature=1.0, top_p=0.95, max_tokens=1500))  # high entropy = diversity
    for out in outs:
        text = out.outputs[0].text.strip()
        k = dedup_key(text)
        if k in seen:          # cheap near-dup guard; real dedup is downstream
            continue
        seen.add(k)
        yield {"kind": "synthetic_textbook", "text": text}

Note the deliberately high temperature and top_p: for expository generation we want entropy, because diversity is the whole game. This is the opposite of reasoning distillation (Recipe 4), where we will often want low temperature and verified correctness.

Recipe 3 — Instruction- and QA-Augmented Pretraining¶

A third recipe blurs the line between pretraining and post-training: inject instruction-formatted data (questions, tasks, and their answers) directly into the pretraining mix. The UL2 / FLAN line of work and, more pointedly, the “instruction pretraining” results show that interleaving QA pairs derived from raw text during pretraining improves both the base model’s quality and its downstream instruction-following — the model arrives at the SFT stage already fluent in the question-answer shape.

The mechanism is reading-comprehension synthesis: for each raw passage, generate questions whose answers are grounded in that passage, then train on passage → questions+answers. Because the answers come from the passage, grounding is strong and hallucination is low — this is closer to WRAP than to free generation.

"""
Instruction-augmented pretraining: turn raw passages into grounded
reading-comprehension QA, so the base model sees instruction-shaped data
DURING pretraining. Answers are extracted from the passage -> low hallucination.
"""
QA_SYNTH_PROMPT = """\
From the passage, write {k} diverse question/answer pairs. Each ANSWER must be
fully supported by the passage -- do not use outside knowledge. Mix factual,
inferential, and "why/how" questions. Format strictly as:
Q: ...
A: ...

PASSAGE:
{passage}
"""

def synth_qa(passages, llm, k=3):
    prompts = [QA_SYNTH_PROMPT.format(k=k, passage=p[:4000]) for p in passages]
    outs = llm.generate(prompts, sampling_params=dict(
        temperature=0.7, max_tokens=512))
    for p, out in zip(passages, outs):
        qa_text = out.outputs[0].text.strip()
        # Interleave the ORIGINAL passage with its QA so the model learns the
        # link between source text and answerable questions.
        yield {"kind": "instruct_pretrain",
               "text": p.strip() + "\n\n" + qa_text}

This recipe is the cheapest way to give a base model “a head start” on instruction following, and it composes with everything in Supervised Fine-Tuning & Instruction Tuning. The key constraint is the grounding instruction (“ANSWER must be supported by the passage”) — drop it and you slide from augmentation into free generation, with all the hallucination risk that entails.

Recipe 4 — Reasoning-Trace Distillation for Post-Training¶

The post-training half of synthetic data is where the most dramatic recent gains live. The recipe: take a strong teacher that can reason (produce chain-of-thought, write and check code), have it solve a large bank of problems, filter to keep only verified-correct traces, and fine-tune a student on those traces. This is rejection-sampling fine-tuning (sometimes called STaR-style after Zelikman et al.’s “Self-Taught Reasoner,” or RFT), and it is how reasoning capability gets compressed from a frontier model into a small one. It connects deeply to Distillation, Model Compression & Knowledge Transfer, Reasoning, Chain-of-Thought & Test-Time Compute, and RL with Verifiable Rewards (RLVR) & The Reasoning Recipe.

The crucial difference from the pretraining recipes: here we can often verify the output objectively. A math answer either matches the gold answer or it does not; code either passes the unit tests or it does not. Verification turns noisy generation into a high-precision data source — we over-generate and filter hard.

Rejection-sampling fine-tuning, end to end¶

"""
Reasoning-trace distillation via rejection sampling (STaR/RFT-style).
For each problem with a known answer, sample K chain-of-thought traces from a
teacher, KEEP only traces whose final answer is verified correct, dedup, and
build an SFT dataset of (problem -> correct trace).
"""
import re
from collections import defaultdict

def extract_final_answer(trace: str):
    """Pull the boxed/'final answer' from a CoT trace. Robust parsing matters;
    a bad extractor silently throws away good traces or keeps bad ones."""
    m = re.search(r"\\boxed\{([^}]*)\}", trace)
    if m:
        return m.group(1).strip()
    m = re.search(r"(?:final answer|answer)\s*[:=]\s*(.+)", trace, re.I)
    return m.group(1).strip() if m else None

def verify(pred, gold) -> bool:
    """Domain-specific. For math, normalize then compare; for code, run tests
    in a sandbox (see the reward-verifiers chapter). Here: normalized string eq."""
    if pred is None:
        return False
    norm = lambda s: re.sub(r"\s+", "", str(s)).rstrip(".").lower()
    return norm(pred) == norm(gold)

def distill_reasoning(problems, llm, K=8, max_keep_per_problem=2):
    """
    problems: list of dicts {"question": str, "gold": str}
    Over-generate K traces/problem, verify, keep up to `max_keep` correct +
    DIVERSE traces per problem.
    """
    # Sample K completions per problem in one big batched call.
    prompts, owners = [], []
    for i, p in enumerate(problems):
        cot = (p["question"] +
               "\n\nThink step by step, then give the final answer "
               "in \\boxed{}.")
        for _ in range(K):
            prompts.append(cot); owners.append(i)

    outs = llm.generate(prompts, sampling_params=dict(
        temperature=0.8, top_p=0.95, max_tokens=2048))  # entropy -> diverse traces

    kept = defaultdict(list)
    for owner, out in zip(owners, outs):
        trace = out.outputs[0].text
        pred  = extract_final_answer(trace)
        if verify(pred, problems[owner]["gold"]):
            kept[owner].append(trace.strip())

    dataset = []
    for i, traces in kept.items():
        # Light diversity: dedup identical first lines, cap per problem so easy
        # problems don't swamp the dataset (a curriculum/balance concern).
        seen, chosen = set(), []
        for t in traces:
            key = t.split("\n", 1)[0]
            if key in seen:
                continue
            seen.add(key); chosen.append(t)
            if len(chosen) >= max_keep_per_problem:
                break
        for t in chosen:
            dataset.append({"question": problems[i]["question"],
                            "trace": t, "kind": "reasoning_sft"})
    return dataset

Three design points are doing the heavy lifting:

• Over-generate, filter hard. Sampling \(K=8\) traces and keeping only correct ones is the verification asymmetry in action. Even if the teacher is only ~40% correct per attempt, with \(K=8\) you cover a large fraction of solvable problems at least once: the probability of at least one success is \(1-(1-0.4)^8 \approx 0.98\).
• Cap per problem. Easy problems get solved every time and would dominate the dataset; capping enforces balance, a curriculum concern shared with RL Data, Curriculum & Replay Management.
• Keep diverse correct traces, not just one. Multiple valid solution paths teach the student that reasoning is a search, not a single memorized script.

The one line above that you should not ship as written is verify. Normalized string equality marks 0.5 wrong against 1/2 and (x+1)^2 wrong against x^2+2x+1, silently discarding correct traces and teaching the student to avoid whole answer formats. Use HuggingFace’s math-verify (parse the prediction and the gold, then verify them with a SymPy-backed equivalence check) for math, and a real sandboxed executor for code — see Reward Engineering, Verifiers & Sandboxes. Downstream, the resulting {"question", "trace"} records are just an SFT dataset: format them with the student’s chat template and hand them to TRL’s SFTTrainer (TRL: HuggingFace’s RL Library). huggingface/open-r1 is the fully open reference implementation of this exact loop — generation with vLLM, verification, filtering, then SFT and GRPO — and is the best code to read before building your own. Our capstone runs a miniature version of it in Post-Training: SFT, DPO, and Narrow RLVR (GRPO).

Distillation vs. RL: where synthetic SFT stops¶

Rejection-sampling distillation is off-policy SFT on filtered samples. It is cheap, stable, and gets you most of the way. But it has a ceiling: the student can only imitate traces the teacher already produces, and SFT on someone else’s tokens can teach style without competence (the student parrots “Let me think step by step” without the underlying search). When you want the student to exceed its imitation ceiling, you move to RL with verifiable rewards, where the student generates its own traces and is rewarded for correctness — see GRPO, RLOO & Critic-Free RL and RL with Verifiable Rewards (RLVR) & The Reasoning Recipe. A common modern pipeline is: rejection-sampling SFT to bootstrap, then RLVR to push past the teacher. The landmark demonstration is DeepSeek-R1 (2025): the reasoning traces of an RL-trained frontier model were distilled by plain SFT into small dense students that inherited much of the teacher’s reasoning — the clearest evidence to date that this recipe transfers frontier capability into compact models.

Recipe 5 — Synthesizing Instructions and Preferences¶

Recipe 4 assumed you already had a bank of problems with known answers. For general assistant behavior you have neither the problems nor a verifier, and the instructions themselves must be manufactured. Three generations of technique, each removing more human input than the last:

• Self-Instruct (Wang et al., 2022). Start from ~175 hand-written seed tasks; few-shot the generator to invent new instructions; generate responses; drop any new instruction too similar (by ROUGE overlap) to one already in the pool. Alpaca’s 52k dataset is the canonical output, and this is the ancestor of nearly every open instruction set.
• Evol-Instruct (WizardLM, Xu et al.). Take an existing instruction and evolve it with a rewrite prompt: “add one more constraint,” “make the reasoning deeper,” “replace a general requirement with a specific one” (in-depth evolution), or “generate a new instruction in a rarer domain” (in-breadth evolution). Iterating builds a difficulty ladder from a tiny seed set — the cheapest way to get hard prompts, which is what SFT mixes are usually starved of.
• Magpie (Xu et al., 2024). The cleverest and cheapest. Prompt an already-aligned chat model with only the chat template’s user-turn prefix and nothing after it. The model is now sitting at a position where a user message belongs, so it writes one. Feed that instruction back as a normal prompt to get the response. No seeds, no prompt engineering, and the instruction distribution comes straight out of the model’s own post-training distribution rather than out of your imagination.

"""
Magpie-style instruction synthesis: two batched passes over the same engine.
Pass 1 gives the model ONLY the user-turn prefix, so it completes the USER's
message -> a synthetic instruction. Pass 2 answers it normally.
"""
def user_turn_prefix(tokenizer) -> str:
    """Everything the chat template emits BEFORE the user's content starts.
    Rendering a sentinel and splitting on it is template-agnostic."""
    rendered = tokenizer.apply_chat_template(
        [{"role": "user", "content": "\x00"}],
        tokenize=False, add_generation_prompt=False)
    return rendered.split("\x00")[0]        # e.g. "<|im_start|>user\n"

def magpie_pairs(llm, n=1024):
    tok = llm.tokenizer
    prefix = user_turn_prefix(tok)
    # The end-of-turn marker closes the synthetic user message. It is the EOS
    # for most chat templates; add the literal token if your template differs.
    stop = [s for s in [tok.eos_token] if s]
    # n copies of the SAME prompt: sampling entropy is what makes them differ.
    outs = llm.generate([prefix] * n, sampling_params=dict(
        temperature=1.0, top_p=0.95, max_tokens=256, stop=stop))
    instrs = [o.outputs[0].text.strip() for o in outs]
    # Yield is well under 100%: drop empties, fragments, and runaway lengths.
    instrs = [i for i in instrs if 8 <= len(i) <= 2000]

    prompts = [llm.chat_prompt(i) for i in instrs]
    outs = llm.generate(prompts, sampling_params=dict(
        temperature=0.7, max_tokens=1024))
    return [{"instruction": i, "response": o.outputs[0].text.strip()}
            for i, o in zip(instrs, outs)]

Magpie only works on a model that has been aligned — a base model given the same prefix will produce web text, not an instruction. That is the point: you are extracting a distilled shadow of somebody’s post-training data. Check the generator’s license before shipping the result.

Synthetic preference pairs for DPO¶

Preference tuning needs (prompt, chosen, rejected) triples, and those can be synthesized too. Three constructions, in increasing order of usefulness:

1. Cross-model pairs (UltraFeedback-style): sample responses to the same prompt from several models of differing strength, score each with a judge or reward model on helpfulness/honesty/etc., and pair the best against the worst.
2. Best-of-\(N\) from one model: sample \(N\) responses from a single policy and pair the judge’s top against its bottom.
3. On-policy best-of-\(N\): do (2) using the very model you are about to DPO.

Construction 3 is the one that works. DPO’s gradient raises \(\log \pi_\theta(y_w) - \log \pi_\theta(y_l)\); if the rejected response is something your model would never have produced, the loss spends its capacity on a contrast the model has already learned, and you get style transfer instead of improvement. On-policy pairs put the contrast exactly where the policy’s own failure mass lives (Direct Preference Optimization & Its Variants, and the DPO stage of Post-Training: SFT, DPO, and Narrow RLVR (GRPO)).

"""
On-policy synthetic preference pairs: sample N responses from the SFT model we
are about to DPO, score with a judge, pair best vs worst. Skip near-ties -- a
pair whose members the judge cannot separate is label noise, not signal.
"""
def synth_preference_pairs(prompts, policy_llm, judge_score, N=4, min_gap=2.0):
    """judge_score(prompt, response) -> float (a judge LLM or a reward model)."""
    batch, owners = [], []
    for i, p in enumerate(prompts):
        cp = policy_llm.chat_prompt(p)
        batch += [cp] * N; owners += [i] * N

    outs = policy_llm.generate(batch, sampling_params=dict(
        temperature=1.0, top_p=0.95, max_tokens=1024))  # entropy -> real spread

    cands = {}
    for owner, o in zip(owners, outs):
        cands.setdefault(owner, []).append(o.outputs[0].text.strip())

    pairs = []
    for i, responses in cands.items():
        scored = sorted(((judge_score(prompts[i], r), r) for r in responses),
                        key=lambda t: t[0])
        (lo, worst), (hi, best) = scored[0], scored[-1]
        if hi - lo >= min_gap:            # discard uninformative near-ties
            pairs.append({"prompt": prompts[i],
                          "chosen": best, "rejected": worst})
    return pairs

Both loops above are exactly what distilabel packages as first-class tasks (SelfInstruct, EvolInstruct, Magpie, UltraFeedback), so in production you usually declare the pipeline rather than write these functions — but you should understand what the task is doing before you trust its output.

Mixing, Scaling, and the Math of Model Collapse¶

We now turn to the central question: how much synthetic data, mixed how, before things go wrong?

The accumulate-vs-replace distinction¶

The most important result in the model-collapse literature (Shumailov et al., “The Curse of Recursion”; Gerstgrasser et al., “Is Model Collapse Inevitable?”) is a sharp dichotomy:

• Replace. If each generation trains only on the previous generation’s synthetic output, discarding the original data, errors compound and the distribution narrows generation after generation. Variance shrinks, the tails vanish, and eventually the model produces near-constant gibberish. This is true model collapse.
• Accumulate. If you keep the original real data and add synthetic data on top — train each generation on real + all-synthetic-so-far — collapse is largely avoided. The real data acts as an anchor that pins the distribution in place.

This is exactly why every good recipe above keeps the original: WRAP keeps the source document, instruction-augmentation interleaves the real passage, FineWeb-Edu is real data. The failure mode is recursive replacement, not synthetic data per se.

A toy model of collapse: the shrinking Gaussian¶

The cleanest way to feel collapse is the Gaussian re-estimation chain. Suppose generation 0 is the true distribution \(\mathcal{N}(\mu, \sigma^2)\). Each generation draws \(n\) samples, fits a Gaussian by maximum likelihood, and the next generation samples from that fitted Gaussian — the “replace” regime. The fitted variance \(\hat\sigma^2_{t}\) is an unbiased estimate of \(\sigma^2_{t-1}\), but it has sampling noise, and that noise systematically erodes variance over time. The expected variance decays:

so after \(t\) generations the variance has shrunk by a factor \((1-1/n)^t\). With finite samples the tails progressively disappear; the distribution collapses toward its mean. Now contrast the accumulate regime, where each fit uses the original real samples plus all synthetic so far — the real samples keep re-injecting the true variance, and the decay halts. Let us watch it happen.

"""
Toy model of model collapse: recursive Gaussian re-estimation.
REPLACE: each generation fits only on the previous generation's samples.
ACCUMULATE: each generation fits on the ORIGINAL data plus all synthetic so far.
"""
import numpy as np

def collapse_chain(mode="replace", n=50, generations=30, seed=0):
    rng = np.random.default_rng(seed)
    true_mu, true_sigma = 0.0, 1.0
    real = rng.normal(true_mu, true_sigma, size=n)   # generation-0 REAL data

    mu, sigma = real.mean(), real.std()
    history = [(mu, sigma)]
    pool = list(real)  # for ACCUMULATE we grow this with synthetic samples

    for _ in range(generations):
        synth = rng.normal(mu, sigma, size=n)        # sample from current model
        if mode == "replace":
            fit = synth                              # forget the real data
        else:  # accumulate
            pool.extend(synth)                       # keep real + all synthetic
            fit = np.array(pool)
        mu, sigma = fit.mean(), fit.std()
        history.append((mu, sigma))
    return history

for mode in ("replace", "accumulate"):
    h = collapse_chain(mode=mode)
    s0, s_end = h[0][1], h[-1][1]
    print(f"{mode:>10}: sigma {s0:.3f} -> {s_end:.3f}  "
          f"({100*(s0-s_end)/s0:+.0f}% change after 30 generations)")

Running this, the replace chain shows the standard deviation steadily shrinking toward zero (variance collapse), while the accumulate chain holds its standard deviation near 1.0 — the real anchor prevents collapse. The same dynamic plays out in language models, with “variance” standing in for diversity of vocabulary, syntax, and ideas: replacement-trained models drift toward generic, repetitive, low-perplexity text, losing the rare words and unusual constructions that live in the tails.

Scaling behavior of mixed corpora¶

Two empirical regularities guide the mixing ratio (and connect to Data Mixing, Domain Weighting & Curriculum):

1. Synthetic data improves the effective token efficiency — you reach a target loss with fewer total tokens — but the gains saturate as the synthetic fraction grows. Past some fraction (often cited in the 30–50% range for rephrase-style data, higher for verified reasoning data), adding more synthetic stops helping and starts narrowing the distribution.
2. The optimal synthetic fraction depends on the quality gap. If your generator is much stronger than your target student (reasoning distillation from a frontier teacher), a high synthetic fraction is great. If your generator is the same class as your target (self-rephrasing), keep the fraction moderate and always anchor with real data, because you cannot distill capability the generator does not have — you can only restyle.

A simple, defensible default for pretraining mixes: 1–2 parts natural to 1 part synthetic, with synthetic capped around one-third to one-half of the blend. For post-training reasoning SFT, the dataset can be almost entirely synthetic because every example is verified — verification breaks the collapse dynamic by removing the error-accumulation channel.

Verification, Filtering, and Contamination Control¶

Synthetic data is only as good as its filters. The generation step is cheap; the verification and decontamination steps are where quality is won or lost. Treat the whole thing as a high-throughput data pipeline with quality gates (the philosophy of Data Cleaning, Deduplication & Quality Filtering and Data Flywheels & Continuous Improvement).

The verification ladder¶

Order your filters cheapest-first so you discard junk before paying for expensive checks:

"""
A cheapest-first verification ladder for a synthetic shard. Each stage drops
records so later, costlier stages process fewer items. Returns survivors + stats.
"""
def verify_shard(records, edu_head, embed_fn, judge_llm=None):
    stats = {"in": len(records)}
    out = []
    seen_ngrams = set()

    for r in records:
        t = r["text"]
        # Stage 1: format / length / language (cheap string ops)
        if not (50 <= len(t.split()) <= 4000):
            continue
        # Stage 2: cheap near-dup via 13-gram fingerprint set
        toks = t.lower().split()
        fp = hash(" ".join(toks[:13]))
        if fp in seen_ngrams:
            continue
        seen_ngrams.add(fp)
        out.append(r)
    stats["after_dedup"] = len(out)

    # Stage 3: quality classifier (one batched embedding + linear head)
    embeds = embed_fn([r["text"] for r in out])
    keep = edu_head.predict(embeds) >= 3.0
    out = [r for r, k in zip(out, keep) if k]
    stats["after_quality"] = len(out)

    # Stage 4: correctness is already enforced upstream for reasoning traces
    # (rejection sampling); for free-gen we may run an LLM judge here.
    if judge_llm is not None:
        out = [r for r in out if llm_judge_ok(judge_llm, r["text"])]
    stats["out"] = len(out)
    return out, stats

def llm_judge_ok(judge_llm, text: str) -> bool:
    """Ask a judge model for a factuality/coherence verdict. Expensive -> last."""
    prompt = ("Is the following passage coherent and free of obvious factual "
              "errors? Answer YES or NO.\n\n" + text[:3000])
    verdict = judge_llm.generate([prompt], sampling_params=dict(
        temperature=0.0, max_tokens=4))[0].outputs[0].text.strip().upper()
    return verdict.startswith("YES")

For verifiable domains (math, code), stage 4 is a real executor in a sandbox — see Reward Engineering, Verifiers & Sandboxes. For everything else, stage 5 is an LLM-as-judge pass; understand its biases before you trust it (LLM-as-a-Judge & Automated Evaluation).

Contamination: the silent killer of synthetic data¶

The single most dangerous failure mode is benchmark contamination. When your generator was itself trained on, or has memorized, the test sets you evaluate on (MMLU, GSM8K, HumanEval, …), its synthetic output can leak those test items verbatim or near-verbatim into your training data. You then score brilliantly on the benchmark and learn nothing — a contaminated number is worse than no number, because it actively misleads. Synthetic data amplifies this risk because the generator can regurgitate memorized eval items even when you never asked for them.

Decontamination is mandatory and must run against your eval suites:

"""
Decontamination: drop any synthetic record that overlaps an eval item by a
high-n-gram match. Use a generous n (e.g. 13-grams) and check against ALL
benchmarks you report on. This is non-negotiable for credible numbers.
"""
def ngrams(text, n=13):
    toks = text.lower().split()
    return {tuple(toks[i:i+n]) for i in range(len(toks) - n + 1)}

def build_eval_ngram_index(eval_items, n=13):
    idx = set()
    for item in eval_items:           # questions AND answers from every benchmark
        idx |= ngrams(item, n)
    return idx

def decontaminate(records, eval_index, n=13, max_overlap=0):
    """Drop a record if it shares more than `max_overlap` 13-grams with any
    eval item -- i.e. it has likely memorized/leaked a test question."""
    clean = []
    for r in records:
        overlap = len(ngrams(r["text"], n) & eval_index)
        if overlap <= max_overlap:
            clean.append(r)
    return clean

At corpus scale the loop above becomes a distributed set-intersection, and you should not write it yourself: datatrove ships n-gram decontamination and MinHash-LSH stages that run sharded on SLURM or Ray, the Dolma toolkit provides Bloom-filter-backed equivalents, and lm-evaluation-harness exposes the exact n-gram sets for the benchmarks you are about to report — which is the right source for eval_items, because a decontamination pass run against a different rendering of the benchmark than your evaluator uses will silently miss the leaks.

Distributional narrowing and fact drift¶

Two subtler failure modes round out the catalog:

• Distributional narrowing. Even without full collapse, synthetic data tends to over-represent the center of the distribution — common topics, standard phrasings, the generator’s stylistic tics — and under-represent rare entities, dialects, code-switching, and long-tail facts. Mitigate by seeding generation from real data (so topic frequency tracks reality) and by measuring diversity directly: track type-token ratio, self-BLEU (lower is more diverse), and embedding-space coverage of your synthetic corpus against the natural one.
• Fact drift / error laundering. A wrong fact generated once, kept, and trained on becomes “knowledge” the next generation confidently repeats and elaborates. This is how a single hallucination metastasizes across a model lineage. The defense is grounding (WRAP, instruction-augmentation), verification (reasoning traces), and source seeding — and never running a recursive self-generation loop without real-data anchoring and decontamination at every step.

"""
Diversity monitoring: a cheap self-BLEU proxy (lexical overlap among samples)
plus type-token ratio. Run these on every synthetic shard; a falling TTR or a
rising self-overlap is an early warning of narrowing/collapse.
"""
def type_token_ratio(texts):
    toks = [w for t in texts for w in t.lower().split()]
    return len(set(toks)) / max(1, len(toks))   # higher = more diverse

def self_overlap(texts, k=200):
    """Average fraction of shared unigrams between random pairs. Higher = more
    repetitive (a collapse warning). Sampled for speed."""
    import random
    sets = [set(t.lower().split()) for t in texts[:k]]
    pairs, tot = 0, 0.0
    for i in range(len(sets)):
        for j in range(i + 1, len(sets)):
            inter = len(sets[i] & sets[j])
            union = len(sets[i] | sets[j]) or 1
            tot += inter / union; pairs += 1
    return tot / max(1, pairs)                  # ~ Jaccard self-similarity

Wire these into your pipeline as monitored metrics: if type-token ratio falls or self-overlap rises shard-over-shard, your generator is narrowing and you should raise temperature, broaden seeds, or cut the synthetic fraction.

Interview Corner¶

Putting It Together: A Synthetic-Data Playbook¶

Stitching the recipes into one decision procedure:

The throughline across all six: synthetic data converts compute into targeted, high-quality tokens, and its safety is governed by grounding, verification, and the refusal to ever recursively replace real data. Generation is the easy 10%; the verification ladder, decontamination, and diversity monitoring are the 90% that decides whether your synthetic corpus is a moat or a landmine.

Further reading¶

• Maini, Seto, et al., Rephrasing the Web (WRAP): A Recipe for Compute and Data-Efficient Language Modeling.
• Gunasekar et al., Textbooks Are All You Need (Phi-1); Li et al., Textbooks Are All You Need II (phi-1.5); the broader Microsoft Phi technical reports.
• Ben Allal, Lozhkov, et al., Cosmopedia (HuggingFace) — open reproduction of Phi-style synthetic textbook data; and the FineWeb-Edu dataset and educational-classifier report.
• Shumailov et al., The Curse of Recursion: Training on Generated Data Makes Models Forget (model collapse).
• Gerstgrasser et al., Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data.
• Zelikman et al., STaR: Bootstrapping Reasoning with Reasoning; and rejection-sampling fine-tuning as used in the Llama and DeepSeek-R1 post-training reports.
• Muennighoff et al., Scaling Data-Constrained Language Models (how many epochs of real data are worth fresh tokens).
• Cheng et al., Instruction Pre-Training: Language Models Are Supervised Multitask Learners.
• Wang et al., Self-Instruct; Xu et al., WizardLM (Evol-Instruct); Xu et al., Magpie; Cui et al., UltraFeedback — the instruction- and preference-synthesis lineage of Recipe 5.
• The distilabel and open-r1 repositories (HuggingFace/Argilla) — production implementations of every generation loop in this chapter.

Exercises¶

1. A colleague sets up a training loop where each new model generation is trained only on text sampled from the previous generation, discarding the original web corpus to “save storage.” After a dozen rounds the outputs are bland and repetitive. Name the failure mode, explain the mechanism in one or two sentences, and describe the single change that would have prevented it — connecting your answer to why WRAP keeps the source document and why FineWeb-Edu is safe by construction.

2. You are doing rejection-sampling reasoning distillation. Your teacher solves a given problem on any single attempt with probability \(p = 0.25\) (independent attempts). (a) With \(K = 8\) samples per problem, what is the probability you obtain at least one correct trace? (b) How many samples \(K\) would you need so that the probability of at least one correct trace reaches \(0.99\)? © In one sentence, explain why this “over-generate, filter hard” scheme still yields a high-precision SFT dataset even though the teacher is wrong 75% of the time per attempt.

3. A 1-trillion-parameter model is Chinchilla-optimal at 20 tokens per parameter, so it wants \(20\text{T}\) training tokens. Your deduplicated, clean, English web supply is \(10\text{T}\) tokens. (a) If you simply repeat this corpus to reach \(20\text{T}\) tokens, how many epochs is that, and does the chapter consider it within the “safe” repetition zone? (b) You instead adopt a 1:1 natural:synthetic WRAP mix: keep each real token once and generate an equal volume of rephrasings. How many total tokens do you get, what fraction is synthetic, and how does that fraction compare to the chapter’s recommended cap? © Why is the WRAP option preferable to the repetition option even though both reach \(20\text{T}\) tokens?

4. Consider the “replace”-regime toy model of collapse with expected variance \(\sigma^2 (1 - 1/n)^t\) after \(t\) generations, where \(n\) is the number of samples drawn per generation. (a) With \(n = 25\) samples per generation, after how many generations does the expected variance fall to half its original value? (b) You want to slow collapse so that the half-life is at least 100 generations. What is the smallest \(n\) that achieves this? © What does this tell you about controlling collapse purely by increasing \(n\), versus the accumulate regime?

5. The distill_reasoning function keeps correct traces but reports nothing about how hard the problems were. Write a function coverage_report(problems, llm, K=8) that, reusing the chapter’s extract_final_answer and verify, samples \(K\) traces per problem and returns, for each problem, the empirical per-attempt solve rate \(\hat p\) (fraction of the \(K\) traces that were correct), plus two corpus-level numbers: the coverage (fraction of problems solved at least once) and the count of “too-easy” problems (solved on every attempt, which the chapter says should be capped so they do not swamp the dataset). Keep the batched-generation style of the chapter.

def coverage_report(problems, llm, K=8):
    """
    Sample K traces per problem, verify each against gold, and report
    per-problem empirical solve rate plus corpus-level difficulty stats.
    Reuses extract_final_answer() and verify() from the chapter.
    """
    # Build K identical CoT prompts per problem, one big batched call.
    prompts, owners = [], []
    for i, p in enumerate(problems):
        cot = (p["question"] +
               "\n\nThink step by step, then give the final answer "
               "in \\boxed{}.")
        for _ in range(K):
            prompts.append(cot); owners.append(i)

    outs = llm.generate(prompts, sampling_params=dict(
        temperature=0.8, top_p=0.95, max_tokens=2048))

    # Count correct traces per problem.
    correct = [0] * len(problems)
    for owner, out in zip(owners, outs):
        pred = extract_final_answer(out.outputs[0].text)
        if verify(pred, problems[owner]["gold"]):
            correct[owner] += 1

    per_problem = []
    solved = too_easy = 0
    for i, c in enumerate(problems):
        p_hat = correct[i] / K
        per_problem.append({"question": problems[i]["question"],
                            "p_hat": p_hat,
                            "n_correct": correct[i]})
        if correct[i] > 0:
            solved += 1
        if correct[i] == K:          # solved on every single attempt
            too_easy += 1

    return {
        "per_problem": per_problem,
        "coverage": solved / max(1, len(problems)),   # >=1 correct
        "n_too_easy": too_easy,                        # p_hat == 1.0
    }