13.4 Watermarking, Provenance & AI-Content Detection¶

As generative models reach human-level fluency, the question “did a machine write this?” becomes legally, commercially, and politically loaded. A journalism outlet needs to know whether a submitted opinion piece is AI-generated. A school needs evidence for an academic-integrity decision. A regulator enforcing the EU AI Act needs a disclosure mechanism that works at scale. This chapter develops three complementary answers to that question: proactive watermarking embedded during generation, content-provenance standards that cryptographically chain a file to its origin, and post-hoc detection applied to content whose origin is unknown. We will see why each approach is necessary, how each works at the algorithmic level, and why none of them is sufficient alone.

Related chapters that establish useful background: Privacy, Memorization & Differential Privacy for LLMs covers what information leaks from model outputs and is a natural companion here; Safety, Guardrails & Content Moderation covers the runtime enforcement layer; Mechanistic Interpretability & Model Internals provides the foundational lens on how models work internally; AI Governance, Compliance & Regulation treats the legal landscape in depth.

Why Watermarking Exists: The Motivation¶

Consider a model that can produce text indistinguishable from human writing with essentially zero marginal cost. Without a provenance signal, every downstream consumer of that text must either (a) trust the claimed source, (b) run a statistical detector that is probabilistically unreliable, or © give up. The asymmetry is extreme: a determined attacker can polish AI text with trivial paraphrasing, but a defender running a post-hoc classifier faces a distribution shift problem — they cannot enumerate every possible model or every possible attack.

Watermarking inverts this asymmetry. The model operator embeds a signal at generation time. Detection is then a verification problem (does this signal match our key?) rather than a classification problem (does this text look AI-generated?). The analogy is a cryptographic signature: the hardness of verification scales with the strength of a secret key, not with the adversary’s sophistication at mimicking text.

Three desiderata tension against each other:

1. Detectability — the signal must be recoverable with high statistical confidence from a short passage.
2. Imperceptibility — watermarked text must be indistinguishable (in quality, style, content) from unwatermarked text.
3. Robustness — the signal must survive paraphrase, translation, summarization, or selective editing by an adversary.

These three cannot all be maximized simultaneously, and the tradeoffs define the design space.

Statistical Text Watermarking: The Green-List Scheme¶

The Kirchenbauer et al. Construction¶

The seminal paper by Kirchenbauer, Geiping, Wen, Kirchenbauer, Goldblum, and Goldstein (2023) — commonly called the KGW watermark — embeds a signal by biasing the token sampling distribution at each step.

The construction works as follows. Let \(V\) be the vocabulary of size \(|V|\). At each generation step \(t\), the preceding context (or a hash of it) is used to seed a pseudorandom function \(f\) that partitions \(V\) into a green list \(G_t \subseteq V\) of size \(\gamma |V|\) and a red list \(R_t = V \setminus G_t\).

During generation, the logit for every token \(v \in G_t\) is boosted by a hardness parameter \(\delta > 0\) before softmax:

The model therefore preferentially samples green tokens without requiring any modification to the model weights. An unmodified greedy or nucleus-sampled text would use \(\gamma |V|\) green tokens in expectation. A watermarked text uses them at a much higher rate.

Detection. Given a candidate text of \(T\) tokens and a secret key \(k\), the detector reconstructs each \(G_t\) and counts the number of green tokens \(g\). Under the null hypothesis (human text), \(g \sim \text{Binomial}(T, \gamma)\). The z-score is:

A threshold \(z^* \approx 4\) corresponds to a false-positive rate on the order of \(10^{-5}\) per document.

Be careful with that null, though: it assumes each scored token is green independently with probability \(\gamma\), and repeated \(n\)-grams break the assumption. If a passage repeats the phrase “the quarterly revenue figure” five times, the same context hashes to the same green list every time and the same token is scored green five times — correlated draws that inflate \(z\) and quietly raise the real false-positive rate above the nominal one. Production detectors therefore deduplicate: Hugging Face’s WatermarkDetector exposes exactly this as ignore_repeated_ngrams=True, which scores each distinct context \(n\)-gram once. Always report which convention you used when you quote a \(z\)-score.

Context Hashing and Key Security¶

The green list \(G_t\) is typically derived by hashing the previous \(h\) tokens together with a secret key \(k\):

where TopGamma selects the \(\lfloor \gamma |V| \rfloor\) tokens with the lowest hash values (a deterministic pseudo-random selection). Setting \(h=1\) (hash only the previous token) creates a Markov-1 watermark that is easy to implement but can be approximated by an adversary who observes many samples. Setting \(h \geq 4\) dramatically increases the adversary’s work because the number of possible contexts is \(|V|^h \sim 32000^4 \approx 10^{18}\).

Distortion-Free Watermarks¶

A limitation of the KGW scheme is that boosting green-list logits changes the output distribution — it is not distortion-free. For tasks where output quality is critical (medical summarization, legal reasoning), this is unacceptable.

Kuditipudi, Thickstun, Hashimoto, and Liang (2023) introduced distortion-free watermarking. The key idea: rather than modifying the logit distribution, use the secret key to generate a sequence of random numbers \(\{r_t\}\) and then select the token whose inverse-CDF transform under the model distribution corresponds to \(r_t\):

where \(F_t^{-1}\) is the quantile function of the model’s next-token distribution. Because we apply a monotone transformation, the marginal distribution of each token is unchanged — the text is statistically identical to unmodified sampling. Yet the sequence of tokens is correlated with the known random sequence, giving a detectable signal.

Detection uses a test based on the rank statistics of the observed tokens under the model’s predicted distribution: under watermarking, observed tokens should cluster near \(r_t\) in probability space, while human text is uniform.

A closely related distortion-free trick, usually credited to Scott Aaronson’s 2022 description of an (unreleased) OpenAI scheme, is Gumbel or exponential-minimum sampling: draw a key-derived uniform \(r_v \in (0,1)\) for every token \(v\) and emit \(\arg\max_v r_v^{1/p_v}\). This is exactly the Gumbel-max trick with the randomness supplied by the key instead of a fresh RNG, so the emitted token is distributed exactly as \(p\); but the detector, which can recompute \(\{r_v\}\), sees that the chosen tokens have systematically high \(r_v\) and sums \(-\log(1 - r_{w_t})\) into a test statistic. The catch shared by all such schemes: the marginal per-token distribution is preserved, but generation becomes deterministic given the key and the context, so the same prompt regenerates the same text. Practical systems break this by mixing a per-request nonce into the key (see the key-derivation exercise at the end of this chapter).

Tournament Sampling: SynthID-Text¶

The scheme actually deployed at production scale on text is SynthID-Text (Dathathri et al., Scalable watermarking for identifying large language model outputs, Nature 2024), used in Gemini. It is worth understanding because it sits between KGW and the distortion-free family and because its implementation is open source.

Instead of perturbing logits, SynthID-Text perturbs the sampling step with a knockout tournament. At step \(t\):

1. Draw \(2^\ell\) candidate tokens i.i.d. from the model’s true next-token distribution \(p_t\) (so far, nothing is distorted).
2. Hash the previous \(n-1\) tokens (an \(n\)-gram context) together with a set of watermarking keys to produce \(\ell\) pseudorandom scoring functions \(g_1,\dots,g_\ell\), each mapping a candidate token to a value in \(\{0,1\}\) (or a small finite set).
3. Run the tournament: in layer \(i\), pair the surviving candidates and keep, from each pair, the one with the larger \(g_i\) value; break ties uniformly at random.
4. Emit the single survivor.

With one layer (\(\ell = 1\)) the scheme is non-distortionary: averaged over the random keys, the emitted token is still distributed as \(p_t\), because both members of each pair were drawn from \(p_t\) and the tie-breaking is symmetric. Adding layers strictly increases detectability — the emitted token wins more and more key-derived comparisons — at the cost of a small, controlled deviation from \(p_t\). That single knob, “how many tournament layers,” is the cleaner analogue of KGW’s \(\delta\).

Detection is a mean test: compute \(\bar{g} = \frac{1}{\ell T}\sum_{t,i} g_i(w_t)\) over the passage. Under the null this concentrates around its chance value; under watermarking it is biased upward. The Nature paper reports that a learned Bayesian detector — a small classifier trained on watermarked and unwatermarked samples from the same model — is substantially more sample-efficient than the training-free weighted-mean test, which matters because you want confident detection from a few dozen tokens, not a few hundred. Two implementation details are load-bearing: repeated-context masking (skip positions whose \(n\)-gram context has already been seen, so a repeated phrase does not get watermarked twice and does not get counted twice) and the fact that, exactly as with KGW, low-entropy spans carry no signal because all \(2^\ell\) candidates are the same token.

From-Scratch Green-List Watermark: Code¶

"""
green_list_watermark.py
From-scratch implementation of a KGW-style token watermark.
We simulate the core algorithm without a full language model,
using a random vocabulary and a toy probability distribution.
"""

import hashlib
import struct
import math
import random
from dataclasses import dataclass


# ---------------------------------------------------------------------------
# Vocabulary and pseudo-logits (stand-in for a real LM)
# ---------------------------------------------------------------------------

VOCAB_SIZE = 32_000
VOCAB = list(range(VOCAB_SIZE))  # tokens are just integers

def fake_lm_logits(prev_token: int, seed: int = 42) -> list[float]:
    """
    Returns random logits for demonstration. A real implementation
    would call model.forward() and extract the next-token logit vector.
    """
    rng = random.Random(seed ^ prev_token)
    return [rng.gauss(0, 1) for _ in VOCAB]


# ---------------------------------------------------------------------------
# Green-list construction
# ---------------------------------------------------------------------------

def get_green_list(
    prev_token: int,
    secret_key: bytes,
    gamma: float = 0.5,
) -> set[int]:
    """
    Derive the green list for position t given the previous token and key.
    Uses HMAC-SHA256 as the PRF; maps each vocabulary token to a hash value
    and selects the lowest gamma*|V| hashes as the green list.
    """
    green_size = int(gamma * VOCAB_SIZE)

    # Score every token by hashing (key || prev_token || token_id)
    scores = []
    for tok in VOCAB:
        # Pack prev_token and tok as little-endian 32-bit ints
        data = secret_key + struct.pack("<II", prev_token, tok)
        h = hashlib.sha256(data).digest()
        # Interpret first 8 bytes as a uint64 for a uniform [0, 2^64) score
        score = struct.unpack("<Q", h[:8])[0]
        scores.append((score, tok))

    # The green list is the gamma fraction with lowest hash scores
    scores.sort()
    green_set = {tok for _, tok in scores[:green_size]}
    return green_set


# ---------------------------------------------------------------------------
# Watermarked sampler
# ---------------------------------------------------------------------------

def softmax(logits: list[float]) -> list[float]:
    """Numerically stable softmax."""
    m = max(logits)
    exps = [math.exp(l - m) for l in logits]
    s = sum(exps)
    return [e / s for e in exps]


def sample_token(probs: list[float], rng: random.Random) -> int:
    """Sample a token index from a probability distribution."""
    r = rng.random()
    cumulative = 0.0
    for i, p in enumerate(probs):
        cumulative += p
        if r < cumulative:
            return i
    return len(probs) - 1  # fallback


def generate_watermarked(
    seed_token: int,
    length: int,
    secret_key: bytes,
    delta: float = 2.0,
    gamma: float = 0.5,
    generation_seed: int = 0,
) -> list[int]:
    """
    Generate a watermarked token sequence of given length.

    Args:
        seed_token: The token preceding the generation window (context).
        length: Number of tokens to generate.
        secret_key: The watermark secret key (kept by the operator).
        delta: Green-list logit boost.
        gamma: Fraction of vocabulary in the green list.
        generation_seed: Seed for token sampling (mimics temperature sampling).

    Returns:
        List of generated token ids.
    """
    rng = random.Random(generation_seed)
    tokens = []
    prev = seed_token

    for step in range(length):
        # Get base logits from the (fake) language model
        logits = fake_lm_logits(prev, seed=step)

        # Construct green list for this step
        green = get_green_list(prev, secret_key, gamma=gamma)

        # Boost green-list logits
        boosted_logits = [
            l + delta if tok in green else l
            for tok, l in enumerate(logits)
        ]

        probs = softmax(boosted_logits)
        tok = sample_token(probs, rng)
        tokens.append(tok)
        prev = tok

    return tokens


# ---------------------------------------------------------------------------
# Detector
# ---------------------------------------------------------------------------

def detect_watermark(
    tokens: list[int],
    seed_token: int,
    secret_key: bytes,
    gamma: float = 0.5,
) -> dict:
    """
    Compute the z-score for a sequence of tokens.

    Returns a dict with z_score, green_count, total_tokens, and p_value.
    The null hypothesis is that the text is human-written (each token is
    in the green list with probability gamma, independently).
    """
    T = len(tokens)
    if T == 0:
        return {"z_score": 0.0, "green_count": 0, "total": 0, "p_value": 1.0}

    green_count = 0
    prev = seed_token

    for tok in tokens:
        green = get_green_list(prev, secret_key, gamma=gamma)
        if tok in green:
            green_count += 1
        prev = tok

    # z-score under Binomial(T, gamma) null
    mu = gamma * T
    sigma = math.sqrt(T * gamma * (1 - gamma))
    z = (green_count - mu) / sigma

    # One-sided p-value (standard normal CDF approximation via erfc)
    p_value = 0.5 * math.erfc(z / math.sqrt(2))

    return {
        "z_score": round(z, 4),
        "green_count": green_count,
        "total_tokens": T,
        "gamma_expected": round(mu, 1),
        "p_value": round(p_value, 8),
        "flagged": z > 4.0,
    }


# ---------------------------------------------------------------------------
# Demo
# ---------------------------------------------------------------------------

if __name__ == "__main__":
    KEY = b"supersecret-operator-key-2024"
    SEED_TOKEN = 1234
    N = 200  # tokens to generate

    print("=== Watermarked text ===")
    wm_tokens = generate_watermarked(SEED_TOKEN, N, KEY, delta=2.0, gamma=0.5)
    result_wm = detect_watermark(wm_tokens, SEED_TOKEN, KEY)
    print(result_wm)
    # Expected: z_score >> 4, flagged=True

    print("\n=== Human (random) text ===")
    rng = random.Random(99)
    human_tokens = [rng.randint(0, VOCAB_SIZE - 1) for _ in range(N)]
    result_human = detect_watermark(human_tokens, SEED_TOKEN, KEY)
    print(result_human)
    # Expected: z_score near 0, flagged=False

    print("\n=== Paraphrase attack: replace 40% tokens randomly ===")
    attacked = list(wm_tokens)
    rng2 = random.Random(7)
    for i in range(N):
        if rng2.random() < 0.4:
            attacked[i] = rng2.randint(0, VOCAB_SIZE - 1)
    result_atk = detect_watermark(attacked, SEED_TOKEN, KEY)
    print(result_atk)
    # Expected: z_score substantially reduced from the unattacked value;
    # in this synthetic (random-logit) simulation it drops below the z*=4.0
    # threshold, though with a real LM's peakier distributions it typically
    # stays flagged until 70-80% of tokens are replaced (see prose below).

Running this demo (with the fixed seeds shown above) produces exactly this output:

=== Watermarked text ===
{'z_score': 11.4551, 'green_count': 181, 'total_tokens': 200, 'gamma_expected': 100.0, 'p_value': 0.0, 'flagged': True}

=== Human (random) text ===
{'z_score': -0.7071, 'green_count': 95, 'total_tokens': 200, 'gamma_expected': 100.0, 'p_value': 0.76024994, 'flagged': False}

=== Paraphrase attack: replace 40% tokens randomly ===
{'z_score': 3.3941, 'green_count': 124, 'total_tokens': 200, 'gamma_expected': 100.0, 'p_value': 0.00034426, 'flagged': False}

Here the 40% substitution attack already drops \(z\) below the 4.0 threshold for this particular random draw — a reminder that in this toy simulation (random logits standing in for a real LM, so the model has no genuine preference among tokens) the watermark carries less signal than it would in a real deployment, where a language model’s confident, low-entropy continuations mean a much larger fraction of tokens must be destroyed before \(z\) drops below threshold. With a real model and this \(\delta,\gamma\) setting, published results show an adversary typically needs to replace on the order of 70–80% of tokens to reliably evade detection, at which point the original content is largely gone.

The Same Watermark With Real Libraries¶

You would not ship the loop above. The KGW scheme is built into Hugging Face transformers as a LogitsProcessor plus a matching detector, so watermarking a real model is a two-object change to your generation call. The parameter names map one-to-one onto the mechanism we just implemented: greenlist_ratio is \(\gamma\), bias is \(\delta\), context_width is \(h\), and hashing_key is the secret \(k\).

# pip install "transformers>=4.46" torch
import torch
from transformers import (
    AutoModelForCausalLM, AutoTokenizer,
    WatermarkingConfig, WatermarkDetector,
)

model_id = "HuggingFaceTB/SmolLM2-135M-Instruct"   # any causal LM; ~100M-class here
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id).eval()

# The KGW green-list watermark, i.e. exactly the scheme coded from scratch above.
wm_cfg = WatermarkingConfig(
    greenlist_ratio=0.25,       # gamma
    bias=2.0,                   # delta (logit boost on green tokens)
    hashing_key=15485863,       # the SECRET key -> HSM in production
    seeding_scheme="lefthash",  # hash the h previous tokens; "selfhash" also available
    context_width=1,            # h
)

prompt = tok("The economics of small language models are", return_tensors="pt")
out = model.generate(
    **prompt, do_sample=True, top_p=0.95, temperature=1.0,
    max_new_tokens=200,
    watermarking_config=wm_cfg,   # merged into the generation config -> logits processor
)

# Detection needs the IDENTICAL config: same key, gamma, scheme and context width.
detector = WatermarkDetector(
    model_config=model.config, device="cpu",
    watermarking_config=wm_cfg,
    ignore_repeated_ngrams=True,  # see the independence caveat above
)
res = detector(out, z_threshold=3.0, return_dict=True)
print(res.num_tokens_scored, res.num_green_tokens, res.green_fraction)
print("z =", res.z_score, "p =", res.p_value, "flagged =", res.prediction)

Two things are worth noticing. First, seeding_scheme="selfhash" hashes the candidate token along with the context, which makes the green list depend on the token being scored and materially raises the cost of the adaptive attacks described below — at the price of a \(|V|\)-way hash per step instead of one. Second, the detector takes the full sequence including the prompt; if you score prompt tokens the model never chose, you dilute \(z\) exactly like the copy-paste splicing attack in Exercise 3.

SynthID-Text lives in the same API surface:

from transformers import SynthIDTextWatermarkingConfig

synthid_cfg = SynthIDTextWatermarkingConfig(
    keys=[654, 400, 836, 123, 340, 443, 597, 160, 57],  # one key per tournament layer
    ngram_len=5,                                        # context n-gram width
)
out = model.generate(**prompt, do_sample=True, max_new_tokens=200,
                     watermarking_config=synthid_cfg)

Detection here is not free: transformers ships SynthIDTextWatermarkDetector wrapping a BayesianDetectorModel that you must first fit on watermarked and unwatermarked samples from your own model — the detector is model- and key-specific. Budget for that training step (and for holding out a clean corpus of your model’s unwatermarked output) when you plan a deployment. Google’s google-deepmind/synthid-text repo carries the reference implementation and the mean/weighted-mean detectors alongside the Bayesian one.

For research and evaluation rather than serving, MarkLLM (THU-BPM/MarkLLM, EMNLP 2024 demo) is the standard harness: it implements two dozen schemes — KGW, Unigram, SynthID-Text, SIR, EXP/Gumbel, and more — behind one interface, together with attack and robustness pipelines so you can compare them on the same model:

# Installed from source; see the repo README for the exact config schema.
from watermark.auto_watermark import AutoWatermark

wm = AutoWatermark.load("KGW",
                        algorithm_config="config/KGW.json",
                        transformers_config=my_transformers_config)
text = wm.generate_watermarked_text("Explain rotary position embeddings.")
print(wm.detect_watermark(text))       # -> {"is_watermarked": True, "score": ...}

In the serving stack. A watermark is just a logits processor, so it belongs at the same point in the pipeline as top-p and repetition penalty — see Sampling Strategies & Decoding Algorithms. vLLM exposes a logits-processor extension point for exactly this, though the interface changed with the V1 engine (per-request logits_processors callables in the older engine; a batched LogitsProcessor class registered as a plugin in V1), so pin your version and check its docs before writing the hook. The ordering rule is version-independent and easy to get wrong: apply the watermark bias after temperature and truncation warpers, otherwise top-\(k\)/top-\(p\) may prune the green tokens you just promoted — transformers appends the watermark processor last for precisely this reason.

If you are building Stack-100M in Part XIV, this is the hook to add to its serving path (Evaluation & Serving: Honest Benchmarks, int4 Quantization, and Running on a Laptop). Do measure before you trust it: a 100M-parameter model produces shorter, lower-entropy, more repetitive text than a frontier model, which is the regime where \(z\) accumulates slowest, so empirically calibrate how many tokens you need for \(z > 4\) on your model rather than importing a number from the paper.

Robustness to Attacks¶

A watermark that fails under modest editing provides only false assurance. The main attack classes are:

The context-window hash choice (\(h\), the number of preceding tokens hashed) determines how an insertion/deletion attack propagates. With \(h=1\) (Markov-1), inserting one token shifts the green list for only one future token. With \(h=4\), it shifts four. Sliding-window detection (try all possible insertion offsets) partially recovers.

Adaptive adversaries who can query the model repeatedly — and observe which tokens land in the green list — can, in principle, reconstruct the green list and mask it. This motivates multi-bit and multi-key watermarks: rather than a single binary signal, embed a message identifier drawn from a space of \(2^{32}\) or more possible keys, so that cracking one key does not transfer.

Multimodal Watermarking: SynthID and Its Relatives¶

Text watermarking exploits the discrete token distribution. Image, audio, and video watermarking must operate in continuous, high-dimensional pixel/waveform space and must survive JPEG compression, resizing, screen-capture, and codec re-encoding.

SynthID-Class Image Watermarking¶

Google DeepMind’s SynthID (first announced in 2023 for Imagen, later extended across Gemini’s modalities) embeds a learned, imperceptible pattern into image pixel values. Rather than modifying pixel intensities post-hoc (the classical LSB or DWT approach), SynthID pairs a small neural encoder that injects the signal with a decoder network that recovers it, both trained end-to-end against a differentiable stack of augmentations (JPEG, resize, crop, noise) so the signal learns to survive them. The closest published academic construction is The Stable Signature (Fernandez et al., ICCV 2023), which fine-tunes a latent diffusion model’s decoder so that every image the model emits carries a per-user binary signature — the watermark is rooted in the weights rather than bolted on afterwards, so an attacker cannot simply delete a post-processing step. Such learned encoders aim to add a near-zero-magnitude steganographic signal that:

• Survives lossy compression down to moderate JPEG quality factors (on the order of Q=50).
• Survives resizing and cropping.
• Is visually imperceptible — the perturbation is well below the just-noticeable-difference threshold on natural images, though the exact reported fidelity numbers vary by system and are not always published.

Detection uses the same decoder network: the residual embedding is projected onto a learned detection vector, and a threshold test is applied.

Spectral domain techniques. Earlier (non-neural) approaches work in the frequency domain. The DWT-DFT watermark embeds a pseudo-random bit sequence into the mid-frequency DCT or DFT coefficients of an image. These are robust to JPEG (which discards high-frequency coefficients) but remain in the preserved mid-frequencies. The trade-off is that neural-network detectors can learn to strip these signals when given enough examples.

Audio and Video¶

For speech synthesis (text-to-speech), audio watermarking can be applied in the mel-spectrogram domain before vocoding or via a neural encoder-decoder analogous to SynthID. Google’s SynthID for audio (deployed with the Lyria music models from 2024) converts the waveform to a spectrogram, embeds the signal there, and converts back; Google reports robustness to the common transformations audio actually undergoes — lossy compression such as MP3, added noise, and moderate speed/tempo changes — though the precise operating envelope is not fully published. Meta’s AudioSeal (open source, facebookresearch/audioseal) is the equivalent you can actually run: a localized watermark whose detector emits a per-sample probability, so it can find a watermarked span spliced into a longer human recording rather than scoring the clip as a whole.

Video watermarking faces an additional adversary: temporal resampling (frame-rate change, slow-motion). Frame-level image watermarks compound across frames, so detection uses majority voting across sampled frames, which is robust to partial frame replacement.

Content Provenance: C2PA and Signed Manifests¶

Watermarking answers “was this generated by model X?” but not “who generated it, when, and with what inputs?”. Content provenance standards answer the richer question by attaching a cryptographically signed manifest to the file at the moment of creation.

The C2PA Standard¶

The Coalition for Content Provenance and Authenticity (C2PA) is a joint effort of Adobe, Microsoft, BBC, Intel, Sony, and others. The specification is now in its 2.x line (2.0 landed in 2024, with further 2.x point releases since — check spec.c2pa.org for the current version, and note that the specification has also been taken into formal international standardization). It defines:

• Content Credentials: a JSON-LD manifest embedded in the file’s XMP/JUMBF metadata. It records the asset’s identity, creation tool, timestamp, and a list of actions (crop, generate, edit) performed.
• Hard-binding: a cryptographic hash of the asset bytes (SHA-256) is included in the manifest, binding the manifest to exactly this file. Any modification breaks the hash.
• Soft-binding: a perceptual hash (e.g., pHash) allows approximate matching after lossy transformations. If the hard-binding hash fails but the perceptual hash matches, the tool reports “modified from a C2PA-signed original.”
• X.509 certificate chain: the manifest is signed with an operator certificate, whose trust roots back to a C2PA-approved CA. This gives a verifiable, non-repudiable binding between the manifest and the signer’s identity.

{
  "alg": "sha256",
  "assertions": [
    {
      "label": "c2pa.actions",
      "data": {
        "actions": [
          {
            "action": "c2pa.created",
            "softwareAgent": "MyAIImageGen v2.1",
            "when": "2025-04-10T14:23:00Z",
            "digitalSourceType": "trainedAlgorithmicMedia"
          }
        ]
      }
    },
    {
      "label": "c2pa.hash.data",
      "data": {
        "alg": "sha256",
        "hash": "a3f4...c291"
      }
    }
  ],
  "claim_generator": "MyAIImageGen/2.1 c2pa-rs/0.25.0",
  "signature_info": {
    "issuer": "MyAI Corp",
    "cert_serial_number": "0x4A2F..."
  }
}

Signing and Verifying in Practice: c2pa-rs and c2patool¶

You do not implement C2PA yourself. The reference implementation is the Rust crate c2pa-rs from the Content Authenticity Initiative, with a CLI (c2patool) and bindings for Python, JavaScript, and C. The manifest above is the input you hand to it; the tool does the JUMBF embedding, the hard-binding hash, and the COSE/X.509 signing.

# Read whatever credentials an asset already carries (exits non-zero if invalid).
c2patool ./candidate.jpg --detailed

# Sign an asset with the manifest definition shown above.
# The manifest JSON references a signing cert chain + private key
# (in production: a cert from a C2PA-recognised CA, key held in an HSM/KMS).
c2patool ./render.png --manifest ./manifest.json --output ./signed.png

# pip install c2pa-python   -- Reader/Builder mirror the CLI's read/sign paths.
from c2pa import Reader

with Reader.from_file("signed.png") as reader:
    manifest_store = reader.json()   # validation status + full assertion list
print(manifest_store)

Check the crate’s README for the exact flag and API names, which have moved across the 0.x releases. The engineering point that does not move: verification is a certificate-chain problem, not an image-processing problem. A valid signature tells you only that some holder of some trusted certificate asserted these actions; the trust decision is about which roots you accept, and revocation and expiry apply exactly as they do for TLS. This is also why provenance and watermarking are deployed together — an operator generating an image typically embeds a SynthID-class pixel watermark and signs a C2PA manifest, so that stripping the metadata leaves the pixel signal and re-encoding the pixels leaves the manifest.

Limitations of Provenance¶

C2PA solves honest-party verification: if a trustworthy party generates content and signs it, downstream consumers can verify the claim. It does not prevent a malicious party from generating content with a modified tool that forges the manifest, or from distributing content through channels that strip metadata.

The key insight: C2PA and watermarking are complementary. C2PA provides a rich, human-readable, auditable record for compliant workflows. Watermarking provides a signal that survives metadata stripping. Together they cover the main attack surface.

Post-Hoc AI-Text Detection: What Works and What Doesn’t¶

When proactive watermarks are absent, operators sometimes fall back to post-hoc detectors that classify text as AI- or human-written based on statistical features. We must be honest about their limitations.

Likelihood-Based Detectors¶

The simplest family uses the generating model’s own perplexity. If a text has unusually low perplexity under GPT-4, it was probably written by GPT-4. DetectGPT (Mitchell et al., 2023) formalizes this: it computes the model’s log-probability on the candidate text, then samples small perturbations (via a masking model) and asks whether the original is a local maximum of log-probability — an expected property of model-sampled text.

If \(\Delta(x) > 0\) by a large margin, the text is flagged as AI-generated. The method requires access to the generating model’s probability, which is unavailable for closed-source models.

Trained Classifier Detectors¶

Fine-tuned classifiers (e.g., GPTZero, or OpenAI’s since-withdrawn AI Text Classifier) are trained on (human, AI) pairs. They achieve high accuracy on in-distribution examples but degrade severely under:

• Domain shift: a classifier trained on Reddit posts misclassifies academic AI writing.
• Prompt conditioning: AI text generated with unusual system prompts is less “typical” and evades detection.
• Short texts: below roughly 100 words, false-positive rates jump dramatically.
• Paraphrase: light editing by a human or a second model is often sufficient to evade commercial detectors.

A rigorous audit by Weber-Wulff et al. (2023) found that widely-used commercial tools had false-positive rates of 2–10% on human text and true-positive rates as low as 30–70% on real AI-generated content — worse than their advertised numbers. This has profound fairness implications: non-native English speakers tend to use more regular, predictable sentence structures that resemble AI text, and their work is flagged at higher rates.

Why Post-Hoc Detection Is Fundamentally Limited¶

A language model that is fine-tuned post-hoc to evade a specific detector can do so while maintaining utility on the downstream task. This is a Goodhart’s Law dynamic: as soon as a measure of AI-ness becomes a target, it ceases to be a reliable measure. Watermarking avoids this problem by putting the signal under the operator’s private-key control.

The EU AI Act: Article 50 and Transparency Obligations¶

The EU AI Act’s transparency obligations for generative AI take effect on 2 August 2026 — though the 2026 “Omnibus” simplification reform granted generative-AI systems already on the market a grace period until 2 December 2026 to meet the machine-readable-marking requirement. Article 50 specifies:

Note who each obligation binds — the Article splits duties between providers (who build and ship the system) and deployers (who use it), and the split is the part engineers most often get wrong:

1. Chatbot disclosure — Article 50(1), on providers. Systems intended to interact directly with natural persons must make it clear to the person that they are interacting with an AI, unless that is obvious from the context.

2. Machine-readable marking — Article 50(2), on providers. Providers of AI systems that generate synthetic audio, image, video, or text must ensure the outputs are marked in a machine-readable format and detectable as artificially generated or manipulated. The Act requires the solutions to be “effective, interoperable, robust and reliable as far as this is technically feasible” — deliberately technology-neutral wording that watermarks, C2PA-style credentials, and metadata all aim at.

3. Deepfake and public-interest-text disclosure — Article 50(4), on deployers. A deployer publishing AI-generated or manipulated image, audio, or video constituting a deepfake must disclose it. Separately, a deployer publishing AI-generated or AI-manipulated text in order to inform the public on matters of public interest must disclose that too — unless the content underwent human review and a natural or legal person holds editorial responsibility for it.

The Act leaves the specific technical implementation to the Commission (via standards bodies, likely ETSI and CEN/CENELEC), and C2PA-style content credentials are the leading candidate for a compliant machine-readable marking. Non-compliance with the Article 50 transparency obligations falls under the second fine tier: up to EUR 15 million or 3% of total worldwide annual turnover, whichever is higher.

Practically, this means:

• LLM API providers serving EU users must expose a watermark API (or embed watermarks by default) for public-facing text generation.
• Image and video generation services must embed C2PA credentials or equivalent.
• The editorial-responsibility exemption is narrower than it first looks: it relieves the deployer of the disclosure duty for public-interest text, but it does not relieve the provider of the Article 50(2) machine-readable marking duty. If you build the model, human review downstream does not get you out of marking.

Cross-reference: AI Governance, Compliance & Regulation covers the full EU AI Act risk-tier framework and the broader global compliance landscape.

System-Level Architecture for a Production Watermark Pipeline¶

Deploying watermarking at scale requires integrating it into the inference serving stack without unacceptable latency overhead.

For high-throughput serving, the hash computation can be parallelized on GPU. The green-list can be represented as a bitmask of \(|V|\) bits (32000 / 8 = 4 KB), which fits comfortably in L1 cache. The logit-addition is a vectorized CUDA kernel that adds delta * mask to the logit vector, adding negligible latency.

A real implementation also needs:

• Key management: the secret key must be stored in a hardware security module (HSM) or equivalent; compromise of the key allows an adversary to construct text that passes detection.
• Per-request key rotation: rather than a global key, derive a per-request subkey from the master key and a request ID. This limits the exposure from any single leaked subkey.
• Audit log: record the request ID → watermark key mapping so that a flagged document can be traced back to the originating request (with appropriate legal authorization).

Practical Limitations and Open Problems¶

Despite their elegance, current watermarking schemes face several real-world limitations.

Low-entropy text. If the next token is highly predictable — e.g., completing a fixed template, writing code with rigid syntax, or generating a phone number — the model has essentially no degrees of freedom. The green list is irrelevant because the only viable token may be red. The effective green-token rate collapses, and the \(z\)-score is diluted. This is a fundamental tension: watermarking works best when the model has many plausible continuations.

Multi-model provenance. When a document is partly human-written, partly AI-generated, and partly AI-edited, neither a single \(z\)-score nor a binary classifier can give a nuanced answer. Forensic attribution — “this paragraph is AI-generated; this one is not” — requires token-level rather than document-level watermarking, an active research area.

Open-weight models. Any open-weight model (LLaMA, Mistral, etc.) can be deployed without watermarking. The operator can remove the watermark processor trivially. This means watermarking mandates (like EU AI Act Article 50) apply only to API-based closed models; open-weight models present a policy gap. One proposed mitigation is embedding watermarks in the model weights (via fine-tuning to prefer green tokens without an explicit processor), but this is easily circumvented by further fine-tuning. This is the honest position for a project like Stack-100M: once you release checkpoints, you cannot make anyone else’s copy watermark its output. What you can do — and what a responsible release does — is watermark the endpoint you host, sign the artifacts you publish, and document the model card so downstream users know what they are re-hosting.

Semantic watermarks. An emerging line of work embeds watermarks at the semantic level — in the choice of which concepts to mention, which synonyms to use — rather than at the token level. Semantic watermarks are more robust to paraphrase but harder to detect without access to a semantic model, and their statistical properties are less well understood.

Key Takeaways¶

Further Reading¶

• Kirchenbauer, J., Geiping, J., Wen, Y., Kirchenbauer, K., Goldblum, M., and Goldstein, T. — A Watermark for Large Language Models (2023). The foundational green-list watermark paper.
• Kuditipudi, R., Thickstun, J., Hashimoto, T., and Liang, P. — Robust Distortion-Free Watermarks for Language Models (2023). Introduces the distortion-free inverse-CDF construction.
• Fernandez, P., Couairon, G., Jégou, H., Douze, M., and Furon, T. — The Stable Signature: Rooting Watermarks in Latent Diffusion Models (NeurIPS 2023). Neural watermarking for latent diffusion.
• Mitchell, E., Lee, Y., Khazatsky, A., Manning, C. D., and Finn, C. — DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature (ICML 2023).
• Weber-Wulff, D. et al. — Testing of Detection Tools for AI-Generated Text (2023). Rigorous empirical audit of commercial detectors.
• C2PA Technical Specification v2.0 — Coalition for Content Provenance and Authenticity (2024). The normative standard for content credentials.
• Google DeepMind — SynthID: Identifying AI-Generated Content (Nature, 2024). Details of SynthID image and audio watermarking.
• Dathathri, S. et al. — Scalable Watermarking for Identifying Large Language Model Outputs (Nature, 2024). The SynthID-Text tournament-sampling scheme, its non-distortionary configuration, and the learned Bayesian detector, evaluated in a live Gemini deployment.
• Zhao, X., Ananth, P., Li, L., and Wang, Y. — Provably Robust Multi-bit Watermarking for AI-Generated Text (2023). Multi-bit extensions with information-theoretic robustness proofs.

Exercises¶

1. The chapter lists three desiderata for a watermark — detectability, imperceptibility, and robustness — and claims they cannot all be maximized at once. Using the KGW green-list scheme, explain concretely how turning the hardness knob \(\delta\) up trades one desideratum against another. Then explain why a hospital deploying an AI clinical-summary tool might prefer the Kuditipudi et al. distortion-free construction over KGW even though both are “detectable.”

2. A detector receives a candidate passage of \(T = 400\) tokens produced with a watermark parameter \(\gamma = 0.25\) (a quarter of the vocabulary is green at each step). It counts \(g = 140\) green tokens. Compute the mean and standard deviation of the green count under the human-text null, the \(z\)-score, and decide whether the passage is flagged at \(z^\* = 4.0\). Roughly what is the one-sided \(p\)-value?

3. Copy-paste splicing. An attacker takes a strongly watermarked snippet of \(60\) tokens (assume, optimistically for the attacker’s target, that every one of those 60 is green) and pastes it into \(240\) tokens of genuinely human-written text, giving a document of \(T = 300\) tokens. The watermark uses \(\gamma = 0.5\). Compute the expected \(z\)-score of the spliced document. Then show that no 60-token watermarked snippet, however strong, can push this 300-token document over \(z^\* = 4.0\), and explain the lesson.

4. The “System-Level Architecture” section recommends per-request key derivation: instead of a single global secret, derive a per-request subkey from a master key and a request ID, so a leaked subkey does not compromise the whole system. Implement a derive_subkey(master_key, request_id) helper (HMAC-SHA256 based) and show the minimal changes needed to generate_watermarked and detect_watermark so both use the derived subkey. Verify that detection succeeds only when the correct request ID is supplied.

import hmac
import hashlib

def derive_subkey(master_key: bytes, request_id: str) -> bytes:
    """
    Deterministically derive a per-request watermark subkey.
    HMAC-SHA256(master_key, request_id) -> 32-byte subkey.
    Leaking one subkey does not reveal master_key or any other subkey.
    """
    return hmac.new(master_key, request_id.encode("utf-8"),
                    hashlib.sha256).digest()


MASTER_KEY = b"master-operator-key-in-HSM"
REQ_ID = "req-2026-07-000123"

# --- generation: derive, then reuse the existing sampler unchanged ---
subkey = derive_subkey(MASTER_KEY, REQ_ID)
wm_tokens = generate_watermarked(
    seed_token=1234, length=200, secret_key=subkey,
    delta=2.0, gamma=0.5,
)

# --- detection with the CORRECT request id -> flagged ---
correct = detect_watermark(
    wm_tokens, seed_token=1234,
    secret_key=derive_subkey(MASTER_KEY, REQ_ID),
)
print("correct id:", correct["z_score"], correct["flagged"])
# e.g. z_score ~ 11.5, flagged=True

# --- detection with the WRONG request id -> not flagged ---
wrong = detect_watermark(
    wm_tokens, seed_token=1234,
    secret_key=derive_subkey(MASTER_KEY, "req-2026-07-000999"),
)
print("wrong id:  ", wrong["z_score"], wrong["flagged"])
# z_score ~ 0, flagged=False

5. Extend the from-scratch code with a robustness_sweep(...) function that, for a fixed watermarked sequence, measures the detector \(z\)-score after randomly replacing a fraction \(p\) of tokens, for \(p \in \{0.0, 0.1, \dots, 0.9\}\). Report the smallest \(p\) at which \(z\) drops below the \(z^\*=4.0\) threshold in this toy simulation, and reconcile the result with the chapter’s claim that against a real LM an adversary must destroy roughly 70–80% of tokens.

import random

def robustness_sweep(
    wm_tokens: list[int],
    seed_token: int,
    secret_key: bytes,
    gamma: float = 0.5,
    fractions=None,
    trials: int = 20,
    base_seed: int = 0,
) -> list[tuple[float, float]]:
    """
    For each replacement fraction p, average the detector z-score over
    `trials` random substitutions, and return [(p, mean_z), ...].
    """
    if fractions is None:
        fractions = [i / 10 for i in range(10)]  # 0.0 .. 0.9
    n = len(wm_tokens)
    out = []
    for p in fractions:
        zs = []
        for t in range(trials):
            rng = random.Random(base_seed + t)
            attacked = list(wm_tokens)
            for i in range(n):
                if rng.random() < p:
                    attacked[i] = rng.randint(0, VOCAB_SIZE - 1)
            res = detect_watermark(attacked, seed_token, secret_key,
                                   gamma=gamma)
            zs.append(res["z_score"])
        out.append((p, sum(zs) / len(zs)))
    return out


if __name__ == "__main__":
    KEY = b"supersecret-operator-key-2024"
    wm = generate_watermarked(1234, 200, KEY, delta=2.0, gamma=0.5)
    for p, mean_z in robustness_sweep(wm, 1234, KEY):
        flag = "flagged" if mean_z > 4.0 else "clear"
        print(f"p={p:.1f}  mean_z={mean_z:6.2f}  {flag}")