4.7 Quantization I: Post-Training Quantization (GPTQ, AWQ, SmoothQuant)¶

A 70-billion-parameter model in 16-bit floating point weighs 140 GB. That does not fit on a single 80 GB A100 or H100, and even if it did, you would have nothing left for the KV cache or activations. Now store the same weights in 4 bits: 35 GB. Suddenly the model fits on one GPU, the weight-loading traffic from HBM (high-bandwidth memory) drops by 4x, and — because LLM decoding is memory-bandwidth bound, not compute bound (see The Roofline Model & Performance Engineering) — your tokens-per-second can nearly quadruple. The price you pay is a tiny, often imperceptible, degradation in quality.

This is the promise of quantization: represent weights (and sometimes activations) with fewer bits per number. This chapter covers post-training quantization (PTQ) — methods that take an already-trained FP16 model and squeeze it down without any gradient-based retraining, using at most a few hundred calibration examples and a few GPU-minutes. We build the arithmetic from scratch, confront the central villain of LLM quantization — activation outliers — and then dissect the three algorithms every LLM engineer must know: GPTQ, AWQ, and SmoothQuant, plus the mixed-precision trick LLM.int8(). The companion chapter Quantization II: INT4/INT8/FP8, GGUF, bitsandbytes & QAT covers number formats, file formats, and quantization-aware training.

The Arithmetic of Quantization¶

Quantization maps a continuous (or high-precision) value \(x\) to one of a small finite set of integers, then maps it back. The forward map is quantize, the inverse is dequantize. For a \(b\)-bit integer grid we have \(2^b\) levels.

Affine (asymmetric) quantization¶

The general affine scheme uses two parameters: a positive scale \(s\) (a floating-point number, the size of one quantization step) and an integer zero-point \(z\) (which integer code maps to the real value 0).

To quantize a real value \(x\):

and to dequantize back to an approximate real value \(\hat{x}\):

For unsigned \(b\)-bit codes, \(q_{\min}=0\) and \(q_{\max}=2^b-1\) (so \(0\ldots15\) for INT4, \(0\ldots255\) for INT8). Given the observed range \([\,x_{\min}, x_{\max}\,]\) of the tensor, we choose

The zero-point lets an asymmetric range like \([-0.2, 1.8]\) use the full integer grid. This matters for activations after ReLU/GELU, which are skewed and non-negative-ish.

Symmetric quantization¶

If the distribution is roughly symmetric about zero — which is true of most weight matrices — we drop the zero-point (\(z=0\)) and use a signed grid. For INT4 that is \([-8, 7]\); for INT8, \([-128, 127]\). The scale is set from the maximum absolute value:

Symmetric quantization is cheaper at inference time: dequantization is a single multiply \(\hat{x}=s\cdot q\) with no add, and the integer matmul does not need a zero-point correction term. Almost all modern LLM weight-quantization (GPTQ, AWQ) uses symmetric or near-symmetric per-group schemes.

Here is the whole idea in runnable code:

import torch

def quantize_symmetric(x: torch.Tensor, n_bits: int = 4):
    """Symmetric per-tensor quantization. Returns (codes, scale)."""
    qmax = 2 ** (n_bits - 1) - 1          # e.g. 7 for INT4, 127 for INT8
    qmin = -qmax                           # symmetric grid, drop the -8 slot for simplicity
    s = x.abs().max() / qmax               # one scalar scale for the whole tensor
    s = s.clamp(min=1e-8)                  # avoid divide-by-zero on all-zero tensors
    q = torch.clamp(torch.round(x / s), qmin, qmax)
    return q.to(torch.int8), s

def dequantize_symmetric(q: torch.Tensor, s: torch.Tensor):
    return q.to(torch.float32) * s

def quantize_affine(x: torch.Tensor, n_bits: int = 4):
    """Asymmetric (affine) quantization with a zero-point. Good for activations."""
    qmin, qmax = 0, 2 ** n_bits - 1        # unsigned grid: 0..15 for INT4
    xmin, xmax = x.min(), x.max()
    s = (xmax - xmin) / (qmax - qmin)
    s = s.clamp(min=1e-8)
    z = torch.round(qmin - xmin / s)        # the integer code that represents real 0.0
    q = torch.clamp(torch.round(x / s) + z, qmin, qmax)
    return q.to(torch.uint8), s, z

def dequantize_affine(q, s, z):
    return (q.to(torch.float32) - z) * s

# --- sanity check ---
w = torch.randn(4096) * 0.05               # a slice of a typical weight column
q, s = quantize_symmetric(w, n_bits=4)
err = (dequantize_symmetric(q, s) - w).abs().mean()
print(f"INT4 mean abs error: {err:.4e}  (scale={s:.4e})")

Why rounding is the enemy¶

The error introduced by round is the quantization error. For a uniform grid of step \(s\), if the rounding error were uniformly distributed on \([-s/2, s/2]\), its variance would be \(s^2/12\) — the classic quantization-noise formula. The take-away: error is proportional to the step size \(s\), which is proportional to the range of the tensor. One giant value in the tensor inflates \(\max|x|\), inflates \(s\), and corrupts the precision of every other value. This single observation drives the entire chapter.

Granularity: Per-Tensor, Per-Channel, Per-Group¶

We do not have to use one scale for an entire matrix. Finer granularity = more scales = better fidelity (each scale is fit to a smaller, more homogeneous slice) at the cost of a little more memory and bookkeeping.

• Per-tensor: a single \((s, z)\) for the matrix. Fastest kernels, but one outlier ruins everything. Common for activations where you want a cheap dynamic scale.
• Per-channel (per-row for weights): one scale per output channel. The matmul \(y = Wx\) produces output \(y_j = \sum_i W_{ji} x_i\); a per-row scale \(s_j\) factors out cleanly because the whole row shares it: \(y_j \approx s_j \sum_i q^{W}_{ji} x_i\).
• Per-group: split each row into contiguous groups of \(G\) weights (typically \(G=128\)) and give each group its own scale. This is the dominant scheme for 4-bit LLM weights (GPTQ, AWQ, GGUF all default to group size 128). It costs \(\frac{16}{G} = \frac{16}{128} = 0.125\) extra bits per weight to store the FP16 scales — negligible.

def quantize_per_group(W: torch.Tensor, n_bits=4, group_size=128):
    """Per-group symmetric quantization of a weight matrix [out, in]."""
    out_f, in_f = W.shape
    assert in_f % group_size == 0
    Wg = W.reshape(out_f, in_f // group_size, group_size)   # [out, n_groups, G]
    qmax = 2 ** (n_bits - 1) - 1
    s = Wg.abs().amax(dim=-1, keepdim=True) / qmax           # one scale per (row, group)
    s = s.clamp(min=1e-8)
    q = torch.clamp(torch.round(Wg / s), -qmax, qmax)
    return q.to(torch.int8), s, (out_f, in_f, group_size)

def dequantize_per_group(q, s, shape):
    out_f, in_f, G = shape
    W = (q.to(torch.float32) * s).reshape(out_f, in_f)
    return W

The Outlier Problem¶

Here is the fact that breaks naive quantization of large transformers. As models scale past roughly 6–7B parameters, a small number of feature dimensions in the activations develop enormous magnitudes — often 10x to 100x larger than the typical activation. These are activation outliers. They are not noise: they are systematic, they live in the same handful of channels across many tokens, and ablating them destroys the model’s accuracy. This phenomenon was documented carefully in the LLM.int8() paper (Dettmers et al., 2022).

Why are outliers fatal? Recall that per-tensor quantization error scales with \(\max|x|\). If one channel is 100x larger than the rest, the per-tensor scale \(s\) becomes 100x too coarse for the other 99% of values, and they all collapse toward zero. Consider quantizing the vector \([0.1, -0.2, 0.15, 70.0]\) to INT8 per-tensor:

The three normal values all round to zero. The single outlier ate the entire dynamic range.

Crucially, the outliers are predominantly in activations, not weights. Weight distributions are well-behaved and quantize beautifully to 4 bits with per-group scales. Activations are the hard part. This asymmetry explains the design space:

• Weight-only quantization (GPTQ, AWQ): keep activations in FP16, quantize only weights (to INT4/INT3). The matmul dequantizes weights on the fly. Great for memory-bound decoding, which is where LLM inference spends most of its time.
• Weight + activation quantization (SmoothQuant, LLM.int8()): quantize both so the matmul itself runs in INT8 on tensor cores. Needed for compute-bound prefill and maximum throughput, but you must tame the activation outliers first.

Weight-Only Quantization I: GPTQ¶

The simplest weight quantization is round-to-nearest (RTN): independently round each weight to its grid. RTN is shockingly good for INT8 and decent for INT4 with group size 128, but it leaves accuracy on the table because it ignores the interactions between weights. GPTQ (Frantar et al., GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers, 2022) does better by asking a sharper question.

The objective: minimize the output error, not the weight error¶

We do not actually care whether each quantized weight is close to the original. We care that the layer’s output is close. For a linear layer with weight row \(\mathbf{w}\) and a batch of calibration inputs \(X\) (shape \([\,d_{\text{in}}, N\,]\) of \(N\) tokens), we want a quantized \(\hat{\mathbf{w}}\) minimizing the reconstruction error:

This is the Optimal Brain Quantization formulation, descended from the classic Optimal Brain Surgeon pruning work. The key object is the Hessian of this least-squares loss:

\(H\) captures the correlations between input features. The crucial insight: when we quantize one weight and incur an error, we can compensate by nudging the not-yet-quantized weights to absorb that error, and the optimal nudge is dictated by \(H^{-1}\).

The GPTQ algorithm¶

OBS/OBQ says: quantize weights one at a time; after fixing weight \(i\) to its grid value, update all remaining weights to compensate. The optimal update when quantizing column \(i\) is

where \(w_i - q_i\) is the rounding error you just committed. GPTQ makes this practical at LLM scale with three engineering moves:

1. Same quantization order for all rows. OBQ picks a different greedy order per row; GPTQ proves that quantizing columns in a fixed left-to-right order is nearly as good, which lets all rows share one \(H^{-1}\) and turns the algorithm into batched matrix operations.
2. Cholesky reformulation. Repeated \(H^{-1}\) updates are numerically unstable. GPTQ pre-computes a Cholesky factorization of \(H^{-1}\) once and reads off the needed columns, which is both stable and fast.
3. Lazy block updates. Update remaining weights in blocks of ~128 columns to keep memory traffic cache-friendly.

A faithful-but-readable core of GPTQ:

import torch

def gptq_quantize_layer(W, H, n_bits=4, group_size=128, percdamp=0.01):
    """
    GPTQ for one linear layer.
      W : [out_features, in_features]  original weights (float)
      H : [in_features, in_features]   Hessian  = X X^T  over calibration tokens
    Returns the quantized-then-dequantized weight (ready to use in FP matmul,
    or to be packed into INT4). This mirrors the real algorithm's structure.
    """
    W = W.clone().float()
    out_f, in_f = W.shape
    qmax = 2 ** (n_bits - 1) - 1

    # 1) Dampen the Hessian diagonal for invertibility (some inputs are dead).
    dead = torch.diag(H) == 0
    H[dead, dead] = 1.0
    W[:, dead] = 0
    damp = percdamp * torch.mean(torch.diag(H))
    H[range(in_f), range(in_f)] += damp

    # 2) Cholesky of the INVERSE Hessian -> upper triangular factor we walk along.
    Hinv = torch.cholesky_inverse(torch.linalg.cholesky(H))
    Hinv = torch.linalg.cholesky(Hinv, upper=True)   # upper-tri Cholesky of H^{-1}

    Q = torch.zeros_like(W)
    Err = torch.zeros_like(W)

    # 3) Walk columns left-to-right; quantize, then propagate the error forward.
    for i in range(in_f):
        w = W[:, i].clone()                # current column across all output rows
        d = Hinv[i, i]                     # diagonal entry => error-scaling factor

        # per-group symmetric scale (recomputed at each group boundary)
        if i % group_size == 0:
            g = W[:, i:i + group_size]
            scale = g.abs().amax(dim=1, keepdim=True).clamp(min=1e-8) / qmax

        q = torch.clamp(torch.round(w / scale.squeeze(1)),
                        -qmax, qmax) * scale.squeeze(1)
        Q[:, i] = q

        err = (w - q) / d                  # the OBS-optimal scaled error
        # compensate every not-yet-quantized column j>i using H^{-1} row i.
        W[:, i:] -= err.unsqueeze(1) * Hinv[i, i:].unsqueeze(0)
        Err[:, i] = err

    return Q

The Hessian itself is accumulated during a forward pass over a calibration set (e.g. 128 sequences of 2048 tokens from C4 or WikiText):

def accumulate_hessian(layer, calibration_inputs):
    """H = sum over tokens of x x^T, normalized. x is the layer's input."""
    in_f = layer.weight.shape[1]
    H = torch.zeros(in_f, in_f)
    n = 0
    for X in calibration_inputs:           # X: [n_tokens, in_f]
        X = X.float()
        H += X.t() @ X                     # accumulate outer products
        n += X.shape[0]
    return (2.0 / n) * H

Where do those calibration_inputs come from? You do not have them lying around — you have to record the tensor each nn.Linear actually sees, with a forward pre-hook:

def capture_layer_inputs(model, linear, batches):
    """Record the inputs one nn.Linear sees over a calibration set."""
    store = []

    def hook(module, args):
        x = args[0].detach()                       # [batch, seq, in_f]
        store.append(x.reshape(-1, x.shape[-1]).float())   # -> [tokens, in_f]

    h = linear.register_forward_pre_hook(hook)
    with torch.no_grad():
        for input_ids in batches:                  # e.g. 128 seqs x 2048 tokens
            model(input_ids)
    h.remove()
    return store

Two practical details that real implementations get right and a naive one does not. First, work block by block, not model-wide: hold the hidden states entering transformer block \(\ell\), quantize every linear in that block, then re-run those hidden states through the already quantized block to produce the inputs for block \(\ell+1\). Later layers then compensate for the error earlier layers introduced — this is the true_sequential option in GPTQModel, and it is worth roughly a tenth of a perplexity point at INT4. Second, budget the Hessians: \(H\) is \(d_{\text{in}}\times d_{\text{in}}\) in FP32, so for \(d_{\text{in}}=4096\) that is 67 MB per linear — and the MLP down-projection, whose input is the much wider intermediate dimension, costs several times that again. Accumulate them one block at a time and free them before moving on.

GPTQ runs in minutes to a couple of hours for a 7B–70B model on a single GPU and reliably hits INT4 (and often INT3) with small perplexity loss. Its weakness: it is sequential per layer and Hessian-heavy, and it can struggle on the very narrowest bit-widths without a group size.

Weight-Only Quantization II: AWQ¶

AWQ — Activation-aware Weight Quantization (Lin et al., 2023) — starts from a different observation: not all weights are equally important, and the important ones are revealed by the activations. If a weight column multiplies an input channel that frequently carries large activations, errors in that column get amplified into the output. AWQ identifies these salient weight channels (typically the ~1% aligned with activation outliers) and protects them.

The scaling trick¶

The naive way to protect salient weights is mixed precision (keep 1% in FP16), but mixed-precision INT4/FP16 matmuls are kernel-unfriendly. AWQ’s elegant alternative: keep everything INT4, but rescale. Consider a single weight \(w\) multiplied by input \(x\): \(y = w \cdot x\). We can scale the weight up by \(\alpha > 1\) and the input down by the same factor without changing the product:

Quantizing \(w\cdot\alpha\) instead of \(w\) shrinks its relative rounding error by roughly \(\alpha\), because the error is \(s/2\) in absolute terms but the value is now \(\alpha\) times bigger. The cost is that the input \(x/\alpha\) must be rescaled — but \(x\) stays in FP16 (weight-only), so we can fold \(1/\alpha\) into the previous layer’s normalization weights for free. AWQ searches for a per-input-channel scale vector \(\mathbf{s}\) that minimizes the layer output error, using a simple grid search over a hyperparameter that interpolates between “scale by activation magnitude” and “scale by weight magnitude”:

import torch

def awq_search_scales(W, X, n_bits=4, group_size=128, grid=20):
    """
    AWQ per-input-channel scale search for one linear layer.
      W : [out, in]   weights
      X : [n_tokens, in]   calibration activations into this layer
    Returns scale vector s of shape [in]; you then quantize (W * s) and fold
    (1/s) into the upstream layernorm / previous linear.
    """
    W = W.float()
    x_mag = X.abs().float().mean(dim=0)          # [in]  per-channel activation scale
    w_mag = W.abs().mean(dim=0).clamp(min=1e-6)  # [in]  per-channel weight scale

    def quant_dequant(w):                         # group-wise INT4 RTN
        out, inf = w.shape
        wg = w.reshape(out, inf // group_size, group_size)
        qmax = 2 ** (n_bits - 1) - 1
        s = wg.abs().amax(-1, keepdim=True).clamp(min=1e-8) / qmax
        q = torch.clamp(torch.round(wg / s), -qmax, qmax) * s
        return q.reshape(out, inf)

    ref = X.float() @ W.t()                       # FP16 reference output
    best_err, best_s = float("inf"), torch.ones_like(x_mag)
    for i in range(grid):
        beta = i / grid                           # interpolate importance exponent
        s = (x_mag ** beta) / (w_mag ** (1 - beta))
        s = s.clamp(min=1e-4)
        # Normalize by the geometric mean of the extremes (as the official AWQ
        # code does) so the scales straddle 1.0 instead of all shrinking.
        s = s / (s.max() * s.min()).sqrt()
        Wq = quant_dequant(W * s)                 # quantize the *scaled* weights
        out = (X.float() / s) @ Wq.t()            # input rescaled by 1/s
        err = (out - ref).pow(2).mean().item()
        if err < best_err:
            best_err, best_s = err, s
    return best_s

AWQ is faster than GPTQ (no Hessian inverse, no sequential per-column loop — just a grid search over a scale vector), has no overfitting to the calibration set (it never solves a least-squares problem against specific tokens, only a magnitude statistic), and tends to generalize better across domains. In practice GPTQ and AWQ are both excellent at INT4-g128, and their kernels remain popular in vllm/sglang serving stacks — though by 2026 the original standalone AutoGPTQ and AutoAWQ libraries have both been retired in favor of actively-maintained successors: GPTQModel for broad multi-backend GPTQ/AWQ support, and the vLLM Project’s llm-compressor for producing GPTQ/AWQ/SmoothQuant checkpoints tuned for vLLM deployment.

Activation Quantization: SmoothQuant & LLM.int8()¶

Weight-only quantization halves or quarters memory but the matmul still runs in FP16. To use INT8 tensor cores (2x–4x the FLOPs of FP16 on many GPUs) we must quantize activations too — and that means facing the outliers head-on. Two complementary answers:

SmoothQuant: migrate the difficulty¶

SmoothQuant (Xiao et al., 2022) makes a beautiful observation: activations have outliers and are hard to quantize; weights are smooth and easy. So move some of the difficulty from activations into weights using the same scale-invariance AWQ exploits. For \(y = X W\), insert a per-channel diagonal scale \(\operatorname{diag}(\mathbf{s})\):

Choose \(\mathbf{s}\) so the smoothed activation \(\hat{X}\) has no outliers and the smoothed weight \(\hat{W}\) is still quantizable. The per-channel scale that balances the two difficulties is:

The migration strength \(\alpha\) trades off: \(\alpha=1\) pushes all difficulty into weights, \(\alpha=0\) leaves it in activations, \(\alpha=0.5\) splits it. Because \(\operatorname{diag}(\mathbf{s})^{-1}\) multiplies \(X\) and \(X\) is the output of a LayerNorm/RMSNorm, the scale folds into the norm’s affine weights at zero runtime cost — exactly like AWQ. After smoothing, both \(\hat{X}\) and \(\hat{W}\) quantize cleanly to INT8, and the matmul runs entirely on INT8 tensor cores with per-tensor or per-channel scales.

import torch

@torch.no_grad()
def smoothquant_scales(act_abs_max, weight, alpha=0.5):
    """
    Compute the SmoothQuant per-channel migration scale.
      act_abs_max : [in]   per-channel max |activation| from calibration
      weight      : [out, in]
    Returns s of shape [in]: divide activations by s, multiply weight cols by s.
    """
    w_abs_max = weight.abs().amax(dim=0).clamp(min=1e-5)   # [in]
    a_abs_max = act_abs_max.clamp(min=1e-5)
    s = (a_abs_max.pow(alpha) / w_abs_max.pow(1 - alpha)).clamp(min=1e-5)
    return s

@torch.no_grad()
def apply_smoothing(weight, ln_weight, s):
    """Fold 1/s into the upstream norm, s into this linear's weight columns."""
    ln_weight = ln_weight / s          # activations get divided by s for free
    weight = weight * s.unsqueeze(0)   # weight columns multiplied by s
    return weight, ln_weight

LLM.int8(): keep the outliers in FP16¶

LLM.int8() (Dettmers et al., 2022) takes the bluntest possible approach and it works: decompose the matmul. Detect the outlier feature dimensions at runtime (channels whose magnitude exceeds a threshold, e.g. 6.0). Run the ~99.9% of “normal” dimensions through a fast INT8 matmul, and run the handful of outlier columns through a small FP16 matmul. Sum the two. Mathematically, splitting the input feature set into outlier indices \(O\) and the rest \(R\):

where the first term is INT8 and the second is FP16. Because \(|O|\) is tiny (often a few dozen channels out of thousands), the FP16 part is cheap, and the INT8 part captures the bulk of the FLOPs with no accuracy loss on the well-behaved channels.

import torch

@torch.no_grad()
def llm_int8_matmul(X, W, threshold=6.0):
    """
    Mixed INT8 / FP16 matmul, the LLM.int8() decomposition.
      X : [tokens, in]   activations (fp16)
      W : [in, out]      weights (fp16)
    """
    # 1) find outlier feature columns by magnitude across the batch
    col_max = X.abs().amax(dim=0)                # [in]
    outlier = col_max > threshold               # boolean mask
    reg = ~outlier

    # 2) FP16 path for the few outlier dimensions (exact, no quant error)
    y_fp16 = X[:, outlier] @ W[outlier, :]

    # 3) INT8 path for the regular dimensions
    Xr, Wr = X[:, reg], W[reg, :]
    sx = Xr.abs().amax(dim=1, keepdim=True).clamp(min=1e-8) / 127   # per-row act scale
    sw = Wr.abs().amax(dim=0, keepdim=True).clamp(min=1e-8) / 127   # per-col wgt scale
    Xq = torch.clamp(torch.round(Xr / sx), -127, 127).to(torch.int8)
    Wq = torch.clamp(torch.round(Wr / sw), -127, 127).to(torch.int8)
    acc = (Xq.float() @ Wq.float())             # stands in for an INT8 tensor-core matmul
    y_int8 = acc * sx * sw                       # dequantize with outer-product of scales

    return y_int8 + y_fp16

LLM.int8() is zero-tuning — no calibration, no Hessian, no scale search — and is the default behind load_in_8bit=True in bitsandbytes. Its downside is latency: the dynamic outlier detection and the split matmul add overhead, so it is better for fitting a model in memory than for maximum throughput. SmoothQuant, by contrast, is a static transform that yields a clean INT8-everywhere model with fast kernels.

Rotation: making the outliers disappear¶

SmoothQuant moves the outliers and LLM.int8() routes around them. The third answer — the basis of essentially every W4A4 and FP4 pipeline in 2026 — is to rotate them away. Insert an orthogonal matrix and its inverse into the matmul, which changes nothing mathematically:

Choose \(Q\) to be a random-sign Hadamard matrix (\(Q = \tfrac{1}{\sqrt d} H D\), applicable in \(O(d\log d)\) by the fast Hadamard transform) and every output coordinate becomes an equally-weighted \(\pm\) mix of all \(d\) input coordinates. A single channel that was 100x the others no longer has anywhere to hide: after rotation the per-channel magnitudes are near-Gaussian and no coordinate is much larger than the vector’s RMS. This is incoherence processing, introduced for weight quantization in QuIP and used by FlashAttention-3 to make FP8 attention viable (see FlashAttention 2 & 3).

# Why rotation kills an outlier: a Hadamard mix spreads it over all d coords.
import torch
d = 1024
H = torch.tensor([[1.0]])
while H.shape[0] < d:                                  # Sylvester construction
    H = torch.cat([torch.cat([H,  H], 1),
                   torch.cat([H, -H], 1)], 0)
Q = (H / d ** 0.5) * torch.randint(0, 2, (d,)).mul(2).sub(1).float()  # random signs
x = torch.randn(64, d); x[:, 17] *= 60.0               # one outlier channel
print(f"max|x| before rotation: {x.abs().max():.2f}")
print(f"max|x| after  rotation: {(x @ Q).abs().max():.2f}")   # far smaller

QuaRot (Ashkboos et al., 2024) and SpinQuant (which learns the rotation instead of sampling it) fold these rotations into the checkpoint wherever the residual stream is linear, so most of them cost nothing at runtime. Two structural conditions make that folding legal, and they are worth internalizing: the normalization must be RMSNorm without mean subtraction (a rotation commutes with scaling by \(\|x\|\) but not with subtracting a mean), and the norm’s learned \(\gamma\) must first be folded into the following linear so what remains is a pure scalar normalization. What cannot be folded — typically one rotation on the attention output and one on the MLP down-projection input — runs as an online fast Hadamard kernel costing a few percent of layer time. The payoff is large: rotated models quantize to W4A4 including the KV cache with roughly a point or less of perplexity loss, where SmoothQuant-style W8A8 was previously the floor. The same trick is what makes 4-bit float formats (NVFP4/MXFP4) usable; those formats and their kernels live in Quantization II.

Putting It Together: A Decision Guide¶

The shorthand W4A16 means 4-bit weights, 16-bit activations (GPTQ/AWQ territory); W8A8 means 8-bit weights and activations (SmoothQuant territory). A practical pipeline:

1. Pick a target: W4A16 for single-GPU decoding, W8A8 for throughput serving.
2. Gather ~128 calibration sequences from in-domain data.
3. Run GPTQ or AWQ (weight-only) or SmoothQuant+per-tensor INT8 (W8A8).
4. Evaluate: perplexity on held-out text and at least one task eval. A 0.1–0.3 perplexity rise is normal and usually invisible downstream; a >1.0 rise or a benchmark cliff means revisit group size, calibration data, or keep more layers in higher precision (the first and last layers, and lm_head, are often kept in FP16).

Doing it with real libraries¶

You should understand the algorithms above; you should almost never re-implement them for production. The 2026 default path is llm-compressor, which applies PTQ as a one-shot recipe and writes a compressed-tensors checkpoint that vLLM and SGLang load directly:

from datasets import load_dataset
from transformers import AutoModelForCausalLM, AutoTokenizer
from llmcompressor import oneshot          # older releases: llmcompressor.transformers
from llmcompressor.modifiers.quantization import GPTQModifier
from llmcompressor.modifiers.smoothquant import SmoothQuantModifier

MODEL_ID = "meta-llama/Llama-3.1-8B-Instruct"
model = AutoModelForCausalLM.from_pretrained(MODEL_ID, torch_dtype="auto", device_map="auto")
tok = AutoTokenizer.from_pretrained(MODEL_ID)

# Calibration: a few hundred in-domain sequences, tokenized like deployment.
ds = load_dataset("HuggingFaceH4/ultrachat_200k", split="train_sft").shuffle(seed=0).select(range(512))
ds = ds.map(lambda ex: tok(tok.apply_chat_template(ex["messages"], tokenize=False),
                          truncation=True, max_length=2048))

# W4A16 (GPTQ, group 128) — the memory-bound-decode target.
recipe = GPTQModifier(targets="Linear", scheme="W4A16", ignore=["lm_head"])

# W8A8 instead? Compose SmoothQuant *before* the quantizer, exactly as in the paper:
# recipe = [SmoothQuantModifier(smoothing_strength=0.8),
#           GPTQModifier(targets="Linear", scheme="W8A8", ignore=["lm_head"])]

oneshot(model=model, dataset=ds, recipe=recipe,
        max_seq_length=2048, num_calibration_samples=512)
model.save_pretrained("Llama-3.1-8B-Instruct-W4A16", save_compressed=True)
tok.save_pretrained("Llama-3.1-8B-Instruct-W4A16")

Notice how the API mirrors the theory: ignore=["lm_head"] is the “keep the output layer high-precision” rule; SmoothQuantModifier(smoothing_strength=0.8) is the migration strength \(\alpha\); swapping GPTQModifier for AWQModifier (from llmcompressor.modifiers.awq) swaps the algorithm without touching the pipeline. Serving needs no flags — vLLM reads the quantization config out of the checkpoint:

vllm serve ./Llama-3.1-8B-Instruct-W4A16   # compressed-tensors is auto-detected

The alternative front-end is GPTQModel, the maintained successor to AutoGPTQ (and, per its migration guide, to AutoAWQ), which emits classic gptq/awq checkpoints and ships Marlin/BitBLAS kernels:

from gptqmodel import GPTQModel, QuantizeConfig

cfg = QuantizeConfig(bits=4, group_size=128)          # the INT4-g128 default
model = GPTQModel.load("meta-llama/Llama-3.1-8B-Instruct", cfg)
model.quantize(calibration_dataset)                    # list of tokenized dicts
model.save("Llama-3.1-8B-Instruct-gptq-4bit")

And for the zero-calibration path, bitsandbytes needs no separate step at all — AutoModelForCausalLM.from_pretrained(..., quantization_config=BitsAndBytesConfig(load_in_8bit=True)) gives you LLM.int8() at load time (details, plus the 4-bit NF4 variant, in Quantization II).

To watch this run end to end on a model you built yourself — RTN INT8 and INT4-per-group applied to every nn.Linear of a 100M-parameter model, a QuantizedLinear that unpacks and dequantizes on the fly, and the honest before/after eval deltas on a laptop CPU — see Evaluation & Serving: Honest Benchmarks, int4 Quantization, and Running on a Laptop; the GPTQ and AWQ ideas from this chapter are the upgrade path it measures itself against. For the actual file formats, packing, and how these integrate with bitsandbytes, GGUF, and quantization-aware training, continue to Quantization II. For how quantized models slot into serving systems, see vLLM: Architecture, PagedAttention & Internals; the floating-point background lives in Numerical Computing, Floating Point & Precision, and quantization for training in Mixed Precision, bf16 & FP8 Training.

Further reading¶

• Frantar, Ashkboos, Hoefler, Alistarh — GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers (2022). The Hessian-based, Cholesky-stabilized method; see also the gptq and AutoGPTQ repositories.
• Lin, Tang, Tang, et al. — AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration (2023). The salience/scaling idea; the llm-awq repository.
• Xiao, Lin, Seznec, Wu, Demouth, Han — SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models (2022). Migrating difficulty for W8A8.
• Dettmers, Lewis, Belkada, Zettlemoyer — LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale (2022). Outlier features and mixed-precision decomposition; implemented in bitsandbytes.
• Ashkboos et al. — QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs (2024). Hadamard rotation as the third answer to outliers; the spcl/QuaRot repository.
• Frantar & Alistarh — Optimal Brain Compression / Optimal Brain Quantization (the OBS/OBQ lineage GPTQ builds on).
• Nagel et al. — A White Paper on Neural Network Quantization (Qualcomm AI Research). An excellent, rigorous primer on scales, zero-points, granularity, and PTQ vs QAT.

Exercises¶

1. (Quantitative — by hand.) Quantize the weight vector \(\mathbf{w} = [\,0.02,\ -0.05,\ 0.08,\ -0.03\,]\) to symmetric INT4 using the chapter’s convention \(q_{\max}=2^{b-1}-1=7\), \(q_{\min}=-7\). Compute the scale \(s\), the integer codes, the dequantized values \(\hat{w}\), and the mean absolute error. Which value is quantized exactly, and why?

2. (Conceptual + small calc.) The chapter shows that quantizing \([\,0.1,\,-0.2,\,0.15,\,70.0\,]\) per-tensor to INT8 collapses the three small values to zero. (a) Explain in one or two sentences why a single outlier does this. (b) Now split the vector into two per-group groups of size 2, \([\,0.1,\,-0.2\,]\) and \([\,0.15,\,70.0\,]\), and quantize each group symmetrically to INT8 (\(q_{\max}=127\)). Show that the small values survive in the first group. © What does this tell you about why weights quantize well to INT4-g128 but per-tensor activation quantization does not?

3. (Quantitative.) You have a 7B-parameter model in FP16. (a) Compute the weight memory in FP16. (b) Compute the weight memory after INT4 with group size 128, including the FP16 group scales. © What is the compression ratio? (d) If HBM bandwidth is 2 TB/s and decoding is bandwidth-bound (stream all weights once per token), estimate the per-token weight-load time before and after, and the speedup.

4. (Conceptual + calc — the AWQ scaling trick.) A salient input channel carries a weight \(w = 0.02\) that sits inside a group whose max-absolute weight is \(0.08\) (dominated by other channels), so the group’s INT4 step is \(s = 0.08/7 \approx 0.0114\). (a) What is the worst-case relative rounding error on \(w\)? (b) AWQ multiplies this channel’s weight by \(\alpha = 3\) (and divides the corresponding activation by 3, folded upstream). Assuming the scaled weight \(0.06\) is still below the group max \(0.08\) so \(s\) is unchanged, what is the new worst-case relative error? © State the general rule and the one condition under which it can break.

5. (Implementation.) Implement SmoothQuant’s per-channel migration and empirically verify its two defining properties on a synthetic layer with an injected activation outlier: (i) the transform is mathematically exact — \(\hat{X}\hat{W}^\top = XW^\top\) — and (ii) it reduces the activation’s max-absolute value. Use \(\alpha = 0.5\).

import torch
torch.manual_seed(0)

X = torch.randn(32, 8)            # [tokens, in]  activations
X[:, 3] *= 40.0                   # inject one outlier channel
W = torch.randn(16, 8) * 0.05     # [out, in]  weights

@torch.no_grad()
def smoothquant_scales(act_abs_max, weight, alpha=0.5):
    w_abs_max = weight.abs().amax(dim=0).clamp(min=1e-5)   # [in]
    a_abs_max = act_abs_max.clamp(min=1e-5)
    return (a_abs_max.pow(alpha) / w_abs_max.pow(1 - alpha)).clamp(min=1e-5)

a_max = X.abs().amax(dim=0)              # [in]  per-channel act max
s = smoothquant_scales(a_max, W, alpha=0.5)

Xhat = X / s                             # activations divided by s
What = W * s                             # weight columns multiplied by s

# (i) exact invariance: Xhat @ What.t() == X @ W.t()
err = (Xhat @ What.t() - X @ W.t()).abs().max().item()
print(f"max reconstruction error: {err:.2e}")   # ~1e-6 (float noise)

# (ii) outlier tamed
print(f"activation max before: {X.abs().amax(0).max().item():8.3f}")
print(f"activation max after : {Xhat.abs().amax(0).max().item():8.3f}")

6. (Implementation — hard.) Empirically demonstrate GPTQ’s core claim: that Hessian-based error compensation yields lower layer-output error than plain round-to-nearest (RTN) when the calibration inputs are correlated (a non-diagonal Hessian). Build a synthetic layer with correlated inputs, quantize it both ways, and compare \(\|WX^\top - \hat{W}X^\top\|_2^2\).

import torch
torch.manual_seed(0)

in_f, out_f, N = 256, 128, 512
A = torch.randn(in_f, in_f)              # mixing matrix -> correlated features
X = torch.randn(N, in_f) @ A.t()         # [tokens, in]  correlated calibration
W = torch.randn(out_f, in_f) * 0.05      # [out, in]

H = (2.0 / N) * (X.t() @ X)              # input Hessian, non-diagonal

def rtn(W, n_bits=4, group_size=128):
    out, inf = W.shape
    wg = W.reshape(out, inf // group_size, group_size)
    qmax = 2 ** (n_bits - 1) - 1
    s = wg.abs().amax(-1, keepdim=True).clamp(min=1e-8) / qmax
    return (torch.clamp(torch.round(wg / s), -qmax, qmax) * s).reshape(out, inf)

# gptq_quantize_layer(W, H, ...) is the function defined earlier in this chapter.
Wq_rtn  = rtn(W.clone())
Wq_gptq = gptq_quantize_layer(W.clone(), H.clone(), n_bits=4, group_size=128)

ref = W @ X.t()                          # FP16 reference output [out, tokens]
e_rtn  = (Wq_rtn  @ X.t() - ref).pow(2).mean().item()
e_gptq = (Wq_gptq @ X.t() - ref).pow(2).mean().item()
print(f"RTN  output MSE: {e_rtn:.4e}")
print(f"GPTQ output MSE: {e_gptq:.4e}")
print(f"GPTQ is {e_rtn / e_gptq:.2f}x lower")