7.12 Inference Economics: Latency, Throughput & Cost¶

Every LLM serving system is a negotiation between three forces: latency (how fast a single request finishes), throughput (how many requests the system handles per second), and cost (how much you pay per token produced). These three forces are coupled in fundamental ways — you cannot optimize all three simultaneously, and understanding the trade-offs quantitatively is what separates engineers who ship sustainable inference systems from those who run out of GPU budget six weeks after launch.

This chapter is the capstone of Part VII. We assume you have read about the mechanics of prefill and decode in The Anatomy of LLM Inference: Prefill, Decode & The KV Cache, continuous batching in Continuous Batching & Request Scheduling, and the memory machinery in PagedAttention & KV-Cache Memory Management. Here we focus on the economics: the math of dollars per million tokens, the hardware selection calculus, and the operational playbook for keeping costs under control.

The Latency–Throughput–Cost Triangle¶

The fundamental constraint is that a GPU can do roughly a fixed number of floating-point operations per second (FLOP/s) and move a fixed number of bytes per second across its memory bus (bandwidth). Every inference workload maps to a point on the roofline (see The Roofline Model & Performance Engineering):

Decode is almost always memory-bandwidth-bound: for a batch of size \(B\) generating one token, we read all model weights once (roughly \(2P\) bytes for an \(FP16\) model with \(P\) parameters) but do only \(2PB\) FLOPs — arithmetic intensity is \(B\). An H100 has roughly 3,350 GB/s of HBM bandwidth and ~989 TFLOP/s of BF16 MMA throughput. The breakeven arithmetic intensity is:

Until batch size exceeds ~295, decode is bandwidth-bound, and adding more arithmetic units does not help — you are just waiting for weights to stream from HBM.

Prefill, by contrast, is compute-bound at large sequence lengths (the attention FLOPs scale as \(O(L^2)\), eventually dominating weight-loading). This asymmetry drives the disaggregated prefill/decode architectures described in Disaggregated Prefill/Decode & Chunked Prefill.

The Three-Way Trade-off Stated Clearly¶

There is no free lunch. Your job is to find the operating point that satisfies the latency SLO while maximizing GPU utilization — because utilization is the primary driver of cost efficiency.

The Math of Dollars Per Million Tokens¶

Let’s derive the cost formula from scratch and then plug in real numbers.

GPU Rental Cost¶

Suppose you rent a node with \(G\) GPUs at a price of \(\$R\) per hour. The node delivers \(T_{\text{sustained}}\) output tokens per second (measured at your operating batch size). In one hour you produce:

The cost per token is therefore:

And per million tokens:

Measuring \(T_{\text{sustained}}\) Instead of Guessing It¶

Every number in the formula above except \(T_{\text{sustained}}\) is known exactly: you read \(R\) off your cloud invoice. \(T_{\text{sustained}}\) is the one term you must measure, at your own request-rate, prompt-length and output-length distribution — analytical models like the ones in this chapter set expectations, they do not replace a load test. Both production serving engines ship a load generator for exactly this.

# 1. Start the server (vLLM). Two flags here move the cost needle directly:
#    --gpu-memory-utilization raises the HBM given to KV cache (larger B_eff),
#    --enable-prefix-caching stops you re-paying for shared system prompts.
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.92 \
  --enable-prefix-caching

# 2. Drive it with a realistic open-loop arrival process and measure.
#    'vllm bench serve' is the modern CLI; older releases ship the same tool
#    as benchmarks/benchmark_serving.py in the repo.
#    --request-rate is in requests/s; sweep it to trace the SLO frontier.
vllm bench serve \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --dataset-name random \
  --random-input-len 1024 --random-output-len 256 \
  --request-rate 8 \
  --num-prompts 500

# SGLang exposes an equivalent harness:
#   python -m sglang.bench_serving --backend sglang --num-prompts 500 --request-rate 8

The report gives you output-token throughput plus mean/median/p99 TTFT, TPOT (time per output token) and ITL (inter-token latency). Sweep --request-rate upward and you trace the SLO frontier: throughput climbs, then TTFT and TPOT cross your SLO. The last rate that still satisfies the SLO is your goodput, and the \(T_{\text{sustained}}\) you should put in the cost formula is the output-token rate at that point — not the unconstrained maximum, which corresponds to an operating point no user would tolerate. This distinction is the central argument of the DistServe paper cited at the end of the chapter.

Input Tokens vs. Output Tokens¶

Output tokens are expensive because they require an autoregressive decode step — one weight-loading pass per token. Input (prefill) tokens are cheap relative to output because they are processed in parallel. As a rough rule of thumb, at large batch sizes and typical prompt/completion ratios, output tokens cost roughly 3–5× more in wall-clock GPU-time — and hence dollars — per token than input tokens (the exact ratio depends on sequence lengths and batch size). Note the asymmetry is not about arithmetic: the FLOPs per token are essentially identical (~2P, where P is the parameter count) for a prefill token and a decode token. It is about bandwidth. A decode step reads the entire weight matrix from HBM to emit a single token, so it is memory-bound and poorly utilized; parallel prefill reads those same weights once and amortizes them across every prompt token at near-peak FLOP utilization. Output tokens are expensive because each one pays for its own weight-loading pass, not because it does more math.

Public API providers charge differently for input and output tokens for exactly this reason. When optimizing your prompt, shortening the completion (e.g., via structured generation or speculative decoding — see Speculative Decoding: Draft Models, Medusa, EAGLE & Lookahead) has a higher ROI than shortening the prompt.

Batching vs. Latency SLOs¶

Batching is the primary mechanism by which you trade latency for efficiency. Understanding the math helps you set the right operating point.

Continuous Batching and Effective Batch Size¶

With continuous batching (see Continuous Batching & Request Scheduling), the effective batch size at any instant is the number of sequences that are currently in the decode phase. Call this \(B_{\text{eff}}\). The decode throughput scales approximately linearly with \(B_{\text{eff}}\) until the arithmetic intensity breakeven, after which it saturates at compute capacity.

For a model with \(P\) parameters in FP16/BF16, the time to generate one token for a batch of size \(B\) is approximately:

where BW is HBM bandwidth. The first term is the bandwidth-bound floor (weight streaming), and the second is the compute-bound ceiling. The knee of the curve — where bandwidth and compute balance — is at \(B^* = \text{FLOP/s} / \text{BW}\) (the arithmetic intensity breakeven from above).

Latency SLOs in Practice¶

Typical production SLOs take two forms: - TTFT (time-to-first-token): often 200–500 ms for interactive chat. - TBT (time-between-tokens): often 30–80 ms/token for a streaming UX that feels “fast” (roughly 12–33 tokens/second perceived). - P99 end-to-end latency for a fixed-length response.

A key insight is that batching affects TBT but not TTFT in the same way. TTFT is dominated by prefill time (which scales with prompt length and is bounded by compute), whereas TBT is dominated by decode speed (which scales with batch size up to the bandwidth floor).

The scheduler must balance these. The simplest heuristic: saturate batch up to \(B^*\), which gives maximum throughput without exceeding the compute-bound regime. Beyond \(B^*\) you pay in latency without gaining more tokens per dollar.

# Illustrative scheduler: fill batch up to the bandwidth breakeven point.
# This is a simplified model; real systems use token budgets and KV cache limits.

import math

def bandwidth_breakeven(flops_per_sec: float, bandwidth_bytes_per_sec: float) -> int:
    """
    Return the batch size B* at which decode transitions from bandwidth-bound
    to compute-bound. Below B*, adding more sequences to the batch is free in
    terms of time (you're already waiting for weights to stream from HBM).
    Above B*, decode time grows linearly with batch size.

    Args:
        flops_per_sec: Peak BF16 FLOP/s (e.g., 989e12 for H100 SXM5)
        bandwidth_bytes_per_sec: HBM bandwidth in bytes/s (e.g., 3.35e12 for H100)

    Returns:
        B*: arithmetic intensity breakeven batch size
    """
    return int(flops_per_sec / bandwidth_bytes_per_sec)


def decode_step_time_ms(
    n_params: int,
    batch_size: int,
    flops_per_sec: float,
    bandwidth_bytes_per_sec: float,
    bytes_per_param: int = 2,  # BF16 / FP16
) -> float:
    """
    Estimate the wall-clock time for one decode step (one new token per sequence).

    The model weights must be streamed from HBM once per step regardless of batch
    size (bandwidth-bound regime). At large batch sizes the MMA units become the
    bottleneck (compute-bound regime).
    """
    # Bytes read: all parameters once (both weight read and result write)
    bytes_read = n_params * bytes_per_param
    # FLOPs: 2 MACs per parameter per batch element
    flops = 2 * n_params * batch_size

    # Time in each regime (seconds)
    t_bandwidth = bytes_read / bandwidth_bytes_per_sec
    t_compute   = flops / flops_per_sec

    # Actual time is the max; report in ms
    return max(t_bandwidth, t_compute) * 1000.0


# --- Hardware constants ---
H100_FLOPS = 989e12      # BF16 tensor core FLOP/s (H100 SXM5)
H100_BW    = 3.35e12     # HBM bandwidth bytes/s

# --- Model: Llama 3 70B (70e9 params, BF16) ---
N_PARAMS = 70e9

B_STAR = bandwidth_breakeven(H100_FLOPS, H100_BW)
print(f"H100 arithmetic intensity breakeven batch size: {B_STAR}")

for batch in [1, 4, 16, 64, 128, 256, B_STAR]:
    t = decode_step_time_ms(N_PARAMS, batch, H100_FLOPS, H100_BW)
    regime = "bandwidth-bound" if batch <= B_STAR else "compute-bound"
    print(f"  batch={batch:4d}  step_time={t:.2f} ms  regime={regime}")

Running this produces output along the lines of:

H100 arithmetic intensity breakeven batch size: 295
  batch=   1  step_time=41.79 ms  regime=bandwidth-bound
  batch=   4  step_time=41.79 ms  regime=bandwidth-bound
  batch=  16  step_time=41.79 ms  regime=bandwidth-bound
  batch=  64  step_time=41.79 ms  regime=bandwidth-bound
  batch= 128  step_time=41.79 ms  regime=bandwidth-bound
  batch= 256  step_time=41.79 ms  regime=bandwidth-bound
  batch= 295  step_time=41.79 ms  regime=bandwidth-bound

Below \(B^* \approx 295\) the decode step time is flat at ~42 ms per step (bandwidth-bound floor). Each new request added below this threshold contributes zero extra latency but produces one more output token — a pure win. Above this, each additional request adds proportional latency.

The practical lesson: on an H100-class bandwidth budget serving a 70B BF16 model at negligible context length, you can pack up to ~295 concurrent decoding sequences before latency starts climbing. That is the regime where tokens/s/GPU is maximized. But this model has a missing term, and at realistic context lengths that term dominates everything else in this chapter.

The Missing Term: KV-Cache Reads¶

The model above counts only the bytes of weights streamed per decode step. It ignores the fact that attention must also read every active sequence’s KV cache from HBM, every single step. Weights are shared across the batch; KV caches are not. The honest bandwidth-bound decode model is:

where \(\text{kv}(S) = 2\, n_{\text{kv}}\, d\, L\, S \cdot b\) bytes is the per-sequence KV cache (the formula derived later in this chapter, with \(b\) bytes per element) and \(F_{\text{attn}}(S) = 4\, n_q\, d\, L\, S\) FLOPs is the per-sequence attention work — two matmuls (\(QK^\top\) and \(AV\)), each \(2 n_q d L S\) FLOPs across all \(n_q\) query heads and \(L\) layers.

Now compute the arithmetic intensity of the attention part on its own, in BF16 (\(b = 2\)):

For Llama-3 70B (\(n_q = 64\) query heads, \(n_{\text{kv}} = 8\) GQA key/value heads) that is exactly 8 — independent of \(S\), and, crucially, independent of \(B\). Batching amortizes weight reads across sequences, but it does not amortize KV reads, because every sequence carries its own cache. The attention component of decode is therefore pinned at an arithmetic intensity of 8, hopelessly below the H100’s \(\text{AI}^\ast \approx 295\), no matter how well you batch. This is the real reason MQA, GQA and MLA exist (see Multi-Head Attention, MQA, GQA & MLA): the only levers on that ratio are shrinking \(n_{\text{kv}}\) or, as in MLA, compressing the cache into a low-rank latent.

Three consequences follow, and they overturn the naive picture:

1. The compute-bound knee never arrives. Setting the two branches equal and solving for \(B\) yields a negative root for any \(S > 0\) on H100/70B: the KV term in the bandwidth branch grows faster in \(B\) than the compute branch does. Decode does not transition from bandwidth-bound to compute-bound; it transitions from weight-bandwidth-bound to KV-bandwidth-bound.
2. Batching stops being free at \(B_{1/2} = 2P / \text{kv}(S)\) — the batch size at which KV reads equal weight reads and step time has already doubled. This, not \(B^\ast\), is the number your scheduler actually collides with.
3. Per-node decode throughput has a hard ceiling of \(\text{BW} / \text{kv}(S)\) tokens/s as \(B \to \infty\). No amount of batching beats it, because in the limit every step is pure KV streaming.

At 32k context the useful batch size is not 295 but roughly a dozen, and node throughput saturates around 300 tokens/s — a 16× collapse in tokens/s/dollar relative to the short-context regime, purely from KV traffic. (Tensor parallelism splits both weights and KV heads across GPUs, so \(B_{1/2}\) is unchanged while the ceiling scales with the group’s aggregate bandwidth.)

# Extend the decode model with the KV-cache read term and attention FLOPs.
# Reuses H100_FLOPS, H100_BW and N_PARAMS from the block above.

def kv_bytes_per_seq(seq_len, n_kv=8, head_dim=128, n_layers=80, bytes_per_elt=2):
    """K and V, for every layer, for every cached token. Llama-3 70B defaults."""
    return 2 * n_kv * head_dim * n_layers * seq_len * bytes_per_elt


def attn_flops_per_seq(seq_len, n_q=64, head_dim=128, n_layers=80):
    """Decode attention: two matmuls (QK^T and AV), 2 FLOPs per MAC, all query heads."""
    return 2 * (2 * n_q * head_dim * n_layers * seq_len)


def decode_step_time_ms_v2(n_params, batch_size, seq_len,
                           flops_per_sec, bandwidth_bytes_per_sec,
                           bytes_per_param=2):
    """Decode step time including per-sequence KV-cache traffic.

    Weights are read once per step for the whole batch; KV caches are read
    once per step *per sequence*. That asymmetry is the whole story.
    """
    bytes_moved = n_params * bytes_per_param + batch_size * kv_bytes_per_seq(seq_len)
    flops = batch_size * (2 * n_params + attn_flops_per_seq(seq_len))
    return max(bytes_moved / bandwidth_bytes_per_sec, flops / flops_per_sec) * 1000.0


for S in (2048, 8192, 32768):
    kvb = kv_bytes_per_seq(S)
    print(f"S={S:6d}  kv/seq={kvb/1e9:5.2f} GB  "
          f"B_half={2*N_PARAMS/kvb:6.1f}  ceiling={H100_BW/kvb:7.0f} tok/s")
    for B in (1, 16, 64, 256):
        t = decode_step_time_ms_v2(N_PARAMS, B, S, H100_FLOPS, H100_BW)
        print(f"    B={B:4d}  step={t:7.2f} ms  node throughput={B/(t/1000):7.0f} tok/s")

S=  2048  kv/seq= 0.67 GB  B_half= 208.6  ceiling=   4992 tok/s
    B=   1  step=  41.99 ms  node throughput=     24 tok/s
    B=  16  step=  45.00 ms  node throughput=    356 tok/s
    B=  64  step=  54.61 ms  node throughput=   1172 tok/s
    B= 256  step=  93.07 ms  node throughput=   2750 tok/s
S=  8192  kv/seq= 2.68 GB  B_half=  52.2  ceiling=   1248 tok/s
    B=   1  step=  42.59 ms  node throughput=     23 tok/s
    B=  16  step=  54.61 ms  node throughput=    293 tok/s
    B=  64  step=  93.07 ms  node throughput=    688 tok/s
    B= 256  step= 246.92 ms  node throughput=   1037 tok/s
S= 32768  kv/seq=10.74 GB  B_half=  13.0  ceiling=    312 tok/s
    B=   1  step=  45.00 ms  node throughput=     22 tok/s
    B=  16  step=  93.07 ms  node throughput=    172 tok/s
    B=  64  step= 246.92 ms  node throughput=    259 tok/s
    B= 256  step= 862.32 ms  node throughput=    297 tok/s

This single term explains most of the interventions in the rest of this chapter: prefix caching (don’t rebuild caches you already paid for), KV quantization (halve \(\text{kv}(S)\), double the ceiling), MLA and GQA (shrink \(n_{\text{kv}}\)), sliding-window and hybrid attention (cap the \(S\) in \(\text{kv}(S)\)), and paged allocation (stop wasting the capacity you do have — see PagedAttention & KV-Cache Memory Management).

GPU Selection and the Hardware Cost Function¶

Key GPU Metrics for Inference¶

When choosing a GPU for inference, the relevant specs are different from those for training:

Note: BF16 figures are dense (non-sparse) tensor-core peak, for consistency across rows. Prices are illustrative order-of-magnitude estimates and vary significantly by provider, region, and contract. As of 2026 the Blackwell generation is the volume frontier for inference — the B200 above, plus the newer Blackwell Ultra (B300/GB300) with 288 GB of HBM3e per GPU for larger models and longer-context KV caches — and its native FP4 tensor cores add a low-precision serving tier below FP8 (see below).

For inference on a fixed model, the figures of merit are: 1. Tokens/s/dollar: dominated by HBM bandwidth for decode-heavy workloads. 2. Max batch size before KV-cache OOM: dominated by HBM capacity. 3. TTFT for long prompts: dominated by compute (FLOP/s).

The H100’s jump in bandwidth (3.35 vs. 2.00 TB/s for A100) directly translates to a 1.67× speedup in decode throughput per GPU when bandwidth-bound — before any software optimization.

Tensor Parallelism Across GPUs¶

Splitting a model across \(N\) GPUs with tensor parallelism (TP) reduces per-GPU memory by \(1/N\) but also splits both FLOPs and bandwidth by roughly \(1/N\). The bandwidth-bound decode time becomes:

The all-reduce adds a fixed per-step communication overhead (typically a few ms over NVLink). Since the weight-streaming time halves with TP=2, the effective batch size breakeven also halves — you saturate compute with fewer concurrent requests.

For very large models (>70B), TP is often mandatory to fit in memory. The key point is that TP does not improve tokens/s/dollar beyond what’s needed to fit the model — it just allows serving. Sequence parallelism and disaggregated architectures are needed for further scaling (see Multi-GPU & Multi-Node Inference).

The Other End of the Curve: Serving a ~100M Model¶

Everything above assumes weight streaming is the dominant cost. For a small model it simply is not, and the economics invert in a way that is worth internalizing — not least because it is the regime the capstone model of Evaluation & Serving: Honest Benchmarks, int4 Quantization, and Running on a Laptop lives in.

Take Stack-100M: 100M parameters, 0.2 GB in BF16. The bandwidth floor on an H100 is \(0.2\,\text{GB} / 3.35\,\text{TB/s} \approx 60\ \mu\text{s}\) per decode step. But a small transformer still launches on the order of a hundred CUDA kernels per forward pass, and each launch plus the Python scheduling around it costs a few microseconds. The fixed per-step overhead is now an order of magnitude larger than the physics. The consequences:

• CUDA Graphs and torch.compile are the dominant optimization, not quantization. Capturing the decode step as a graph collapses hundreds of launches into one replay; vLLM does this by default (it is what --enforce-eager turns off, which is why eager mode is so much slower on small models). See Kernel Fusion, torch.compile, CUDA Graphs & Compilers.
• The arithmetic-intensity breakeven is trivially reachable. \(B^\ast \approx 295\) sequences of a 100M model cost almost no HBM, so you should batch aggressively; per-token cost falls until the CPU-side scheduler, tokenizer, and HTTP layer become the bottleneck. At this scale a meaningful share of the bill is CPU, not GPU.
• The GPU may be the wrong hardware entirely. Quantized to 4-bit GGUF, Stack-100M is roughly 55 MB of weights. On a laptop with ~80 GB/s of DDR5 bandwidth the weight-streaming floor is under a millisecond per token, so llama.cpp on CPU delivers interactive speeds at zero marginal GPU cost. Marginal-cost-per-token analysis is the wrong frame here; total cost of ownership is dominated by whether you need a GPU at all.
• Fixed costs dominate at low volume. A model this small serves so many tokens per GPU-hour that the \(\$/1M\) figure becomes a rounding error next to the engineering time, the eval harness, and the one always-warm replica. The capstone’s full ledger — training plus serving — is worked through in Retrospective: Cost Accounting, Reproducibility, and the Path to 1B.

The transferable lesson is that the roofline method, not any particular number, is what generalizes: recompute \(2P/\text{BW}\) and \(\text{kv}(S)/\text{BW}\) for your model on your memory system (HBM, DDR, or unified memory), compare against the fixed per-step overhead, and optimize whichever term is largest.

Why Decode-Heavy Workloads Are Expensive¶

Reasoning models, coding assistants, and long-form generation are all examples of decode-heavy workloads: the ratio of output tokens to input tokens is high (sometimes 10:1 or more). This matters economically for three reasons:

1. Compute cost scales with output length. Each output token requires one full forward pass through the decode stage (weight loading from HBM). A 10× longer output costs roughly 10× more GPU time.

2. KV cache memory scales with context length. For a model with \(n_{\text{heads}}\) KV heads, head dimension \(d\), \(L\) layers, in BF16, the KV cache for one sequence of length \(S\) is:

For Llama-3 70B: \(n_{\text{kv}}=8\) (GQA), \(d=128\), \(L=80\). At \(S=32{,}768\) tokens:

A TP=2 node of two 80 GB H100s has 160 GB of HBM; 140 GB goes to BF16 weights and a few more GB to activations and CUDA workspace, leaving on the order of 15–20 GB for KV cache. That is one or two concurrent 32k-context sequences. Batch size is not limited by the arithmetic-intensity knee here — it is limited by capacity, long before you get near it. Going to TP=4 (320 GB) or FP8 weights (70 GB) is what buys back the batch.

3. Speculative decoding helps but has limits. Speculative decoding (see Speculative Decoding: Draft Models, Medusa, EAGLE & Lookahead) can recover 2–3× speed for predictable outputs (code, factual answers) by verifying multiple tokens per step, but the draft model adds HBM bandwidth overhead and is less effective for creative or reasoning-heavy outputs.

Chain-of-Thought Tax¶

The reasoning models popularized by OpenAI o1 and DeepSeek-R1 generate long internal reasoning traces before the final answer. If a reasoning model generates 2,000 thinking tokens before a 200-token answer, the effective output-to-answer token ratio is 11:1. At $3/1M output tokens, reasoning for 1,000 queries costs:

vs. $0.60 for a direct-answer model with 200 tokens. The quality premium has a concrete dollar tag. See Reasoning, Chain-of-Thought & Test-Time Compute for the quality vs. cost trade-off in detail.

Capacity Planning and Autoscaling¶

The Capacity Planning Formula¶

Capacity planning starts from traffic projections. Given: - \(\lambda\) = peak requests per second - \(\bar{o}\) = average output tokens per request - \(T_{\text{decode}}\) = sustained decode tokens/s per GPU - \(U_{\text{target}}\) = target GPU utilization (e.g., 0.7 to leave headroom)

The number of GPUs required for decode is:

For prefill-heavy workloads, a similar formula applies based on TTFT SLO and prefill FLOP/s.

Autoscaling Strategies¶

Autoscaling for LLM serving is different from stateless web services because:

1. Startup latency is high. Loading a 70B model from disk to GPU memory takes tens of seconds. Cold-start latency makes traditional reactive autoscaling painful.

2. The batch size effect creates sudden non-linearity. At low traffic, fewer GPUs may be just as fast (since decode is bandwidth-bound and batch size doesn’t affect per-step latency until saturation). The breakeven batch size gives you a natural hysteresis threshold.

3. GPU memory is the binding constraint, not CPU. You cannot partially load a model; you need the whole thing in HBM.

Practical approaches:

• Predictive (proactive) scaling: use historical traffic patterns to pre-scale 5–10 minutes ahead of predicted ramp-up.
• Scale to zero with warm pools: maintain 1 “warm” replica always; scale-to-zero on truly idle queues.
• Model sharding and multiplexing: run multiple smaller models on the same GPU (e.g., several 7B fine-tunes with different LoRA adapters via LoRA adapter hot-swap, as in PEFT I: LoRA, QLoRA, DoRA & The Adapter Family).

# Minimal autoscaler mock: decides how many replicas to run based on queue depth
# and current tokens/s, with hysteresis to avoid thrashing.

from dataclasses import dataclass, field
from collections import deque
import time

@dataclass
class AutoscalerConfig:
    min_replicas: int = 1
    max_replicas: int = 16
    target_tokens_per_sec_per_replica: float = 2000.0  # sustained decode tps/replica
    scale_up_queue_threshold: int = 100    # tokens queued → add replica
    scale_down_idle_seconds: float = 120.0  # idle for 2 min → remove replica
    cooldown_seconds: float = 60.0          # min time between scale events

@dataclass
class AutoscalerState:
    n_replicas: int = 1
    last_scale_time: float = field(default_factory=time.time)
    idle_since: dict = field(default_factory=dict)  # replica_id → time became idle

def autoscale_step(
    config: AutoscalerConfig,
    state: AutoscalerState,
    queue_depth_tokens: int,     # tokens currently waiting in the request queue
    active_tokens_per_sec: float, # current measured throughput
    now: float = None,
) -> int:
    """
    Return the desired number of replicas. Does not actually spin up/down anything —
    the orchestrator (Kubernetes, Ray Serve, etc.) handles the actual scaling.
    """
    if now is None:
        now = time.time()

    # Don't scale more often than the cooldown window
    if now - state.last_scale_time < config.cooldown_seconds:
        return state.n_replicas

    desired = state.n_replicas

    # Scale UP: queue is building faster than current replicas can drain it
    tokens_capacity = state.n_replicas * config.target_tokens_per_sec_per_replica
    if queue_depth_tokens > config.scale_up_queue_threshold:
        # Estimate replicas needed to drain queue within 30 seconds
        needed = int((active_tokens_per_sec + queue_depth_tokens / 30.0)
                     / config.target_tokens_per_sec_per_replica) + 1
        desired = min(needed, config.max_replicas)

    # Scale DOWN: we have spare capacity
    elif active_tokens_per_sec < 0.5 * tokens_capacity and state.n_replicas > config.min_replicas:
        desired = max(config.min_replicas, state.n_replicas - 1)

    if desired != state.n_replicas:
        state.n_replicas = desired
        state.last_scale_time = now

    return desired

Quantization and Its Cost Impact¶

Quantization (see Quantization I: Post-Training Quantization (GPTQ, AWQ, SmoothQuant) and Quantization II: INT4/INT8/FP8, GGUF, bitsandbytes & QAT) is one of the most powerful tools in the inference economist’s toolkit.

How Quantization Changes the Economics¶

Going from BF16 to INT8 halves the number of bytes loaded from HBM per decode step — directly halving the bandwidth-bound decode time (for the same batch size) or equivalently, halving the number of GPUs needed to sustain the same throughput.

Going from BF16 to INT4 (e.g., via GPTQ or AWQ) approximately quarters the decode memory bandwidth, but introduces some quality degradation and requires dequantization overhead.

For a 70B model:

FP8 inference is supported natively on H100 and Blackwell hardware via the FP8 matmul units, offering near-BF16 quality with roughly 2× throughput improvement — a straightforward win for most production workloads. On Blackwell, NVIDIA’s NVFP4 4-bit format (a micro-scaled FP4 storing one FP8 scale per 16 values) pushes this further: post-training quantization to NVFP4 has been shown to stay within about 1% of FP8 accuracy on models like DeepSeek-R1 while delivering roughly 3× the FP8 (up to 4× the BF16) peak throughput on Blackwell, and it is now deployable via TensorRT-LLM and vLLM. In 2026 this makes hardware-native 4-bit a viable production tier rather than a research curiosity.

KV Cache Quantization¶

Beyond weight quantization, the KV cache itself can be quantized. From the 70B example above (10.7 GB per sequence at 32k context in BF16), switching KV cache to INT8 halves this to 5.35 GB/sequence — roughly doubling the number of concurrent long-context sequences a single node can handle.

Optimizing the Bill: A Practitioner Playbook¶

Pulling all the above together, here is a concrete checklist for reducing inference costs.

Tier 1: Cheapest wins (deploy immediately)¶

1. Maximize continuous-batch utilization. If your average GPU utilization during peak is below 70%, you have headroom to serve more traffic at zero incremental cost. Profile with nvidia-smi dmon or vLLM’s built-in metrics.

2. Enable FP8 or INT8 quantization where quality is acceptable. Use AWQ or GPTQ for INT4 on older hardware. This halves or quarters your GPU memory footprint, often allowing you to cut replicas.

3. Enable prefix caching (see Prefix Caching & KV-Cache Reuse) for workloads with shared system prompts or few-shot examples. Recomputing a 2,000-token system prompt on every request wastes significant compute.

4. Choose the right model size. A well-fine-tuned 13B model may outperform a generic 70B on your specific task at ⅕th the compute cost. Evaluate empirically.

Tier 2: Architectural changes (higher leverage, more work)¶

5. Speculative decoding for predictable outputs (code, templates, FAQ answers). Properly configured, a 2–3× throughput gain is achievable with no quality loss.

6. Disaggregated prefill/decode for mixed-length workloads. Prefill-heavy requests (long documents) run on FLOP-optimized nodes; decode runs on bandwidth-optimized nodes. This avoids decode latency spikes caused by large prefill batches blocking the decode queue.

7. Knowledge distillation into a smaller task-specific model. If your deployment uses a general 70B model but only for one narrow task, distilling a fine-tuned 7B model can yield 10× cost reduction. See Distillation, Model Compression & Knowledge Transfer.

Tier 3: Systemic cost management¶

8. Prompt compression (prompt caching, summarization, retrieval rather than full context) reduces input tokens. Though cheaper per token, input tokens still consume compute and KV cache memory.

9. Routing to model tiers. Route simple queries (keyword lookup, formatting) to small/fast/cheap models, and complex queries to large models. See Caching, Routing & Cost Control in Production.

10. Spot/preemptible instances for batch inference workloads (embeddings, offline scoring, eval runs). Spot pricing is typically 60–70% cheaper; use checkpoint-and-resume for fault tolerance.

# Cost-aware router: send queries to a small or large model based on estimated complexity.
# In production, use a lightweight classifier; here we use a heuristic proxy.

from typing import Literal

ModelTier = Literal["fast_small", "capable_large"]

# Illustrative cost per 1M output tokens (update to your actual prices)
COST_PER_1M = {
    "fast_small":   0.50,   # e.g., a 7B fine-tuned model on cheap hardware
    "capable_large": 8.00,  # e.g., a 70B frontier model
}

def classify_complexity(prompt: str, n_few_shot: int = 0) -> float:
    """
    Estimate query complexity as a float in [0, 1].
    Real systems train a small classifier on human-labeled routing decisions.
    Here we use proxy features: prompt length, question words, code keywords.
    """
    words = prompt.lower().split()
    n_words = len(words)

    code_keywords  = {"def", "class", "import", "function", "algorithm",
                      "implement", "debug", "explain", "analyze"}
    hard_keywords  = {"compare", "contrast", "design", "evaluate", "synthesize",
                      "critique", "reason", "proof", "derive"}

    code_signal  = sum(1 for w in words if w in code_keywords) / max(n_words, 1)
    hard_signal  = sum(1 for w in words if w in hard_keywords) / max(n_words, 1)
    length_signal = min(n_words / 200.0, 1.0)  # normalize at 200 words

    return min(1.0, code_signal * 2 + hard_signal * 2 + length_signal * 0.5)


def route_query(prompt: str, complexity_threshold: float = 0.25) -> ModelTier:
    """
    Return which model tier to use for this prompt.
    Below the threshold, the small fast model suffices.
    """
    score = classify_complexity(prompt)
    if score < complexity_threshold:
        return "fast_small"
    return "capable_large"


# Simulate routing 1000 queries and compute expected cost
import random

def simulate_cost(n_queries: int = 1000,
                  avg_output_tokens: int = 300,
                  threshold: float = 0.25) -> dict:
    random.seed(42)
    total_small = 0
    total_large = 0

    # Synthetic prompts: mix of simple and complex
    prompts = (
        ["What is 2+2?"] * 400              # trivial
        + ["List the top 5 European capitals"] * 200  # easy
        + ["Implement a red-black tree in Python with full docstrings"] * 200  # hard
        + ["Compare DPO vs PPO for RLHF alignment"] * 200  # hard
    )
    random.shuffle(prompts)

    for p in prompts[:n_queries]:
        tier = route_query(p, threshold)
        if tier == "fast_small":
            total_small += 1
        else:
            total_large += 1

    cost_small = (total_small * avg_output_tokens / 1e6) * COST_PER_1M["fast_small"]
    cost_large = (total_large * avg_output_tokens / 1e6) * COST_PER_1M["capable_large"]
    cost_all_large = (n_queries * avg_output_tokens / 1e6) * COST_PER_1M["capable_large"]

    return {
        "routed_small": total_small,
        "routed_large": total_large,
        "cost_with_routing": cost_small + cost_large,
        "cost_all_large": cost_all_large,
        "savings_pct": 100.0 * (1 - (cost_small + cost_large) / cost_all_large),
    }

result = simulate_cost()
print(f"Routed to small: {result['routed_small']} / Routed to large: {result['routed_large']}")
print(f"Cost with routing: ${result['cost_with_routing']:.4f}")
print(f"Cost all-large:    ${result['cost_all_large']:.4f}")
print(f"Savings:           {result['savings_pct']:.1f}%")

The Full Cost Stack: Beyond GPU Compute¶

GPU compute is the dominant but not the only cost. A complete cost breakdown for a production LLM API includes:

At small scale, the marginal dollar usually goes further when spent on engineering (prompt compression, fine-tuning a smaller model, prefix caching implementation) than on more hardware.

At large scale (> $1M/month), negotiated reserved instance pricing, custom silicon (Google TPUs, AWS Trainium2, NVIDIA Hopper/Blackwell multi-year reservations), and distillation programs pay off significantly.

Putting It All Together: A Reference Dashboard¶

A good inference cost dashboard tracks these key metrics in real time:

You do not have to invent most of this. vLLM and SGLang both expose a Prometheus /metrics endpoint out of the box (--disable-log-stats turns vLLM’s off), so scraping the engine gets you the raw series; your job is the derived cost layer on top. The right-hand column below names the upstream vLLM series each dashboard panel should be built from.

# Reference metrics to instrument in your serving stack (e.g., via Prometheus).
# The comment on each line names the upstream vLLM series it derives from —
# scrape http://<host>:8000/metrics and build these on top rather than
# re-instrumenting from scratch. (Series names track vLLM's current metrics
# schema; confirm against your deployed version.)

INFERENCE_METRICS = {
    # Throughput  <- rate() over vllm:generation_tokens_total / vllm:prompt_tokens_total
    "output_tokens_per_second":   "gauge",    # rate(vllm:generation_tokens_total[5m])
    "input_tokens_per_second":    "gauge",    # rate(vllm:prompt_tokens_total[5m])

    # Latency  <- histogram_quantile() over the engine's latency histograms
    "ttft_p50_ms":                "gauge",    # vllm:time_to_first_token_seconds
    "ttft_p99_ms":                "gauge",    # vllm:time_to_first_token_seconds
    "tbt_p50_ms":                 "gauge",    # vllm:time_per_output_token_seconds
    "tbt_p99_ms":                 "gauge",    # vllm:time_per_output_token_seconds

    # Efficiency
    "gpu_utilization_pct":        "gauge",    # DCGM exporter (not the engine)
    "kv_cache_utilization_pct":   "gauge",    # vllm:gpu_cache_usage_perc
    "batch_size_mean":            "gauge",    # vllm:num_requests_running
    "batch_size_p99":             "gauge",    # vllm:num_requests_running

    # Cost
    "cost_per_1m_output_tokens":  "gauge",    # Derived: $/1M output tokens
    "gpu_cost_per_hour":          "gauge",    # Cluster cost rate (from cloud API or fixed)
    "requests_per_dollar":        "gauge",    # Inverse cost efficiency

    # Quality proxy
    "generation_errors_per_min":  "gauge",    # Truncations, OOM aborts, timeouts
    "queue_depth_tokens":         "gauge",    # vllm:num_requests_waiting (as a proxy)
}

def compute_cost_per_1m_tokens(
    gpu_cost_per_hour: float,
    output_tokens_per_second: float,
) -> float:
    """
    Compute the current effective cost per 1M output tokens from live metrics.
    This number should be tracked and alerted on if it exceeds budget.
    """
    if output_tokens_per_second <= 0:
        return float("inf")
    return (gpu_cost_per_hour * 1e6) / (output_tokens_per_second * 3600.0)

The same formula as a single PromQL expression, so the headline number is a live panel rather than a spreadsheet:

# $/1M output tokens, computed directly from vLLM's counters.
# gpu_cluster_cost_per_hour is a constant you record (a recording rule or a
# static series); everything else comes from the engine's /metrics endpoint.
(gpu_cluster_cost_per_hour * 1e6)
  / (sum(rate(vllm:generation_tokens_total[5m])) * 3600)

# KV-cache pressure — the leading indicator of preemption and of the
# KV-bandwidth wall from earlier in this chapter:
max(vllm:gpu_cache_usage_perc)

# Goodput proxy: request rate that met a 500 ms TTFT SLO over the window.
sum(rate(vllm:time_to_first_token_seconds_bucket{le="0.5"}[5m]))

Alert thresholds to set: - cost_per_1m_output_tokens > 2× your baseline → something is wrong (traffic spike, failed GPU, KV cache thrashing). - kv_cache_utilization_pct > 90% → risk of request preemption/OOM; scale out or reduce max context. - ttft_p99_ms > SLO → prefill queue is backing up; consider Disaggregated Prefill/Decode & Chunked Prefill. - batch_size_mean consistently < 10% of \(B^*\) at peak hours → you are over-provisioned; scale down.

Further Reading¶

• Kwon et al., “Efficient Memory Management for Large Language Model Serving with PagedAttention”, SOSP 2023. The foundational paper behind vLLM; introduces the KV cache memory problem and paged allocation.
• Yu et al., “Orca: A Distributed Serving System for Transformer-Based Generative Models”, OSDI 2022. Introduces continuous batching (iteration-level scheduling) and quantifies the throughput gains.
• Sheng et al., “FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU”, ICML 2023. Shows how to trade latency for throughput on memory-constrained hardware via offloading.
• Pope et al., “Efficiently Scaling Transformer Inference”, MLSys 2023 (Google). Detailed analysis of model parallelism strategies and hardware trade-offs for serving large models at scale.
• Agrawal et al., “Sarathi-Serve: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills”, OSDI 2024. Quantifies the prefill-decode interference problem and the benefit of chunked prefill for latency.
• vLLM project (github.com/vllm-project/vllm): The reference open-source implementation; its metrics endpoint is the best way to observe the concepts in this chapter in a real system.
• LLM-Perf Leaderboard (HuggingFace): Community benchmarks for tokens/s and cost across models, hardware, and quantization levels — useful for calibrating the numbers in this chapter against real measurements.

Exercises¶

1. (Conceptual) On a single H100 serving a 70B BF16 model, the chapter shows the decode step time is a flat ~42 ms per step for any batch size from 1 up to \(B^* \approx 295\). Explain why adding a 50^th concurrent decoding sequence to a batch that already has 49 adds zero extra latency, yet adding a sequence to a batch that already has 295 adds proportional latency. In one sentence, what physical resource is the system waiting on in each regime?

2. (Quantitative) You rent a single H100 for $5.00/hour. At a healthy peak batch it sustains 2,000 output tokens/second. Compute the cost per 1M output tokens. Then a lull drops traffic so the effective batch falls and sustained throughput drops to 700 tokens/second (the GPU is now poorly utilized). Recompute the cost per 1M tokens and state the multiplier between the two.

3. (Quantitative) A reasoning model emits an average of 1,500 internal chain-of-thought tokens before a 300-token final answer. A direct-answer model produces only the 300-token answer. Output tokens are billed at $4.00 per 1M. For a workload of 5,000 queries, compute the output-token cost of each model and the ratio between them. What is the effective output-to-answer token ratio for the reasoning model?

4. (Quantitative) Using the chapter’s Llama-3 70B KV-cache parameters (\(n_{\text{kv}} = 8\) GQA heads, head dim \(d = 128\), \(L = 80\) layers, BF16 = 2 bytes), compute the KV-cache size for one sequence at \(S = 8{,}192\) tokens. A TP=2 node (2 × 80 GB = 160 GB) holding 140 GB of BF16 weights plus activations and workspace leaves roughly 20 GB for KV cache. How many such sequences fit concurrently in BF16? How many if the KV cache is quantized to INT8?

5. (Quantitative) Your API sees peak traffic of 30 requests/second, each generating an average of 500 output tokens. You serve a 13B model at TP=1 on A100 80GB GPUs that sustain 6,000 decode tokens/second each. Targeting 70% utilization for headroom, how many GPUs do you need? At $3.50/GPU-hour, what does the peak cluster cost per hour?

6. (Implementation) Extend the chapter’s decode_step_time_ms model to answer the economic question directly: write a function dollars_per_1m_tokens(...) that returns the cost per 1M output tokens for a given batch size, GPU rental rate, and weight precision. It should reuse decode_step_time_ms (converting one decode step into a sustained tokens/second rate) and the chapter’s $/1M formula. Then use it to compare BF16 (bytes_per_param=2) against INT8 (bytes_per_param=1) for a 70B model on a single H100 at $5.00/hour with batch size 64, and confirm the speedup matches the bandwidth-ratio prediction.

# Reuses decode_step_time_ms, H100_FLOPS, H100_BW, N_PARAMS from the chapter.

def sustained_tokens_per_sec(
    n_params: float,
    batch_size: int,
    flops_per_sec: float,
    bandwidth_bytes_per_sec: float,
    bytes_per_param: int = 2,
) -> float:
    """One token per sequence per decode step -> batch_size tokens per step."""
    step_s = decode_step_time_ms(
        n_params, batch_size, flops_per_sec,
        bandwidth_bytes_per_sec, bytes_per_param,
    ) / 1000.0
    return batch_size / step_s


def dollars_per_1m_tokens(
    n_params: float,
    batch_size: int,
    flops_per_sec: float,
    bandwidth_bytes_per_sec: float,
    gpu_cost_per_hour: float,
    bytes_per_param: int = 2,
) -> float:
    """$/1M output tokens = (R * 1e6) / (T_sustained * 3600)."""
    tps = sustained_tokens_per_sec(
        n_params, batch_size, flops_per_sec,
        bandwidth_bytes_per_sec, bytes_per_param,
    )
    return (gpu_cost_per_hour * 1e6) / (tps * 3600.0)


COST_PER_HR = 5.00
BATCH = 64

for label, bpp in [("BF16", 2), ("INT8", 1)]:
    tps = sustained_tokens_per_sec(N_PARAMS, BATCH, H100_FLOPS, H100_BW, bpp)
    cost = dollars_per_1m_tokens(N_PARAMS, BATCH, H100_FLOPS, H100_BW,
                                 COST_PER_HR, bpp)
    print(f"{label}: {tps:7.0f} tok/s  ->  ${cost:.3f} / 1M output tokens")