12.1 Designing an LLM Serving System¶

A single model running under vllm serve on one GPU is a demo. A serving system is what stands between a million daily users and a fleet of GPUs: it terminates connections, authenticates and meters traffic, decides which replica of which model handles each request, protects the GPUs from overload, keeps tail latency inside a contractual budget, and scales the fleet up and down as load swings by 10x between 3 a.m. and 3 p.m. This chapter is about the system — the boxes and arrows around the model — and the quantitative reasoning that sizes each box.

It assumes you already understand what happens inside one engine. If prefill, decode, and the KV cache are not yet second nature, read The Anatomy of LLM Inference: Prefill, Decode & The KV Cache first; the batching machinery comes from Continuous Batching & Request Scheduling; the cost arithmetic comes from Inference Economics: Latency, Throughput & Cost. Here we zoom out and design the thing that wraps dozens of those engines into a service with an SLO. This chapter also doubles as a system-design interview script — see ML System Design: A Framework for the meta-framework and ML System Design: Worked Cases for adjacent cases.

Why LLM Serving Is Not Just “Microservices With a GPU”¶

If you have served a REST API before, your instincts are mostly right and dangerously incomplete. Three properties make LLM serving a different animal.

Requests are not uniform-cost. A normal web request takes a few milliseconds and returns a fixed-size payload. An LLM request streams for seconds to minutes and its cost is dominated by an output length you do not know in advance. A 20-token classification and a 4,000-token essay enter the same queue. This means classical load balancers that distribute by request count will badly imbalance GPU work. You must balance by tokens, or better, by predicted compute.

The unit of work is a long-lived, stateful stream. Each generation holds a KV cache that grows with every token, pinning a slice of HBM (high-bandwidth memory) for the request’s entire lifetime. You cannot freely move an in-flight request to another replica — its KV cache lives in one GPU’s memory. Statelessness, the thing that makes web services trivially scalable, is gone.

Latency has two numbers, not one. Users perceive two distinct quantities: TTFT (time to first token), how long until words start appearing, and TPOT / ITL (time per output token / inter-token latency), how fast they then stream. These trade against each other and against throughput. A single “latency” SLO is meaningless; you need both.

The job of a serving system is to keep both of these numbers inside a budget while keeping the GPUs — the expensive part — busy. Everything below is in service of that tension.

The Reference Architecture¶

Let us lay out the full stack top to bottom, then dissect each layer. Requests flow downward; tokens stream back upward.

A few non-obvious points about this picture. The gateway is cheap, stateless, CPU-bound, and trivially horizontally scalable; you run many of them behind a standard L4/L7 load balancer. The router is the brain — it makes per-request placement decisions and is where most of the interesting policy lives. The model pools are groups of identical replicas of one model configuration; each replica is a full inference engine with its own continuous-batching scheduler and its own KV cache. The autoscaler watches metrics and changes the number of replicas per pool. Below we walk these in order.

The Gateway: The Cheap, Stateless Front Door¶

The gateway does everything that is not model inference and is per-request bookkeeping. Keep it dumb and fast so it never becomes the bottleneck.

• TLS termination and HTTP/2 or gRPC handling. Streaming responses go out as Server-Sent Events (SSE) or chunked transfer; the gateway must support streaming without buffering the whole response.
• Authentication and authorization: validate API keys / OAuth tokens, resolve the caller’s org and tier.
• Rate limiting and quota: per-key requests-per-minute and tokens-per-minute. LLM rate limits must be token-aware, because one request can be 1,000x another.
• Usage metering: count prompt and completion tokens for billing and for the data flywheel (Data Flywheels & Continuous Improvement).
• Request validation and normalization: enforce max_tokens caps, reject malformed JSON early, apply default sampling params.

A token-aware rate limiter is the one piece worth showing, because the naive “N requests per minute” limiter is wrong for LLMs. We use a token bucket keyed on tokens, refilling continuously.

import time
import threading

class TokenBucketLimiter:
    """
    Token-aware rate limiter. The 'tokens' here are LLM tokens (prompt+completion),
    not generic request credits. We refill `rate` tokens/sec up to `capacity`.

    Each LLM request first RESERVES an estimate (prompt_len + max_new_tokens),
    then RECONCILES with the true completion length when the stream finishes.
    This prevents a flood of long-generation requests from blowing the budget.
    """
    def __init__(self, rate_per_sec: float, capacity: float):
        self.rate = rate_per_sec          # e.g. 50_000 tokens/sec for a tier
        self.capacity = capacity          # burst ceiling, e.g. 100_000 tokens
        self.tokens = capacity            # start full
        self.last = time.monotonic()
        self.lock = threading.Lock()

    def _refill(self) -> None:
        now = time.monotonic()
        elapsed = now - self.last
        self.tokens = min(self.capacity, self.tokens + elapsed * self.rate)
        self.last = now

    def try_reserve(self, estimated_tokens: float) -> bool:
        """Reserve up-front. Returns False (429 -> client backs off) if over budget."""
        with self.lock:
            self._refill()
            if self.tokens >= estimated_tokens:
                self.tokens -= estimated_tokens
                return True
            return False

    def reconcile(self, estimated_tokens: float, actual_tokens: float) -> None:
        """Refund or charge the difference once the true length is known."""
        with self.lock:
            self._refill()
            # If we over-estimated, give tokens back (capped at capacity).
            self.tokens = min(self.capacity, self.tokens + (estimated_tokens - actual_tokens))

The crucial idea is reserve-then-reconcile: because output length is unknown, you charge a pessimistic estimate up front (so a burst of long requests cannot overrun the budget) and refund the slack when the stream ends. This same pattern reappears in admission control and in cost accounting.

In production you seldom write this gateway yourself. Two open-source options implement exactly the layer above: LiteLLM proxy (BerriAI/litellm) — an OpenAI-compatible server with per-key virtual budgets, token-per-minute limits, fallbacks and spend tracking, covered further in Caching, Routing & Cost Control in Production — and Envoy AI Gateway (envoyproxy/ai-gateway), which extends Envoy’s rate-limit API so that a request’s cost is its token count (input, output, or a CEL-weighted combination) extracted from the OpenAI-schema response, rather than a flat “1”. Recognizing that both are just reserve-then-reconcile token buckets with a control plane is the point of writing the 30 lines above.

The Router: Where the Intelligence Lives¶

The router answers two questions for every request: which model and which replica of that model. Both are policy decisions, and both matter for latency and cost.

Model routing¶

Sometimes the caller names the model explicitly ("model": "llama-3-70b") and routing is a lookup. More interesting is policy routing, where the system chooses the model: send easy queries to a cheap 8B model and hard ones to an expensive 70B, gated by a small classifier or by the prompt’s characteristics. This cascade pattern is a major cost lever and is developed in depth in Caching, Routing & Cost Control in Production; here we note only that the router is the natural home for it, and that any routing classifier must itself be cheap (sub-millisecond) or you have just added latency to every request.

Replica selection: do not use round-robin¶

The seductive default is round-robin or random. For LLMs this is a mistake, because replicas are stateful and unevenly loaded. Two better signals:

1. Least outstanding load. Pick the replica with the fewest queued + running tokens (not requests). Because cost scales with tokens, balancing tokens balances work.
2. Prefix-cache affinity. If a request shares a long prefix (system prompt, few-shot examples, a document) with a recent request, routing it to the replica that already has those KV blocks cached turns an expensive prefill into a near-free cache hit. This is the central idea of Prefix Caching & KV-Cache Reuse and of SGLang’s RadixAttention (SGLang: RadixAttention & Structured Programs).

These two goals conflict: affinity says “send it where the cache is,” load-balancing says “send it where it’s quiet.” The router blends them. A simple, effective scheme is power-of-two-choices with a cache bonus: sample two candidate replicas at random, and pick the one with the lower effective load, where effective load subtracts a bonus for cached prefix overlap.

import random
from dataclasses import dataclass, field

@dataclass
class ReplicaState:
    id: str
    queued_tokens: int = 0        # tokens waiting (prefill not yet started)
    running_tokens: int = 0       # tokens of active sequences (KV held)
    cached_prefixes: set = field(default_factory=set)  # hashes of cached prefix blocks

    def effective_load(self, req_prefix_hashes: set) -> float:
        # Base load = work in flight. Tokens are the right unit, not request count.
        load = self.queued_tokens + 0.5 * self.running_tokens
        # Cache bonus: each overlapping prefix block we DON'T have to recompute
        # is real prefill work saved, so we subtract it from effective load.
        overlap = len(self.cached_prefixes & req_prefix_hashes)
        CACHE_BLOCK_TOKENS = 16
        return load - overlap * CACHE_BLOCK_TOKENS

def pick_replica(replicas, req_prefix_hashes, d=2):
    """Power-of-d-choices, load- and cache-aware. O(d), no global scan needed."""
    candidates = random.sample(replicas, k=min(d, len(replicas)))
    return min(candidates, key=lambda r: r.effective_load(req_prefix_hashes))

Power-of-two-choices is a beautiful result from balls-into-bins theory: sampling two random replicas and picking the less loaded one reduces the maximum load from \(\Theta(\log n / \log\log n)\) (pure random) to \(\Theta(\log\log n)\) — an exponential improvement — while needing only local state for two replicas, not a global scan. It is the workhorse of large fleets precisely because it avoids the herd behavior and central-state bottleneck of “always pick the global minimum.”

The open-source routers that implement this¶

effective_load above is a teaching model of a component you can deploy off the shelf. Three implementations are worth knowing, all of them scoring endpoints on the same signals — queue depth, KV-cache utilization, prefix overlap:

• vLLM production-stack (vllm-project/production-stack) — vLLM’s Kubernetes-native reference deployment. Its router offers round-robin, session-sticky, prefix-aware, KV-aware and disaggregated-prefill routing policies, with LMCache integration for cross-replica KV reuse. There is also a standalone high-performance vLLM Router written in Rust that does power-of-two-choices, consistent-hash prefix affinity, and prefill/decode-aware dispatch.
• Gateway API Inference Extension (kubernetes-sigs/gateway-api-inference-extension) — the Kubernetes-standard answer. It adds an InferencePool resource (a set of model-server pods) plus an Endpoint Picker (EPP), an ext-proc extension that scrapes each replica’s metrics — pending-queue length, KV-cache utilization, which LoRA adapters are resident — and tells the gateway which pod to send the request to. Supported backends include vLLM, SGLang and TensorRT-LLM; it is what GKE’s Inference Gateway and several Istio/Envoy-based gateways build on.
• NVIDIA Dynamo (ai-dynamo/dynamo) — a datacenter-scale orchestration layer that coordinates disaggregated prefill/decode pools and KV-aware routing across nodes, running on top of vLLM, SGLang or TensorRT-LLM.

The lesson to carry into a design review: the routing policy is the interesting part, and the reason these projects exist is that a stock L7 load balancer has no way to see queue depth or KV-cache state, so it cannot compute anything like effective_load.

SLOs: Defining “Fast Enough” Precisely¶

You cannot design a system without a target. For LLM serving the target is a set of SLOs (service-level objectives) on latency, stated as percentiles, because averages hide the tail that users actually feel.

The canonical four metrics:

A real SLO names a percentile and a number, e.g. “p50 TTFT ≤ 300 ms, p99 TTFT ≤ 1,000 ms, p90 TPOT ≤ 50 ms, for prompts ≤ 2k tokens.” The percentile matters enormously: the gap between p50 and p99 is almost entirely queueing delay under load, which is why batching and admission control (below) are SLO tools, not just throughput tools.

Once you have SLOs, the quantity you actually optimize stops being throughput and becomes goodput: the rate of requests that complete while meeting both their TTFT and TPOT targets. A fleet driven to saturation shows a beautiful tokens/sec number and near-zero goodput, because every request violates the SLO. Goodput is the objective function behind disaggregation and chunked-prefill scheduling (DistServe, Sarathi-Serve) and behind every sizing decision below; the measurement procedure — sweep the offered rate, take the throughput at the last rate that still meets the SLO — is in Inference Economics: Latency, Throughput & Cost.

The fundamental tension is throughput vs. latency, mediated by batch size. Bigger batches amortize weight loads across more tokens, raising throughput (lower cost), but they make each decode step take longer (worse TPOT) and make a newly arrived request wait behind a larger in-flight batch (worse TTFT). Continuous batching (Continuous Batching & Request Scheduling) and chunked prefill (Disaggregated Prefill/Decode & Chunked Prefill) exist to move this frontier outward, but the tradeoff never disappears. The serving system’s batching policy is the knob that places you on the curve where your SLOs are met at the lowest cost.

Queueing: Little’s Law and Why p99 Explodes¶

Here is the single most useful piece of quantitative reasoning for a serving interview. Treat one model pool as a queueing system. Little’s Law relates the three core quantities of any stable queue:

where \(L\) is the mean number of requests in the system, \(\lambda\) is the arrival rate (requests/sec), and \(W\) is the mean time a request spends in the system. It holds for any stable system regardless of arrival or service distribution — which is what makes it so powerful for capacity planning.

Now define utilization \(\rho = \lambda / (c\,\mu)\), where \(c\) is the number of parallel “servers” (think: batch slots / replicas) and \(\mu\) is the service rate per server. The brutal fact of queueing theory is that waiting time does not grow linearly as you approach saturation — it grows like \(1/(1-\rho)\). For a simple M/M/1 model the mean time in system is:

As \(\rho \to 1\), \(W \to \infty\). This is the mathematical reason a cluster that looks “80% utilized and fine” falls off a cliff at 95%: the queueing term \(1/(1-\rho)\) goes from \(5\) to \(20\) — a 4x latency increase from a 15-point utilization change. The tail percentiles explode even faster than the mean. This is why you provision LLM clusters to run at 60–75% utilization, not 95%. The headroom is not waste; it is the budget that keeps p99 bounded.

Every symbol in that arithmetic — \(\mu\), the per-replica service rate, and the replica_token_capacity the autoscaler later divides by — is something you measure, never guess, because it depends on your model, GPU, precision and prompt-length distribution. The standard instrument is an open-loop load generator that issues requests at a fixed Poisson rate regardless of whether earlier ones finished (a closed-loop tool with a fixed concurrency can never build a queue, so it cannot find your cliff):

# In one shell: start ONE replica exactly as production would run it.
#   vllm serve meta-llama/Meta-Llama-3-8B-Instruct --port 8000

# In another shell: sweep the offered rate against the SAME prompt/output length distribution
# your traffic has, and read off TTFT/TPOT percentiles at each rate.
for rate in 4 8 12 16 20 24; do
  vllm bench serve \
    --backend vllm --host 127.0.0.1 --port 8000 \
    --model meta-llama/Meta-Llama-3-8B-Instruct \
    --dataset-name random --random-input-len 800 --random-output-len 200 \
    --num-prompts 500 --request-rate "$rate" \
    --percentile-metrics ttft,tpot,itl --metric-percentiles 50,90,99
done

The highest --request-rate whose p99 TTFT and p90 TPOT still sit inside the SLO is that replica’s goodput; dividing it by the target utilization gives the \(\mu\) you plug into the sizing formulas. (SGLang ships the equivalent as python -m sglang.bench_serving; NVIDIA’s genai-perf does the same for TensorRT-LLM and Triton.)

The admission-control corollary: when a queue starts to build, it is usually better to shed load fast (return HTTP 429 so the client retries elsewhere or backs off) than to admit a request that will violate its SLO anyway and consume a KV slot that degrades everyone behind it. A queue that grows without bound is a worse outcome than a clean rejection.

def admit(request, replica, slo_ttft_s=1.0, prefill_tok_per_s=10_000):
    """
    Predictive admission control. Estimate the TTFT this request WOULD see if
    admitted to `replica` right now; reject (429) if it can't meet the SLO.
    This protects the requests already in flight from a late-arriving straggler.
    """
    # Work ahead of us in the queue (tokens), plus our own prefill cost.
    work_ahead = replica.queued_tokens
    our_prefill = request.prompt_len
    predicted_ttft = (work_ahead + our_prefill) / prefill_tok_per_s
    if predicted_ttft > slo_ttft_s:
        return False   # shed load: 429, let the client retry / route elsewhere
    replica.queued_tokens += our_prefill
    return True

Replica and Cluster Sizing: Memory Is the Real Constraint¶

Throughput sizing (above) tells you how many replicas you need for speed. A second, independent constraint sets how many concurrent requests fit: HBM capacity, almost entirely consumed by model weights plus KV cache. If you do not budget memory, the autoscaler will happily place a replica that OOMs on the first big batch.

The memory budget for one replica is:

The KV-cache term per token, for a model with \(L\) layers, \(H_{kv}\) key/value heads, head dimension \(d_h\), in bytes_per_elem precision, storing both K and V, is:

The factor \(H_{kv}\) (not the full head count \(H\)) is exactly why grouped-query attention matters so much for serving — see Multi-Head Attention, MQA, GQA & MLA. Fewer KV heads means more concurrent requests fit. The number of requests you can serve concurrently is then the leftover memory divided by per-request KV footprint:

def kv_bytes_per_token(num_layers, num_kv_heads, head_dim, bytes_per_elem=2):
    # 2 for K and V; bytes_per_elem=2 for fp16/bf16, 1 for fp8.
    return 2 * num_layers * num_kv_heads * head_dim * bytes_per_elem

def max_concurrent_requests(hbm_gb, weight_gb, reserve_gb,
                            num_layers, num_kv_heads, head_dim,
                            avg_ctx_tokens, bytes_per_elem=2):
    """How many simultaneous sequences fit in one replica's HBM."""
    free_bytes = (hbm_gb - weight_gb - reserve_gb) * 1e9
    per_tok = kv_bytes_per_token(num_layers, num_kv_heads, head_dim, bytes_per_elem)
    per_request = per_tok * avg_ctx_tokens
    return int(free_bytes // per_request)

# Llama-3-70B-ish on one 80GB H100 shard would not fit weights alone (140GB bf16),
# so 70B needs tensor parallelism. Let's size an 8B model on a single 80GB H100:
#   8B bf16 weights ~= 16 GB. L=32, num_kv_heads=8 (GQA), head_dim=128.
n = max_concurrent_requests(
    hbm_gb=80, weight_gb=16, reserve_gb=4,
    num_layers=32, num_kv_heads=8, head_dim=128,
    avg_ctx_tokens=2048, bytes_per_elem=2,
)
print(n)  # -> 223 concurrent 2k-token sequences fit in KV cache

For a 70B model the weights alone (≈140 GB in bf16) exceed one 80 GB GPU, forcing tensor parallelism (TP) across GPUs — covered in Multi-GPU & Multi-Node Inference. The sizing rule then operates per-shard: with TP=4, each GPU holds a quarter of the weights and a quarter of each layer’s KV, and the four GPUs together form one replica. PagedAttention (PagedAttention & KV-Cache Memory Management) is what lets you actually pack memory to near this theoretical \(N_{\text{concurrent}}\) instead of wasting it on fragmentation and worst-case pre-allocation.

Run the same arithmetic at the other end of the scale and the regime flips. Stack-100M — the ~100M-parameter model built end-to-end in Part XIV — has bf16 weights of roughly 0.2 GB and, with its handful of layers and KV heads, a KV footprint per token measured in kilobytes; a single consumer GPU holds thousands of concurrent sequences, so nothing is memory-bound and the binding constraint is purely prefill/decode compute. Serving it (including int4 quantization and a laptop-scale deployment) is Evaluation & Serving: Honest Benchmarks, int4 Quantization, and Running on a Laptop; the value of doing the arithmetic there is that you can verify the formulas above against a model you can actually hold in one GPU.

Batching Policy: The Throughput Engine¶

Inside each replica, the scheduler decides which sequences run in each forward pass. The serving system’s job is to configure that policy, not reinvent it. Three levers, in increasing sophistication.

Continuous (in-flight) batching. Static batching — wait for \(B\) requests, run them lockstep until all finish — wastes the GPU because short sequences sit idle waiting for the longest one. Continuous batching instead adds and evicts sequences every decode step: the moment one finishes, a waiting request takes its slot. This is the single biggest throughput win in modern serving and is the default in vLLM, SGLang, and TGI. The mechanism is detailed in Continuous Batching & Request Scheduling; the serving system mainly tunes its admission limits (max_num_seqs, max_num_batched_tokens).

Chunked prefill. Prefill is compute-heavy and bursty; a single long prompt’s prefill can stall the decode loop, spiking the TPOT of everyone else (a “decode stall”). Chunked prefill splits a long prefill into token-sized chunks and interleaves them with ongoing decode steps, smoothing TPOT at a small TTFT cost. See Disaggregated Prefill/Decode & Chunked Prefill.

Prefill/decode disaggregation. Because prefill is compute-bound and decode is memory-bandwidth-bound, they want different hardware and different batch sizes. Disaggregation runs them on separate replica pools and streams the KV cache between them. This eliminates prefill-vs-decode interference entirely at the cost of a KV transfer over the network — worth it at scale, overkill for small deployments.

class BatchPolicy:
    """
    Serving-side knobs that bound a replica's continuous-batching scheduler.
    These are the parameters the autoscaler and SLO budget actually control.
    """
    def __init__(self,
                 max_num_seqs=256,           # concurrency cap (KV-memory bound)
                 max_num_batched_tokens=8192,# per-step token budget (TPOT bound)
                 enable_chunked_prefill=True,
                 prefill_chunk_size=512):     # split long prefills into 512-tok chunks
        self.max_num_seqs = max_num_seqs
        self.max_num_batched_tokens = max_num_batched_tokens
        self.enable_chunked_prefill = enable_chunked_prefill
        self.prefill_chunk_size = prefill_chunk_size

    def step_budget(self, running_seqs, waiting_prefills):
        """
        Decide this step's work. Decode tokens are cheap (1 tok/seq); reserve the
        rest of the token budget for prefill chunks. This is the policy that
        trades TTFT (admit prefills) against TPOT (keep decode steps small).
        """
        decode_tokens = len(running_seqs)            # 1 token per running seq
        prefill_budget = self.max_num_batched_tokens - decode_tokens
        chunks = []
        for req in waiting_prefills:
            if prefill_budget <= 0 or len(running_seqs) >= self.max_num_seqs:
                break
            take = min(self.prefill_chunk_size, req.remaining_prefill, prefill_budget)
            chunks.append((req, take))
            prefill_budget -= take
        return decode_tokens, chunks

Every field of that toy class is a real flag on a real engine. This is what the policy looks like when you actually launch a replica:

# --max-num-seqs            concurrency cap  -> KV-memory bound
# --max-num-batched-tokens  per-step budget  -> TPOT bound
# --max-model-len           hard context cap (also caps KV per sequence)
# --gpu-memory-utilization  fraction of HBM vLLM may claim; the rest is reserve
# --kv-cache-dtype fp8      halves KV bytes/token -> ~2x concurrency
# --tensor-parallel-size    >1 shards weights AND KV across GPUs (still ONE replica)
# --enable-prefix-caching   prefix/KV reuse (already the default in vLLM's V1 engine)
vllm serve meta-llama/Meta-Llama-3-8B-Instruct \
  --max-num-seqs 256 \
  --max-num-batched-tokens 8192 \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.90 \
  --kv-cache-dtype fp8 \
  --tensor-parallel-size 1 \
  --enable-prefix-caching

One honest difference from the toy class: vLLM has no separate prefill_chunk_size knob. In its V1 engine chunked prefill is on by default and the chunk size is --max-num-batched-tokens — a long prefill is sliced to fill whatever token budget remains after the running sequences take one decode token each, which is exactly the step_budget policy above. SGLang exposes the same two levers as --max-running-requests and --chunked-prefill-size.

The deep lesson: the batch policy is how you spend your latency budget. A bigger max_num_batched_tokens raises throughput (cheaper) but lengthens each decode step (worse TPOT); enabling chunked prefill protects TPOT at the cost of slightly higher TTFT. There is no universally correct setting — there is only the setting that meets your SLO at the lowest cost, found by load-testing against your real traffic distribution.

Autoscaling on GPUs: Scaling a Resource You Cannot Get Instantly¶

Autoscaling a stateless web service is easy: CPU goes up, add pods, pods are ready in seconds. GPU autoscaling for LLMs is hard for three reasons, and a good system designer names all three.

1. Cold starts are minutes, not seconds. A new replica must acquire a GPU node (possibly from a cloud quota or a cluster autoscaler that itself takes minutes), pull a multi-gigabyte container, load tens of gigabytes of weights from object storage into HBM, warm up CUDA graphs / torch.compile, and prime the KV allocator. A 70B replica can take several minutes to become ready.
2. The scaling signal must be a leading indicator. Scaling on GPU utilization is too late — by the time utilization saturates, the queue is already building and p99 is already violated. Scale on queue depth / waiting-tokens or on a predicted-TTFT signal, which rise before the SLO breaks.
3. Scale-down must drain, not kill. You cannot SIGKILL a replica with 200 in-flight streaming requests; their KV caches and partial generations vanish. Scale-down marks a replica as draining (router stops sending new work), waits for in-flight requests to finish (with a timeout), then releases the GPU.

Because cold starts are slow, the standard pattern is predictive + buffered autoscaling: keep a warm headroom of \(k\) idle replicas (or a small pool of pre-warmed standby nodes) so demand spikes are absorbed instantly, while the slow path provisions more capacity in the background. The control law is a target-tracking loop on queue-derived load.

import math

def desired_replicas(current_replicas,
                     waiting_tokens, running_tokens,
                     replica_token_capacity,   # tokens/sec one replica sustains
                     target_utilization=0.7,
                     warm_buffer=2,
                     min_replicas=2, max_replicas=64):
    """
    Target-tracking autoscaler driven by QUEUE load (a leading indicator),
    not GPU utilization (a lagging one).

    offered_load: tokens of work the fleet currently owes (queued + in-flight).
    We size so that this load sits at `target_utilization` of total capacity,
    then add a warm buffer to absorb spikes during the slow cold-start window.
    """
    offered_load = waiting_tokens + running_tokens
    total_capacity_needed = offered_load / target_utilization
    raw = math.ceil(total_capacity_needed / replica_token_capacity)
    desired = raw + warm_buffer
    # Hysteresis: clamp and let the caller apply scale-down delay separately.
    return max(min_replicas, min(max_replicas, desired))

Two operational guardrails make this stable in practice. Hysteresis / asymmetric timing: scale up aggressively (spikes hurt users now) but scale down slowly (e.g. only after load stays low for several minutes), so a brief dip does not trigger an expensive teardown-then-rebuild cycle. Cost-aware bounds: GPUs are expensive enough that the min_replicas floor and warm_buffer are real budget decisions, not afterthoughts — a warm H100 sitting idle still bills by the second. Scale-to-zero is attractive for rarely-used models but pays the full multi-minute cold start on the next request, so reserve it for latency-tolerant or batch workloads.

In a Kubernetes deployment you do not run that control law yourself; you express it declaratively. Kubernetes’ built-in HPA scales on CPU and memory, which are useless signals for a GPU server (CPU stays near-idle while the GPU melts), so the standard pattern is KEDA — it manages an HPA underneath but drives it from an arbitrary metric, here vLLM’s own Prometheus gauge:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: vllm-llama3-8b
spec:
  scaleTargetRef:
    name: vllm-llama3-8b          # the Deployment running `vllm serve`
  minReplicaCount: 2              # never scale to zero for a latency-sensitive model
  maxReplicaCount: 64
  cooldownPeriod: 300             # scale DOWN slowly: 5 min of quiet before shrinking
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://prometheus-operated.monitoring.svc:9090
        metricName: vllm_num_requests_waiting
        # The leading indicator: work already queued, summed over the pool.
        query: sum(vllm:num_requests_waiting)
        threshold: "5"            # ~5 waiting requests per replica is the target

vllm:num_requests_waiting is one of the gauges vLLM exports on /metrics alongside vllm:num_requests_running, vllm:gpu_cache_usage_perc and the vllm:time_to_first_token_seconds / vllm:time_per_output_token_seconds histograms (see Continuous Batching & Request Scheduling and Observability, Logging & LLMOps). Asymmetric timing is configured through KEDA’s advanced.horizontalPodAutoscalerConfig.behavior (fast scaleUp, throttled scaleDown), and draining is bought with a generous terminationGracePeriodSeconds plus a preStop hook that deregisters the pod from the router before the engine is asked to stop. KServe and Ray Serve package the same loop (including scale-to-zero and request-driven autoscaling) at a higher level if you would rather not assemble it from primitives.

Multi-Model Serving: Many Models, Finite GPUs¶

Production fleets serve dozens of models: base + fine-tunes, multiple sizes, embedding models, a moderation classifier, several customer-specific adapters. Giving each its own always-on dedicated GPUs is simple but ruinously expensive when most are lightly used. Three strategies, from cheapest-but-slowest to most-isolated.

LoRA / adapter multiplexing (best for fine-tunes). If twenty models are LoRA fine-tunes (PEFT I: LoRA, QLoRA, DoRA & The Adapter Family) of one base, you load the base weights once and swap only the small low-rank adapter matrices per request — even within a single batch, different requests can use different adapters (multi-LoRA batching, as in vLLM’s S-LoRA-style support). One GPU’s worth of base weights serves twenty “models” at near-zero marginal memory. This is the highest-leverage multi-model trick when it applies. Operationally it is vllm serve <base> --enable-lora --max-loras 8 --max-lora-rank 32 --lora-modules sql=/adapters/sql legal=/adapters/legal, after which clients pass "model": "sql" and vLLM batches both adapters’ requests together; the mechanics and its scaling limits are in Multi-Tenant LoRA & Adapter Serving at Scale.

Time-sharing with hot-swap (best for many independent models, spiky traffic). Keep a pool of GPUs and load/evict full model weights on demand, treating HBM like a cache with an LRU policy keyed on recent traffic. The cost is the cold-start latency on a cache miss; mitigate it by keeping the top-\(k\) models pinned and streaming weights fast from a local NVMe tier.

Static partitioning (best for a few high-QPS models). Give each big, busy model its own dedicated pool. Simplest to reason about, best isolation (no noisy-neighbor), worst utilization. Reserve it for the handful of models that are busy enough to justify full GPUs.

from collections import OrderedDict

class ModelLRUCache:
    """
    HBM-as-cache for whole-model hot-swapping. Keeps total loaded weights under
    `capacity_gb`; evicts the least-recently-used model on a miss. Production
    systems pin the top-k busiest models so they are never evicted (anti-thrash).
    """
    def __init__(self, capacity_gb, pinned=()):
        self.capacity_gb = capacity_gb
        self.loaded = OrderedDict()    # model_id -> size_gb, MRU at the end
        self.pinned = set(pinned)
        self.used_gb = 0.0

    def get(self, model_id, size_gb, load_fn):
        if model_id in self.loaded:
            self.loaded.move_to_end(model_id)   # mark most-recently-used
            return  # hot path: already resident, zero cold-start
        # Cache miss: evict LRU (never a pinned model) until it fits.
        while self.used_gb + size_gb > self.capacity_gb:
            for victim in list(self.loaded):
                if victim not in self.pinned:
                    self.used_gb -= self.loaded.pop(victim)
                    break
            else:
                raise MemoryError("cannot fit model; all residents pinned")
        load_fn(model_id)                # SLOW: weights -> HBM (the cold start)
        self.loaded[model_id] = size_gb
        self.used_gb += size_gb

The decision tree is: Are they adapters of one base? → multiplex. Are they distinct but individually low-QPS? → time-share with hot-swap. Are a few of them high-QPS? → give those static pools and time-share the long tail. Most real fleets do all three at once.

Putting It Together: An End-to-End Request Trace¶

To consolidate, here is the life of one streaming chat request through the whole system.

Every numbered step maps to a layer we designed. Notice how the same request touches quota (gateway), placement and cache affinity (router), admission and SLO prediction (admit), batching policy (replica), and feeds the autoscaler and metering — these are not separate systems, they are one control loop around the GPUs.

Further reading¶

• Yu et al., Orca: A Distributed Serving System for Transformer-Based Generative Models (OSDI 2022) — the paper that introduced iteration-level (continuous) batching.
• Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention (the vLLM paper, SOSP 2023).
• Zheng et al., SGLang: Efficient Execution of Structured Language Model Programs — RadixAttention and prefix-cache-aware serving.
• Patel et al., Splitwise: Efficient Generative LLM Inference Using Phase Splitting — the case for prefill/decode disaggregation.
• Agrawal et al., Sarathi-Serve (Taming Throughput-Latency Tradeoff with chunked prefill) — chunked prefill and the TTFT/TPOT tradeoff.
• Sheng et al., S-LoRA: Serving Thousands of Concurrent LoRA Adapters — multi-adapter batching for multi-model serving.
• Mitzenmacher, The Power of Two Choices in Randomized Load Balancing — the load-balancing result behind power-of-d-choices routing.
• Kleinrock, Queueing Systems, Volume 1: Theory — Little’s Law and the \(1/(1-\rho)\) behavior, the foundation of capacity planning.

Exercises¶

1. The chapter insists that “a single latency SLO is meaningless” for LLM serving and that you need two numbers. Name the two user-perceived latency quantities, say which layer/phase dominates each, and explain why an SLO written only as “p99 end-to-end latency \(\le\) 10 s” could be satisfied by a system that users nonetheless experience as broken.

2. Using the M/M/1 approximation from the chapter, \(W = (1/\mu)\,/\,(1-\rho)\), take a replica whose mean service time is \(1/\mu = 80\) ms. Compute the mean time in system \(W\) at utilizations \(\rho = 0.70\), \(0.90\), and \(0.95\). By what factor does \(W\) grow going from \(\rho=0.70\) to \(\rho=0.95\)? Use the result to justify the chapter’s rule of provisioning to 60-75% utilization rather than 95%.

3. Size the KV cache for a hypothetical model on one 80 GB GPU. The model has \(L = 48\) layers, \(H_{kv} = 8\) KV heads (GQA), head dimension \(d_h = 128\), served in bf16 (2 bytes/element). Weights occupy 30 GB and you reserve 4 GB for activations and allocator overhead. For an average context of 4,096 tokens per request, how many concurrent requests fit? Then state what changes if you switch the KV cache to fp8.

4. Combine the chapter’s two independent sizing constraints. A streaming chat model sees peak \(\lambda = 40\) requests/sec, and the average request lives in the system for \(W = 20\) s (long generations). Each replica prefills at a service rate of \(\mu = 12.5\) requests/sec and, from Exercise 3, holds \(N_{\text{concurrent}} = 57\) sequences in KV. Compute (a) the throughput-bound replica count at target utilization \(\rho = 0.70\), and (b) the memory/concurrency-bound replica count via Little’s Law. Which constraint binds, and what does that tell you about the workload’s regime?

5. (Implementation.) The chapter gives max_concurrent_requests(...) for memory sizing but leaves the “take the max of the two constraints” step (from the aside and from Exercise 4) as prose. Implement a function fleet_size(...) that computes both replica counts and returns the binding one plus a label of the regime. Reuse max_concurrent_requests from the chapter, and drive the throughput side from \(\lambda\), per-replica \(\mu\), and a target utilization; drive the memory side from Little’s Law (\(L = \lambda W\)). Show it on the Exercise 4 numbers.

import math

def kv_bytes_per_token(num_layers, num_kv_heads, head_dim, bytes_per_elem=2):
    return 2 * num_layers * num_kv_heads * head_dim * bytes_per_elem

def max_concurrent_requests(hbm_gb, weight_gb, reserve_gb,
                            num_layers, num_kv_heads, head_dim,
                            avg_ctx_tokens, bytes_per_elem=2):
    free_bytes = (hbm_gb - weight_gb - reserve_gb) * 1e9
    per_tok = kv_bytes_per_token(num_layers, num_kv_heads, head_dim, bytes_per_elem)
    per_request = per_tok * avg_ctx_tokens
    return int(free_bytes // per_request)

def fleet_size(arrival_rate,          # lambda, requests/sec (peak)
               mean_time_in_system,   # W, seconds (enqueue -> last token)
               service_rate,          # mu, requests/sec one replica sustains
               target_utilization,    # rho ceiling, e.g. 0.70
               n_concurrent_per_replica):
    """
    Take the MAX of the two independent sizing constraints:
      - throughput:  size so peak load sits at target utilization
      - memory:      Little's Law concurrency / per-replica KV capacity
    Returns (replicas, binding_regime).
    """
    # Throughput-bound: c >= lambda / (rho * mu)
    c_tput = math.ceil(arrival_rate / (target_utilization * service_rate))

    # Memory-bound: L = lambda * W concurrent seqs, each replica holds N.
    concurrency = arrival_rate * mean_time_in_system      # Little's Law
    c_mem = math.ceil(concurrency / n_concurrent_per_replica)

    if c_mem >= c_tput:
        return c_mem, "memory-bound"
    return c_tput, "throughput-bound"

# --- Exercise 4 numbers -------------------------------------------------
N = max_concurrent_requests(
    hbm_gb=80, weight_gb=30, reserve_gb=4,
    num_layers=48, num_kv_heads=8, head_dim=128,
    avg_ctx_tokens=4096, bytes_per_elem=2,
)
print(N)  # -> 57 concurrent sequences per replica

replicas, regime = fleet_size(
    arrival_rate=40, mean_time_in_system=20,
    service_rate=12.5, target_utilization=0.70,
    n_concurrent_per_replica=N,
)
print(replicas, regime)   # -> 15 memory-bound