7.3 vLLM: Architecture, PagedAttention & Internals¶

If there is a single piece of open-source software that defined the era of practical LLM serving, it is vLLM. Born from a 2023 Berkeley paper (Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention), it took an idea borrowed from operating systems — paging — and applied it to the one resource that dominates LLM inference memory: the KV cache. The result was a throughput jump of several times over the contemporary baselines, and within a year vLLM became the default backend for an enormous fraction of self-hosted inference, RL rollout engines, and managed APIs.

This chapter is a reference on how vLLM actually works inside. We assume you already understand the mechanics of prefill, decode, and the KV cache from The Anatomy of LLM Inference: Prefill, Decode & The KV Cache, and the idea of running many requests in one forward pass from Continuous Batching & Request Scheduling. PagedAttention itself — the kernel and the memory-management algorithm — is developed in depth in PagedAttention & KV-Cache Memory Management; here we recap it just enough to understand vLLM’s system, then spend most of our time on the block manager, the scheduler, the engine/executor stack, the rewritten V1 architecture, and the features that make vLLM a production engine: prefix caching, speculative decoding, and multi-LoRA. We close with how to actually run and tune it.

Why the KV cache is the bottleneck (and what PagedAttention fixes)¶

Recall the shape of autoregressive inference. Prefill runs the whole prompt through the network in one parallel pass and produces a key and value vector for every layer and every prompt token; those are stored in the KV cache. Decode then generates one token at a time, and each new token appends one more K and V vector per layer. The cache grows by one token-slot per step and is read in full on every step.

The per-request KV cache size is exact and worth memorizing:

where \(L\) is the number of layers, \(H_{kv}\) the number of key/value heads (after GQA/MQA — see Multi-Head Attention, MQA, GQA & MLA), \(d_h\) the head dimension, \(S\) the sequence length, \(b\) the bytes per element, and the leading \(2\) counts keys and values.

The systems problem is not the total — it is the dynamics. A request’s final length is unknown when it arrives. Pre-vLLM engines handled this by pre-allocating a contiguous buffer for each request sized to max_seq_len. This is catastrophic for memory:

• Internal fragmentation. A request that generates 50 tokens but was allocated 2048 slots wastes 97% of its reservation for its entire lifetime.
• External fragmentation. Contiguous buffers of different sizes leave unusable gaps between them, exactly like a poorly managed heap.
• No sharing. Two requests with an identical system prompt each keep their own full copy.

The vLLM paper measured that in such systems only ~20–40% of KV memory held actual tokens; the rest was reserved-but-empty or fragmented. Since KV memory caps the number of concurrent requests, and concurrency caps throughput, this waste translated almost linearly into lost throughput.

PagedAttention in one paragraph¶

The operating-systems answer to fragmentation is virtual memory with paging: divide memory into fixed-size pages, let a process see a contiguous virtual address space, and map each virtual page to any physical page through a page table. PagedAttention applies this exactly. The KV cache is carved into fixed-size blocks (vLLM’s term for pages), each holding the K and V vectors for a fixed number of tokens — the block size, commonly 16. A request’s logical sequence of tokens maps, via a block table, to a list of physical blocks that need not be contiguous in GPU memory.

The payoff:

• Near-zero internal waste. Only the last partially-filled block of each sequence is under-utilized — at most block_size - 1 slots, independent of max_seq_len. With block size 16 the worst-case waste per sequence is 15 token-slots, versus thousands before.
• No external fragmentation. All blocks are the same size, so any free block fits any request. Allocation is O(1) off a free list.
• Copy-on-write sharing. Two sequences can point their block tables at the same physical block. This is the mechanism behind shared prompts and parallel sampling.

The cost is a custom attention kernel: instead of reading K and V from one contiguous tensor, the kernel must gather them block-by-block using the block table. That gather is the PagedAttention CUDA kernel. The arithmetic of attention is unchanged — only the addressing is indirected. We develop that kernel in PagedAttention & KV-Cache Memory Management; here we treat it as a primitive and build the engine around it.

The block manager: paging for the KV cache¶

The block manager (in V1, the KVCacheManager) is vLLM’s memory allocator. It owns the pool of physical KV blocks and hands them out to sequences. Think of it as malloc/free plus a page-table — but tuned for the append-only, highly-shareable access pattern of LLM decode.

How many blocks exist?¶

At startup vLLM runs a memory profiling pass. It loads the model weights, runs a dummy forward at the configured maximum batch, and measures peak activation memory. Whatever is left of GPU memory — up to a fraction set by gpu_memory_utilization (default ~0.9) — is dedicated to the KV cache. Dividing that by the bytes-per-block gives the total number of physical blocks. This is why vLLM “grabs” most of your GPU on launch: it is pre-carving the entire KV pool so allocation at runtime is just popping from a free list.

# Sketch of vLLM's startup KV-budget computation (numbers illustrative).
total_gpu_bytes      = 80 * 1024**3          # 80 GB GPU (e.g. A100/H100)
weight_bytes         = 14 * 1024**3          # ~14 GB for a 7B model in bf16
peak_activation      = 4  * 1024**3          # measured by a profiling forward pass
util                 = 0.90                  # gpu_memory_utilization

usable               = int(total_gpu_bytes * util)        # ~72 GB
kv_cache_bytes       = usable - weight_bytes - peak_activation

# Bytes for one block: 2 (K and V) * block_size * num_kv_heads * head_dim
#                       * num_layers * dtype_bytes
block_size, n_kv_heads, head_dim = 16, 8, 128
n_layers, dtype_bytes            = 32, 2     # bf16
bytes_per_block = 2 * block_size * n_kv_heads * head_dim * n_layers * dtype_bytes

num_gpu_blocks = kv_cache_bytes // bytes_per_block
print(f"KV pool: {kv_cache_bytes/1024**3:.1f} GB -> {num_gpu_blocks} blocks "
      f"({num_gpu_blocks * block_size} token-slots)")

The total token-slots (num_gpu_blocks * block_size) is the hard ceiling on how many tokens of KV cache can exist across all running requests at once. The scheduler’s whole job is to keep total demand under this ceiling.

The free list and the block table¶

The allocator maintains a free list of physical block IDs. Each sequence keeps a block table: an ordered list mapping its logical block index to a physical block ID. The core operations:

class BlockManager:
    """Simplified KV-block allocator (V0-flavoured for clarity)."""

    def __init__(self, num_blocks: int, block_size: int):
        self.block_size = block_size
        self.free_blocks = list(range(num_blocks))   # the free list
        self.ref_count = {}                          # phys_block -> refcount

    def can_allocate(self, seq_len: int) -> bool:
        n = (seq_len + self.block_size - 1) // self.block_size
        return len(self.free_blocks) >= n

    def allocate(self, seq_len: int) -> list[int]:
        n = (seq_len + self.block_size - 1) // self.block_size
        table = []
        for _ in range(n):
            blk = self.free_blocks.pop()             # O(1)
            self.ref_count[blk] = 1
            table.append(blk)
        return table                                  # this is the block table

    def can_append(self, seq) -> bool:
        """Room for one more decode token? A fresh block is only needed when
        the current tokens exactly fill the last block."""
        boundary = seq.num_tokens % self.block_size == 0
        return len(self.free_blocks) >= (1 if boundary else 0)

    def append_slot(self, block_table: list[int], cur_len: int) -> int | None:
        """Called each decode step. `cur_len` is the number of tokens already
        stored; the token about to be generated lands at position `cur_len`.
        Grow the table by one block ONLY when that position starts a fresh
        block (cur_len % block_size == 0); otherwise the last block still has
        a free slot and we return None. (Mirrors 4.6's BlockManager.append_token:
        without this guard we would pop one block per sequence *per step*, a
        block_size-fold over-allocation.)"""
        if cur_len % self.block_size != 0:
            return None                              # room in the last block
        blk = self.free_blocks.pop()                 # boundary: need a new block
        self.ref_count[blk] = 1
        block_table.append(blk)
        return blk

    def free(self, block_table: list[int]) -> None:
        for blk in block_table:
            self.ref_count[blk] -= 1
            if self.ref_count[blk] == 0:             # last owner releases it
                self.free_blocks.append(blk)

Two design points carry their weight:

• Lazy, incremental growth. A sequence allocates one block at a time as decode crosses block boundaries (append_slot). It never reserves space for tokens it has not yet produced. This is the direct cure for internal fragmentation.
• Reference counting enables sharing. A physical block can be referenced by multiple sequences. free only returns a block to the pool when its refcount hits zero. This single mechanism powers parallel sampling, beam search, and prefix caching.

Copy-on-write for shared blocks¶

When two sequences share a block (refcount > 1) and one of them needs to write into it — e.g. two samples diverged and one wants to append a token into a block the other still reads — vLLM does copy-on-write: allocate a fresh block, copy the shared block’s contents into it, point the writer’s block table at the copy, and decrement the original’s refcount. Identical to COW in fork(). It means shared prefixes cost memory only once until the moment of divergence.

def append_with_cow(self, block_table, idx):
    """Ensure block_table[idx] is writable; copy if shared."""
    blk = block_table[idx]
    if self.ref_count[blk] > 1:                # shared -> must not clobber
        new_blk = self.free_blocks.pop()
        # GPU-side: copy KV contents of `blk` into `new_blk`
        self.ref_count[blk]   -= 1
        self.ref_count[new_blk] = 1
        block_table[idx] = new_blk             # writer now owns its private copy
        return ("copy", blk, new_blk)          # scheduler emits a copy op
    return ("noop",)

The scheduler: continuous batching under a memory budget¶

vLLM’s scheduler is the brain. On every engine step it decides which sequences run in the next forward pass, subject to the block budget. It implements continuous batching (also called iteration-level scheduling): requests join and leave the running batch at token granularity, so a finished sequence frees its slot mid-flight and a waiting request fills it on the very next step. The general technique is covered in Continuous Batching & Request Scheduling; here we look at vLLM’s specific policy.

The three queues and the step loop¶

Sequences live in three states:

• WAITING — admitted but no KV blocks yet (never run, or preempted-and-recomputed).
• RUNNING — has KV blocks; will be in the next forward pass.
• SWAPPED — preempted, its KV blocks evicted to CPU RAM (V0 swap path).

A schedule step, conceptually:

def schedule_step(self):
    scheduled, blocks_to_copy = [], []

    # 1. Keep RUNNING sequences going; each decode step may need +1 block.
    for seq in self.running:
        while not self.block_mgr.can_append(seq):
            # Out of memory: preempt the lowest-priority running seq.
            victim = self.running.pop()              # tail = newest/lowest prio
            self._preempt(victim)                    # recompute or swap
            if victim is seq:
                break
        if seq in self.running:
            self.block_mgr.append_slot(seq.block_table, seq.num_tokens)
            scheduled.append(seq)

    # 2. Admit WAITING sequences (prefill) if budget and token quota allow.
    while self.waiting and self._budget_left(scheduled):
        seq = self.waiting[0]
        if not self.block_mgr.can_allocate(seq.prompt_len):
            break                                    # not enough free blocks
        self.waiting.popleft()
        seq.block_table = self.block_mgr.allocate(seq.prompt_len)
        seq.state = RUNNING
        scheduled.append(seq)

    return scheduled, blocks_to_copy

The subtlety in step 1 is preemption. Decode is monotonic: every running sequence needs at most one new block per step, and once you are decoding you cannot pause a sequence without losing forward progress. So when the pool is exhausted, vLLM evicts a whole sequence to make room, with two recovery strategies:

• Recomputation (default in V1). Drop the victim’s KV blocks entirely and move it back to WAITING. When rescheduled, re-run prefill over its prompt plus tokens generated so far. Wastes compute but frees memory instantly and needs no CPU transfer. Because prefill is compute-bound and fast, this is usually the better choice.
• Swapping (V0). Copy the victim’s KV blocks to pinned CPU memory, free the GPU blocks, and copy back when rescheduled. Saves recompute but pays PCIe bandwidth twice.

Prefill, decode, and chunked prefill¶

A naive scheduler runs either a batch of prefills or a batch of decodes. The trouble: a long prefill (say a 32k-token prompt) monopolizes a forward pass and stalls every decoding request, spiking inter-token latency for everyone — a head-of-line blocking problem.

Chunked prefill fixes this by splitting a long prompt into chunks of at most max_num_batched_tokens and processing one chunk per step, interleaved with decodes of other requests in the same batch. A single forward pass thus mixes a slice of prefill tokens with many one-token decodes. This smooths inter-token latency and keeps the GPU busy (decode alone is memory-bound and under-utilizes compute; mixing in prefill tokens raises arithmetic intensity — see the roofline view in The Roofline Model & Performance Engineering). In V1 chunked prefill is on by default and the prefill/decode distinction largely dissolves into a single “token budget” per step. The disaggregation alternative — running prefill and decode on separate machines — is covered in Disaggregated Prefill/Decode & Chunked Prefill.

The engine and executor stack¶

Around the scheduler sits the request-handling and execution machinery. From the outside in:

• API server / AsyncLLM is the outermost layer when you run vllm serve: a FastAPI app implementing the OpenAI /v1/completions, /v1/chat/completions, /v1/embeddings and /metrics routes, backed by the asynchronous engine wrapper (AsyncLLM in V1, AsyncLLMEngine in V0) that turns each HTTP request into a per-request async generator of output tokens. It applies the chat template, runs the tokenizer, and — for JSON-schema or grammar-constrained requests — attaches the logit-processing backend covered in Structured & Constrained Generation. In offline mode the LLM class replaces this layer with a synchronous loop over a list of prompts.

• LLMEngine / EngineCore is the orchestrator. It owns the scheduler and the block manager, accepts requests, drives the step loop, runs the tokenizer/detokenizer, and streams outputs. In V1 the heavy loop runs in a dedicated EngineCore process, talking to the front end over a ZeroMQ IPC channel, so Python overhead (tokenization, HTTP, detokenization, scheduling) overlaps with GPU execution.

• Executor abstracts where and how the model runs: a single process for one GPU, a multiprocessing or Ray-based executor for tensor/pipeline parallelism across GPUs and nodes. It broadcasts the ExecuteModelRequest to every worker and gathers results. Multi-GPU details are in Multi-GPU & Multi-Node Inference.

• Worker owns one GPU’s slice of the model and its KV blocks. It holds the ModelRunner, which turns the scheduler’s abstract plan into concrete GPU work: it flattens token IDs and positions into tensors, assembles per-sequence block tables and slot mappings, runs the forward pass through the model with the chosen attention backend (FlashAttention, FlashInfer, or the PagedAttention kernel), and samples the next tokens.

The forward step in detail¶

Each engine step, the ModelRunner builds inputs for a batch that may mix prefill chunks and decodes. The crucial tensors are the slot mapping (for each token being written, the exact physical KV slot to write its K and V into) and the block tables (for each sequence, where to read prior KV from).

# Conceptual ModelRunner input assembly for one step.
def prepare_inputs(scheduled_seqs, block_size):
    input_ids, positions, slot_mapping = [], [], []
    block_tables = []                         # padded [num_seqs, max_blocks]

    for seq in scheduled_seqs:
        new_tokens = seq.tokens_to_process()  # prompt chunk (prefill) or 1 (decode)
        start = seq.num_computed_tokens
        for i, tok in enumerate(new_tokens):
            pos = start + i
            input_ids.append(tok)
            positions.append(pos)
            # Where does this token's K/V get written?
            logical_block = pos // block_size
            offset        = pos %  block_size
            phys_block    = seq.block_table[logical_block]
            slot_mapping.append(phys_block * block_size + offset)
        block_tables.append(seq.block_table)

    return {
        "input_ids":    torch.tensor(input_ids,    device="cuda"),
        "positions":    torch.tensor(positions,    device="cuda"),
        "slot_mapping": torch.tensor(slot_mapping, device="cuda"),
        "block_tables": pad_and_stack(block_tables, device="cuda"),
    }

Inside each attention layer, the freshly computed K and V are scattered into the cache at slot_mapping, then PagedAttention reads K and V for the whole context by gathering blocks via block_tables. The MLP and norms are ordinary dense ops. Because the same kernel handles any mix of prefill and decode tokens, the scheduler is free to pack them however it likes.

CUDA graphs and torch.compile¶

Decode steps are tiny — one token per sequence — so per-kernel launch overhead (the CPU cost of dispatching each CUDA kernel) becomes a large fraction of step time. vLLM captures the decode forward pass into a CUDA graph: the sequence of kernel launches is recorded once for a given batch shape and replayed as a single GPU submission, slashing CPU overhead. Because graphs are shape-specialized, vLLM captures a set of graphs for a ladder of batch sizes and pads each real batch up to the nearest captured size. V1 deepens this with a torch.compile-based path (piecewise compilation that leaves attention as a custom op while compiling the rest), further fusing kernels. See Kernel Fusion, torch.compile, CUDA Graphs & Compilers. The flags enforce_eager=True (disable graphs, easier debugging, slower) and --cuda-graph-sizes control this.

V1: the rewritten architecture¶

In 2024–2025 vLLM was substantially re-architected as V1, now the default engine. V0 had accreted features over a fast-moving codebase, and its single-process design left GPU bubbles whenever the CPU was busy tokenizing, scheduling, or detokenizing. V1’s themes:

• Isolated EngineCore process. The scheduler + block manager + model execution loop run in their own process, communicating with the API/tokenizer side over a fast IPC channel. This overlaps CPU work with GPU execution — while the GPU runs step \(t\), the front end is already tokenizing requests and detokenizing outputs for step \(t-1\)/\(t+1\). The result is fewer GPU idle bubbles and higher utilization, especially at high request rates.

• Unified scheduler with a token budget. V1 removes the rigid “prefill batch vs decode batch” split. The scheduler simply allocates a per-step token budget (max_num_batched_tokens) across whatever sequences want to run, mixing prefill chunks and decodes freely. Chunked prefill becomes the default, not an option.

• Prefix caching on by default. V1’s KV-cache manager treats cached prefixes as a first-class citizen (next section), with low enough overhead that it is enabled out of the box rather than opt-in.

• torch.compile + piecewise CUDA graphs as the default execution path, replacing much hand-written V0 glue and improving portability across hardware backends.

• Cleaner extension points for new attention backends, hardware (the Platform abstraction for NVIDIA/AMD/TPU/CPU), speculative decoding, and structured output.

For the user, V1 is mostly transparent: the LLM(...) and OpenAI-server interfaces are unchanged. What you observe is higher throughput, lower overhead at small batch sizes, and prefix caching helping for free. If you need a V0-only feature during a migration you can sometimes force the old engine, but new development targets V1.

Prefix caching: reuse KV across requests¶

Many requests share a prefix: a long system prompt, a few-shot exemplar block, a chat history that grows by one turn, or — in agentic and RL workloads — a giant fixed instruction reused across thousands of rollouts. Recomputing that prefix’s KV every time is pure waste. Automatic prefix caching (APC) lets vLLM reuse the KV blocks of a previously computed prefix across requests, turning an expensive prefill into a cheap cache hit.

How it works: hashing blocks¶

Because the KV cache is already block-structured, sharing is natural. vLLM computes a hash for each full block that incorporates the token IDs in that block and the hash of all preceding blocks — a rolling hash that makes the block ID a function of the entire prefix up to that point. (Two prefixes that diverge at token 5 get different hashes from block 0 onward; two identical prefixes get identical hashes block-for-block.)

def block_hash(prev_hash, token_ids_in_block, extra=None):
    """Hash of a *full* block = (hash of everything before) + (this block's
    tokens) + optional extras (LoRA id, multimodal hash, cache salt)."""
    return hash((prev_hash, tuple(token_ids_in_block), extra))

def hash_prompt_blocks(prompt_ids, block_size, lora_id=None):
    hashes, h = [], None
    for i in range(0, len(prompt_ids) - block_size + 1, block_size):
        block = prompt_ids[i:i + block_size]      # only FULL blocks are hashable
        h = block_hash(h, block, extra=lora_id)
        hashes.append(h)
    return hashes

The block manager keeps a map from block_hash -> physical_block_id for blocks whose contents are “committed” (full and immutable). When a new request arrives:

1. Hash its prompt blocks.
2. For each leading block whose hash is already in the map, reuse that physical block: point the new request’s block table at it and bump the refcount. No compute, no new memory.
3. At the first block that misses, stop matching; allocate fresh blocks for the remaining (uncached) tokens and prefill only those.

A new request with a 4000-token cached system prompt and 50 new tokens prefills only ~50 tokens instead of 4050 — a roughly \(80\times\) reduction in prefill work for that request, and a large drop in time-to-first-token.

Eviction and correctness¶

Cached blocks still occupy the pool. When the pool is full and a new allocation is needed, vLLM evicts cached blocks that no running sequence references (refcount via the cache, not active use), typically LRU with awareness of how deep in a prefix the block sits (evict leaves before roots so popular shared prefixes survive). Eviction is lossless: a re-request just recomputes.

Correctness hinges on the hash covering everything that affects the KV: token IDs, position (implied by block order), the active LoRA adapter (different adapters produce different KV — see multi-LoRA below), and a cache salt you can set to isolate tenants so one user’s cached prefix can never be served to another. Multimodal inputs hash their image/audio features too. Get this wrong and you would serve one request’s KV to another — a correctness and security bug — which is why the hashing is conservative. Prefix caching has its own dedicated chapter, Prefix Caching & KV-Cache Reuse; here the point is that it falls out almost for free from the paged design plus reference counting.

Speculative decoding support¶

Decode is memory-bound: each step you load the entire model’s weights from HBM to produce one token. The hardware could do far more arithmetic per byte loaded. Speculative decoding exploits this by having a cheap drafter propose several future tokens, then verifying them with the big model in a single forward pass; accepted tokens come “for free” because verifying \(k\) tokens costs about the same memory traffic as generating one. The algorithms (draft models, Medusa, EAGLE, n-gram/lookahead) are the subject of Speculative Decoding: Draft Models, Medusa, EAGLE & Lookahead; here we note vLLM’s plumbing.

vLLM supports several proposers behind a common interface:

• Draft model — a small model of the same family proposes tokens.
• N-gram / prompt lookahead — propose by matching recent context against the prompt; free, great for tasks with copying (summarization, code editing).
• EAGLE / Medusa-style — lightweight heads on the target model predict multiple future tokens.

Each step, vLLM runs the proposer to get \(k\) draft tokens, then runs the target model on all \(k+1\) positions at once. A rejection-sampling verifier accepts the longest prefix of drafts consistent with the target’s distribution and corrects the first rejected token, so the output distribution is provably identical to plain sampling from the target — speculation changes speed, not what is generated.

# Engine-level shape of one speculative step (target distribution preserved).
draft_tokens, draft_probs = proposer.propose(seq, k)          # cheap
target_logits = target_model(seq.context + draft_tokens)      # ONE big fwd, k+1 pos
accepted = rejection_sample(draft_tokens, draft_probs, target_logits)  # 0..k accepted
seq.extend(accepted)
seq.extend([sample(target_logits[len(accepted)])])            # +1 bonus/correction token
# Net: up to k+1 tokens emitted per single target forward pass.

The system challenge is that speculation must coexist with paged KV and continuous batching: the KV for rejected draft tokens must be discarded (their blocks/slots rolled back), and batches mix sequences accepting different numbers of tokens. V1 re-implemented spec decode to fit the unified scheduler cleanly. The speedup depends on the acceptance rate \(\alpha\) and draft cost; expected tokens per target step is roughly \(\frac{1-\alpha^{k+1}}{1-\alpha}\), so high-\(\alpha\), predictable text (code, structured output) benefits most, while creative high-entropy text benefits least — and a poor drafter can even slow you down because of the wasted draft compute.

Multi-LoRA serving¶

LoRA (low-rank adaptation — PEFT I: LoRA, QLoRA, DoRA & The Adapter Family) fine-tunes a model by adding a low-rank update \(\Delta W = BA\) to selected weight matrices, where \(B \in \mathbb{R}^{d\times r}\), \(A \in \mathbb{R}^{r\times d}\), and the rank \(r\) is tiny (8–64). The adapter is a few megabytes versus tens of gigabytes for the base model. vLLM’s multi-LoRA serving exploits this: keep one copy of the base model in GPU memory and serve many different fine-tunes simultaneously by swapping in the small adapters, even mixing requests for different adapters in the same batch.

The key kernel insight is that you do not merge adapters into the weights (that would force one adapter per batch). Instead you keep the base forward pass shared and add each request’s LoRA contribution as a batched, gathered low-rank matmul. For a layer’s output \(y = Wx\), the LoRA-augmented output for request \(i\) using adapter \(a(i)\) is:

vLLM’s Punica/SGMV-style kernels compute the \(B(Ax)\) term for a whole batch where each row may use a different adapter, gathering the right \(A, B\) per row. This makes the marginal cost of serving \(N\) adapters close to serving one, as long as the adapters fit in memory.

from vllm import LLM
from vllm.lora.request import LoRARequest

# Base model loaded once; enable LoRA with a max rank and a cap on
# how many distinct adapters may be active in a single batch.
llm = LLM(model="meta-llama/Meta-Llama-3-8B",
          enable_lora=True,
          max_loras=4,        # distinct adapters per *batch*
          max_lora_rank=16,   # must be >= rank of any adapter you load
          max_cpu_loras=32)   # adapters parked in CPU, paged to GPU on demand

# Each request names the adapter it wants. Different adapters can be
# batched together in the same forward pass.
out = llm.generate(
    ["Translate to French: Hello", "Summarize: ...long doc..."],
    lora_request=[
        LoRARequest("fr-translator", 1, "/adapters/fr_lora"),
        LoRARequest("summarizer",    2, "/adapters/sum_lora"),
    ],
)

Adapters themselves are paged like the KV cache: a pool of GPU adapter slots, with inactive adapters held in CPU memory (max_cpu_loras) and copied in on demand. max_loras caps distinct adapters per batch (kernel/memory limit); max_cpu_loras caps how many are kept warm. This is the backbone of multi-tenant “one base model, hundreds of customer fine-tunes” serving. Note the interaction with prefix caching: the LoRA ID is part of the block hash, so two requests can only share cached prefix blocks if they use the same adapter.

Running and tuning vLLM¶

Two entry points¶

Offline batched inference — drive the engine directly from Python; best for evals, dataset generation, and RL rollouts:

from vllm import LLM, SamplingParams

llm = LLM(
    model="meta-llama/Meta-Llama-3-8B-Instruct",
    tensor_parallel_size=2,          # shard across 2 GPUs (TP)
    gpu_memory_utilization=0.90,     # fraction of GPU for weights + KV
    max_model_len=8192,              # caps KV per request; lower = more concurrency
    enable_prefix_caching=True,      # default in V1; reuse shared prefixes
    dtype="bfloat16",
)
params = SamplingParams(temperature=0.7, top_p=0.95, max_tokens=256)
for o in llm.generate(["Explain PagedAttention in one sentence."], params):
    print(o.outputs[0].text)

Online OpenAI-compatible server — a drop-in replacement for the OpenAI API, used by virtually every “self-host an LLM” deployment:

# Launch an OpenAI-compatible HTTP server.
vllm serve meta-llama/Meta-Llama-3-8B-Instruct \
    --tensor-parallel-size 2 \
    --gpu-memory-utilization 0.90 \
    --max-model-len 8192 \
    --max-num-seqs 256 \
    --max-num-batched-tokens 8192 \
    --port 8000
# (chunked prefill and prefix caching are already on by default under V1.)

# Then call it exactly like the OpenAI API:
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"meta-llama/Meta-Llama-3-8B-Instruct",
       "messages":[{"role":"user","content":"Hello!"}]}'

# Scheduler and cache health, as Prometheus counters:
curl -s http://localhost:8000/metrics | grep -E 'preemption|prefix_cache|kv_cache_usage'

Serving a model vLLM has never seen¶

Everything above assumes vLLM already knows your architecture. When you have trained your own model — as we do in Evaluation & Serving: Honest Benchmarks, int4 Quantization, and Running on a Laptop for Stack-100M — you have two options: emit a checkpoint whose config.json declares an architecture vLLM already supports (LlamaForCausalLM is the pragmatic choice, since a standard pre-norm/RoPE/SwiGLU/GQA model is weight-compatible with it), or register your architecture out of tree.

The registration path is worth understanding because it exposes the model-side contract of the whole paged design. The crucial point: your model never touches the KV cache. You instantiate vllm.attention.Attention layers; vLLM allocates their KV blocks in the profiling pass and injects the slot mapping and block tables through a per-step forward context, so the same module works for a prefill chunk, a decode step, or any mix of the two.

# stack100m_vllm.py — teach vLLM an architecture it has never seen.
import torch
from torch import nn
from vllm import ModelRegistry
from vllm.attention import Attention          # the paged-KV attention layer


class Stack100MAttention(nn.Module):
    def __init__(self, cfg, prefix: str = ""):
        super().__init__()
        self.n_q, self.n_kv, self.d_h = cfg.n_heads, cfg.n_kv_heads, cfg.head_dim
        self.qkv = nn.Linear(cfg.hidden, (self.n_q + 2 * self.n_kv) * self.d_h, bias=False)
        self.o_proj = nn.Linear(self.n_q * self.d_h, cfg.hidden, bias=False)
        # vLLM owns this layer's KV cache: the block manager sizes and allocates
        # it, and the ModelRunner supplies slot_mapping / block_tables via the
        # forward context. We only hand it q, k, v.
        self.attn = Attention(num_heads=self.n_q, head_size=self.d_h,
                              scale=self.d_h ** -0.5, num_kv_heads=self.n_kv,
                              prefix=f"{prefix}.attn")

    def forward(self, positions: torch.Tensor, hidden: torch.Tensor) -> torch.Tensor:
        splits = [self.n_q * self.d_h, self.n_kv * self.d_h, self.n_kv * self.d_h]
        q, k, v = self.qkv(hidden).split(splits, dim=-1)
        q, k = apply_rope(positions, q, k)                  # your RoPE
        return self.o_proj(self.attn(q, k, v))              # paged attention


class Stack100MForCausalLM(nn.Module):
    """Three methods are the contract vLLM calls."""

    def __init__(self, *, vllm_config, prefix: str = ""):
        super().__init__()
        self.config = vllm_config.model_config.hf_config
        ...                                                  # build the stack

    # Flat token tensors, NOT [batch, seq]: the ModelRunner concatenates every
    # scheduled sequence's tokens into one ragged 1-D batch.
    def forward(self, input_ids, positions, intermediate_tensors=None,
                inputs_embeds=None) -> torch.Tensor:
        ...                                                  # -> hidden states

    def compute_logits(self, hidden_states, sampling_metadata=None):
        ...                                                  # -> [num_seqs, vocab]

    def load_weights(self, weights):
        """Consume an iterator of (checkpoint_name, tensor) and copy into params,
        handling any fused/sharded layouts (e.g. q,k,v -> one qkv matrix)."""
        ...


ModelRegistry.register_model("Stack100MForCausalLM", Stack100MForCausalLM)

Two practical notes. First, registration must happen before the engine starts and in every worker process — importing the module in your script works for a single process, but for tensor parallelism (which spawns workers) package it as a plugin exposed through the vllm.general_plugins entry point so each worker imports it too. Second, set "architectures": ["Stack100MForCausalLM"] in your checkpoint’s config.json so vLLM looks the class up. The exact forward/compute_logits signatures drift between releases; the honest way to get them right is to copy the structure of vllm/model_executor/models/llama.py from the version you have installed.

vLLM inside an RL loop: sleep mode and weight sync¶

RL post-training (veRL: HybridFlow & The Single-Controller Architecture, and Post-Training: SFT, DPO, and Narrow RLVR (GRPO) That Works at 100M) alternates between generating rollouts and training on them. Both phases want the whole GPU, and after each update the rollout engine is holding stale weights. vLLM exposes two mechanisms for exactly this, and they are why veRL, OpenRLHF and TRL’s online trainers can co-locate the actor and the sampler on one device:

from vllm import LLM

llm = LLM(model="./stack100m-sft", enable_sleep_mode=True)

rollouts = llm.generate(prompts, params)      # generation phase
llm.sleep(level=1)                            # free the KV pool, offload weights to CPU
# ... trainer now owns the GPU: compute GRPO advantages, backprop, step ...
llm.wake_up()                                 # weights back on GPU, KV pool re-carved

# Push the freshly updated policy into every TP worker without restarting:
llm.collective_rpc("update_weights", args=(handles,))

sleep(level=1) offloads weights to CPU and discards the KV cache (level 2 discards the weights too, for when the trainer will supply them anyway); wake_up() restores. collective_rpc broadcasts a method call to all workers, and frameworks register the receiving method by passing a worker_extension_cls at construction — typically one that copies tensors in place from the trainer’s sharded parameters, often over NCCL or CUDA IPC handles rather than through disk. Invalidate the prefix cache after a weight update: cached blocks hold KV computed by the old policy, and the block hash covers token IDs, not weights. Frameworks handle this by resetting the prefix cache (llm.reset_prefix_cache()) as part of the sync; if you build your own loop, do it explicitly or you will silently mix policy versions inside a single rollout.

The knobs that actually matter¶

A practical tuning loop:

1. Pick max_model_len honestly. This single number sets the worst-case KV per request. Cutting an unnecessary 32k cap down to 8k can multiply concurrency.
2. Push gpu_memory_utilization up until you see activation OOMs, then back off a notch.
3. Decide your objective. Throughput-first: large max_num_batched_tokens, large max_num_seqs. Latency-first (low inter-token latency for chat): smaller max_num_batched_tokens with chunked prefill so long prefills don’t stall decodes.
4. Watch the metrics, and measure with a load generator. The server’s /metrics endpoint exports Prometheus counters and histograms: KV-cache utilization, running/waiting counts, num_preemptions, prefix-cache hit rate, time-to-first-token and time-per-output-token distributions. If preemptions are nonzero in steady state, you’re oversubscribed — lower max_num_seqs or max_model_len, or quantize to enlarge the pool. Do not tune by eyeballing single requests: drive the server with vLLM’s own harness (vllm bench serve in recent versions, or benchmarks/benchmark_serving.py in the repo) against a real request-rate and length distribution, which reports throughput alongside TTFT/TPOT percentiles so you can see the latency cost of every throughput gain.
5. Quantize to buy concurrency. FP8/INT4 weights and an FP8 KV cache both enlarge the effective KV pool; for many workloads that concurrency gain outweighs the tiny quality cost. Inference economics — the latency/throughput/cost trade-off you are navigating — is the subject of Inference Economics: Latency, Throughput & Cost.

vLLM is not the only serving engine — SGLang: RadixAttention & Structured Programs pushes prefix sharing further with a radix tree, and TensorRT-LLM, TGI & Other Serving Stacks trades flexibility for hand-tuned NVIDIA kernels. But vLLM’s combination of a clean paged-memory core, broad model and hardware coverage, an active community, and the OpenAI-compatible surface has made it the default. Understanding its internals — the block manager, the scheduler, the executor, and how prefix caching, speculation, and multi-LoRA bolt onto the paged core — is understanding how modern open LLM serving works.

Further reading¶

• Kwon, Li, Zhuang, Sheng, Zheng, Yu, Gonzalez, Zhang, Stoica — Efficient Memory Management for Large Language Model Serving with PagedAttention (SOSP 2023). The founding vLLM paper.
• Yu, Jeong, Kim, Kim, Chun — Orca: A Distributed Serving System for Transformer-Based Generative Models (OSDI 2022). Origin of iteration-level / continuous batching.
• Dao, Fu, Ermon, Rudra, Ré — FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. The attention kernel vLLM builds on.
• Chen, Borgeaud, Irving, Lespiau, Sifre, Jumper — Accelerating Large Language Model Decoding with Speculative Sampling; and Leviathan, Kalman, Matias — Fast Inference from Transformers via Speculative Decoding.
• Chen, Ye, Zheng, et al. — Punica: Multi-Tenant LoRA Serving, and S-LoRA: Serving Thousands of Concurrent LoRA Adapters. The basis of vLLM’s multi-LoRA kernels.
• The vLLM project repository and documentation (vllm-project/vllm), including the V1 architecture design notes.

Exercises¶

1. (Conceptual.) A colleague argues that PagedAttention “just moves the fragmentation around” and cannot really help, because you still have to store the same number of KV tokens. Explain precisely why paging reduces wasted KV memory, and name the one remaining source of internal waste and its worst-case size per sequence.

2. (Quantitative — block budget.) You serve a model with \(L = 40\) layers, \(H_{kv} = 8\) KV heads (GQA), head dimension \(d_h = 128\), in bf16 (\(b = 2\) bytes), with block_size = 16. After loading weights and reserving activations, the profiling pass leaves 48 GB for the KV pool. Compute (a) the bytes per token of KV, (b) the bytes per block, © the total number of physical blocks, and (d) the total token-slots. Use \(1\text{ GB} = 1024^3\) bytes.

3. (Quantitative — concurrency and preemption.) Continuing from Exercise 2 (314,560 token-slots), suppose the average request holds 4,000 tokens of context (prompt + generated) in steady state.

(a) How many concurrent requests fit? (b) A pre-paging engine reserves max_seq_len = 16384 per request. How many fit for it, and what is the concurrency ratio? © You set max_num_seqs = 120. Argue whether this risks a preemption storm, and what metric you would watch.

4. (Implementation — fix a subtle allocator bug.) A teammate writes the per-step slot allocator below, intending to grow a sequence’s block table by one block whenever decode needs it. It over-allocates. Identify the bug, state the consequence in concrete numbers for block_size = 16, and give a corrected append_slot consistent with the chapter’s BlockManager.

def append_slot(self, block_table, cur_len):
    # `cur_len` = number of tokens already stored; new token lands at index cur_len.
    blk = self.free_blocks.pop()
    self.ref_count[blk] = 1
    block_table.append(blk)
    return blk

def append_slot(self, block_table, cur_len):
    """Grow the block table by one block ONLY when the incoming token
    starts a fresh block; otherwise the last block still has a slot."""
    if cur_len % self.block_size != 0:
        return None                          # room in the last block
    blk = self.free_blocks.pop()             # boundary: need a new block
    self.ref_count[blk] = 1
    block_table.append(blk)
    return blk

5. (Implementation — prefix-cache hashing.) Using the chapter’s rolling block_hash / hash_prompt_blocks, implement prefix_cache_lookup(prompt_ids, block_size, cache, lora_id=None) that returns (num_reused_blocks, first_miss_token_index): it walks the prompt’s full blocks in order, counts how many leading blocks are already present in cache (a dict block_hash -> physical_block_id), and stops at the first miss. Then explain, using the numbers in the chapter, why a 4000-token cached system prompt plus 50 new tokens prefills only ~50 tokens.

def block_hash(prev_hash, token_ids_in_block, extra=None):
    return hash((prev_hash, tuple(token_ids_in_block), extra))

def prefix_cache_lookup(prompt_ids, block_size, cache, lora_id=None):
    """Return (num_reused_blocks, first_miss_token_index).
    Only FULL blocks are hashable/cacheable; a trailing partial block
    is never a cache hit."""
    num_reused, h = 0, None
    n_full = len(prompt_ids) // block_size
    for i in range(n_full):
        block = prompt_ids[i * block_size:(i + 1) * block_size]
        h = block_hash(h, block, extra=lora_id)   # rolling: folds in prefix
        if h in cache:
            num_reused += 1                        # leading hit: reuse block
        else:
            break                                  # first miss: stop matching
    return num_reused, num_reused * block_size

6. (Quantitative — speculative decoding.) A drafter proposes \(k = 4\) tokens per step with per-token acceptance rate \(\alpha = 0.8\). The chapter gives the expected number of tokens emitted per target forward pass as \(\frac{1 - \alpha^{k+1}}{1 - \alpha}\).

(a) Compute the expected tokens per target step, and the speedup over plain decoding (1 token/step) ignoring draft cost. (b) The drafter costs 20% of a target forward pass per step. If a spec step costs \(1 + 0.20 = 1.2\) target-passes of time, what is the net speedup? © Give one qualitative reason a low-\(\alpha\) (e.g. creative, high-entropy) workload can make speculation a net loss.