7.14 Multi-Tenant LoRA & Adapter Serving at Scale¶

Imagine you run a platform where every customer fine-tunes their own model. A legal-tech firm wants a contracts assistant; a game studio wants a lore-keeping NPC; a hospital wants a discharge-summary drafter. The naïve answer is “give each customer their own GPU running their own fine-tuned model.” With a 13-billion-parameter base in bf16 weighing roughly 26 GB, that means one (or several) dedicated 80 GB GPU per tenant, sitting near-idle whenever that tenant is quiet. At a thousand tenants this is economically absurd: you are paying for thousands of GPUs to serve a request rate that, aggregated, might fit on twenty.

The escape hatch is Low-Rank Adaptation (LoRA). Each tenant’s fine-tune is not a fresh 26 GB of weights — it is a handful of tiny low-rank matrices, often 10–200 MB, layered on top of a single shared base model. If we can keep one copy of the base resident on the GPU and swap in the right small adapter per request, we can serve hundreds-to-thousands of distinct fine-tuned models from a single base deployment. The catch: a production batch contains requests for many different adapters at once. Serving them efficiently — without looping over adapters one at a time and destroying GPU utilization — is the central systems problem of this chapter.

This chapter assumes you understand LoRA’s math from PEFT I: LoRA, QLoRA, DoRA & The Adapter Family and the inference anatomy from The Anatomy of LLM Inference: Prefill, Decode & The KV Cache and Continuous Batching & Request Scheduling. We build up from the LoRA forward pass, derive the batched heterogeneous-adapter kernel (Punica’s SGMV), study the S-LoRA / LoRAX architectures and their adapter registries, hot-swapping and tiered storage, adapter sharding under tensor parallelism, unified vs. disaggregated execution, cross-model prefix-cache reuse, and the throughput/SLO/fairness trade-offs that govern real deployments. We close with a from-scratch batched multi-adapter forward in PyTorch.

7.14.1 The LoRA Forward Pass and Why Multi-Tenancy Is Hard¶

Recall the LoRA reparameterization. For a frozen base weight matrix \(W_0 \in \mathbb{R}^{d_\text{out} \times d_\text{in}}\), LoRA adds a low-rank update:

where \(A \in \mathbb{R}^{r \times d_\text{in}}\) and \(B \in \mathbb{R}^{d_\text{out} \times r}\), the rank \(r \ll \min(d_\text{out}, d_\text{in})\) (typically 8–64), and \(\alpha\) is a scaling hyperparameter. For an input activation \(x \in \mathbb{R}^{d_\text{in}}\), the layer output is

The crucial structural fact is the parenthesization: we never materialize \(\Delta W = BA\) (that would be a full \(d_\text{out} \times d_\text{in}\) matrix, as large as the base). Instead we compute \(Ax\) first — a cheap projection down to dimension \(r\) — then \(B(Ax)\), a projection back up. The LoRA path costs \(r(d_\text{in} + d_\text{out})\) multiply-adds versus \(d_\text{in} d_\text{out}\) for the base; for \(r=16\) on a \(4096 \times 4096\) projection, that is about 0.8% extra compute.

The single-adapter trap: merging¶

If you only ever serve one adapter, the optimal move is to merge it into the base: precompute \(W = W_0 + \frac{\alpha}{r}BA\) once, and from then on you run a plain dense model with zero LoRA overhead. This is what peft’s merge_and_unload() does. Merging is perfect for single-tenant serving.

But merging is a trap for multi-tenancy. If a batch contains requests for adapters \(\mathcal{A}_1, \mathcal{A}_2, \dots, \mathcal{A}_k\), a merged weight can only embody one of them. You would have to either (a) run \(k\) separate forward passes, one per adapter, each with batch size \(\tfrac{B}{k}\) — collapsing the batch and wrecking throughput — or (b) re-merge and un-merge the base weights between micro-batches, which means rewriting tens of gigabytes of GPU memory per step. Both are catastrophic. The entire field of multi-tenant LoRA serving exists to avoid them.

The batched-heterogeneous problem, stated precisely¶

Continuous batching (the foundation of vLLM/SGLang/TGI) assembles a batch of tokens from many requests at different decode positions. In multi-tenant LoRA, each request carries an adapter ID. So a single batched matrix multiply now looks like this: we have a stacked activation tensor \(X \in \mathbb{R}^{B \times d_\text{in}}\) where row \(i\) belongs to adapter \(a_i \in \{1, \dots, N\}\). The base contribution \(X W_0^\top\) is shared and trivially batched. The LoRA contribution is not:

Every row may use a different \(A\), \(B\), rank \(r\), and scale \(\alpha\). This is a grouped or segmented matrix multiply: the batch is partitioned into segments by adapter, and each segment multiplies against its own pair of small matrices. Standard torch.bmm cannot express it efficiently when segment sizes are wildly uneven (one popular adapter might have 200 rows in the batch; a cold one, 1). The kernel that solves this is the heart of the chapter.

7.14.2 Punica & SGMV: The Batched Multi-Adapter Kernel¶

Punica (Chen et al., Punica: Multi-Tenant LoRA Serving, 2023) identified the grouped matmul as the bottleneck and introduced SGMV — Segmented Gather Matrix-Vector multiplication (more precisely a segmented gather matrix–matrix multiply). The insight: rather than launching one kernel per adapter (which serializes and underuses the GPU), launch one kernel that processes the whole batch, with each thread block told which adapter’s weights to gather.

The data layout¶

Punica keeps all adapter weights in a single contiguous tensor, indexed by adapter ID:

  A_all:  [N_adapters, num_layers, r, d_in]   # all "down" projections, stacked
  B_all:  [N_adapters, num_layers, d_out, r]  # all "up" projections, stacked

  For a batch of B tokens, a per-token index vector:
  lora_idx: [B]  e.g. [3, 3, 7, 0, 3, 7, ...]  # which adapter each token uses
                 (value -1 or a reserved slot means "no adapter / base only")

Because requests for the same adapter tend to arrive together (and the scheduler can sort the batch by adapter to make segments contiguous), the batch decomposes into segments:

  sorted batch:  [ A=3 | A=3 | A=3 | A=7 | A=7 | A=0 ]
  segment ptrs:  seg_start = [0, 3, 5]   # adapter 3: rows 0..2, adapter 7: 3..4, adapter 0: 5
  seg_adapter  = [3, 7, 0]

What SGMV computes¶

SGMV fuses the two LoRA matmuls with the gather. Conceptually, for each segment \(s\) with adapter \(a\) and rows \([\text{start}_s, \text{end}_s)\):

The kernel is launched as a 2-D grid: one axis over segments, one over output tiles. Each CTA (cooperative thread array / thread block) reads its segment’s adapter ID, indexes into A_all/B_all to find the right weight tile, and accumulates. Crucially, the base matmul \(XW_0^\top\) is a separate, fully-batched GEMM that runs at peak GPU efficiency; SGMV only computes the small low-rank additive correction and writes it into the same output buffer.

This split is what makes the scheme cheap: 99% of the FLOPs (the base GEMM) are a dense batched operation independent of adapters, and the 1% LoRA correction is handled by a specialized grouped kernel.

Two variants: BGMV vs SGMV¶

Punica distinguishes the decode and prefill regimes:

• BGMV (Batched Gather Matrix-Vector) — used in decode, where each request contributes exactly one token per step. The batch is “one row per request,” so it is a batched matrix-vector product: each row gathers a different adapter. Memory-bandwidth bound (we stream each adapter’s weights to multiply by a single vector).
• SGMV (Segmented Gather Matrix-multiply) — used in prefill, where each request contributes many tokens (its whole prompt). Now each adapter’s segment has multiple rows, so we get a real (small) GEMM per segment with arithmetic intensity high enough to be compute-bound. SGMV tiles over the rows of a segment to reuse the adapter weights loaded into shared memory.

7.14.3 S-LoRA & LoRAX: Serving Thousands of Adapters¶

Punica gave us the kernel. S-LoRA (Sheng et al., S-LoRA: Serving Thousands of Concurrent LoRA Adapters, 2023) gave us the system: how to keep thousands of adapters straight in memory, schedule them fairly, and not run out of GPU DRAM. LoRAX (Predibase’s open-source server) and the multi-LoRA paths now in vLLM and SGLang are direct descendants.

Unified Paging: adapters live in the KV-cache pool¶

The signature S-LoRA idea is Unified Paging. The KV cache (see PagedAttention & KV-Cache Memory Management) already manages GPU DRAM as a pool of fixed-size pages allocated and freed dynamically as requests come and go. S-LoRA stuffs adapter weights into the same paged memory pool. Both KV pages and adapter weight tensors are variable in size and lifetime; unifying them avoids fragmentation and lets the system trade memory fluidly between “more concurrent requests” (KV) and “more resident adapters” (LoRA weights).

Because adapters are of heterogeneous rank, S-LoRA pads them to a common rank for the kernel or, better, uses rank-aware tiling so a rank-8 adapter does not waste a rank-64 slot. Its custom kernels (an evolution of Punica’s MBGMV/SGMV) handle the non-uniform ranks directly.

Tiered storage: GPU → CPU → disk/object store¶

You cannot fit thousands of adapters in GPU DRAM at once, and you do not need to: at any instant only the active adapters (those with in-flight requests) must be resident. S-LoRA and LoRAX organize adapters in a storage hierarchy:

When a request arrives for adapter \(a\), the scheduler checks whether \(a\) is GPU-resident. If not, it triggers an asynchronous prefetch (HBM ← CPU ← SSD/object store) while the request waits in the queue, and admits the request to a batch only once the weights have landed. The cost of a cold adapter is dominated by this transfer, not by compute.

Hot-swapping and the prefetch pipeline¶

The art is overlapping adapter transfer with ongoing compute so swaps are invisible. A good server runs a copy engine (DMA over a separate CUDA stream) that streams the next batch’s needed adapters into GPU pages while the current batch is still executing on the compute stream. Get a feel for the magnitude: a rank-16 adapter on \(q,k,v,o\) of a 40-layer, \(d{=}5120\) model holds \(40 \times 4 \times 2 \times 16 \times 5120 \approx 26\)M parameters \(\approx 52\) MB in bf16; over a PCIe Gen4 x16 link at an effective ~20 GB/s that is roughly 2–3 ms — a few decode steps’ worth of time. That is small enough to hide completely if you prefetch, and far too large to pay on the critical path of a single request (and a rank-64 adapter, or one that also adapts the MLP, is 4–8× worse). By 2026 this overlap has become a first-class, opt-in engine feature rather than a hand-rolled stream (see §7.14.6 for SGLang’s and vLLM’s flags).

# Sketch: overlap adapter prefetch with the current forward pass.
# copy_stream does H2D DMA; compute_stream runs the model.
copy_stream = torch.cuda.Stream()

def prefetch_adapters(needed_ids, registry, gpu_pool):
    """Stream weights for `needed_ids` into GPU pages on a side stream."""
    with torch.cuda.stream(copy_stream):
        for aid in needed_ids:
            if aid in gpu_pool:                  # already resident -> skip
                continue
            cpu_weights = registry.fetch_to_cpu(aid)   # CPU/SSD/object store
            slot = gpu_pool.alloc(aid)                  # may evict an LRU adapter
            slot.copy_(cpu_weights, non_blocking=True)  # async H2D into the slot

# Each scheduler step:
#   1. pick the batch (requests + their adapter IDs)
#   2. prefetch_adapters(...) on copy_stream
#   3. run base+LoRA forward on compute_stream
#   4. compute_stream.wait_stream(copy_stream)  before the LoRA kernel reads weights

7.14.4 The Adapter Registry, Eviction & Fairness¶

Behind every multi-LoRA server is an adapter registry: the control-plane component that maps a tenant-facing adapter name to its physical weights and current residency tier, reference-counts in-flight usage, and decides what to evict.

What the registry tracks¶

Eviction policy¶

When the GPU pool is full and a new adapter must be loaded, the registry evicts a resident adapter with ref_count == 0 (never one in active use). LRU is the default; LFU or a cost-aware policy (weight the recency by adapter size and fetch cost) can be better when a few adapters dominate traffic. Because an evicted adapter still lives in CPU DRAM (or at least the object store), eviction is cheap — it just frees GPU pages. This is structurally identical to KV-cache eviction in Prefix Caching & KV-Cache Reuse, and in S-LoRA’s unified pool it is literally the same allocator.

Per-tenant fairness¶

A naïve “first-come-first-served, sort by adapter” scheduler is throughput-optimal but unfair: one tenant who floods the queue can starve everyone else, and a popular adapter that is always resident gets a latency advantage over a cold one. Production servers add fairness controls:

• Per-tenant token-bucket rate limits to cap any single tenant’s share of the batch.
• Max-adapters-per-batch so a single step’s LoRA kernel does not degenerate into thousands of tiny segments (each segment has fixed launch overhead; too many tanks efficiency).
• Weighted fair queuing across tenants so batch slots are allocated proportionally to entitlements, not arrival order.
• Reserved/pinned adapters for premium tenants guaranteed GPU residency (a latency SLO), traded against shared pool capacity.

There is a genuine tension here: sorting the batch by adapter maximizes kernel efficiency (few, large segments) but can violate per-request latency fairness; honoring strict FCFS order maximizes fairness but can produce a batch with many tiny adapter segments. Real schedulers interpolate — they sort within a fairness-bounded window.

7.14.5 Unified vs. Disaggregated Execution & Cross-Model Prefix Reuse¶

Unified execution¶

The default architecture — vLLM, SGLang, LoRAX, S-LoRA — is unified: base GEMM and LoRA correction run on the same GPU(s) in the same forward pass. The base matmul output is computed, then the SGMV/BGMV kernel adds the per-adapter correction in place before the activation flows on. One process, one weight set, one batch. This is simplest and has the lowest latency because there is no cross-machine hop.

Sharding adapters under tensor parallelism¶

A 13B+ base is usually served with tensor parallelism (TP), so each adapted linear layer is already sharded across GPUs (see Multi-GPU & Multi-Node Inference). How do \(A\) and \(B\) follow? The answer falls out of linearity, and it differs by layer type:

• Column-parallel base (\(q,k,v\), gate/up — sharded along \(d_\text{out}\)). Every rank sees the full input \(x\), so it can compute the full \(v = A x\) with a replicated \(A\), then multiply by its own shard \(B^{(k)} \in \mathbb{R}^{(d_\text{out}/\text{TP}) \times r}\) to produce its slice of the correction. No extra collective — the LoRA path adds nothing to the communication schedule.
• Row-parallel base (\(o\), down — sharded along \(d_\text{in}\)). Each rank holds only a slice of \(x\), so \(A\) is sharded along \(d_\text{in}\) and each rank computes a partial \(v^{(k)} = A^{(k)} x^{(k)}\). Because \(B\left(\sum_k v^{(k)}\right) = \sum_k B v^{(k)}\), each rank can apply a replicated \(B\) to its partial \(v\) and add the result into the base layer’s partial output — the correction then rides the base’s existing all-reduce. Again no extra collective.

The cost of this default is memory: the replicated matrix (\(A\) for column-parallel, \(B\) for row-parallel) is stored TP times over, which matters when you are holding hundreds of adapters resident. Fully-sharded LoRA (from S-LoRA, exposed in vLLM as --fully-sharded-loras) shards both matrices on every layer and pays a small extra collective on the rank-\(r\) intermediate instead. Since \(r \ll d\), that all-gather/reduce moves \(O(Br)\) elements versus the base’s \(O(Bd)\) — typically well under 1% of the layer’s traffic — so at TP \(\ge 4\) with a large adapter pool it is usually the right trade: you buy back a factor of TP in adapter memory for a nearly free collective.

Disaggregated execution¶

A disaggregated design separates concerns across machines or pools. There are two distinct axes, easily conflated:

1. Prefill/decode disaggregation (see Disaggregated Prefill/Decode & Chunked Prefill) — prefill (compute-bound, batches large) runs on one pool, decode (bandwidth-bound) on another, with the KV cache shipped between them. With LoRA, both pools must hold the relevant adapters, and the SGMV vs BGMV split maps naturally onto the two pools.
2. Base/adapter disaggregation — a more exotic design where a fleet of “base servers” runs the shared dense forward and a separate “adapter service” applies LoRA corrections, or where adapters are sharded across nodes. This can help when you have so many high-rank adapters that they exceed any single node’s memory, or when adapter compute is offloaded to cheaper hardware. The cost is an extra activation round-trip per layer, which is usually prohibitive for the decode path; it is mostly of interest at extreme adapter counts.

For the overwhelming majority of deployments, unified execution with tiered adapter storage wins: it keeps the hot path local and pays transfer cost only on cold misses.

Cross-model prefix-cache reuse¶

Here is a subtle and valuable interaction. Prefix caching reuses KV blocks for shared token prefixes (system prompts, few-shot headers). But KV tensors depend on the weights, and LoRA changes the weights — so in general two requests on different adapters produce different KV tensors for the same prefix tokens. Can they share a prefix cache?

The answer hinges on which modules the adapter touches:

• If the LoRA does not adapt the \(k\) and \(v\) projections (e.g., it only adapts \(q,o\) or the MLP), then \(K\) and \(V\) for the prefix are computed purely from base weights and are identical across all such adapters. The prefix KV cache can be shared across every tenant — a large win, since system prompts are often shared.
• If the LoRA does adapt \(k\) or \(v\), the cached KV blocks are adapter-specific. They are still cacheable, but only reusable by requests on the same adapter (and only if the base prefix is the same). The cache must therefore be keyed by (prefix_hash, adapter_id).

  Prefix-cache key strategy:
    adapter touches k or v  →  key = hash(prefix_tokens) ⊕ adapter_id   (per-adapter)
    adapter leaves k,v base →  key = hash(prefix_tokens)                (shared!)

Designing adapters to avoid the \(k,v\) projections (when accuracy permits) is therefore not just a quality choice — it directly enables cross-tenant KV reuse. This is a concrete example of co-designing the fine-tuning recipe with the serving system. See SGLang: RadixAttention & Structured Programs for the trie-based cache this plugs into; the radix tree can store per-adapter subtrees off a shared base-prefix root.

7.14.6 The vLLM, SGLang & TensorRT-LLM Multi-LoRA Paths¶

Every major open-source engine now ships production multi-LoRA support built on the Punica/S-LoRA lineage; they differ mainly in how dynamic the adapter set is allowed to be.

vLLM exposes LoRA as a first-class serving feature. You launch with --enable-lora, set --max-loras (max distinct adapters per batch/step) and --max-cpu-loras (the CPU warm-pool size), and bound rank with --max-lora-rank. Adapters can be registered statically at launch (--lora-modules name=path ...) or loaded dynamically at runtime via the API, which is what makes a true multi-tenant platform possible — tenants upload adapters and route to them by name without restarting the server. Internally vLLM uses Punica-style SGMV/BGMV kernels (and Triton variants), the paged allocator holds adapter weights, and an LRU manager handles GPU↔CPU residency. Requests carry a LoRARequest(name, id, path) so the scheduler knows which adapter each belongs to.

SGLang similarly supports multi-LoRA, sorting requests by adapter to form efficient SGMV segments and integrating adapter residency with its RadixAttention KV cache (so the cross-model prefix-reuse story above is native). You launch with --lora-paths name=path ..., cap adapter cardinality with --max-loras-per-batch (the knob of §7.14.4), bound rank with --max-lora-rank, and select the kernel backend with --lora-backend (a Triton SGMV implementation is the default). Adapters can also be added and removed at runtime through /load_lora_adapter and /unload_lora_adapter HTTP endpoints. Recent releases add an opt-in overlapped adapter loading mode that streams adapter weights on a side CUDA stream to hide cold-adapter transfer behind compute (§7.14.3), reported to cut median TTFT substantially on large-adapter workloads at the cost of occasionally fragmenting multi-adapter prefill batches — check python -m sglang.launch_server --help for the current flag name, since these LoRA server args are still moving. Its structured-program model means a single program can fan out across adapters, and its scheduler co-optimizes the LoRA batch with prefix sharing.

TensorRT-LLM supports multi-LoRA too, but with an ahead-of-time twist worth internalizing: because the engine is compiled, the LoRA plugin, the set of target modules, and the maximum rank must be declared at trtllm-build time and are baked into the engine. Adapter weights are still dynamic at runtime (its LoRA manager keeps GPU and CPU adapter caches, sized like vLLM’s max_loras/max_cpu_loras), but a tenant who trains a rank-128 adapter for an engine built at rank 64, or who adapts the MLP when only attention modules were compiled in, cannot be served without rebuilding. If you run a fine-tuning SaaS on TensorRT-LLM, publish the supported rank ceiling and target-module set as part of your product contract. See TensorRT-LLM, TGI & Other Serving Stacks.

# vLLM: serve a base model with multi-LoRA, dynamic loading enabled.
#   --max-loras 8       : up to 8 distinct adapters per scheduler step (§7.14.4)
#   --max-lora-rank 64  : kernels/slots sized for ranks up to 64
#   --max-cpu-loras 256 : CPU warm pool, 256 adapters resident off-GPU
#   --enable-prefix-caching : share base-prefix KV where adapters allow (§7.14.5)
# NOTE: keep comments on their own lines — a `#` after a trailing `\` silently
# breaks the line continuation and the rest of the flags become bogus commands.
VLLM_ALLOW_RUNTIME_LORA_UPDATING=1 vllm serve meta-llama/Llama-2-13b-hf \
  --enable-lora \
  --max-loras 8 \
  --max-lora-rank 64 \
  --max-cpu-loras 256 \
  --enable-prefix-caching

# Register a tenant's adapter at runtime — no restart, the key to multi-tenancy.
curl -s http://localhost:8000/v1/load_lora_adapter \
  -H 'Content-Type: application/json' \
  -d '{"lora_name": "acme-contracts-v3", "lora_path": "/adapters/acme-v3"}'

# Then route to it exactly like a model name on the OpenAI-compatible API:
curl -s http://localhost:8000/v1/completions \
  -H 'Content-Type: application/json' \
  -d '{"model": "acme-contracts-v3", "prompt": "Summarize clause 4:", "max_tokens": 64}'

# ...and release it when the tenant churns (frees the CPU/GPU pool entry):
curl -s http://localhost:8000/v1/unload_lora_adapter \
  -H 'Content-Type: application/json' -d '{"lora_name": "acme-contracts-v3"}'

# vLLM offline batched multi-LoRA: one base, many adapters, one batch.
from vllm import LLM, SamplingParams
from vllm.lora.request import LoRARequest

llm = LLM(model="meta-llama/Llama-2-13b-hf",
          enable_lora=True, max_loras=4, max_lora_rank=16)

sp = SamplingParams(temperature=0.0, max_tokens=64)

# Each prompt is tagged with its own adapter; vLLM batches them together
# and applies the per-row SGMV correction in a single fused forward.
prompts = [
    ("Summarize this contract clause: ...", LoRARequest("contracts", 1, "/adapters/contracts")),
    ("Generate NPC dialogue for a dragon:", LoRARequest("game-lore", 2, "/adapters/game-lore")),
    ("Draft a discharge summary for:",      LoRARequest("med-notes", 3, "/adapters/med-notes")),
]
outs = llm.generate([p for p, _ in prompts], sp,
                    lora_request=[lr for _, lr in prompts])

from peft import PeftModel
model = PeftModel.from_pretrained(base, "/adapters/contracts", adapter_name="contracts")
model.load_adapter("/adapters/game-lore", adapter_name="game-lore")
out = model.generate(**batch,   # one batch, three different behaviours
                     adapter_names=["contracts", "game-lore", "__base__"])

7.14.7 From Scratch: A Batched Multi-Adapter Forward¶

Let us build the core mechanism end to end: a single adapted linear layer that, given a batch of activations each tagged with an adapter ID, computes the base GEMM once and adds the correct per-row LoRA correction — a CPU/GPU-portable PyTorch model of SGMV/BGMV. We then wrap it in a tiny registry with eviction to show the control plane.

import torch
import torch.nn.functional as F
from dataclasses import dataclass, field
from collections import OrderedDict

# ---------------------------------------------------------------------------
# 1. The batched multi-adapter linear layer (the SGMV/BGMV idea, vectorized).
# ---------------------------------------------------------------------------
class MultiLoRALinear:
    """A frozen base weight W0 plus a *stack* of LoRA adapters, applied
    per-row according to a per-token adapter index. This is the math model
    of Punica's SGMV; a real kernel fuses the gather + matmuls in CUDA/Triton.
    """
    def __init__(self, d_in, d_out, max_adapters, rank, device="cpu"):
        self.d_in, self.d_out, self.rank = d_in, d_out, rank
        # Frozen base weight: [d_out, d_in]
        self.W0 = torch.randn(d_out, d_in, device=device) / d_in**0.5
        # Adapter stacks. Slot 0 is reserved as the "no adapter" identity:
        # its A,B are zero so the correction is exactly zero (base-only rows).
        # A_all: [S, r, d_in]   B_all: [S, d_out, r]
        S = max_adapters + 1
        self.A_all = torch.zeros(S, rank, d_in, device=device)
        self.B_all = torch.zeros(S, d_out, rank, device=device)
        self.scale = torch.ones(S, device=device)  # alpha/r per slot; slot 0 -> 0
        self.scale[0] = 0.0
        self.device = device

    def set_adapter(self, slot, A, B, alpha):
        """Install adapter weights into a GPU slot. A:[r,d_in] B:[d_out,r]."""
        assert slot >= 1, "slot 0 is reserved as base-only"
        self.A_all[slot].copy_(A)
        self.B_all[slot].copy_(B)
        self.scale[slot] = alpha / self.rank

    def forward(self, x, lora_idx):
        """
        x:        [B, d_in]   stacked activations (one row per token)
        lora_idx: [B] long    adapter slot for each row (0 == base only)
        returns:  [B, d_out]
        """
        # (a) Base GEMM — fully batched, peak-efficiency, adapter-independent.
        y = F.linear(x, self.W0)                      # [B, d_out]

        # (b) Gather each row's adapter matrices. In a real kernel this gather
        #     is fused; here we use advanced indexing for clarity.
        A = self.A_all[lora_idx]                       # [B, r, d_in]
        B = self.B_all[lora_idx]                       # [B, d_out, r]
        s = self.scale[lora_idx].unsqueeze(-1)         # [B, 1]

        # (c) Shrink then expand:  v = A x   (per-row mat-vec), then B v.
        #     einsum expresses the per-row low-rank product without a Python loop.
        v = torch.einsum("brd,bd->br", A, x)           # [B, r]   (down-proj)
        delta = torch.einsum("bor,br->bo", B, v)       # [B, d_out](up-proj)
        y = y + s * delta                              # rows with slot 0 add 0
        return y


# ---------------------------------------------------------------------------
# 2. A segment-sorted variant: sort the batch by adapter so identical
#    adapters are contiguous (what the scheduler does to form SGMV segments).
# ---------------------------------------------------------------------------
def forward_segmented(layer, x, lora_idx):
    """Demonstrate adapter-sorting: group rows by adapter, apply per segment,
    then scatter results back to original order. Mirrors how a real server
    sorts a continuous batch by adapter id before launching SGMV."""
    order = torch.argsort(lora_idx)                    # stable grouping
    x_sorted, idx_sorted = x[order], lora_idx[order]
    y_sorted = layer.forward(x_sorted, idx_sorted)     # same result, contiguous
    # scatter back to the caller's order
    y = torch.empty_like(y_sorted)
    y[order] = y_sorted
    return y


# ---------------------------------------------------------------------------
# 3. A tiny adapter registry with GPU-slot LRU eviction + CPU warm pool.
# ---------------------------------------------------------------------------
@dataclass
class AdapterMeta:
    A: torch.Tensor
    B: torch.Tensor
    alpha: float
    ref_count: int = 0

class AdapterRegistry:
    def __init__(self, layer: "MultiLoRALinear", n_gpu_slots: int):
        self.layer = layer
        self.n_gpu_slots = n_gpu_slots
        self.cpu_pool: dict[str, AdapterMeta] = {}     # warm pool (CPU DRAM)
        self.gpu: "OrderedDict[str,int]" = OrderedDict()  # name -> slot (LRU order)
        self.free_slots = list(range(1, n_gpu_slots + 1))  # slot 0 reserved

    def register(self, name, A, B, alpha):
        """Add an adapter to the (CPU) warm pool — the source of truth here."""
        self.cpu_pool[name] = AdapterMeta(A=A, B=B, alpha=alpha)

    def ensure_resident(self, name) -> int:
        """Return the GPU slot for `name`, loading + evicting as needed."""
        if name in self.gpu:                           # GPU hit
            self.gpu.move_to_end(name)                 # mark most-recently-used
            return self.gpu[name]
        meta = self.cpu_pool[name]                     # CPU-warm hit (else KeyError)
        if not self.free_slots:                        # need to evict an LRU adapter
            victim, vslot = next(iter(self.gpu.items()))
            if self.cpu_pool[victim].ref_count > 0:
                raise RuntimeError("LRU victim is in use; need a richer policy")
            del self.gpu[victim]
            self.free_slots.append(vslot)
        slot = self.free_slots.pop()
        self.layer.set_adapter(slot, meta.A, meta.B, meta.alpha)  # H2D copy
        self.gpu[name] = slot
        return slot

    def build_index(self, request_adapter_names):
        """Map a batch's per-request adapter names to GPU slot indices,
        ensuring each is resident first."""
        return torch.tensor([self.ensure_resident(n) for n in request_adapter_names],
                            dtype=torch.long, device=self.layer.device)


# ---------------------------------------------------------------------------
# 4. End-to-end sanity check: correctness vs an explicit per-adapter reference.
# ---------------------------------------------------------------------------
if __name__ == "__main__":
    torch.manual_seed(0)
    d_in, d_out, r = 64, 48, 8
    layer = MultiLoRALinear(d_in, d_out, max_adapters=4, rank=r)
    reg = AdapterRegistry(layer, n_gpu_slots=3)        # exactly enough for this batch's
                                                        # 3 concurrent adapters; a 4th
                                                        # distinct adapter in a later batch
                                                        # would force an eviction

    # Register 3 adapters into the CPU warm pool (exactly filling GPU slots).
    refs = {}
    for name in ["contracts", "game-lore", "med-notes"]:
        A = torch.randn(r, d_in) * 0.02
        B = torch.randn(d_out, r) * 0.02
        reg.register(name, A, B, alpha=16.0)
        refs[name] = (A, B, 16.0 / r)

    # A heterogeneous batch: 6 tokens over 3 adapters + 1 base-only row.
    batch_names = ["contracts", "game-lore", "contracts",
                   "med-notes", "game-lore", "__base__"]
    x = torch.randn(len(batch_names), d_in)

    # __base__ maps to reserved slot 0 (no correction); others get real slots.
    def to_slot(n):
        return 0 if n == "__base__" else reg.ensure_resident(n)
    lora_idx = torch.tensor([to_slot(n) for n in batch_names], dtype=torch.long)

    y = layer.forward(x, lora_idx)
    y_seg = forward_segmented(layer, x, lora_idx)      # sorted path, same answer

    # Reference: compute each row independently with merged math y = W0 x + (a/r) B A x
    y_ref = torch.empty_like(y)
    for i, n in enumerate(batch_names):
        base = F.linear(x[i], layer.W0)
        if n == "__base__":
            y_ref[i] = base
        else:
            A, B, s = refs[n]
            y_ref[i] = base + s * (B @ (A @ x[i]))

    print("batched  vs reference  max err:", (y - y_ref).abs().max().item())
    print("segmented vs reference max err:", (y_seg - y_ref).abs().max().item())
    # Both errors are ~1e-6 (float rounding): the fused multi-adapter forward
    # is numerically identical to per-adapter merged math.

Running this prints max errors on the order of \(10^{-6}\) — float rounding — confirming that the single batched forward over a heterogeneous mix of adapters is exactly equivalent to merging each adapter and running it alone, but at a fraction of the cost and with one shared base in memory. The forward_segmented path shows the scheduler trick of sorting by adapter; the registry shows GPU-slot LRU eviction with a CPU warm pool and a reserved base-only slot.

What a tenant actually uploads: the adapter artifact¶

The registry above was handed bare A/B tensors. In a real platform the tenant uploads a PEFT adapter directory — the de-facto interchange format that peft, vLLM, SGLang, LoRAX and TensorRT-LLM all read:

  /adapters/acme-contracts-v3/
    adapter_config.json        # rank, alpha, target_modules, base model id, variant flags
    adapter_model.safetensors  # the A/B matrices, one pair per adapted linear layer

The weight keys follow PEFT’s wrapped-module naming, and the shapes are exactly the \(A \in \mathbb{R}^{r \times d_\text{in}}\), \(B \in \mathbb{R}^{d_\text{out} \times r}\) of §7.14.1:

  base_model.model.model.layers.0.self_attn.q_proj.lora_A.weight   [r, d_in]
  base_model.model.model.layers.0.self_attn.q_proj.lora_B.weight   [d_out, r]
  base_model.model.model.layers.0.self_attn.v_proj.lora_A.weight   ...

Parsing this — and validating it before it reaches a kernel — is the unglamorous half of the control plane:

import json, os, re, torch

def load_peft_adapter(path, max_rank=64, allowed_modules=("q_proj", "k_proj",
                                                          "v_proj", "o_proj")):
    """Parse a PEFT adapter dir into {(layer_idx, module): (A, B, scale)}.

    Returns tensors in the exact layout the SGMV kernel wants, plus the
    per-adapter scale so the kernel never has to know about alpha or rsLoRA.
    """
    from safetensors.torch import load_file          # imported lazily
    cfg = json.load(open(os.path.join(path, "adapter_config.json")))

    # --- Admission validation: reject before allocating a GPU slot. ---------
    if cfg.get("peft_type", "LORA") != "LORA":
        raise ValueError(f"unsupported peft_type {cfg.get('peft_type')}")
    if cfg.get("use_dora") or cfg.get("bias", "none") != "none":
        # DoRA adds a per-column magnitude vector and bias-tuning adds a bias
        # term: neither is a pure additive low-rank correction, so a plain
        # SGMV path cannot serve them. Check your engine before accepting.
        raise ValueError("DoRA / bias tuning not supported by this SGMV path")
    r, alpha = int(cfg["r"]), float(cfg["lora_alpha"])
    if r > max_rank:
        raise ValueError(f"rank {r} exceeds engine ceiling {max_rank}")
    bad = set(cfg["target_modules"]) - set(allowed_modules)
    if bad:
        raise ValueError(f"adapter adapts unsupported modules: {sorted(bad)}")

    # rsLoRA rescales by alpha/sqrt(r) instead of alpha/r. Getting this wrong
    # silently changes the adapter's effective strength -- a real serving bug.
    scale = alpha / (r ** 0.5) if cfg.get("use_rslora") else alpha / r

    sd = load_file(os.path.join(path, "adapter_model.safetensors"))
    pat = re.compile(r"layers\.(\d+)\..*\.(\w+_proj)\.lora_(A|B)\.weight")
    out = {}
    for k, v in sd.items():
        m = pat.search(k)
        if m is None:                       # e.g. embedding/lm_head adapters
            continue
        layer, module, which = int(m.group(1)), m.group(2), m.group(3)
        out.setdefault((layer, module), [None, None, scale])
        out[(layer, module)][0 if which == "A" else 1] = v.to(torch.float16)
    for key, (A, B, _) in out.items():
        assert A is not None and B is not None, f"incomplete LoRA pair at {key}"
        assert A.shape[0] == B.shape[1] == r, f"rank mismatch at {key}"
    return out

# Usage (guarded so this file runs with or without a real adapter on disk):
ADAPTER_DIR = "/adapters/acme-contracts-v3"
if os.path.isdir(ADAPTER_DIR):
    weights = load_peft_adapter(ADAPTER_DIR)
    # Each (layer, module) pair goes into that layer's A_all/B_all slot:
    #   layer_stack[(l, mod)].set_adapter(slot, A, B, alpha=scale * r)
    print(f"loaded {len(weights)} adapted projections, "
          f"{sum(A.numel() + B.numel() for A, B, _ in weights.values())/1e6:.1f}M params")

Two of these checks are the ones that bite in production. Rank ceiling: engines size their kernels and adapter slots for max_lora_rank at startup (or, for TensorRT-LLM, at build time), so an over-rank adapter must be rejected at upload with a clear error, not at first request. rsLoRA: use_rslora changes the scaling from \(\alpha/r\) to \(\alpha/\sqrt{r}\); a server that ignores the flag serves a silently mis-scaled model that evaluates worse than the tenant’s own local test — one of the nastiest support tickets in this business.

To turn this toy into the real thing you would: (1) replace the einsum gather with a fused SGMV/BGMV Triton or CUDA kernel that never materializes the gathered A/B tensors; (2) apply it to all adapted projections in every transformer block, not one layer; (3) move the registry’s CPU↔GPU copies onto a side stream for overlap; and (4) wire the adapter index through the continuous-batching scheduler so it is rebuilt every step as requests join and leave.

7.14.8 Throughput, SLOs & When Not to Use Multi-LoRA¶

Multi-tenant LoRA serving is a throughput-per-dollar machine, but it is not free. A summary of the trade-offs:

When not to use multi-LoRA serving: if a single tenant dominates traffic, just merge their adapter and serve a dedicated dense model — you avoid the LoRA tax entirely. If adapters are high-rank (hundreds) or full fine-tunes, the “tiny adapter” assumption breaks and per-model deployment may be cheaper. And if tenants need different base models (not just different adapters on one base), multi-LoRA doesn’t apply at all — you are back to multi-model serving and routing (see Caching, Routing & Cost Control in Production). The sweet spot is many low-rank adapters over one shared base, with skewed-but-not-degenerate traffic — exactly the SaaS fine-tuning platform we opened with.

For the broader economics of latency vs. throughput vs. cost that frame these decisions, see Inference Economics: Latency, Throughput & Cost and the system-design view in Designing an LLM Serving System.

Further Reading¶

• Chen, L. et al. “Punica: Multi-Tenant LoRA Serving.” MLSys 2024 / arXiv:2310.18547. — Introduces SGMV/BGMV, the batched gather matmul for heterogeneous adapters over a shared base.
• Sheng, Y. et al. “S-LoRA: Serving Thousands of Concurrent LoRA Adapters.” arXiv:2311.03285, 2023. — Unified Paging, tiered adapter storage, and rank-aware kernels for thousands of adapters.
• Hu, E. et al. “LoRA: Low-Rank Adaptation of Large Language Models.” ICLR 2022. — The original low-rank reparameterization underpinning everything in this chapter.
• Kwon, W. et al. “Efficient Memory Management for Large Language Model Serving with PagedAttention.” SOSP 2023. — The paged allocator S-LoRA’s Unified Paging extends to adapter weights.
• Zheng, L. et al. “SGLang: Efficient Execution of Structured Language Model Programs.” arXiv:2312.07104, 2023. — RadixAttention and the structured-program serving model that hosts SGLang’s multi-LoRA path.
• Predibase, “LoRAX: Multi-LoRA inference server.” Open-source repository — production reference implementation of dynamic adapter loading and tiered storage.
• vLLM documentation, “Using LoRA adapters” — configuration and the runtime dynamic-loading API for multi-tenant adapter serving.

Exercises¶

1. A colleague proposes speeding up your multi-tenant server by calling peft’s merge_and_unload() on every adapter at load time so that “the LoRA math disappears at runtime.” A single batch on your server typically contains requests for 15 different adapters. Explain concretely why merging breaks multi-tenancy, and describe the two fallback strategies you would be forced into if you insisted on serving merged weights — and why both are catastrophic.

2. Consider one adapted projection with \(d_\text{in} = d_\text{out} = 4096\) and LoRA rank \(r = 32\). Using the parenthesized forward \(y = W_0 x + \frac{\alpha}{r}B(Ax)\), compute the extra multiply-accumulate (MAC) cost of the LoRA path per token as a fraction of the base GEMM. Then state what this fraction would become if you (naively) first materialized \(\Delta W = \frac{\alpha}{r}BA\) and added it to \(W_0\).

3. You run a decode step on a shared 13B-shaped model. LoRA is applied to the four attention projections \(q,k,v,o\), each with \(d_\text{in}=d_\text{out}=4096\) and rank \(r=16\), weights in fp16 (2 bytes). The decode batch has 48 tokens drawn from 24 distinct adapters. Because decode is memory-bandwidth bound, estimate the “bandwidth tax” the LoRA weights impose: the bytes of LoRA weights that must be streamed, as a fraction of the base attention-projection weights streamed. Contrast this with the FLOP fraction of the same correction.

4. Prefix caching lets requests share KV-cache blocks for a common token prefix (e.g. a shared system prompt). Tenant Alpha’s adapter adapts only the \(q\) and \(o\) projections and the MLP; Tenant Beta’s adapter adapts \(q,k,v,o\). Both send requests that begin with the same 400-token shared system prompt over the same base model. For each tenant, can the prefix KV blocks be shared with other tenants? State the correct cache key in each case and explain the underlying reason.

5. The toy MultiLoRALinear.forward in Section 7.14.7 uses self.A_all[lora_idx] and self.B_all[lora_idx], which materializes a [B, r, d_in] gathered tensor — physically copying a popular adapter’s weight once per row that uses it. The chapter’s warning notes a real SGMV kernel instead loads each adapter’s weight tile once per segment. Implement a forward_grouped(layer, x, lora_idx) that reproduces this: it must compute the correction by iterating over the distinct adapters present in the batch, loading each adapter’s A/B exactly once and applying it to all of that adapter’s rows with a single small GEMM. Skip the reserved base-only slot 0. Your function must be numerically equivalent to layer.forward.

import torch
import torch.nn.functional as F

def forward_grouped(layer, x, lora_idx):
    """Batched multi-adapter forward that loads each adapter's A,B once
    per segment (no per-row gather of weights). Equivalent to
    MultiLoRALinear.forward but bandwidth-friendly, mirroring SGMV.
    x:        [B, d_in]
    lora_idx: [B] long   (0 == reserved base-only slot)
    """
    # (a) Shared base GEMM, fully batched, adapter-independent.
    y = F.linear(x, layer.W0)                       # [B, d_out]

    # (b) One low-rank correction per distinct adapter in the batch.
    for a in torch.unique(lora_idx).tolist():
        if a == 0:
            continue                                # base-only rows: no correction
        rows = (lora_idx == a).nonzero(as_tuple=True)[0]   # this adapter's segment
        xa = x[rows]                                # [n_a, d_in]
        A  = layer.A_all[a]                         # [r, d_in]  loaded ONCE
        B  = layer.B_all[a]                         # [d_out, r] loaded ONCE
        s  = layer.scale[a]                         # scalar alpha/r
        v  = xa @ A.t()                             # [n_a, r]     shrink
        delta = v @ B.t()                           # [n_a, d_out] expand
        y[rows] += s * delta
    return y

torch.manual_seed(0)
d_in, d_out, r = 64, 48, 8
layer = MultiLoRALinear(d_in, d_out, max_adapters=4, rank=r)
for slot in (1, 2, 3):
    A = torch.randn(r, d_in) * 0.02
    B = torch.randn(d_out, r) * 0.02
    layer.set_adapter(slot, A, B, alpha=16.0)

x = torch.randn(6, d_in)
lora_idx = torch.tensor([1, 2, 1, 3, 2, 0])     # 6 tokens, 3 adapters + base
y_ref = layer.forward(x, lora_idx)
y_grp = forward_grouped(layer, x, lora_idx)
print((y_ref - y_grp).abs().max().item())        # ~1e-7: numerically identical

6. The chapter’s AdapterRegistry.ensure_resident evicts by LRU and refuses to touch any adapter still in use. Section 7.14.4 notes that LFU (evict the least-frequently-used) can be better when a few adapters dominate traffic. Modify the registry to (a) track a per-adapter hit_count, and (b) evict, when the GPU pool is full, the resident adapter with the smallest hit_count among those with ref_count == 0. If every resident adapter is in use, raise an error. Give the code and explain when LFU beats LRU here.

from dataclasses import dataclass
from collections import OrderedDict
import torch

@dataclass
class AdapterMeta:
    A: torch.Tensor
    B: torch.Tensor
    alpha: float
    ref_count: int = 0
    hit_count: int = 0          # NEW: LFU frequency signal

class LFUAdapterRegistry:
    def __init__(self, layer, n_gpu_slots: int):
        self.layer = layer
        self.n_gpu_slots = n_gpu_slots
        self.cpu_pool: dict[str, AdapterMeta] = {}
        self.gpu: "OrderedDict[str,int]" = OrderedDict()   # name -> slot
        self.free_slots = list(range(1, n_gpu_slots + 1))  # slot 0 reserved

    def register(self, name, A, B, alpha):
        self.cpu_pool[name] = AdapterMeta(A=A, B=B, alpha=alpha)

    def ensure_resident(self, name) -> int:
        meta = self.cpu_pool[name]
        meta.hit_count += 1                     # count every use for LFU
        if name in self.gpu:                    # GPU hit
            return self.gpu[name]
        if not self.free_slots:                 # pool full -> LFU eviction
            candidates = [
                (self.cpu_pool[n].hit_count, n, slot)
                for n, slot in self.gpu.items()
                if self.cpu_pool[n].ref_count == 0
            ]
            if not candidates:
                raise RuntimeError("all resident adapters in use; cannot evict")
            _, victim, vslot = min(candidates)  # smallest hit_count wins
            del self.gpu[victim]
            self.free_slots.append(vslot)
        slot = self.free_slots.pop()
        self.layer.set_adapter(slot, meta.A, meta.B, meta.alpha)   # H2D copy
        self.gpu[name] = slot
        return slot