7.13 Serving Mixture-of-Experts: Expert Parallelism & All-to-All Inference¶

A dense 70-billion-parameter model and a sparse 671-billion-parameter Mixture-of-Experts (MoE) model can have identical per-token FLOPs. DeepSeek-V3 has 671B total parameters but activates only ~37B per token; Kimi K2 pushes the same recipe to ~1T total with only ~32B active; Llama-4 Maverick has ~400B total but ~17B active; Qwen3-235B-A22B activates ~22B of 235B. The promise of MoE — explained architecturally in Mixture-of-Experts (MoE) Architectures — is that quality tracks total parameters while compute tracks active parameters. That promise is what makes these models attractive to train. But it creates a serving problem that has almost nothing in common with serving a dense model of equivalent speed.

The trouble is memory and movement, not arithmetic. Those 671B parameters still have to live on GPUs — roughly 671 GB in FP8, over 1.3 TB in BF16 — even though any single token touches a tiny fraction of them. And because experts are scattered across dozens of GPUs, every MoE layer becomes a two-phase all-to-all communication: send each token to the GPUs holding its chosen experts (dispatch), then send the results back (combine). A dense model with the same active parameter count does a couple of cheap all-reduces per layer; a large MoE does two cluster-wide shuffles whose cost is set by the slowest, most overloaded GPU in the group. Serving MoE well is the art of making that shuffle cheap and keeping every expert busy.

This chapter is the inference-specific companion to the architecture chapter and to Multi-GPU & Multi-Node Inference, which introduced expert parallelism (EP) as one of four parallelism axes. Here we go deep on EP at decode time: the all-to-all kernel itself (DeepEP), hot-expert skew and why it is worse in serving than in training, the attention-DP + expert-EP hybrid layout that frontier stacks now use, prefill-vs-decode EP asymmetry, expert and KV offloading, and the economics of “dense-equivalent throughput.” We assume you know prefill/decode and the KV cache (The Anatomy of LLM Inference) and collective communication (Parallel Computing & Collective Communication).

Why MoE Serving Is a Different Problem¶

Start from the resource that dominates decode. A single decode step is memory-bandwidth-bound: the GPU must stream the weights it needs through its compute units, and at batch sizes typical of low-latency serving it does very little arithmetic per byte loaded. For a dense model the working set is the whole weight matrix every step. For an MoE model the per-token working set is small — the router, the attention block, the shared expert (if any), and the \(k\) routed experts the token selected — but the resident working set is enormous because every expert must be in memory in case some token in the batch picks it.

This is the first inversion. In a dense model, parameters loaded \(\approx\) parameters resident. In an MoE model, with \(E\) experts and top-\(k\) routing, a single token loads \(k/E\) of the expert weights but the batch may load all of them. Consider a batch of \(B\) tokens, \(E=256\) experts, \(k=8\). Expected distinct experts touched is

For \(B=64\), \(k=8\), \(E=256\) this is \(256(1-(1-1/32)^{64})\approx 256(1-0.132)\approx 222\) — already 87% of all experts touched by a modest batch. So even though each token is sparse, a realistic decode batch is nearly dense in expert coverage. You cannot avoid keeping (almost) all experts resident and ready.

The second inversion is communication. EP places experts on different GPUs; routing is data-dependent and changes every token. The required primitive is all-to-all: GPU \(i\) has a variable number of tokens destined for each expert, hence for each GPU. Unlike the all-reduce of tensor parallelism (TP) — whose volume is fixed at \(\sim 2\,d_{\text{model}}\) bytes per token per layer regardless of routing — all-to-all volume is \(\sim k\,d_{\text{model}}\) bytes per token per layer and is load-imbalanced and irregular: the message sizes are unknown until the router fires.

The third inversion is who is the bottleneck. In dense decode, the bottleneck is HBM bandwidth on a balanced set of identical GPUs. In MoE decode, the bottleneck is whichever GPU happens to hold a hot expert that the current batch over-selects, because the all-to-all cannot complete until that GPU finishes its disproportionate share. Routing skew, which is a mild efficiency annoyance during training (you fix it with an auxiliary load-balance loss; see Mixture-of-Experts Architectures), becomes a tail-latency weapon in serving, where each decode step is a synchronization barrier.

Expert Parallelism Mechanics: Dispatch, Compute, Combine¶

Let us pin down the layout. Suppose \(E\) experts spread over \(G\) GPUs in the EP group, \(E/G\) experts per GPU (assume it divides). Each GPU also holds a replica of the non-expert parts (attention, router, shared expert, norms) — or, in the hybrid layout of the next section, a shard of attention. A token resident on GPU \(i\) is processed as:

1. Router runs locally; produces top-\(k\) expert ids and gate weights for each local token.
2. Dispatch (all-to-all #1): tokens are permuted so that all tokens destined for expert \(e\) are contiguous, then sent to the GPU owning \(e\). Because token-to-GPU counts vary, this is an all-to-all-v (variable length).
3. Expert compute: each GPU runs its local experts’ FFNs over the tokens it received — a grouped GEMM (one matmul per expert, batched).
4. Combine (all-to-all #2): expert outputs are sent back to each token’s home GPU and weighted-summed by the gate weights. (With top-\(k>1\), a token’s home GPU receives \(k\) partial results to combine.)

Here is a minimal but correct single-process reference that makes the data movement explicit. It simulates EP across G virtual ranks with plain tensors so you can run it on one GPU or CPU and inspect the permutations.

import torch
import torch.nn.functional as F

def moe_ep_layer(x, router_w, expert_w1, expert_w2, k, G):
    """
    Reference EP forward for ONE MoE layer, simulating G ranks in-process.
      x          : (T, d)         tokens for the whole EP group (gathered for clarity)
      router_w   : (E, d)         router weights
      expert_w1  : (E, d, h)      per-expert up-projection
      expert_w2  : (E, h, d)      per-expert down-projection
      k          : top-k          experts per token
      G          : number of EP ranks; expert e lives on rank (e // (E//G))
    Returns y : (T, d). The point is to expose dispatch/compute/combine, not speed.
    """
    T, d = x.shape
    E = router_w.shape[0]
    experts_per_rank = E // G

    # ---- 1. ROUTER (runs on each token's home rank) ----
    logits = x @ router_w.t()                      # (T, E)
    gate = F.softmax(logits, dim=-1)               # softmax over all experts
    topv, topi = gate.topk(k, dim=-1)              # (T, k) weights and expert ids
    topv = topv / topv.sum(-1, keepdim=True)       # renormalize the chosen-k weights

    y = torch.zeros_like(x)

    # ---- 2-4. For each rank, gather its tokens (DISPATCH), compute, scatter back (COMBINE)
    for r in range(G):
        lo, hi = r * experts_per_rank, (r + 1) * experts_per_rank
        # Which (token, slot) pairs route to an expert owned by rank r?
        mask = (topi >= lo) & (topi < hi)          # (T, k) boolean
        if not mask.any():
            continue
        tok_idx, slot_idx = mask.nonzero(as_tuple=True)   # variable length -> all-to-all-v
        local_eid = topi[tok_idx, slot_idx] - lo          # expert index within this rank
        gw = topv[tok_idx, slot_idx]                      # gate weight for the combine

        # Grouped expert compute: loop experts on this rank (real kernels fuse this GEMM)
        for le in range(experts_per_rank):
            sel = (local_eid == le)
            if not sel.any():
                continue
            rows = tok_idx[sel]
            xin = x[rows]                                  # tokens sent to this expert
            hmid = F.gelu(xin @ expert_w1[lo + le])        # up-proj + act
            out = hmid @ expert_w2[lo + le]                # down-proj
            # COMBINE: weighted scatter-add back to the home token
            y.index_add_(0, rows, out * gw[sel].unsqueeze(-1))
    return y

# --- tiny smoke test ---
torch.manual_seed(0)
T, d, h, E, k, G = 12, 16, 32, 8, 2, 4
x  = torch.randn(T, d)
rw = torch.randn(E, d) * 0.1
w1 = torch.randn(E, d, h) * (d ** -0.5)
w2 = torch.randn(E, h, d) * (h ** -0.5)
y = moe_ep_layer(x, rw, w1, w2, k, G)
print(y.shape)   # torch.Size([12, 16])

The mask.nonzero step is the heart of the cost model: its length on each rank is how many tokens that rank must process, and it varies wildly with routing. The grouped GEMM in step 3 is a single fused kernel (e.g., CUTLASS grouped GEMM, Triton’s grouped-GEMM tutorial kernel, MegaBlocks’ block-sparse kernels, or DeepSeek’s FP8 DeepGEMM) that runs all local experts’ matmuls with one launch, padding or masking ragged group sizes.

Steps 2 and 4 in a real deployment are torch.distributed.all_to_all_single calls. It is worth writing that out once, because the variable-length version has a subtlety the in-process simulation hides: you must exchange the counts before you can exchange the payload.

# real_ep_alltoall.py — the actual collective. Run with:
#   torchrun --nproc_per_node=2 real_ep_alltoall.py
import torch
import torch.distributed as dist

dist.init_process_group("nccl" if torch.cuda.is_available() else "gloo")
rank, G = dist.get_rank(), dist.get_world_size()
if torch.cuda.is_available():
    torch.cuda.set_device(rank)
dev = f"cuda:{rank}" if torch.cuda.is_available() else "cpu"

T, d, E, k = 8, 16, 8, 2
experts_per_rank = E // G
x    = torch.randn(T, d, device=dev)                        # this rank's local tokens
topi = torch.randint(0, E, (T, k), device=dev)              # stand-in for router output

# --- 1. Permute so tokens for the same DESTINATION RANK are contiguous ---
flat_tok = torch.arange(T, device=dev).repeat_interleave(k)  # (T*k,) home token id
flat_eid = topi.reshape(-1)                                  # (T*k,) chosen expert id
dst_rank = flat_eid // experts_per_rank                      # which GPU owns that expert
order    = torch.argsort(dst_rank)                           # group rows by destination
send_buf = x[flat_tok[order]].contiguous()                   # (T*k, d) payload, permuted
send_counts = torch.bincount(dst_rank, minlength=G)          # rows destined for each rank

# --- 2. METADATA all-to-all: every rank must learn how much it will RECEIVE ---
recv_counts = torch.empty_like(send_counts)
dist.all_to_all_single(recv_counts, send_counts)              # fixed-size, one int per rank
sc, rc = send_counts.tolist(), recv_counts.tolist()           # <-- device->host SYNC

# --- 3. DISPATCH: the payload all-to-all-v, variable splits on both sides ---
recv_buf = torch.empty(sum(rc), d, device=dev, dtype=x.dtype)
dist.all_to_all_single(recv_buf, send_buf,
                       output_split_sizes=rc, input_split_sizes=sc)

# ... local grouped GEMM over recv_buf (each rank's own experts) goes here ...

# --- 4. COMBINE: mirror image — swap the split sizes, send results home ---
back = torch.empty_like(send_buf)
dist.all_to_all_single(back, recv_buf,
                       output_split_sizes=sc, input_split_sizes=rc)
# back[i] belongs to local token flat_tok[order][i]; scatter-add with its gate weight.
print(f"rank {rank}: sent {sc}, received {rc}")
dist.destroy_process_group()

Two things in that listing are the reason production stacks do not just call NCCL. First, the metadata collective is a second round trip on the critical path — two barriers per MoE layer instead of one. Second, send_counts.tolist() is a device-to-host synchronization: the CPU must wait for the router to finish on the GPU before it can even size the receive buffer or launch the next kernel, which drains the pipeline and makes the step impossible to capture in a CUDA graph. Removing both is a central design goal of the kernels in the next section.

The Two Costs: Bytes Moved and the Barrier¶

Per MoE layer, the all-to-all moves about \(k\,d_{\text{model}}\) bytes per token out (dispatch) and the same back (combine), times bytes-per-element. For BF16, \(d_{\text{model}}=7168\) (DeepSeek-V3), \(k=8\):

That is per token, per MoE layer, in each direction — and DeepSeek-V3 has 58 MoE layers. The volume is not the killer on fast interconnect (NVLink at ~900 GB/s, or 8×400 Gb/s InfiniBand per node); the killer is that each all-to-all is a barrier. Decode is one token per request per step, so the batch must cross the all-to-all 2× per MoE layer, and the step cannot finish until the last byte from the slowest rank arrives. Latency, not bandwidth, is what large-EP engineering fights — which is exactly what DeepEP is built for.

DeepEP: A Production All-to-All Kernel¶

all_to_all_single from a generic NCCL build works, but it leaves a lot on the table for MoE decode: it does not overlap dispatch with router computation, it does not exploit the two-tier topology (NVLink within a node, RDMA across nodes), and it synchronizes the GPU with the host to compute split sizes. DeepEP (open-sourced by DeepSeek in early 2025 during its Open Source Week, and the communication library behind DeepSeek-V3/R1 serving) is a purpose-built expert-parallel communication library that addresses all three. The mechanisms generalize to any large-EP serving stack (vLLM and SGLang both integrate DeepEP-style kernels), so they are worth understanding even if you never call the library directly.

DeepEP provides two families of kernels:

• High-throughput (normal) kernels for prefill and training, where batches are large and you want to saturate bandwidth. These do NVLink+RDMA forwarding: a token going to a remote node’s expert is sent over RDMA once, then forwarded over NVLink to the right GPU inside that node, instead of doing \(G\) independent point-to-point sends. This is the all-to-all analogue of a hierarchical all-reduce.
• Low-latency kernels for decode, where batches are tiny and the all-to-all latency directly sets time-per-output-token (TPOT). These use pure RDMA with no NVLink hop (lower latency, fewer synchronization points) and a hook-based overlap scheme: the kernel returns control to Python before the receive completes, lets you run unrelated compute (the attention of the next micro-step, or the router), and you call a hook later that consumes the arrived data. This hides the all-to-all latency behind useful work — communication and computation truly overlap rather than serialize.

Three further tricks matter:

1. FP8 dispatch. DeepEP can quantize the dispatched activations to FP8 (E4M3) on the fly, halving the bytes moved versus BF16. The combine comes back in BF16 (gate-weighted sums are precision-sensitive). This nearly halves dispatch latency at negligible quality cost — the same logic as FP8 KV cache.
2. Communication–computation overlap via the SM-free design. The low-latency kernels are designed to use few or no streaming multiprocessors for the RDMA path (the NIC does the work), leaving the SMs free to compute experts on already-arrived tokens. The classic DeepSeek serving picture is a software pipeline where, at any instant, one micro-batch’s tokens are in flight over RDMA while another micro-batch’s experts are computing.
3. Fixed-size buffers, so no host sync and no metadata round trip. The low-latency path pre-allocates a communication buffer sized by a configured maximum dispatch tokens per rank and pads to it, rather than sizing receives from live counts. That removes both problems we hit in the raw all_to_all_single listing above — the extra metadata collective and the device-to-host sync — and, because every kernel now has static shapes, it lets the whole decode step (attention → dispatch → grouped GEMM → combine) be captured in a CUDA graph (Kernel Fusion, torch.compile, CUDA Graphs & Compilers), collapsing per-layer launch overhead that at decode batch sizes can rival the collective itself. The natural partner on the compute side is DeepGEMM, DeepSeek’s FP8 grouped-GEMM library, whose “masked” layout is designed to consume exactly these padded ragged buffers without a shape-dependent relaunch.

# Sketch of the low-latency decode pattern DeepEP enables (pseudo-API).
# The real library: deep_ep.Buffer.low_latency_dispatch / _combine with a hook.

def moe_decode_step(hidden, router, experts, ep_buffer):
    topk_idx, topk_w = router(hidden)                  # local routing

    # Kick off dispatch; returns immediately with a 'hook' you call later.
    recv_x, recv_layout, hook = ep_buffer.low_latency_dispatch(
        hidden, topk_idx, num_experts=experts.E, use_fp8=True
    )
    # ---- OVERLAP WINDOW: do work that doesn't depend on recv_x ----
    #   e.g., prefetch next layer's expert weights, compute shared-expert FFN,
    #         run the next request stream's attention, etc.
    shared_out = experts.shared_expert(hidden)         # dense path, no all-to-all
    # --------------------------------------------------------------
    hook()                                             # block only now, on arrival

    expert_out = experts.grouped_gemm(recv_x, recv_layout)   # local expert FFNs (FP8 in)
    # Combine: send results home and weighted-sum by gate weights.
    y = ep_buffer.low_latency_combine(expert_out, topk_idx, topk_w)
    return y + shared_out

The single most important idea: on the decode critical path, the all-to-all should not be visible. If your profiler shows the GPU idle inside the EP collective, your overlap is broken, and your effective decode throughput will be a fraction of dense-equivalent. Getting the overlap right is the difference between a large-EP deployment that hits 80%+ of dense-equivalent throughput and one that hits 30%.

Hot-Expert Skew at Decode and Load Balancing¶

Routing is never perfectly uniform, and uniformity is harder to guarantee at inference than in training. Three things make decode skew dangerous.

First, no auxiliary loss is acting. During training, a load-balancing loss (or DeepSeek’s auxiliary-loss-free bias-correction) actively pushes the router toward balance. At inference the weights are frozen; whatever imbalance the deployment-time traffic induces is simply suffered. Worse, real traffic is not the training distribution: a burst of code requests, or one language, or one prompt template repeated across a batch (think a RAG system with a fixed system prompt) can correlate routing decisions and spike a few experts.

Second, decode batches are small, so the law of large numbers does not save you. With \(B=2048\) tokens spread over \(E=256\) experts at \(k=8\), the expected load per expert is \(Bk/E = 2048\cdot 8/256 = 64\) tokens, and the relative fluctuation (\(\propto 1/\sqrt{64}\)) is small. With a low-latency decode batch of \(B=64\) the mean is just \(2\) tokens per expert, so a single popular expert can receive several times the mean, and the all-to-all barrier waits for it.

Third, every step is a barrier. In prefill you process thousands of tokens per request; skew averages out over the prompt and the cost is amortized over a big GEMM. In decode you cross the all-to-all once per token per layer; a hot expert taxes every single step.

Quantify skew with the imbalance factor — the max load over the mean load:

where \(L_g\) is the number of tokens GPU \(g\)’s experts must process. Because the all-to-all completes at the slowest rank, the effective expert-compute time scales with IF. An IF of 1.0 is perfect; an IF of 2.0 means you are wasting half your aggregate expert FLOPs waiting on one GPU.

import numpy as np

def imbalance_factor(token_expert_ids, E, G):
    """token_expert_ids: (T, k) selected expert ids for a decode batch."""
    counts = np.bincount(token_expert_ids.reshape(-1), minlength=E)
    experts_per_rank = E // G
    rank_load = counts.reshape(G, experts_per_rank).sum(axis=1)
    return rank_load.max() / rank_load.mean()

# Uniform routing vs. a skewed batch (one rank's experts are 4x popular)
rng = np.random.default_rng(0)
E, G, T, k = 256, 32, 2048, 8
uniform = rng.integers(0, E, size=(T, k))
print("uniform IF:", round(imbalance_factor(uniform, E, G), 3))    # ~1.05

p = np.ones(E); p[:8] *= 4; p /= p.sum()                            # experts 0..7 hot
skewed = rng.choice(E, size=(T, k), p=p)
print("skewed  IF:", round(imbalance_factor(skewed, E, G), 3))     # > 1.5

The production countermeasures are layered:

• Expert replication / redundant experts. Identify the busiest experts offline (from traffic logs) and place an extra copy on an under-loaded GPU; the dispatch then load-balances tokens for that expert across its replicas. DeepSeek’s deployment calls these redundant experts; a handful of replicas of the top-K hottest experts can cut IF dramatically. The cost is a little extra memory.
• Dynamic expert rebalancing. Periodically (every few seconds) recompute expert placement from recent load and migrate experts to flatten the histogram. This is a control loop, not a per-step decision — migrating experts mid-decode is too expensive.
• Capacity factor and token dropping. As in training, you can cap tokens per expert at \(C = \text{cf} \cdot \frac{B k}{E}\) and drop overflow (the token skips that expert, keeping only its other top-\(k\) choices, or falls back to the shared expert). Dropping bounds the worst-case all-to-all but degrades quality; in serving most stacks prefer no drop with replication instead, because a dropped token is a silently worse answer.
• Shared experts as a relief valve. Models like DeepSeek-V3 route every token through one or two always-on shared experts (a dense path) in addition to the routed ones. The shared expert carries the “common” computation, reduces what the routed experts must specialize in, and — crucially for serving — its FFN runs locally with no all-to-all, giving you free overlap work to hide dispatch latency behind.

The Hybrid Layout: Attention-DP + Expert-EP¶

Here is the layout subtlety that separates a 2024-era MoE deployment from a 2025/2026 one. A large MoE has two very different sublayers with opposite parallelism preferences:

• Attention has few parameters but a large, per-request KV cache. You want to shard it in a way that does not replicate the KV cache and keeps it bandwidth-efficient. With Multi-head Latent Attention (MLA), as used by DeepSeek (see Multi-Head Attention, MQA, GQA & MLA), the per-token KV is a small compressed latent, so each GPU can cheaply hold the full KV for its own requests.
• Experts have most of the parameters and need to be spread (EP) so they fit and so aggregate expert bandwidth is high.

If you naively apply tensor parallelism (TP) to the whole model, you replicate the KV cache across all TP ranks — wasteful — and you split attention heads, which for MLA or low-head-count attention can underutilize each GPU. The modern answer is a hybrid:

Attention runs data-parallel (attention-DP): each GPU handles a disjoint set of requests and keeps those requests’ full KV cache locally. Experts run expert-parallel (EP): the FFN/MoE sublayer is sharded across the same GPUs.

The payoff is large. Attention does zero cross-GPU communication (no all-reduce, no KV replication), so the only collective in the whole layer is the MoE all-to-all. The KV cache is partitioned by request, so total KV capacity scales with GPU count — you can serve more concurrent requests. And because attention is local and cheap, the GPUs have spare time to overlap the MoE all-to-all behind attention of the next micro-batch.

The catch is balance across the DP attention ranks. Each GPU now owns a different set of requests with different sequence lengths and different arrival times. If GPU0’s requests are long (big prefill, big KV) and GPU7’s are short, GPU0 becomes the straggler — and since the MoE all-to-all is a group barrier, everyone waits for GPU0. So the scheduler must balance tokens-in-flight across attention-DP ranks, not just balance experts. This couples request scheduling (Continuous Batching & Request Scheduling) to the parallelism layout in a way dense serving never has to think about.

A common refinement combines a small TP degree for attention (to fit MLA/attention compute on fast GEMMs and to bound per-GPU KV) with DP across TP groups, and EP for experts — written as TP×DP for attention and EP for the MoE, all multiplexed on the same physical GPUs. The exact recipe depends on model shape and interconnect; the invariant is attention and experts are parallelized differently, on the same devices.

Prefill vs. Decode: Why EP Needs Two Modes¶

Prefill and decode stress EP in opposite ways, and a serious deployment configures (or even physically disaggregates; see Disaggregated Prefill/Decode & Chunked Prefill) them differently.

Prefill sends many tokens per expert, so the grouped GEMM is fat and compute-bound; the all-to-all is a bandwidth problem you solve with the high-throughput kernel. Skew matters less because thousands of tokens average out per expert.

Decode sends ~\(Bk/E\) tokens per expert per step — often a handful or even zero. The grouped GEMM is skinny and dominated by overhead and padding; the bottleneck is the all-to-all latency and the per-step barrier. This is why decode wants large EP: spreading experts over more GPUs means each GPU holds fewer experts, so the local grouped GEMM is faster and more aggregate HBM bandwidth feeds expert weights, while DeepEP’s low-latency kernels keep the wider all-to-all from dominating. Counterintuitively, for decode of a huge MoE, more GPUs in the EP group can lower per-token latency — the opposite of dense TP, where adding ranks eventually hurts because the all-reduce grows.

This asymmetry is the core argument for disaggregating prefill and decode onto separate GPU pools for MoE serving: run prefill on a smaller-EP, high-throughput pool, then hand the KV cache to a large-EP, low-latency decode pool. Each pool uses the matching DeepEP kernel and a matching EP degree, and neither compromises for the other.

Offloading: Experts and KV When You Can’t Fit¶

Not everyone has two H100 nodes. The defining property of MoE — most parameters are cold on any given step — makes MoE unusually amenable to offloading: keep hot/shared parameters in HBM and stream cold experts from CPU RAM (or NVMe) on demand. The math that makes this viable: with top-\(k\) routing, a single token needs only \(k\) experts; if you can predict or prefetch which experts the batch will use, you move only those.

The techniques, roughly in order of aggressiveness:

• Keep attention + router + shared expert in HBM; offload routed experts to CPU. Each step, after the router fires, the needed experts are copied HBM-ward over PCIe/NVLink-C2C just-in-time. This works when the per-step expert set is small (small batch, low \(k\)) and the PCIe transfer overlaps compute. It collapses for large batches that touch most experts.
• Expert caching with an LRU. Treat HBM as a cache of experts; keep the most-recently/most-frequently-used experts resident and fault in the rest from CPU. Because real traffic is skewed (the same hot experts again and again), a cache far smaller than the full expert set can achieve a high hit rate. The cache eviction policy is the router’s load histogram — keep the hot experts pinned.
• Pre-gating / look-ahead prefetch. Run the router (or a cheap proxy) a step early so you know which experts the next step needs and start the CPU→GPU copy before you need them, hiding transfer latency behind the current step’s compute.
• NVMe tier (ZeRO-Inference / FlexGen style). For truly memory-starved setups, experts live on NVMe and stream through CPU RAM to GPU. Throughput is low (this is for offline batch, not interactive serving), but it lets a single GPU “run” a model far larger than its HBM. See Memory-Efficient Training for the offload hierarchy and ZeRO-Infinity mechanics.

# Illustrative expert offload with an LRU resident cache (single GPU).
import torch
from collections import OrderedDict

class ExpertCache:
    """Keep `capacity` experts on GPU; fault the rest in from CPU on demand."""
    def __init__(self, cpu_experts, capacity, device="cuda"):
        self.cpu = cpu_experts                  # dict: eid -> (w1_cpu, w2_cpu) pinned
        self.capacity = capacity
        self.device = device
        self.resident = OrderedDict()           # eid -> (w1_gpu, w2_gpu), LRU order

    def get(self, eid):
        if eid in self.resident:
            self.resident.move_to_end(eid)      # mark most-recently-used
            return self.resident[eid]
        if len(self.resident) >= self.capacity:
            old, (w1, w2) = self.resident.popitem(last=False)   # evict LRU
            del w1, w2
        w1c, w2c = self.cpu[eid]
        # non_blocking copy overlaps with compute if src is pinned memory
        w1 = w1c.to(self.device, non_blocking=True)
        w2 = w2c.to(self.device, non_blocking=True)
        self.resident[eid] = (w1, w2)
        return w1, w2

    def prefetch(self, eids):
        """Look-ahead: kick off copies for next step's experts."""
        for e in eids:
            if e not in self.resident:
                self.get(e)                     # warms the cache before it's needed

The honest caveat: offloading trades latency for capacity. PCIe 5.0 moves ~64 GB/s; an expert FFN in DeepSeek-V3 is on the order of tens of MB, so faulting in a handful per step costs hundreds of microseconds to milliseconds of transfer — easily dominating a decode step if not overlapped, and impossible to fully hide once a large batch touches most experts. Offloading is a cost-saving for low-QPS or batch workloads, not a path to low-latency high-throughput serving. For that, you buy the HBM.

The Economics of Dense-Equivalent Throughput¶

Why do labs build these monsters if serving them is this hard? Because the unit economics — discussed in general in Inference Economics: Latency, Throughput & Cost — favor sparsity when you can keep the experts busy.

Frame it as cost per million output tokens. The compute per token is set by active parameters: DeepSeek-V3 at 37B active does roughly the arithmetic of a dense 37B. But it answers with the quality of something far larger than 37B dense. So if you compare at equal quality, the MoE serves each token with a fraction of the FLOPs — provided you reach high GPU utilization. The serving overheads we have catalogued — all-to-all, skew, padding, the memory to hold all experts — are the “tax” on that FLOP advantage. Define dense-equivalent throughput efficiency:

\(\eta = 1\) means the MoE serves as fast as its dense-active twin — you got the quality of a giant model for the price of a small one. A naive single-node MoE deployment, hammered by exposed all-to-all and IF \(\approx 2\), might land at \(\eta \approx 0.3\). A well-engineered large-EP deployment — attention-DP, redundant experts, DeepEP overlap, FP8 dispatch, prefill/decode disaggregation — recovers most of the gap, \(\eta \approx 0.7\)–\(0.9\). That last factor of 2–3× is the entire ballgame; it is why DeepEP, redundant experts, and the hybrid layout exist.

The other half of the economics is batch size. MoE needs large batches to be efficient: only a big batch keeps every expert’s grouped GEMM fat enough to be compute-bound, and only a big batch amortizes the all-to-all overhead per token. A dense model degrades gracefully at small batch (it is just bandwidth-bound); an MoE at small batch is doubly punished — skinny expert GEMMs and a per-step all-to-all barrier with little to overlap. The practical consequence: MoE serving lives at high concurrency. If your traffic cannot fill big batches, a dense model of equal active size may be cheaper to run despite “wasting” parameters. The break-even concurrency is a real number you should compute for your traffic before committing to an MoE in production.

Putting It Together: An EP Serving Checklist¶

A production MoE serving stack (vLLM, SGLang, TensorRT-LLM all converge here) assembles the pieces from this chapter:

1. Layout. Attention-DP (local KV, possibly small TP for MLA) + expert-EP, multiplexed on the same GPUs. Size EP so all experts fit resident with headroom for KV.
2. Communication. DeepEP (or equivalent): high-throughput kernels for prefill, low-latency hook kernels for decode, FP8 dispatch, NVLink+RDMA forwarding across nodes.
3. Balance. Redundant experts for the hot ones, a rebalancing control loop, and a scheduler that balances tokens-in-flight across attention-DP ranks (not just across experts).
4. Disaggregation. Separate prefill (smaller EP, throughput kernels) and decode (large EP, latency kernels) pools; hand off the KV cache between them.
5. Overlap. Drive exposed all-to-all toward zero by scheduling shared-expert compute, attention, and weight prefetch into the dispatch/combine windows.
6. Memory tiering. If HBM is tight, LRU expert caching / offload — but only for low-QPS or batch workloads; interactive low-latency serving needs experts resident.
7. Measure the right thing. Report \(\eta\) (dense-equivalent efficiency), p99 TPOT, and the routing imbalance factor on replayed production traffic, not synthetic uniform routing.

In practice you do not implement any of this: you select it with launch flags. Both mainstream open-source engines expose the same four decisions — EP on/off, the all-to-all backend, DP attention, and redundant experts:

# --- vLLM: attention-DP + expert-EP across 2 nodes (16 GPUs), DeepEP low-latency backend.
# The EP degree is data_parallel_size x tensor_parallel_size; --enable-expert-parallel
# switches the MoE sublayer from TP-sharded to EP-sharded on those same ranks.
VLLM_ALL2ALL_BACKEND=deepep_low_latency \
vllm serve deepseek-ai/DeepSeek-V3 \
  --data-parallel-size 16 --tensor-parallel-size 1 \
  --enable-expert-parallel \
  --enable-eplb          # Expert Parallel Load Balancer: redundant + periodically rebalanced experts

# --- SGLang: the same layout, DeepEP low-latency kernels, 32 redundant expert replicas.
python -m sglang.launch_server --model-path deepseek-ai/DeepSeek-V3 \
  --tp 16 --dp 16 --enable-dp-attention \
  --enable-deepep-moe --deepep-mode low_latency \
  --ep-num-redundant-experts 32

Flag spellings move between releases (check the vLLM and SGLang docs linked below); the four concepts they select are the stable part, and each maps to a section of this chapter. For prefill pools, flip the backend to the high-throughput kernel (deepep_high_throughput / --deepep-mode normal) and use a smaller EP degree, then wire the two pools together with the KV transfer described in Disaggregated Prefill/Decode & Chunked Prefill.

Scope note. None of this machinery is needed at capstone scale: a ~100M-parameter model fits on one GPU, so a small MoE variant of Stack-100M (Evaluation & Serving) keeps every expert resident on a single device and this whole chapter collapses to step 3 of the reference implementation — a grouped GEMM, with no all-to-all at all. Expert parallelism is what you reach for when the expert weights stop fitting in one device’s (then one node’s) HBM; everything above starts to bind at that threshold, not before.

def ep_decode_step_pipeline(batch, layers, ep, sched):
    """
    End-to-end shape of one decode step in a hybrid attention-DP + expert-EP stack.
    Each `layer` interleaves LOCAL attention with an EP MoE, overlapping the all-to-all.
    """
    sched.balance_tokens_across_dp_ranks(batch)        # keep attention ranks even
    h = batch.hidden
    for layer in layers:
        # --- attention: purely local, owns this rank's request KV (MLA) ---
        h = layer.attention(h, batch.kv_cache)         # NO cross-GPU collective
        # --- router + MoE with overlapped all-to-all ---
        topk_idx, topk_w = layer.router(h)
        recv, layout, hook = ep.low_latency_dispatch(   # async; returns a hook
            h, topk_idx, use_fp8=True)
        shared = layer.shared_expert(h)                 # overlap work, no all-to-all
        hook()                                          # block only on arrival
        expert_out = layer.grouped_gemm(recv, layout)   # local experts
        h = ep.low_latency_combine(expert_out, topk_idx, topk_w) + shared
    return layer_norm_and_lm_head(h)

Every line of that loop is a decision this chapter argued for: local attention because of attention-DP, FP8 dispatch and an async hook because of DeepEP overlap, a shared expert sitting in the overlap window, and a scheduler that keeps the DP ranks balanced so the all-to-all barrier never waits on a straggler. That is what it takes to make a trillion-scale sparse model serve at the speed of a small dense one — which is the whole reason MoE exists.

Further reading¶

• DeepSeek-AI, DeepSeek-V3 Technical Report, 2024 — the canonical large-EP MoE: MLA, auxiliary-loss-free balancing, redundant experts, and production EP serving across nodes.
• DeepSeek-AI, DeepEP (open-source repository) — purpose-built expert-parallel all-to-all kernels: high-throughput NVLink+RDMA forwarding, low-latency hook-overlap kernels, and FP8 dispatch.
• Lepikhin et al., GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding, ICLR 2021 — first formalization of expert sharding and all-to-all dispatch/combine at scale.
• Fedus, Zoph & Shazeer, Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity, JMLR 2022 — capacity factor, token dropping, and load-balancing loss that this chapter’s serving countermeasures echo.
• Rajbhandari et al., DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale, ICML 2022 — early systematic treatment of MoE inference optimization and expert parallelism.
• Aminabadi et al., DeepSpeed-Inference / Rajbhandari et al., ZeRO-Infinity, SC 2021 — the offload hierarchy (HBM/CPU/NVMe) underpinning expert and KV offloading.
• vLLM (Kwon et al., 2023) and SGLang (Zheng et al., 2024) project documentation — open-source references for production EP serving, large-EP deployment, and DeepEP integration; see vLLM: Architecture, PagedAttention & Internals and SGLang: RadixAttention & Structured Programs.

Exercises¶

1. (Conceptual) The chapter insists that “a sparse token makes a dense batch.” Explain, in your own words, why almost all experts must stay resident in HBM even though any single token touches only \(k/E\) of them. Then explain why this is the first inversion relative to a dense model, and what it implies for whether expert offloading can ever deliver low-latency, high-concurrency serving.

2. (Quantitative) Take DeepSeek-V3-like numbers: \(E=256\) routed experts, top-\(k=8\), \(d_{\text{model}}=7168\), 58 MoE layers, and a low-latency decode batch of \(B=32\) tokens. (a) Estimate the number of distinct experts the batch touches, and express it as a fraction of \(E\). (b) Compute the dispatch bytes moved per token per MoE layer in BF16 and in FP8 (E4M3). © Compute the total dispatch bytes moved per decode step (all 58 MoE layers) for the whole batch in FP8.

3. (Quantitative) A decode step routes tokens to \(E=8\) experts laid out over \(G=4\) GPUs (2 experts/GPU: experts 0–1 on rank 0, 2–3 on rank 1, 4–5 on rank 2, 6–7 on rank 3). The per-expert token counts this step are:

| expert | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 |
|---|---|---|---|---|---|---|---|---|
| tokens | 30 | 10 | 12 | 8 | 5 | 5 | 5 | 5 |

(a) Compute the imbalance factor IF $=\max_g L_g \,/\, \tfrac{1}{G}\sum_g L_g$.
(b) What fraction of aggregate expert-compute capacity is wasted waiting on the barrier?
(c) The *same* IF of 2.0 would be far less damaging in prefill than in decode. Why?

4. (Conceptual) In dense tensor parallelism, adding more ranks to the TP group eventually raises per-token decode latency. In MoE decode, the chapter claims that adding more GPUs to the EP group can lower per-token latency. Explain both directions and why they point opposite ways.

5. (Implementation) The chapter’s imbalance_factor reports skew but offers no remedy. Implement the chapter’s primary countermeasure — redundant experts — as a function imbalance_factor_replicated(token_expert_ids, E, G, hot_experts, replicas) that places replicas extra copies of each hot expert on other GPUs, splits that expert’s load evenly across its copies, and greedily assigns every expert copy to the least-loaded rank. Show that replicating the hot experts drops IF on the chapter’s skewed example.

import numpy as np

def imbalance_factor_replicated(token_expert_ids, E, G, hot_experts, replicas):
    """
    Rank-level imbalance factor after placing `replicas` extra copies of each
    expert in `hot_experts`. Each replicated expert's load is split evenly across
    its (1 + replicas) copies; copies are greedily assigned to the least-loaded
    rank (LPT). token_expert_ids: (T, k) selected expert ids for a decode batch.
    """
    counts = np.bincount(token_expert_ids.reshape(-1), minlength=E).astype(float)
    hot = {int(e) for e in hot_experts}

    # One entry per expert *copy*; a hot expert contributes several equal shares.
    loads = []
    for e in range(E):
        n_copies = 1 + (replicas if e in hot else 0)
        loads.extend([counts[e] / n_copies] * n_copies)

    # Greedy LPT: biggest copy first onto the currently least-loaded rank.
    rank_load = np.zeros(G)
    for load in sorted(loads, reverse=True):
        r = int(rank_load.argmin())
        rank_load[r] += load
    return rank_load.max() / rank_load.mean()

# --- reproduce the chapter's skewed batch and fix it ---
rng = np.random.default_rng(0)
E, G, T, k = 256, 32, 2048, 8
p = np.ones(E); p[:8] *= 4; p /= p.sum()          # experts 0..7 hot
skewed = rng.choice(E, size=(T, k), p=p)

from numpy import bincount
def imbalance_factor(ids, E, G):                  # chapter's baseline (fixed layout)
    c = bincount(ids.reshape(-1), minlength=E)
    return c.reshape(G, E // G).sum(1).max() / c.reshape(G, E // G).sum(1).mean()

print("baseline  IF:", round(imbalance_factor(skewed, E, G), 3))
print("greedy    IF:", round(imbalance_factor_replicated(skewed, E, G, [], 0), 3))
print("replicated IF:", round(imbalance_factor_replicated(skewed, E, G, range(8), 2), 3))

6. (Quantitative / economics) DeepSeek-V3 has 671B total parameters. (a) In FP8 (1 byte/param), does the weight set fit on a single 8×H100-80GB node? On two nodes (16 GPUs)? Reserve 30 GB/GPU for KV + activations + overhead; how many H100s are the minimum for FP8 weights under that reserve? (b) A naive single-node-style MoE deployment achieves dense-equivalent efficiency \(\eta \approx 0.3\); a well-engineered large-EP deployment reaches \(\eta \approx 0.85\). If the dense-active twin (a hypothetical dense 37B on the same GPUs) serves 2000 output tok/s, what does each MoE deployment serve, and what is the ratio of their cost-per-million-output-tokens?