12.3 Caching, Routing & Cost Control in Production¶

LLM inference is expensive. A single frontier-model call can cost anywhere from a fraction of a cent to several dollars depending on context length and model tier, and at even modest traffic (say, ten thousand daily active users sending five messages each) you are looking at meaningful infrastructure spend before you have written a line of business logic. This chapter is about systematically engineering that bill down — not by compromising quality, but by routing work to the cheapest system that can do it well, avoiding redundant computation wherever possible, and squeezing utilisation out of every GPU-second you pay for.

The techniques here sit at the intersection of distributed systems, economics, and ML: semantic caching, prompt-prefix caching, model routing cascades, speculative routing, quantised fallbacks, intelligent batching, and spot/preemptible GPU scheduling. We will cover the mechanism of each, when to reach for it, and how to wire them together into a coherent cost-control stack.

Cross-references: Inference Economics: Latency, Throughput & Cost covers the per-token cost model; Prefix Caching & KV-Cache Reuse covers the low-level KV-cache reuse mechanism; Continuous Batching & Request Scheduling covers the scheduler side of batching; Quantization I and Quantization II cover the quantisation methods we invoke here as fallbacks. When we serve the Stack-100M model we build in Part XIV, the levers in this chapter are exactly the ones we pull — see Evaluation & Serving: Honest Benchmarks, int4 Quantization, and Running on a Laptop.

Why Does This Problem Exist?¶

Token economics follow a simple formula. Let \(C_\text{input}\) and \(C_\text{output}\) be per-token prices (in USD), and let a request have \(n_p\) prompt tokens and \(n_g\) generated tokens:

Output tokens are typically 3–5× more expensive than input tokens because decoding is memory-bandwidth-bound and fundamentally sequential (see The Anatomy of LLM Inference: Prefill, Decode & The KV Cache). For a 128-token system prompt plus 500-token user message feeding a 300-token response, and illustrative prices of $0.003/1K input and $0.015/1K output, one call costs roughly:

That is small. But at 50,000 calls per day the monthly bill is around USD 9,600 — and that is a single, modest product. Production applications routinely run at 10× to 100× that volume, and frontier models are significantly pricier. The levers are: (a) call fewer tokens, (b) reuse previously computed results, © route to a cheaper model when you can, (d) spread load to reduce idle GPU time.

Exact Caching¶

The simplest optimisation is to remember the answer to a query you have already answered. If you can guarantee that two requests are byte-for-byte identical, you can return the cached response with zero model compute.

What to key on¶

The cache key must cover everything that would change the model’s output:

• Model ID (and version/commit, not just name)
• Full serialised messages array (role + content)
• Sampling hyperparameters (temperature, top-p, top-k, max tokens)
• Any injected system prompt variables

A common mistake is to key only on the user message and miss that the system prompt varies per tenant, producing cross-tenant cache poisoning.

Storage and eviction¶

Redis with a TTL is the standard choice. A SHA-256 hash of the canonicalised request body fits in 32 bytes; the response blob is typically 1–10 KB. With a 90-day TTL and 50,000 RPD the steady-state working set is on the order of a few hundred megabytes — trivially cacheable.

import hashlib, json, redis

client = redis.Redis(host="localhost", port=6379, decode_responses=False)

def _cache_key(model: str, messages: list[dict], params: dict) -> str:
    """
    Deterministic SHA-256 key over the full request.
    sort_keys=True ensures dict ordering never matters.
    """
    payload = json.dumps(
        {"model": model, "messages": messages, **params},
        sort_keys=True, ensure_ascii=False,
    ).encode("utf-8")
    return "llm:exact:" + hashlib.sha256(payload).hexdigest()

def exact_cache_get(model, messages, params, ttl_seconds=86400 * 30):
    key = _cache_key(model, messages, params)
    blob = client.get(key)
    if blob is not None:
        return json.loads(blob)          # cache hit: no model call
    return None

def exact_cache_set(model, messages, params, response, ttl_seconds=86400 * 30):
    key = _cache_key(model, messages, params)
    # SETEX: set with expiry atomically
    client.setex(key, ttl_seconds, json.dumps(response))

Exact caching has a narrow hit rate for conversational workloads (few requests are truly identical) but extremely high value for specific patterns: FAQ bots, templated document generation, CI/CD pipeline prompts, and embedding calls (which are purely deterministic). For embeddings, exact caching can eliminate 60–90% of API calls if users resubmit the same documents.

Handling non-determinism¶

If temperature > 0, returning a cached stale response is semantically wrong for freshness-sensitive queries. A common compromise is to cache only when temperature == 0, or to cache with a short TTL (a few hours) to capture burst traffic while not serving stale creative content for long.

Semantic Caching¶

Exact caching misses near-duplicate queries. Semantic caching embeds the user query and checks whether the embedding is close enough to a previously answered query that the cached answer is still valid.

Similarity threshold selection¶

Let \(\hat{q}\) be the unit-normalised query embedding and \(\hat{c}\) be a cached entry’s embedding. The cosine similarity is \(s = \hat{q} \cdot \hat{c}\). You admit a cache hit when \(s \geq \tau\).

Choosing \(\tau\) is a calibration problem. Too low: you return wrong answers (a FAQ about “cancel subscription” matches “delete my account” — maybe OK) or dangerously wrong ones (“What is the dosage of aspirin” matches “What is the dosage of ibuprofen” — very much not OK). Too high: the hit rate collapses toward exact matching.

A practical approach:

1. Collect a held-out set of (query, gold answer) pairs from your domain.
2. For each \(\tau \in [0.85, 0.99]\) sweep, measure precision (fraction of returned hits that are semantically correct) and recall (fraction of queries served from cache).
3. Pick the \(\tau\) that hits your precision floor (typically 0.97+) while maximising recall.

For general chat bots a threshold near 0.92–0.95 is common. For medical, legal, or financial applications a higher bar (0.97+) or a human-review loop on borderline hits is appropriate.

That domain gap is not arbitrary — it falls out of a decision-theoretic view of \(\tau\). Let \(h(\tau)\) be the hit rate, \(e(\tau)\) the fraction of admitted hits that are semantically wrong, \(c_\text{call}\) the money a hit saves, and \(c_\text{err}\) the cost of serving a wrong answer (refund, escalation, harm). Expected saving per query is

which is positive only when the admitted-hit precision satisfies \(1 - e(\tau) > \dfrac{c_\text{err}}{c_\text{call} + c_\text{err}}\). With \(c_\text{err} \approx c_\text{call}\) you need barely 50% precision and can run a loose threshold; with \(c_\text{err} = 30 c_\text{call}\) you need better than 96.8% precision, which is precisely why regulated domains land at 0.97+ or add a review loop. Sweep \(\tau\), measure \(h\) and \(e\) on held-out data, and pick the \(\tau\) that maximises \(S\) rather than the one that maximises hit rate.

import numpy as np
from typing import Optional

class SemanticCache:
    """
    Minimal semantic cache using cosine similarity.
    In production, replace the linear scan with a FAISS/Qdrant ANN index.
    """
    def __init__(self, embed_fn, threshold: float = 0.93):
        self.embed_fn = embed_fn      # callable: str -> np.ndarray (unit-normed)
        self.threshold = threshold
        self.index: list[tuple[np.ndarray, dict]] = []  # (embedding, entry)

    def _cosine(self, a: np.ndarray, b: np.ndarray) -> float:
        return float(np.dot(a, b))    # assumes unit-norm inputs

    def get(self, query: str) -> Optional[dict]:
        """Return the best cached entry if similarity >= threshold."""
        if not self.index:
            return None
        q_emb = self.embed_fn(query)
        best_score, best_entry = max(
            ((self._cosine(q_emb, emb), entry) for emb, entry in self.index),
            key=lambda x: x[0],
        )
        if best_score >= self.threshold:
            return best_entry          # cache hit
        return None

    def put(self, query: str, response: dict) -> None:
        """Store a new (query, response) pair."""
        q_emb = self.embed_fn(query)
        self.index.append((q_emb, {"query": query, "response": response}))

Embedding model selection and latency¶

The embedding call itself introduces latency. A 100 ms embedding call eats into the savings if the cached path is supposed to be fast. Use a small, locally-hosted embedding model (e.g., a 22M-parameter sentence-transformer) so the embedding call completes in 1–5 ms on CPU. The ANN lookup in a vector store like Qdrant or FAISS is another 1–10 ms at typical scales. Compare that to a frontier model call at 500 ms to 5 s: the speedup is 100×.

In production you rarely hand-roll this. GPTCache (Zilliz) wraps the whole pattern — embedding function, vector backend (FAISS, Qdrant, Milvus), similarity evaluator, eviction policy — behind a drop-in wrapper around the OpenAI/LangChain client, and LiteLLM (below) exposes both an exact Redis cache and a vector-store-backed semantic cache at the gateway. Build the toy version above once to understand the failure modes, then adopt a library so you inherit eviction, TTL, and hit-rate metrics for free.

Semantic caching integrates naturally with RAG systems (see Retrieval-Augmented Generation Architectures) — you can cache at both the retrieval step and the generation step.

Prompt-Prefix Caching (Provider-Side KV Reuse)¶

Distinct from the application-level caches above, providers like Anthropic (prompt caching) and OpenAI (caching) offer server-side KV-cache reuse for repeated prompt prefixes. If you send a 2,000-token system prompt on every call, the provider can skip recomputing the key-value tensors for that prefix after the first request.

The economics are significant. Anthropic’s prompt caching charges roughly 10% of the normal input price for cache-hit tokens — a rate that has held steady through 2026 — plus a modest premium (roughly 1.25× to 2× the base input price, depending on cache TTL) on the initial write. For a 2,000-token system prompt at $0.003/1K tokens:

• Without caching: 2,000 tokens × $0.003/1K = $0.006 per call
• With caching (after first call): 2,000 tokens × $0.0003/1K = $0.0006 per call

At 10,000 calls per day this saves approximately USD 18 per day on the system prompt alone — over USD 6,500 per year.

To maximise prefix cache hits, keep the stable part of your prompt at the top (system instructions, few-shot examples, retrieved context) and put the variable part at the bottom (user message). This is covered in more depth in Prefix Caching & KV-Cache Reuse and Context Engineering & Management.

# Anthropic prompt caching API usage (Python SDK, 2024+)
import anthropic

client = anthropic.Anthropic()

# Mark the static system prompt for caching.
# The provider will reuse KV tensors for this block across calls.
response = client.messages.create(
    model="claude-opus-4-5",
    max_tokens=1024,
    system=[
        {
            "type": "text",
            "text": STATIC_SYSTEM_PROMPT,          # 4000+ tokens
            "cache_control": {"type": "ephemeral"}, # request caching
        }
    ],
    messages=[
        {"role": "user", "content": user_message}  # variable part
    ],
)

# Inspect whether you got a cache hit
usage = response.usage
print(f"Input tokens: {usage.input_tokens}")
print(f"Cache read tokens: {usage.cache_read_input_tokens}")   # billed at 10%
print(f"Cache write tokens: {usage.cache_creation_input_tokens}")  # first call

Self-hosted prefix caching (vLLM, SGLang)¶

If you serve your own weights — a quantised 7B fallback tier, or the Stack-100M model of Part XIV — you get the same mechanism without any provider, because the open-source servers implement it directly. vLLM calls it automatic prefix caching (APC): KV blocks are hashed by their token contents so any request sharing a prefix reuses them; SGLang calls it RadixAttention and keeps the KV blocks in a radix tree so branching prefixes (few-shot templates, multi-turn threads, parallel samples of the same prompt) share ancestors.

# vLLM: automatic prefix caching. Enabled by default in recent (V1-engine)
# releases; the flag is written out explicitly here so the intent is visible.
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --enable-prefix-caching \
  --gpu-memory-utilization 0.90

# SGLang: RadixAttention is on by default. Disable it to A/B measure the win.
python -m sglang.launch_server \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --port 30000
  # add --disable-radix-cache for the no-cache control run

Two differences from the provider case matter operationally. First, the saving is not a line item on an invoice: a hit removes prefill FLOPs you would otherwise buy with GPU-seconds, so it shows up as lower time-to-first-token and higher tokens/second per GPU — you must divide it into your own cost-per-token model. Second, the cache lives in the same GPU memory as the KV cache for running requests and is LRU-evicted, so the hit rate is a function of (working-set of distinct prefixes) versus (free KV blocks): raising --gpu-memory-utilization or shrinking --max-model-len buys hit rate. Both servers export prefix-cache hit counters on their Prometheus /metrics endpoint; treat that number as a first-class dashboard tile alongside your application cache hit rate. Mechanism details are in vLLM: Architecture, PagedAttention & Internals and SGLang: RadixAttention & Structured Programs.

Model Routing and Cascades¶

Not every query needs your most capable (and most expensive) model. A cascade routes each request to the cheapest model that can answer it correctly, escalating to stronger models only when needed.

The cascade can be implemented in two ways:

1. Quality-based escalation (sequential)¶

Call the cheap model first; if its output meets a quality gate, return it. Otherwise call the expensive model. This introduces latency for the escalated fraction, so it is best for workloads where most queries are simple (high hit rate on the cheap tier).

import asyncio
from dataclasses import dataclass
from typing import Callable, Awaitable

@dataclass
class ModelTier:
    name: str
    call_fn: Callable[..., Awaitable[dict]]   # async function
    cost_per_1k_tokens: float                 # illustrative combined cost
    quality_gate: Callable[[dict], bool]      # returns True iff output is good enough

async def cascade(prompt: str, tiers: list[ModelTier]) -> dict:
    """
    Try each tier in order (cheapest first).
    Return the first response that passes its quality gate,
    or the last tier's response unconditionally.
    """
    for i, tier in enumerate(tiers):
        response = await tier.call_fn(prompt)
        is_last = (i == len(tiers) - 1)
        if is_last or tier.quality_gate(response):
            response["_tier_used"] = tier.name
            return response
    # unreachable, but satisfies type checker
    raise RuntimeError("Empty tier list")

# ------- Example quality gates -------

def confidence_gate(response: dict, min_logprob: float = -0.15) -> bool:
    """
    Accept the cheap model's output if average log-probability of
    output tokens is high (model is 'confident').
    Requires logprobs=True in the API call.
    """
    logprobs = response.get("logprobs", [])
    if not logprobs:
        return False
    avg = sum(logprobs) / len(logprobs)
    return avg >= min_logprob

def length_gate(response: dict, max_tokens: int = 200) -> bool:
    """
    Reject cheap model if it hit the token limit — likely incomplete.
    """
    return response.get("finish_reason") != "length"

2. Classifier-based routing (parallel or pre-dispatch)¶

Train a small classifier that predicts which model tier a query belongs to, and route before calling any LLM. This avoids the latency overhead of sequential calls but requires labelled training data (which you can bootstrap from cascade logs: label the query with the cheapest tier that gave good output in the sequential cascade).

# Lightweight routing classifier using a small embedding model + logistic regression.
# In practice you might use a fine-tuned DistilBERT or even a rule-based system.

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import LabelEncoder

class RoutingClassifier:
    """
    Maps a query embedding to a model tier label.
    Train offline on cascade logs; serve online with ~1 ms latency.
    """
    def __init__(self, embed_fn, labels: list[str]):
        self.embed_fn = embed_fn
        self.enc = LabelEncoder().fit(labels)
        self.clf = LogisticRegression(max_iter=500, C=1.0)

    def fit(self, queries: list[str], tier_labels: list[str]) -> None:
        X = np.stack([self.embed_fn(q) for q in queries])
        y = self.enc.transform(tier_labels)
        self.clf.fit(X, y)

    def predict(self, query: str) -> tuple[str, float]:
        """Returns (tier_name, confidence)."""
        emb = self.embed_fn(query).reshape(1, -1)
        proba = self.clf.predict_proba(emb)[0]
        idx = int(np.argmax(proba))
        return self.enc.inverse_transform([idx])[0], float(proba[idx])

Cascade economics¶

Let \(p_s\) be the fraction of queries routed to the small model (hit rate), \(c_s\) the small-model cost per query, and \(c_l\) the large-model cost per query. Expected cost per query:

If the small model handles 70% of traffic (on the sequential cascade that also pays \(c_s\) before escalating), and \(c_s = \$0.002\), \(c_l = \$0.020\):

Versus paying \(c_l\) for everything: USD 0.020. A 2.5× cost reduction for a 70% hit rate.

Speculative Routing¶

Speculative routing is the routing analogue of speculative decoding (see Speculative Decoding: Draft Models, Medusa, EAGLE & Lookahead). Instead of waiting for a sequential cascade decision, you fire both the cheap and expensive models in parallel and discard the expensive result if the cheap one passes the quality gate. This cuts latency to approximately the cheap model’s latency for the common case, while guaranteeing large-model quality for the rest.

The economics are worse than sequential cascade (you always pay both) unless the large model can be cancelled mid-generation when the small model succeeds. Streaming APIs with cancellation make this viable.

import asyncio

async def speculative_route(query: str, small_fn, large_fn, gate_fn):
    """
    Fires both models concurrently. Returns the small model's output if it
    passes the gate; otherwise waits for (and returns) the large model.
    Cancels the large model task if small passes early.
    """
    small_task = asyncio.create_task(small_fn(query))
    large_task = asyncio.create_task(large_fn(query))

    # Await the small model first (it should finish sooner)
    small_resp = await small_task
    if gate_fn(small_resp):
        large_task.cancel()             # stop paying for large model
        try:
            await large_task            # let cancellation propagate cleanly
        except asyncio.CancelledError:
            pass
        return small_resp, "small"

    # Small model failed quality gate; wait for large model
    large_resp = await large_task
    return large_resp, "large"

Speculative routing works best when: (a) the cheap model is 3–10× faster than the expensive one, (b) the quality gate can be evaluated quickly (e.g., a short confidence check, not a slow LLM-as-judge), and © the cancellation saves meaningful tokens (longer outputs).

Quantised Fallbacks¶

Running a smaller quantised model is not just about model routing to a different API endpoint — you can also host a quantised version of the same model locally as a fallback that trades quality for cost and latency. The quantisation taxonomy is covered in depth in Quantization I: Post-Training Quantization (GPTQ, AWQ, SmoothQuant) and Quantization II; here we focus on deployment economics.

Memory and throughput impact¶

A 70B parameter model at FP16 requires approximately 140 GB of GPU memory (2 bytes per parameter). The same model quantised to INT4 (with GPTQ or AWQ) requires around 35 GB — fitting on a single 40 GB A100, versus four A100s for FP16. The cost impact of that memory reduction is dramatic:

For a fallback tier receiving queries that were already routed away from the frontier model, the 1–3% MMLU degradation is often acceptable.

# Loading a GPTQ-quantised model with vLLM for cost-effective fallback serving
from vllm import LLM, SamplingParams

# vLLM natively supports GPTQ/AWQ via the quantization parameter.
# This model fits on a single A100 (40GB) vs 4x A100 for FP16.
fallback_llm = LLM(
    model="TheBloke/Llama-2-70B-Chat-GPTQ",   # example quantised checkpoint
    quantization="gptq",
    dtype="float16",
    gpu_memory_utilization=0.92,               # leave 8% for KV cache headroom
    max_model_len=4096,
)

fallback_params = SamplingParams(
    temperature=0.7,
    top_p=0.9,
    max_tokens=512,
)

def run_fallback(prompts: list[str]) -> list[str]:
    outputs = fallback_llm.generate(prompts, fallback_params)
    return [o.outputs[0].text for o in outputs]

Producing the quantised checkpoint is its own step, and by 2026 you generally quantise your own weights rather than hunting for a community upload (the once-ubiquitous TheBloke repositories stopped being refreshed in 2024, so they only cover older model generations). The standard open-source paths are llm-compressor (Red Hat / Neural Magic), which emits compressed-tensors checkpoints that vLLM loads with quantization="compressed-tensors" and which covers INT8/INT4 weight-only, W8A8, and FP8 recipes; AutoAWQ and GPTQModel for classic AWQ/GPTQ weights; bitsandbytes for zero-effort NF4 loading in HF Transformers; and llama.cpp’s quantisers for GGUF when the fallback tier is CPU or a laptop. All of them want a small calibration set (typically a few hundred in-domain sequences) — use samples of your production traffic, not generic web text, or you will calibrate the activation scales for the wrong distribution. The capstone walks the whole loop at 100M scale — quantise to int8 then int4, re-measure the benchmark deltas, and export for CPU serving with these same tools as the upgrade path: Evaluation & Serving.

When to invoke a quantised fallback¶

Quantised fallbacks fit into the routing cascade as the cheap-but-hosted tier, between the semantic cache and the frontier API call. You can also use them for latency degradation gracefully: when the frontier API returns a 429 rate-limit or a timeout, fall back to the local quantised model rather than returning an error to the user.

import httpx

async def call_with_quantised_fallback(prompt: str) -> dict:
    try:
        # Attempt frontier model first
        resp = await call_frontier_api(prompt, timeout=3.0)
        return resp
    except (httpx.TimeoutException, httpx.HTTPStatusError) as e:
        # Fall back to locally hosted quantised model
        text = run_fallback([prompt])[0]
        return {"text": text, "_source": "quantised_fallback"}

Batching for Cost¶

Batching is the primary lever for maximising GPU utilisation and therefore amortising fixed GPU cost across more tokens. There are two distinct batching strategies relevant to cost control.

Continuous (in-flight) batching¶

The inference server (vLLM, SGLang, TGI) uses continuous batching (also called iteration-level batching) to keep the GPU full across requests with different lengths. This is covered architecturally in Continuous Batching & Request Scheduling; the cost angle is that higher effective batch size directly reduces cost per token:

At batch size 1 a modern GPU may generate on the order of 1,000–5,000 tokens/second for a 7B model. At batch size 32 the same GPU generates 10,000–30,000 tokens/second — a 5–6× throughput improvement for the same hardware cost. This is why GPU utilisation is the KPI: each percentage point of utilisation is free capacity.

Offline batching for async workloads¶

Not all LLM workloads are latency-sensitive. Nightly report generation, bulk document summarisation, evaluation runs, and data labelling jobs can tolerate minutes of latency. For these, the cloud provider’s batch inference API (Anthropic Batches API, OpenAI Batch API) offers a significant discount — on the order of 50% — in exchange for up to 24-hour turnaround.

import anthropic, json, time

batch_client = anthropic.Anthropic()

def run_batch_job(requests: list[dict]) -> list[dict]:
    """
    Submit a batch of requests to Anthropic Batches API.
    Up to 50% cheaper; results available within 24 hours.
    """
    # Build the batch request list
    batch_requests = [
        {
            "custom_id": f"req-{i}",
            "params": {
                "model": "claude-opus-4-5",
                "max_tokens": 1024,
                "messages": req["messages"],
            },
        }
        for i, req in enumerate(requests)
    ]

    # Submit the batch
    batch = batch_client.messages.batches.create(requests=batch_requests)
    print(f"Batch created: {batch.id}, status: {batch.processing_status}")

    # Poll until complete (in production: use a webhook or async poller)
    while batch.processing_status == "in_progress":
        time.sleep(60)
        batch = batch_client.messages.batches.retrieve(batch.id)
        print(f"  Status: {batch.processing_status}, "
              f"succeeded: {batch.request_counts.succeeded}, "
              f"errored: {batch.request_counts.errored}")

    # Collect results
    results = []
    for result in batch_client.messages.batches.results(batch.id):
        if result.result.type == "succeeded":
            results.append({
                "custom_id": result.custom_id,
                "text": result.result.message.content[0].text,
            })
    return results

The batch API is the right choice for any pipeline that is not user-interactive: bulk summarisation, scheduled report generation, embedding generation for new documents, and offline evaluation suites.

Spot and Preemptible GPUs¶

Cloud GPUs come in two flavours: on-demand (always available, full price) and spot/preemptible (heavily discounted — typically 60–80% cheaper — but can be reclaimed by the cloud provider with 30–120 seconds notice). For a well-designed LLM serving system, spot instances are highly tractable.

Architecture for spot resilience¶

Key design decisions:

1. Stateless inference workers. Each worker loads the model from shared storage (EFS, GCS) at startup; no local mutable state. Preemption loses nothing.
2. Short request timeouts + retries. Set a 10-second timeout per inference request. If a spot instance is reclaimed mid-request, the load balancer retries on another instance. For a 2-second P95 SLA, this is usually invisible.
3. Keep a minimum on-demand floor. Even 1 on-demand instance per deployment ensures some capacity remains during a spot shortage.
4. Preemption signal handling. Cloud providers send a SIGTERM (GCP) or a metadata flag (AWS) 30–120 seconds before reclamation. The worker should stop accepting new requests and drain in-flight ones.

import signal, sys

class ServiceUnavailableError(Exception):
    """Raised when the worker is draining and cannot accept new requests."""

class InferenceWorker:
    def __init__(self):
        self.accepting_new = True
        self.in_flight = 0
        # Register spot preemption handler
        signal.signal(signal.SIGTERM, self._handle_preemption)

    def _handle_preemption(self, signum, frame):
        """
        Called ~30-120s before the spot instance is reclaimed.
        Stop accepting new requests; let in-flight ones complete.
        """
        print("SIGTERM received — draining inference worker", file=sys.stderr)
        self.accepting_new = False
        # Health check endpoint will start returning 503,
        # causing the load balancer to stop sending new requests.

    async def handle_request(self, request):
        if not self.accepting_new:
            raise ServiceUnavailableError("Worker is draining")
        self.in_flight += 1
        try:
            return await self._run_inference(request)
        finally:
            self.in_flight -= 1

Spot instance cost modelling¶

Suppose you have a serving system requiring 4 GPU-hours per hour at steady state. On-demand A10G: on the order of USD 1.50/GPU-hour; spot A10G: on the order of USD 0.45/GPU-hour (70% discount). A mixed fleet of 75% spot, 25% on-demand costs:

Versus all on-demand at USD 6.00/hour. A 2.1× cost reduction purely from instance type selection, with near-transparent resilience.

Putting It All Together: A Production Cost-Control Stack¶

The techniques above are not independent — they compose into a layered stack, and the order matters.

import asyncio
from dataclasses import dataclass
from typing import Optional

@dataclass
class CostControlStack:
    exact_cache: "ExactCache"
    semantic_cache: "SemanticCache"
    router: "RoutingClassifier"
    simple_model_fn: callable
    medium_model_fn: callable
    frontier_model_fn: callable

    async def handle(self, request: dict) -> dict:
        messages = request["messages"]
        query = messages[-1]["content"]          # last user turn

        # --- Layer 1: Exact cache ---
        hit = self.exact_cache.get(request)
        if hit:
            return {**hit, "_source": "exact_cache"}

        # --- Layer 2: Semantic cache ---
        hit = self.semantic_cache.get(query)
        if hit:
            return {**hit["response"], "_source": "semantic_cache"}

        # --- Layer 3: Route to cheapest capable model ---
        tier, confidence = self.router.predict(query)

        if tier == "simple":
            resp = await self.simple_model_fn(request)
        elif tier == "medium":
            resp = await self.medium_model_fn(request)
        else:  # "complex"
            resp = await self.frontier_model_fn(request)

        resp["_tier"] = tier
        resp["_router_confidence"] = confidence

        # --- Fill both caches for future requests ---
        self.exact_cache.set(request, resp)
        self.semantic_cache.put(query, resp)

        return resp

Doing this with a real gateway (LiteLLM)¶

The stack above is written out longhand so that nothing is a black box, but in production most teams put a gateway in front of everything: one OpenAI-compatible endpoint that owns keys, retries, caching, routing, fallbacks, and per-tenant spend limits, so application code never learns which provider answered. LiteLLM is the de-facto open-source choice for that layer as of 2026 (it speaks 100+ providers, and any vLLM/SGLang server you run yourself is just another OpenAI-compatible backend in its model list).

# pip install "litellm[proxy]"
import litellm
from litellm import Router

# Layer 1: exact response cache, keyed on the full request (model + messages +
# params) — the same hashing discipline as our _cache_key, done for us.
litellm.cache = litellm.Cache(type="redis", host="localhost", port="6379")

router = Router(
    model_list=[
        # Cheap tier: our own quantised model served by vLLM on spot GPUs.
        {"model_name": "cheap",
         "litellm_params": {"model": "openai/local-7b-int4",
                            "api_base": "http://vllm-spot:8000/v1",
                            "api_key": "EMPTY"}},
        # Frontier tier: a hosted API.
        {"model_name": "frontier",
         "litellm_params": {"model": "anthropic/claude-opus-4-5"}},
    ],
    fallbacks=[{"cheap": ["frontier"]}],   # 429 / timeout / 5xx on cheap -> frontier
    num_retries=2,
    routing_strategy="latency-based-routing",   # among replicas of one model_name
)

# Our classifier from above still decides the tier; the gateway executes it,
# caches the result, records spend, and handles the failure path we hand-coded.
router_clf = RoutingClassifier(embed_fn, labels=["cheap", "frontier"])
router_clf.fit(train_queries, train_tier_labels)   # from cascade logs

messages = [{"role": "user", "content": query}]
tier, _confidence = router_clf.predict(query)      # "cheap" | "frontier"
resp = router.completion(model=tier, messages=messages, caching=True)

Note the division of labour. The gateway owns the mechanical concerns — caching, retries, fallback on 429/timeout, load-balancing across replicas, per-key budgets — which is exactly the quantised-fallback wrapper and exact cache we wrote by hand above. What it does not own is the judgement: which tier a query deserves, what the quality gate is, and what similarity threshold is safe for your domain. Those stay yours. RouteLLM plugs into the same seam as a trained preference-data router if you would rather not train your own classifier, and LiteLLM’s semantic-cache mode covers Layer 2 when you do not want a separate service.

No cost-control stack is set-and-forget. Wire up the following metrics:

• Cache hit rate (exact and semantic, separately) — alert if it drops, investigate if query distribution shifted.
• Tier distribution (fraction of traffic to each model tier) — regressions in routing quality show up as unexpectedly high frontier usage.
• Cost per request — tagged by product feature, user cohort, model tier. Allows per-feature cost attribution.
• Quality-gate pass rate — tracks whether the cheap tier is regressing (model update, distribution shift).

See Observability, Logging & LLMOps for the observability infrastructure that powers these dashboards.

Key Takeaways¶

Further Reading¶

• Kang et al., “LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion,” ACL 2023 — foundational work on ensembling and routing across LLMs.
• Chen et al., “FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance,” 2023 — introduces the LLM cascade framework and cost-quality trade-off analysis.
• Vllm project (Kwon et al., “Efficient Memory Management for Large Language Model Serving with PagedAttention,” SOSP 2023) — the continuous batching and KV-cache management paper underlying most open-source serving stacks.
• Dao et al., “FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning,” ICLR 2024 — understanding IO-efficient attention is prerequisite to understanding why KV-cache reuse saves so much.
• Lin et al., “AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration,” MLSys 2024 — the quantisation method most commonly used in quantised fallback deployments.
• SGLang RadixAttention (Zheng et al., “SGLang: Efficient Execution of Structured Language Model Programs,” 2024) — prefix cache reuse at the serving-system level, complementing provider-side caching.
• GPTCache (GitHub: zilliztech/GPTCache) — open-source semantic cache library with pluggable vector stores and embedding backends, useful as a reference implementation.

Exercises¶

1. (Conceptual) The _cache_key function in the Exact Caching section hashes a JSON payload containing model, messages, and all sampling params. Suppose an engineer “simplifies” it to key only on the last user message string. Describe two distinct failure modes this introduces, and explain why the chapter insists the model version (not just the name) belongs in the key.

2. (Quantitative) You run a documentation assistant with a fixed 3,000-token system prompt. Prices are $0.003/1K input, $0.0003/1K cached input, and traffic is 15,000 calls/day. With provider-side prompt caching the system prompt is a cache hit on 90% of calls and a full-price miss (first call of a fresh prefix) on the remaining 10%. Ignoring the cache-write premium (as the chapter’s worked example does) and considering only the system-prompt portion of the bill, compute the daily cost with and without prompt caching, and the annual saving.

3. (Quantitative) Using the sequential-cascade cost model from the chapter, \(\mathbb{E}[\text{cost}] = c_s + (1 - p_s)\,c_l\), take a cheap tier at \(c_s = \$0.001\) and a frontier tier at \(c_l = \$0.015\). (a) Compute the expected cost and the cost-reduction factor versus always calling the frontier model when the cheap tier handles \(p_s = 0.60\) of traffic. (b) Derive the break-even hit rate \(p_s^\star\) below which the sequential cascade is more expensive than just calling the frontier model directly. Comment on what makes the break-even so low.

4. (Quantitative) A serving system needs 8 GPU-hours per hour at steady state. On-demand A10G costs $1.20/GPU-hour; spot A10G costs $0.36/GPU-hour. (a) Compute the hourly cost of an 80%-spot / 20%-on-demand fleet and the reduction versus all on-demand. (b) Your finance team caps GPU spend at $5.00/hour. What is the minimum fraction of the fleet that must run on spot to stay within budget? Express the answer as a fraction and comment on whether it still leaves room for an on-demand floor.

5. (Implementation) The chapter’s “Handling non-determinism” note argues you should not cache high-temperature responses forever, and suggests caching only when temperature == 0, or otherwise using a short TTL. Modify exact_cache_set (and add a small guard to exact_cache_get if needed) so that: requests with temperature == 0 are cached with the default 30-day TTL, requests with temperature > 0 are cached with a short 2-hour TTL, and the caller can opt out entirely by passing cache_nonzero_temp=False (in which case non-deterministic requests are never written). Keep the chapter’s Redis/SHA-256 style.

import hashlib, json, redis

client = redis.Redis(host="localhost", port=6379, decode_responses=False)

DETERMINISTIC_TTL = 86400 * 30   # 30 days
NONDETERMINISTIC_TTL = 3600 * 2  # 2 hours

def _cache_key(model: str, messages: list[dict], params: dict) -> str:
    payload = json.dumps(
        {"model": model, "messages": messages, **params},
        sort_keys=True, ensure_ascii=False,
    ).encode("utf-8")
    return "llm:exact:" + hashlib.sha256(payload).hexdigest()

def exact_cache_set(model, messages, params, response,
                    cache_nonzero_temp: bool = True) -> bool:
    """
    Write the response with a freshness-appropriate TTL.
    Returns True if the entry was written, False if skipped.
    """
    temperature = params.get("temperature", 0.0)
    if temperature == 0:
        ttl = DETERMINISTIC_TTL          # safe to cache long: reproducible
    else:
        if not cache_nonzero_temp:
            return False                 # caller opted out of caching sampling
        ttl = NONDETERMINISTIC_TTL       # burst capture only, expires quickly
    key = _cache_key(model, messages, params)
    client.setex(key, ttl, json.dumps(response))
    return True

def exact_cache_get(model, messages, params):
    # No change needed: a stale non-deterministic entry simply expires via
    # its short TTL, so a miss after 2 hours is automatic. The key already
    # includes temperature, so temperature==0 and temperature>0 never mix.
    key = _cache_key(model, messages, params)
    blob = client.get(key)
    return json.loads(blob) if blob is not None else None

6. (Implementation) The chapter’s SemanticCache does a Python-level linear scan and stores entries forever. For a cache of \(N\) entries, that is \(O(N)\) Python-object cosine calls per lookup and unbounded memory growth. Rewrite get/put to (a) hold all embeddings in a single contiguous NumPy matrix and score every entry with one vectorised matrix-vector product, and (b) support TTL-based expiry so stale entries (e.g., after a knowledge-base update) drop out. Preserve the unit-norm cosine assumption and the threshold semantics.

import time
import numpy as np
from typing import Optional

class SemanticCache:
    """
    Vectorised semantic cache with TTL expiry.
    Embeddings are assumed unit-normed, so cosine == dot product.
    """
    def __init__(self, embed_fn, threshold: float = 0.93,
                 ttl_seconds: float = 86400.0, dim: Optional[int] = None):
        self.embed_fn = embed_fn
        self.threshold = threshold
        self.ttl = ttl_seconds
        self._mat = None if dim is None else np.empty((0, dim), dtype=np.float32)
        self._entries: list[dict] = []      # parallel to rows of _mat
        self._timestamps: list[float] = []  # insertion time per row

    def _expire(self) -> None:
        """Drop rows older than the TTL (lazy compaction)."""
        if not self._entries:
            return
        now = time.time()
        ts = np.asarray(self._timestamps)
        keep = ts >= (now - self.ttl)
        if keep.all():
            return
        self._mat = self._mat[keep]
        self._entries = [e for e, k in zip(self._entries, keep) if k]
        self._timestamps = [t for t, k in zip(self._timestamps, keep) if k]

    def get(self, query: str) -> Optional[dict]:
        self._expire()
        if self._mat is None or self._mat.shape[0] == 0:
            return None
        q = self.embed_fn(query).astype(np.float32)   # unit-normed
        scores = self._mat @ q                          # one mat-vec, (N,)
        idx = int(np.argmax(scores))
        if scores[idx] >= self.threshold:
            return self._entries[idx]                   # cache hit
        return None

    def put(self, query: str, response: dict) -> None:
        emb = self.embed_fn(query).astype(np.float32).reshape(1, -1)
        if self._mat is None:
            self._mat = np.empty((0, emb.shape[1]), dtype=np.float32)
        self._mat = np.vstack([self._mat, emb])
        self._entries.append({"query": query, "response": response})
        self._timestamps.append(time.time())