15.6 Serving, Quantizing and Evaluating: vLLM, lm-eval-harness, llama.cpp¶

Chapter 15.5 handed off an aligned checkpoint — ckpts/stack-100m-grpo, the output of SFT → DPO → narrow GRPO run through TRL — sitting on disk in Hugging Face format. Chapter 14.11 already did the conceptual work this chapter is about: it built a token-level generate() loop, hand-wrote round-to-nearest int8/int4 quantization with a bespoke QuantizedLinear, and assembled a small, honest capability battery from scratch. Every mechanism in that chapter — the KV cache byte-counting, the group-wise quantization math, the “re-run the eval after every edit” discipline — still holds. What changes here is the plumbing: instead of a Python for loop calling model.generate() one request at a time, you get continuous batching serving hundreds of concurrent requests; instead of a hand-rolled QuantizedLinear only stacklm’s own loader understands, you get a calibration-driven quantizer that writes a format three different serving engines already read; instead of five bespoke probes in a Python list, you get a standardized harness whose numbers are comparable to every other model’s numbers.

This is the last chapter of Part XV, and it closes the loop the part opened in 15.1: the same Stack-100M, the same architecture, the same aligned checkpoint — now shipped with vLLM for GPU serving, llm-compressor/AutoAWQ for production post-training quantization, GGUF/llama.cpp for the CPU/laptop path, and lm-evaluation-harness for evaluation you can defend to someone who wasn’t in the room when you wrote the probes. Three separate, substantial open-source systems own three rows of the crosswalk table from 15.1; this chapter gives each the room it needs.

1. Where We Pick Up, and What This Chapter Adds¶

Recall the state of the checkpoint. Ch. 15.5 exported stacklm’s custom nn.Module — RoPE with every-4^th-layer NoPE, GQA, QK-norm, SwiGLU, tied embeddings (Ch. 14.4) — into a LlamaConfig/LlamaForCausalLM-shaped checkpoint via a remap_state_dict that renames parameters, gated by a fixed-batch assertion that the exported model’s logits match the original’s to under 1e-4. Read that gate carefully: only an export that preserves the architecture can pass it, which means path (1) below. Two of Stack-100M’s architectural choices, NoPE and QK-norm, do not fit in a stock LlamaConfig; 15.5 flagged this explicitly and offered two honest paths: (1) ship a modeling_stacklm.py alongside the checkpoint and load it with trust_remote_code=True, preserving bit-exact fidelity, or (2) accept the faithful-enough Llama export, which drops those two knobs but is readable by every tool in this chapter without a line of custom code.

This chapter takes path (2) as the default, and says exactly why: llm-compressor, AutoAWQ, and llama.cpp’s GGUF converter are all written against well-known HF architectures (Llama, Qwen2, Mistral, …). Feeding them a trust_remote_code=True custom model either fails outright or forces you to also write custom kernels for whichever tool you’re using — a real cost, not a hypothetical one. If your production model leans harder on exotic architecture choices than Stack-100M’s two small ones, path (1) is still there, and vLLM in particular can run a trust_remote_code HF model through its Transformers backend — a fallback execution path, distinct from vLLM’s natively optimized model classes, that trades some throughput for “it just runs your AutoModelForCausalLM.” That escape hatch is not free, though, and it is not universal: the modeling file has to satisfy vLLM’s compatibility contract — attention dispatched through Transformers’ ALL_ATTENTION_FUNCTIONS registry (so vLLM can substitute its own paged-attention implementation), the class advertising _supports_attention_backend = True, and **kwargs threaded through the forward chain. A modeling_stacklm.py lifted straight from Ch. 14.4, whose Attention.forward calls F.scaled_dot_product_attention directly, is rejected rather than silently run slower until that one path is rewritten. Which flag enables it (--model-impl transformers as of some 2025 releases) is exactly the kind of surface that moves between vLLM versions; treat the name below as illustrative and check vllm serve --help against your pin.

Here is the piece of the crosswalk table this chapter expands, first sketched in 15.1:

Every row below follows the same shape: name the library, show it running against the real checkpoint, and say plainly what it buys you that the from-scratch version did not.

2. Serving with vLLM: Continuous Batching, PagedAttention, and the OpenAI API¶

vLLM is the 2026 default for GPU-backed open-weight serving, and the reason is architectural, not fashion. Ch. 14.11’s generate_text processes one request fully — prefill, then decode token by token — before starting the next. That is fine for an eval probe; it is disastrous for a server, because requests arrive at different times, finish at different lengths, and a naive batcher either waits to fill a static batch (adding latency) or runs GPUs at low utilization (wasting money). Continuous batching fixes this at the scheduler level: at every decode step, the server admits newly arrived requests and evicts newly finished ones from the same running batch, so the GPU is never idle waiting for the slowest request in a static group. PagedAttention (Ch. 4.6) is the memory mechanism that makes this practical: it manages the KV cache in fixed-size blocks (like OS virtual memory pages) instead of one contiguous per-request allocation, so blocks can be allocated, freed, and — crucially for prefix caching — shared, on demand as the batch composition churns every step. Chapter 7.3 derives both mechanisms in full; here we use them.

Install and an offline batch run¶

# CUDA wheel; vLLM releases roughly biweekly and the exact pin below will be
# stale by the time you read this — check `pip index versions vllm` and pin
# whatever you tested against, together with the transformers/torch versions
# vLLM's own compatibility matrix names for that release.
pip install "vllm==0.8.5"   # illustrative pin — verify against vllm's release notes

"""
offline_batch.py -- vLLM's non-server API. Good for eval sweeps (Section 5) and
for sanity-checking a checkpoint before you stand up a server.
"""
from vllm import LLM, SamplingParams

# gpu_memory_utilization is the fraction of the GPU's TOTAL memory vLLM budgets
# for weights + KV cache + activation scratch. Treat it as memory the process
# really does take, not a ceiling it might stay under: vLLM profiles the
# activation peak, then EAGERLY allocates the KV block pool to fill whatever is
# left of the budget, and holds it for the engine's lifetime. At 101M params
# there is no reason to hand vLLM the whole card; 0.3 leaves headroom for
# anything else running on the box (Section 2.3 works out why this barely
# matters for a model this small).
llm = LLM(
    model="ckpts/stack-100m-grpo",
    dtype="bfloat16",
    max_model_len=2048,           # matches Stack-100M's pretrain context (Ch. 14.4)
    gpu_memory_utilization=0.3,
)

params = SamplingParams(temperature=0.0, max_tokens=64)
prompts = [
    "Compute 37 + 8. Give the final integer after '####'.",
    "Compute 91 - 46. Give the final integer after '####'.",
]
outputs = llm.generate(prompts, params)   # continuous batching applies even to
                                            # a Python list -- vLLM schedules the
                                            # whole list as it would concurrent
                                            # HTTP requests
for o in outputs:
    print(o.prompt, "->", o.outputs[0].text)

This one call replaces stacklm’s entire generate_text + manual batching loop. There is no code here implementing continuous batching, prefix caching, or memory management — that is exactly the point: those are the twelve person-years the library absorbed.

vllm serve: the OpenAI-compatible endpoint¶

vllm serve ckpts/stack-100m-grpo \
    --served-model-name stack100m \
    --dtype bfloat16 \
    --max-model-len 2048 \
    --gpu-memory-utilization 0.3 \
    --port 8000

The chat template you registered on the tokenizer in Ch. 15.5 — the Jinja STACK_CHAT_TEMPLATE with its {% generation %} markers — ships inside tokenizer_config.json and travels with the checkpoint, so vllm serve applies it automatically to every /v1/chat/completions request. No extra flag is needed for the common case; you only reach for --chat-template if you want to override what the checkpoint carries.

curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
      "model": "stack100m",
      "messages": [
        {"role": "user", "content": "Compute 37 + 8. Give the final integer after #### ."}
      ],
      "temperature": 0,
      "max_tokens": 64
    }'

The response is a standard OpenAI chat-completion object — choices[0].message.content, usage.prompt_tokens/usage.completion_tokens — the same shape a client written against a hosted frontier API expects. This is the practical payoff of the OpenAI-compatible surface: any tool built to talk to a commercial API (a LangChain/LlamaIndex client, an evaluation harness, your own agent loop from Ch. 14.10) talks to ckpts/stack-100m-grpo with a one-line base-URL change.

vLLM also exposes Prometheus-format metrics:

curl -s localhost:8000/metrics | grep '^vllm:'

The metric names are one of vLLM’s faster-moving surfaces — check your pinned version’s actual output rather than trusting a list in a book — but the categories are stable across releases: running/waiting request counts (num_requests_running, num_requests_waiting), KV-cache occupancy (gpu_cache_usage_perc), and the two latency histograms that matter for user-perceived quality, time-to-first-token and time-per-output-token. Wiring these into Grafana/Prometheus is the standard way to watch a serving fleet — see Observability, Logging & LLMOps for the dashboard side of this, and Designing an LLM Serving System for how these numbers feed capacity planning.

A tiny model on a big GPU: why you are never memory-bound here¶

3. Production Quantization: llm-compressor and AutoAWQ¶

Chapter 14.11 §6–7 built round-to-nearest (RTN) quantization from scratch and sketched GPTQ’s column-by-column Hessian-guided reconstruction and AWQ’s activation-aware channel rescaling by hand — read those sections for the derivations; we do not repeat them here. What changes in production is threefold: (1) a maintained implementation that drives the whole error-propagation path from a single Cholesky factor of \(H^{-1}\) instead of the pedagogical inv(H) sketch (the same \(O(d_{in}^3)\) order, but numerically stable in fp32 and free of the per-column \(H^{-1}\) updates a literal Optimal-Brain-Surgeon transcription would redo \(d_{in}\) times), handles every layer of a real transformer (not five hand-picked ones) with the correct cross-layer sharing constraints AWQ needs, and is tested against dozens of real architectures; (2) calibration-set plumbing — tokenizing, batching, capturing activations — that you no longer write yourself; and (3) a standard on-disk format, compressed-tensors, that vLLM (and increasingly other engines) load directly, with no custom loader — unlike stacklm’s bespoke int4 safetensors format from Ch. 14.11 §7.4, which only stacklm.serve.quantize.load_quantized knows how to read.

GPTQ via llm-compressor¶

llm-compressor is vLLM’s own project for producing quantized checkpoints; it is the maintained successor to the earlier standalone GPTQ tooling and shares the compressed-tensors format with vLLM’s native loader.

"""
gptq_quantize.py -- production INT4 GPTQ, operating on the HF checkpoint from
Ch. 15.5. Replaces stacklm.serve.quantize's gptq_quantize_column_by_column
sketch (Ch. 14.11 Section 6.2) with the maintained implementation.

pip install "llm-compressor==0.3.1"   # young, fast-moving package -- check the
                                        # repo's examples/ directory against
                                        # whatever you pin; the oneshot() API
                                        # shape below has already changed once.
"""
from llmcompressor.transformers import oneshot
from llmcompressor.modifiers.quantization import GPTQModifier
from transformers import AutoModelForCausalLM, AutoTokenizer
from datasets import load_dataset

MODEL_DIR = "ckpts/stack-100m-grpo"
model = AutoModelForCausalLM.from_pretrained(MODEL_DIR, torch_dtype="auto")
tokenizer = AutoTokenizer.from_pretrained(MODEL_DIR)

# GPTQ's Hessian is built from REAL activations run through the layer -- the
# X_calib argument in Ch. 14.11's from-scratch gptq_quantize_column_by_column().
# llm-compressor handles tokenize-and-forward bookkeeping; the DATA CHOICE is
# still yours. Calibrate on a held-out slice of the SAME mix the model trained
# on (Ch. 14.2's FineWeb-Edu/Cosmopedia/code/math blend) -- calibrating on
# out-of-distribution text (e.g. pure code) measurably skews which channels
# GPTQ decides are salient, the same failure mode Ch. 14.11 warns about for the
# hand-rolled version.
calib = load_dataset(
    "json", data_files="data/holdout_calib.jsonl", split="train"
).select(range(512))

recipe = GPTQModifier(
    targets="Linear",
    scheme="W4A16",          # 4-bit weights, activations stay bf16/fp16
    ignore=["lm_head"],      # leave the tied embedding/head unquantized (bf16) --
                              # the same instinct as Ch. 14.11's embedding_bits=8,
                              # though strictly higher precision than it, and for
                              # the same reason: the 32768x512 table is 17% of the
                              # params and its rare-token rows are thinly trained.
                              # For int8 parity instead, target lm_head with its
                              # own W8A16 config_groups entry rather than ignoring it
    dampening_frac=0.01,     # the same Hessian damping (`damp`) as Ch. 14.11's
                              # `H += damp * I`
)

oneshot(
    model=model,
    tokenizer=tokenizer,
    dataset=calib,
    recipe=recipe,
    max_seq_length=2048,
    num_calibration_samples=512,
)

model.save_pretrained("ckpts/stack-100m-gptq-w4a16", save_compressed=True)
tokenizer.save_pretrained("ckpts/stack-100m-gptq-w4a16")

Serving it needs no export step, unlike the hand-rolled path:

vllm serve ckpts/stack-100m-gptq-w4a16 --served-model-name stack100m-int4

vLLM detects the compressed-tensors quantization config in the checkpoint’s config.json and dispatches to the matching int4 kernel automatically. Compare this to Ch. 14.11’s §7, where QuantizedLinear.forward dequantizes to fp32 and does a standard matmul — a memory win with no matmul speedup, explicitly flagged there as “not a fast path.” vLLM’s GPTQ/AWQ kernels (Marlin and friends) instead fuse the dequantization into the matmul: the int4 weights are unpacked and scaled in registers/shared memory, tile by tile, and the full-precision weight tensor is never materialized in HBM. The MMA math is still bf16/fp16 — W4A16 means exactly what it says, and the speedup comes from moving 4× fewer weight bytes per decode step in a regime that is memory-bandwidth-bound, not from cheaper arithmetic. Integer-native arithmetic would require quantizing the activations too (W8A8-INT8, or the newer W4A8/FP8 schemes llm-compressor also emits), which is a different recipe with a different accuracy budget.

AWQ via AutoAWQ¶

One caveat before the code: AutoAWQ is the original standalone AWQ implementation, and it was retired by its author in 2025 — AWQ production has moved into llm-compressor’s AWQModifier (and GPTQModel), which write the same compressed-tensors/awq checkpoints vLLM already loads (Ch. 4.7). The recipe below is still the clearest statement of AWQ’s interface — a quant config, one quantize() call, one save_quantized() — and pinned releases still run, but for new work prefer the AWQModifier path through the same oneshot() call the GPTQ recipe above uses, which keeps one tool and one format for both methods.

"""
pip install "autoawq==0.2.7"   # illustrative pin of an archived project; see the
                               # note above before adopting this for new work
"""
from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer
from datasets import load_dataset

MODEL_DIR = "ckpts/stack-100m-grpo"
model = AutoAWQForCausalLM.from_pretrained(MODEL_DIR)
tokenizer = AutoTokenizer.from_pretrained(MODEL_DIR)

quant_config = {
    "zero_point": True,
    "q_group_size": 64,   # matches Ch. 14.11's group_size=64, for an apples-to-
                           # apples comparison in the sweep below
    "w_bit": 4,
    "version": "GEMM",
}
# AWQ's search needs real activations too, and `calib_data` DEFAULTS to a
# `pileval` slice AutoAWQ downloads for you -- 128 samples of generic web text
# truncated to 512 tokens, i.e. exactly the out-of-distribution calibration the
# GPTQ recipe above takes pains to avoid, plus a silent network dependency.
# Hand it the SAME held-out slice, at the same length, or the two checkpoints
# you are about to A/B differ in their calibration set as much as in their
# method.
calib_texts = load_dataset(
    "json", data_files="data/holdout_calib.jsonl", split="train"
).select(range(512))["text"]

# Runs the same salient-channel grid search as Ch. 14.11's
# awq_search_channel_scales() -- find a per-channel scale s that minimizes
# post-quantization OUTPUT error, then fold s into the preceding RMSNorm --
# batched across every linear layer in the model, with the joint-scale
# constraint Ch. 14.11 flagged (layers reading the SAME normalized hidden
# state, like wq/wk/wv, must share one s).
model.quantize(
    tokenizer,
    quant_config=quant_config,
    calib_data=calib_texts,   # a list[str]; AutoAWQ also accepts a Dataset or a
                               # Hub dataset name (check your pin's signature)
    max_calib_samples=512,
    max_calib_seq_len=2048,   # match the GPTQ recipe's max_seq_length=2048;
                               # the default here is 512
)
model.save_quantized("ckpts/stack-100m-awq-w4g64")
tokenizer.save_pretrained("ckpts/stack-100m-awq-w4g64")

vllm serve ckpts/stack-100m-awq-w4g64 --served-model-name stack100m-awq

4. GGUF and llama.cpp: CPU and Laptop Serving¶

vLLM assumes a GPU. The other half of Ch. 14.11’s payoff — generating text on the CPU of the laptop you’re reading this on, with zero cloud dependency — has its own standard 2026 tool: llama.cpp (Georgi Gerganov and contributors), a C/C++ inference engine with no PyTorch dependency at runtime, and its companion checkpoint format GGUF. This is the production version of the exact demo Ch. 14.11 closed with, and it maps directly onto Ch. 14.11’s laptop path the way vLLM maps onto Ch. 14.11’s server-side generation loop.

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
pip install -r requirements.txt   # deps for the Python converter scripts

# Convert the HF checkpoint (Ch. 15.5's Llama-shaped export) to GGUF at f16.
# The converter script's NAME has moved across llama.cpp's history --
# convert.py -> convert-hf-to-gguf.py -> convert_hf_to_gguf.py -- run
# `ls convert*` in your checkout and match whatever is actually there.
python convert_hf_to_gguf.py ../ckpts/stack-100m-grpo \
    --outfile stack100m-f16.gguf \
    --outtype f16

# Build (the current CMake path; the older Makefile build is deprecated).
cmake -B build && cmake --build build --config Release -j

# Quantize the f16 GGUF down to a K-quant. The Q4_K block layout packs 256
# weights per super-block as eight 32-weight sub-blocks with 6-bit scales AND
# 6-bit mins -- exactly 4.5 bits/weight, the scheme Ch. 14.11's memory-budget
# example gestured at. Q4_K_M is the standard "good default": Q4_K almost
# everywhere, with the sensitive tensors promoted to Q6_K, so the file averages
# a few tenths of a bit MORE than 4.5 (worked example below).
./build/bin/llama-quantize stack100m-f16.gguf stack100m-Q4_K_M.gguf Q4_K_M

# An OpenAI-compatible endpoint, CPU-only, on your own machine.
./build/bin/llama-server -m stack100m-Q4_K_M.gguf --port 8080 -c 2048

# Or drive it directly, no server process:
./build/bin/llama-cli -m stack100m-Q4_K_M.gguf \
    -p "Compute 37 + 8. Give the final integer after '####'." -n 64

For embedding in a Python application rather than shelling out, llama-cpp-python wraps the same engine:

"""pip install llama-cpp-python"""
from llama_cpp import Llama

llm = Llama(model_path="stack100m-Q4_K_M.gguf", n_ctx=2048)
out = llm(
    "Compute 37 + 8. Give the final integer after '####'.",
    max_tokens=64, temperature=0.0,
)
print(out["choices"][0]["text"])

5. Honest Evaluation with lm-evaluation-harness¶

Everything above produced artifacts — a served endpoint, a quantized checkpoint, a GGUF file. None of them is worth anything without the discipline Ch. 14.11 §3–5 built by hand: measure honestly, disclose what the model cannot do, and re-measure after every edit. EleutherAI’s lm-evaluation-harness (lm-eval) is the 2026 standard implementation of that discipline — the same harness behind the Hugging Face Open LLM Leaderboard and most published small-model papers, which matters because it makes your numbers comparable to someone else’s, not just internally consistent.

pip install "lm-eval==0.4.7"   # pin, and re-check task YAMLs after upgrading --
                                 # metric names occasionally change between releases

# Point it at the HF checkpoint directly.
lm_eval --model hf \
    --model_args pretrained=ckpts/stack-100m-grpo,dtype=bfloat16 \
    --tasks arc_easy,piqa \
    --device cpu \
    --batch_size 8 \
    --output_path results/stack100m_base

# Or route through the vLLM backend -- the harness's `vllm` model type wraps
# exactly the offline-batch LLM() call from Section 2, so continuous batching
# speeds up EVALUATION, not just serving. This is the same call Ch. 15.5's
# post-training gate uses, forward-referenced there as "the same engine you
# serve with, Ch. 15.6."
lm_eval --model vllm \
    --model_args pretrained=ckpts/stack-100m-grpo,dtype=bfloat16,gpu_memory_utilization=0.3 \
    --tasks arc_easy,piqa \
    --batch_size auto

Be honest, as Ch. 14.11’s capability report insists: arc_easy and piqa are real standardized tasks, but a 100M-parameter model trained on ~20B tokens will land close to chance on most broad-knowledge benchmarks in the harness’s catalog. The value of running them is not a leaderboard number — it is methodology: the same battery every paper reports, with the exact prompt formatting and scoring convention (cloze scoring for multiple choice, the same BPE-boundary discipline Ch. 14.11’s sequence_logprob implements by hand) applied consistently, so a future rerun on a better checkpoint is actually comparable, which the bespoke probes in Ch. 14.11 were explicitly not designed to be against anyone else’s model.

Porting the hand-rolled arithmetic probe into a custom task¶

Ch. 14.11 §4.1 built make_arithmetic_probe/eval_arithmetic, graded with the exact #### <int> format the RLVR stage trained. The harness supports exactly this kind of bespoke task via a YAML file — porting it is mechanical, and it is worth doing precisely because it turns a script only you can run into an artifact anyone with the harness installed can rerun, diff against a baseline, and wire into CI.

# tasks/stack_arithmetic/arithmetic_2digit.yaml
task: stack_arithmetic_2digit
dataset_path: json
dataset_kwargs:
  data_files: {test: "data/arith_probe_2digit.jsonl"}   # same seed=0, n=200 set
                                                          # Ch. 14.11 generates
test_split: test
output_type: generate_until
doc_to_text: "{{question}}"      # Ch. 14.9's exact prompt: "Compute {a} {op} {b}.
                                  # Give the final integer after '####'."
doc_to_target: "{{answer}}"
generation_kwargs:
  until: ["<|end|>"]
  max_gen_toks: 64               # matches Ch. 14.9's RLVR rollout budget --
                                  # truncate this and you cut off the '####'
                                  # line and measure your own budget, not the
                                  # model, exactly the pitfall Ch. 14.11 flags
  do_sample: false
filter_list:
  - name: "extract_answer"
    filter:
      - function: "regex"
        regex_pattern: "#### (-?\\d+)"   # the SAME parse Ch. 14.9's
                                          # exact_match_reward uses
      - function: "take_first"
metric_list:
  - metric: exact_match
    aggregation: mean
    higher_is_better: true

lm_eval --model hf \
    --model_args pretrained=ckpts/stack-100m-grpo,dtype=bfloat16 \
    --tasks stack_arithmetic_2digit \
    --include_path tasks/ \
    --limit 200 \
    --log_samples --output_path results/arith

--log_samples writes every prompt, generation, and parsed prediction to disk — inspect it, especially the first time, to confirm the harness’s regex filter is matching the same #### <int> your RLVR verifier matches. --limit caps the number of examples for a fast smoke test before a full run. --num_fewshot controls in-context examples for tasks that use them (0 for the arithmetic task above, since Ch. 14.9 trained zero-shot).

For agentic evaluation — the ReAct loop from Ch. 14.10, which lm_eval’s log-likelihood/generate-until primitives do not naturally decompose — lighteval (Hugging Face) and especially inspect_ai (UK AI Safety Institute) are the right tools, with solvers, tool sandboxes, and per-step trace logging built for exactly the format-validity/tool-choice/turns/EM decomposition Ch. 14.11 §4.4 implemented by hand. See Reasoning, Coding & Agentic Evals and Building Eval Harnesses.

6. The Full Pipeline, and Living With Version Drift¶

Stitching everything in this chapter into one script makes the shape of the production pipeline concrete: one checkpoint in, three shipped artifacts out, each independently evaluated.

#!/usr/bin/env bash
# ship_stack100m.sh -- the production pipeline for Part XV, end to end.
set -euo pipefail

CKPT=ckpts/stack-100m-grpo     # Ch. 15.5's final, aligned HF checkpoint

# 1. GPU path: quantize with AWQ, serve with vLLM.
#    NOTE the explicit budget on the backgrounded server. Without an explicit
#    --gpu-memory-utilization it defaults to 0.9 and eagerly reserves ~72GB of
#    an 80GB card, and every lm_eval below (each of which spins up its OWN
#    in-process vLLM engine on the same GPU) then fails at init with "free
#    memory is less than desired GPU memory utilization" -- which, under
#    `set -e`, kills the script before `kill $VLLM_PID` ever runs and leaves an
#    orphaned server holding the card. Budget the whole GPU explicitly
#    (0.3 server + 0.3 for one eval at a time) and clean up via trap.
python awq_quantize.py --model $CKPT --out ckpts/stack-100m-awq-w4g64
vllm serve ckpts/stack-100m-awq-w4g64 --served-model-name stack100m \
    --gpu-memory-utilization 0.3 --port 8000 &
VLLM_PID=$!
trap 'kill "$VLLM_PID" 2>/dev/null || true' EXIT

# 2. CPU/laptop path: convert to GGUF, quantize with a K-quant, serve with
#    llama.cpp -- independent of the GPU path above, same source checkpoint.
python llama.cpp/convert_hf_to_gguf.py $CKPT --outfile stack100m-f16.gguf
./llama.cpp/build/bin/llama-quantize stack100m-f16.gguf stack100m-Q4_K_M.gguf Q4_K_M

# 3. Evaluate BOTH shipped artifacts against the same battery -- fp16 baseline,
#    AWQ, and (via llama-cpp-python's HF-compatible wrapper or a small custom
#    lm_eval model class) the GGUF checkpoint. Never ship without this step.
lm_eval --model vllm \
    --model_args pretrained=$CKPT,dtype=bfloat16,gpu_memory_utilization=0.3 \
    --tasks arc_easy,piqa,stack_arithmetic_2digit --include_path tasks/ \
    --output_path results/base.json

lm_eval --model vllm \
    --model_args pretrained=ckpts/stack-100m-awq-w4g64,dtype=bfloat16,gpu_memory_utilization=0.3 \
    --tasks arc_easy,piqa,stack_arithmetic_2digit --include_path tasks/ \
    --output_path results/awq.json

# 4. Smoke-test the endpoint that will actually take production traffic --
#    the served artifact, not just the offline engine the evals used.
for _ in $(seq 60); do curl -sf localhost:8000/health >/dev/null && break; sleep 2; done
curl -sf http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{"model":"stack100m","messages":[{"role":"user","content":"Compute 37 + 8. Give the final integer after #### ."}],"temperature":0,"max_tokens":64}' \
    > results/serve_smoke.json

# The EXIT trap above stops the server, including on failure.

Notice what is not in this script: any code implementing continuous batching, GPTQ’s Hessian, K-quant packing, or cloze scoring. All of that lives in Ch. 14.11, where you built it once, by hand, to understand it — and none of it belongs in a script you run every release.

That gap is also exactly where version drift lives. Chapter 15.1 named the general discipline (pin, freeze, verify fields against your actual installed version before trusting a snippet); this chapter’s specific hazards are: vllm serve’s CLI flags and its Transformers-backend fallback flag, which move roughly on vLLM’s release cadence; llm-compressor’s oneshot() signature, which has already changed shape once since the project’s early releases; llama.cpp’s converter script name, which has been renamed twice; and lm-evaluation-harness’s task-YAML schema, where filter/metric keys occasionally shift between major versions. None of this is a reason to avoid these tools — it is a reason to pin every one of them together in one lockfile, record that lockfile next to the shipped checkpoint (the same reproducibility discipline Ch. 14.12 demands for the training side), and treat each library’s own examples/ directory, not this chapter, as the ground truth the day a flag stops working.

Further Reading¶

• Kwon, Li, Zhuang, Sheng, et al., Efficient Memory Management for Large Language Model Serving with PagedAttention (vLLM, SOSP 2023) — the memory mechanism behind vLLM’s serving.
• Yu, Jeong, Kim, Kim, Chun, Orca: A Distributed Serving System for Transformer-Based Generative Models (OSDI 2022) — the origin of iteration-level continuous batching.
• Frantar, Ashkboos, Hoefler, Alistarh, GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers (2022).
• Lin, Tang, Tang, Yang, Dang, Han, AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration (MLSys 2024 Best Paper).
• Gao, Tow, Abbasi, et al., The Language Model Evaluation Harness (EleutherAI, lm-evaluation-harness) — the harness and its task registry.
• Gerganov et al., llama.cpp / ggml — the CPU inference engine and tensor library behind GGUF.
• GGUF format specification (ggml-org/ggml) — the on-disk layout, including K-quant super-block packing.
• llm-compressor (vLLM project / Neural Magic) and compressed-tensors — the maintained quantization toolchain and on-disk format vLLM reads natively.
• AutoAWQ (Casper Hansen) — the original standalone AWQ implementation, used above; retired/archived in 2025, with AWQ production now flowing through llm-compressor’s AWQModifier (and GPTQModel), per Ch. 4.7.
• inspect_ai (UK AI Safety Institute) — the agentic-evaluation framework referenced for the ReAct-loop grading Ch. 14.11 built by hand.