9.6 Multimodal & Visual-Document Retrieval: ColPali & Late Interaction¶

Most of Part IX has quietly assumed that a “document” is text — a paragraph you can tokenize, embed, and stuff into a context window. But an enormous fraction of the world’s high-value knowledge does not live as clean text. It lives in PDFs full of multi-column layouts, financial reports dense with tables, slide decks where the meaning is in the figure, scanned contracts, scientific papers with equations, invoices, engineering diagrams, and screenshots. The moment you try to retrieve over these with the standard pipeline, you hit a wall that has nothing to do with embeddings and everything to do with parsing.

The traditional answer is an OCR-and-layout pipeline: run optical character recognition, detect tables and reading order, reconstruct a linearized text stream, chunk it, embed it, and proceed as in Chunking, Reranking & Hybrid Search. This works, but it is brittle and lossy. OCR mangles multi-column reading order, drops table structure, and silently discards the very thing that made a chart informative — its visual form. Every error compounds downstream: a misread row in a table becomes a wrong retrieval becomes a wrong answer.

This chapter is about the alternative that emerged in 2024–2025 and rapidly became a default for visually rich corpora: treat each page as an image and retrieve directly over the pixels, skipping OCR entirely. We will build up from cross-modal dense retrieval (CLIP/SigLIP), then develop the key idea — late interaction, borrowed from ColBERT and extended to vision in ColPali and ColQwen2 — and finally engineer the full system: how to index thousands of multi-vector page embeddings (PLAID, HNSW-over-patches), how to rerank, what it costs, and how to evaluate it on ViDoRe. The generation half — feeding retrieved page images to a vision-language model (VLM) — connects directly to Vision-Language Models.

Cross-Modal Dense Retrieval: CLIP and SigLIP¶

Before late interaction, the first question is simpler: can we put an image and a text query into the same vector space so that nearest-neighbor search retrieves the right image for a textual query? This is cross-modal dense retrieval, and the canonical answer is CLIP.

The Dual-Encoder Contrastive Recipe¶

CLIP (Radford et al., Learning Transferable Visual Models From Natural Language Supervision, 2021) trains two encoders — an image encoder \(f_\text{img}\) (a Vision Transformer; see Vision Transformers & Image Encoders) and a text encoder \(f_\text{txt}\) — to map their inputs into a shared \(d\)-dimensional space. Each image and each text is reduced to a single normalized vector. Given a batch of \(N\) matched (image, caption) pairs, we compute all \(N \times N\) cosine similarities and apply a symmetric contrastive loss (InfoNCE) that pulls matched pairs together and pushes mismatched pairs apart.

Let \(u_i = f_\text{img}(I_i) / \lVert f_\text{img}(I_i)\rVert\) and \(v_j = f_\text{txt}(T_j) / \lVert f_\text{txt}(T_j)\rVert\). With a learned temperature \(\tau\), the image-to-text loss is

and the full loss symmetrizes over text-to-image as well. The mechanics are exactly the contrastive learning from Embeddings & Representation Learning, just with two modalities sharing one space.

SigLIP (Zhai et al., Sigmoid Loss for Language Image Pre-Training, 2023) replaces the softmax-over-the-batch with an independent sigmoid loss per pair, treating every (image, text) pair as a binary classification:

where \(\sigma\) is the logistic sigmoid, \(t\) is a learned scale, and \(b\) a learned bias initialized negative (since most pairs are negatives). The practical win is that the sigmoid loss does not require a global softmax normalization across the batch, so it scales to very large batches without the all-gather coupling that softmax needs — and it tends to give better small-batch behavior. SigLIP’s vision tower later became the backbone of several VLMs and, crucially for us, of ColPali.

Using CLIP/SigLIP for Visual-Document Retrieval¶

import torch
import torch.nn.functional as F
from PIL import Image
from transformers import AutoModel, AutoProcessor

# Load a SigLIP checkpoint (image + text towers sharing one space).
model = AutoModel.from_pretrained("google/siglip-base-patch16-224").eval()
proc = AutoProcessor.from_pretrained("google/siglip-base-patch16-224")

@torch.no_grad()
def embed_images(paths):
    imgs = [Image.open(p).convert("RGB") for p in paths]
    inp = proc(images=imgs, return_tensors="pt")
    emb = model.get_image_features(**inp)        # [N, d]
    return F.normalize(emb, dim=-1)              # unit vectors -> cosine = dot

@torch.no_grad()
def embed_text(queries):
    inp = proc(text=queries, padding="max_length", return_tensors="pt")
    emb = model.get_text_features(**inp)         # [M, d]
    return F.normalize(emb, dim=-1)

# Index: one vector per page image.
page_paths = ["page_001.png", "page_002.png", "page_003.png"]
page_vecs  = embed_images(page_paths)            # [3, d]

q = embed_text(["What was Q3 revenue growth?"]) # [1, d]
scores = q @ page_vecs.T                         # [1, 3] cosine similarities
best = scores.argmax(dim=-1).item()
print("Best page:", page_paths[best], "score:", scores[0, best].item())

This is genuinely useful for natural-image retrieval (“find the photo of a dog on a beach”). But it is weak for dense documents, and the reason is structural, not a tuning problem. CLIP/SigLIP compress an entire page into one vector. A page of a 10-K filing contains dozens of distinct facts: a revenue figure, a footnote about currency hedging, a segment table, a risk paragraph. A single 768-dimensional vector cannot simultaneously preserve all of them in a way that a specific keyword-like query can latch onto. This is the information-bottleneck problem of single-vector (“bi-encoder”) retrieval, and it is exactly the problem that late interaction was invented to solve in the text world.

Late Interaction: From ColBERT to ColPali¶

The ColBERT Idea¶

ColBERT (Khattab & Zaharia, ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT, 2020) sits between two extremes. A cross-encoder concatenates query and document, runs full attention across both, and scores them jointly — maximally expressive but \(O(\text{queries} \times \text{docs})\) forward passes, far too slow to scan a corpus. A bi-encoder (single vector each) is fast — you precompute document vectors and do a dot product — but throws away token-level detail.

Late interaction is the middle path. Encode the query into one vector per query token \(\{q_1, \dots, q_n\}\) and the document into one vector per document token \(\{d_1, \dots, d_m\}\). Precompute and store all document token vectors offline. At query time, score with the MaxSim operator: for each query token, find its best-matching document token, then sum:

Intuitively, each query term gets to “go shopping” across the whole document for its single best evidence, and the score adds up that evidence. There is no cross-attention between query and document — the only interaction is the cheap MaxSim dot products at the very end, hence late interaction. This preserves token granularity (a rare query term can find its exact match) while keeping documents fully precomputable.

import torch

def maxsim(Q, D):
    """
    Q: [n, d] query token embeddings (L2-normalized)
    D: [m, d] document token embeddings (L2-normalized)
    Returns the ColBERT late-interaction score (a scalar).
    """
    sim = Q @ D.T                  # [n, m] all query-token x doc-token sims
    per_query = sim.max(dim=1).values   # [n] best doc token per query token
    return per_query.sum()         # sum over query tokens

# Toy: 4 query tokens, 6 doc tokens, d=8
Q = torch.nn.functional.normalize(torch.randn(4, 8), dim=-1)
D = torch.nn.functional.normalize(torch.randn(6, 8), dim=-1)
print("MaxSim score:", maxsim(Q, D).item())

The cost is storage: instead of one vector per document you store \(m\) (often 100–300 for a passage). This is the central trade-off of all multi-vector retrieval, and most of the systems engineering later in this chapter exists to make it affordable.

ColPali: Late Interaction Over Page Patches¶

ColPali (Faysse et al., ColPali: Efficient Document Retrieval with Vision Language Models, 2024) makes one conceptually clean move: replace ColBERT’s BERT-token document encoder with a vision-language model that turns a page image into a grid of patch embeddings. The query is still text; the document is now an image.

The pipeline:

1. Render each PDF page to an image (e.g., at ~150 DPI). No OCR.
2. Feed the image to a VLM’s vision encoder + projection. The original ColPali used PaliGemma (a SigLIP vision tower feeding a Gemma language model). The image becomes a sequence of patch tokens — for PaliGemma, a \(32\times 32\) grid yields 1024 patch embeddings per page.
3. Project each patch embedding down to a low dimension \(d\) (ColPali uses \(d = 128\), matching ColBERT) with a linear layer, and L2-normalize. Store these 1024 vectors as the page’s representation.
4. The text query is tokenized and run through the same model’s language tower to produce one \(d\)-vector per query token. Following ColBERT, the query is also augmented: colpali-engine’s process_queries prepends a short instruction-style prefix and appends a handful of padding tokens that are not masked out of the MaxSim. Those extra slots behave as learned query-expansion vectors — they can latch onto page evidence the literal query words never mention — and removing them typically costs a little recall.
5. Score with MaxSim, identical to ColBERT.

The magic is in step 2. Because the patch embeddings come from a VLM that was pretrained to read, each patch vector is contextualized: a patch sitting on the cell of a table that says “Revenue: USD 4.2B” carries a representation that a query token for “revenue” can match against — without anyone ever running OCR. The model learned to associate visual glyphs and layout with textual meaning during pretraining, and ColPali’s fine-tuning sharpens that for the retrieval objective.

ColQwen2 and the Family¶

ColQwen2 swaps the PaliGemma backbone for Qwen2-VL. Qwen2-VL’s vision encoder supports dynamic resolution — it does not force every page into a fixed \(448\times 448\) box but processes the native aspect ratio, producing a variable number of patch tokens. For a tall, dense page this means more patches (finer evidence) and for a sparse slide fewer (cheaper). This typically lifts retrieval quality on dense documents at the cost of variable, sometimes larger, per-page storage. The broader family keeps the same skeleton — VLM patch encoder + low-dim projection + MaxSim — and only swaps the document encoder. By 2026 the strong open defaults have moved to the Qwen2.5-VL backbone: ColQwen2.5 reaches roughly 89 nDCG@5 on the original ViDoRe benchmark, ColSmol (256M/500M) targets edge/CPU, ColNomic-embed-multimodal (also Qwen2.5-VL) leads on the harder ViDoRe-v2, and the first Qwen3-VL-based checkpoints are already pushing the frontier further. The contrast with ColBERT-text is still only in the document encoder.

Training ColPali¶

ColPali is fine-tuned with a contrastive in-batch loss on (query, positive-page) pairs, where the negatives are the other pages in the batch. The score is the MaxSim, and the loss is the same InfoNCE softmax over MaxSim scores. A common refinement adds a margin-aware or in-batch hard-negative term. The training data is the crux: synthetic and curated document-query pairs spanning tables, figures, infographics, and full pages, so the model learns the document query distribution that CLIP never saw.

import torch
import torch.nn.functional as F

def colbert_scores(Qb, Db, q_mask, d_mask):
    """
    Vectorized MaxSim for a batch (used both in training and scoring).
    Qb: [B, n, d] query token embeddings (padded)
    Db: [B, m, d] doc patch embeddings (padded)
    q_mask: [B, n] 1 for real query tokens, 0 for padding
    d_mask: [B, m] 1 for real patches
    Returns: [B, B] score matrix S[i, j] = MaxSim(query_i, doc_j)
    """
    # Pairwise sims across the cross product of the batch:
    #   [B, n, d] x [B, m, d] -> [B(query), B(doc), n, m]
    sim = torch.einsum("ind,jmd->ijnm", Qb, Db)
    # Mask out padded doc patches before the max over patches.
    sim = sim.masked_fill(~d_mask[None, :, None, :].bool(), -1e4)
    sim = sim.max(dim=-1).values                 # [B, B, n] max over patches
    # Zero out padded query tokens before summing over query tokens.
    sim = sim * q_mask[:, None, :]
    return sim.sum(dim=-1)                        # [B, B]

def colpali_loss(Qb, Db, q_mask, d_mask):
    S = colbert_scores(Qb, Db, q_mask, d_mask)   # [B, B], diagonal = positives
    labels = torch.arange(S.size(0), device=S.device)
    # Standard in-batch InfoNCE over MaxSim scores (both directions optional).
    return F.cross_entropy(S, labels)

A Worked Example: Storage and Scoring Cost¶

Indexing Multi-Vector Embeddings at Scale¶

The brute-force MaxSim above is fine for a few thousand pages but collapses at corpus scale. The whole field of efficient multi-vector retrieval is about avoiding the full \(O(\text{pages} \times \text{patches} \times \text{q-tokens})\) scan. There is also a structural reason you cannot simply hand MaxSim to an off-the-shelf ANN library: MaxSim is a Chamfer-style set similarity, not an inner product between two single vectors, and it is not a metric (it is asymmetric and violates the triangle inequality), so the geometric assumptions HNSW and IVF are built on do not hold for it. Every practical system therefore reduces MaxSim to something indexable — per-token ANN over individual patches, centroid assignments, or a fixed-dimensional encoding whose dot product approximates MaxSim — and computes true MaxSim only on a shortlist. There are two dominant approaches.

PLAID: ColBERT’s Native Engine¶

PLAID (Santhanam et al., PLAID: An Efficient Engine for Late Interaction Retrieval, 2022) is the production indexing system designed for ColBERT, and it carries over to ColPali. Its pipeline:

1. Residual compression. Cluster all patch/token vectors in the corpus with k-means into a codebook of centroids. Store each vector as (centroid_id, residual), where the residual (vector minus its centroid) is quantized to 1–2 bits per dimension. This is the dominant memory saving.
2. Centroid-based candidate generation. Each query token retrieves the nearest centroids. Any document that has patches assigned to those centroids becomes a candidate. This avoids touching documents with no plausible matching patch.
3. Centroid-pruned scoring. Approximate MaxSim using centroid similarities first to prune, then refine only promising candidates with the decompressed residuals.
4. Full re-ranking. Decompress the top candidates and compute exact MaxSim for the final ordering.

The staged funnel — cheap-and-approximate to expensive-and-exact — is the recurring pattern of all large-scale retrieval, and you saw it for single-vector ANN in Vector Databases & Approximate Nearest Neighbor Search.

HNSW Over Flattened Patch Vectors¶

A simpler, store-agnostic approach: flatten every patch vector of every page into one giant single-vector ANN index, tagging each with its page id. Build an HNSW graph (see the ANN chapter) over all \(100{,}000 \times 1024 \approx 10^8\) patch vectors. Then:

1. For each of the \(n\) query token vectors, run an ANN search to retrieve the top-\(k\) nearest patches (across all pages).
2. Collect the union of page ids those patches belong to — this is the candidate page set.
3. For each candidate page, fetch its full patch matrix and compute exact MaxSim.
4. Sort by exact score; return top pages.

This is the strategy used by libraries that build ColPali on top of generic vector databases (Qdrant’s multivector support, Vespa’s tensor/MaxSim ranking, Weaviate’s MUVERA-style approaches). The candidate-generation step is approximate (you might miss a page whose best patch did not make any query token’s top-\(k\)), but in practice recall is high because a relevant page usually has several strongly matching patches.

import numpy as np
import hnswlib

class PatchHNSWIndex:
    """Flatten all page patches into one HNSW index; rerank pages with exact MaxSim."""
    def __init__(self, dim=128):
        self.dim = dim
        self.index = hnswlib.Index(space="ip", dim=dim)  # inner product
        self.page_patches = {}   # page_id -> [m, d] float32 patch matrix
        self.label_to_page = {}  # global patch label -> page_id
        self._next = 0

    def init(self, max_patches):
        self.index.init_index(max_elements=max_patches, ef_construction=200, M=16)

    def add_page(self, page_id, patches):
        patches = patches.astype(np.float32)
        n = patches.shape[0]
        labels = np.arange(self._next, self._next + n)
        self.index.add_items(patches, labels)
        for lab in labels:
            self.label_to_page[int(lab)] = page_id
        self.page_patches[page_id] = patches
        self._next += n

    def search(self, query_patches, k_per_token=50, topn=10):
        query_patches = query_patches.astype(np.float32)
        # 1) candidate generation: nearest patches per query token -> union of pages
        candidates = set()
        for qtok in query_patches:
            labels, _ = self.index.knn_query(qtok, k=k_per_token)
            for lab in labels[0]:
                candidates.add(self.label_to_page[int(lab)])
        # 2) exact MaxSim rerank over candidate pages only
        scored = []
        for pid in candidates:
            D = self.page_patches[pid]               # [m, d]
            sim = query_patches @ D.T                # [n, m]
            scored.append((pid, sim.max(axis=1).sum()))   # MaxSim
        scored.sort(key=lambda x: -x[1])
        return scored[:topn]

In production you rarely hand-roll this. Qdrant stores a whole matrix of patch vectors per point and evaluates MaxSim natively, so the funnel above becomes a few lines of client code:

from qdrant_client import QdrantClient, models

client = QdrantClient(":memory:")                 # or url="http://localhost:6333"
client.create_collection(
    collection_name="pages",
    vectors_config=models.VectorParams(
        size=128,                                  # ColPali projection dim
        distance=models.Distance.COSINE,
        # Each point holds [m, 128] patch vectors; scoring is MaxSim, not a dot product.
        multivector_config=models.MultiVectorConfig(
            comparator=models.MultiVectorComparator.MAX_SIM),
    ),
)
client.upsert("pages", points=[
    models.PointStruct(id=i, vector=emb.tolist(), payload={"page": i})
    for i, emb in enumerate(page_embeddings)       # emb: [m, 128] per page
])
hits = client.query_points("pages", query=q_emb[0].tolist(), limit=5).points

Vespa expresses the same scoring declaratively as a tensor rank-profile (a reduce(..., max, patch) over the query-token × patch tensor product, then a sum over query tokens), and Weaviate exposes multi-vector fields with an optional MUVERA encoder for the candidate stage. The trade-off is identical everywhere: the database can compute MaxSim for you, but you still decide how the candidate stage narrows millions of pages down to hundreds.

Token Pooling and Compression¶

Because 1024 patches per page is the cost driver, a cheap win is token pooling: cluster a page’s patch vectors (e.g., hierarchical agglomerative clustering) and keep a smaller set of representative vectors — say 256 instead of 1024 — before indexing. Empirically you can drop a large fraction of patches with little retrieval-quality loss, because many patches (margins, whitespace, repeated background) are redundant. It is about ten lines:

import numpy as np
from scipy.cluster.hierarchy import linkage, fcluster
from scipy.spatial.distance import pdist

def pool_patches(D, keep=128):
    """D: [m, d] L2-normalized patch vectors -> [<=keep, d] pooled, renormalized.
    Average-link agglomerative clustering on cosine distance, cut at `keep` clusters;
    each cluster collapses to the (renormalized) mean of its members."""
    if D.shape[0] <= keep:
        return D
    Z = linkage(pdist(D, metric="cosine"), method="average")
    labels = fcluster(Z, t=keep, criterion="maxclust")     # at most `keep` clusters
    pooled = np.stack([D[labels == c].sum(0) for c in np.unique(labels)])
    return pooled / np.linalg.norm(pooled, axis=1, keepdims=True)

D = np.random.default_rng(0).standard_normal((1024, 128))
D /= np.linalg.norm(D, axis=1, keepdims=True)
print(pool_patches(D, keep=128).shape)                     # (128, 128)

MaxSim over the pooled set is a lower-resolution version of the original: a query token now matches a region of the page rather than a single patch, which is exactly why quality degrades gracefully. colpali-engine ships an equivalent hierarchical token pooler (parameterized by a pool factor) so you can apply this at index time without writing it yourself. Combined with binary quantization of what remains, real ColPali deployments routinely shrink the index by 10–30× from the naive 256 KiB/page figure. MUVERA (Multi-Vector Retrieval via Fixed Dimensional Encodings) goes further: it deterministically projects the whole multi-vector set into a single fixed-dimensional vector whose dot product approximates MaxSim, letting you reuse a standard single-vector ANN index for candidate generation and only fall back to exact MaxSim for reranking.

The OCR-Free Page-as-Image RAG Pipeline¶

Retrieval is half the system. The other half feeds the retrieved page images to a vision-language model to generate the answer — never reconstructing text at all. This is the OCR-free RAG loop, and it composes cleanly with everything in Retrieval-Augmented Generation Architectures, with images replacing text chunks.

# End-to-end sketch with the `colpali-engine` library + a VLM generator.
import torch
from colpali_engine.models import ColPali, ColPaliProcessor
from pdf2image import convert_from_path

device = "cuda" if torch.cuda.is_available() else "cpu"
model = ColPali.from_pretrained("vidore/colpali-v1.3",
                                torch_dtype=torch.bfloat16).to(device).eval()
processor = ColPaliProcessor.from_pretrained("vidore/colpali-v1.3")

# --- OFFLINE: render pages and embed them ---
# pdf2image shells out to poppler; `pypdfium2` is a pure-wheel alternative with no
# system dependency, which matters when you containerize the indexer.
pages = convert_from_path("annual_report.pdf", dpi=150)  # list[PIL.Image]
page_embeddings = []
with torch.no_grad():
    for batch_start in range(0, len(pages), 4):
        batch = pages[batch_start:batch_start + 4]
        inp = processor.process_images(batch).to(device)
        emb = model(**inp)                  # [b, num_patches, 128]
        page_embeddings.extend(list(emb.to(torch.float16).cpu()))
# (Index page_embeddings into PLAID/HNSW here; see previous section.)

# --- ONLINE: embed the query, score with MaxSim, retrieve top pages ---
query = "How did operating margin change from 2021 to 2022?"
with torch.no_grad():
    q_inp = processor.process_queries([query]).to(device)
    q_emb = model(**q_inp)                  # [1, n_tokens, 128]

# Exact MaxSim against all pages (small corpus; use the index at scale).
def maxsim(q, d):
    return (q @ d.T).max(dim=1).values.sum()
scores = torch.tensor([maxsim(q_emb[0].float(), d.float()) for d in page_embeddings])
top = scores.topk(3).indices.tolist()
retrieved_images = [pages[i] for i in top]

# --- GENERATION: hand the raw page images to a VLM (no OCR text!) ---
# Any capable VLM works here; swap in a newer Qwen2.5-VL / Qwen3-VL checkpoint
# (or serve it behind vLLM/SGLang) without touching the retrieval half.
from transformers import AutoModelForImageTextToText, AutoProcessor
vlm = AutoModelForImageTextToText.from_pretrained(
    "Qwen/Qwen2-VL-7B-Instruct", torch_dtype=torch.bfloat16).to(device).eval()
vproc = AutoProcessor.from_pretrained("Qwen/Qwen2-VL-7B-Instruct")

messages = [{"role": "user", "content":
    [{"type": "image", "image": im} for im in retrieved_images]
    + [{"type": "text", "text": query
        + " Cite the page number you used."}]}]
prompt = vproc.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
vin = vproc(text=[prompt], images=retrieved_images, return_tensors="pt").to(device)
with torch.no_grad():
    out = vlm.generate(**vin, max_new_tokens=256)
print(vproc.batch_decode(out, skip_special_tokens=True)[0])

Why “no OCR” Is a Feature, Not a Hack¶

The instinct of a careful engineer is to distrust skipping OCR — surely text is more reliable than pixels? But consider what survives the round-trip. A bar chart’s meaning is the relative height of bars; OCR captures only the axis labels and legend, discarding the comparison. A complex table’s meaning is its 2-D structure; linearized OCR text frequently scrambles which number belongs to which row/column header. The page-as-image approach keeps the layout, the chart geometry, the equation rendering, and the spatial relationships intact, and a modern VLM can read all of it. You trade OCR’s parsing errors for the VLM’s vision — and for visually rich documents that is a strongly favorable trade.

The cost is on the generation side: page images are expensive in tokens. A single high-resolution page can consume hundreds to over a thousand vision tokens in the VLM’s context, so retrieving \(k=10\) pages can blow a context budget that \(k=10\) text chunks would not. This makes precision of retrieval more important here than in text RAG — you want the top 1–3 pages to be right, because you cannot afford to dump 20 page images into the VLM. It also makes reranking valuable.

Visual Reranking and Hybrid Strategies¶

Late-interaction retrieval already does fine-grained matching, so do we still need a reranker? Often yes — for the same staged-funnel reason as text RAG (see Chunking, Reranking & Hybrid Search). Three flavors of visual reranking exist.

1. Exact-MaxSim rerank. If candidate generation was approximate (HNSW union, MUVERA, or PLAID centroid pruning), recomputing exact MaxSim on the shortlist is itself a reranker. Cheap and almost always worth it.

2. Cross-encoder / VLM-as-judge rerank. Pass each candidate page image and the query jointly into a VLM and ask for a relevance score. This is the visual analogue of a text cross-encoder: maximally expressive (full cross-attention between query and page) but expensive, so apply it only to the top ~20 candidates. A prompt like “On a scale of 0–10, does this page contain the information to answer: <query>?” turns any capable VLM into a reranker.

3. Hybrid with text/OCR signals. Nothing forbids running OCR in addition for a BM25 lexical channel (see hybrid search). For documents with exact identifiers — invoice numbers, part codes, legal citations — a lexical match on OCR’d text is unbeatable, while ColPali handles the semantic/visual layout. Fuse the rankings with Reciprocal Rank Fusion (RRF):

def reciprocal_rank_fusion(rankings, k=60):
    """rankings: list of ranked lists, each a list of page_ids best-first."""
    scores = {}
    for ranking in rankings:
        for rank, pid in enumerate(ranking):
            scores[pid] = scores.get(pid, 0.0) + 1.0 / (k + rank + 1)
    return sorted(scores, key=lambda p: -scores[p])

colpali_order = ["p17", "p4", "p23", "p9"]
bm25_order    = ["p4", "p17", "p88", "p23"]   # from OCR + BM25
print(reciprocal_rank_fusion([colpali_order, bm25_order]))

Evaluation: ViDoRe and What to Measure¶

You cannot tune what you cannot measure, and visual-document retrieval needed its own benchmark because text-retrieval datasets do not exist as page images. ViDoRe (the Visual Document Retrieval Benchmark, introduced alongside ColPali) is the standard. It contains query–page tasks spanning academic papers, figures, infographics, tables, and multi-domain documents (medical, energy, government, AI), in multiple languages, with both synthetic and human-curated queries. Crucially, each task is page-level: given a query, retrieve the correct page(s) from a corpus of page images.

The primary metric is nDCG@k (Normalized Discounted Cumulative Gain), with Recall@k and MRR as companions. nDCG rewards placing relevant pages near the top and discounts gains logarithmically by rank:

where \(\text{rel}_i\) is the relevance of the page at rank \(i\) and IDCG is the DCG of the ideal ordering (so nDCG \(\in [0, 1]\)).

import numpy as np

def dcg(relevances):
    relevances = np.asarray(relevances, dtype=float)
    discounts = np.log2(np.arange(2, relevances.size + 2))
    return np.sum((2**relevances - 1) / discounts)

def ndcg_at_k(retrieved_rels, ideal_rels, k=5):
    """retrieved_rels: relevance of each retrieved page, in retrieved order.
       ideal_rels: all true relevances sorted descending (for the ideal ranking)."""
    actual = dcg(retrieved_rels[:k])
    ideal  = dcg(sorted(ideal_rels, reverse=True)[:k])
    return actual / ideal if ideal > 0 else 0.0

# Retrieved pages had relevances [1, 0, 1, 0, 0]; one other relevant page (rel 1)
# existed but was missed, so the ideal top-5 is [1, 1, 1, 0, 0].
print(round(ndcg_at_k([1, 0, 1, 0, 0], [1, 1, 1, 0, 0], k=5), 4))

You should not re-implement the harness around that metric. The ViDoRe authors ship vidore-benchmark (a Python package plus CLI) that pulls the benchmark’s page-image datasets from the Hugging Face Hub, runs any colpali-engine retriever — or your own, by registering a class that exposes forward_queries / forward_passages and a scoring function — across every task and reports per-task nDCG@5; the ViDoRe tasks are also reachable through MTEB’s multimodal retrieval suite. Wiring your retriever into it is the same argument as using lm-evaluation-harness for text evals rather than hand-rolling one (see Building Eval Harnesses): comparability with published numbers is the whole point.

A few measurement subtleties specific to this setting:

• Report retrieval and end-to-end separately. A page can be retrieved correctly yet the VLM still answers wrong, or vice versa. Measure nDCG@k for retrieval and an answer-correctness metric (exact match, LLM-as-judge; see LLM-as-a-Judge & Automated Evaluation) for the full pipeline.
• Watch DPI and resolution. Rendering pages too low-res hurts ColPali’s patch evidence and the VLM’s reading; too high-res inflates cost. ViDoRe-style ablations across DPI catch this.
• Beware synthetic-query leakage. Synthetic queries generated by the same family of models used in retrieval can inflate scores; trust human-curated splits more. ViDoRe-v2 specifically broadened domains and tightened this. By 2026 the original ViDoRe-v1 is largely saturated (top systems ~89 nDCG@5), so ViDoRe-v2 — where the best models still sit in the low-60s nDCG@5 — is the more discriminating benchmark to report.
• Language and OCR-hardness. ColPali’s biggest wins over OCR pipelines are on non-Latin scripts, handwriting, and heavy-layout pages where OCR degrades; stratify your eval by document difficulty to see where the visual approach actually pays off.

Putting It Together: When to Reach for This¶

The decision is a corpus question, not a fashion question. Reach for ColPali-style visual-document retrieval when:

• Your corpus is PDFs, slides, scans, or screenshots with meaningful layout, tables, figures, or charts.
• OCR is lossy or failing on your documents (multi-column, non-Latin scripts, handwriting, complex tables).
• You can afford a multi-vector index (or the compression to make it affordable) and you want to skip the brittle OCR-and-layout engineering entirely.

Stay with text RAG (and OCR if needed) when the corpus is born-digital clean text, when index size is tightly constrained, or when you need exact lexical matching on identifiers as the dominant signal (though hybrid fusion lets you have both). And remember the cost asymmetry: visual retrieval shifts expense from a parsing pipeline (offline, one-time) to storage (multi-vector index) and generation tokens (page images in the VLM). For high-value, layout-heavy corpora that trade is usually worth it; for commodity text it usually is not. The capstone makes exactly this call and lands on the text side: Stack-100M has no vision tower and its research corpus is born-digital, so A Narrow Auto-Research Agent uses a single-vector text index. Late interaction is the upgrade path the moment that corpus becomes scanned PDFs — the MaxSim scorer, the candidate-then-exact funnel, and the two-tier storage split all port over unchanged; only the document encoder changes.

This chapter closes Part IX. The retrieval mechanisms here — dual encoders, late interaction, multi-vector indexing, staged reranking — are the same primitives you have seen throughout the part, recombined for pixels instead of tokens. The generation side hands off directly to Part X: see Vision-Language Models for how the VLM actually reads those retrieved pages, and Vision Transformers & Image Encoders for the patch encoders that make any of this possible.

Further Reading¶

• Faysse et al., ColPali: Efficient Document Retrieval with Vision Language Models, 2024 — the foundational paper and source of the ViDoRe benchmark.
• Khattab & Zaharia, ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT, SIGIR 2020.
• Santhanam et al., ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction, NAACL 2022.
• Santhanam et al., PLAID: An Efficient Engine for Late Interaction Retrieval, CIKM 2022.
• Radford et al., Learning Transferable Visual Models From Natural Language Supervision (CLIP), ICML 2021.
• Zhai et al., Sigmoid Loss for Language Image Pre-Training (SigLIP), ICCV 2023.
• Beyer et al., PaliGemma: A versatile 3B VLM for transfer, 2024 — the backbone of the original ColPali.
• Jayaram et al., MUVERA: Multi-Vector Retrieval via Fixed Dimensional Encodings, 2024.
• The colpali-engine and ColBERT open-source repositories for reference implementations.

Exercises¶

1. (Conceptual) A colleague proposes speeding up ColPali by mean-pooling the query’s per-token vectors into a single 128-dim vector before scoring, arguing “the pages are still multi-vector, so we keep the fine-grained matching.” Explain (a) why single-vector CLIP/SigLIP already struggles on dense document pages, and (b) why your colleague’s change silently throws away ColPali’s advantage. What retrieval regime does the modified system collapse to?

2. (Quantitative) A query has \(n=3\) tokens and a page has \(m=4\) patches. All vectors are L2-normalized, and the resulting query-token by patch dot-product matrix (rows = query tokens, columns = patches) is

q1: [0.2, 0.9, 0.1, 0.4]
q2: [0.7, 0.3, 0.5, 0.2]
q3: [0.1, 0.2, 0.8, 0.6]

Compute the MaxSim score. Then compute the score you would get if you (wrongly) mean-pooled the three query rows into one row before taking a single max over patches, and comment on the difference.

3. (Quantitative) You index 50,000 pages with ColQwen2, which (dynamic resolution) produces on average 768 patch vectors per page, each \(d=128\), stored at fp16 (2 bytes/element). (a) What is the raw index size? (b) You apply token pooling to reduce each page to 128 representative patches. New size? © You then binary-quantize the pooled vectors to 1 bit/dimension. Final size, and the overall reduction factor from (a)?

4. (Quantitative) On a ViDoRe-style task, your system retrieves 3 pages with binary relevances (in retrieved order) \([1, 0, 1]\). The corpus contains exactly 2 relevant pages for this query. Compute nDCG@3 using the chapter’s definition. Show DCG and IDCG.

5. (Implementation) The chapter notes binary quantization (1 bit/dim) is “often the single highest-leverage optimization.” Implement it. Write binarize(V) that sign-quantizes and bit-packs a patch matrix, and approx_maxsim(q_bits, d_bits, dim) that computes an approximate MaxSim using XOR + popcount (Hamming distance) instead of float dot products. Verify on random unit vectors that the approximate ranking tracks the exact MaxSim.

import numpy as np

def binarize(V):
    """V: [m, d] float -> packed sign bits [m, ceil(d/8)] uint8 (1 = nonneg)."""
    signs = (V >= 0).astype(np.uint8)      # 1 bit per dimension
    return np.packbits(signs, axis=-1)

def approx_maxsim(q_bits, d_bits, dim):
    """q_bits: [n, bytes], d_bits: [m, bytes]. Approx MaxSim via Hamming."""
    xor = np.bitwise_xor(q_bits[:, None, :], d_bits[None, :, :])  # [n, m, bytes]
    # Cast to signed: unpackbits.sum returns uint64, and dim - 2*ham goes
    # negative when >half the bits differ, which would underflow uint64.
    ham = np.unpackbits(xor, axis=-1).sum(-1).astype(np.int64)   # [n, m]
    sim = dim - 2 * ham                     # [n, m] surrogate for cos * dim
    return sim.max(axis=1).sum()            # MaxSim: best patch per q-token

def exact_maxsim(Q, D):
    return float((Q @ D.T).max(axis=1).sum())

# --- sanity check: approximate ranking should track exact ranking ---
rng = np.random.default_rng(0)
dim = 128
def unit(m): 
    X = rng.standard_normal((m, dim)); return X / np.linalg.norm(X, axis=1, keepdims=True)

Q = unit(8)                                  # 8 query tokens
pages = [unit(rng.integers(40, 80)) for _ in range(20)]

exact = np.array([exact_maxsim(Q, D) for D in pages])
qb = binarize(Q)
approx = np.array([approx_maxsim(qb, binarize(D), dim) for D in pages])

# Compare orderings (top pages should mostly agree).
print("exact top5 :", np.argsort(-exact)[:5])
print("approx top5:", np.argsort(-approx)[:5])

6. (Implementation) Reproduce the chapter’s HNSW-over-patches funnel without an ANN library, so the candidate-generation logic is explicit. Given a query’s patch matrix and a dict of page_id -> [m, d] patch matrices, implement (1) candidate generation as the union of pages owning each query token’s top-\(k\) nearest patches across the whole corpus, then (2) exact-MaxSim reranking over only those candidate pages. Return the top-\(n\) pages.

import numpy as np

def maxsim(Q, D):
    return float((Q @ D.T).max(axis=1).sum())

def candidate_pages(q_patches, pages, k_per_token=5):
    # Flatten all patches, remembering which page owns each.
    mats, owner = [], []
    for pid, D in pages.items():
        mats.append(D)
        owner.extend([pid] * D.shape[0])
    A = np.vstack(mats).astype(np.float32)     # [total_patches, d]
    owner = np.asarray(owner)
    cands = set()
    for q in q_patches:                        # one ANN probe per query token
        sims = A @ q                           # [total_patches]
        top = np.argpartition(-sims, k_per_token)[:k_per_token]
        cands.update(owner[top].tolist())      # union of owning pages
    return cands

def search(q_patches, pages, k_per_token=5, topn=3):
    q_patches = q_patches.astype(np.float32)
    cands = candidate_pages(q_patches, pages, k_per_token)      # stage 1: approx
    scored = [(pid, maxsim(q_patches, pages[pid])) for pid in cands]  # stage 2: exact
    scored.sort(key=lambda x: -x[1])
    return scored[:topn]

# --- demo ---
rng = np.random.default_rng(1)
d = 128
def unit(m):
    X = rng.standard_normal((m, d)); return (X / np.linalg.norm(X, axis=1, keepdims=True)).astype(np.float32)
pages = {f"p{i}": unit(int(rng.integers(60, 100))) for i in range(50)}
Q = unit(10)
print(search(Q, pages, k_per_token=5, topn=3))