9.3 Retrieval-Augmented Generation Architectures¶

A language model trained six months ago does not know about last week’s earnings call. A 7-billion-parameter model that has memorized Wikipedia cannot tell you what is in your company’s internal runbook. Even a frontier model with a million-token context window will confidently confabulate a citation if the relevant fact falls outside its training distribution. Retrieval-Augmented Generation (RAG) addresses all three problems with a single architectural shift: rather than forcing the model to answer from parametric memory alone, we first retrieve the most relevant documents from an external store and inject them into the context before the model generates its answer.

This chapter dissects the full RAG pipeline — from corpus ingestion to final generation — explains why each design decision matters, catalogs the most important failure modes, and shows you how to build a minimal but production-faithful implementation from scratch. We also cover how to measure whether your RAG system is actually working with the RAGAS framework of faithfulness, answer relevance, and context precision.

Related chapters: Embeddings & Representation Learning covers how dense encoders produce the vectors we retrieve against. Vector Databases & Approximate Nearest Neighbor Search covers the indexing and search algorithms in depth. Chunking, Reranking & Hybrid Search extends the ideas here with more sophisticated retrieval pipelines. Advanced RAG: GraphRAG, Agentic RAG & Long-Context vs RAG covers the frontier.

Why RAG Exists: Three Hard Problems¶

The Parametric Memory Wall¶

A transformer’s weights are a lossy, distributed compression of the training corpus. Factual knowledge is stored implicitly across billions of parameters, which means:

1. Temporal staleness. Pre-training is expensive and infrequent. A model trained on data through mid-2024 cannot answer questions about events after that date.
2. Capacity limits. Even a 70 B-parameter model has finite capacity. Long-tail facts, private documents, and domain-specific knowledge may never be reliably memorized.
3. No attribution. When a model answers from parametric memory, there is no provenance trail — no page number, no document, no timestamp. For enterprise or legal contexts, this is often a blocker.

What RAG Changes¶

RAG augments each inference call with a retrieval step. The model’s parametric knowledge becomes a reasoning engine rather than a fact store:

where \(q\) is the user query and \(\mathcal{D}_{q} = \{d_1, d_2, \ldots, d_k\}\) is the set of \(k\) documents retrieved for that specific query. The generation is conditioned on fresh, verifiable context. Updating the knowledge base requires only re-indexing documents, not re-training the model.

This trades off context-window tokens for factual accuracy, freshness, and citability — a trade that is almost always worth making for knowledge-intensive tasks.

Frozen Concatenation vs. Marginalization¶

Writing \(P(y \mid q, \mathcal{D}_q)\) hides a modelling choice. The original RAG paper (Lewis et al., 2020) treats the retrieved document \(z\) as a latent variable and marginalizes over the top-\(k\) retrieved documents, in one of two ways. RAG-Sequence commits to a single document for the whole answer and marginalizes over sequences:

while RAG-Token lets every generated token draw on a different document:

Because \(p_\eta(z \mid q)\) is differentiable through the query encoder, this objective trains the retriever and the generator jointly with only answer supervision — the document encoder and index stay frozen, since re-embedding the corpus every few steps is prohibitive. Two descendants matter: Fusion-in-Decoder (Izacard & Grave, 2021) encodes each of the \(k\) passages separately with the query and lets the decoder cross-attend over the concatenated encoder states, which scales to far more passages than stuffing them into one encoder; REPLUG (Shi et al., 2023) keeps the LM a black box and ensembles its output distributions weighted by retrieval scores, training only the retriever against LM likelihood.

Essentially all production RAG in 2026 uses a third, degenerate variant: frozen concatenation — retrieve top-\(k\), paste the passages into the prompt, run a single forward pass, no marginalization and no retriever training. It is \(k\times\) cheaper than RAG-Sequence (one generation pass instead of \(k\)), works with any API-served model, and with modern instruction-tuned generators it is usually good enough. Knowing the marginalized form still matters: it tells you exactly what you gave up — a calibrated \(p_\eta(z \mid q)\) weighting, and gradient signal that would otherwise teach the retriever what the generator actually finds useful. RAFT (below) recovers part of that signal on the generator side by fine-tuning on retrieved contexts with distractors.

The RAG Pipeline: Five Stages¶

<g class="mono">

  <rect x="338" y="92"  width="124" height="26" rx="5" class="v-accent" fill="var(--accent)" fill-opacity="0.18" stroke="var(--accent)" stroke-width="1.4"/>
  <text x="350" y="109" class="mono" fill="var(--accent)">c12</text>
  <text x="452" y="109" text-anchor="end" class="mono" fill="var(--accent)">0.83</text>

  <rect x="338" y="122" width="124" height="26" rx="5" class="v-accent" fill="var(--accent)" fill-opacity="0.18" stroke="var(--accent)" stroke-width="1.4"/>
  <text x="350" y="139" class="mono" fill="var(--accent)">c07</text>
  <text x="452" y="139" text-anchor="end" class="mono" fill="var(--accent)">0.79</text>

  <rect x="338" y="152" width="124" height="26" rx="5" class="v-accent" fill="var(--accent)" fill-opacity="0.18" stroke="var(--accent)" stroke-width="1.4"/>
  <text x="350" y="169" class="mono" fill="var(--accent)">c31</text>
  <text x="452" y="169" text-anchor="end" class="mono" fill="var(--accent)">0.74</text>


  <rect x="338" y="186" width="124" height="22" rx="5" class="v-fill v-grid" stroke-width="1"/>
  <text x="350" y="201" class="mono" fill="var(--muted)">c45</text>
  <text x="452" y="201" text-anchor="end" class="mono" fill="var(--muted)">0.41</text>

  <rect x="338" y="212" width="124" height="22" rx="5" class="v-fill v-grid" stroke-width="1"/>
  <text x="350" y="227" class="mono" fill="var(--muted)">c02</text>
  <text x="452" y="227" text-anchor="end" class="mono" fill="var(--muted)">0.38</text>
</g>

<text x="400" y="256" text-anchor="middle" class="sub">… N indexed chunk vectors …</text>


<rect x="356" y="272" width="88" height="22" rx="11" fill="var(--accent)" fill-opacity="0.14" stroke="var(--accent)" stroke-width="1.2"/>
<text x="400" y="287" text-anchor="middle" class="mono" fill="var(--accent)">k = 3</text>

<text x="400" y="316" text-anchor="middle" class="sub">offline: chunk -&gt; embed -&gt;</text>
<text x="400" y="330" text-anchor="middle" class="sub">index (HNSW / IVF)</text>

<rect x="528" y="76" width="242" height="20" rx="4" class="v-fill v-grid" stroke-width="1"/>
<text x="538" y="90" class="mono">system: answer from context only</text>

<rect x="528" y="100" width="242" height="60" rx="4" fill="var(--accent)" fill-opacity="0.12" stroke="var(--accent)" stroke-width="1.2"/>
<text x="538" y="115" class="mono" fill="var(--accent)">context:</text>
<text x="544" y="130" class="mono" fill="var(--accent)">[1] c12  [2] c07  [3] c31</text>
<text x="538" y="148" class="sub">retrieved chunks, numbered</text>

<rect x="528" y="164" width="242" height="20" rx="4" class="v-fill v-grid" stroke-width="1"/>
<text x="538" y="178" class="mono">user: {query}</text>

<text x="649" y="206" text-anchor="middle" class="sub">grounding + numbered passages</text>

The full pipeline from raw documents to final answer has five conceptually distinct stages. Each is a design space in its own right.

Stage 1 — Chunking¶

A raw document (PDF, HTML, code file, transcript) must be split into chunks that fit inside a retrieval unit. Chunks that are too large decrease retrieval precision because a 2,000-token chunk retrieved for one sentence drags in irrelevant text. Chunks that are too small lose local context — a sentence about “the acquisition” means nothing without the surrounding paragraph.

Common strategies:

Chapter Chunking, Reranking & Hybrid Search covers the design space in full detail.

Stage 2 — Embedding¶

Each chunk is encoded into a dense vector \(\mathbf{v} \in \mathbb{R}^d\) by a bi-encoder (also called a dual-encoder). The same encoder maps the query to \(\mathbf{q} \in \mathbb{R}^d\). Retrieval is then a nearest-neighbor search in this space. See Embeddings & Representation Learning for the full treatment of encoder architectures and training.

Popular open-source choices include models from the sentence-transformers family, e5-large-v2, and bge-m3; as of 2026 the Qwen3-Embedding family (0.6B/4B/8B, Apache-2.0, released mid-2025) tops the multilingual MTEB leaderboard and offers Matryoshka-style truncatable dimensions. Typical dimensionality \(d\) ranges from 384 to 4096.

Stage 3 — Indexing¶

The chunk vectors are stored in a vector database (FAISS, Pinecone, Weaviate, Qdrant, pgvector, etc.) with an Approximate Nearest Neighbor (ANN) index such as HNSW or IVF. See Vector Databases & Approximate Nearest Neighbor Search for internals.

At query time, the query vector is compared against all indexed vectors, and the top-\(k\) most similar chunks (by cosine similarity or dot product) are returned in sub-millisecond to low-millisecond time even for corpora of tens of millions of chunks.

Stage 4 — Retrieve (and optionally Rerank)¶

The ANN index returns approximate top-\(k\) results, typically \(k = 5\)–\(20\). A cross-encoder reranker (e.g., bge-reranker-v2-m3 or the 2025 Qwen3-Reranker series) then scores the query alongside each candidate chunk jointly, producing a more accurate relevance ranking. The top-\(k'\) (typically \(k' = 3\)–\(5\)) chunks after reranking form the retrieved context.

Stage 5 — Generate¶

The retrieved chunks are concatenated into a context block, formatted with a prompt template, and sent to the LLM. The model generates an answer grounded in the retrieved evidence.

System: You are a helpful assistant. Answer based on the context below.
        If the answer is not in the context, say "I don't know."

Context:
[Chunk 1 text]
---
[Chunk 2 text]
---
[Chunk 3 text]

User: {query}

A Minimal RAG Implementation¶

The following implementation is self-contained and runnable. It uses sentence-transformers for embedding, faiss-cpu for indexing, and the openai client for generation. Every design decision is annotated.

"""
Minimal RAG from scratch.
Dependencies: sentence-transformers, faiss-cpu, openai
  pip install sentence-transformers faiss-cpu openai
"""

import re
import textwrap
from typing import List, Tuple

import faiss
import numpy as np
from openai import OpenAI
from sentence_transformers import SentenceTransformer

# ─────────────────────────────────────────────
# 1. Corpus (replace with your real documents)
# ─────────────────────────────────────────────
CORPUS = [
    """FlashAttention is an IO-aware exact attention algorithm. It tiles the
    Q, K, V matrices into blocks that fit in SRAM and computes attention
    without materializing the full N×N attention matrix, reducing memory
    from O(N²) to O(N). Published by Dao et al. in 2022.""",

    """RLHF (Reinforcement Learning from Human Feedback) is a post-training
    technique that fine-tunes a language model to align with human
    preferences. It requires a reward model trained on comparison data and
    uses PPO to optimize the policy against that reward signal.""",

    """RAG (Retrieval-Augmented Generation) was introduced by Lewis et al.
    in 2020. It combines a dense retriever (DPR) with a seq2seq generator
    (BART), marginalizing over the top-k retrieved documents. The query
    encoder and generator are trained jointly with answer supervision while
    the document encoder and index stay frozen.""",

    """The Chinchilla scaling law (Hoffmann et al., 2022) showed that for a
    given compute budget, training tokens should scale roughly 1:1 with
    model parameters. A 70 B model is optimally trained on about 1.4 T
    tokens, not the ~300 B tokens used for the original GPT-3 scale models.""",

    """FAISS (Facebook AI Similarity Search) is a library for efficient
    similarity search. Its IVF index (Inverted File Index) partitions
    vectors into Voronoi cells using k-means clustering. At query time,
    only the closest nprobe cells are searched, trading recall for speed.""",
]


# ─────────────────────────────────────────────
# 2. Chunking (trivial for this demo — each doc
#    is already one chunk; in practice you would
#    split long docs into overlapping windows)
# ─────────────────────────────────────────────
def chunk_text(text: str, max_tokens: int = 200, overlap: int = 20) -> List[str]:
    """
    Simple word-level sliding window chunker.
    For production use a sentence boundary splitter or spacy.
    """
    words = text.split()
    chunks, start = [], 0
    while start < len(words):
        end = min(start + max_tokens, len(words))
        chunks.append(" ".join(words[start:end]))
        if end == len(words):
            break
        start = end - overlap  # overlap keeps context across boundaries
    return chunks


# Flatten corpus into chunks
all_chunks: List[str] = []
for doc in CORPUS:
    all_chunks.extend(chunk_text(doc))

print(f"Total chunks: {len(all_chunks)}")


# ─────────────────────────────────────────────
# 3. Embed all chunks (bi-encoder)
# ─────────────────────────────────────────────
EMBED_MODEL = "sentence-transformers/all-MiniLM-L6-v2"
encoder = SentenceTransformer(EMBED_MODEL)

# encode() returns (n_chunks, d) float32 numpy array
chunk_embeddings = encoder.encode(
    all_chunks,
    batch_size=64,
    show_progress_bar=True,
    normalize_embeddings=True,   # unit-norm so dot product == cosine similarity
)
d = chunk_embeddings.shape[1]   # embedding dimensionality (384 for MiniLM)
print(f"Embedding dim: {d}, corpus size: {chunk_embeddings.shape[0]}")


# ─────────────────────────────────────────────
# 4. Build FAISS index
# ─────────────────────────────────────────────
# IndexFlatIP = exact inner product (cosine on unit vectors) — fine for small corpora.
# For production use IndexIVFPQ for speed and memory efficiency at scale.
index = faiss.IndexFlatIP(d)
index.add(chunk_embeddings.astype(np.float32))
print(f"FAISS index has {index.ntotal} vectors")


# ─────────────────────────────────────────────
# 5. Retrieval function
# ─────────────────────────────────────────────
def retrieve(query: str, k: int = 3) -> List[Tuple[str, float]]:
    """
    Encode the query and return the top-k (chunk, score) pairs.
    Score is cosine similarity (inner product of unit vectors), so it lies in
    [-1, 1] — in practice text embeddings almost always score in ~[0.0, 0.95].
    """
    q_vec = encoder.encode(
        [query],
        normalize_embeddings=True,
    ).astype(np.float32)

    scores, indices = index.search(q_vec, k)
    # scores shape: (1, k), indices shape: (1, k)
    results = []
    for score, idx in zip(scores[0], indices[0]):
        if idx >= 0:  # FAISS returns -1 for missing results
            results.append((all_chunks[idx], float(score)))
    return results


# ─────────────────────────────────────────────
# 6. Generation with injected context
# ─────────────────────────────────────────────
SYSTEM_PROMPT = textwrap.dedent("""\
    You are a precise technical assistant. Answer the user's question using
    ONLY the provided context passages. If the answer cannot be found in the
    context, respond with "I don't know based on the provided context."
    Cite the relevant passage(s) inline as [1], [2], etc.
""")


def build_context_block(retrieved: List[Tuple[str, float]]) -> str:
    lines = []
    for i, (chunk, score) in enumerate(retrieved, 1):
        lines.append(f"[{i}] (relevance={score:.3f})\n{chunk}")
    return "\n\n".join(lines)


def rag_query(query: str, k: int = 3) -> str:
    # Step A: retrieve
    retrieved = retrieve(query, k=k)
    context_block = build_context_block(retrieved)

    # Step B: generate.
    # Any OpenAI-compatible endpoint works, so nothing here is tied to a vendor.
    # To run the whole pipeline locally, serve an open-weights model yourself
    # with vLLM or SGLang and repoint the client:
    #   $ vllm serve Qwen/Qwen3-4B-Instruct-2507 --port 8000
    #   client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
    client = OpenAI()  # reads OPENAI_API_KEY from env
    messages = [
        {"role": "system", "content": SYSTEM_PROMPT},
        {
            "role": "user",
            "content": (
                f"Context:\n{context_block}\n\n"
                f"Question: {query}"
            ),
        },
    ]
    response = client.chat.completions.create(
        model="gpt-4o-mini",   # or the local model id you served above
        messages=messages,
        temperature=0.0,   # deterministic for RAG — no creativity needed
        max_tokens=512,
    )
    return response.choices[0].message.content


# ─────────────────────────────────────────────
# 7. Demo
# ─────────────────────────────────────────────
if __name__ == "__main__":
    query = "How does FlashAttention reduce memory usage?"
    print(f"\nQ: {query}")
    answer = rag_query(query)
    print(f"A: {answer}")

The Same Pipeline in a Framework¶

You wrote the 60 lines above so you know what every stage does; in production most teams let a framework own the plumbing. LlamaIndex is the most direct fit for indexing-heavy work (parent-child, summary and knowledge-graph indexes, agentic retrievers), LangChain covers loaders/splitters/retrievers inside a broader agent-orchestration platform, and Haystack (deepset) offers an explicit component-graph pipeline that is pleasant to unit-test. The identical five stages collapse to:

# pip install llama-index llama-index-embeddings-huggingface
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader, Settings
from llama_index.core.node_parser import SentenceSplitter
from llama_index.embeddings.huggingface import HuggingFaceEmbedding

Settings.embed_model = HuggingFaceEmbedding("BAAI/bge-small-en-v1.5")  # stage 2
Settings.node_parser = SentenceSplitter(chunk_size=512, chunk_overlap=64)  # stage 1

docs = SimpleDirectoryReader("./corpus").load_data()
index = VectorStoreIndex.from_documents(docs)            # stage 3 (in-memory ANN)
engine = index.as_query_engine(similarity_top_k=5)       # stages 4 + 5
print(engine.query("How does FlashAttention reduce memory usage?"))

The trade is the usual one: you get loaders for 100+ file types, vector-store adapters, and streaming/citation machinery for free, at the cost of a large dependency tree and abstractions that make it harder to see which prompt actually reached the model. A reasonable default is to prototype with a framework, then, once the design is settled, inline the two or three components you actually use. Whichever you choose, keep the retriever behind a narrow search(query, k) -> list[(text, score)] interface so you can swap FAISS for Qdrant, or rank_bm25 for OpenSearch, without touching the generator.

That narrow interface is exactly how the capstone wires retrieval into a 100M model: A Narrow Auto-Research Agent: ReAct, Tool-Use & Retrieval by Distillation exposes BM25 plus a dense encoder as a single search tool over a few hundred passages, and Evaluation & Serving scores the resulting retrieval-QA probe. Two lessons transfer from that scale: at 100M parameters the embedding model may legitimately be larger than the generator (grounding is the load-bearing component, so spend there), and the corpus must be held out of the pretraining mix or the “open-book” evaluation silently becomes a memorization test.

Naive RAG and Its Failure Modes¶

“Naive RAG” refers to the minimal pipeline: fixed-size chunks, single-pass dense retrieval, no reranking, concatenate-and-prompt generation. Practitioners who deploy this baseline quickly run into a characteristic set of failure modes.

Retrieval Failures¶

Semantic mismatch between query and chunk vocabulary. Dense embeddings capture meaning, not keywords. A query “What is the interest rate hike in Q3?” may miss a chunk that uses the phrase “Federal Reserve raised the federal funds rate by 75 bps in the third quarter” because the two sentences are in different parts of the semantic space if the encoder was not fine-tuned on financial text.

Chunking boundary problems. A chunk that begins mid-sentence (“…increased revenue, driven by strong performance in the cloud segment”) provides poor context to the retriever and the reader. The embedding may not capture the key entity because it appears in the previous chunk.

Top-k may not contain the answer at all. For long-tail or multi-hop questions (“Who acquired the company that made the chip used in the iPhone 6?”), a single retrieval step may not find all necessary evidence. The answer requires stitching together two or more documents.

Stale index. If documents are updated but the index is not rebuilt, retrieved chunks may contain outdated information.

Context Utilization Failures¶

Lost-in-the-middle. Liu et al. (2023) documented that LLMs are significantly worse at using information in the middle of a long context compared to information at the beginning or end. If you inject \(k = 10\) chunks, the ones in positions 4–6 are likely to be ignored.

Context-answer inconsistency (hallucination despite context). The model may generate information that is not in the retrieved chunks, especially when the retrieved context is ambiguous or partially relevant.

Prompt over-crowding. Too many retrieved chunks consume the context budget, leaving insufficient room for chain-of-thought reasoning or multi-turn history. For agents, this competes with tool call outputs and conversation history. See Context Engineering & Management for strategies.

Ranking and Diversity Failures¶

Semantic redundancy. ANN retrieval can return five near-duplicate chunks from the same section of a document. Each chunk has a high similarity score to the query, but they provide no additional information. A Maximal Marginal Relevance (MMR) or diversity filter is needed.

No cross-encoder reranking. Bi-encoders embed query and document independently, so they cannot capture fine-grained interaction. A query “disadvantages of batch normalization” may retrieve the highest-scoring chunk being a general overview of batch norm that only tangentially mentions its disadvantages.

RAG Evaluation: RAGAS and the Three Metrics¶

Measuring whether your RAG system is working requires disaggregating the pipeline into retrieval quality and generation quality. The RAGAS framework (Es et al., 2023) defines three core metrics that cover both concerns, all computable without human labels using an LLM-as-judge.

Before reaching for a judge, measure the retriever on its own with classical information-retrieval metrics — Recall@\(k\), MRR, and nDCG@\(k\) — on a small labelled query set; they are cheap, deterministic, and tell you whether the ceiling is set by retrieval or by generation. The mteb package (which subsumes the BEIR retrieval suite) gives you standardized zero-shot retrieval tasks and is the right tool for choosing an encoder; see Embeddings & Representation Learning for MTEB/BEIR and Chunking, Reranking & Hybrid Search for stage-by-stage Recall@\(k\)/MRR measurement. RAGAS is what you add on top once retrieval is decent and the open question is whether the generator is using the context. Alternative open-source RAG-eval libraries with similar judge-based metric sets include deepeval and TruLens; whichever you pick, validate the judge against a few hundred human labels before trusting its absolute numbers (LLM-as-a-Judge & Automated Evaluation).

Faithfulness¶

Definition. Given the retrieved context \(\mathcal{D}_{q}\) and the generated answer \(y\), faithfulness measures whether every claim in \(y\) is entailed by \(\mathcal{D}_{q}\). It is computed as:

A faithfulness score of 1.0 means the answer is fully grounded. A score below 0.8 is a strong signal of hallucination.

Implementation. An LLM (the judge) first decomposes \(y\) into a list of atomic claims, then classifies each claim as supported or unsupported by the context passages.

Answer Relevance¶

Definition. Answer relevance measures whether the generated answer is relevant to the query, independent of whether it is faithful. A system could achieve high faithfulness (all claims supported by context) but low answer relevance (the context and answer are about a different aspect of the query).

Formally, the judge generates \(n\) hypothetical questions \(q_1, \ldots, q_n\) that the answer \(y\) appears to address, then measures:

where \(\mathbf{e}_{q}\) is the embedding of the original query. A high score means the hypothetical questions regenerated from the answer closely resemble the original query.

Context Precision and Recall¶

Two retrieval-quality metrics round out the picture:

• Context Precision. Of the \(k\) retrieved chunks, how many are actually useful for answering the question? Precision penalizes noisy retrieval.
• Context Recall. Given ground-truth relevant passages, how many were retrieved? Recall penalizes missed evidence.

In the LLM-as-judge variant (no ground truth), the judge is asked: “Is this chunk necessary to produce the correct answer?” for each retrieved chunk.

RAGAS in Practice¶

"""
Minimal RAGAS-style evaluation loop.
Requires: pip install ragas openai

API note: ragas 0.2 renamed the dataset columns. Modern versions use
  user_input / retrieved_contexts / response / reference
whereas 0.1.x examples you will find online use
  question / contexts / answer / ground_truth
with a plain `datasets.Dataset`. Check the version you installed.
"""

from ragas import evaluate, EvaluationDataset
from ragas.metrics import (
    faithfulness,        # generation: are the answer's claims entailed by context?
    answer_relevancy,    # generation: does the answer address the question?
    context_precision,   # retrieval: how much of the retrieved context is useful?
    context_recall,      # retrieval: was the needed evidence retrieved? (needs reference)
)

eval_data = [
    {
        "user_input": "How does FlashAttention reduce memory?",
        "response": "FlashAttention tiles Q, K, V into SRAM blocks and avoids materializing the O(N²) attention matrix, reducing memory to O(N).",
        "retrieved_contexts": [
            "FlashAttention is an IO-aware exact attention algorithm. It tiles the Q, K, V matrices into blocks that fit in SRAM...",
        ],
        # `reference` (the gold answer) is optional; context_recall needs it.
        "reference": "FlashAttention avoids storing the full N×N attention matrix by using tiled SRAM computation.",
    },
]

dataset = EvaluationDataset.from_list(eval_data)
results = evaluate(
    dataset=dataset,
    metrics=[faithfulness, answer_relevancy, context_precision, context_recall],
    # llm=..., embeddings=...  # pass wrapped judge/embedder to pin the judge model
)
print(results)
# Example output:
# {'faithfulness': 0.97, 'answer_relevancy': 0.92,
#  'context_precision': 1.00, 'context_recall': 0.95}

The RAG Design Space¶

Naive RAG is a starting point, not a destination. The design space is large and each axis can be tuned independently.

Retrieval Strategy¶

Hybrid search combines a BM25 sparse index (exact keyword matching) with a dense embedding index, fusing scores via Reciprocal Rank Fusion (RRF). This is almost universally better than either alone for general-purpose corpora:

where \(k = 60\) is a smoothing constant, \(R_1\) is the BM25 ranking, and \(R_2\) is the dense ranking.

HyDE (Hypothetical Document Embedding), introduced by Gao et al. (2022), flips the retrieval problem: instead of embedding the short query directly, the LLM generates a hypothetical answer document, and that document is used as the retrieval query. Since the hypothetical document uses the same vocabulary and style as the corpus, the retrieval often improves substantially for queries where the question and answer have very different surface forms.

Indexing Strategies¶

Parent-child chunks. Index small child chunks (128 tokens) for high retrieval precision, but when a child chunk is retrieved, look up and return its parent chunk (512 tokens) for generation context. This avoids losing context while keeping retrieval sharp.

Summary indexes. For each document, store a summary chunk alongside the raw chunks. The retriever can match against the summary (capturing document-level semantics), then return the full document or relevant sections.

Knowledge graph indexes (GraphRAG). Convert the corpus into a knowledge graph of entities and relationships, enabling retrieval by entity traversal rather than pure semantic similarity. See Advanced RAG: GraphRAG, Agentic RAG & Long-Context vs RAG.

Generator Configuration¶

The way retrieved context is formatted and prompted has a large impact on generation quality:

1. Positional bias mitigation. Shuffle retrieved chunks between runs, or put the most relevant chunk first and last.
2. Explicit grounding instruction. “Answer ONLY using the provided passages. If the answer is not there, say ‘I don’t know’.” This reduces hallucination significantly compared to implicit grounding.
3. Citation format. Instructing the model to cite “[chunk index]” inline allows post-processing to verify every cited claim.
4. Temperature. For factual QA, temperature 0.0 or very low temperatures are preferred — we do not want creative paraphrasing of facts.

Putting It Together: A More Complete Pipeline¶

Here is a pipeline that incorporates hybrid retrieval, parent-child chunk lookup, and explicit faithfulness checking. This is close to a production-quality open-source RAG system.

"""
Production-style RAG pipeline:
  - BM25 + dense hybrid retrieval (RRF fusion)
  - Parent-child chunk hierarchy
  - Faithfulness check (NLI-based)
Dependencies: rank_bm25, sentence-transformers, faiss-cpu, transformers
  pip install rank_bm25 sentence-transformers faiss-cpu transformers
"""

import numpy as np
from rank_bm25 import BM25Okapi
from sentence_transformers import SentenceTransformer, CrossEncoder
from transformers import pipeline as hf_pipeline
import faiss
from typing import List, Dict, Tuple

# ─────────────────────────────────────────────
# Data structures
# ─────────────────────────────────────────────
class Chunk:
    def __init__(self, text: str, chunk_id: int, parent_id: int):
        self.text = text
        self.chunk_id = chunk_id
        self.parent_id = parent_id   # index into parent_docs list

class Document:
    def __init__(self, full_text: str, doc_id: int):
        self.full_text = full_text
        self.doc_id = doc_id


# ─────────────────────────────────────────────
# Build parent-child index
# ─────────────────────────────────────────────
def build_parent_child_index(
    documents: List[str],
    child_size: int = 128,    # words per child chunk
    overlap: int = 16,
) -> Tuple[List[Document], List[Chunk]]:
    parent_docs = [Document(doc, i) for i, doc in enumerate(documents)]
    child_chunks = []
    chunk_id = 0
    for doc in parent_docs:
        words = doc.full_text.split()
        start = 0
        while start < len(words):
            end = min(start + child_size, len(words))
            text = " ".join(words[start:end])
            child_chunks.append(Chunk(text, chunk_id, doc.doc_id))
            chunk_id += 1
            if end == len(words):
                break
            start = end - overlap
    return parent_docs, child_chunks


# ─────────────────────────────────────────────
# Hybrid retrieval: BM25 + dense, fused by RRF
# ─────────────────────────────────────────────
class HybridRetriever:
    def __init__(self, chunks: List[Chunk], embed_model: str):
        self.chunks = chunks
        self.texts = [c.text for c in chunks]

        # BM25
        tokenized = [t.lower().split() for t in self.texts]
        self.bm25 = BM25Okapi(tokenized)

        # Dense (FAISS)
        self.encoder = SentenceTransformer(embed_model)
        embeddings = self.encoder.encode(
            self.texts, normalize_embeddings=True, show_progress_bar=True
        ).astype(np.float32)
        d = embeddings.shape[1]
        self.index = faiss.IndexFlatIP(d)
        self.index.add(embeddings)

    def _rrf(self, *rank_lists, k: int = 60) -> Dict[int, float]:
        """Reciprocal Rank Fusion over multiple ranked lists of chunk indices."""
        scores: Dict[int, float] = {}
        for ranked in rank_lists:
            for rank, idx in enumerate(ranked):
                scores[idx] = scores.get(idx, 0.0) + 1.0 / (k + rank + 1)
        return scores

    def retrieve(self, query: str, top_n: int = 10) -> List[Tuple[Chunk, float]]:
        # BM25 ranking
        bm25_scores = self.bm25.get_scores(query.lower().split())
        bm25_ranked = [int(i) for i in np.argsort(bm25_scores)[::-1][:top_n]]

        # Dense ranking
        q_vec = self.encoder.encode(
            [query], normalize_embeddings=True
        ).astype(np.float32)
        _, dense_indices = self.index.search(q_vec, top_n)
        # FAISS pads with -1 when fewer than top_n results exist; -1 would index
        # the LAST chunk via Python's negative indexing, so drop it explicitly.
        dense_ranked = [int(i) for i in dense_indices[0] if i >= 0]

        # Fuse
        fused = self._rrf(bm25_ranked, dense_ranked, k=60)
        sorted_ids = sorted(fused.items(), key=lambda x: -x[1])[:top_n]
        return [(self.chunks[idx], score) for idx, score in sorted_ids]


# ─────────────────────────────────────────────
# Cross-encoder reranker
# ─────────────────────────────────────────────
class Reranker:
    def __init__(self, model: str = "cross-encoder/ms-marco-MiniLM-L-6-v2"):
        self.model = CrossEncoder(model)

    def rerank(
        self,
        query: str,
        candidates: List[Tuple[Chunk, float]],
        top_k: int = 3,
    ) -> List[Tuple[Chunk, float]]:
        pairs = [(query, c.text) for c, _ in candidates]
        scores = self.model.predict(pairs)
        ranked = sorted(
            zip([c for c, _ in candidates], scores),
            key=lambda x: -x[1],
        )
        return ranked[:top_k]


# ─────────────────────────────────────────────
# NLI faithfulness checker (optional post-filter)
# ─────────────────────────────────────────────
class FaithfulnessChecker:
    """
    Uses an NLI model to check whether the generated answer
    is entailed by the retrieved context.
    Returns a float in [0, 1]; <0.5 suggests hallucination.
    """
    def __init__(self):
        # mnli model: entailment, neutral, contradiction
        self.nli = hf_pipeline(
            "text-classification",
            model="cross-encoder/nli-deberta-v3-small",
        )

    def score(self, premise: str, hypothesis: str) -> float:
        """Returns probability of entailment."""
        # Pass a {"text", "text_pair"} dict, NOT a manually "[SEP]"-joined string:
        # the tokenizer must build the sentence-pair encoding (special tokens and
        # token_type_ids) itself, or the cross-encoder sees a single malformed
        # sequence and the entailment score is meaningless.
        result = self.nli(
            {"text": premise, "text_pair": hypothesis},
            truncation=True,
            max_length=512,
            top_k=None,  # return scores for all labels, not just the top-1
        )
        # With top_k=None the pipeline returns a list of {label, score} dicts.
        label_score = {r["label"].lower(): r["score"] for r in result}
        return label_score.get("entailment", 0.0)

    def check_answer(self, answer: str, context: str, threshold: float = 0.5) -> bool:
        """Return True if answer appears to be grounded in context."""
        return self.score(premise=context, hypothesis=answer) >= threshold


# ─────────────────────────────────────────────
# Assemble the full pipeline
# ─────────────────────────────────────────────
def run_pipeline(
    query: str,
    retriever: HybridRetriever,
    reranker: Reranker,
    parent_docs: List[Document],
    faithfulness_checker: FaithfulnessChecker,
    generate_fn,  # callable(prompt: str) -> str
) -> Dict:
    # Step 1: Retrieve child chunks
    candidates = retriever.retrieve(query, top_n=20)

    # Step 2: Rerank
    top_chunks = reranker.rerank(query, candidates, top_k=3)

    # Step 3: Look up parent documents for richer context
    seen_parents = set()
    context_parts = []
    for chunk, score in top_chunks:
        parent_id = chunk.parent_id
        if parent_id not in seen_parents:
            context_parts.append(parent_docs[parent_id].full_text)
            seen_parents.add(parent_id)

    context = "\n\n---\n\n".join(context_parts)

    # Step 4: Generate
    prompt = (
        f"Context:\n{context}\n\n"
        f"Answer the question based ONLY on the context above.\n"
        f"Question: {query}\nAnswer:"
    )
    answer = generate_fn(prompt)

    # Step 5: Faithfulness check
    is_faithful = faithfulness_checker.check_answer(answer, context)

    return {
        "answer": answer,
        "context": context,
        "top_chunks": [(c.text, s) for c, s in top_chunks],
        "faithful": is_faithful,
    }

Long-Context LLMs vs RAG: The Design Decision¶

A recurring question for practitioners is: “If the model has a million-token context window — the norm for frontier models by 2026 — why do I need RAG at all — can’t I just stuff the whole knowledge base into the prompt?”

The honest answer is: it depends.

For corpora that fit in a long context (e.g., a single legal contract, a codebase under 100 k tokens), long-context prompting is simpler and more reliable. For large, dynamic, multi-document corpora (knowledge bases, enterprise wikis, customer support databases), RAG is the right tool. See Advanced RAG: GraphRAG, Agentic RAG & Long-Context vs RAG for a detailed comparison.

The two approaches also compose: iterative RAG retrieves a small context, reasons over it, then decides whether to retrieve more, effectively using long-context reasoning to stitch together multi-hop evidence. The Agentic Loop: ReAct, Plan-Execute & Reflection covers this pattern.

Serving a RAG System in Production¶

A few operational concerns that come up in every production RAG deployment:

Indexing latency vs freshness. Real-time re-embedding and re-indexing is expensive. Common patterns: batch re-index nightly; maintain a “hot” real-time index for recent documents and a “cold” indexed corpus for historical ones; serve queries against both and merge results.

Multi-tenancy and access control. Retrieved documents must respect document-level permissions. FAISS alone has no ACL support. Solutions include: metadata-filtered retrieval (Qdrant, Weaviate, Pinecone all support filter expressions), per-tenant index shards, or post-retrieval filtering.

Serving the encoder and reranker, not just the LLM. Calling SentenceTransformer.encode() inside your request handler ties an ML model’s lifecycle to your web process and wastes the GPU on batch-size-1 work. In production, put the encoder and the cross-encoder behind their own inference server: HuggingFace’s Text Embeddings Inference (TEI, a Rust server with continuous batching that serves both embedding and reranker models), Infinity, or vLLM’s pooling/embedding endpoint if you already run vLLM for generation. The sparse half deserves real infrastructure too — rank_bm25 is a teaching implementation that rescores the whole corpus in Python per query; ship bm25s (sparse-matrix BM25), Pyserini (Lucene), or an OpenSearch/Elasticsearch cluster when the corpus outgrows memory. See TensorRT-LLM, TGI & Other Serving Stacks.

Caching. Embedding the same query repeatedly is wasteful. Cache query embeddings keyed by the (normalized) query string. Also cache retrieval results for frequent queries. See Caching, Routing & Cost Control in Production.

Monitoring. Log every query, retrieved chunk IDs, faithfulness scores, and user feedback. A RAGAS-style offline eval batch should run on a sample of production logs nightly. See Observability, Logging & LLMOps.

Embedding model upgrades. If you switch encoder models, you must re-embed the entire corpus — the new model’s embedding space is incompatible with the old one. Plan for full re-indexing as a first-class operation. Dual-write during migration: index new documents with both old and new models until the old index is retired.

Further Reading¶

• Lewis et al., “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks”, NeurIPS 2020 — the original RAG paper combining DPR retrieval with BART generation end-to-end.
• Karpukhin et al., “Dense Passage Retrieval for Open-Domain Question Answering” (DPR), EMNLP 2020 — the bi-encoder retrieval model that underlies most dense RAG systems.
• Gao et al., “Precise Zero-Shot Dense Retrieval without Relevance Labels” (HyDE), ACL 2023 — hypothetical document embedding for improved query-document alignment.
• Es et al., “RAGAS: Automated Evaluation of Retrieval Augmented Generation”, 2023 — defines the faithfulness, answer relevance, and context precision metrics implemented by the ragas library.
• Liu et al., “Lost in the Middle: How Language Models Use Long Contexts”, TACL 2024 — empirical study of positional bias in long-context generation, directly relevant to multi-chunk RAG.
• Izacard & Grave, “Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering” (Fusion-in-Decoder), EACL 2021 — encodes each retrieved passage separately and fuses them in the decoder, scaling to far more passages than single-context concatenation.
• Shi et al., “REPLUG: Retrieval-Augmented Black-Box Language Models”, NAACL 2023 — treats the LLM as a black box and trains only the retriever via LM likelihood signals.
• Zhang et al., “RAFT: Adapting Language Model to Domain Specific RAG”, 2024 — fine-tuning recipe for making models better at extracting answers from retrieved context while ignoring distractor documents.
• LangChain and LlamaIndex open-source repos — the two most widely used RAG orchestration frameworks, with extensive examples of advanced retrieval patterns.

Exercises¶

1. (Conceptual) A colleague deploys naive RAG and, to be safe, retrieves the top \(k = 15\) chunks and concatenates all of them into the prompt. They observe that answer quality is worse than when they retrieved only \(k = 3\). Drawing on the chapter’s failure-mode taxonomy, give at least two distinct reasons why raising \(k\) can hurt, and name one prompt-formatting mitigation from the chapter that partially addresses one of them.

2. (Quantitative) You are sizing a flat (exact) index. Your corpus has \(N = 2{,}000{,}000\) chunk vectors produced by OpenAI text-embedding-3-small (\(d = 1536\)), stored as float32 in an IndexFlatIP.

(a) How much memory does the raw vector store require? (b) You consider switching to MiniLM (\(d = 384\)). What is the new footprint, and by what factor does it shrink? © The chapter quotes exact IndexFlatIP search over \(10^6\) vectors at \(d = 384\) as taking roughly 20-50 ms per query. If retrieval takes ~40 ms and LLM generation of the answer takes ~3 s, what fraction of end-to-end latency is retrieval, and what does this imply about where to optimize first?

3. (Quantitative) You run hybrid search over four candidate chunks \(\{A, B, C, D\}\) and fuse the two rankings with Reciprocal Rank Fusion using the chapter’s smoothing constant \(k = 60\), treating the top result as rank 1:

• BM25 ranking \(R_1\): \(A, B, C\)
• Dense ranking \(R_2\): \(C, A, D\)

Compute the RRF score of every chunk and give the final fused order. Which chunk wins, and what property of RRF makes it win even though it is never ranked first by either retriever?

4. (Quantitative) An LLM judge decomposes a generated answer into 5 atomic claims and finds 4 of them supported by the retrieved context. Separately, of the \(k = 5\) retrieved chunks, only 2 are judged necessary to answer the question.

(a) Compute the RAGAS faithfulness score and state whether it clears the chapter’s hallucination threshold. (b) Compute ContextPrecision@5. © The system has faithfulness \(= 0.8\) but the single unsupported claim happens to be the exact fact the user asked about. Which additional RAGAS metric would flag this, and why can faithfulness alone miss it?

5. (Implementation) The chapter warns that ANN retrieval can return several near-duplicate chunks (“semantic redundancy”), and names Maximal Marginal Relevance (MMR) as the fix. Modify the minimal RAG implementation’s retrieve function into a retrieve_mmr variant that over-fetches fetch_k candidates from FAISS and then greedily selects k of them by MMR, balancing query relevance against novelty. Use the MMR objective

$$ \text{MMR} = \lambda\,\cos(\mathbf{q}, \mathbf{d}) \;-\; (1-\lambda)\,\max_{s \in S}\cos(\mathbf{d}, \mathbf{s}), $$

where \(S\) is the set of already-selected chunks. Assume the same globally available encoder, index, all_chunks, and unit-normalized chunk_embeddings from the chapter’s minimal implementation.

import numpy as np
from typing import List, Tuple

def retrieve_mmr(
    query: str,
    k: int = 3,
    fetch_k: int = 10,
    lambda_mult: float = 0.7,   # 1.0 = pure relevance, 0.0 = pure diversity
) -> List[Tuple[str, float]]:
    """
    Retrieve top-k chunks with Maximal Marginal Relevance re-ranking.
    Over-fetches fetch_k candidates from FAISS, then greedily selects k
    that trade off query relevance against novelty vs already-picked chunks.
    """
    # Encode query (unit-norm so dot product == cosine similarity)
    q_vec = encoder.encode([query], normalize_embeddings=True).astype(np.float32)
    q = q_vec[0]

    # Over-fetch candidates from the exact index
    _, indices = index.search(q_vec, fetch_k)
    cand_idx = [int(i) for i in indices[0] if i >= 0]   # drop -1 padding
    cand_vecs = chunk_embeddings[cand_idx]              # (n, d), unit-normed

    # Cosine sim of each candidate to the query
    query_sim = cand_vecs @ q                           # (n,)

    selected_local: List[int] = []   # positions within cand_idx
    results: List[Tuple[str, float]] = []
    remaining = list(range(len(cand_idx)))

    while remaining and len(selected_local) < k:
        best_local, best_score = None, -np.inf
        for j in remaining:
            if selected_local:
                # max similarity to anything already chosen (redundancy penalty)
                diversity = max(
                    float(cand_vecs[j] @ cand_vecs[s]) for s in selected_local
                )
            else:
                diversity = 0.0
            mmr = lambda_mult * float(query_sim[j]) - (1 - lambda_mult) * diversity
            if mmr > best_score:
                best_score, best_local = mmr, j
        selected_local.append(best_local)
        remaining.remove(best_local)
        # report the query-relevance score for consistency with retrieve()
        results.append(
            (all_chunks[cand_idx[best_local]], float(query_sim[best_local]))
        )
    return results