RAG, from naive to measured
The short version
RAG makes a language model answer only from retrieved text instead of guessing, and this series measures each improvement step by step.
A gentle, chapter-by-chapter tour of Retrieval-Augmented Generation — start here, then follow any link that catches you for the deep dive.
Imagine handing someone a closed-book exam on a 1,000-page tax manual they have never read. They will either stay silent or make something up. That is a language model answering questions about your documents. Now hand them the same exam, but let them flip to the right pages first. The answers get honest. That open-book trick is RAG — Retrieval-Augmented Generation.
The mechanics are three small moves: fetch the text that looks relevant to the question, paste it into the prompt, and ask the model to answer only from what it was given. No retraining, no fine-tuning — you change what the model reads, not what it is.
Most writing about RAG hands you a bag of tricks — hybrid search, rerankers, HyDE — and promises each one helps. This series does the opposite. We built the most naive RAG system that could possibly work, then improved it one idea at a time, measuring every step on the same hard test. Some tricks delivered. Some didn't. A couple helped one thing while quietly hurting another.
This page is the map. Each section below is a one-screen, gentle explanation of one chapter: what the idea is, the problem it solves, and the one number it moved. When a section makes you want more, follow the Read the full chapter link at its end.
📓 Want to check the work? Every number below comes from one runnable notebook — the exact pipeline (OpenAI + Qdrant + BM25 + FlashRank + RAGAS), the full IRS corpus, the 50-question golden set. Open it in Google Colab and reproduce the whole climb.
The whole journey is eight steps. You start naive and measure it — that's the floor. Every step after makes the same system measurably more powerful, until a Novice pipeline is a Ninja one. Here's the climb:
The test, and the villain
RAG is scored on a hard IRS-forms corpus using 50 fixed questions, where forms are nearly indistinguishable from their own instructions.
Before any chapter, two things to meet — because everything below is scored against them.
Everything is measured on a deliberately unforgiving corpus: ~1,400 real IRS tax forms — the form f1040, its nonresident cousin f1040nr, dozens of schedules, and the instruction booklets that look almost identical to the forms they explain. We score against a frozen set of 50 golden questions. We picked a corpus where naive RAG is bad on purpose, so there is real room to climb.
That corpus has a recurring villain, and it stalks every chapter. Ask "What is Form 1040 used for?" and naive search returns i1040gi — the instructions for Form 1040 — not the form itself. Why? To a system that searches by meaning, "Form 1040" and "Instructions for Form 1040" are nearly the same thing. The same blind spot confuses 1040 with 1040-NR. Watch how each layer either lands a punch on this villain or pointedly fails to — that is the through-line of the whole series.
How do you actually measure a RAG system?
Score retrieval and generation separately with recall, precision, faithfulness, and hit-rate so changes are measured, not guessed.
You can't improve what you can't score. This is the single most-skipped step in the whole field, and skipping it is how teams ship for months on vibes — you change a setting, read three answers, decide they "feel" sharper, and push to production.
The fix is a scorecard. Split the system in two — retrieval (did we fetch the right text?) and generation (did the model answer faithfully from it?) — and put a meaning-aware number on each: recall, precision, faithfulness, and a hit-rate. Now every change is a measured before/after, not a guess.
Every number on this page comes from that scorecard, so this is the chapter to read first.
Read the full chapter: how do you actually measure a RAG system?
What does naive RAG look like?
Naive RAG runs the default chunk-embed-retrieve-generate loop with top-4 similarity, giving a rough but honest baseline of 0.65 recall and 0.75 faithfulness.
Naive RAG is the three-move loop with every default left untouched: chunk every document (cut it into pieces), embed the chunks (turn each into a vector that captures its meaning), retrieve the top 4 by similarity, drop them into a strict prompt, and generate. On real libraries it is almost one line:
from llama_index.core import VectorStoreIndex
index = VectorStoreIndex.from_documents(docs)
engine = index.as_query_engine(similarity_top_k=4, text_qa_template=STRICT_PROMPT)
print(engine.query("What is Form 1040 used for?"))
It works — and it's rough: 0.65 recall, 0.75 faithfulness, and it walks straight into the form-versus-instructions trap. This is the honest floor. Everything after earns its way up from here.
How does chunking change RAG?
Shrinking chunks from 1,024 to 512 tokens sharpened embeddings and lifted recall, precision, and faithfulness, but did not fix the underlying retrieval problem.
Chunking is the step where you cut documents into smaller pieces before embedding them. It is the quietest knob in RAG: most systems set it to a library default and never touch it again, yet it silently decides how much of your retrieval works at all.
Naive RAG used big 1,024-token chunks; we tried 512. Smaller chunks aren't diluted by surrounding filler, so the line that actually answers a question gets a sharper embedding and ranks higher.
What moved: every metric — recall 0.65 → 0.70, precision and faithfulness up with it. What didn't, as expected: the hit-rate stayed stuck near 0.56, because the villain is a retrieval problem, and no chunk size teaches search that 1040 and 1040-NR are different strings. A layer fixes the problem it's shaped for and leaves the rest.
Read the full chapter: how does chunking change RAG?
How does hybrid search work?
Hybrid search combines keyword matching and meaning-based retrieval, but blending them can dilute the precision keyword search already had on identifier-heavy text.
To finally separate 1040 from 1040-NR, hybrid search runs two retrievers over the same corpus at once: one that reads exact words (BM25 — the classic keyword-matching algorithm) and one that reads meaning, then fuses their rankings. Picture two researchers with opposite blind spots — the translator who understands "nonresident" ≈ "alien individual," and the clerk who knows 1040-NR is simply a different token.
What moved: faithfulness jumped to 0.87 — keyword matching pulls the actual form more often, so answers are better grounded. The surprise: on this identifier-heavy corpus, pure keyword search actually beat the fused version on recall, because blending the fuzzy meaning-search back in diluted the keyword precision that was already winning. Fusion is not a free lunch — and we only knew because we measured.
Read the full chapter: how does hybrid search work?
How does a reranker work?
A reranker reorders an already-retrieved shortlist to push the best answers to the top, but it cannot fix recall if the right document was never retrieved.
A reranker takes the rough pile your search pulled back and reorders it so the documents that truly answer the question rise to the top — a slow, careful second reader let loose on the shortlist before the answer is written. We pulled a wide pool of 20 candidates and let a cross-encoder read each one together with the question and keep the best 4.
What moved: faithfulness again, to 0.91, the best yet — a cleaner top-4 means fewer look-alike booklets to trip the model. What didn't, and why it is not a failure: recall stayed flat at 0.73. A reranker can only reorder the shortlist it was handed; if the right form isn't in the top-20, no reordering invents it. Recall is set upstream. This is the most important "no" in the series, and it sets up the next two chapters.
Read the full chapter: how does a reranker work?
How does query rewriting help (and hurt)?
Query rewriting boosts recall by searching multiple rephrasings, but more than doubles latency since it is paid on every request.
If the right document never even enters the pool, stop tweaking the ranking and change the question. Query rewriting treats the user's words as a first draft: the LLM rewrites the question several ways and searches with all of them — a wider net that catches documents a single phrasing missed.
What moved: recall, off its two-chapter plateau, 0.73 → 0.78 — exactly the pool problem the reranker couldn't touch. The cost, as expected: latency more than doubled to 6.5s, because every query now triggers an LLM rewrite plus four retrievals. A real gain, rented on every single request. (We also tried HyDE, which matched the recall but quietly damaged faithfulness — a worse trade.)
Read the full chapter: how does query rewriting help (and hurt)?
How does contextual retrieval work?
Prepending an LLM-written context line to each chunk before embedding improved every metric and cut latency, making query rewriting unnecessary.
Here is the turn. Query rewriting fixed the pool by reshaping the question every time. Contextual retrieval fixes the same pool by reshaping the documents once: before embedding, we prepend an LLM-written "you are here" line to each chunk — "Form 1040-NR, Nonresident Alien return; this part covers…" — so a chunk stops being an orphan and starts describing itself.
What moved: nearly everything, hard — recall 0.78 → 0.83, precision 0.85 → 0.95, faithfulness 0.91 → 0.97, the biggest jumps in the series. Self-describing chunks are easier to find and easier for the reranker to tell apart, so every downstream layer got sharper. And latency fell (6.5s → 2.6s), because it beat multi-query outright — so we deleted query rewriting. The system got simpler and better at once. This was the biggest win in the series.
Read the full chapter: how does contextual retrieval work?
How does a semantic cache work?
A semantic cache cuts the cost and latency of repeated questions by matching meaning, but never improves answer accuracy.
Accuracy was solved; now economics. A semantic cache remembers the questions already answered and hands back the stored answer when a new one means the same thing — matching on meaning, so it catches the paraphrases an exact-match cache misses.
What moved: latency, by 92% on repeats (2.6s → 0.2s), because a cache lookup is a millisecond dot-product instead of a full pipeline run. What it deliberately doesn't touch: accuracy — a cache can't make an answer better, only cheaper to serve again. Its real danger is the threshold: set it too loose and two different-but-similar questions collide (the villain, one last time).
Read the full chapter: how does a semantic cache work?
The whole climb, in one table
Across chunking, hybrid search, reranking, and contextual retrieval, scores climbed from 0.65/0.78/0.75 to 0.83/0.95/0.97, nearly solving retrieval for this corpus.
Before the numbers, a word on what the columns mean — because the whole table is just five metrics moving. Four are meaning-aware scores from RAGAS, and one is a blunt deterministic counter. Each gets its own short deep-dive in the Measurements series:
- Context Recall — did we even fetch the facts the answer needs? (The ceiling on everything else.)
- Context Precision — are those facts ranked above the noise, where the model reads?
- Faithfulness — is the answer grounded in what we fetched, with nothing made up?
- Answer Relevancy — does it actually answer the question that was asked?
- Retrieval hit-rate (the
hit column) — did retrieval pull at least one expected form? A model can't be charmed into faking this one.
The last column, latency, is just end-to-end seconds per question — the cost we weigh every gain against. Now the climb:
Every row is one chapter, one change, scored on the same 50 golden questions over the same ~1,400 forms.
| layer |
Recall |
Precision |
Faithfulness |
hit |
latency |
| Naive (1024) |
0.65 |
0.78 |
0.75 |
0.50 |
1.0s |
| Chunk @ 512 |
0.70 |
0.84 |
0.80 |
0.56 |
0.9s |
| Hybrid |
0.73 |
0.85 |
0.87 |
0.58 |
1.4s |
| + Reranker |
0.73 |
0.85 |
0.91 |
0.60 |
2.6s |
| + Multi-query |
0.78 |
0.85 |
0.91 |
0.66 |
6.5s |
| Contextual (−multi) |
0.83 |
0.95 |
0.97 |
0.68 |
2.6s |
| sparse baseline (for reference) |
0.85 |
0.74 |
0.90 |
0.66 |
1.3s |
From a naive 0.65 / 0.78 / 0.75 to 0.83 recall, 0.95 precision, 0.97 faithfulness — beating the strong pure-keyword baseline on everything except recall (by 0.02), where it crushes it on precision (0.95 vs 0.74). For this corpus, the retrieval problem is essentially solved.
What the series is really teaching
Each RAG layer fixes only its own problem, so measure first, then match the layer to the actual failure.
Step back from the numbers and there's a pattern worth more than any single technique. Each layer fixes the problem it is shaped to fix, and is powerless on the others. Chunking sharpens chunks you already find; it can't find new ones. Reranking reorders the pool; it can't retrieve into it. Query rewriting and contextual retrieval widen the pool; caching doesn't touch quality at all. Knowing which problem you have tells you which layer to reach for — and which layers won't help no matter how hard you turn them up.
That is why the measuring came first, and why we reported the flops as loudly as the wins. A reranker that moved faithfulness but not recall taught us more than a clean victory would have.
Five lessons carry the whole series:
- Measure before you tune. Split RAG into retrieval and generation, score four meaning-aware metrics on a frozen golden set, and never trust "it looks better."
- Each layer has a shape. Chunking and contextual retrieval sharpen and widen the pool; reranking reorders it; caching only makes serving cheaper. Match the layer to the failure.
- Recall is set upstream. No reranker invents a document that retrieval never fetched — when recall is stuck, fix the pool, not the ranking.
- The cheap fix can win. Contextual retrieval (pay once at index time) beat query rewriting (pay forever per query) on every axis — and let us delete a whole component.
- Report the flops. BM25 beating hybrid, HyDE hurting faithfulness, the reranker not moving recall — the honest "no" is where the real understanding lives.
What's next
Upcoming work covers production hardening and agentic RAG, which lets the system decide when to retrieve, reformulate, and verify.
Two arcs remain. Production hardening — persistence, invalidating the cache when documents change, warming, query routing. Then agentic RAG — letting the system decide when to retrieve, reformulate, and verify, instead of running one fixed pipeline for every question. Those are coming.
For now, start with how we measure a RAG system, then follow the climb chapter by chapter — and run the code that proves every number in the companion Colab notebook.
The short version
RAG systems need a scorecard splitting retrieval and generation into measurable metrics, otherwise every change is just a guess.
A RAG system retrieves relevant documents and then asks a language model to write an answer from them. Measuring it means putting a number on whether that answer is actually any good — and, just as important, on where it went wrong when it isn't.
This is the single most-skipped step in the whole field, and skipping it is how teams ship for months on vibes. You change a setting, read three answers, decide they "feel" sharper, and push to production. Without a scorecard, every change is a guess, and you only learn you guessed wrong when your users do. This post is the scorecard the rest of the series leans on.
In this post, we will cover the following:
- Why "it looks better" is a trap
- Splitting RAG into retrieval and generation
- The four metrics, with plain-English intuition
- Why we let an LLM be the judge — and how RAGAS does it in code
- The deterministic counters that keep the judge honest
- The frozen 50-question discipline
- What "good" looks like on a hard corpus
- Summary and what's next
Why "it looks better" is a trap
One good-looking answer on an easy question hides failures that only separate measurement can distinguish and fix.
Let's slow down here, because this trap is subtle and almost everyone falls into it at least once. You change your chunk size, ask your favourite test question, and the answer reads beautifully. A win, surely? Except your favourite question was always easy. The change you just made quietly broke a harder class of questions you never thought to check — and you won't find out until much later.
The deeper problem is that a RAG answer can be wrong in several different ways, and they do not look the same from the outside. The system might have fetched the wrong documents. It might have fetched the right ones but buried them under junk. It might have had perfect context and still made something up. Or it might have answered a different question than the one you asked.
Here is where it gets concrete. Our running corpus is real IRS tax forms, and the classic confusion is Form 1040 versus its instruction booklet, i1040gi. A user asks how to fill in a line, and the honest answer lives in the instructions — but a careless retriever grabs the form itself, which states the line without explaining it. A different failure: someone asks about Form 1040 and the system serves up 1040-NR, the non-resident version, which looks almost identical and is wrong for most filers. "It looks better" cannot tell these failures apart, and each one gets fixed by a different change. So we need numbers that pull them apart.
Split the problem in two
RAG quality splits into retrieval (recall, precision) and generation (faithfulness, relevancy), each failing and measured differently.
The first useful move is to stop treating "RAG quality" as one thing. A RAG answer is produced in two stages — first retrieval finds the context, then generation writes an answer from it — and each stage fails in its own way. So we measure them separately:
RETRIEVAL — did we find the right stuff?
Context Recall did we fetch the facts the answer needs?
Context Precision are those facts ranked above the noise?
GENERATION — did we use it well?
Faithfulness is the answer grounded in what we fetched?
Answer Relevancy does it actually answer the question asked?
Keep this split in your head — it is the most valuable mental model in this entire series. Almost every technique we cover targets exactly one of these four metrics. Knowing which one a change is supposed to move tells you what to expect, and what not to expect, before you even run the experiment. If you add a reranker hoping to fix recall, you are going to be disappointed, and the split is what tells you why in advance.
The four metrics, with intuition
RAG quality splits into retrieval (recall, precision) and generation (faithfulness, relevancy), and each failure mode shows up in a different metric.
Let me define each one plainly and tie it back to the tax assistant, because intuition matters more than the formula here.
Context Recall — did we even fetch the needed facts? Recall simply asks: of the facts the correct answer requires, how many showed up in what we retrieved? This is the ceiling on everything else. A model cannot use a fact it never saw. If recall is low, no amount of clever prompting or reranking will save you, because the right information never made it into the room. If the answer needs the i1040gi instructions and you only pulled the bare 1040 form, recall is where that shows up. When the series later mentions "the recall ceiling," this is the number we mean.
Context Precision — are the relevant chunks near the top? You can retrieve the right document and still bury it under four irrelevant ones. Precision asks whether the good chunks are ranked high, where the model will actually read them, rather than padding the bottom of the context window. The clean way to remember the pair: recall is "is it in the pile?", precision is "is it on top of the pile?"
Faithfulness — did the answer stick to the context? Faithfulness checks whether every claim in the answer is actually supported by the retrieved chunks, or whether the model invented some. This is your hallucination detector. On a tax assistant it is the metric that matters most — a fluent, confident, completely made-up answer about someone's taxes is the worst failure this system can produce.
Answer Relevancy — did it answer the actual question? A response can be perfectly grounded in real context and still wander off, or quietly answer a neighbouring question. Relevancy catches the answer that is technically true but unhelpful — the kind that recites what Form 1040-NR is when you asked about 1040.
Four numbers, two stages. When a change lifts faithfulness but leaves recall flat, that is not a mystery — it is telling you the change improved how the model used context, not what context it found. The scorecard turns surprises into explanations.
Why we let an LLM be the judge
RAGAS uses an LLM judge with four metrics, scored against human-written reference answers, to grade RAG outputs for meaning rather than exact text.
How do you score "is this claim supported by this context" automatically? Exact string matching falls apart instantly. "April 15th" and "the fifteenth of April" are the same answer with zero characters in common. You need a grader that reads for meaning, not characters.
A quick note for you on one term you'll see everywhere: cosine similarity simply measures how close two vectors of text point — 1.0 means identical in meaning, 0 means unrelated. That handles meaning-matching, but scoring a full claim against a paragraph of context needs more judgement than a single similarity number can give. So we use RAGAS (Retrieval-Augmented Generation Assessment), a library that grades with an LLM acting as the judge.
In code, the four metrics are exactly the retrieval/generation split made concrete. Here is the list, lifted straight from our eval.py:
from ragas.metrics import (
Faithfulness,
LLMContextPrecisionWithReference,
LLMContextRecall,
ResponseRelevancy,
)
metrics = [
LLMContextRecall(),
LLMContextPrecisionWithReference(),
Faithfulness(),
ResponseRelevancy(),
]
Notice LLMContextPrecisionWithReference. The "WithReference" part is the whole defence against the obvious worry — isn't this the model grading its own homework? Every golden question in our set ships with a human-written reference answer (the ground truth). Recall and precision are judged against that human ground truth, not against the model's own opinion of itself.
You can see that contract in how each evaluation sample is assembled. We run the pipeline on a question and pack four things into one SingleTurnSample:
from ragas import SingleTurnSample
out = pipeline.answer(row["question"])
sample = SingleTurnSample(
user_input=row["question"],
retrieved_contexts=out["contexts"], # what retrieval found
response=out["answer"], # what generation wrote
reference=row["ground_truth"], # the human-written answer
)
The judge sees the question, the retrieved context, the generated response, and the reference, and grades all four metrics from that. All of them run on a 0..1 scale, higher is better.
The counters that keep the judge honest
LLM judges can be fooled, so quality scores are always paired with two deterministic, ungameable checks: retrieval hit-rate and latency.
An LLM judge is powerful, but it is still an LLM, and it can be wrong or be charmed by a fluent answer. We do not take its word alone. We pair the LLM-judged metrics with two cheap, deterministic numbers that cannot be talked into anything:
- Retrieval hit-rate — did we pull at least one of the expected source forms? A blunt yes/no. In code it is just a set intersection between the docs we expected and the docs we retrieved — no model involved, so a persuasive answer cannot game it.
- Latency — end-to-end seconds per question. Every "improvement" in the series is weighed against what it costs here, because a 3% quality gain that triples latency is usually a bad trade.
That hit-rate is computed right in the evaluation loop, alongside building the samples:
if set(row.get("expected_docs", [])) & set(out.get("retrieved_docs", [])):
hits += 1
Between a reference-grounded judge and two un-foolable counters, we trust the direction the scorecard points — and we stay honest about the LLM-as-judge risk by never relying on it in isolation.
The rule that keeps us honest
Every RAG change is tested against the same fixed 50-question set so results stay comparable and honest.
Here is the discipline that makes the whole series mean something. We freeze one golden set of 50 questions over the same ~1,400 real IRS forms. From chapter to chapter we change exactly one thing and re-score on that identical set. No moving the goalposts, no quietly swapping the test when a change underperforms.
And when a change doesn't help — or helps one metric while hurting another — we report it. That is not a footnote, it is the point. The chapter where a reranker lifts faithfulness but moves recall by precisely zero teaches you more about how RAG actually works than any clean victory could. Holding the question set frozen is what makes those deltas trustworthy: the only thing that changed was the one thing you changed.
What "good" looks like here
A hard tax-form corpus pushed recall from 0.65 to 0.83 and faithfulness from 0.75 to 0.97 through six reproducible one-change-at-a-time improvements.
We deliberately chose a hard corpus — dense, near-duplicate tax forms where 1040 and 1040-NR sit side by side — so the naive baseline is honestly mediocre: 0.65 recall, 0.75 faithfulness. That is the floor, and it leaves real room to climb.
Six chapters of one-change-at-a-time improvements later, the system sits at 0.83 recall, 0.97 faithfulness. Every step of that climb is a number you can reproduce on your own machine, not a claim you have to take on faith. That is the whole idea: read the intuition, then look at the receipt.
Measured output
The naive RAG baseline retrieves the right form only half the time, though its answers stay fairly grounded and fast.
The exact RAGAS scorecards this chapter's run printed — 50 golden questions over the full IRS corpus, gpt-5.4-mini judge + text-embedding-3-small embeddings. Reproduce them with the companion Colab notebook (it inlines the exact pipeline code and embeds these JSON scorecards).
📊 RAGAS scorecard — 00 · naive baseline
Context Recall 0.650 did we fetch the needed facts?
Context Precision 0.779 relevant context ranked high?
Faithfulness 0.751 grounded, no hallucination?
Answer Relevancy 0.584 answers the actual question?
Retrieval hit-rate 0.500 ≥1 expected form retrieved
Avg latency 1.01s per question, end-to-end
Summary
Measuring RAG well means splitting retrieval and generation into four meaning-aware metrics, checked against deterministic counters on a frozen golden set.
You cannot improve what you cannot measure, and "it looks better" is a measurement that lies to you. The fix is to split RAG into its two real stages, score four meaning-aware metrics against a frozen golden set, and guard the LLM judge with deterministic counters. Do that and every change becomes an explanation instead of a guess.
Key takeaways:
- A RAG answer fails in distinct ways — wrong docs, buried docs, hallucination, off-topic — and each needs a different fix, so measure them separately.
- Retrieval is graded by Context Recall (is the fact in the pile?) and Context Precision (is it on top of the pile?); recall is the ceiling on everything downstream.
- Generation is graded by Faithfulness (grounded, no hallucination) and Answer Relevancy (answers the actual question).
- RAGAS uses an LLM as judge, made reliable by a human-written reference answer per question (
LLMContextPrecisionWithReference).
- Retrieval hit-rate and latency are deterministic counters that cannot be charmed by a fluent answer — they keep the judge honest.
- One frozen set of 50 questions, one change per chapter, and honest reporting of changes that didn't help are what make the numbers mean something.
What's next
The next chapter tests how changing document chunking affects the four scorecard numbers.
Now that you have the scorecard, you can watch it move. The next chapter takes the cheapest improvement in the whole book — changing how documents are split into chunks — and shows exactly which of these four numbers it lifts and which it leaves alone. Read on in how chunking changes RAG.
The short version
Changing only chunk size, the quietest and cheapest RAG setting, beat the baseline on every measured metric.
Chunking is the step where you cut your documents into smaller pieces before you embed them. An embedding is a single fixed-size list of numbers that captures the meaning of a span of text, and chunking decides how much text each one has to summarize.
This matters because chunk size is the quietest knob in RAG. Most systems set it to a library default and never touch it again, yet it silently decides how much of your retrieval works at all. It is also the cheapest improvement in this whole series. In this chapter we change nothing but the size of the pieces, and we beat the baseline on every metric we measure.
In this post, we will cover the following:
- Why we cut documents into pieces at all instead of embedding whole forms
- What a chunk, a token, and an overlap actually are
- The Goldilocks problem: why too big and too small both fail
- The sweep we ran over the full IRS corpus, with exact numbers
- What moved when we halved the default, and the reason it moved
- What chunking deliberately did not fix, and why
- The code that produced these results
- Where we go next
Why you can't skip this step
Embeddings capture meaning best for a focused span of text, so documents must be split into chunks before retrieval can work.
You might wonder why we chop documents up at all. Why not embed each whole tax form as one vector and search those?
Because an embedding is a single fixed-size summary of meaning, and a whole tax form is far too much meaning to fit in one. Cram an entire 30-page instruction booklet into a single vector and you get a blurry average of everything it talks about, which is useless for finding the one paragraph that answers a specific question. Embeddings work best on a focused span of text that is about one thing.
So you split each document into chunks, embed each chunk, and search those. A chunk is just one of those slices of a document, and it is the unit your retriever actually returns. When a user asks a question, you embed the question, find the chunks whose embeddings sit closest to it, and hand those chunks to the language model as context. Everything the model knows about the answer, it learns from the chunks you retrieved, which is why their size matters so much.
The question this whole chapter answers is simple to state and surprisingly deep to get right: how big should a chunk be? It feels like a throwaway setting, the kind you accept from the library and forget. It is actually the dial that sets the resolution of your entire pipeline, and getting it wrong caps how well every later component can possibly perform.
The Goldilocks problem
Chunk size must balance two failures: chunks too big dilute the key sentence, chunks too small lose the context that gives it meaning.
Chunk size is a balance between two opposite failures, and you feel both as soon as you picture them.
Make the chunks too big and the one line that answers the question drowns. A 1,000-token chunk might contain the sentence you need plus nine sentences you don't, so its embedding gets diluted, pulled toward the average of all ten, and it ranks lower than it should. A token here is roughly a word-piece, the unit both the embedding model and the language model count text in, so a 1,000-token chunk is about 750 words. You also waste the model's limited prompt budget shipping all that filler into the answer step.
Make the chunks too small and you lose the context that gives a line meaning. Picture a chunk that is just $13,850. Retrieved on its own it is almost worthless. Which form? Which filing status? Which year? The number got orphaned from the rule it belongs to. A chunk has to be small enough to stay focused but large enough to stand on its own.
There is a useful analogy here: slicing bread for sandwiches. The whole loaf is one giant chunk. It technically contains everything, but it is useless to eat in one bite. Crumbs are tiny chunks. You can chew them, but you have lost which slice the filling was ever on. Even slices with a little overlap are token-aware chunks done right. Each bite is coherent, and the overlap means a word split at the edge of one slice still shows up whole in its neighbour.
The sweet spot is corpus-specific, and the only honest way to find it is to measure. There is no universal best chunk size, only the best size for your documents and your questions, and a number that wins on prose can lose badly on dense tabular tax forms. If you have not yet set up a scorecard, measuring a RAG system is the chapter that builds the one we use here.
The sweep
Our naive baseline used the library's default 1,024-token chunks.
Our naive baseline used the library's default 1,024-token chunks. We swept three sizes over the full IRS corpus, scoring the same 50 golden questions each time so that the only thing changing between runs was the chunk size.
One more term before the table. The overlap is how many tokens neighbouring chunks share, so a sentence that falls on a boundary is not sliced clean in half and lost. A small overlap means a fact split at the edge of one chunk still appears whole in its neighbour.
| chunk / overlap |
Recall |
Precision |
Faithfulness |
hit-rate |
| 1024 / 200 (default) |
0.65 |
0.78 |
0.75 |
0.50 |
| 512 / 128 |
0.70 |
0.84 |
0.80 |
0.56 |
| 256 / 64 |
0.69 |
0.80 |
0.77 |
0.55 |
What moved, and why
512-token chunks beat both 1024 and 256, showing there's a sweet spot between dilution and orphaned context.
Halving the default to 512 tokens beat 1024 on everything. Recall went 0.65 to 0.70, precision 0.78 to 0.84, faithfulness 0.75 to 0.80. The reason is exactly the "too big" failure above. Tighter chunks are less diluted, so the chunk that actually contains the answer has a sharper embedding and ranks higher in the search. And because the model now reads four focused chunks instead of four bloated ones, its answers are better grounded, which is the faithfulness bump you see in the last column.
Let's slow down here, because the next row is where the lesson lives. Notice what happened at 256 tokens. Recall held, but precision and faithfulness slipped back below the 512 mark. That is the "too small" failure showing up in the numbers. At 256 tokens, chunks start orphaning context, splitting a rule from the figure it applies to, exactly like the lonely $13,850 from earlier. There is a floor, and for this corpus 512 sat right above it. That is the whole Goldilocks story, in one table: tighter helps until it doesn't, and the turn is something you can see rather than guess.
What it deliberately did not fix
Chunking improves precision on retrieved chunks but cannot fix recall failures caused by embeddings confusing near-identical form numbers like 1040 and 1040-NR.
Now look at the hit-rate column again. Even at the winning size it only crept to 0.56. Nearly half the questions still failed to retrieve a single correct form. And that is expected, because the thing breaking those questions isn't chunk size at all.
It is our recurring villain. Dense search cannot tell 1040 from 1040-NR, because to an embedding those two strings mean almost the same thing. They sit right next to each other in vector space no matter how you slice the document. No chunk size on earth teaches a retriever the difference between two near-identical form numbers. That is a retrieval recall problem, not a granularity one.
This is the pattern worth internalising early. Each layer fixes the problem it is shaped to fix, and leaves the others for the layer that is shaped to fix them. Chunking sharpened every chunk we already retrieve, raising precision and faithfulness on the questions that were already landing. It could not, and was never supposed to, retrieve the chunks we were missing entirely. Keeping those two jobs separate in your head is what lets you debug a RAG system instead of guessing at it.
The code
The same evaluation harness scores three chunking strategies that differ only in chunk size and overlap, and 512 wins on recall without hurting faithfulness.
The whole experiment is one dictionary and one parser. Each strategy is a row on the scorecard, and the only thing that varies between rows is how documents get cut.
STRATEGIES = {
"sentence_1024": {"kind": "sentence", "chunk_size": 1024, "chunk_overlap": 200}, # default
"sentence_512": {"kind": "sentence", "chunk_size": 512, "chunk_overlap": 64}, # tuned tighter
"sentence_256": {"kind": "sentence", "chunk_size": 256, "chunk_overlap": 32}, # small & focused
}
Turning a named strategy into an actual splitter is a one-liner. The SentenceSplitter is token-aware: it splits on sentence boundaries, packs sentences together up to chunk_size tokens, and overlaps neighbours by chunk_overlap tokens so an edge-split fact still appears whole somewhere.
from llama_index.core.node_parser import SentenceSplitter
def make_parser(cfg):
return SentenceSplitter(
chunk_size=cfg["chunk_size"],
chunk_overlap=cfg["chunk_overlap"],
)
Scoring a strategy means building an index with that parser and running it through the exact same evaluation harness from the previous chapter. Nothing else about the system changes, which is what makes the comparison fair.
for strategy in STRATEGIES:
rag = ChunkedRAG(strategy=strategy).build()
scorecard = run_eval(rag, name=f"02_{strategy}")
scorecards.append(scorecard)
That loop is the entire A/B test. Build each strategy, score it against the 50 golden questions, and read the deltas against the baseline. The winner is the strategy with the highest recall that doesn't sacrifice faithfulness, which is how we landed on 512.
One detail worth noting from the real code: each strategy builds its own index once and is then reused, instead of re-embedding the corpus on every run. Embedding is the slow, paid step, so you pay it a single time per strategy and load the saved vectors afterward. For a 40-form corpus this is convenience. In production, where the corpus is large and the embedding bill is real, it is the difference between a system you can iterate on and one you can't afford to touch.
So, how does chunking change RAG?
Chunk size sets the resolution of a RAG system and is the cheapest, most foundational parameter to tune.
It sets the resolution of everything downstream. Too coarse and your answers drown in filler. Too fine and they lose the context that made them mean anything. Finding the right size is the cheapest win available to you, a single parameter with no new components, and it is the foundation the rest of the system is built on. We carried 512 / 128 forward as that foundation, and every later chapter is measured on top of it.
Measured output
Across 50 golden questions, chunk size 512 gave the best overall RAGAS scores, beating 1024 and 256 on most metrics.
The exact RAGAS scorecards this chapter's run printed — 50 golden questions over the full IRS corpus, gpt-5.4-mini judge + text-embedding-3-small embeddings. Reproduce them with the companion Colab notebook (it inlines the exact pipeline code and embeds these JSON scorecards).
📊 RAGAS scorecard — 02 · chunk 1024
Context Recall 0.650 did we fetch the needed facts?
Context Precision 0.779 relevant context ranked high?
Faithfulness 0.751 grounded, no hallucination?
Answer Relevancy 0.584 answers the actual question?
Retrieval hit-rate 0.500 ≥1 expected form retrieved
Avg latency 1.01s per question, end-to-end
📊 RAGAS scorecard — 02 · chunk 512 (win)
Context Recall 0.700 did we fetch the needed facts?
Context Precision 0.838 relevant context ranked high?
Faithfulness 0.799 grounded, no hallucination?
Answer Relevancy 0.664 answers the actual question?
Retrieval hit-rate 0.560 ≥1 expected form retrieved
Avg latency 1.10s per question, end-to-end
📊 RAGAS scorecard — 02 · chunk 256
Context Recall 0.690 did we fetch the needed facts?
Context Precision 0.745 relevant context ranked high?
Faithfulness 0.698 grounded, no hallucination?
Answer Relevancy 0.511 answers the actual question?
Retrieval hit-rate 0.560 ≥1 expected form retrieved
Avg latency 0.96s per question, end-to-end
Summary
Chunk size controls precision and faithfulness in RAG, but smaller chunks can't fix retrieval misses like distinguishing similar form numbers.
Chunking is where you decide the granularity of retrieval, and that decision quietly governs how much of your RAG system works.
- An embedding summarizes one span of text, so you split documents into focused chunks before embedding rather than embedding whole forms.
- Chunk size is a Goldilocks problem: too big dilutes the answer, too small orphans its context.
- On the IRS corpus, halving the default to 512 / 128 beat 1024 on every metric (recall 0.65 → 0.70, precision 0.78 → 0.84, faithfulness 0.75 → 0.80).
- Going further to 256 held recall but slipped precision and faithfulness back, marking the floor where chunks start losing context.
- Chunking did not lift hit-rate past ~0.56, because it cannot make dense search distinguish
1040 from 1040-NR.
- Each layer fixes only the problem it is shaped to fix. Chunking sharpens chunks you already retrieve; it does not retrieve the ones you miss.
What's next
When the hit-rate ceiling appears, the bottleneck shifts from chunking to retrieval itself, and hybrid search is the fix.
The hit-rate ceiling is the loud signal that retrieval, not granularity, is now the bottleneck. The villain is two near-identical form numbers that mean the same thing to an embedding, and the fix is a search that reads exact words alongside meaning. That is how hybrid search works, where we finally start retrieving the chunks chunking left on the table.
The short version
Rewriting the user's query with an LLM, not just reranking, is the lever that finally broke a stuck recall plateau, at a real cost.
Query rewriting is the idea that the question a user types is just a first draft of what you should actually search for. Instead of feeding the user's exact words into retrieval, you let the LLM reshape them into something that lands closer to the right document, and you search with that.
This matters because of a wall we keep hitting. Your Context Recall sits frozen, your reranker has nothing left to give, and the right document never even makes it into the shortlist. When ranking can't help you, the only lever left is the question itself. This is the chapter where our recall finally broke off a two-chapter plateau, and it came with a bill worth being honest about.
In this post, we will cover the following:
- Why your retrieval can be stuck even with hybrid search and a reranker in place
- The blunt insight: the user's words aren't always the best search
- Multi-query: casting a wider net with several phrasings
- HyDE: searching with a fake answer
- The one honesty trick that keeps the whole experiment fair
- What we measured on the IRS corpus, with exact numbers
- Why a real recall win can still be the wrong long-term choice
- Where we go next
The wall we hit
Context Recall stayed stuck at 0.73 because reranking can't retrieve documents that were never shortlisted; only the query itself can fix that.
For two chapters our Context Recall sat frozen at 0.73. We had added hybrid search, then a cross-encoder reranker, and neither one budged it. The reranker chapter told us exactly why. A reranker can only reorder the shortlist it was handed. For the questions we keep missing, the right document was never in that shortlist to begin with. The ceiling is set by what retrieval pulls, and reordering a pool that lacks the answer cannot conjure the answer back in.
So if reordering can't help, there is only one place left to push. You stop touching how you rank, and you change what you ask. If the question the user typed doesn't surface the right document, send a better question.
The blunt insight: the user's words aren't always the best search
User questions and the documents that answer them often share almost no vocabulary, so literal search can miss the right document entirely.
Picture the villain we have been fighting all along — the IRS corpus, where the document that answers a question almost never uses the same words as the question. A user asks: "If I live abroad and I'm not a citizen, which form do I file?" The document that answers it talks about "nonresident alien individuals" and the form 1040-NR.
Notice that the question and the answer share almost no vocabulary. The user wrote in plain English. The form is written in tax English. A literal search of the user's phrasing can sail right past the document that holds the answer, because the embedding of "non-citizen abroad" simply does not sit near the embedding of "nonresident alien individual."
A query transform is an LLM step that rewrites the user's question into one or more better search queries before retrieval runs. It is the recognition that the question is just your opening guess at what to search for, and the LLM can make better guesses. There are two common ways to do it, and we will look at both.
Multi-query: cast a wider net
Searching with several LLM-generated paraphrases of a query, plus the original, widens the net and catches documents a single phrasing would miss.
Multi-query is the transform where the LLM rewrites your one question into several different phrasings, you search with all of them, and you merge the results into a single pool. A paraphrase here is just one of those alternate phrasings — same meaning, different words.
The intuition is a wider net. Any single phrasing might miss the document. Four different phrasings, casting in slightly different directions, are far more likely to catch it between them. If "nonresident" doesn't land but "alien individual living abroad" does, that rewrite pulls in the 1040-NR chunk the original could never reach. You are directly attacking the pool problem the reranker couldn't touch — you are making the shortlist contain the right document in the first place.
Here is the fan-out, with our non-citizen-abroad question going in:
"which form does a non-citizen abroad file?"
│ LLM fans it out
▼
┌─ "nonresident alien filing requirements"
├─ "tax return for foreign individuals living abroad"
├─ "form 1040-NR who must file"
└─ "US tax form for non-citizens with US income"
│ search each, fuse the hits
▼
one combined shortlist (wider, more diverse)
The code that does the fan-out is small. The LLM is asked for n alternative phrasings, one per line, and the original question is always prepended so the paraphrases can't completely take over the merge:
def _multi_query(self, q: str) -> list[str]:
raw = chat(_MULTI_SYS.format(n=self.n_multi), q)
paraphrases = [line.strip() for line in raw.splitlines() if line.strip()]
# ALWAYS include the original — paraphrases shouldn't dominate the merge.
return [q] + paraphrases[: self.n_multi]
That [q] + paraphrases line is doing quiet work. The original question stays in the search set, so even if every rewrite drifts, the user's real intent is still being searched for directly.
HyDE: search with a fake answer
HyDE searches using an LLM's fabricated answer paragraph, because its vocabulary and style match real documents better than the bare question does.
The second transform is cleverer and a little strange. HyDE (Hypothetical Document Embeddings) does not rewrite the question at all. It asks the LLM to write a hypothetical answer to the question — a made-up paragraph of what the answer might say — and then you search with that paragraph.
Why on earth would searching with a fabricated answer help? Because a hypothetical answer lives in the same vocabulary and style as the real document. A terse question and a dense tax paragraph look quite different to an embedding. A plausible fake answer and the real answer look very similar. For the non-citizen-abroad case, the LLM might invent a paragraph full of "nonresident alien," "Form 1040-NR," and "effectively connected income" — exactly the tax English the real document uses. The fake answer is a better-shaped probe into meaning-space than the bare question ever was, even when the fake answer gets its facts slightly wrong.
The prompt that drives it is unapologetic about this — it explicitly tells the model the paragraph does not need to be correct, because it exists only to steer retrieval. The honesty there is deliberate: HyDE is trading factual accuracy of the probe for vocabulary match, and that trade is the whole reason it works.
The one trick that keeps the experiment honest
Query rewriting only widens retrieval; reranking and generation always use the original question, so recall changes are attributable to retrieval alone.
Both transforms change only the query that goes into retrieval. They do not change what the reranker sees, and they do not change what the generator answers. This is the single most important detail in the whole chapter.
Look at the answer() flow. The query gets reshaped, retrieval runs once per query and unions the results, but then the reranker and the generator both go back to the original question:
def answer(self, question: str) -> dict:
# 1. RESHAPE the query (or queries) using the LLM.
queries = self._transform(question)
# 2. RETRIEVE for each; union by node id, keep the best score.
pool: dict[str, object] = {}
for q in queries:
for n in self.retriever.retrieve(q):
k = n.node.node_id
if k not in pool or (pool[k].score or 0) < (n.score or 0):
pool[k] = n
# 3. RERANK with the ORIGINAL question — that's the real user intent.
reranked = self.reranker.postprocess_nodes(
list(pool.values()), query_bundle=QueryBundle(query_str=question)
)
# 4. GENERATE — answer the ORIGINAL question from the reranked context.
The reranker scores the merged pool against the question the user actually typed, not against any paraphrase or hypothetical. The generator answers the user's real question, never the transform. The transforms exist purely to widen the net during retrieval, and then you snap straight back to the user's true intent. That separation is what makes the recall delta attributable to retrieval alone — if recall moves, you know it moved because the right documents reached the pool, not because you quietly changed the question being graded.
What we measured
Multi-query rewriting raised recall without hurting faithfulness, but more than doubled latency; HyDE matched recall while damaging faithfulness.
Both transforms run on top of the same hybrid + reranker stack from the previous chapter. The only thing that changes is the query transform. Same IRS corpus, same questions, same generator.
| config |
Recall |
Precision |
Faithfulness |
latency |
| rerank only |
0.73 |
0.851 |
0.909 |
2.56s |
| + multi-query |
0.78 |
0.852 |
0.909 |
6.50s |
| + HyDE |
0.78 |
0.849 |
0.815 |
4.12s |
Read that table slowly, because every column is telling you something.
Multi-query moved recall 0.73 → 0.78. That is the first recall gain in two chapters. The wider net worked exactly as the intuition predicted: more phrasings reached more of the right documents, so the shortlist finally started containing the chunks we had been missing — the 1040-NR kind of chunk that the user's plain English could never reach. Faithfulness held at 0.909. Precision held. This is a clean, real gain on the one metric that had been stuck.
Now look at the latency column for multi-query: 2.56s → 6.50s. It more than doubled. Of course it did. Every query now triggers an LLM call to generate the rewrites, and then runs retrieval four times instead of once. That cost is not a one-time setup expense. It is paid on every single request, forever. You did not buy this recall gain. You are renting it, and the rent comes due on every call your users ever make.
HyDE matched the recall at 0.78 but quietly damaged faithfulness, 0.909 → 0.815. This is the "hurts" in the title, and it is instructive. HyDE's probe is, by construction, a made-up answer. When that fabricated text leaks into the retrieved context and the model leans on it, grounding suffers — the system starts trusting its own guess instead of the document. On a tax assistant, where a confidently wrong answer about which form a non-citizen files is worse than no answer at all, trading away faithfulness for recall is a bad deal. HyDE was out.
A good fix can still be the wrong one
Multi-query improved recall but cost an LLM rewrite plus four retrievals on every query forever, until contextual retrieval matched it at a fraction of the latency.
Multi-query was the right idea for this moment. It was the only lever we had to widen the pool, and it delivered a genuine recall gain that nothing else had managed. You should reach for it when your right documents exist but your phrasing can't find them.
Step back and look at what it costs, though. An LLM rewrite plus four retrievals, on every query, indefinitely. The gain is real, but the bill never stops arriving. That framing is the whole setup for the next chapter, because there is another way to make sure the right document is in the pool.
Instead of reshaping the question every time someone asks, you can reshape the documents once, when you index them. You pay the cost a single time at build, not forever at query. When contextual retrieval did exactly that — matching this recall at a fraction of the latency — we removed query rewriting entirely. A good idea, retired by a better one. That is exactly what a scorecard is for.
Measured output
Multi-query and HyDE both raise recall and hit-rate, but multi-query roughly triples latency while HyDE lowers faithfulness.
The exact RAGAS scorecards this chapter's run printed — 50 golden questions over the full IRS corpus, gpt-5.4-mini judge + text-embedding-3-small embeddings. Reproduce them with the companion Colab notebook (it inlines the exact pipeline code and embeds these JSON scorecards).
📊 RAGAS scorecard — 05 · rerank only
Context Recall 0.730 did we fetch the needed facts?
Context Precision 0.851 relevant context ranked high?
Faithfulness 0.909 grounded, no hallucination?
Answer Relevancy 0.643 answers the actual question?
Retrieval hit-rate 0.600 ≥1 expected form retrieved
Avg latency 2.56s per question, end-to-end
📊 RAGAS scorecard — 05 · + multi-query
Context Recall 0.780 did we fetch the needed facts?
Context Precision 0.852 relevant context ranked high?
Faithfulness 0.909 grounded, no hallucination?
Answer Relevancy 0.665 answers the actual question?
Retrieval hit-rate 0.660 ≥1 expected form retrieved
Avg latency 6.50s per question, end-to-end
📊 RAGAS scorecard — 05 · + HyDE
Context Recall 0.780 did we fetch the needed facts?
Context Precision 0.849 relevant context ranked high?
Faithfulness 0.815 grounded, no hallucination?
Answer Relevancy 0.642 answers the actual question?
Retrieval hit-rate 0.620 ≥1 expected form retrieved
Avg latency 4.12s per question, end-to-end
Summary
Query rewriting can raise recall but costs latency or faithfulness, so the trade must fit the use case.
You stop treating the user's exact words as the search and treat them as a first draft. Multi-query fans the question into several phrasings and casts a wider net. HyDE searches with a hypothetical answer that is shaped more like the target document. Both can pull in documents a literal search misses, and both make the LLM work before retrieval even starts, so you pay in latency on every call.
- Reranking is capped by what retrieval pulls; when the right document isn't in the pool, you have to fix the question, not the ranking.
- Multi-query lifted recall 0.73 → 0.78 with faithfulness and precision held — a real, clean win on the IRS corpus.
- That win cost latency 2.56s → 6.50s, paid on every request forever. You rent the gain; you don't buy it.
- HyDE matched the recall but dropped faithfulness 0.909 → 0.815, because a fabricated probe can leak into the context. Bad trade for a tax assistant.
- The reranker and the generator always see the ORIGINAL question. Only retrieval is shifted, which is what keeps the recall delta honest.
What's next
Contextual retrieval rewrites chunks once at index time, matching multi-query recall at far lower ongoing latency cost.
Multi-query fixes the question every single time someone asks one. The next chapter asks the obvious follow-up: what if you fix the documents once instead? In contextual retrieval, we rewrite each chunk at index time so it carries its own context — so the 1040-NR chunk already reads like the answer to "non-citizen living abroad" before any user types a thing. Pay once at build, not forever at query. It matched this recall at a fraction of the latency, and it is why query rewriting did not survive the scorecard.
The short version
Adding a one-line LLM-written header to each chunk before embedding fixes lost context, cheaply, once at index time.
Contextual retrieval is a technique where, before you embed a chunk, you ask the LLM to write a one-line "you are here" note about that chunk and glue it to the front. A chunk is a small slice of a document, and an embedding is the numeric fingerprint of meaning you search against. The note tells search where the chunk came from, so the chunk can finally describe itself.
Why does this matter so much? Because it fixes the deepest weakness in your whole pipeline — chunks that forgot which document they belong to — and it fixes it once, when you build the index, instead of forever, on every query. On the IRS corpus, this single change beat the expensive per-query trick from the last chapter on every metric, and it was cheaper. This was the biggest win in the series.
In this post, we will cover the following:
- Why a chunk loses its identity the moment you slice it
- The orphaned
f1040nr chunk, and why search sails right past it
- The fix: a one-line header, written by the LLM, glued on before you embed
- Why you pay for this once at index time, not forever per query
- The one honest compromise: per-document context vs per-chunk context
- The code: one LLM call per document, returning a JSON array of contexts
- The key move: context that is embedded with the chunk but hidden from the generator
- The results, with exact numbers — and why precision and faithfulness leapt
- The satisfying ending where we delete a whole component
A chunk with amnesia
Slicing documents into chunks strips away the parent context, so a chunk often no longer describes itself well enough to be found.
Go back to how chunking works. You slice a document into pieces and embed each piece on its own. That slicing has a hidden cost you have been paying the whole time: a chunk forgets which document it came from. An orphaned chunk is a piece of text that no longer carries any reference to its parent — its form, its section, its topic are all gone.
Picture a chunk pulled from the middle of the nonresident return, f1040nr:
"Line 8: enter the amount from Schedule NEC, line 15…"
Read that cold. Which form is it from? You cannot tell. Is it about nonresidents? No way to know. The one fact that would make this chunk findable — that it belongs to Form 1040-NR — lived in the document's title and header, three pages up, and the slicing threw it away.
When you embed this chunk, you embed an orphan. A user asking "what does a nonresident file?" sails right past it, because nothing in the chunk's text says "nonresident." This is the same villain from every chapter, seen from a new angle. It is not that search is dumb. You handed search a chunk that does not describe itself.
The fix: give every chunk a header before you embed it
Prepending a one-line summary of a chunk's context before embedding sharply cuts retrieval failures, especially combined with keyword search and reranking.
The idea is almost embarrassingly simple once you see the problem. Before you embed a chunk, show it to the LLM along with its parent document and ask one question: in one line, where does this chunk live and what is it about? Then you prepend that line to the chunk.
Here is the orphan before and after:
BEFORE (what we used to embed):
"Line 8: enter the amount from Schedule NEC, line 15…"
AFTER (what we embed now):
"Form 1040-NR, U.S. Nonresident Alien Income Tax Return — this part
covers reporting income not effectively connected with a US trade.
Line 8: enter the amount from Schedule NEC, line 15…"
That is it. The orphan now carries its own identity. Its embedding lands near "nonresident return" in meaning-space, and a keyword search can finally match "1040-NR" because those characters literally appear in the text now. The bare, invisible line became a self-describing, findable fact — and you never touched the user's question to do it.
This is not a brand-new idea you have to take on faith. Anthropic published the technique in September 2024 and reported how much it cut the retrieval failure rate versus naive RAG: contextual embeddings alone gave a -35% reduction, contextual embeddings plus BM25 (the keyword half of hybrid search, which you already run) gave -49%, and adding the reranker on top gave -67%. Those are big, real numbers from a controlled study, and they map cleanly onto the stack you have already built.
Pay once, not forever
Contextual retrieval fixes retrieval failures at index time, paid once, instead of at query time, paid on every request.
Here is the part to really absorb, because it is the reason this chapter beats the last one.
The previous chapter fixed the same "the right document never made it into the pool" problem by rewriting the question — and that cost an LLM call plus four retrievals on every query, forever. Index time is when you build your search index, once, before any user shows up. Query time is when a user asks something, and you pay for whatever you do there on every single request.
Contextual retrieval fixes the orphan problem at index time. It rewrites the documents, once, when you build the index. After that, every query is cheap again. You move the LLM work from query time, where you pay endlessly, to index time, where you pay once. Same problem, fundamentally cheaper place to solve it. When you can choose where to spend a cost, prefer the place you pay it once.
One honest compromise
Using one situating summary per document instead of per chunk cuts API calls by about 27x while still capturing most of the benefit on form-identity corpora.
I want to be straight with you about a trade we made. The original technique, as Anthropic published it, situates every chunk with its own LLM call. On the IRS corpus that is 38,000 calls — and the first time we tried it, that saturated the rate limit and drained the API budget mid-run. So we use the scalable variant: one situating summary per document, about 1,400 calls, roughly 27x fewer, prepended to all of that document's chunks.
Is that cheating? A little — but it is an honest, deliberate trade. On a corpus where the core problem is document identity ("which form is this?"), a per-document summary captures most of the benefit, because every chunk of f1040nr wants the same header anyway: this is the nonresident return.
When would you reach for the full per-chunk version instead? When your chunks vary wildly within a single document — a long, heterogeneous report where page 2 is about revenue and page 80 is about litigation risk. There, one summary cannot place every chunk, and you would pay for the per-chunk calls because they earn their keep. Know your corpus, then pick the variant that fits it.
The code: one LLM call per document
Sending a document with all its chunks in a single LLM call generates per-chunk context cheaply by amortizing the cost across the whole document.
Let me show you how cheap this actually is in code. For each document, you send the whole document plus all of its chunks in a single call, and you ask the model to return a JSON array with one short context per chunk, in order.
def _gen_contexts_for_doc(doc_text: str, chunks: list[str]) -> list[str]:
n = len(chunks)
chunk_block = "\n\n".join(f"CHUNK {i + 1}: {c[:400]}" for i, c in enumerate(chunks))
user = (
f"DOCUMENT:\n{doc_text}\n\n"
f"CHUNKS ({n} total):\n{chunk_block}\n\n"
f"For EACH chunk, write ONE short sentence (max 25 words) that places it "
f"within the document — what form, what section, what topic. Return a JSON "
f"array of EXACTLY {n} strings in the same order as the chunks."
)
raw = chat(_CONTEXT_SYS, user, temperature=0.0).strip()
arr = json.loads(raw)
return [str(x).strip() for x in arr]
One call, one document, N contexts back. That is the whole reason the per-document variant is 27x cheaper than per-chunk: you amortize a document's worth of chunks over a single round trip. For very long documents you truncate to the first 16k tokens of context — enough to place the chunks, not enough to bill you for a novel.
The key move: embed it, but hide it from the generator
Synthetic context should boost embedding-based retrieval but stay hidden from the generator so answers stay grounded in the original text.
Now the subtle part, and the one most people get wrong. The context you generated is meant for the embedding model, not for the model that writes the final answer. If the generator saw your synthetic "this is the nonresident return" line, it might quote it or drift onto LLM-invented context that was never in the real document.
LlamaIndex lets you control this per metadata key on a TextNode. You put the context in metadata, then you exclude it from the LLM's view while including it in the embedded text.
TextNode(
text=c.get_content(),
metadata={"doc_id": doc_id, "source": source, "context": ctx},
# KEY MOVE: context is EMBEDDED with the chunk but HIDDEN from the LLM.
excluded_embed_metadata_keys=["doc_id", "source"],
excluded_llm_metadata_keys=["context", "source"],
)
Read those last two lines slowly. The context key is absent from excluded_embed_metadata_keys, so it goes into the embedding — that is what lifts retrieval. The context key is present in excluded_llm_metadata_keys, so the generator never sees it — that is what keeps your answers grounded in the real text. The chunk you search for is augmented; the chunk you answer from is the clean original. You get the recall lift without paying for it in hallucinations.
What it did — the cheap fix wins outright
Contextual retrieval alone beat multi-query on recall, precision, and faithfulness while running more than twice as fast.
We ran contextual retrieval on the same hybrid plus reranker stack from earlier chapters, and we also tried stacking the previous chapter's multi-query on top of it, to see if they compound.
| config |
Recall |
Precision |
Faithfulness |
latency |
| rerank + multi-query (Ch 5) |
0.78 |
0.852 |
0.909 |
6.50s |
| contextual (Ch 6) |
0.83 |
0.945 |
0.968 |
2.56s |
| contextual + multi-query |
0.83 |
0.923 |
0.956 |
5.54s |
Read the middle row. Contextual retrieval, by itself, beat last chapter's multi-query on recall (0.83 vs 0.78), precision (0.945 vs 0.852), and faithfulness (0.968 vs 0.909) — at 2.56 seconds instead of 6.50. Cheaper and better on every axis.
Why precision and faithfulness leapt, not just recall
Self-describing chunks let the reranker separate look-alike documents, so top-4 context is cleaner and the model hallucinates less.
The recall gain you can predict — self-describing chunks are easier to find. But look at how hard precision (0.85 to 0.95) and faithfulness (0.91 to 0.97) jumped. Those are the biggest single-chapter gains in the series, and they come from a nice second-order effect.
Remember the whole saga of f1040 versus i1040gi — the form versus its look-alike instruction booklet. Once each chunk announces its own identity in its text, the reranker can finally tell them apart cleanly, because the distinguishing words are right there for it to read. So the top-4 it produces is less polluted with near-duplicates, which means the model sees cleaner context, which means it hallucinates less.
Fixing the chunks did not just help search find them — it made every downstream layer sharper. The same logic explains the f1040nr orphan: once its chunk says "nonresident return" out loud, it stops competing with the resident f1040 chunks that used to crowd it out. Good context is upstream of everything.
When the right move is to remove a component
Adding multi-query on top of contextual retrieval didn't help recall and hurt precision, faithfulness, and latency, so it was removed.
The bottom row of the table is the quiet hero. Stacking multi-query on top of contextual did not help — same recall, and precision, faithfulness, and answer relevancy all slipped, while latency climbed back to 5.54s. Contextual had already cleaned up the pool, so the extra rewrites just dragged borderline chunks back in.
So we deleted multi-query from the pipeline. Think about how satisfying that is: the system got simpler and better at the same time. We retired a whole component — and its permanent per-query cost — because a cheaper idea did its job better. That is the best outcome a chapter can have, and you only ever find it by measuring whether your shiny components still earn their place.
Measured output
Contextual retrieval alone beats multi-query on quality and speed, while combining it with multi-query only boosts hit-rate further.
The exact RAGAS scorecards this chapter's run printed — 50 golden questions over the full IRS corpus, gpt-5.4-mini judge + text-embedding-3-small embeddings. Reproduce them with the companion Colab notebook (it inlines the exact pipeline code and embeds these JSON scorecards).
📊 RAGAS scorecard — 06 · multi-query (Ch5)
Context Recall 0.780 did we fetch the needed facts?
Context Precision 0.852 relevant context ranked high?
Faithfulness 0.909 grounded, no hallucination?
Answer Relevancy 0.665 answers the actual question?
Retrieval hit-rate 0.660 ≥1 expected form retrieved
Avg latency 6.50s per question, end-to-end
📊 RAGAS scorecard — 06 · contextual
Context Recall 0.830 did we fetch the needed facts?
Context Precision 0.945 relevant context ranked high?
Faithfulness 0.968 grounded, no hallucination?
Answer Relevancy 0.713 answers the actual question?
Retrieval hit-rate 0.680 ≥1 expected form retrieved
Avg latency 2.56s per question, end-to-end
📊 RAGAS scorecard — 06 · contextual+multi
Context Recall 0.830 did we fetch the needed facts?
Context Precision 0.923 relevant context ranked high?
Faithfulness 0.956 grounded, no hallucination?
Answer Relevancy 0.688 answers the actual question?
Retrieval hit-rate 0.720 ≥1 expected form retrieved
Avg latency 5.54s per question, end-to-end
Summary
Prepending a one-line LLM-written context to each chunk before embedding fixes orphaned chunks and sharply improves retrieval accuracy at a one-time indexing cost.
You hand each chunk a one-line header that says where it lives and what it is about, written by the LLM, glued on before you embed. The orphaned f1040nr chunk becomes self-describing, so search finds it and the reranker stops confusing it with its resident-form neighbors — and because you do this once at index time, every query stays cheap forever.
- A chunk forgets its document the moment you slice it; an orphan like
f1040nr's "Line 8…" is unfindable because nothing in it says "nonresident."
- The fix is a one-line LLM-written context, prepended before embedding — Anthropic reports -35% / -49% / -67% failure-rate reductions as you stack it with hybrid and reranking.
- You pay for it once at index time, not forever per query — that is why it beats query rewriting.
- One LLM call per document returns a JSON array of contexts: ~1,400 calls instead of 38,000, an honest compromise that fits an identity-driven corpus.
- Embed the context with the chunk but hide it from the generator, via
excluded_llm_metadata_keys, so recall lifts without hurting grounding.
- Contextual alone hit 0.83 recall, 0.945 precision, 0.968 faithfulness at 2.56s — better and cheaper than multi-query — so we deleted multi-query.
What's next
The next step is building a semantic cache so near-duplicate queries reuse past answers instead of repeating the full, costly LLM pipeline.
Accuracy is, for our purposes, solved. The harder question now is economics: you have a pipeline that retrieves well, but you are still paying for an LLM call on every query, and many of those queries are near-duplicates of ones you have already answered. The next move is not about finding better chunks — it is about not doing the same expensive work twice. We will build a semantic cache that recognizes when a new question means the same thing as an old one, and serves the cached answer instead of paying for the whole pipeline again.
The short version
A semantic cache returns stored answers for meaning-equivalent questions, cutting repeated-query latency sharply without changing accuracy.
A semantic cache remembers the questions your system has already answered, and when a new question means the same thing as an old one, it hands back the stored answer instead of running your whole pipeline again. A cache is just a fast lookup table that returns a saved result instead of recomputing it.
Here is why you should care. Every chapter before this one made your answers better — better chunks, better retrieval, better reranking. None of them made answering cheaper. In the real world, questions repeat constantly, and recomputing the same answer four hundred times a day is pure waste. A semantic cache cut our average latency by 92% on repeated questions, and it does that without touching accuracy. It also hides one knife-edge setting that this entire post is really about.
In this post, we will cover the following:
- The waste a normal RAG pipeline pays on every repeated question
- Why an exact-text cache barely helps real users
- How a semantic cache matches on meaning instead of spelling
- The threshold — the single number that decides everything — and the code behind it
- The collision trap: how Form 1040 vs 1040-NR can poison your cache
- The measured cost and quality numbers, with exact figures
- What the cache changes (economics) and what it deliberately does not (accuracy)
- How to take this to production
The waste we are cleaning up
Without caching, every repeated or reworded question pays the full embed-search-rerank-LLM cost again, which is wasteful at real traffic.
Your pipeline is good by now, but it is not cheap. Every single question runs the full machine: embed the query, search 38,000 chunks two ways, run a slow reranker over the shortlist, then call the LLM to write an answer. That is fine to pay once. An embedding turns a piece of text into a vector — a list of numbers — so that text with similar meaning lands at nearby points.
The problem is that questions repeat. Ten different users ask the same thing this morning. The same person rephrases and asks again. A support bot gets "how do I file as a nonresident?" four hundred times a day. For every one of those, you pay the full cost again to produce an answer you already produced. At any real traffic, that waste is enormous.
This is the IRS corpus we have been using all along, and the questions users ask of it are deeply repetitive. The same handful of forms — 1040, 1040-NR, Schedule C — come up again and again, worded a hundred different ways. That repetition is exactly the opportunity a cache exists to capture.
Why a normal cache is not enough
A normal cache only matches identical text, so it almost never hits because people rarely phrase questions the same way.
The obvious fix is a cache: store each question with its answer, and when a question comes in, check whether you have seen it. A plain cache matches exact text, though, and that barely helps here. Look at these two:
"What is Form 1040 used for?"
"What's the 1040 form used for?"
Same question. Zero characters that a string match would treat as equal — the words, punctuation, and order all differ. A normal cache sees two completely different keys and misses. People almost never phrase things identically, so an exact cache hits almost never.
What you need is a cache that recognizes meaning, not spelling.
The mechanism: match on meaning
A semantic cache compares a new question's embedding to stored questions via cosine similarity and returns the cached answer if the best match clears a threshold.
You already have the perfect tool for "do these two pieces of text mean the same thing" — embeddings. So a semantic cache works like this. Store a vector for every question you have answered. When a new question arrives, embed it and find the closest stored question by cosine similarity. Cosine similarity is a score from -1 to 1 that measures the angle between two vectors — close to 1 means the meanings line up.
new question ──embed──► compare (cosine) against every cached question
│
best match ≥ threshold? ──yes──► return its stored answer (~0.2s, $0)
│
no ────────────────► run the full pipeline, store it (~2.6s)
If the closest stored question is close enough, you have seen this question before in different clothes — return its answer. If nothing is close enough, run the full pipeline and add this one to the cache for next time. A lookup is a single dot-product against your stored vectors — microseconds, against the seconds a full pipeline run takes.
Here is the heart of the lookup. The cached questions live in one matrix mat, every row L2-normalized, so a plain dot-product against the normalized query embedding is the cosine similarity. You take the best match and compare it to the threshold:
def lookup(self, q_emb, threshold):
if self._mat is None or len(self._mat) == 0:
return 0.0, None
sims = self._mat @ q_emb # both sides normalized → dot == cosine
i = int(sims.argmax())
best = float(sims[i])
if best < threshold:
return best, None # nothing close enough → miss
return best, self._cache.get(self._keys[i])
The whole pipeline then sits behind one answer method. You embed the query, look it up, and on a hit you return the stored answer immediately. On a miss you run the real base pipeline and store the result so the next caller gets the fast path:
def answer(self, question):
q_emb = self._embed_query(question)
sim, entry = self.cache.lookup(q_emb, self.threshold)
if entry is not None: # HIT — stored answer, no LLM
out = dict(entry["answer_dict"])
out["cache_hit"] = True
return out
out = self.base.answer(question) # MISS — full pipeline, then store
self.cache.store(question, q_emb, out)
out["cache_hit"] = False
return out
Notice there is no special intelligence in either snippet. The cache is dumb on purpose. All of its behavior — when it helps and when it hurts — comes down to that one comparison, best < threshold.
The one number that decides everything: the threshold
The cosine similarity threshold determines whether a semantic cache saves work or silently returns wrong answers.
That phrase "close enough" is doing all the work, and it hides the entire difficulty of semantic caching. The threshold is the minimum cosine similarity a new question must reach before the cache will reuse an old answer. It is a knife-edge.
Set it too high and real paraphrases never clear the bar — the cache almost never fires, and you have saved almost nothing. Set it too low and two genuinely different questions get treated as the same, and you confidently serve the wrong answer. The whole skill is finding the line between "the same question reworded" and "a different question that happens to sound similar." That line lives in your own data, so you have to measure it.
You warm the cache with your 50 golden questions, then ask 50 reworded versions and sweep the threshold:
| threshold |
hit-rate |
correct hits |
wrong hits |
| 0.80 |
1.00 |
50 |
0 |
| 0.84 |
0.98 |
49 |
0 |
| 0.88 |
0.90 |
45 |
0 |
| 0.92 |
0.66 |
33 |
0 |
As the threshold climbs, coverage falls — fewer paraphrases clear the bar. Across the board, zero wrong answers. So on this set, 0.80 looks free: every reworded question is recognized, none misfires.
The trap hiding in "zero wrong"
A clean 'zero wrong' cache result only holds for a question set with no near-duplicate topics; real data can collide at high similarity.
You have to be careful here, because that clean result is a property of this question set, not of semantic caching in general.
Your 50 golden questions cover distinctly different topics, so each paraphrase maps cleanly back to its own original — no two are close enough to collide. A real production stream is not so polite. Watch the IRS villain do its damage one more time, now to the cache:
Warm the cache with: "Who must file Form 1040-NR?"
Then ask: "Who must file Form 1040?"
Those are different questions — nonresident versus resident — but to an embedding they are nearly identical (cosine ≈ 0.76, because they share almost every word). Set your threshold below that, chasing more cache hits, and the cache will cheerfully serve the 1040-NR answer to the 1040 question. Confidently. Wrongly. The same form-versus-form confusion that haunted retrieval now haunts the cache.
The lesson is not "0.80 is safe." The lesson is "measure where your own collision line sits, and stay above it." On a corpus where every question is about a wholly separate topic, a low threshold is harmless. On the IRS corpus, where 1040 and 1040-NR live a cosine hair apart, a low threshold is a loaded gun. Your data decides, not a blog post.
Quality and cost, measured
The semantic cache cut average latency by 92% while keeping faithfulness flat and even improving recall, at a small cost to answer relevancy.
At the chosen threshold, answering 50 paraphrases without the cache versus with it:
| metric |
no-cache |
cache |
| Context Recall |
0.76 |
0.85 |
| Faithfulness |
0.917 |
0.920 |
| Answer Relevancy |
0.703 |
0.679 |
| avg latency |
2.62s |
0.21s |
Average latency collapsed from 2.62s to 0.21s — a 92% drop — and faithfulness held essentially flat at ~0.92. Recall even ticked up, from 0.76 to 0.85, which has a tidy explanation: a cache hit returns the answer computed for the original question, whose cleaner phrasing retrieved slightly better than the paraphrase would have. Serving the canonical answer can beat re-answering a reworded one from scratch. The only real cost was a hair of answer relevancy, 0.703 down to 0.679, because the cached answer was written for the original wording, not the new phrasing.
What it does not do, on purpose
A semantic cache only speeds up and cheapens repeated answers; it never improves answer accuracy, which must be fixed first.
Notice what the cache did not change: the core accuracy of a fresh answer. It cannot. A cache never makes an answer better — it can only hand back a good answer faster. Its job is not quality, it is economics. Those accuracy numbers were set by contextual retrieval and the chapters before it; the cache just stops you from paying for them twice.
That is the right way to slot it into your mental model. The accuracy chapters built a good answer. The cache makes serving that answer, again and again, nearly free. If your answers are wrong, a cache will faithfully serve wrong answers faster — fix accuracy first, then cache.
Taking it to production
Moving a semantic cache to production means swapping storage (diskcache to Redis) and tuning the similarity threshold against measured eval results, not copying defaults.
The cache also has a notion of freshness so it never serves a stale answer forever. TTL — time to live — is the number of seconds a cached entry stays valid before it expires and gets recomputed. Tax guidance changes; a one-hour TTL means a popular question gets re-answered against the live pipeline at least once an hour, while still absorbing the flood of repeats in between.
Locally you can back the whole thing with diskcache — zero infrastructure, works on any laptop, persists across runs. For production you swap that one storage class for Redis + RedisVL and keep the exact same lookup and store contract. Nothing about the matching logic changes; you just move from a file on one machine to a shared, networked store every replica can read.
For the threshold itself, start around 0.90, raise it to 0.92, and watch your eval scorecard the whole way. The numbers above are why: 0.92 still recovered two-thirds of paraphrases with zero wrong hits, and a slightly higher floor keeps you comfortably above collision pairs like 1040 vs 1040-NR. Treat the threshold as a dial you tune against measured results, never a constant you copy from someone else's corpus.
Measured output
A warm semantic cache cut average latency from 2.62s to 0.21s per question while slightly improving context recall and hit-rate.
The exact RAGAS scorecards this chapter's run printed — 50 golden questions over the full IRS corpus, gpt-5.4-mini judge + text-embedding-3-small embeddings. Reproduce them with the companion Colab notebook (it inlines the exact pipeline code and embeds these JSON scorecards).
📊 RAGAS scorecard — 07 · no cache
Context Recall 0.760 did we fetch the needed facts?
Context Precision 0.918 relevant context ranked high?
Faithfulness 0.917 grounded, no hallucination?
Answer Relevancy 0.703 answers the actual question?
Retrieval hit-rate 0.640 ≥1 expected form retrieved
Avg latency 2.62s per question, end-to-end
📊 RAGAS scorecard — 07 · cache (warm)
Context Recall 0.850 did we fetch the needed facts?
Context Precision 0.912 relevant context ranked high?
Faithfulness 0.920 grounded, no hallucination?
Answer Relevancy 0.679 answers the actual question?
Retrieval hit-rate 0.680 ≥1 expected form retrieved
Avg latency 0.21s per question, end-to-end
Summary
A semantic cache matches question meaning, not exact wording, to reuse past answers cheaply, with a single similarity threshold controlling the coverage-versus-wrong-answer tradeoff.
A semantic cache stores a vector for every question you have answered, and when a new question's meaning lands close enough to a stored one, it returns the old answer instead of recomputing it. It matches on meaning, not spelling, so it catches the paraphrases a plain cache misses — and it lives or dies by a single threshold that trades coverage against the risk of serving a similar-but-different question the wrong answer.
- A semantic cache changes your economics, not your accuracy — same answers, served far cheaper.
- It cut average latency from 2.62s to 0.21s (~92%) while faithfulness held at ~0.92 and recall even rose 0.76 → 0.85.
- The threshold is the whole game: too high saves nothing, too low serves wrong answers.
- Measure your collision line on your own data — on the IRS corpus, 1040 vs 1040-NR sit at cosine ≈ 0.76.
- Use TTL so popular answers refresh; start the threshold near 0.90, raise to 0.92, watch the scorecard.
- Back it with diskcache locally and swap to Redis + RedisVL in production behind the same contract.
What's next
The next steps are hardening the pipeline for production and then enabling multi-step agentic retrieval.
You now have a pipeline that is both accurate and cheap to serve. The remaining arcs are about making it robust. Production hardening comes first — handling failures, monitoring drift, guarding against the cache and retrieval collisions you just met at real scale. After that comes agentic RAG, where the system stops answering one question at a time and starts planning multi-step retrieval, deciding for itself when to search, when to reuse, and when to ask again.