Measuring a RAG system
The short version
Context Recall measures how many required facts retrieval actually fetched, and it caps everything else the system can achieve.
Of all the facts the right answer needs, how many did retrieval actually fetch? Recall is the ceiling everything else sits under.
💡 In one line: don't miss the good stuff.
Context Recall answers one blunt question: of the facts a correct answer requires, how many actually showed up in the chunks you retrieved? It is the first of the two retrieval metrics in the RAG scorecard, and it is the most important number in the whole pipeline — because a model cannot use a fact it never saw.
That is why we call recall the ceiling. Reranking, prompting, a bigger model — none of it can recover information that retrieval left on the shelf. If recall is 0.6, then 40% of the time the answer the user needs is simply not in the room, and no downstream cleverness invents it.
The intuition: "is it in the pile?"
Context recall only checks whether the needed fact entered the retrieved pile at all, not where it sits.
Picture retrieval dumping a pile of chunks on the desk. Recall asks only one thing: is the needed fact somewhere in that pile? It does not care whether the fact is on top or buried at the bottom — that is precision's job. Recall only cares whether it made it onto the desk at all.
On our IRS tax corpus, the classic recall failure is this: a user asks how to fill in a line on Form 1040, and the real explanation lives in the i1040gi instruction booklet — but retrieval only pulled the bare form. The fact never entered the pile, so recall is where that miss shows up.
How it's scored
Context Recall checks whether each claim in a human-written reference answer is supported by retrieved context, avoiding self-graded scoring.
RAGAS computes recall against a human-written reference answer, not the model's own opinion (the metric is LLMContextRecall). It breaks the reference into individual claims, then checks each claim: can it be attributed to the retrieved context?
Context Recall = (reference claims supported by retrieved context)
────────────────────────────────────────────────
(total claims in reference)
Scored 0..1, higher is better. Grading against the human ground truth is what stops this from being the model marking its own homework.
What good and bad look like
Context recall rises not from reranking but from retrieval fixes like query rewriting and contextual retrieval that widen the retrieved pool.
On our hard corpus, naive RAG starts at 0.650 recall — a third of the needed facts simply missing. The fixes that move recall are the ones that widen the pool: query rewriting casts a wider net by rephrasing the question, and contextual retrieval makes chunks easier to find by letting them describe themselves. By the end of the series recall climbs to 0.83.
Note what does not move it: a reranker only reorders the pile it was handed — if the fact was never retrieved, no reordering invents it. When recall is stuck, fix the retrieval, not the ranking.
The trap: recall vs precision
Raising recall by retrieving more chunks adds noise that collapses precision, so both metrics must be measured together.
You can always raise recall by retrieving more chunks — fetch 50 instead of 4 and you will catch more needed facts. But now the pile is full of noise, precision collapses, and the model drowns the right answer in junk. The two metrics pull against each other, which is exactly why you measure both. Recall is "is it in the pile?"; precision is "is it on top?".
Part of the Measurements series. Next: Context Precision — is the right chunk ranked high, or buried? See all four metrics working together in how to measure a RAG system.
The short version
Context precision checks whether the relevant retrieved chunks are ranked near the top, not just present in the retrieved set.
You fetched the right chunk — but is it near the top where the model reads, or buried under noise? That's precision.
💡 In one line: don't bring the junk.
Context Precision is the second retrieval metric in the RAG scorecard. Where recall asks whether the needed fact made it into the pile at all, precision asks the sharper follow-up: of the chunks you retrieved, how many are actually relevant — and are the relevant ones ranked high?
This matters because a language model only really reads the first few chunks you hand it. You can retrieve the perfect document and still fail if it sits at position 8, padded out by four near-duplicates the model reads first.
The intuition: "is it on top of the pile?"
Context precision measures whether the correct retrieved fact is ranked near the top of the pile, not merely present somewhere in it.
The clean way to hold the pair in your head:
- Recall — is the right fact in the pile?
- Precision — is it on top of the pile?
On the IRS corpus, low precision looks like this: you ask about Form 1040, retrieval does pull the right form, but it also drags in 1040-NR (the nonresident version) and three instruction booklets that all look almost identical. The right chunk is in there — it's just not on top, and the model reads the look-alikes first.
How it's scored
Context Precision checks each retrieved chunk against the reference answer and rewards relevant chunks for ranking near the top.
RAGAS judges precision against the human-written reference answer (LLMContextPrecisionWithReference). For each rank position it asks "is this chunk relevant?", then rewards relevant chunks that appear early more than relevant chunks that appear late:
Context Precision = weighted mean of precision@k over the ranking,
giving more credit to relevant chunks ranked higher
Scored 0..1, higher is better. A high score means the useful context sits at the top of the window, not the bottom.
What good and bad look like
Naive RAG scores 0.779 precision with many near-duplicate chunks, but reranking and contextual retrieval push precision up to 0.95.
Naive RAG starts at 0.779 precision on our corpus — decent, but with too many look-alikes near the top. The fix that moves precision most is the reranker: a slow, careful second reader that re-sorts a wide candidate pool so the genuinely relevant chunks rise. Contextual retrieval helps too, by making near-duplicate chunks easier to tell apart. By the end of the series precision reaches 0.95.
The trap: precision vs recall
Optimizing precision alone, without watching recall, can produce a system that looks right but silently fails on most questions.
Precision is easy to game in the wrong direction. Retrieve only one chunk and if it's relevant your precision is perfect — but you've almost certainly missed needed facts, so recall craters. The two trade off, which is the whole reason the scorecard tracks both. Tuning one blind to the other is how teams ship a system that's confidently right about a third of questions and silently wrong about the rest.
Part of the Measurements series. Next: Faithfulness — did the answer stick to the context, or make something up? See all four metrics in how to measure a RAG system.
The short version
Faithfulness checks whether every claim in an answer is actually supported by the retrieved context, catching invented content even when retrieval works perfectly.
Did the answer stick to what was retrieved, or did the model quietly invent some of it? Faithfulness is your hallucination detector.
💡 In one line: don't make it up.
Faithfulness is the first of the two generation metrics in the RAG scorecard. Retrieval can do its job perfectly — the right context lands on the desk — and the model can still make something up. Faithfulness catches exactly that: it checks whether every claim in the answer is actually supported by the retrieved context.
This is the single most important metric on a high-stakes assistant. A fluent, confident, completely made-up answer about someone's taxes is the worst failure a RAG system can produce — worse than saying "I don't know," because it's wrong in a way that looks right.
The intuition: "did it stay grounded?"
Faithfulness measures whether the model's answer stuck to the retrieved context instead of inventing details from memory.
Where recall and precision grade the retrieval stage, faithfulness grades what the model did with what it was handed. Imagine an open-book exam: faithfulness asks whether the student answered from the book, or wandered off and wrote from memory (and got it wrong).
On the IRS corpus, an unfaithful answer might confidently state a filing deadline or a dollar threshold that appears nowhere in the retrieved forms. The context was fine; the model embellished.
How it's scored
Faithfulness is scored by splitting an answer into claims and checking what fraction are supported by the retrieved context.
RAGAS (Faithfulness) works in two steps:
- Break the generated answer into individual factual claims.
- Check each claim: is it supported by the retrieved context?
Faithfulness = (answer claims supported by the retrieved context)
──────────────────────────────────────────────────
(total claims in the answer)
Scored 0..1, higher is better. A 0.9 means nine of every ten claims trace back to the context; the rest are the model going off-script.
What good and bad look like
Faithfulness improves mainly through cleaner, less confusing context, not through retrieving more documents.
Naive RAG starts at 0.751 faithfulness on our corpus — one claim in four not grounded. What lifts it is cleaner, less confusing context: the reranker removes look-alike booklets that tempt the model into mixing up forms, and contextual retrieval makes each chunk unambiguous about which form it belongs to. By the end of the series faithfulness reaches 0.97 — near-zero hallucination.
Notice the pattern: faithfulness is moved by what tidies the generation stage, not by what widens the pool. A change that lifts recall but leaves faithfulness flat is telling you it improved what was found, not how it was used.
The trap: faithful but useless
A faithful answer can still fail to actually answer the question, so faithfulness and answer relevancy must both be measured.
An answer can be perfectly faithful and still bad. "I cannot determine that from the provided context" is 100% grounded and often correct to say — but if the context did contain the answer, that's a failure of a different metric: answer relevancy. Faithfulness guards against making things up; it says nothing about whether you actually answered the question. You need both.
Part of the Measurements series. Next: Answer Relevancy — did it answer the actual question? See all four metrics in how to measure a RAG system.
The short version
Answer Relevancy checks that a response actually addresses the question asked, not just that it's truthful.
A response can be perfectly grounded in real context and still answer the wrong question. Relevancy catches the helpful-sounding miss.
💡 In one line: answer the actual question.
Answer Relevancy is the second generation metric in the RAG scorecard. Faithfulness makes sure the answer didn't invent anything; relevancy makes sure the answer is actually about the question you asked — complete, on-topic, and not padded with detours.
It catches the answer that is technically true but unhelpful: the kind that recites what Form 1040-NR is when you asked about Form 1040, or that buries the one line you needed under three paragraphs you didn't.
The intuition: "did it answer my question?"
A faithful answer can still score low on relevancy if it is off-topic, incomplete, or padded with filler.
Three things can go wrong even with a faithful answer:
- Off-topic — it drifts to a neighbouring question.
- Incomplete — it answers half of what you asked.
- Padded — it's stuffed with hedging and filler that dilutes the point.
Relevancy is low for all three. On the IRS corpus, a low-relevancy answer might correctly describe a form's purpose when the user clearly asked how to fill in line 12 — grounded, but beside the point.
How it's scored
RAGAS scores answer relevancy by having an LLM reverse-generate questions from the answer and comparing them to the original question via cosine similarity.
RAGAS (ResponseRelevancy) scores this with a clever reverse trick: it asks an LLM to generate several questions that the answer would be a good response to, then measures how close those reconstructed questions are to the original question using cosine similarity (how close two pieces of text point in meaning — 1.0 identical, 0 unrelated).
Answer Relevancy = average cosine similarity between the original question
and the questions an LLM reverse-generates from the answer
Scored 0..1, higher is better. The logic: if the answer truly addresses your question, an LLM reading only the answer should be able to guess your question back. If it can't, the answer wandered.
What good and bad look like
Naive RAG scores lowest on answer relevancy because weak context makes the model hedge or miss the question.
Naive RAG starts at the lowest score on the whole scorecard — 0.584 answer relevancy — because with weak, noisy context the model often hedges or answers a near-miss question. As the retrieval stages improve the context (hybrid search, reranking, contextual retrieval), the model has cleaner material to work from and its answers land on-target far more often.
This is the metric that most reveals a quiet truth of RAG: fixing retrieval fixes generation. Better context doesn't just stop hallucinations — it lets the model answer the actual question.
The trap: relevancy without faithfulness
An answer can be on-topic but made up, or accurate but off-topic, so relevancy and faithfulness must both be measured together.
Relevancy and faithfulness are a pair, and you need both. An answer can be perfectly on-topic and entirely made up (high relevancy, low faithfulness), or perfectly grounded and beside the point (high faithfulness, low relevancy). Either alone ships a bad assistant. The full scorecard tracks all four metrics precisely because each one hides a failure the others can't see.
Part of the Measurements series. Start with the overview: how to measure a RAG system, then see the metrics in action across the RAG climb.
The short version
Retrieval Hit-Rate is a deterministic yes/no check for whether retrieval pulled at least one known-correct document, with no LLM judgment involved.
A blunt, un-foolable yes/no: did retrieval pull at least one of the documents we know the answer lives in?
💡 In one line: did you fetch any right doc at all?
Retrieval Hit-Rate is the odd one out on the RAG scorecard. Where recall, precision, faithfulness, and answer relevancy are all meaning-aware metrics judged by an LLM, hit-rate is a deterministic counter — a plain set intersection with no model in the loop. That is exactly why it exists: an LLM judge can be charmed by a fluent answer, but a set intersection cannot be talked into anything.
The intuition: "did we land at least one right doc?"
Hit-rate simply checks whether retrieval pulled in at least one of the expected source documents for a question, nothing more.
For every golden question we know, ahead of time, which source forms should be retrieved — the expected_docs. Hit-rate asks the bluntest possible question: did the retrieved set contain at least one of them?
It does not care how many, or where they ranked, or how good the answer was. One expected form in the pile = a hit. Zero = a miss. On our IRS corpus, if a question's answer lives in i1040gi and retrieval pulled anything but that, it's a miss — full stop.
How it's scored
Retrieval hit-rate is a simple, LLM-free set-intersection check scored 0 to 1, immune to persuasive but wrong answers.
It's computed right in the evaluation loop, no LLM involved:
if set(row.get("expected_docs", [])) & set(out.get("retrieved_docs", [])):
hits += 1
hit_rate = hits / total_questions
Scored 0..1 (the fraction of questions with at least one expected doc retrieved), higher is better. Because it's a raw set intersection, a persuasive but wrong answer cannot game it — which is the whole point of keeping it on the board next to the LLM-judged metrics.
Hit-rate vs Context Recall
Hit-rate checks if any expected document appeared, while recall measures what fraction of needed facts were actually retrieved.
These two look similar and are easy to confuse, so pin the difference:
- Hit-rate is all-or-nothing per question and document-level: did any expected doc show up? It's a coarse, cheap, deterministic smoke alarm.
- Recall is graded and fact-level: of all the facts the answer needs, what fraction were retrieved? It's the fine-grained, LLM-judged ceiling.
You can have a hit (one right doc retrieved) and still have mediocre recall (you missed three other facts the answer needed). Hit-rate tells you the door was open; recall tells you how much of the room you actually got into.
What good and bad look like
Naive RAG retrieval fails on half the questions, while better retrieval stages raise hit-rate to 0.68.
Naive RAG starts at 0.500 hit-rate on our corpus — half the questions never even retrieved a single expected form. That's a brutal floor, and it's the deterministic proof that the naive baseline's retrieval is genuinely broken, not just unlucky with the judge. As the retrieval stages improve (hybrid search, contextual retrieval), hit-rate climbs to 0.68 — fewer questions where the right document never made it into the room at all.
Why keep a blunt metric at all?
Pairing LLM-judged metrics with deterministic counters like hit-rate and latency prevents regressions from hiding behind a well-phrased but wrong answer.
Because the LLM-judged metrics, for all their nuance, share a single point of failure: the judge is itself an LLM. Pairing them with two deterministic counters — hit-rate and latency (end-to-end seconds) — means a real regression can't hide behind a charming answer. When the smart metrics and the dumb counters agree, you trust the direction. When they disagree, you've found something worth investigating.
Part of the Measurements series. See it alongside the meaning-aware metrics in how to measure a RAG system, and watch it move across the RAG climb.